Physicists and statisticians met in Banff for the latest PHYSTAT workshop, where they discussed how to tackle upper limits, significance and separating signal from background.
When analysing data from particle-physics experiments, the best statistical techniques can produce a better quality result. Given that statistical computations are not expensive, while accelerators and detectors are, it is clearly worthwhile investing some effort in the former. The PHYSTAT series of conferences and workshops, which started at CERN in January 2000, has been devoted to just this topic (CERN Courier May 2000 p17). The latest workshop was at Banff in the Canadian Rockies in July and was also a culminating part of the spring 2006 programme of astrostatistics which had taken place earlier in the year at the Statistical and Applied Mathematical Sciences Institute (SAMSI) in the Research Triangle Park, North Carolina.
The initiative for the Banff workshop came from Nancy Reid, a statistician from Toronto who has delivered invited talks at the PHYSTAT conferences at SLAC and Oxford (see CERN Courier March 2004 p22 and (see CERN Courier January/February 2006 p35). The Banff International Research Station sponsors workshops on a variety of mathematical topics, including statistics. The setting for these meetings is the Banff Center, an island of tranquillity and vigorous intellectual activity in the town of Banff. Most of the activities at the centre are in the arts, but science and mathematics are found there too.
Thirty-three people attended the workshop, of whom 13 were statisticians, the remainder being mostly experimental particle physicists, with astrophysicists making up the total. It concentrated on three specific topics: upper limits, in situations where there are systematic effects (nuisance parameters); assessing the significance of possible new effects, in the presence of nuisance parameters; and the separation of events that are signal from those that are caused by background, a classification process that is required in almost every statistical analysis in high-energy physics. For each of these topics there were two coordinators, a physicist and a statistician.
The three topics, of course, interact with each other. Searches for new physics will result in an upper limit when little or no effect is seen, but will need a significance calculation when a discovery is claimed. The multivariate techniques are generally used to provide the enriched subsample of data on which these searches are performed. Just as for limits or significance, nuisance parameters can be important in multivariate separation methods.
As this was a workshop, the organizers encouraged participants to be active in the weeks before the meeting. Reading material was circulated as well as some simulated data, on which participants could run computer programmes that incorporated their favourite algorithms. This enabled all participants to become familiar with the basic issues before the start of the meeting. The workshop began with introductory talks on particle physics and typical statistical analyses, and Monte Carlo simulations in high-energy physics. These primarily described for statisticians the terminology, the sort of physics issues that we try to investigate in experimental particle physics, what the statistical problems are and how we currently cope with them, and so on.
Jim Linnemann of Michigan State University publicized a new website, www.phystat.org, which provides a repository for software that is useful in statistical calculations for physics. Everyone is encouraged to contribute suitable software, which can range from packages that are suitable for general use, to the code that is specifically used in preparing a physics publication.
The convenors of the various subgroups led the remaining sessions on the first day. Very few talks were scheduled for subsequent days, specifically to leave plenty of time for discussions and for summary talks and to provide an opportunity for exploring fundamental issues.
Limits and significance
The discussion about limits ranged from Bayesian techniques, via profile likelihood to pure frequentist methods. Statisticians made the interesting suggestion that hierarchical Bayes might be a good approach for a search for new physics in several related physics channels. There was a lively discussion about the relative merits of the possible approaches, and about the relevant criteria for the comparison. After a late evening session, it was decided that data would be made available by the limits convenor, Joel Heinrich of the University of Pennsylvania, so that participants could try out their favourite methods, and Heinrich would compare the results. This work is expected to continue until November.
Discussions considered the significance issue within particle physics, with several other examples in astrophysics. Indeed it arises in a range of subjects in which anomalous effects are sought. Luc Demortier of Rockefeller University in New York, the physics convenor on significance, detailed eight ways in which nuisance parameters can be incorporated into these calculations, and discussed their performance. This will be a crucial issue for new particle searches at the Large Hadron Collider at CERN, where the exciting discoveries that may be made include the Higgs boson, supersymmetric particles, leptoquarks, pentaquarks, free quarks or magnetic monopoles, extra spatial dimensions, technicolour, the substructure of quarks and/or leptons, mini black holes, and so on. In all cases some of the backgrounds will be known only approximately and it will be necessary to distinguish among peaks that are merely statistical fluctuations, errors and genuine signals of new physics.
Demortier also addressed the issues of whether it is possible to assess the significance of an interesting effect, which is obtained by physicists adjusting selection procedures while looking at the data; and why particle physics usually demands the equivalent of a 5 σ fluctuation of the background before claiming a new discovery. (The probability of obtaining such a large fluctuation by chance is less than one part in a million.)
Signal or background?
The sessions on multivariate signal–background separation resulted in positive discussions between physicists and statisticians. Byron Roe of the University of Michigan explained the various techniques that are used for separating signal from background. He described how for the MiniBooNE experiment at Fermilab, Monte Carlo studies showed that boosted decision trees yielded good separation, and coped with more than 100 input variables. An important issue concerned assessing the effect on the physical result, in this case neutrino oscillation parameters, of possible systematic effects. One of the conventional methods for doing this is to vary each possible systematic effect by one standard deviation, and to see how much this affects the result; the different sources are then combined. Roe pointed out that there is much to recommend an alternative procedure, which investigated the effect on the result of simultaneously varying all possible systematic sources at random.
Radford Neal, a statistician from Toronto University, took up this theme in more detail, and also emphasized the need for any statistical procedure to be robust against possible uncertainties on its input assumptions. One of Neal’s favourite methods uses Bayesian neural nets. He also described graphical methods for showing which of the input variables were most useful in providing the separation of signal and background.
Ilya Narsky of Caltech gave a survey of the various packages that exist for performing signal–background separation, including R, WEKA, MATLAB, SAS, S+ and his own StatPatternRecognition. Narsky suggested that the criteria for judging the usefulness of such packages should include versatility, ease of implementation, documentation, speed, size and graphics capabilities. Berkeley statistician Nicolai Meinshausen gave a useful demonstration of the statistical possibilities within R.
The general discussion in this sub-group covered topics such as the identification of variables that were less useful, and whether to remove them by hand or in the programme; the optimal approach when there are several different sources of background; the treatment of categorical variables; and how to compare the different techniques. This last issue was addressed by a small group of participants working one evening using several different classifiers on a common simulated data set. Clearly there was not the time to optimize the adjustable parameters for each classification method, but it was illuminating to see how quickly a new approach could be used and comparative performance figures produced. Reinhard Schweinhorst of Michigan State University then presented the results.
As far as the workshop as a whole was concerned, it was widely agreed that it was extremely useful having statisticians present to discuss new techniques, to explain old ones and to point out where improvements could be made in analyses. It was noted, however, that while astrophysics has been successful in involving statisticians in their analyses to the extent that their names appear on experimental papers, this is usually not the case in particle physics.
Several reasons for this have been suggested. One is that statisticians enjoy analysing real data, with its interesting problems. Experimental collaborations in particle physics tend to be very jealous about their data, however, and are unwilling to share it with anyone outside the collaboration until it is too old to be interesting. This results in particle physicists asking statisticians only very general questions, which the statisticians regard as unchallenging. If we really do want better help from statisticians, we have to be prepared to be far more generous in what we are ready to share with them. A second issue might be that in other fields scientists are prepared to provide financial support to a statistics post-doc to devote his/her time and skills to helping analyse the data. In particle physics this is, at present, very unusual.
There was unanimous agreement among attendees that the Banff meeting had been stimulating and useful. The inspiring location and environment undoubtedly contributed to the dynamic interaction of participants. The sessions were the scene of vigorous and enlightening discussion, and the work continued late into the evenings, with many participants learning new techniques to take back with them to their analyses. There was real progress in understanding practical issues that are involved in the three topics discussed, and everyone agreed that it would be useful and enjoyable to return to Banff for another workshop.
Further reading
The most recent PHYSTAT conference was in Oxford (see www.physics.ox.ac.uk/phystat05/, which has links to the earlier meetings). Details about the Banff meeting are at www.pims.math.ca/birs/birspages.php?task=displayevent&event_id=06w5054.
