Whenever we perform an analysis of our data, whether measuring a physical quantity of interest or testing some hypothesis, it is necessary to assess the accuracy of our result. Statistical uncertainties arise from the limited accuracy with which we can measure anything, or from the natural Poisson fluctuations involved in counting independent events. They have the property that repeated measurements result in greater accuracy.
Systematic uncertainties, on the other hand, arise from many sources and may not cause a spread in results when experiments are repeated, but merely shift them away from the true value. Accumulating more data usually does not reduce the magnitude of a systematic effect. As a result, estimating systematic uncertainties typically requires much more effort than for statistical ones, and more personal judgement and skill is involved. Furthermore, statistical uncertainties between different analyses usually are independent; this often is not so for systematics.
The November event saw the largest number of statisticians at any PHYSTAT meeting
In particle-physics analyses, many systematics are related to detector and analysis effects. Examples include trigger efficiency; jet energy scale and resolution; identification of different particle types; and the strength of backgrounds and their distributions. There are also theoretical uncertainties which, as well as affecting predicted values for comparison with measured ones, can also influence the experimental variables extracted from the data. Another systematic comes from the intensity of accelerator beams (the integrated luminosity at the LHC for example), which is likely to be correlated for the various measurements made using the same beams.
At the LHC, it is in analyses with large amounts of data where systematics are likely to be most relevant. For example, a measurement of the mass of the W boson published by the ATLAS collaboration in 2018, based on a sample of 14 million W-boson decays, had a statistical uncertainty of 7 MeV but a systematic uncertainty of 18 MeV.
PHYSTAT-Systematics
Two big issues for systematics are how the magnitudes of the different sources are estimated, and how they are then incorporated in the analysis. The PHYSTAT-Systematics meeting concentrated on the latter, as it was thought that this was more likely to benefit from the presence of statisticians – a powerful feature of the PHYSTAT series, which started at CERN in 2000.
The 20 talks fell into three categories. The first were those devoted to analyses in different particle-physics areas: the LHC experiments; neutrino-oscillation experiments; dark-matter searches; and flavour physics. A large amount of relevant information was discussed, with interesting differences in the separate sub-fields of particle physics. For example, in dark-matter searches, upper limits sometimes are set using Yellin’s Maximum Gap method when the expected background is low, or by using Power Constrained Limits, whereas these tend not to be used in other contexts.
The second group followed themes: theoretical systematics; unfolding; mis-modelling; an appeal for experiments to publish their likelihood functions; and some of the many aspects that arise in using machine learning (where the machine-learning process itself can result in a systematic, and the increased precision of a result should not be at the expense of accuracy).
Finally, there was a series of talks and responses by statisticians. The November event saw the largest number of statisticians at any PHYSTAT meeting, and the efforts that they made to understand our intricate analyses and the statistical procedures that we use were much appreciated. It was valuable to have insights from a different viewpoint on the largely experimental talks. David van Dyk, for instance, emphasised the conceptual and practical differences between simply using the result of a subsidiary experiment’s estimate of a systematic to assess its effect on a result, and using the combined likelihood function for the main and the subsidiary measurements. Also, in response to talks about flavour physics and neutrino-oscillation experiments, attention was drawn to the growing impact in cosmology of non-parametric, likelihood-free (simulation-based likelihoods) and Bayesian methods. Likelihood-free methods came up again in response to a modelling talk based on LHC-experiment analyses, and the role of risk estimation was emphasised by statisticians. Such suggestions for alternative statistical strategies open the door to further discussions about the merits of new ideas in particular contexts.
A novel feature of this remote meeting was that the summary talks were held a week later, to give speakers Nick Wardle and Sara Algeri more time. In her presentation, Algeri, a statistician, called for improved interaction between physicists and statisticians in dealing with these interesting issues.
Overall, the meeting was a good step on the path towards having a systematic approach to systematics. Systematics is an immense topic, and it was clear that one meeting spread over four afternoons was not going to solve all the issues. Ongoing PHYSTAT activities are therefore planned, and the organisers welcome further suggestions.
