The bumpy ride to the bump

19 September 2008: the LHC was without beam because of a transformer problem. The hardware commissioning team were finishing off powering tests of the main dipole magnet circuit in sector 3–4 when, at 11:18, an electrical fault resulted in considerable physical damage, the release of helium, and debris in a long section of the machine. In the control room, the alarms came swamping in. The cryogenics team grappled to make sense of what their systems were telling them, and there was frantic effort to interpret the data from the LHC’s quench protection system. I called LHC project leader Lyn Evans: “looks like we’ve got a serious problem here”.

Up to this point, 2008 had been non-stop but things were looking good. First circulating beam had been established nine days earlier in a blaze of publicity. Beam commissioning had started in earnest, and the rate of progress was catching some of us by surprise.

It is hard to describe how much of a body blow the sector 3–4 incident was to the community. In the following days, as the extent of the damage became clearer, I remember talking to Glyn Kirby of the magnet team and being aghast when he observed that “it’s going to take at least a year to fix”. He was, of course, right.

What followed was a truly remarkable effort by everyone involved. A total of 53 cryomagnets (39 dipoles and 14 quadrupoles) covering most of the affected 700 m-long zone were removed and brought to the surface for inspection, cleaning and repair or reuse. Most of the removed magnets were replaced by spares. All magnets whatever their origin had to undergo full functional tests before being installed.

Soot in the vacuum pipes, which had been found to extend beyond the zone of removed magnets, was cleared out using endoscopy and mechanical cleaning. The complete length of the beam pipes was inspected for contamination by flakes of multilayer insulation, which were removed by vacuum cleaning. About 100 plug-in modules installed in the magnet interconnects were replaced. 

Following an in-depth analysis of the root causes of the incident, and an understanding of the risks posed by the joints in the magnet interconnects, a new worst-case Maximum Credible Incident was adopted and a wide range of recommendations and mitigation measures were proposed and implemented. These included a major upgrade of the quench protection system, new helium pressure-release ports, and new longitudinal restraints for selected magnets. 

One major consequence of the 19 September incident was the decision to run at a lower-than-design energy until full consolidation of the joints had been performed – hence the adoption of an operational beam energy of 3.5 TeV for Run 1. Away from the immediate recovery, other accelerator teams took the opportunity to consolidate and improve controls, hardware systems, instrumentation, software and operational procedures. As CMS technical coordinator Austin Ball famously noted, come the 2009 restart, CMS, at least, was in an “unprecedented state of readiness”. 

Take two

Beam was circulated again on 20 November 2009. Progress thereafter was rapid. Collisions with stable-beam conditions were quickly established at 450 + 450 GeV, and a ramp to the maximum beam energy at the time (1.18 TeV, compared to the Tevatron’s 0.98 TeV) was successfully performed on 30 November. The first ramps were a lot of fun – there’s a lot going on behind the scenes, including compensation of significant field dynamics in the superconducting dipoles. Cue much relief when beam made it up the ramp for the first time. All beam-based systems were at least partially commissioned and LHC operations started a long process to master the control of a hugely complex machine. Following continued deployment of the upgraded quench protection system during the 2009 year-end technical stop, commissioning with beam started again in the new year. Progress was good, with first colliding beams at 3.5 + 3.5 TeV being established under the watchful eye of the media on 30 March 2010. With scheduled collisions delayed by two unsuccessful ramps, it was a gut-knotting experience in the control room. Nonetheless, we finally got there about three hours late. “Stable Beams” was declared, the odd beer was had, and we were off. 

Essentially 2010 was then devoted to commissioning and establishing confidence in operational procedures and the machine protection system, before starting to increase the number of bunches in the beam. In June the decision was taken to go for bunches with nominal population (~1.2 × 10¹¹ protons), which involved another extended commissioning period. Up to this point, in deference to machine-protection concerns, only around one fifth of the nominal bunch population had been used. To further increase the number of bunches, the move to a bunch separation of 150 ns was made and the crossing angle bumps spanning the experiments’ insertion regions were deployed. After a carefully phased increase in total intensity, the proton run finished with beams of 368 bunches of around 1.2 × 10¹¹ protons per bunch, and a peak luminosity of 2.1 × 10³² cm^–2s^–1.

Looking back, 2010 was a profoundly important year for a chastened and cautious accelerator sector. The energy stored in the magnets had demonstrated its destructive power, and it was clear from the start that the beam was to be treated with the utmost respect; safe exploitation of the machine was necessarily an underlying principle for all that followed. The LHC became magnetically and optically well understood (judged by the standards at the time – impressively surpassed in later years), and was stunningly magnetically reproducible. The performance of the collimation system was revelatory and accomplished its dual role of cleaning and protection impeccably throughout the full cycle. The injectors were doing a great job throughout in reliably providing high-intensity bunches with unforeseen low transverse emittances.

2010 finished with a switch from protons to operations with lead ions for the first time. Diligent preparation and the experience gained with protons allowed a rapid execution of the ion commissioning programme and Stable Beams for physics was declared on 7 November. 

Homing in 

The beam energy remained at 3.5 TeV in 2011, with the bunch spacing switched from 75 to 50 ns. A staged ramp in the number of bunches then took place up to a maximum of 1380 bunches, and performance was further increased by reducing the transverse size of the beams delivered by the injectors and by gently increasing the bunch population. The result was a peak luminosity of 2.4 × 10³³ cm^–2s^–1 and some healthy delivery rates that topped 90 pb^–1 in 24 hours. The next step-up in peak luminosity followed a reduction in the β^* parameter in ATLAS and CMS from 1.5 to 1 m (the transverse beam size at the interaction point is directly related to the value of β*). Along with further gentle increases in bunch population, this produced a peak luminosity of 3.8 × 10³³ cm^–2s^–1 – well beyond expectations at the start of the year. Coupled with a concerted effort to improve availability, the machine went on to deliver a total of around 5.6 fb^–1 for the year to both ATLAS and CMS. 

Meanwhile, excitement was building in the experiments. A colloquium at the end of 2011 showed a strengthening significance of an excess at around 125 GeV. The possible discovery of the Higgs boson in 2012 was recognised, and corresponding LHC running scenarios were discussed in depth – first at the Evian workshop (where we heard the plea from CMS spokesperson Guido Tonelli to “gimme 20” [inverse femtobarns]) and crystallised at the 2012 Chamonix workshop, where CERN Director-General Rolf Heuer stated: as a top priority the LHC machine must produce enough integrated luminosity to allow the ATLAS and CMS experiments an independent discovery of the Higgs before the start of long shutdown 1 (LS1). Soon after the workshop, Council president Michel Spiro sent a message to CERN’s member states: “After a brilliant year in 2011, 2012 should be historic, with either the discovery of the Standard Model Higgs boson or its exclusion.”

An important decision concerned the energy. A detailed risk evaluation concluded that the probability of a splice burn-out at 4 TeV per beam in 2012 was equal to, or less than, the probability that had been estimated in 2011 for 3.5 TeV per beam. The decision to run at 4 TeV helped in a number of ways: higher cross-sections for Higgs-boson production, reduced emittance and the possibility for a further reduction of β^*.

Discovery year 

And so 2012 was to be a production year at an increased beam energy of 4 TeV. The choice was made to continue to exploit 50 ns bunch spacing, which offered the advantages of less electron cloud and higher bunch charge compared with 25 ns, and to run with 1380 bunches. Based on the experience of 2011, it was also decided to operate with tight collimator settings, enabling a more aggressive squeeze to β^* = 0.6 m. The injectors continued to provide exceptional quality beam and routinely delivered 1.7 × 10¹¹ protons per bunch. The peak luminosity quickly rose to its maximum for the year, followed by determined and long running attempts to improve peak performance. Beam instabilities, although never debilitating, were a reoccurring problem and there were phases when they cut into operational efficiency. Nonetheless by the middle of the year another 6 fb^–1 had been delivered to both ATLAS and CMS. Combined with the 2011 dataset, this paved the way for the announcement of the Higgs-boson discovery. 

After a brilliant year in 2011, 2012 should be historic, with either the discovery of the Standard Model Higgs boson or its exclusion

2012 was a very long operational year and included the extension of the proton–proton run until December to allow the experiments to maximise their 4 TeV data before LS1. Integrated-luminosity rates were healthy at around 1 fb^–1 per week, and the total for the year came in at about 23 fb^–1 to both ATLAS and CMS. Run 1 finished with four weeks of proton–lead operations at the start of 2013.

It is impossible to do justice to the commitment and effort that went into establishing, and then maintaining, the complex operational performance of the LHC that underpinned the Higgs-boson discovery: RF, power converters, collimation, injection and beam-dump systems, vacuum, transverse feedback, machine protection, cryogenics, magnets, quench detection and protection, accelerator physics, beam instrumentation, beam-based feedbacks, controls, databases, software, survey, technical infrastructure, handling engineering, access, radiation protection plus material science, mechanical engineering, laboratory facilities … and the coordination of all that!