During two periods in summer and autumn 2006, the CMS collaboration took advantage of the near-complete assembly of the detector above ground to test its performance with cosmic rays, from support systems through to data-acquisition.
The strategy for building the CMS detector is unique among the four major experiments for the LHC at CERN. The collaboration decided from the beginning that assembling the large units of the detector would take place in a surface hall before lowering complete sections into the underground cavern. At the time the main driving factor was the attempt to cope with delivery of the underground cavern late in the schedule as a result of running the previous accelerator, LEP, together with civil-engineering works that were complicated by the geology of the terrain. Another goal was to minimize the large underground assembly operations, which would inevitably take more time and be more complex and risky in the confined space of the cavern. As construction and assembly progressed above ground, however, it became clear that there would a valuable opportunity for system integration and commissioning on the surface.
The complexity of CMS and the other LHC experiments is unprecedented. For this reason, the collaboration believed that the early combined operation of the various subsystems would be an important step towards a working experiment capable of taking data as soon as the LHC provides colliding beams. Initial plans focused on testing the state-of-the-art 4 T solenoid. This would require closing the yoke, already substantially instrumented with muon chambers. Since final elements of other subsystems would also be available by this stage, installed in their final locations, the idea of staging a combined system test in the surface hall became an attractive possibility.
Such a test also required the presence of the full magnet control system and scaled-down versions of the detector control, data-acquisition (DAQ) and safety systems. After much brainstorming and pragmatic criticism, the idea developed into the “cosmic challenge” for which the overall benchmark of success was the recording, and ultimate reconstruction, of cosmic-muon tracks passing through all sub-detectors simultaneously. This objective alone placed a big demand on the compatibility and interoperability of the sub-detectors, the magnet, the central DAQ, the control and monitoring systems and the offline software. The groups working on the Electromagnetic Calorimeter (ECAL) and the Tracker decided to find the resources to contribute active elements, rather than passive mechanical structures. This was a major factor in the positive feedback that eventually led virtually all systems, which will be needed to operate CMS in the LHC pilot run, to participate in the Magnet Test and Cosmic Challenge (MTCC).
In more detail, the objectives of the cosmic challenge were to: check closure tolerances, movement under a magnetic field, and the muon alignment system; check the field tolerance of yoke-mounted components; check the installation and cabling of the ECAL, the Hadron Calorimeter (HCAL), and Tracker inside the coil; test combined sub-detectors in 20° slice(s) of CMS with the magnet, using as near as possible final readout and auxiliary systems to check noise and interoperability; and last but not least, trigger and record cosmic muons and try out operational procedures.
In addition, the cosmic tests had to make no significant impact on progress in assembling the detector, and had to take place in the shadow of the work on commissioning and field-mapping the magnet. The tests also had to complement the trigger-system (high rate) tests taking place in the electronics-integration centre. Moreover, the aim was to use final systems as far as possible, that is with no (or very few) specific developments for the cosmic test. Another important aspect was to build a fully functional commissioning and operations team of experts from a collaboration that brings together more than 2000 people from laboratories worldwide, transcending linguistic and cultural backgrounds.
In order not to interfere with assembly work, electronics racks and control rooms for the tests were installed just outside the surface-assembly building in a large control barrack recovered from the OPAL experiment at LEP. Substantial investments were nonetheless needed in the surface hall, general and sub-system infrastructure, the triggering system, some temporary power supply systems, and in the tracker “slice” that was specially made for the cosmic challenge within a full replica of the final containment tube.
As the project progressed, the collaboration began to recognize its importance as a first test of intra-collaboration communication and remote participation, and the original scope expanded to include more substantial objectives for offline as well as online systems. A series of Run Workshops, culminating in a readiness review in June 2006, established the final objectives of the project. Weekly Run Meetings open to all CMS, eventually becoming daily, also ensured coordination. Ultimately the diligent work of hundreds of people aided by a little good fortune transformed the cosmic challenge into a cosmic success for CMS.
Four sub-detectors took part in the challenge. The silicon tracker system comprised 75 modules of the Tracker Inner Barrel in two partially populated layers, 24 modules of the Tracker Outer Barrel, and 34 modules of the Tracker Endcap system in two partially populated “petals”. By normal standards these 133 modules were a substantial system, comparable to any silicon detector used at LEP. It is worth remarking that this represents only 1% of the final CMS system, by far the largest ever built using silicon detectors. In addition, there were two barrel supermodules comprising 3400 lead tungstate crystals of the ECAL, or about 5% of the total; eight barrel sectors (22%), four endcap sectors (11%), and four sectors of the outer barrel section of the HCAL. For muon detection there were three (out of 60) muon barrel sectors, consisting of drift tube (DT) and resistive plate chambers (RPC), together with cathode strip chambers (CSCs) forming endcap muon chambers – in all, 8% of the total system.
As was the case for the sub-detectors, all common support systems were tested in close to final versions, using in most cases production hardware and software. The first priority was the definition and implementation of elements of the Detector Safety System. The teams had also to integrate sub-detectors with the central Detector Control System and a scaled-down version of the trigger system. The tests used the central DAQ with its final architecture and approximately 1% of the final computing power, and successfully operated the integrated run control, event builder, event filter, data storage and transfer to the CERN Advanced Storage manager (CASTOR). Throughout the whole exercise a fully functional event display enabled a simple and quick feedback on the status of different sub-detectors.
Other important organizational components of the operations were the consistent use of an electronic logbook, webcams, video-conferencing tools and Wiki-based documentation, as well as web-based monitoring, which was extensively tested. The challenge involved data transfers from Tier-0 (at CERN) to some Tier-1 centres (at CNAF/Bologna, PIC/Madrid and Fermilab) through the Physics Experiment Data Export (PHEDEX) protocol exercising the fast offline analysis and remote monitoring at the Meyrin CERN site as well as at the Fermilab Remote Operations Center.
There were two distinct phases of the cosmic challenge: the first phase in July and August 2006 was parasitic to the commissioning of the magnet. During this phase around 25 million “good” events were recorded with, at least, DT triggers and the ECAL and Tracker slice in the readout. Of these, 15 million events were at a stable magnetic field of at least 3.8 T – close to the maximum field of 4 T. A few thousand of the events corresponded to the benchmark where a cosmic ray was recorded in all four CMS sub-detector systems – Tracker, the ECAL, the HCAL and muon system – with nominal magnetic field. The image of the first of these events rapidly became a symbol of the success of the cosmic challenge and the demonstration of the CMS detector “as built”. During the challenge, data-taking efficiency reached more than 90% for extended periods. Data transfer to some Tier-1 centres, online event display, quasi-online analysis on the Meyrin site, and fast offline data-checking at Fermilab were some of the highlights of Phase I, which in this way offered a first taste of the full running experience. One example of an encouraging result was the good agreement between the predicted and measured cosmic muon spectra, both in momenta and angular distributions, using the new CMS software, CMSSW.
Phase II took place during October and November, after an efficiently executed “cosmic shutdown”, during which the tracker slice and the ECAL were removed and replaced with a field-mapper. While not as glamorous as the first phase, Phase II provided a wealth of solid information relevant to commissioning and operating CMS as an instrument for physics. For this phase, the team corrected and tested several minor faults found in Phase I in the magnet, detectors and central systems, and took more data with the HCAL and muon systems. Phase II recorded about 250 million events for studies of calibration, alignment and efficiency. The measurements made of the effect of the magnetic field on the response of the HCAL and on the drift paths in the muon barrel DTs were particularly crucial. Integration work on aspects of the trigger also allowed data to be recorded with some final systems.
Less than two weeks after the end of the magnet tests, the CMS detector was fully re-opened so the major elements could begin to be lowered into the experiment cavern. Meanwhile, work on analysing the millions of cosmic-ray events recorded in the cosmic challenge continues in many of the institutes in the collaboration. Now, as attention turns to completing the remaining assembly and installation of the muon, tracking and ECAL systems, the whole collaboration is looking forward eagerly, and with confidence, to re-assembling the detector underground and repeating the exciting and successful accomplishments of 2006, but this time with tracks from collisions of LHC beams.
