Vinod Chohan looks back at the results of a unique collaboration.
The cold testing of 1706 superconducting magnets for the LHC came to a successful completion early this year. This important milestone for the project marked the end of an operation that had begun in 2001, meeting considerable challenges along the way. By the end of 2003 only 95 dipole magnets had been tested, but the effort and innovative ideas that came from the Operations Team enabled the team eventually to meet the target. The majority of the personnel for the tests came from India, for a year at a time, as part of the CERN–India Collaboration for the LHC. Their success provides a unique example of international collaboration in the accelerator domain on an unprecedented scale.
The LHC consists of two interleaved synchrotron rings, 26.7 km in circumference. The main elements of the rings are the 2-in_1 superconducting dipole and quadrupole magnets operating in superfluid helium at 1.9 K. The total number of cryogenic magnet assemblies – or cryomagnets – includes 1232 dipoles with correctors, 360 short straight sections (SSS) for the arcs with quadrupoles and integrated high-order poles, and 114 special SSS for the insertion regions (IR-SSS) with magnets for matching and dispersion suppression. All of these magnets had to be tested at low temperatures before they could be installed in the tunnel, and for this purpose a superconducting magnet test facility, equipped with 12 test benches and the necessary cryogenic infrastructure, was set up in building SM18 just across the border in France from CERN’s Meyrin site.
The magnet testing had several aspects. For each magnet the tests had to verify the integrity of the cryogenics, mechanics and electrical insulation; qualify the performance of the protection systems; train the magnet up to the nominal field or higher; characterize the field; ensure that the magnet met the design criteria; and finally accept the magnet according to its performance in quenches and in training. The workforce to do this consisted of three main teams – the Operation Team, who performed the tests and measurements, supported by the Cryogenics Team and the Magnet Connect/Disconnect Team (known as ICS). In addition, a team known as Equipment Support looked after improvements and on-call trouble-shooting of hardware and software, and a sub-team of ICS handled the movement of the magnets with a remote-controlled vehicle (the ROCLA).
The complexity of the magnets implied a high level of complexity in the test facility, which required its own specific infrastructure, from cryogenic feed boxes and high-current circuits to data acquisition for measurement and control. Assembling the full facility involved several groups from CERN’s AB, AT and TS departments working over many years, with the final test bench commissioned in June 2004.
The first testing of series production magnets began in 2001, with two test benches and a limited cryogenic infrastructure. Undertaken by specialists in the magnets and the related equipment, the operation at this stage was more laboratory R&D than a well-defined, structured approach to the test procedure. The first sets of dipoles, comprising around 30 samples from each of the three suppliers, had to be thoroughly tested, with full magnetic and other measurements. This extensive testing, together with the limited operational experience and support tools, meant that some 20–30 days were required to test a magnet during 2001–2002, and only 21 magnets were tested during this period.
To increase throughput, the test facility began to operate round the clock early in 2003. With a final set-up of 12 test benches and a minimum of 4 people a shift, this required a minimum team of 24. The initial plan had been to outsource, but by early 2002 it was clear that this was no longer an option, and also that only a few non-expert CERN staff were available to run the test facility. It was at this time that the Department of Atomic Energy (DAE), India, offered technical human resources for SM18. A collaboration agreement between India and CERN had been in place since the 1990s, including a 10 man-year arrangement for tests and measurements during the magnet prototyping phase. This eventually allowed more than 90 qualified personnel from four different Indian establishments to participate in the magnet tests on a one-year rotational basis (a condition requested by India) starting around 2002.
It was also clear that proper strategies and support tools were needed to meet the target of testing all magnets by the end of 2006. In addition to several reviews aimed at streamlining the process, an extensive study took place to define a selective, reduced set of magnetic measurements needed to qualify and accept a magnet.
A testing renaissance
The overall magnet-test operation involved significant manual effort, and while this remained the case throughout the tests, from mid-2003 the operational process underwent a renaissance, from basic manual data-logging to an efficient, sophisticated and highly automated test-management system. The Operation Team, and in particular the Indian personnel, provided essential feedback for framing new strategies and made significant proposals for improving the throughput. In addition, CERN’s web-based network backbone and computer facilities were widely used to develop supporting tools.
A “to-do list” of the minimum set of tests for a magnet was created, with methods developed to reduce human error in the process as much as possible. The list of tests was reflected in templates for a magnet test report. A new website included all of the important documentation, from manuals and templates to troubleshooting procedures and the shift plan. This proved immensely helpful in training new staff as well as in managing the daily activities.
A web-based system called the SM18 Test Management System (SMTMS), based on the to-do list, was developed to generate test-sequences and reports automatically and to store all of the relevant data. This enabled fast, reliable and error-free generation of crucial data. It also made it possible to keep track of the times taken for various phases of the tests, and everyone concerned could keep track of the tests from different locations both within CERN and further afield. An electronic log-book approach, using the network backbone at CERN, ensured easy access and helped to categorize and record faults that occurred during the tests.
Another web-based tool, the e-traveller, ensured a smooth interaction between the teams during setting up and at the end of the tests for a particular magnet. This tool informed the relevant teams about the need for their services on the magnet, using mobile-phone alerts in the appropriate language. This helped the Indian personnel to overcome difficulties in verbal communication with the exclusively French-speaking teams, while maintaining the work rhythm, as well as automatically recording the phases in the tests.
The SM18 community celebrated the Hindu festival of Diwali with a party in October 2006, and lit candles to represent the remaining magnets still to be tested. Diwali takes place on the date of the new moon, between the months of Asvina and Kartika on the Hindu calendar (usually in October or November).
To fulfil their roles at CERN, the technical engineers, all specialists in their own fields, had to take a leave of absence from their regular jobs. For example, Praveen Deshpande who was at CERN in 2006 designs instruments for accelerators and lasers, while Sampathkumar Raghunathan and Charudatta Kulkarni work on the R&D of nuclear instrumentation. For Deshpande, the best part about working at CERN was the interaction with different people. “I’m exposed to many people and agencies here, which I don’t get at home working in a small design section,” he explained in an interview for the CERN Bulletin. The logistics of implementing large-scale projects at CERN proved an eye opener. “The template approach with documentation and foresight is important in a big job like this,” said Raghunathan. Kulkarni agreed, and further identified excellent leadership skills as an important lesson that he learned.
The technical engineers could also use their free time to further their own knowledge. “Everyone was given the name of someone at CERN who works in a similar field to contact, if we wished. This is for our intellectual development. Because we work in R&D, it is important that we are up to date with new developments, so we are not left behind when we return a year later,” explained Kulkarni.
On their days off, the SM18 community organized group visits to tourist destinations, celebrated Indian festivals, such as Diwali, and even formed teams for cricket matches. Many of the technical engineers brought their families over, and some of their children attended local schools. For example, Raghunathan’s wife, two children and his mother moved to Switzerland for nine months. His seven-year-old daughter attended a French school for a full academic year. “They all enjoyed living here. My daughter also learned a lot of French. It’s an added asset.”
At the end of their year at CERN, when the technical engineers returned home, enriched by their experiences, they hoped to incorporate what they had learnt at CERN into their work, in particular the methods of coordinating and managing large-scale projects. However, the experience gained extends far beyond a professional level. The magnet test has facilitated international friendships and even reunited lost ones from home. Deshpande met friends that he knew from 13 years ago when he was undergoing training, but with whom he had since lost touch, as well as colleagues at the same establishment in India whom he has never met owing to the size of the organization.
• Extract from the CERN Bulletin http://bulletin.cern.ch/eng/bulletin.php?bullno=48/2006.
To attain a high throughput it was also necessary to reduce the time and cryogenic resources taken in testing the magnets, including the “training” required to reach the operational magnetic field. In this, the current is increased until the magnet quenches (reverts from its superconducting state to a normally conducting state) and then the process is repeated. In the early stages, each dipole was trained to reach a field about 8% higher than required for LHC operation – a major time-consuming activity. During 2003, the Operation Team observed that the majority of magnets cross their nominal field (8.33 T or 11,850 A) on the second attempt, and that not much additional information on the quality of a magnet came from a third quench or more. This led to a “two-quench rule” being agreed by the magnet experts, in which a magnet was accepted after two quenches providing it crossed the nominal field by a small margin. Later a “three-quench rule” allowed a magnet to be accepted if it had failed the two-quench rule, but crossed a field of 8.6 T (12,250 A) in the third quench. This strategy drastically reduced the overall time for the cold tests.
Another important step towards reducing the overall test time was the introduction of a rapid on-bench thermal cycle for magnets that had a poor performance in the first run. Further time-saving came from the round-the-clock decision-making on the performance of a magnet by the operator, based on the results in the Magnet Appraisal and Performance Sheet, provided by the web-based SMTMS.
Figure 1 shows the cumulative number of magnet tests, including repeats, since 2003, both for dipoles and for SSS. While throughput was low until the end of 2003, it increased sharply after the introduction of the new tools and strategies. The flat regions at the end of each year are due to the annual shutdown of the cryogenic infrastructure, typically for seven weeks. More details are shown in table 1.
Testing the SSS magnets was a challenging task until the end of 2004, when all of the necessary information had finally been gathered and collated. The special IR-SSS magnets were even more of a challenge as they have a wide variety of types, structures and temperature regimes, and required the collection of a large amount of information for the tests. Each of the 114 magnets needed its own dedicated to-do list. As the table shows, the majority of the special SSS magnets were tested in 2006, together with significant numbers of standard SSS and dipole magnets, marking altogether a remarkable achievement for the year. While delays in delivery of the magnets to SM18 meant that not all of the magnets had been tested by the end of 2006, the target was achieved only a few weeks later by 23 February 2007.
Around 9% of the dipoles required the test procedure to be repeated, with repeat rates of a little over 12% for the SSS and IR-SSS magnets. In addition, some 3% of dipoles and 6% of the SSS had to be repaired or were rejected after the cold tests. These results alone justify the effort required for testing all of the magnets under the real cryogenic conditions. Moreover, the successful completion of this huge operation has been a unique example of international collaboration on an unprecedented scale in the accelerator domain.
