The longest journey: the LHC dipoles arrive on time

Nearly three years ago we celebrated the lift-off of the industrial production of the superconducting dipole magnets for CERN’s Large Hadron Collider (LHC), marked by the delivery of the first octant on 3 December 2003 (see CERN Courier January/February 2004 p30). It had taken about 10 years of R&D, models and prototyping to launch the first large call for tender, and a further five years of model refinement, industrialization and pre-series construction (the first 90 dipoles) to surpass a production rate of 10 dipoles a month in summer 2003. In contrast, it took only three years at the maximum rate to produce the remaining 90% of the dipoles. Indeed, since December 2005, the production has begun to slow and will finish well before the end of 2006, in perfect time for installation in the LHC ring.

Only four years ago success was far from certain. The preparation time for the largest and most complex hi-tech production ever tried in particle physics had been so long that scepticism was palpable. Moreover, the scheme devised by the LHC management – that CERN would supply the magnet manufacturers with all of the main components – seemed to many to be doomed to fail. One component or another would cause delays and/or technical problems, which would be charged to CERN. Now this story looks set to have a happy ending, and we should know for sure by November 2006.

The route to success

There have been five key ingredients for this success. First, we ensured adequate preparation and training of the companies during the industrialization phase. This was set up through pilot orders and technology transfer in the CERN Magnet Assembly Facility in Building 181, where about 25 dipole “cold masses” went through the final stages of assembly. It was preceded by a fairly long period of careful preparation and prototyping.

This first stage allowed detailed technical specifications to be produced. The dipoles were “build to print” and “build to process”, with only minor degrees of freedom left to the companies in certain areas, notably in the coil winding and pole assembly. This reflected a change in strategy from the start of the project, when CERN management was inclined to buy an almost turnkey product, where the supplier takes responsibility for how production is implemented. In the end, CERN retained a kind of intellectual property in the dipole project.

A third important point was that CERN procured all the main components for the magnet assembly (see figure 1 and “The incredible supply chain”). This put an additional burden on CERN’s shoulders, involving managing and taking responsibility for the interfaces; liability for delays, such that the famous just-in-time curves of the LHC “dashboard” might have been (and almost were in a few cases) a nightmare; additional workloads that were difficult to cope with; and transport, storage and logistics: we moved 120,000 tonnes of material around Europe, with five international road transport operations a day for more than four years.

Despite these difficulties, this procurement policy has proved to be to CERN’s advantage, and has probably been the cornerstone of our success. It meant that technical homogeneity was assured with contracts issued by one client. Quality assurance (QA) was guaranteed, with follow-up by the final client, that is, by the body most interested in the final product and the only one qualified to balance quality against realism as dictated by the schedule. The procurement policy also enabled economy of scale, through large contracts and the possibility of placing them ahead of the magnet-assembly contracts, which was critical for items requiring long lead times and technical uniformity, such as the superconducting cables and the low-carbon steel for the yoke. In addition, the procurement by CERN of many of the components has helped a balanced industrial return, a goal set by the CERN Council that has been met thanks to the Purchasing Service.

A fourth ingredient for success has been the continuous supervision by a team of CERN physicists, engineers and technicians, reinforced by external professional QA inspectors, who have been resident on the manufacturing sites. Many people in CERN’s Accelerator Technology (AT) department, as well as in the Accelerator Beams (AB) and Technical Support (TS) departments, have played key roles in measurement, analysis and validation of the work undertaken by industry.

Lastly, most of the major tooling for construction and monitoring was supplied under CERN’s responsibility. This allowed tooling development and procurement to carry on in parallel with development of the optimal assembly procedure for the dipoles. This constituted a risk, especially in the case of the big welding presses. However, this strategy, devised as early as 1998, allowed us to gain at least two years, allowing the LHC schedule to be met.

Industrial organization

CERN awarded the contract for manufacturing the LHC dipole magnets – or more precisely, what we call the cold masses, before they are installed in their cryostats at CERN and completed with other items, such as the beam screen – to three firms or consortia. These suppliers were the French Alstom MSA–Jeumont Areva consortium, the Italian company AS-G (previously Ansaldo Superconduttori Genova) and the German firm Babcock Noell (previously BNN). The tender process was difficult and lasted about three years (1999–

2001). Eventually, the companies agreed to lower substantially the tendered price, with no compromise on the technical quality. In exchange CERN had to take most of the risk and responsibility, except for manufacturing errors. The price reduction meant that there was no margin for big errors or for delays, which inevitably incur extra costs.

The three suppliers structured the work-flow, the production process and the logistics in different ways. These all worked well and led to no major difference in production flow.

The French consortium has taken care of cable insulation, coil fabrication and pole assembly on one site, Jeumont Areva, the pride of the French nuclear industry near the Belgian border. The poles are then shipped about 500 km south to Belfort in the historical Alstom factory where the French TGV high-speed train was born. Here, where Alstom also successfully produced about 40% of the precious LHC cable, the poles are put together in dipole assemblies and collared. The collared coils then have to pass critical tests based on HV electric integrity and magnetic measurements. The assembly of the cold mass is eventually completed on the same site with yoking, welding of the shells and curvature formation, corrector magnet mounting, finishing of the extremities with electrical connections among coils and bus bars, closure with the endcap, “collarette” and various flange welding, and final assessment of the curvature and geometry at the extremities. Once the final vacuum test has been passed, the cold mass is transported by special lorries – developed by TCT, winner of the 2002 “truck of the year” in France – along the 250 km route to CERN.

With the Italian company Ansaldo Superconduttori, cable insulation is done by a subcontractor, and then everything else is produced on site at Genoa. This is where the huge superconducting coils of the barrel toroid magnet for the ATLAS detector and the giant solenoid for the CMS detector were also manufactured. For the LHC cold masses, every step takes place from coil winding to the last geometrical laser track control, before the cold mass is shipped to CERN, about 400 km away. In total half of the magnetic energy that will be stored in the LHC tunnel and in the detectors will come from magnets constructed in Genoa.

The German company Babcock Noell split production between two sites. The first stage, up to the production of the collared coil and magnetic measurements, took place in Würzburg, near the headquarters, while the cold-mass assembly was done some 300 km east in Zeitz, near Leipzig, in a renovated building previously used for tank repair and maintenance for the Soviet Union army in the former East Germany. The finished cold masses travel 1000 km across Germany and part of France to CERN.

Industrial production

It is interesting to analyse the industrial approach to managing the production, in particular as applied by Babcock Noell, which had already finished its supply of cold masses in November 2005. This company, in principle less experienced than the other two suppliers in such production techniques, first conducted a careful study, in collaboration with the University of Hanover, to evaluate the whole sequence of operations that are necessary to manufacture a dipole cold mass. The study indicated the number of assembly lines and people needed for each operation (or post) on the base of two shifts, five days a week, as a function of production rate. This work allowed the sensitivity of each working post to be assessed with respect to tooling failure or the necessity to ramp up beyond the nominal rate to recover from a stoppage. Not surprisingly, it singled out the winding stage as the ultimate bottleneck in the overall production.

Taking advantage of the fact that in Germany workers can be hired for projects for up to five years, while avoiding high social payments on laying off after the project, Babcock Noell also decided to take on about 30–40% more employees than the other two firms, which allowed them to carry out the production less time than the other suppliers. The company has consistently delivered 3.5–4 dipoles a week, while a rate of 3–3.3 would have been enough to meet the contractual obligation. In this way the firm lowered its general expenses and used tooling and manpower more intensively. From CERN’s point of view this helped to compensate for a general delay in the pre-series production. However, we could not have sustained such an advanced schedule from all three producers; we would have become short of components, with the serious risk of having to pay extra costs for work stoppages: in 2005, at the maximum production rate, we had more than 400 people working at the three suppliers.

Figure 2 shows the “learning” curves for the production at Babcock Noell. It is remarkable how close to the forecast the actual production rate was. The small margin gained with respect to the target was used to correct inevitable errors and mistakes that were generated by manufacturing or by faulty components. Figure 2a highlights the most critical part, the coil winding and curing for the pre-series of 30 magnets; the singularities are clearly shown, while they are hardly visible in the global production plot in figure 2b. Reaching the goal is satisfying for the companies and also for CERN. It confirms that such a technically complex object can be made ready for industrial production.

Figure 3 shows the learning curves of another producer, Jeumont, in coil winding, curing and assembly. In this case the company continues learning and at each injection of new personnel there is a “need” for learning. For all three suppliers, although to different degrees, the earliest part of the production process proved to be the most critical and more subject to mistakes and unexpected events.

CERN has also carried out a study to compare the dipole production with other projects in terms of industrial learning curves. Based on well known industrial production theory, the data can be fitted with certain models (typically based on power curves) and used to determine the learning percentage, as in figure 4 for the dipole production. Compared with other production processes, the learning percentage for the LHC dipole production lies, not surprisingly, between shipbuilding and aerospace production.

Production and QA

In considering QA, we can take the example of the collared coils and the detection of hidden defects, in which magnetic measurements at the suppliers at room temperature (with a current of only 8 A, that is, at a 5 mT field) played a primary role. CERN provided the tooling and the know-how and, once the pre-series was completed, passed the job of measuring to industry for the series production. Each measurement has been analysed almost online at CERN, with a commitment to giving an answer normally within two hours, exceptionally within a day, if the magnet is good. If an anomaly is detected, a thorough analysis is carried out to establish if the magnet can proceed or if it must be disassembled, a decision that is costly and painful because of the interruption to the tight production cycle.

Figure 5 show the total number of collared coils that have been disassembled versus the magnet fabrication number. At the start of production, as expected, there were a number of rejects. Then after a period of good production, a defective series appeared when production was almost stable. After investigation these defects were traced to small details and procedures that did not have sufficient margin to accommodate small mistakes once the slow and strictly monitored early production gave way to mass production, which also involved huge recruitment and training of new staff. Indeed in the mass production the experienced technicians, trained for years during the prototyping period, became the supervisors of newly trained colleagues; the reality is that in series production the best workers are not active players.

A project like this requires pragmatism to focus on the main objective: producing a magnet of sufficient quality in the given time, rather than continuously improving. As stated in the first external review of superconducting cable and magnet production, instituted in autumn 2001 and composed by a panel of international scientists and engineers, “the best is the enemy of the good”. We made this the guiding star of the whole production process. Following other good advice from the review, a thorough audit of QA was carried out on the manufacturing sites; the very low rate of mistakes in the later production (see figure 5) is also a result of these high-quality audits.

Delivery and performance

It is worth remembering that at the call for tender for the series in 2001 – 386 dipole cold masses for each of the three producers – the bidders offered their minimum price, reduced by more than 20%, with a delivery for November 2006, even if the contract demanded it for June 2006 to provide some contingency. Even then, November 2006 was already considered the latest date to respect the LHC schedule. This goal has not shifted, showing that the project, despite the problems encountered in other areas, has not been delayed significantly since the end of 2001. Babcock Noell has completed production seven months early and the other two suppliers are projected to have delivered the dipoles for the tunnel (1232 in total) by October 2006, in line with the LHC schedule.

The performances of the magnets can be qualified to a good extent by three parameters. The first is quench behaviour, that is, the irreversible loss of the superconducting state. Figure 6 shows the number of quenches that are needed to pass the nominal field of 8.3 T at the second thermal cycle. (The magnets will see at least three thermal cycles before they see particle beams.) The number passing is very encouraging; based on this and on the results for a few weak magnets that were tested three or four times, we anticipate that dipole quenches will take up only around 10–15 days during the hardware commissioning, and a negligible time during beam operation.

Regarding the magnetic field, the most important quantity to monitor is the uniformity of bending strength between dipoles, as all of the magnets in a sector of the LHC will be powered in series. Figure 7 shows the exceptionally satisfactory results for the coils already measured (95% of the whole production). Based on these results, magnets from different manufacturers can be mixed, allowing us to drop a constraint that would have made installation even more difficult than it is today.

A third important check concerns the geometry of the magnets. The dipoles are bent to follow the beam trajectory and minimize coil aperture and cost. Each dipole cold mass has a sagitta of 9 mm over its 15 m length and must be very precise at the extremities, where corrector magnets are positioned. This is not easy on equipment that weighs 30 tonnes, has a laminated construction, and the shape of which is determined by friction and welding shrinkage. It required a great deal of supervision in industry and at CERN (where the cold masses are inserted in their cryostats and many other operations preformed), and good collaboration among different teams in the AT, AB and TS departments. Figure 8 shows the actual positions of the sextupole corrector magnets.

Acknowledgements

The production of the LHC dipoles involved hundreds of people at CERN and about 1200 people in industry all around Europe and worldwide. It has been a hi-tech achievement that has shown the capacity of high-energy physics and of CERN in particular to achieve large and difficult industrial projects on time.