Résumé
Les réseaux face aux défis du LHC
On a beaucoup parlé des capacités de stockage et de calcul de la Grille pour les expériences au LHC. Cependant, elles resteraient inopérantes sans l'infrastructure des réseaux qui permettent de transporter les données des expériences aux instituts des collaborations, puis aux physiciens. Tous les instituts reliés 7agrave; l'infrastructure de la Grille de calcul du LHC doivent communiquer et la réseautique doit assurer le trafic à travers les différents réseaux et domaines. Cet article présente brièvement certaines des infrastructures de réseau et leurs relations avec le projet LHC. Toutes les activités sont nécessairement menées à une échelle mondiale.
It is well known that the Grid will provide storage and computing for the experiments at the Large Hadron Collider (LHC), which will generate vast amounts of data. However, this would not work without a networking infrastructure to move data from the experiments to the collaborating institutes and then to the physicists. The LHC Computing Grid (LCG) infrastructure is built on the concept of tiers, with different institutes providing certain services. Briefly, CERN is Tier-0, while Tier-1 institutes are responsible for long-term data storage services, and Tier-2 centres mainly provide CPU and temporary storage services. All institutes connected to the LCG infrastructure have to communicate with each other, so the networking has to deal with traffic crossing a number of networks and network-management domains.
The end-to-end path between a scientist somewhere in the world and the data coming from the detectors comprises the on-site campus network infrastructures as well as the connectivity between the experiments, the CERN Computer Centre and the Tier-1 and Tier-2 centres. This connectivity is provided on infrastructures of various types, and different international collaborative initiatives contribute to a dynamically evolving situation.
The CERN network
The CERN network itself consists of a number of interconnected infrastructures that provide all the connectivity to user desktops, experimental areas, computer-centre farms and the collaborating institutes around the world. The core of the CERN LCG network architecture is based on a set of highly redundant and high-throughput routers interconnected with multiple 10 Gbit/s connections, using a mesh of Force-10 routers, each equipped with 40 10-Gbit/s interfaces.
The local computing farm at CERN, which includes the CERN part of the LCG, is expected to grow to around 6000 processors and about 2000 tape and disk storage devices, and each of these systems is connected to the LCG farm core using 1 Gbit/s Ethernet. This farm core is in turn connected via a dedicated network infrastructure to the Tier-1 centres.
Desktop computers at CERN are connected to the campus core infrastructure using these highly available and redundant routers with 10 Gbit/s links. This campus core is connected both to the LCG farm core and the external network infrastructure to allow complete connectivity of CERN desktops to the LHC physics computing and the worldwide network infrastructures.
The edge of this campus network is being upgraded with 1 Gbit/s switches from HP ProCurve. It is worth comparing this with the situation in 2000, at the end of the era of the Large Electron-Positron collider, where the equivalent core had 100 Mbit/s connectivity, and the edge 10 Mbit/s.
Tier-0 to Tier-1
A prominent networking goal that has been set is to achieve aggregate data flows from CERN to the collaborating institutes of several gigabytes a second on a continuous basis by the time the LHC is up and running. The planned starting date for production traffic on the LHC networking infrastructure is summer 2007, but the links will be tested at full bandwidth during 2006 as part of the service challenges that are now under way. Already, peaks of 1 GB/s have been achieved (see CERN Courier April 2006 p15).
The resources available at the Tier-1 centres are not all the same and therefore the average network load is expected to vary. However, the network should be able to sustain the peak loads at any given time, so as a starting point at least one dedicated 10 Gbit/s transmission path between CERN and each Tier-1 will be recommended. This network is based on permanent 10 Gbit/s light paths, which form a so-called optical private network for the LCG.
The infrastructure that transports the data from CERN to a particular Tier-1 depends on the location of the Tier-1. In general, a combination of national networks, international projects and commercial links are used (see table, p21). The responsibility for providing network equipment, physical connectivity and manpower is distributed among the co-operating parties. CERN provides the interfaces to each Tier-1 link termination point at CERN.
Tier-1 to Tier-2
The Tier-1 centres are connected with CERN through dedicated links to ensure high reliability and high-bandwidth data exchange, but they are also connected to many research networks and to the Internet, a worldwide network providing Internet Protocol communication between computers at research institutions. These infrastructures ensure good connectivity between Tier-2s and Tier-1s, as well as Tier-1 to Tier-1 communication.
As with the general-purpose Internet, the research networks are in fact a set of interconnected networks that link together the national and international networking initiatives through bi-lateral agreements. In Europe, for example, GEANT links together the national research networks of European countries and provides peering with other networks, for example Abeline in the US. The Energy Sciences Network (ESNet) provides a backbone in the US, linking metropolitan area networks.
To understand which of the many and varied initiatives taking place worldwide could be used to provide connectivity for research purposes - not just in high-energy physics, but in a wide range of scientific applications - the Global Lambda Integrated Facility (GLIF) was established in 2003 at a meeting in Iceland that brought together the major players funding and installing research networks.
GLIF may have important implications for the general-purpose connectivity between Tier-1 and Tier-2 centres as well as inter-Tier-2 connectivity, as it helps organize a complex set of research initiatives worldwide. In particular, GLIF promotes the sharing between participants of lambdas - single wavelengths on multiple-wavelength optical fibres - which provides additional connectivity between networks. Other developments, such as falling bandwidth costs, will also have an impact, leading to an increasing number of high-speed direct links (10 Gbit/s or more) between Tier-1 and Tier-2 centres in the near future, as research-network initiatives continue to acquire affordable dark fibre.
The way forward
The Tier-1 and Tier-2 centres already have plans to implement the required infrastructures for connecting at sufficient bandwidth to fulfil the needs of the LHC experiments. The service challenges will continue to exercise the total end-to-end infrastructure, increasing the load and the number of sites involved. The infrastructure and design being implemented at the moment will be adequate for LHC start-up according to the experiments' computing models. Impressive as this recent progress may seem, it is just the beginning. As an increasing volume of data is created by the experiments, the requirements to transfer these data as quickly as possible between sites will expand.
Part of the solution will be so-called open optical exchanges, which will use pure wavelength switching to allow opportunities for networks to interconnect and provide global connectivity for research and education. Such facilities will be commonplace once the underlying optical technologies achieve sufficient maturity. Equally important, as physicists worldwide start to work on the LHC data, is the challenge to ensure that the tremendous resources made available through the many Grid initiatives around the world can be used effectively, even at the edge of the networks where bandwidth is limited. The issue here is also about bridging the digital divide between well-connected and poorly-connected regions of the world.
This evolution will only be possible through a constant, incremental improvement of networking worldwide in terms of capacity, cost and availability. By providing a clear and visible goal both for scientists and for key decision-makers to target, LHC computing is helping to accelerate that process. This benefits scientists in all fields, and ultimately ordinary citizens around the globe.