This article is a high-level overview of the Common Computing Readiness Challenge – known as CCRC'08. For those who have been involved, the most important message is undoubtedly "thanks for all the hard work – and save some energy for May!"
For several years, the four LHC experiments (ALICE, ATLAS, CMS and LHCb) have been running periodic stress tests of their planned computing operations when LHC data arrives. They have written design documents that describe their models for how they will acquire, distribute and process their data. They differ in the details but have a common base in a model of levels or "Tiers" of resources.
Master copies of the raw data are kept on tape at CERN, the Tier0 site. CERN also does a first reconstruction pass on the raw data using approximate calibration constants. The final calibrations are not ready for weeks or even months, but will be used for a second reconstruction that is done entirely at the next level – the Tier1 sites. (One experiment model – LHCb – differs here in that first pass and later reconstruction is done entirely at their Tier1, but CERN is considered part of that.) A complete copy of the raw data is exported to the Tier1 sites, where it is also put on tape. There are 11 such sites but they do not all support all the experiments.
A total of 10 Tier1 sites support ATLAS, the largest experiment in data rates and volumes: seven support CMS, seven support ALICE and six are behind LHCb (the smallest experiment). The Tier1 resources are not evenly distributed, e.g. BNL (near New York) will provide 28% of ATLAS Tier1 resources, while CNAF in Bologna will provide only 5%.
Raw data is sent from the experiment data acquisition system to disks in the CERN computer centre. There it is migrated to tape under the Castor system, reconstructed by batch jobs at the Tier0, and sent (both the raw and reconstructed data) to the experiment Tier1 sites using the Grid file transfer service. At the Tier1 site it is copied to their local mass storage systems (the most common is called dCache) and when the calibration is ready the raw data is recalled from tape and reconstructed using Grid submitted batch jobs. The output data is then ready for further physics analysis.
There is then a third level – the Tier2 sites – which get a copy of the analysis files for their physicists. These comprise about 250 smaller computer centres, mostly at universities. They also generate simulated (Monte Carlo) events that are needed by the experiments and which are stored on tape at Tier1 sites and, usually, at CERN.
Over the years the experiments, as part of the Worldwide LHC Computing Grid project, and together with CERN–IT, have exercised the various components of their computing models. For example, last year over several days ATLAS succeeded in sending simulated raw data to the computer centre at the required 320 MB/s, processing it with several thousand concurrent batch jobs and exporting the raw and processed data to their 10 Tier1 sites at about 1000 MB/s. Some of the Tier1 sites put the required data to tape, but not all did. When all experiments take data together, the aggregate rate out of CERN will be 1600 MB/s and we will be storing about 40 TB of data (about 100 full tape cartridges) into Castor each day. The current LHC plan is to run for 100 days in 2008, from July to December.
Stress testing
A weakness of all the tests done so far is that they have never overlapped all of the experiments together, and so have not reached the full rates expected at CERN and the Tier sites. Also, they have never had to cope with the lack of homogeneity of having the four different experiment data-management and batch-processing models being busy at the same time. CCRC'08 addresses these issues by exercising all the experiments' computing at the target rates and at the same time.
The plan is to run in two phases:
• The first is to test all the functionality involved and attempt to build up to the target rates as far as resources allow (the full 2008 resources at the remote sites will not be available until 1 April and some sites will be late). This was run from 4–28 February. As expected we found many problems in this period, including software, hardware and organizational problems, and are correcting them during March and April.
• In this way we will be ready for a second run during the whole of May, which is planned to reach the full 2008 data rates rapidly over the whole distributed processing and then run stably for many weeks. The May exercise is therefore a full-scale dress rehearsal for the accelerator run, which is due to start in July.
The accelerator should then be in action continuously for many months, with periodic stoppages of several days each. The current operational plans mean that most physics data from the experiments will arrive during a 12 hour period overnight, but the batch processing – both local and over the Grid – and data export will be continuous. The experiment and site support teams will play a vital role in ensuring these continuous operations.
A Twiki providing many details of CCRC'08 is available at https://twiki.cern.ch/twiki/bin/view/LCG/WLCGCommonComputingReadinessChallenges, and an overview of the whole LHC Computing Grid (LCG) project is at http://lcg.web.cern.ch/LCG/.
What happened next?
As described above, the CCRC'08 was designed to bring together all four experiments and to exercise all the computing models from data acquisition through to data analysis at the Tier2 sites. Here we report on progress in the first phase and preparation for the second.
In terms of data transfer, several significant goals were achieved. The total rates transferred from CERN to the Tier1 sites were higher than those achieved in earlier tests, and have been sustained over several weeks. All four experiments have shown sustained rates in excess of their requirements for 2008. Rates of greater than 2.1 GB/s were achieved in aggregate between all experiments from CERN to all 11 Tier1 sites (see figures 1 and 2). Figure 1 shows the distribution to sites by experiment over several weeks and figure 2 shows that on one day. The experiments have found the testing sufficiently useful and so are continuing.
The performance of the Castor 2 system at CERN had also been of concern, but performed reliably at rates well in excess of those needed for data taking. CMS in particular were able to demonstrate aggregate rates in and out of Castor of 3–4 GB/s (figure 3), and sustained rates to tape of 1.3 GB/s. Unfortunately this level of use with several experiments together was not demonstrated, since ATLAS and ALICE were later in starting the challenge. In total during the one-month challenge CMS moved more than 4.5 PB of data between all participating sites. All of their Tier1 sites achieved the targets to receive data from CERN and migrate to tape, and a large fraction of the Tier1–Tier1 and Tier1–Tier2 targets were also achieved.
ATLAS started late as the amount of data generated in their dress rehearsal was rather less than expected. However, using simulated data starting from week three rapidly showed the rates mentioned above. They also validated the use of SRM v2 (the common mass storage interface) and the Tier1 storage system setups. They achieved most of their milestones despite the early problems and external dependencies.
ALICE and LHCb also achieved their data rate targets with sustained rates of 80 MB/s and 70 MB/s, respectively, over several weeks. LHCb tested bulk file deletion with SRM v2. They have tested most of their full computing model, despite the new version of Dirac only being available just before the start of the test.
In conclusion we can state that the February exercise has been a success, with relatively few issues showing up. Some problems of communication – e.g. slow reporting of problems outside of working hours – show that although processes were in place they were not well advertised or used. These points, together with a prioritized list of issues in the storage systems and other middleware services will be addressed for the May challenge. All four experiments expressed the desire to keep running at this level from now on. It is important that the full 2008 resources are in place at the Tier1 sites in time for the May phase so that the complete system can be tested at the full 2008 rates.
Ian Bird, Harry Renshall and Jamie Shiers, IT Department