COMPASS will probably be able to afford up to 10 Tbytes of disk space, while the total data will be of the order of 300 Tbytes per year of operation. A "hierarchical storage manager" (HSM) is needed to move data between disk and tape.

To solve this problem, which will become even more acute for the LHC experiments, CERN is already testing a high performance storage system (HPSS), which may be appropriate for COMPASS. CERN's Information Technology (IT) division is also developing alternative solutions. IT's physics data processing group (PDP) and RD45 are also implementing solutions to reconcile the requirements of the database and the hierarchical storage manager.

The Central Data Recording (CDR) scheme developed by CERN's IT/PDP group aims to connect an experiment's data acquisition system with a high-speed network link and store the data using the CERN computer centre's tape drives. The CDR was launched for the NA48 experiment and is now routinely used by many other CERN experiments.

The present COMPASS data recording scheme forsees:

  • raw data written onto files in byte-stream format by the on-line computers on their local disks (10­20 parallel streams). (The typical sizes of each of these files will range from a few megabytes to a few gigabytes);
  • files transferred by the CDR to the COMPASS computing farm at the computer centre, where they are formatted into a federated database with objects created and stored. After formatting the corresponding on-line files are deleted;
  • the databases written to tape and the corresponding off-line files deleted.

Computing farm

Large physics experiments can be serviced by "farms" of dedicated PCs. In 1998 an IT/PDP task force looked at the requirements for high-rate experiments (NA48, NA45 and COMPASS). The model proposed for the COMPASS computing farm (CCF) is a hybrid farm with a few (fewer than 10) Unix "data servers", to cope with the data rate, and about 200 mass-market PCs as CPU "clients".

In the present data flow scheme the on-line system performs the event building. If necessary it filters out events and then sends them to the CDR and to the off-line farm at a typical rate of 35 Mbyte/s.

In the off-line farm the data servers will handle the network traffic from the CDR, distribute raw data to the CPU clients, receive them back from the PCs and finally send them to the HSM. In parallel, the data servers will receive data from the HSM, send them to the PCs where the data are processed and collect and send the output to the HSM. The server disk space will be of the order of 10 Tbyte.

Since the computing power required by COMPASS depends on the large number of CPU clients, the analysis program should incorporate some degree of parallelism. Parallelism is already present at the on-line level, where a farm of event builders receives the events and sends data to the off-line farm in multiple streams.

Although Objectivity/DB allows the user to access both remote and local data, the CPU-intensive jobs should be scheduled on those machines with an important fraction of local data. The database model can be used to maximize this "data locality".

Such resource allocation should be done outside and before the user code, which is written in the most general way, accessing data via physical queries without entering into details of data location.

The CCF prototype

COMPASS began to investigate the CCF scheme and Objectivity/DB in 1998. The emphasis has been on performances in terms of data rate and on schemes for recovery after hardware or software failures.

Performances are evaluated in a scaled-down configuration using the same architecture as the final one:

  • a prototype of the on-line farm in the COMPASS experimental area sends mock data to the computer centre via a fibre link (Gigabit Ethernet) over multiple streams (about 10) at variable rates;
  • in the CCF farm prototype, consisting of 7 PCs at the computer centre equipped with Fast Ethernet network cards, the data are received by the data server under test;
  • the data are sent to an Objectivity/DB federated database mainly using the PCs;
  • the databases are sent to the HPSS and the data server disk is cleared.

Data rates of up to 7 Mbyte/s (roughly 28 Mbyte/s of total throughput) have been attained for one server. In these tests the data server was receiving on-line data from its local disks, serving them to a PC farm and getting back objects (one object per event created in the PC) with the Objectivity/DB database server feeding the HPSS system.

Windows NT and Linux have been compared on the client computers. Linux is a clear winner in terms of performance (mainly due to superior network access) and Linux PC looks like a promising replacement for the data server machines. The work of merging the database with the prototype reconstruction code (CORAL; Compass Reconstruction and Analysis) has begun. Building and operating such a complex system for data recoding and processing is a major challenge. Using so many new techniques is not without risk. Most problems derive from software and hardware transition, far from a stable and tested environment.

The most evident problems are those connected with learning the new programming language, adopting commercial software tools and managing computing farms consisting of hundreds of PCs.

The transition to the object-oriented technologies, which is far from complete, has several important consequences. As well as learning a new language, physicists also have to adopt new working methods.

The design of the software requires a different way of thinking by programmers, which usually requires substantial time and effort. Despite a major training effort, particularly at CERN, the use of object-oriented technologies can produce a chism between "programmers" (usually adaptable young physicists) and "users" (mainly physicists interested in data analysis and with considerable experience). This artificial separation could have dramatic effects, particularly for the off-line software.

Summarizing, object-oriented programming speeds up program development and, if properly used, improves the maintenance, reusability and modifiability of software. However, a lot of care has to be put into code design and user requirements.

Commercial tools

Migrating to object-oriented software, existing software has to be re-engineered and new libraries constructed. Commercial object-oriented software can help, particularly for data management, graphics and visualization toolkits. However, within the HEP community many people were sceptical about such a decision, mainly because of doubts about the quality and functionality of commercial tools. It is not easy to find products that satisfy physics requirements. When a promising product is found, specific requirements can delay availability with doubts that the HEP market is big enough to warrant the effort. Licensing issues also emerge.

Parallel processing

More specific to the COMPASS experiment is the choice of hardware. The CCF architecture (based on the distributed PC network) implies parallel processing to get to grips with the event rate. Data collected during the runs are distributed to the different CPU clients. Sequentiality in processing, the data are lost and several "standard" techniques have to be abandoned. An example is "self-calibrating" procedures for electromagnetic calorimeters: with parallel processing, the continuous update of the calibration to follow the behaviour of the detector is no longer feasible.

Also, to take advantage of all of the computing features, new procedures have to be set up to control data distribution to the client CPU and to collect the emerging information.

Despite such intimidating challenges, the mood is optimistic. COMPASS is on course.