The Software Development for Experiments Group (SFT) in the CERN Physics Department, together with the Data Management Group of IT are hosting the LCG Applications Area (AA), which provides several projects that form the most fundamental building blocks for LHC experiment software. Examples for such projects are the ROOT analysis framework, the Geant4 simulation toolkit and projects for data handling such as POOL, COOL and CORAL.
These projects that are developed within AA are augmented by a set of "external" software packages to cover a wide area of functionality, such as mathematical libraries, databases, graphics libraries, scripting languages, Monte Carlo Generators, etc (around 80 at the moment). Most of the external packages are retrieved from sources outside CERN, e.g. Python, Boost, Qt, but also from within e.g. the Grid client software provided by IT-GD. All of these constituents are built and tested with the same version on several platforms, i.e. a combination of operating systems (SLC4, SLC5, Windows, Mac OSX), architectures (i686, x86_64) and compilers (gcc 3.4, gcc 4.0, gcc 4.3, VC7.1, VC9). Being in the transition phase between two SLC releases and two windows compilers, a total of 20 different platforms are provided (figure 1).
Releasing AA software
On the requests of LHC experiments this software stack is released all together in so called "LCG Configurations", which are sets of versions of all packages (projects and externals) working together across all of the provided platforms. With this model several challenges for AA developers and LHC experiments emerge:
• LHC experiments would like to test early the latest code changes of the AA projects against their own applications.
• AA software developers need to check and validate their code changes on the different provided platforms.
• The change/upgrade of an external package needs to be validated against the AA stack and experiment code.
The LCG nightly builds
To facilitate all of these tasks a "nightly build and test system" has been implemented by the Software Process and Infrastructure project (SPI), which works in three "dimensions".
• All projects are compiled every night. Usually the building and testing starts around midnight and results are provided later in the morning. The produced binaries and results are kept for one week.
• Projects are being compiled in different configurations, i.e. a set of versions of AA packages that are supposed to work together. A configuration can be set up e.g. to test patches for a production release, the repository "HEAD" of all projects, the upcoming LCG configuration, etc.
• Different platforms. As mentioned above, the compilation of AA software is provided on different platforms. A build machine for each operating system is provided (e.g. for Linux by IT-FIO) and the nightly builds are executed on it.
The nightly build system provides the binary products to experiments for their integration testing and feedback to AA developers about their changes. The results of all building and testing steps are summarized in a webpage that gives a quick overview of the status of the different platforms for each configuration and day (figure 2). The binary products of the build step are provided in afs (see link below) where they can be further picked up by the users. For example, the Atlas and LHCb experiments are using these products to build their own nightly builds on top of these installations. With this "chain of nightlies" we are able to spot any potential problems in the whole LHC software stack early and take appropriate actions.
The nightly build scripts are implemented in Python allowing easy deployment of the tool across different operating systems. The steering of the nightly builds configuration is done via an XML file and the actual build instructions are encapsulated in Python "builder classes", making it easy to extend the system for other users. An open source product, the CMT tool is used for configuration management and software building. This facilitates the handling of dependencies between AA packages and also provides the build instructions for each package. The use of CMT is optional; the nightly builds would also work without it.
Use of the nightlies
The nightly builds are a tool that greatly helped with the process of releasing AA software. Before this, every project had built its own release one after the other discovering possible problems and inconsistencies very late in the process. Nowadays, as we have the overview of all projects on all platforms, when it comes to releasing the AA software we "rubberstamp" the state of the whole AA stack once satisfied with its quality and release immediately. The actual release of the projects is done with exactly the same build instructions as the nightly builds, thereby guaranteeing that the software is built in exactly the same way.
The use of the nightly builds (also for releasing the AA stack itself) allowed for the production of an "LCG Configuration" within hours instead of in the order of days (sometimes weeks). This fast turnaround will be crucial once the LHC data taking has started and any bug fix to the software has to be provided as quickly as possible.
The nightly build system itself is not only used for AA project testing but also by other parties at CERN. It has been adopted by the Geant4 and LHCb Offline Computing team for its software testing, showing that the tool can be easily extended for users outside AA. Moreover each of the parties is not only using the software but also contributing with new developments and ideas to the overall project leading to mutual benefits.
Useful links
LCG-AA Software Elements: http://lcgsoft.cern.ch
Nightlies webpage: http://lcgapp.cern.ch/spi/cgi-bin/nightlies.py
AFS path to nightly build binary products:
/afs/cern.ch/sw/lcg/app/nightlies
CMT tool: http://www.cmtsite.org
Geant4 nightly summaries: http://tinyurl.com/klvfoj
LHCb nightly summaries: http://cern.ch/lhcb-nightlies
PH-SFT homepage: http://cern.ch/ph-sft