Like all of the LHC experiments, LHCb relies on a tremendous amount of CPU power to select interesting events out of the many millions that the LHC produces every second. Indeed, a large part of the ingenuity of the LHCb collaboration goes into developing trigger algorithms that can sift out the interesting physics from a sea of background. The cleverer the algorithms, the better the physics, but often the computational cost is also higher. About 1500 powerful computing servers in an event filter farm are kept 100% busy when LHCb is taking data and still more could be used.
However, this enormous computing power is used less than 20% of the time when averaged over the entire year. This is partly because of the annual shutdown, so preparations are under way to use the power of the filter farm during that period for offline processing of data – the issues to be addressed include feeding the farm with events from external storage. The rest of the idle time is a result of the gaps between the periods when there are protons colliding in the LHC (the “fills”), which typically last between two and three hours, where no collisions take place and therefore no computing power is required.
This raises the question about whether it is somehow possible to borrow the CPU power of the idle servers and use it during physics runs for an extra boost. Such thoughts led to the idea of “deferred triggering”: storing events that cannot be processed online on the local disks of the servers, and later, when the fill is over, processing them on the now idle servers.
The LHCb Online and Trigger teams quickly worked out the technical details and started the implementation of a deferred trigger early this year. As often happens in online computing, the storing and moving of the data is the easy part, while the true challenge lies in the monitoring and control of the processing, robust error-recovery and careful bookkeeping. After a few weeks, all of the essential pieces were ready for the first successful tests using real data.
Depending on the ratio of the fill length to inter-fill time, up to 20% of CPU time can be deferred – limited only by the available disk space (currently around 200 TB) and the time between fills in the LHC. Buying that amount of CPU power would correspond to an investment of hundreds of thousands of Swiss francs. Instead, this enterprising idea has allowed an increase in the performance of its trigger, allowing time for more complex algorithms (such as the online reconstruction of KS decays) to extend the physics reach of the experiment.
