An international team led by the California Institute of Technology (Caltech), with partners from Michigan, Florida, Tennessee, Fermilab, Brookhaven, CERN, Brazil, Estonia, Korea and Pakistan, set new world records for sustained data transfer among storage systems during the successful SuperComputing 2008 (SC08) conference held in Austin, Texas, in November.
Caltech’s exhibit at SC08 by the High-Energy Physics (HEP) group and the Center for Advanced Computing Research (CACR) demonstrated new applications and systems for globally distributed data analysis for the LHC at CERN, together with Caltech’s global monitoring system, MonALISA, and its collaboration system, Enabling Virtual Organizations. A highlight of the exhibit was the HEP team’s record-breaking demonstration of storage-to-storage data transfers. This achieved a bidirectional peak throughput of 114 Gbit/s and a sustained data flow of more than 110 Gbit/s among clusters of servers on the show floor and at Caltech, Michigan, CERN, Fermilab, Brazil (Rio de Janiero, São Paulo), Korea, Estonia and locations in the US LHCNet network in Chicago, New York, Geneva and Amsterdam.
The team used a small fraction of the global LHC Grid to transfer data between the Tier-1, Tier-2 and Tier-3 facilities at the partners’ sites and between a Tier-2-scale computing and storage facility that the HEP and CACR team had constructed on the exhibit floor in fewer than two days. Rates of more than 40 Gbit/s were sustained in both directions for several hours (and up to 71 Gbit/s in one direction). One of the key elements in this demonstration was Fast Data Transfer (FDT), an open-source Java application based on TCP developed by the Caltech team in close collaboration with colleagues at Politechnica Bucharest. FDT works dynamically with Caltech’s MonALISA system to monitor the capability of the storage systems, as well as the network path, in real time. It also sends data out to the network at a rate that is matched to the capacity of long-range network paths.
A second major milestone was achieved by the HEP team working together with Ciena Corporation, which had just completed its first OTU4-standard optical link carrying a 100 Gbit/s payload over a single wavelength with forward-error correction. The teams used a fibre-optic cable with 10 fibre-pairs to link their neighbouring booths together; Ciena’s system to multiplex and demultiplex ten 10 Gbit/s links onto the single OTU4 wavelength running on an 80 km fibre loop; and some of the Caltech nodes used in setting the wide-area network records, together with FDT. Thanks to the system’s high throughput capabilities and the error-free links between the booths, the teams managed to achieve a maximum of 199.90 Gbit/s bidirectionally (memory-to-memory) within minutes, and an average of 191 Gbit/s during a 12 hour period that logged the transmission of 1.02 PB overnight.
