Virtual machines (VMs) can add agility to computing environments by satisfying different execution environment requirements on single machines, providing flexible testing environments and allowing more robust services with Live Migration. In these scenarios the infrastructure needs to manage the VM images being deployed efficiently, which can vary in number and range in size from several hundred megabytes to several gigabytes. Also, up-to-date VM images need to be generated automatically because doing this manually is time consuming and error prone.
Virtual appliances extend the concept of VMs and offer a preconfigured integrated application deployment inside a VM, meaning that it can be configured more easily and has a smaller impact on the hosts of the applications and services. It opens the door to a novel and expanding ecosystem for deploying services and computing, such as cloud computing.
OS Farm
OS Farm is a service similar to rPath, which provides images and virtual appliances for use in VMs. OS Farm takes advantage of a tool and library developed in the IT-GD group, called Libfsimage, that generates VM images with many different flavours of Linux, including Scientific Linux CERN (SLC), Debian and Ubuntu. By using Libfsimage in the backend, OS Farm provides a service to create and store VM images and virtual appliances. This is similar to rPath, but the focus differs in that images are created on demand and are more faithful to the originating OS. OS Farm provides a web interface through which users can configure their images. Users can choose between a number of software repositories, offering a rich set of software packages that can be selected and added to the image configuration. On creation, images are stored in a browseable repository from which they can be retrieved later. Users can also submit a configuration specified in XML or interact through a Simple Object Access Protocol (SOAP) interface.
Automatically generating an image from scratch is a long process, often requiring downloading and installing hundreds of packages, usually taking several minutes. However, parts of images are identical and can be shared between different images. Image generation is optimized by layering the different parts of an image. For example, the "core" of two different images can be a shared layer, while additional configuration can be appended on top. Thanks to the copy-on-write, or "snapshot", capability of Linux's Logical Volume Manager, the application of an existing layer is instantaneous. Using this technique, the generation time can be shortened to around 1 min.
To optimize the generation time further, each image in the repository is tagged with a checksum of its configuration. If an image is requested that already exists in the repository, it can be returned instantly. The repository thus also serves as a cache of recently generated images.
Content-based transfer
Across different VM images there can be a substantial amount of commonality. The way in which the data is laid out in the images allows the commonality to be identified. For example, comparing two images made from different batch systems in the computer centre showed that 84% of their blocks were identical. Comparing two asymmetrical SLC3 and SLC4 images showed that 48% of the blocks contained in SLC3 were found in SLC4, and 22% of SLC4's blocks were found in SLC3.
Content-based transfer (CBT) is a technique that efficiently transfers VM image data from a source host to a target host. It takes advantage of commonality between images to speed up the transfer and reduce the load on the network. We can calculate the hypothetical maximal observed bandwidth from the degree of commonality in the image being transferred, the physical network bandwidth and the disk-read bandwidth. "Observed bandwidth" here means the bandwidth observed from transferring the image, although only the uncommon blocks and identification data have actually been transferred. Moreover, for the common blocks, which do not need to be transmitted, the bandwidth is bound by the disk, and for the differential blocks the bandwidth is bound by the network.
Experimental analysis shows that our implementation of CBT achieves an observed bandwidth close to the theoretical maximum and reduces the load on the network. For example, on a 100 mbit network, given 90% commonality, CBT achieves an observed bandwidth of 28 MB/s – a speed-up of 2.2 times as much as the maximum. At the same time, the impact is reduced to less than 3 MB/s. Both OS Farm and CBT are proofs of concept, but they are freely available for download and can be used under open-source licences.
Useful links
OS Farm project: http://sourceforge.net/projects/osfarm/.
OS Farm demo: http://cern.ch/osfarm.
Content-based transfer library: http://hbjerke.web.cern.ch/hbjerke/cba/cba.xml.