Anatomy of the libvirt virtualization library
An API for easy Linux virtualization
When it comes to scale-out computing (such as cloud computing), libvirt may be one of the most important libraries you've never heard of. Libvirt provides a hypervisor-agnostic API to securely manage guest operating systems running on a host. Libvirt isn't a tool per se but an API to build tools to manage guest operating systems. Libvirt itself is built on the idea of abstraction. It provides a common API for common functionality that the supported hypervisors implement. Libvirt was originally designed as a management API for Xen, but it has since been extended to support a number of hypervisors.
Let's start our discussion of libvirt with a view of the use model, then dig into its architecture and use. Libvirt exists as a set of APIs designed to be used by a management application (see Figure 1). Libvirt, through a hypervisor-specific mechanism, communicates with each available hypervisor to perform the API requests. I explore how this is done with QEMU later in the article.
Figure 1. Comparison and use model of libvirt
Also shown is a comparison of the terminology that libvirt uses. This terminology is important, as these terms are used in API naming. The two fundamental differences are that libvirt calls the physical host a node, and the guest operating system is called a domain. Note here that libvirt (and its application) runs in the domain of the host Linux operating system (domain 0).
Means of control
With libvirt, you have two distinct means of control. The first is demonstrated in Figure 1, where the management application and domains exist on the same node. In this case, the management application works through libvirt to control the local domains. The other means of control exist when the management application and the domains are on separate nodes. In this case, remote communication is required (see Figure 2). This mode uses a special daemon called libvirtd that runs on remote nodes. This daemon is started automatically when libvirt is installed on a new node and can automatically determine the local hypervisors and set up drivers for them (to be discussed shortly). The management application communicates through the local libvirt to the remote libvirtd through a custom protocol. For QEMU, the protocol ends at the QEMU monitor. QEMU includes a monitor console that allows you to inspect a running guest operating system as well as control various aspects of the virtual machine (VM).
Figure 2. Control of remote hypervisors with libvirtd
To support extensibility over a wide variety of hypervisors, libvirt implements a driver-based architecture, which allows a common API to service a large number of underlying hypervisors in a common fashion. This means that certain specialized functionality of some hypervisors is not visible through the API. Additionally, some hypervisors may not implement all API functions, which are then defined as unsupported within the specific driver. Figure 3 illustrates the layering of the libvirt API and associated drivers. Note also here that libvirtd provides the means to access local domains from remote applications.
Figure 3. Driver-based architecture of libvirt
As of this writing, libvirt implements drivers for the hypervisors listed in Table 1. Other drivers will no doubt be available as new hypervisors emerge from the open source communities.
Table 1. Hypervisors that libvirt supports
|Xen||Hypervisor for IA-32, IA-64, and PowerPC 970 architectures|
|QEMU||Platform emulator for various architectures|
|Kernel-based Virtual Machine (KVM)||Linux platform emulator|
|Linux Containers (LXC)||Linux (lightweight) containers for operating system virtualization|
|OpenVZ||Operating system-level virtualization based on the Linux kernel|
|VirtualBox||Hypervisor for x86 virtualization|
|User Mode Linux||Linux platform emulator for various architectures|
|Test||Test driver for a fake hypervisor|
|Storage||Storage pool drivers (local disk, network disk, iSCSI volume)|
Libvirt and the virtualization shell
Now that I've covered some of the architecture of libvirt, let's look at some examples of the use of the libvirt virtualization API. I start by using an application called virsh (virtualization shell), which is built on top of libvirt. This shell permits use of much of the libvirt functionality but in an interactive (shell-based) fashion. In this section, I demonstrate some of the aspects of VM manipulation using virsh.
The first step is to define the domain configuration file (shown in Listing 1, below). This code specifies all the necessary options for defining a domain—from the hypervisor (emulator) to the resources that the domain uses and peripheral configuration (such as the network). Note that this is a very simple configuration: the actual attributes that libvirt supports are much more diverse. For example, you can specify a BIOS and host bootloader, resources to be used by the domain, and devices to be used—from floppy disks and CD-ROMs to USB and PCI devices.
The domain configuration file defines some of the basic metadata to be used for this QEMU domain, including the domain name, maximum memory, and initially available memory (current) as well as the number of virtual processors to be made available to this domain. You don't assign a Universally Unique Identifier (UUID); instead, you allow libvirt to assign one. You define the type of machine to emulate for this platform—in this case, a 686 processor that is fully virtualized (hvm). You define the location of the emulator (in case you need to support multiple of the same type) and the virtual disk for the domain. Note here that you indicate the VM, which is a ReactOS operating system in Virtual Machine Disk (VMDK) format. Finally, you specify the default networking configuration, and you use Virtual Network Computing (VNC) for graphics.
Listing 1. Domain configuration file
<xml version="1.0"?> <domain type='qemu'> <name>ReactOS-on-QEMU<name> <uuid<uuid> <memory>131072<memory> <currentMemory>131072<currentMemory> <vcpu>1<vcpu> <os> <type arch='i686' machine='pc'>hvm<type> <os> <devices> <emulator>usr/bin/qemu<emulator> <disk type='file' device='disk'> <source file='/home/mtj/libvtest/ReactOS.vmdk'/> <target dev='hda'/> <disk> <interface type='network'> <source network='default'/> <interface> <graphics type='vnc' port='-1'/> <devices> <domain>
Now, with the domain configuration file complete, let's start a domain with
the virsh tool. The virsh tool takes a command argument for the particular
action to be taken. In the case of starting a new domain, you use the
create command and the domain configuration
Listing 2. Starting a new domain
mtj@mtj-desktop:~/libvtest$ virsh create react-qemu.xml Connecting to uri: qemu:///system Domain ReactOS-on-QEMU created from react-qemu.xml mtj@mtj-desktop:~/libvtest$
Note here the Universal Resource Indicator (URI) used to attach to the
qemu:///system). This local URI
attaches to the system mode daemon for the local QEMU driver. To attach to
a remote QEMU hypervisor over the Secure Shell (SSH) protocol on host
shinchan, you could use the URI qemu+ssh://shinchan/.
Next, you can list the active domains on a given host using the
list command within virsh. Doing so lists the
active domains, their domain IDs, and their state, as shown below:
Listing 3. Listing active domains
mtj@mtj-desktop:~/libvtest$ virsh list Connecting to uri: qemu:///system Id Name State ---------------------------------- 1 ReactOS-on-QEMU running mtj@mtj-desktop:~/libvtest$
Note that the name defined here is the name you defined in your domain configuration file metadata. You can see that this domain has a domain ID of 1 and is currently running.
You can also suspend a domain with the
command. This command stops the domain from being scheduled, but the
domain continues to reside in memory and can be quickly resumed. The
following example illustrates suspending the domain, performing a list to
see status, and then restarting the domain:
Listing 4. Suspending a domain, checking status, and restarting
mtj@mtj-desktop:~/libvtest$ virsh suspend 1 Connecting to uri: qemu:///system Domain 1 suspended mtj@mtj-desktop:~/libvtest$ virsh list Connecting to uri: qemu:///system Id Name State ---------------------------------- 1 ReactOS-on-QEMU paused mtj@mtj-desktop:~/libvtest$ virsh resume 1 Connecting to uri: qemu:///system Domain 1 resumed mtj@mtj-desktop:~/libvtest$
The virsh utility also supports a number of other commands, such as saving
a domain (
save), restoring a saved domain
restore), rebooting a domain (
reboot), and many others. You can also create a
domain configuration file from a running domain (
So far, you've started and manipulated a domain. But what about attaching to it so that you can see the domain in action. You can do this using VNC. To create a window representing the graphical desktop of the particular domain, you can use VNC as:
Listing 5. Attaching to a domain
mtj@mtj-desktop:~/libvtest$ xvnc4viewer 127.0.0.1 0
Libvirt and Python
The previous example illustrated the control of domains using the command-line utility virsh. Let's now look an example of domain control using Python. Python was the libvirt-supported scripting language and provides a clean, object-oriented interface to the libvirt API.
In this example, I explore some of the same operations that I demonstrated
with the virsh utility (
so on). The Python example script is provided in Listing 6. In this
example, you begin by importing the libvirt module. You then connect to
the local QEMU hypervisor. From here, you iterate through the domain IDs
that are available; for each one, you create a domain object, and then
suspend, resume, and finally destroy the domain.
Listing 6. Sample Python script for domain control (libvtest.py)
import libvirt conn = libvirt.open('qemu:///system') for id in conn.listDomainsID(): dom = conn.lookupByID(id) print "Dom %s State %s" % ( dom.name(), dom.info() ) dom.suspend() print "Dom %s State %s (after suspend)" % ( dom.name(), dom.info() ) dom.resume() print "Dom %s State %s (after resume)" % ( dom.name(), dom.info() ) dom.destroy()
Although this is a simple example, you can see the power that libvirt provides through Python. Through a simple script, you are able to iterate through all of the local QEMU domains, emit some information about the domain, and then control the domain. The output of this script is shown in Listing 7.
Listing 7. Output from the Python script in Listing 6
mtj@mtj-desktop:~/libvtest$ python libvtest.py Dom ReactOS-on-QEMU State 1 Dom ReactOS-on-QEMU State 3 (after suspend) Dom ReactOS-on-QEMU State 1 (after resume) mtj@mtj-desktop:~/libvtest$
At a high level, the libvirt API can be divided into five API sections: the hypervisor connection API, the domain API, the network API, the storage volume API, and finally the storage pool API.
All libvirt communication occurs after a connection is created for a given
hypervisor (for example, as shown with the
call in Listing 6). The connection provides a path
for all other APIs to work through. In the
API, this behavior is provided through the
virConnectOpen call (as well as others for authentication).
The response of these functions is a
virConnectPtr object, which represents a connection to a
hypervisor. This object serves as the basis for all other management
functionality and is therefore a required argument for subsequent API
calls to a given hypervisor. Important subsequent calls are
virConnectGetCapabilities, which returns the
capabilities of the hypervisor and driver, and
virNodeGetInfo, which retrieves information about the node.
This information is returned as an XML document that can be parsed to
understand which behaviors are possible.
Now, having access to a hypervisor, you can iterate through the various
resources on that hypervisor with a set of API calls. The
virConnectListDomains API call returns a list of
domain identifiers representing the active domains on that hypervisor.
The API implements a large number of functions targeted toward domains. To
explore or manage a domain, you first need a
virDomainPtr object. You can get this handle in a number of
ways (using either the ID, UUID, or domain name). Continuing with the
example of iterating domains, you can use the index list that this
function returns and call
to get the domain handle. With the domain handle in hand, you can now
perform a large number of operations, from exploring the domain (
virDomainMemoryPeek) to controlling the domain (
You can also use the API to manage and inspect virtual networks and storage
resources. Following the model of the API, a
virNetworkPtr object is necessary to manage and inspect
virtual networks, and a
(storage pool) or
object is necessary to manage these resources.
The API also supports an event mechanism with which you can register to be notified of particular events (such as a domain being booted, suspended, resumed, or stopped).
The libvirt library was implemented in
C++) and includes direct support
for Python. But it also supports a number of language bindings. Bindings
have been implemented for Ruby, the Java™ language, Perl, and
OCaml. Work has also been done for calling libvirt from
C#. Libvirt supports the most popular system
programming languages (
C++), a variety of scripting languages, and even
a unified functional language (Objective caml). So whatever your language
focus, libvirt provides a path to control your domains.
Applications using libvirt
From just the small amount of capabilities that I've demonstrated in this article, you can see the power that libvirt provides. And as you can expect, there are a number of applications that are being successfully built on libvirt. One of the interesting applications is virsh (demonstrated here), which is a virtualization shell. There's also virt-install, which can be used to provision new domains from operating system distributions. The utility virt-clone can be used to clone a VM from another VM (covering both operating system and disk replication). Some of the higher-level applications include virt-manager, which is a general-purpose desktop-management tool, and virt-viewer, which is a lightweight tool for securely attaching to the graphical console of VMs.
One of the most important tools built on libvirt is called oVirt. The oVirt VM management application was designed to manage a single VM on a single node or thousands of VMs over hundreds of hosts. In addition to simplifying management of large numbers of hosts and VMs, it can be used to automate clustering and load balancing and works across platforms and architectures.
As you can see from this short article, libvirt is a great library for building applications that manage domains in many different hypervisor environments over large networks of systems. Given the growing popularity of cloud computing, libvirt will no doubt grow along with it, finding new applications and users. As of this writing, libvirt is only just over four years old, so it's relatively new in the massively scalable computing space. More is most certainly to come.
- Check out the libvirt Web site for the latest news about libvirt and to download the latest version. You'll also find a complete API reference manual that covers the core interfaces and error-handling interfaces.
- Libvirt supports a large number of hypervisors, including:
- In "LXC: Linux Container Tools" (developerWorks, February 2009), you can learn more about the management tools built specifically for LXC. You'll notice some parallels here with the management methods for libvirt.
- Virtual Network Computing, or VNC, is a method for sharing graphical desktops over a network. This abstraction is ideal for massive scale-out computing, as it means a single station can manage many distributed clients.
- The Linux subsystem for managing file systems is large and complex. You can learn more about the larger file system subsystem in "Anatomy of the Linux file system" (developerWorks, October 2007).
- Red Hat's oVirt open VM management platform is a user of libvirt and demonstrates what kinds of applications can be built. You can use oVirt to manage large numbers of hosts, and the platform easily scales to support thousands of VMs.
- In this article, you used the ReactOS as a means to demonstrate a domain on QEMU. This article provides more information on ReactOS (a free Windows® clone), which is easily executed on the QEMU platform.
- In the developerWorks Linux zone, find more resources for Linux developers, and scan our most popular articles and tutorials.
- See all Linux articles and Linux tutorials on developerWorks.