Track KVM guests with libvirt and the Linux audit subsystem

Audit and filter guest and host events in virtual environments

Libvirt is an important tool for managing virtualized environments on Linux hosts. An audit trail of events that occur on the host during libvirt's execution is often necessary for monitoring, compliance, forensics, and other purposes. This article describes how to use the Linux audit subsystem on the host to track operations performed by libvirt and how to correlate the events with other host events to provide a consistent and comprehensive view of changes performed through libvirt.

Marcelo H. Cerri (mhcerri@br.ibm.com), Staff Software Engineer, IBM

 Photo of Marcelo CerriMarcelo Cerri is a Software Engineer at IBM Linux Technology Center in Brazil where he is as a member of the Security Development Team and works on cloud and virtualization related projects. His background also includes development of cloud computing platforms and telecom billing systems.



05 June 2012

Also available in Chinese Russian Japanese

Overview

Libvirt has been widely adopted to manage virtualized environments in Linux®. It provides a variety of functions including guest life-cycle management, resource assignment, resource management using cgroups, and security enforcement through SELinux, among others. All these operations performed by libvirt deal with host resources being assigned to or released from guests.

Starting from version 0.9.0, libvirt is able to communicate with the Linux auditing subsystem on the host to issue records about several operations. This is an important feature that allows system administrators, auditors or even other applications to have access to a detailed history of changes in the virtualized environment including guest lifecycle operations and changes in the assignments of host resources to the guests.

The Linux audit subsystem is capable of handling many kinds of events from different sources, not just virtualization-related events issued from libvirt. The audit system is often used to track file changes, security denials from SELinux or AppAmor, and system calls. These different sorts of events can contain useful information from the point of view of auditing virtual machines.

For example, an access to a resource assigned to a guest may be forbidden by the SELinux policy and that will generate an AVC (Access Vector Cache) denial record. Although the event is not issued by libvirt, it is tied to a guest. Similarly many events can be related to a guest, among them anomaly events generated by the kernel or by programs residing in user space.

In the next sections, take a quick look at how the auditing system works, how libvirt interacts with it, and how you can get information relevant to the guests from the audit log.

Audit basics

The Linux kernel is responsible for generating most of the audit events such as system call events, file monitoring events, and security events. But it's also possible for user applications to generate events.

The central component of the Linux audit system is the kernel. All the events generated by other components are first sent to the kernel. In the kernel, a series of rules is applied to decide if an event will be discarded or not. Figure 1 shows the components that comprise the Linux audit system.

Figure 1. Auditing system components
Diagram of the components in the auditing subsystem

The audit rules in the kernel can be changed using the command auditctl. Some applications with special capabilities, such as the libvirt daemon, are considered trusted applications and are able to send audit events to the kernel. After the rules are applied, the kernel sends the events to the audit daemon. The audit daemon then writes the records to the disk and potentially dispatches custom actions using the auditspd daemon.

The other components (ausearch, aureport, aulast and auvirt) are userspace tools used to parse the audit logs and to present its information in a simpler form. The auvirt tool is specifically designed to present virtualization-related events and correlated events.

One important thing to notice is the distinction between an audit event and an audit record. An audit event is an action initiated by the user or by some application that can be audited, for example: a file open is an event that can be audited. Furthermore, an audit record contains information related to just one aspect of the event. An event can therefore span one or more records in the audit log. Considering the example of the file open, the event can generate a record with information about the called open syscall and a record with information about the accessed file (see Listing 1 for an example).

Listing 1. Audit records for an open file event (wrapped for clarity)
type=SYSCALL msg=audit(1328292445.908:40194): arch=c000003e syscall=2 
 success=yes exit=3 a0=7fffe3d2e85c a1=941 a2=1b6 a3=7fffe3d2c570 items=3
 ppid=26377 pid=26677 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0
 fsgid=0 tty=pts8 ses=1 comm="touch" exe="/bin/touch"
 subj=unconfined_u:unconfined_r:unconfined_t:s0 key=(null) 

type=CWD msg=audit(1328292445.908:40194):  cwd="/home/mhcerri" 

type=PATH msg=audit(1328292445.908:40194): item=0 name="/tmp/file" 
 inode=1196 dev=fd:01 mode=0100775 ouid=500 ogid=501 rdev=00:00 
 obj=unconfined_u:object_r:user_tmp_t:s0

Notice that each record has a different type and that all three records have the same "1328292445.908:40194" string. This is the event identifier. The first two numbers before the colon (:) are the event timestamp, the first one being a UNIX® time in seconds and the second the milliseconds value. The third number is a generated serial number. Notice also that the format used in the audit records is not simple and because of that the audit system has some tools to assist you in visualizing audit logs.


Libvirt and Audit

Libvirt basically generates three different types of record. They all have one record in common, the "uuid" field. This is a Universal Unique Identifier that identifies a guest in all the records issued by libvirt. The record types used by libvirt are:

  • VIRT_CONTROL: generated at every step in the life-cycle of a virtual machine. It is generated when a guest is started or stopped and it contains fields such as the operation reason and the guest's name.
  • VIRT_MACHINE_ID: generated when libvirt tries to assign a security label to a guest. When using QEMU with sVirt, this record contains the security label of the QEMU process and the security labels of the disks attached to the guest.
  • VIRT_RESOURCE: generated when a change in the guest's resource assignments occurs, for example, when a disk is removed from a guest or a new network device is attached to it.

Auvirt

The auvirt tool searches the audit log for records generated by libvirt to show a list of virtual machine sessions. It also looks at some other events such as host shutdowns, AVC denials related to guests, and anomaly events associated to QEMU processes. Listing 2 is an example of the auvirt output. The first field is the guest's name. The second is the user that started the guest. And the third is the time range during which the guest was running. Notice in the example that an execution of "Arch" has failed and another is still running.

Listing 2. Auvirt output
# auvirt
Fedora                 	root       	Tue Jan 24 16:28 - 16:28  (00:00)
Fedora                 	root       	Wed Jan 25 10:31 - 10:34  (00:02)
CentOS                 	root       	Wed Jan 25 10:34 - 13:50  (03:16)
Fedora                 	root       	Wed Jan 25 13:51 - 13:54  (00:03)
Fedora                 	root       	Wed Jan 25 13:54 - 13:55  (00:00)
Arch                   	root       	Wed Jan 25 13:55 - failed
Arch                   	root       	Wed Jan 25 13:56

It's possible to limit the output to a specific time range using the following options:

  • --start <date> [<time>] : define a start for the time period considered
  • --end <date> [<time>] : define an end for the time period considered

If <time> is omitted, the tool will default to "00:00:00" for the start and "23:59:59" for the end.

Listing 3. Filtering the auvirt output by time
# auvirt --start 01/25/2012 12:00:00 --end  01/25/2012
Fedora                  root        Wed Jan 25 13:51 - 13:54  (00:03)
Fedora                  root        Wed Jan 25 13:54 - 13:55  (00:00)
Arch                    root        Wed Jan 25 13:55 - failed
Arch                    root        Wed Jan 25 13:56

It's also possible to limit the results to just one guest. The guest can be specified using the guest's name or the guest's UUID. By default, the tool doesn't show the UUID, in order to keep the output simple, but you can include it in the output by using the option --show-uuid.

Listing 4. Filtering the auvirt output by guest using name and UUID (wrapped for clarity)
# auvirt --vm Fedora
Fedora                  root        Tue Jan 24 16:28 - 16:28  (00:00)
Fedora                  root        Wed Jan 25 10:31 - 10:34  (00:02)
Fedora                  root        Wed Jan 25 13:51 - 13:54  (00:03)
Fedora                  root        Wed Jan 25 13:54 - 13:55  (00:00)

# auvirt --show-uuid --uuid cabf9532-99f1-3756-930f-59e8f19d4144 
Arch                    cabf9532-99f1-3756-930f-59e8f19d4144    root
 Wed Jan 25 13:55 - failed
Arch                    cabf9532-99f1-3756-930f-59e8f19d4144    root 
 Wed Jan 25 13:56

By default, auvirt shows a simplified output. To show all the virtualization-related events, the option --all-events can be given. With this option, auvirt shows a record for every start and stop event, resource assignment, and correlated event. Listing 5 shows an example using the --all-events option.

Listing 5. A detailed auvirt output (edited for clarity, see the download section for a complete version of this output)
# auvirt --vm CentOS --all-events
down                            root            Wed Jan 25 08:34
res   CentOS                    root            Wed Jan 25 10:34 - 13:50  (03:16)
 cgroup          allow           major   rw      pty

[...]

avc   CentOS                    root            Wed Jan 25 10:34
 relabelto       denied          libvirtd        CentOS.img
 system_u:object_r:svirt_image_t:s0:c6,c883

[...]

res   CentOS                    root            Wed Jan 25 10:34 - 13:50  (03:16)
 disk            start           /var/lib/libvirt/images/CentOS.img
res   CentOS                    root            Wed Jan 25 10:34 - 13:50  (03:16)
 net             start           52:54:00:DB:AE:B4
res   CentOS                    root            Wed Jan 25 10:34 - 13:50  (03:16)
 mem             start           1048576
res   CentOS                    root            Wed Jan 25 10:34 - 13:50  (03:16)
 vcpu            start           1
start CentOS                    root            Wed Jan 25 10:34 - 13:50  (03:16)

[...]

stop  CentOS                    root            Wed Jan 25 13:50

The first field indicates the event type, which can have the following values: start, stop, res, avc, anom, and down (for host shutdowns).

Resource assignments and AVC records have additional fields that provide details about them. The additional fields for resource assignments are:

  • Resource type: describes the different types of resources that can be assigned to a guest such as disks, virtual CPUs, memory, and network devices among others. Additionally, libvirt can be set up to restrict the resources used by each guest using Cgroups and this is also logged as a resource event. The possible values for this field are: vcpu, mem, disk, net, and cgroup.
  • Reason: indicates which action caused the resource assignment event. For example, a disk can be assigned to a guest when the guest is started or when it's running, for these cases the reason value can be "start" or "attach," respectively. For Cgroups events the reason field indicates if a resource is being allowed or denied and its value can be "allow" or "deny."
  • Resource: is the resource that is assigned to a guest. This is a path for disk resources, megabytes for memory, a MAC address for network devices, and the number of VCPUs for virtual CPUs. For Cgroups events, this value is composed of three components: type, ACL and the resource.

The AVC events reported by auvirt are not denied accesses performed by the guest, they are denied accesses to the guest's resources performed by another process in the host. The additional fields for AVC events are:

  • Operation: is the denied operation performed by the process. Some examples are: read, write, open, relabelto, relabelfrom.
  • Operation result: usually denied.
  • Program: the program name of the process that tried to perform the denied operation.
  • Target: the target of the denied operation. For example, if a process inside the host tries to access a guest's disk image file and its access is denied by SELinux, this field will contain the disk image file name.
  • Context: for the relabelto and relabelfrom operations, one more field is included indicating the related security context: the target context for relabelto and the source context for relabelfrom.

Accessing raw data

Auvirt provides user friendly output that doesn't necessarily contain all event information from the audit log. Sometimes this additional information may be useful and access to the raw data is required to get more detailed information. Auvirt has the option --proof that can be used to display the audit event identifiers used to generate each record in the output.

Listing 6. Auvirt --proof option
# auvirt --vm Fedora -ts 01/23/2012 00:00 -te 01/23/2012 17:30 --proof 
Fedora                      root        Mon Jan 23 17:02 - 17:27  (00:24)
    Proof: 1327345338.087:41631, 1327346831.548:41702

Listing 6 shows an example of auvirt displaying the start and stop event identifiers used to create the record. Using this information, you can extract the raw records from audit logs using the ausearch tool (see Listing 7).

Listing 7. Using ausearch to get raw records (wrapped for clarity)
# ausearch -a 41631 --start 01/23/2012 17:02:00 --end 01/23/2012 17:27:59 
---- 
time->Mon Jan 23 17:02:18 2012 
type=VIRT_CONTROL msg=audit(1327345338.087:41631): user pid=2433 uid=0
 auid=4294967295 ses=4294967295 subj=system_u:system_r:initrc_t:s0 msg='virt=kvm 
 op=start reason=booted vm="Fedora" uuid=7bcae45c-c179-6cea-a185-c7abe50520c7: 
 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'

This example uses the -a option, which restricts the output to records with the given serial number, and the --start and --end options that work in the same way as the auvirt options. Notice that the raw records contains additional information such as reason, executable name, and pid.


Going Further

Given the growing popularity of cloud computing, virtualization has become a prominent technology. It is increasingly important to have a clear understanding of what happens in virtualized environments. In addition, auditing can be a mandatory requirement when dealing with certain industry segments, such as government or financial. This article gives an overview of audit tools that can be used for filtering virtualization-related host audit events. With these tools, you can gain a more complete audit trail for a guest by considering guest events that occur both on the guest and on the host.

Resources

Learn

Get products and technologies

  • Evaluate IBM products in the way that suits you best: Download a product trial, try a product online, use a product in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement Service Oriented Architecture efficiently.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Linux on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Linux, Open source, Cloud computing
ArticleID=819218
ArticleTitle=Track KVM guests with libvirt and the Linux audit subsystem
publish-date=06052012