![Close [x]](http://dw1.s81c.com/i/c.gif){kind=small}
Libvirt has been widely adopted to manage virtualized environments in Linux®. It provides a variety of functions including guest life-cycle management, resource assignment, resource management using cgroups, and security enforcement through SELinux, among others. All these operations performed by libvirt deal with host resources being assigned to or released from guests.
Starting from version 0.9.0, libvirt is able to communicate with the Linux auditing subsystem on the host to issue records about several operations. This is an important feature that allows system administrators, auditors or even other applications to have access to a detailed history of changes in the virtualized environment including guest lifecycle operations and changes in the assignments of host resources to the guests.
The Linux audit subsystem is capable of handling many kinds of events from different sources, not just virtualization-related events issued from libvirt. The audit system is often used to track file changes, security denials from SELinux or AppAmor, and system calls. These different sorts of events can contain useful information from the point of view of auditing virtual machines.
For example, an access to a resource assigned to a guest may be forbidden by the SELinux policy and that will generate an AVC (Access Vector Cache) denial record. Although the event is not issued by libvirt, it is tied to a guest. Similarly many events can be related to a guest, among them anomaly events generated by the kernel or by programs residing in user space.
In the next sections, take a quick look at how the auditing system works, how libvirt interacts with it, and how you can get information relevant to the guests from the audit log.
The Linux kernel is responsible for generating most of the audit events such as system call events, file monitoring events, and security events. But it's also possible for user applications to generate events.
The central component of the Linux audit system is the kernel. All the events generated by other components are first sent to the kernel. In the kernel, a series of rules is applied to decide if an event will be discarded or not. Figure 1 shows the components that comprise the Linux audit system.
Figure 1. Auditing system components
The audit rules in the kernel can be changed using the command
auditctl. Some applications with special capabilities, such as
the libvirt daemon, are considered trusted applications and are
able to send audit events to the kernel. After the rules are
applied, the kernel sends the events to the audit daemon. The
audit daemon then writes the records to the disk and
potentially dispatches custom actions using the auditspd
daemon.
The other components (ausearch, aureport, aulast and auvirt) are userspace tools used to parse the audit logs and to present its information in a simpler form. The auvirt tool is specifically designed to present virtualization-related events and correlated events.
One important thing to notice is the distinction between an audit event and an audit record. An audit event is an action initiated by the user or by some application that can be audited, for example: a file open is an event that can be audited. Furthermore, an audit record contains information related to just one aspect of the event. An event can therefore span one or more records in the audit log. Considering the example of the file open, the event can generate a record with information about the called open syscall and a record with information about the accessed file (see Listing 1 for an example).
Listing 1. Audit records for an open file event (wrapped for clarity)
type=SYSCALL msg=audit(1328292445.908:40194): arch=c000003e syscall=2 success=yes exit=3 a0=7fffe3d2e85c a1=941 a2=1b6 a3=7fffe3d2c570 items=3 ppid=26377 pid=26677 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts8 ses=1 comm="touch" exe="/bin/touch" subj=unconfined_u:unconfined_r:unconfined_t:s0 key=(null) type=CWD msg=audit(1328292445.908:40194): cwd="/home/mhcerri" type=PATH msg=audit(1328292445.908:40194): item=0 name="/tmp/file" inode=1196 dev=fd:01 mode=0100775 ouid=500 ogid=501 rdev=00:00 obj=unconfined_u:object_r:user_tmp_t:s0 |
Notice that each record has a different type and that all three records have the same "1328292445.908:40194" string. This is the event identifier. The first two numbers before the colon (:) are the event timestamp, the first one being a UNIX® time in seconds and the second the milliseconds value. The third number is a generated serial number. Notice also that the format used in the audit records is not simple and because of that the audit system has some tools to assist you in visualizing audit logs.
Libvirt basically generates three different types of record. They all have one record in common, the "uuid" field. This is a Universal Unique Identifier that identifies a guest in all the records issued by libvirt. The record types used by libvirt are:
- VIRT_CONTROL: generated at every step in the life-cycle of a virtual machine. It is generated when a guest is started or stopped and it contains fields such as the operation reason and the guest's name.
- VIRT_MACHINE_ID: generated when libvirt tries to assign a security label to a guest. When using QEMU with sVirt, this record contains the security label of the QEMU process and the security labels of the disks attached to the guest.
- VIRT_RESOURCE: generated when a change in the guest's resource assignments occurs, for example, when a disk is removed from a guest or a new network device is attached to it.
The auvirt tool searches the audit log for records generated by libvirt to show a list of virtual machine sessions. It also looks at some other events such as host shutdowns, AVC denials related to guests, and anomaly events associated to QEMU processes. Listing 2 is an example of the auvirt output. The first field is the guest's name. The second is the user that started the guest. And the third is the time range during which the guest was running. Notice in the example that an execution of "Arch" has failed and another is still running.
Listing 2. Auvirt output
# auvirt Fedora root Tue Jan 24 16:28 - 16:28 (00:00) Fedora root Wed Jan 25 10:31 - 10:34 (00:02) CentOS root Wed Jan 25 10:34 - 13:50 (03:16) Fedora root Wed Jan 25 13:51 - 13:54 (00:03) Fedora root Wed Jan 25 13:54 - 13:55 (00:00) Arch root Wed Jan 25 13:55 - failed Arch root Wed Jan 25 13:56 |
It's possible to limit the output to a specific time range using the following options:
-
--start <date> [<time>]: define a start for the time period considered -
--end <date> [<time>]: define an end for the time period considered
If <time> is omitted, the tool
will default to "00:00:00" for the start and "23:59:59" for the
end.
Listing 3. Filtering the auvirt output by time
# auvirt --start 01/25/2012 12:00:00 --end 01/25/2012 Fedora root Wed Jan 25 13:51 - 13:54 (00:03) Fedora root Wed Jan 25 13:54 - 13:55 (00:00) Arch root Wed Jan 25 13:55 - failed Arch root Wed Jan 25 13:56 |
It's also possible to limit the results to just one guest. The
guest can be specified using the guest's name or the guest's
UUID. By default, the tool doesn't show the UUID, in order to
keep the output simple, but you can include it in the output by
using the option --show-uuid.
Listing 4. Filtering the auvirt output by guest using name and UUID (wrapped for clarity)
# auvirt --vm Fedora Fedora root Tue Jan 24 16:28 - 16:28 (00:00) Fedora root Wed Jan 25 10:31 - 10:34 (00:02) Fedora root Wed Jan 25 13:51 - 13:54 (00:03) Fedora root Wed Jan 25 13:54 - 13:55 (00:00) # auvirt --show-uuid --uuid cabf9532-99f1-3756-930f-59e8f19d4144 Arch cabf9532-99f1-3756-930f-59e8f19d4144 root Wed Jan 25 13:55 - failed Arch cabf9532-99f1-3756-930f-59e8f19d4144 root Wed Jan 25 13:56 |
By default, auvirt shows a simplified output. To show all the
virtualization-related events, the option
--all-events can be
given. With this option, auvirt shows a record for every start
and stop event, resource assignment, and correlated event.
Listing 5 shows an example using the
--all-events option.
Listing 5. A detailed auvirt output (edited for clarity, see the download section for a complete version of this output)
# auvirt --vm CentOS --all-events down root Wed Jan 25 08:34 res CentOS root Wed Jan 25 10:34 - 13:50 (03:16) cgroup allow major rw pty [...] avc CentOS root Wed Jan 25 10:34 relabelto denied libvirtd CentOS.img system_u:object_r:svirt_image_t:s0:c6,c883 [...] res CentOS root Wed Jan 25 10:34 - 13:50 (03:16) disk start /var/lib/libvirt/images/CentOS.img res CentOS root Wed Jan 25 10:34 - 13:50 (03:16) net start 52:54:00:DB:AE:B4 res CentOS root Wed Jan 25 10:34 - 13:50 (03:16) mem start 1048576 res CentOS root Wed Jan 25 10:34 - 13:50 (03:16) vcpu start 1 start CentOS root Wed Jan 25 10:34 - 13:50 (03:16) [...] stop CentOS root Wed Jan 25 13:50 |
The first field indicates the event type, which can have the following values: start, stop, res, avc, anom, and down (for host shutdowns).
Resource assignments and AVC records have additional fields that provide details about them. The additional fields for resource assignments are:
- Resource type: describes the different types of resources that can be assigned to a guest such as disks, virtual CPUs, memory, and network devices among others. Additionally, libvirt can be set up to restrict the resources used by each guest using Cgroups and this is also logged as a resource event. The possible values for this field are: vcpu, mem, disk, net, and cgroup.
- Reason: indicates which action caused the resource assignment event. For example, a disk can be assigned to a guest when the guest is started or when it's running, for these cases the reason value can be "start" or "attach," respectively. For Cgroups events the reason field indicates if a resource is being allowed or denied and its value can be "allow" or "deny."
- Resource: is the resource that is assigned to a guest. This is a path for disk resources, megabytes for memory, a MAC address for network devices, and the number of VCPUs for virtual CPUs. For Cgroups events, this value is composed of three components: type, ACL and the resource.
The AVC events reported by auvirt are not denied accesses performed by the guest, they are denied accesses to the guest's resources performed by another process in the host. The additional fields for AVC events are:
- Operation: is the denied operation performed by the process. Some examples are: read, write, open, relabelto, relabelfrom.
- Operation result: usually denied.
- Program: the program name of the process that tried to perform the denied operation.
- Target: the target of the denied operation. For example, if a process inside the host tries to access a guest's disk image file and its access is denied by SELinux, this field will contain the disk image file name.
-
Context: for the
relabeltoandrelabelfromoperations, one more field is included indicating the related security context: the target context forrelabeltoand the source context forrelabelfrom.
Auvirt provides user friendly output that doesn't necessarily
contain all event information from the audit log. Sometimes
this additional information may be useful and access to the raw
data is required to get more detailed information. Auvirt has
the option --proof that can be
used to display the audit event identifiers used to generate
each record in the output.
Listing 6. Auvirt --proof option
# auvirt --vm Fedora -ts 01/23/2012 00:00 -te 01/23/2012 17:30 --proof
Fedora root Mon Jan 23 17:02 - 17:27 (00:24)
Proof: 1327345338.087:41631, 1327346831.548:41702
|
Listing 6 shows an example of auvirt displaying the start and stop event identifiers used to create the record. Using this information, you can extract the raw records from audit logs using the ausearch tool (see Listing 7).
Listing 7. Using ausearch to get raw records (wrapped for clarity)
# ausearch -a 41631 --start 01/23/2012 17:02:00 --end 01/23/2012 17:27:59 ---- time->Mon Jan 23 17:02:18 2012 type=VIRT_CONTROL msg=audit(1327345338.087:41631): user pid=2433 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:initrc_t:s0 msg='virt=kvm op=start reason=booted vm="Fedora" uuid=7bcae45c-c179-6cea-a185-c7abe50520c7: exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success' |
This example uses the -a option,
which restricts the output to records with the given serial
number, and the --start and
--end options that work in the
same way as the auvirt options. Notice that the raw records
contains additional information such as reason, executable name,
and pid.
Given the growing popularity of cloud computing, virtualization has become a prominent technology. It is increasingly important to have a clear understanding of what happens in virtualized environments. In addition, auditing can be a mandatory requirement when dealing with certain industry segments, such as government or financial. This article gives an overview of audit tools that can be used for filtering virtualization-related host audit events. With these tools, you can gain a more complete audit trail for a guest by considering guest events that occur both on the guest and on the host.
Learn
- Check out the Linux Audit website to download the latest version.
- Explore the Libvirt website to get the latest news and to learn more about libvirt.
- In "Guide to the Secure Configuration of Red Hat Enterprise Linux 5" (National Security Agency, September 2010), learn more about security configuration recommendations for the Red Hat Enterprise Linux (RHEL) 5 operating system. The section 2.6.2 describes how to configure and use the Linux Audit Subsystem.
- In the developerWorks Linux zone, find hundreds of how-to articles and tutorials, as well as downloads, discussion forums, and a wealth of other resources for Linux developers and administrators.
- The Open Source developerWorks zone provides a wealth of information on open source tools and using open source technologies.
- developerWorks Web development specializes in articles covering various web-based solutions.
- Stay current with developerWorks technical events and webcasts focused on a variety of IBM products and IT industry topics.
- Attend a free developerWorks Live! briefing to get up-to-speed quickly on IBM products and tools, as well as IT industry trends.
- Watch developerWorks on-demand demos ranging from product installation and setup demos for beginners, to advanced functionality for experienced developers.
- Follow developerWorks on Twitter, or subscribe to a feed of Linux tweets on developerWorks.
Get products and technologies
- Evaluate IBM products in the way that suits you best: Download a product trial, try a product online, use a product in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement Service Oriented Architecture efficiently.
Discuss
- Join the Linux Audit mailing list to discuss or to get questions about the Linux Audit Subsystem answered.
- Check out developerWorks blogs and get involved in the developerWorks community.
- Get involved in the developerWorks community. Connect with other developerWorks users while exploring the developer-driven blogs, forums, groups, and wikis.
Table of contents
Next steps from IBM
Learn more, join the cloud community
