Performance hints and tips summary

Edit online

Tune the performance of your virtual server.

How best to tune a KVM virtual server environment strongly depends on the workloads of the virtual servers and on the available resources. The same measure that enhances performance in one environment can have adverse effects in another. Finding the best balance for a particular setting can be a challenge and often involves experimentation. Use the following suggestions for your consideration.

Use multiple queues for your virtio network interfaces

With multiple virtual CPUs, you can transfer packages in parallel if you provide multiple queues for incoming and outgoing packets. Use the queues attribute of the driver element to configure multiple queues. Specify an integer of at least 2 but not exceeding the number of virtual CPUs of the virtual server.

The following specification configures two input and output queues for a network interface:
<interface type="direct"> 
      <source network="net01"/> 
      <model type="virtio"/> 
      <driver ... queues="2"/> 
</interface>

Multiple queues are designed to provide enhanced performance for a network interface, but they also use memory and CPU resources. Start with defining two queues for busy interfaces. Then, try just one queue for interfaces with less traffic or more than two queues for busy interfaces.

Use the vhost driver for virtio network interfaces

Use the default vhost driver by omitting the name attribute from the driver element or by specifying vhost for the name. Specifying qemu can result in poor performance.

<interface type="direct"> 
      <source network="net01"/> 
      <model type="virtio"/> 
      <driver ... name="vhost"/> 
</interface>

Use I/O threads for your virtual block devices

Two configuration steps are required to make virtual block devices use I/O threads.
  1. You must configure one or more I/O threads for the virtual server.
  2. You must configure each virtual block device to use one of these I/O threads.
The following example uses the specification <iothreads>3</iothreads> to configure three I/O threads, with consecutive decimal thread IDs 1, 2, and 3. The iothread="2" specification for the driver element of the disk device then configures the device to use the I/O thread with ID 2.
<domain>
    <iothreads>3</iothreads>
    ...
    <devices>
        ...
        <disk type="block" device="disk">
            <driver ... iothread="2"/>
            ...
        </disk>
        ...
    </devices>
    ...
</domain>

Threads can increase the performance of I/O operations for disk devices, but they also use memory and CPU resources. You can configure multiple devices to use the same thread. Which mapping of threads to devices yields best results depends on the available resources and the workload.

Start with a small number of I/O threads. Often, a single I/O thread for all disk devices is sufficient. Do not configure more threads than the number of virtual CPUs, and do not configure idle threads.

For a running virtual server, you can use the virsh iothreadadd command to add I/O threads with specific thread IDs.

See also I/O threads.

Exclude the memory balloon device

Unless you need a dynamic memory size, do not define a memory balloon device and ensure that libvirt does not create one for you. Include the following specification as a child of the devices element in your domain configuration-XML.
<memballoon model="none"/>

For more information about the balloon device, see Ballooning.

Disable the cpuset cgroup controller

Note: This setting applies only to KVM hosts with cgroups version 1.
To enable CPU hotplug on the host, disable the cgroup controller.
  1. Open /etc/libvirt/qemu.conf with an editor of your choice.
  2. Find the cgroup_controllers line.
  3. Duplicate this line and from the copy remove the leading number sign (#).
  4. Remove cpuset from the list of values, such that the line reads:
    cgroup_controllers = [ "cpu", "devices", "memory", "blkio", "cpuacct" ]
Restart the libvirtd daemon to make the setting take effect.
  1. Stop all virtual servers.
  2. Issue the following command:
    # systemctl restart libvirtd
  3. Start the virtual servers.

This setting persists across host reboots.

Avoid virtual SCSI devices

Configure virtual SCSI devices only if you need to address the device through SCSI-specific interfaces. In particular, configure disk space as virtual block devices rather than virtual SCSI devices, regardless of the backing on the host.

In the examples that follow you might need SCSI specifics.
  • A LUN for a SCSI-attached tape drive on the host.
  • A DVD ISO file on the host file system that is mounted on a virtual DVD drive.

Configure disk caching

Configure disk devices such that caching is done by the guest and not by the host.

Ensure that the driver element of the disk device includes the specifications cache="none" and io="native".
<disk type="block" device="disk">
    <driver name="qemu" type="raw" cache="none" io="native" iothread="1"/>
    ...
</disk>

Back your virtual servers with huge pages

Unless you need host paging or IBM® Secure Execution for Linux®, back the memory of your virtual servers with huge pages on the host.

Multiple configuration steps are required to use huge pages:
  1. Enable KVM to use huge pages for virtual servers. For example, include the following line in /etc/modprobe.conf/kvm.conf on the host:
    options kvm hpage=1

    To make this setting take effect on a running host, you must temporarily stop all virtual servers and reload the kvm module.

  2. On the host, reserve huge pages by issuing a command like this:
    # sysctl vm.nr_hugepages=30000
    The number of pages depends on the requirements of your virtual servers.
  3. Include the following specification in the domain configuration-XML of your virtual servers:
    <domain type="kvm">
    ...
    <memoryBacking>
        <hugepages/>
    </memoryBacking>
    ...
</domain>

For more information about backing virtual server memory with huge pages, see Using huge pages.

Configure a suitable number of virtual CPUs

Virtual CPUs use host resources. An excessive number of virtual CPUs can adversely affect performance.

For each virtual server, configure just enough virtual CPUs to adequately handle peak loads. Do not configure more virtual CPUs than the number of available host CPUs.

Accommodate growing processing requirements or unusual peaks by configuring virtual CPUs that can be dynamically added to the virtual servers. For example, the following specification in the domain configuration-XML of a virtual server starts the virtual server with 4 virtual CPUs that can then be dynamically increased to up to 8 virtual CPUs.
<domain type="kvm">
    ...
    <vcpu current="4">8</vcpu>
    ...
</domain>

Tune the polling period for idle virtual CPUs

When a virtual CPU becomes idle, KVM polls for wakeup conditions for the virtual CPU before ceding the host resource. The time interval after which polling stops is specified, in nanoseconds, in sysfs at /sys/module/kvm/parameters/halt_poll_ns. During the specified time, 50000 ns by default, polling reduces the wakeup latency for the virtual CPU at the expense of resource usage. Depending on the workload, a longer or shorter time for polling can be advantageous.

To optimize for low CPU consumption, write a small value to /sys/module/kvm/parameters/halt_poll_ns, or write 0 to disable polling.
# echo 0 > /sys/module/kvm/parameters/halt_poll_ns
To optimize for low latency, for example for transactional workloads, write a large value to /sys/module/kvm/parameters/halt_poll_ns, for example:
# echo 80000 > /sys/module/kvm/parameters/halt_poll_ns
Add the following line to /etc/modprobe.d/kvm.conf or change an existing line to make it persistent:
options kvm halt_poll_ns=<polling_halt>
where <polling_halt> is the period, in nanoseconds, after which polling stops.

To make this setting take effect on a running host, you must temporarily stop all virtual servers and reload the kvm module.

Regardless of this setting, KVM disables idle virtual CPUs if contention occurs within KVM or at the LPAR level.

See also the information about halt_poll_ns in KVM Network Performance - Best Practices and Tuning Recommendations [PDF | HTML].

Tune the CPU migration algorithm of the host scheduler

Important: Do not change the scheduler settings unless you are an expert who understands the implications. Do not apply changes to production systems without testing them and confirming that they have the intended effect.

The kernel.sched_migration_cost_ns parameter specifies a time interval in nanoseconds. After the last execution of a task, the CPU cache is considered to have useful content until this interval expires. Increasing this interval results in fewer task migrations. The default value is 500000 ns.

If the CPU idle time is higher than expected when there are runnable processes, try reducing this interval. If tasks bounce between CPUs too often, try increasing it.

To dynamically set the interval to 60000 ns, enter the following command:
# sysctl kernel.sched_migration_cost_ns=60000
Use the following entry in /etc/sysctl.conf, to persistently change the value to 60000 ns:
kernel.sched_migration_cost_ns=60000