General system settings

The following topics describe the general operating system settings used for the testing.

randomize_va_space

This setting controls Address Space Layout Randomization (ASLR). ASLR is a security features that can help protect against certain types of buffer overflow attacks by randomizing the base address of code, stack, heap and other program elements to prevent attacks at known, predicted or assumed locations. This feature has been available since kernel 2.6.12 back in 2005.

ASLR is configurable and the following values are supported:

0: No randomization. Everything is static.
1: Conservative randomization. Shared libraries, stack, mmap(), VDSO and heap are randomized.
2: Full randomization. In addition to elements listed in the previous point, memory managed through brk() is also randomized.

The default setting adopted by most Linux® distributions is full randomization (2). Changing the setting to no randomization (0) permits the greatest chance of sharing and enhanced cache efficiency. This setting was changed to reduce the variations of the measurement results in a controlled lab environment. Users should consider the security implications when changing this setting in their environments.

Note: It is not recommended to change this parameter in a production environment.

Kernel Same-page Merging

Another action that was taken was to disable Kernel Same-page Merging (KSM). Based on prior experience where processor utilization is a critical resource and in the absence of a memory over-commitment scenario, allowing KSM to run serves to introduce overhead in the form of breaking down Transparent Huge Pages (THP) and consuming CPU cycles when scanning memory. Since there is no need for memory over-commitment for the purposes of this paper, the service which runs KSM was disabled, which can be done by executing the following commands:

[root@kvmhost ~] # chkconfig ksm off
[root@kvmhost ~] # service ksm stop
Stopping ksm: [ OK ]

This setting was changed to reduce the variations of the measurement results in a controlled lab environment.

Note: It is not recommended to change this parameter in a production environment.

Receive Packet Steering

Modern system configurations rely heavily on network connectivity for many functions. Additionally, newer network components are getting increasingly faster and faster. This results in an ever increasing load on the OS and subsystems.

While newer processors increase capacity, much of these gains are due to additional cores rather than more powerful cores. In order to keep pace with the growing network requirements and load, many newer network adapters support multiple send and receive queues and can use multiple cores to process these queues concurrently.

For adapters that do not support multiple send / receive queues, there remains a desire and need to distribute large/high network loads across multiple processor cores. To address this, the feature Receive Packet Steering (RPS) was developed.

RPS uses a hash algorithm, based on packet IP addresses and ports, to distribute received network traffic across multiple cores. The hash ensures packets for the same data stream are processed on the same CPU.

RPS is specified and configured separately for each network device. Each device has a setting in sysfs. The location of the settings is:

/sys/class/net/<device/queues/rx-<queue#>/rps_cpus

where:

<device>: specifies the actual name of the interface device
rx-<queue#>: represents the network queue number being set

and

An example output of this value might be:

[root@kvmhost ~] # cat /sys/class/net/eth0/queues/rx-0/rps_cpus 
0000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000

The setting is comma-delimited bitmap of CPUs. The CPU numbers are from 0 to maxcpus-1. CPU 0 is the rightmost or low order bit. The default value of all bits being 0 (off) means to RPS being disabled. Each number in the bitmap is a hex value and specifies four CPU bit locations. To enable RPS for 1 or more CPUs, the individual bitmask for the selected CPUs must be set to 1. For example, if CPUs 0-3 are to be enabled for RPS, the following invocation would be used:

[root@kvmhost ~] # echo 0xf > /sys/class/net/eth0/queus/rx-0/rps_cpus

Display the resulting bitmap:

[root@kvmhost ~] # cat /sys/class/net/eth0/queues/rx-0/rps_cpus 
0000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,0000000F

The benefits realized from setting RPS bitmap will vary depending on workload characteristics. Based on measurements conducted on our KVM test systems, setting the bitmap value corresponding to the total number of CPUs configured for the KVM host has obtained the best overall improvement.

This setting is recommended for the KVM host only.

Sharing an OSA adapter across multiple LPARs

IBM Z systems are designed to share many system resources across multiple LPARs configured on the system. This is true for CPUs, memory, IO and other adapters.

The tests conducted for this paper have shown that while an OSA adapter is easily shared across several LPARs, when multiple LPARs are driving extremely high loads using a shared OSA adapter, that the peak network performance is less than when each LPAR is using a separate OSA adapter.

To obtain the highest level of network performance across multiple LPARs running KVM guests on a single system, it is recommended that each KVM host LPAR be configured to use a separate OSA adapter.