Freeing AP queues for KVM guests

7.1 LPAR mode z/VM guest KVM guest

By default, the zcrypt device driver controls all AP queues on a Linux® instance. Free AP queues from zcrypt to make them available to alternative device drivers, such as vfio_ap.

The vfio_ap device driver controls AP queues on behalf of KVM guests. While vfio_ap is the only eligible alternative device driver, freeing an AP queue from zcrypt makes it available to vfio_ap.

Before you begin

Free AP queues only if your Linux instance is a KVM host that needs to provide these AP queues to its KVM guests.

Attention:

Do not change the settings for adapters or domains that are in use or reserved for another exploiter. In particular, do not bring already freed adapters or queues back under control of the zcrypt device driver while they are assigned to a KVM guest.

About this task

Two masks rule which AP queues are controlled by the zcrypt device driver and which are available to alternative device drivers, such as vfio_ap.

Adapter mask: The adapter mask is a 256-bit value, each bit representing a cryptographic adapter. The leftmost bit represents the adapter with ID 0x00. In sysfs, the mask is available as the value of attribute /sys/bus/ap/apmask. If an adapter bit is set to 0, all AP queues on this adapter are available to alternative device drivers, across all domains.
Domain mask: The domain mask is a 256-bit value, each bit representing a cryptographic domain. The leftmost bit represents the domain with ID 0x0000. In sysfs, the mask is available as the value of attribute /sys/bus/ap/aqmask. If a domain bit is set to 0, all AP queues with this domain are available to alternative device drivers, across all adapters.

The sysfs representation of both masks is a big-endian, 64-bit, hexadecimal value. For example

For an adapter mask 0x8000..., the bit for adapter 0x00 is 1 and all others are 0.
For a adapter mask 0xFF00..., bits for adapters 0x00 to 0x07 are 1 and all others are 0.

zcrypt controls all AP queues for which both the adapter bit and the domain bit are set to 1. The default for both masks is 1 for all bits. Hence, the default value for both masks in sysfs is

0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff

and zcrypt is the default device driver for all AP queues. To free an AP queue for alternative device drivers, the corresponding adapter bit, or the corresponding domain bit, or both must be set to 0.

You can use kernel parameters to set the mask. On a running Linux instance, you can write to the respective sysfs attribute to change the masks.

Procedure

Use the following methods to change a mask on a running Linux instance.

Write a new mask value to the sysfs attribute.

You can write a 1 - 64-digit hexadecimal number to the respective sysfs attribute to replace the mask. If fewer than 64 digits are specified, the number is padded with 0s on the right.
Examples:
- To set the bit for the adapters with ID 0x00 and 0x01 to 0 and all other bits to 1, issue the following command:
```
# echo 0x3fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff > /sys/bus/ap/apmask
```
- To set the bit for the domains with ID 0x0000 and 0x0001 to 1 and all other bits to 0, issue the following command:
```
# echo 0xc > /sys/bus/ap/aqmask
```
Set an individual bit value.

You can set an individual bit by specifying the bit-number, counting from the left, with one of the following prefixes:

-

The minus sign (-) sets the bit to 0.

+

The plus sign (+) sets the bit to 1.

Bit numbers can be in decimal or hexadecimal notation. Hexadecimal numbers must be prefixed with 0x.
Examples:
- To set the bit for the adapter with ID 0x01 to 0, issue the following command:
```
# echo -1 > /sys/bus/ap/apmask
```
- To set the bit for the domain with ID 0x000a to 1, issue the following command:
```
# echo +10 > /sys/bus/ap/aqmask
```
  The following equivalent command uses hexadecimal notation:
  # echo +0xa > /sys/bus/ap/aqmask
Note: Do not omit the leading plus (+) or minus (-) sign. Plain numbers that can be interpreted as hexadecimal values replace the entire mask, even if they are specified without the 0x prefix.
Set a range of bit values.

Using the same prefixes as for individual bits, you can specify a range of bit-numbers to set all bits in the range to the same value. Specify a range by specifying the bit numbers of the first and last bit in the range, separated by a hyphen (-).
Examples:
- To set the bit for the adapters with IDs 0x00, 0x01, 0x02, and 0x03 to 0, issue the following command:
```
# echo -0-3 > /sys/bus/ap/apmask
```
- To set the bit for the domains with ID 0x0008, 0x0009, 0x000a, 0x000b, and 0x000c to 1, issue the following command:
```
# echo +8-12 > /sys/bus/ap/aqmask
```
  The following equivalent command uses hexadecimal notation:
  # echo +0x8-0xc > /sys/bus/ap/aqmask
- To set all domains to 0 issue:
```
# echo -0-255 > /sys/bus/ap/aqmask
```
  This command is equivalent to the following hexadecimal notation:
  # echo -0x0-0xff > /sys/bus/ap/aqmask
  The same results can be achieved by replacing the entire mask with 0s with the following command:
  # echo 0x0 > /sys/bus/ap/aqmask
Set multiple bits and bit ranges with a single command.

You can set multiple bits and bit ranges with the same command, by using a comma-separated list of specifications for individual bits and ranges. The expressions are processed individually from left to right.
Examples:
- To set the bit for the adapters with IDs 0x00 to 0, the range from 0x01 to 0x03 to 1, and 0x0a to 1 issue:
```
# echo -0,+1-3,+10 > /sys/bus/ap/apmask
```
  The following equivalent command uses hexadecimal notation:
  # echo -0x0,+0x1-0x3,+0xa > /sys/bus/ap/apmask
  This command is equivalent to the following sequence of commands:
  # echo -0 > /sys/bus/ap/apmask # echo +1-3 > /sys/bus/ap/apmask # echo +10 > /sys/bus/ap/apmask
- To first set the bits for all domains to zero, then set the range 0x0010 to 0x001f to 1, and then set 0x0014 to 0, issue the following command:
```
# echo -0-255,+16-31,-20 > /sys/bus/ap/aqmask
```
  The following equivalent command uses hexadecimal notation:
  # echo -0x0-0xff,+0x10-1f,-0x14 > /sys/bus/ap/aqmask
  This command is equivalent to the following sequence of commands:
  # echo -0-255 > /sys/bus/ap/aqmask # echo +16-31 > /sys/bus/ap/aqmask # echo -20 > /sys/bus/ap/aqmask
  The resulting bit mask can be read from the sysfs attribute:
  # cat /sys/bus/ap/aqmask 0x0000f7ff00000000000000000000000000000000000000000000000000000000

Results

You can check your results by reading the masks from the sysfs attributes. In the following example, adapter 0x01, across all domains, and domains 0x0001 and 0x0002, across all adapters, are ignored by the zcrypt device driver and thus, free for alternative device drivers.

# cat /sys/bus/ap/apmask
0xbfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
# cat /sys/bus/ap/aqmask
0x9fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff