Setting up cryptographic adapter resources for VFIO pass-through
Free cryptographic adapter resources from host control and assign them to a VFIO AP mediated device.
AP queues
Cryptographic adapter resources are managed as AP queues (see Cryptographic domains). An AP queue corresponds to a specific cryptographic domain on a specific cryptographic adapter.

In Figure 1, the grey table cells indicate the implicitly assigned
AP queues, which are 00.0000
, 00.0002
, 00.0003
,
02.0000
, 02.0002
, and 02.0003
.
Multi-dimensional arrays, like the two-dimensional table of Figure 1, are often called matrices. Therefore, the term matrix is widely used for representations of tables of AP queues, for example, in sysfs attributes.
- LPAR level
- AP queues that are available to the KVM host.
For hosts that run in LPAR mode, these are the AP queues that result from the AP configuration for the LPAR. On a running host, this matrix can change through dynamic changes of the adapter and domain configurations for the LPAR.
- Host level
- AP queues that are controlled by the host's zcrypt device drivers and so can be accessed by
applications on the host.
By default, all queues that are available to the host are controlled by the host's
zcrypt
device drivers. You can change this default with theap.apmask=
andap.aqmask=
kernel parameters, see Kernel parameters. You can dynamically change this matrix by adding or removing adapters and domains, for example, with the chzdev command.If the
vfio_ap
device driver is loaded, it takes control of the AP queues that are not controlled by the host'szcrypt
device drivers. These AP queues can then be passed on to KVM guests through AP mediated devices. - Mediated device level
- AP queues that are assigned to a VFIO AP mediated device.
You make AP queues available to a KVM guest by attaching a VFIO AP mediated device to the KVM guest.
A mediated device cannot include queues that, at the host level, are assigned to the host's
zcrypt
device drivers. Still, it can include AP queues that are unavailable because they are associated with adapters that are not physically plugged into the hardware. Other queues might not be available because the associated adapter, domain, or both are not configured for the LPAR. As a consequence, attaching a mediated device to a KVM guest might not make all queues of the mediated device matrix available to the KVM guest.Read the /sys/devices/vfio_ap/matrix/<uuid>/matrix attribute of a mediated device to list all AP queues that are assigned to it. Read the /sys/devices/vfio_ap/matrix/<uuid>/guest_matrix to list only queues that can be used by KVM guests. In the paths, <uuid> is the UUID that identifies the mediated device.
You can dynamically change the adapter and domain assignment through a mediated device's sysfs attributes, see Table 1.
Figure 2 is a schematic illustration of the three matrix layers as a succession of filters, where grey cells indicate assigned AP queues. An AP queue can be passed on to a KVM guest only if it is suitably assigned at each level.

Hotplug and hot unplug
On a running KVM guest, attaching a mediated device
results in hotplug events for all those AP queues of the mediated device that are controlled by the
vfio_ap
device driver on the host, which implies that both their adapter and domain
is configured for the LPAR.
- Dynamic changes to the AP configuration for the LPAR on the SE or HMC
- Dynamic changes of the mediated device
Detaching a mediated device results in hot unplug events for all AP queues of the mediated device that are used by the KVM guest.
Persistence across host reboots
You can set up your cryptographic resources such that, after a host reboot, your mediated devices are ready to be attached to
KVM guests, without further configuration steps. This setup
requires a persistent assignment of AP queues to the vfio_ap
device driver and
persistent mediated devices with their assignment of AP queues.
Use the chzdev command to persistently assign AP queues to the
vfio_ap
device driver. Use the nodedev-define command to make a
mediated device and its assignment of AP queues persistent.
Procedure
- Ensure that the vfio_ap device driver is loaded. If it is compiled as a separate module, you
might have to load it yourself. The following command loads vfio_ap and any additional modules it
requires.
# modprobe vfio_ap
- Assign AP queues to the
vfio_ap
device driver.Two 256-bit masks, one for adapters and the other for domains, rule which AP queues are controlled by the zcrypt device driver. Unless you change the initial setting with the
ap.apmask=
andap.aqmask=
kernel parameters, both masks have all bits on by default. With all bits set in the masks, all AP queues that are available to the host can be accessed only by the host itself.Setting the bit for a particular adapter to 0 frees all queues of that adapter from the host and makes them available for use by vfio_ap. Similarly, setting the bit for a particular domain to 0 frees all queues of that domain, across all adapters, from the host and makes them available for use by vfio_ap.
Set bits for adapters or domains or both to
0
to assign queues to thevfio_ap
device driver.- Using the chzdev command
- The preferred method for changing the masks is the chzdev command with the --type ap option. With the chzdev command, you can change the masks persistently, for the active configuration, or both. For details, see KVM Virtual Server Management, SC34-2752.
- Writing to the masks in sysfs
- As a non-persistent fallback method, you can directly write to the masks in sysfs, see Freeing AP queues for KVM guests.
- Create and configure a mediated device.
- Using libvirt
- The preferred method for creating and configuring a mediated device is using the virsh nodedev-define command and a node-device XML description. With this command, you can create a persistent mediated device, see KVM Virtual Server Management, SC34-2752.
- Using general Linux commands and sysfs
- As a non-persistent fallback method, you can create VFIO mediated device for the
vfio_ap
device driver by writing a universally unique identifier (UUID) to /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/create.You then configure the mediated device by writing to the attributes within the sysfs representations of VFIO AP mediated device. VFIO AP mediated devices are represented as subdirectories of /sys/devices/vfio_ap/matrix. The directory names match the UUIDs that identify the mediated devices.
Use the attributes in a device directory to configure the device and to obtain information about the device.
Table 1. sysfs attributes of VFIO mediated devices for cryptographic adapter resources Attribute Explanation assign_adapter Write an adapter ID to this attribute to assign the adapter to the mediated device. Specify the adapter ID in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x
.Example:# echo 0x0a > assign_adapter
assign_control_domain Write a domain ID to this attribute to assign the domain as a control domain to the mediated device. Assign a control domain for each usage domain that you assign to the mediated device, so that you can manage your domains from the guest that uses the mediated device. Specify the domain ID in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x
.Example:# echo 0x001b > assign_control_domain
assign_domain Write a domain ID to this attribute to assign a usage domain to the mediated device. Specify the domain ID in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x
.Example:# echo 0x001b > assign_domain
control_domains Read this attribute to list the assigned control domains. Example:# cat control_domains 001b
guest_matrix Read this attribute to list the subset of assigned AP queues that are eligible for KVM guests. matrix Read this attribute to list the assigned AP queues that result from the adapter and domain assignments. Example:# cat matrix 0a.001b
mdev_type Symbolic link that points to the vfio_ap-passthrough directory. remove Write 1
to this attribute to remove the mediated device.Example:# echo 1 > remove
subsystem Symbolic link that points to the matrix bus. unassign_adapter Write an adapter ID to this attribute to remove the adapter from the mediated device. Specify the adapter ID in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x
.Example:# echo 0x0a > unassign_adapter
unassign_control_domain Write a domain ID to this attribute to remove the domain from the control domains of the mediated device. Specify the domain ID in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x
.Example:# echo 0x001b > unassign_control_domain
unassign_domain Write a domain ID to this attribute to remove a the domain from the usage domains of the mediated device. Specify the domain ID in decimal or hexadecimal notation. For hexadecimal notation, use the prefix 0x
.Example:# echo 0x001b > unassign_domain
Scenario: Fallback method for creating and configuring a mediated device
This scenario assumes a KVM host on which 8 AP queues have been freed from control of the zcrypt device driver. A matrix of two of these queues are to be assigned to a mediated device.
- Load the required modules.
# modprobe vfio_ap
- List the eligible AP queues. AP queues are eligible only if they are controlled by the vfio_ap
device driver.
The output shows that 8 AP queues are eligible. The eight queues correspond to a matrix of two adapters,# lszcrypt -V | grep vfio 00.0001 CEX7A Accelerator online 0 0 13 08 -MC-A-NF- vfio_ap 00.0002 CEX7A Accelerator online 0 0 13 08 -MC-A-NF- vfio_ap 00.0004 CEX7A Accelerator online 0 0 13 08 -MC-A-NF- vfio_ap 00.001b CEX7A Accelerator online 0 0 13 08 -MC-A-NF- vfio_ap 0a.0001 CEX7P EP11-Coproc online 0 0 13 08 -----XNF- vfio_ap 0a.0002 CEX7P EP11-Coproc online 0 0 13 08 -----XNF- vfio_ap 0a.0004 CEX7P EP11-Coproc online 0 0 13 08 -----XNF- vfio_ap 0a.001b CEX7P EP11-Coproc online 0 0 13 08 -----XNF- vfio_ap
0x00
and0x0a
and four domains,0x0001
,0x0002
,0x0004
, and0x001b
. These adapters and domains are the only ones that you can assign to the mediated device. - Create a mediated device.
# uuidgen 4b0518fd-9237-493f-93c8-c5597f8006a3 # echo 4b0518fd-9237-493f-93c8-c5597f8006a3 \ > /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/create
- Add adapters
0x00
and0x0a
to the device.
Reading the matrix attribute does not yield any AP queues. To assign AP queues both adapters and domains must be added.# echo 0x00 > /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/assign_adapter # echo 0x0a > /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/assign_adapter # cat /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/matrix 00. 0a.
- Add domain
0x001b
.
The content of the matrix attribute shows that two AP queues are assigned to the mediated device. The two queues correspond to a matrix of two adapters,# echo 0x001b > /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/assign_domain # cat /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/matrix 00.001b 0a.001b # cat /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/control_domains
0x00
and0x0a
and one domain,0x001b
. Reading the control_domains attribute shows that no control domain is configured. - Add domain
0x001b
to the control domains.# echo 0x001b > /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/assign_control_domain # cat /sys/devices/vfio_ap/matrix/4b0518fd-9237-493f-93c8-c5597f8006a3/control_domains 001b