Hot-plugging Crypto Express adapters to a KVM SEL guest

Read how to dynamically add or remove APQNs on a Crypto Express adapter for a KVM SEL guest.

About this task

Video 1:00:00****

Draft comment: hartig@de.ibm.com
Also see "dynamic AP Pass-through demonstration" in file a96lp03_scenario.txt:

With IBM® zSystems earlier than IBM z16®, a mediated device used to be a static container holding all the APQNs that are available to a KVM guest (regular or SE). That means, to change the APQN configuration of a KVM guest, you needed to stop the guest, unassign the mediated device from the guest, change the mediated device, re-assign the mediated device to the guest, and restart the guest again or you could attach or detach the mediated device as a complete entity to or from the KVM guest with the --live option, as shown in the Results section of Assign a mediated device to a KVM guest or KVM SEL guest. With all prerequisites fulfilled, you can dynamically add or remove APQNs from that container while the guest is running.

Procedure

Remove APQNs from the KVM SEL guest by writing the ID of the desired adapter to the unassign_adapter attribute of the matrix of the mediated device. In this scenario, we dynamically remove (unassign) the cryptographic adapter 0f(hex) = 15(dec) from the guest.

Example on the host:

admin@sel_host: # cd sys/devices/vfio_ap/matrix/01010101-0101-0101-0101-010101010101

admin@sel_host:/sys/devices/vfio_ap/matrix/01010101-0101-0101-0101-010101010101# cat matrix
0f.0014
27.0014
28.0014
admin@sel_host:/sys/devices/vfio_ap/matrix/01010101-0101-0101-0101-010101010101# echo 15 >unassign_adapter
admin@sel_host:/sys/devices/vfio_ap/matrix/01010101-0101-0101-0101-010101010101# cat matrix  
27.0014
28.0014

APQN configuration on the KVM SEL guest before removing:

user@sel_guest:/$ lszcrypt -V
CARD.DOM TYPE  MODE        STATUS   ...   ...   HWTYPE QDEPTH FUNCTIONS  DRIVER      SESTAT
--------------------------------------------------------------------------------------------------------
0f       CEX8A Accelerator online                   14     08 -MC-A-N-F- cex4card    -
0f.0014  CEX8A Accelerator online                   14     08 -MC-A-N-F- cex4queue   bound
27       CEX8P EP11-Coproc online                   14     08 -----XN-F- cex4card    -
27.0014  CEX8P EP11-Coproc online                   14     08 -----XN-F- cex4queue   unbound
28       CEX8P EP11-Coproc online                   14     08 -----XN-F- cex4card    -
28.0014  CEX8P EP11-Coproc online                   14     08 -----XN-F- cex4queue   unbound

Result on the KVM SEL guest after removal:

user@sel_guest:/$ lszcrypt -V

CARD.DOM TYPE  MODE        STATUS   ...   ...   HWTYPE QDEPTH FUNCTIONS  DRIVER      SESTAT
----------------------------------------------------------------------------------------------
27       CEX8P EP11-Coproc online                   14     08 -----XN-F- cex4card    -
27.0014  CEX8P EP11-Coproc online                   14     08 -----XN-F- cex4queue   unbound
28       CEX8P EP11-Coproc online                   14     08 -----XN-F- cex4card    -
28.0014  CEX8P EP11-Coproc online                   14     08 -----XN-F- cex4queue   unbound

Add an APQN, by writing the ID of the desired adapter to the unassign_adapter attribute of the matrix of the mediated device.

Example on the host:

admin@sel_host:/sys/devices/vfio_ap/matrix/01010101-0101-0101-0101-010101010101# 
echo 15 >assign_adapter  
admin@sel_host:/sys/devices/vfio_ap/matrix/01010101-0101-0101-0101-010101010101# 
cat matrix
0f.0014
27.0014
28.0014

Result on the KVM SE guest after adding:

user@sel_guest:/$ lszcrypt -V
CARD.DOM TYPE  MODE        STATUS   ...   ...   HWTYPE QDEPTH FUNCTIONS  DRIVER      SESTAT
---------------------------------------------------------------------------------------------
0f       CEX8A Accelerator online                    14     08 -MC-A-N-F- cex4card    -
0f.0014  CEX8A Accelerator online                    14     08 -MC-A-N-F- cex4queue   unbound
27       CEX8P EP11-Coproc online                    14     08 -----XN-F- cex4card    -
27.0014  CEX8P EP11-Coproc online                    14     08 -----XN-F- cex4queue   unbound
28       CEX8P EP11-Coproc online                    14     08 -----XN-F- cex4card    -
28.0014  CEX8P EP11-Coproc online                    14     08 -----XN-F- cex4queue   unbound

You see, that the APQN configuration after adding this adapter is the same as shown in #kvm_se_guest_setup__ex_guest_before_remove (before the removal of this adapter).

What to do next

You can also dynamically add or remove a domain from all adapters:

Example on the host:

admin@sel_host:/sys/devices/vfio_ap/matrix/01010101-0101-0101-0101-010101010101# cat matrix
0f.0014
0f.0015
27.0014
27.0015
28.0014
28.0015
admin@sel_host:/sys/devices/vfio_ap/matrix/01010101-0101-0101-0101-010101010101# echo 21 >unassign_domain
admin@sel_host:/sys/devices/vfio_ap/matrix/01010101-0101-0101-0101-010101010101# cat matrix
0f.0014
27.0014
28.0014
admin@sel_host:/sys/devices/vfio_ap/matrix/01010101-0101-0101-0101-010101010101#

Result of the example on the guest:

user@sel_guest:/$ lszcrypt -V
CARD.DOM TYPE  MODE        STATUS  ...   ...   HWTYPE QDEPTH FUNCTIONS  DRIVER      SESTAT
-------------------------------------------------------------------------------------------
0f       CEX8A Accelerator online                   14     08 -MC-A-N-F- cex4card    -
0f.0014  CEX8A Accelerator online                   14     08 -MC-A-N-F- cex4queue   unbound
27       CEX8P EP11-Coproc online                   14     08 -----XN-F- cex4card    -
27.0014  CEX8P EP11-Coproc online                   14     08 -----XN-F- cex4queue   unbound
28       CEX8P EP11-Coproc online                   14     08 -----XN-F- cex4card    -
28.0014  CEX8P EP11-Coproc online                   14     08 -----XN-F- cex4queue   unbound