Monitoring CPACF activity

Edit online

Use the processor activity instrumentation counter (PAI counter) to monitor CPACF activity in OpenSSL or use the cpacfinfo utility to retrieve information about CPACF functions and instructions installed on the system.

The IBM® z16™ and IBM LinuxONE 4 systems support a processor activity instrumentation counter (PAI counter) to monitor CPACF usage in greater detail than the basic CPACF counters. Furthermore, PAI counters are available on LPAR and VM guests. The cpacfstats utility has support for PAI since version 2.19. Furthermore, the pai_crypto event source must be available in the kernel. To check, run the following command: 

$ ls -d /sys/bus/event_source/devices/pai_crypto/

You should see a directory and not an error. Otherwise, the currently running kernel does not support PAI. With PAI supported and cpacfstats version 2.19 or later, you can enable PAI for user space mode by issuing the following command:

$ cpacfstats -e pai_user

This enables all PAI counters for the user space. However, the counters are global and not localized to one particular application.

Note: Activated PAI counters influence the CPACF performance. They should only be activated when really needed.

To print all non-zero counters, issue the following command:

$ cpacfstats -p pai_user -n

To reset all counters, issue: 

$ cpacfstats -r pai_user -n
Note: Even after a reset, you might see non-zero counters. These counters stem, among others, from the active SSH connection. One way to isolate the system, is to set the cipher used for your SSH connection to chacha20-poly1305 which does not cause any counter increments.

Once counters are no longer needed, deactivate them with the following command:

$ cpacfstats -d pai_user

OpenSSL and PAI:

Considerable amounts of optimization went into the cryptographic algorithms contained in libcrypto for Linux® on IBM Z. Most of the work exploits CPACF and can be monitored via PAI.

AES CBC
This cipher block chaining uses the KMC CPACF instruction. Depending on the used key size, you see increments to the following counters:
  • KMC AES 128bit
  • KMC AES 192bit
  • KMC AES 256bit
AES CCM
The CCM mode exploits the KM, KMA, and KMAC CPACF instructions. Unfortunately the usage of KMA again does not fall into the AEAD fashion required for PAI to count invocation of this instruction. So you only see increments to the following counters:
  • KM AES 128bit
  • KM AES 192bit
  • KM AES 256bit
  • KMAC AES 128bit
  • KMAC AES 192bit
  • KMAC AES 256bit
AES CFB
Cipher feedback mode comes in various flavors. You can either feed back 1 bit, 8 bits, or 128 bits. For 1 bit feedback, only KM is used. Depending on the used key size, you only see increments to the following counters:
  • KM AES 128bit
  • KM AES 192bit
  • KM AES 256bit

For 8 bits or 128 bits feedback, depending on the key size used, you see increments to the following counters:

  • KMF AES 128bit
  • KMF AES 192bit
  • KMF AES 256bit

Furthermore, if EVP_EncryptUpdate or EVP_DecryptUpdate is called with an input size that is not a multiple of the AES block size (16 byte), depending on the used key size, you also see increments to the following counters:

  • KM AES 128bit
  • KM AES 192bit
  • KM AES 256bit
AES CTR
This counter mode is implemented in different ways depending on the size of the input. If the input is not more than 256 bytes, this counter mode operations increments the following counters:
  • KM AES 128bit
  • KM AES 192bit
  • KM AES 256bit
AES ECB
This mode only uses the KM CPACF instruction. Depending on the used key size, it increments the following counters:
  • KM AES 128bit
  • KM AES 192bit
  • KM AES 256bit

For larger input buffer, OpenSSL uses the KMA CPACF instruction. This instruction only counts its usage if used in AEAD fashion. Since this counter mode does not perform AEAD, no counters are incremented.

AES GCM
This Galois Counter Mode uses the KMA CPACF instruction in AEAD fashion. Furthermore, if EVP_EncryptUpdate or EVP_DecryptUpdate is called with plain text respective ciphertext that is not a multiple of the AES block size (16 bytes), then KM is used, too. So depending on your use case and the used key size, you might see the following counter increments:
  • KM AES 128bit
  • KM AES 192bit
  • KM AES 256bit
  • KMA AES 128bit
  • KMA AES 192bit
  • KMA AES 256bit
Note: Typically, SSH connections use AES GCM. So you see some increments of KMA AES 256bit due to SSH connections. It is possible to chose a different cipher when connecting to the target system. Check the SSH documentation (for example, ssh -Q cipher) for a list of available ciphers.

When using the command openssl speed to test GCM mode, remember to request AEAD operations via the -aead option. Otherwise, no counter is incremented.

AES OFB
This output feedback (OFB) mode uses KM and KMO CPACF instructions. You see increments to the following counters depending on the used key size:
  • KM AES 128bit
  • KM AES 192bit
  • KM AES 256bit
  • KMO AES 128bit
  • KMO AES 192bit
  • KMO AES 256bit
AES XTS
The XTS implementation in OpenSSL uses the KM CPACF instruction. Depending on the used key size, you see updates to the following counters:
  • KM AES 128bit
  • KM AES 256bit
  • KM AES-XTS 128bit
  • KM AES-XTS 256bit
  • KM AES-XTS (FULL) 128bit (as of MSA 10 on IBM z17™ and IBM LinuxONE 5)
  • KM AES-XTS (FULL) 256bit (as of MSA 10 on IBM z17 and IBM LinuxONE 5)
Random Number Operations
The default pseudo random number generator of OpenSSL uses AES counter mode. You see increments to the following counter:
  • KM AES 256bit
SHA hashing
SHA hashing on IBM Z® in OpenSSL uses the KIMD CPACF instruction. Depending on the algorithm, you see increments to the following counters:
  • KIMD SHA1
  • KIMD SHA256
  • KIMD SHA512
Note: SHA224 is counted as SHA256 and SHA384 is counted as SHA512.
SHA3 Hashing
SHA3 implementations use KIMD and KLMD CPACF instructions. Depending on the used algorithm, you see increments for the following counters:
  • KIMD SHA3-224
  • KIMD SHA3-256
  • KIMD SHA3-384
  • KIMD SHA3-512
  • KLMD SHA3-224
  • KLMD SHA3-256
  • KLMD SHA3-384
  • KLMD SHA3-512
SHAKE Hashing
SHAKE uses KIMD and KLMD CPACF instructions. Depending on the used algorithm, you see increments for the following counters:
  • KIMD SHAKE 128
  • KIMD SHAKE 256
  • KLMD SHAKE 128
  • KLMD SHAKE 256
GHASH
Computing a GHASH uses the KLMD CPACF instruction. You see increments for the following counter:
  • KIMD GHASH
Note: GHASH is mostly used inside GCM mode which does not increment this counter if the KMA instruction is available.
HMAC (as of MSA 11 on IBM z17 and IBM LinuxONE 5)
HMAC uses the KMAC CPACF instruction. Depending on the used algorithm, you see increments for the following counters:
  • KMAC HMAC SHA 224
  • KMAC HMAC SHA 256
  • KMAC HMAC SHA 384
  • KMAC HMAC SHA 512
EC Key-Pair Generation
For generation of EC key pairs for the NIST curves P256, P384, and P521, depending on the curve, you see increments for the following counters:
  • PCC Scalar Mult P256
  • PCC Scalar Mult P384
  • PCC Scalar Mult P521
Curve25519 and Curve448 key pair generation
For generation of Key-Pairs for Curve25519 or Curve448 (either in Edwards or Montgomery form), depending on the curve and desired form, you see updates for the following counters:
  • PCC Scalar Mult Ed25519
  • PCC Scalar Mult Ed448
  • PCC Scalar Mult X25519
  • PCC Scalar Mult X448
ECDH, X25519, X448
For key derivation via ECDH, X25519, or X448, depending on the curve used in the derivation, you see increments to the following counters:
  • PCC Scalar Mult P256
  • PCC Scalar Mult P384
  • PCC Scalar Mult P521
  • PCC Scalar Mult X25519
  • PCC Scalar Mult X448
ECDSA
For ECDSA sign and verify operations over the NIST curves P256, P384, and P521, depending on the curve and operation, you see increments for the following counters:
  • KDSA ECDSA Verify P256
  • KDSA ECDSA Verify P384
  • KDSA ECDSA Verify P521
  • KDSA ECDSA Sign P256
  • KDSA ECDSA Sign P384
  • KDSA ECDSA Sign P521
Note: You see counter increments also for signatures that fail with verification in a non-trivial way. If, however, OpenSSL does not need any cryptographic operation to see that a given signature is invalid (for example, because it is obviously of wrong size), then no counter is incremented.
EdDSA
For EdDSA operations over ED25519 and ED448, depending on the curve and operation, you see increments for the following counters:
  • KDSA EdDSA Verify Ed25519
  • KDSA EdDSA Verify Ed448
  • KDSA EdDSA Sign Ed25519
  • KDSA EdDSA Sign Ed448
Note: You see counter increments also for signatures that fail with verification in a non-trivial way. If, however, OpenSSL does not need any cryptographic operation to see that a given signature is invalid (for example, because it is obviously of wrong size), then no counter is incremented.

Using the cpacfinfo utility

The cpacfinfo tool, which you can use to retrieve information about CPACF functions and instructions installed on the system, is available as part of the s390-tools package.

Among others, you can see which function (function code) is available for which instruction and thus can be used by OpenSSL for acceleration. For example, if the output of cpacfinfo --functions contains the line 
( 18) [    AVAILABLE] KMCTR-AES-128
this means that for the AES algorithms in counter mode 128 bit keys can be used by OpenSSL for acceleration.

Also, using this tool with the --msa option (cpacfinfo -m/--msa) lets you find out the MSA levels available on the system.

View an excerpt of an output example with a combination of the --msa and the --functions options, which shows you, which functions are available for which MSA level:

$ cpacfinfo --msa --functions
MSA    STFLE bit [  17 ] :     AVAILABLE ( 16 / 16 functions available )
        Cipher Message (KM)
                (  0) [    AVAILABLE] KM-Query
                (  1) [    AVAILABLE] KM-DEA
                (  2) [    AVAILABLE] KM-TDEA-128
                (  3) [    AVAILABLE] KM-TDEA-192
        Cipher Message with Chaining (KMC)
                (  0) [    AVAILABLE] KMC-Query
                (  1) [    AVAILABLE] KMC-DEA
                (  2) [    AVAILABLE] KMC-TDEA-128
                (  3) [    AVAILABLE] KMC-TDEA-192
        Compute Intermediate Message Digest (KIMD)
                (  0) [    AVAILABLE] KIMD-Query
                (  1) [    AVAILABLE] KIMD-SHA-1
        Compute Last Message Digest (KLMD)
                (  0) [    AVAILABLE] KLMD-Query
                (  1) [    AVAILABLE] KLMD-SHA-1
        Compute Message Authentication Code (KMAC)
                (  0) [    AVAILABLE] KMAC-Query
                (  1) [    AVAILABLE] KMAC-DEA
                (  2) [    AVAILABLE] KMAC-TDEA-128
                (  3) [    AVAILABLE] KMAC-TDEA-192
MSA  1                   :     AVAILABLE (  5 / 5  functions available )
        ...
MSA  2                   :     AVAILABLE (  6 / 6  functions available )
...
...
MSA 13                   :     AVAILABLE ( 13 / 13 functions available )
        Cipher Message (KM)
                (127) [    AVAILABLE] KM-Query-Authentication-Information
        Cipher Message with Chaining (KMC)
                (127) [    AVAILABLE] KMC-Query-Authentication-Information
        Compute Intermediate Message Digest (KIMD)
                (127) [    AVAILABLE] KIMD-Query-Authentication-Information
        ...

On IBM z17 and IBM LinuxONE 5 machines, this tool exploits the query authentication function per CPACF instruction introduced with MSA 13. For a detailed description of the cpacfinfo utility, refer to the Device Drivers documentation or to the cpacfinfo man page.