Attestation

Edit online

Attestation is a process that starts by default when the IBM Confidential Computing Containers for Red Hat OpenShift Container Platform (CCCO) virtual machine (VM) is created. It ensures that the CCCO VM image is indeed built by IBM, and that it was not modified. This process also provides information and allows validation of any data that is provided to the CCCO VM at the time of workload deployment.

When you create a CCCO VM by using the IBM CCCO image, the image uses an initial file system that is protected by encryption and signed by IBM Secure Execution (SE). For more information, see Confidential computing with LinuxONE.

To know more about the attestation process, see this video. This video shows the attestation process. The IBM CCCO for Red Hat OpenShift Container Platform uses the similar approach.

The boot process creates a unique root disk encryption key to ensure protection of the root disk. To perform attestation, the IBM CCCO VM image contains an attestation-signing key and the hash of the root partition at build time. The boot process validates the root partition. If the hash of the root partition does not match, the boot process does not continue because it assumes that the image was modified before boot. The attestation signing key is a random RSA 4 K key that is signed by an IBM root key that is maintained in Confidential Computing Containers Crypto Services. The IBM root key is signed by Digicert.

During deployment of the IBM CCCO VM, an attestation record is created. It contains hashes of the following items:

The original base image
The root partition at the moment of the first boot
The root partition at build time

The attestation record is signed by the attestation key. As an extra protection layer, you can provide a public key during deployment, against which the attestation record is encrypted. The hash of this public key is added to the attestation record to ensure that the record can be viewed only by the compliance authority, and the expected authority can be easily identified through that hash.

After a workload is brought up using IBM CCCO, you can validate the attestation records within the created pod. Workload running on IBM CCCO VM will have the records added to /var/hyperprotect directory. Perform the following procedures for further verifications:

The attestation record is signed by the attestation signing key.
The attestation signing key can be confirmed by the IBM intermediate certificate. The IBM intermediate certificate is signed by DigiCert, which is proven by the root certificate of DigiCert, thus completing the chain of trust.

The encryption and attestation certificates are signed by the IBM intermediate certificate and this has been signed by the IBM Digicert intermediate cert (which in turn is signed by DigiCert Trusted Root G4). For more information about the certificates, see DigiCert Trusted Root Authority Certificates.

Use the following procedure to validate the attestation record and hashes:

Obtain the attestation record se-checksums.txt and the signature file se-signature.bin from your IBM CCCO VM. To do so, you can implement your container to provide the attestation record and the signature file. The attestation record and the signature file are made available to your container in the /var/hyperprotect directory.
Get the IBM attestation certificate that is a part of the image TAR file.
Validate the attestation certificate by following the instructions here.

Extract the attestation public key from the attestation certificate by using the following command:

 openssl x509 -pubkey -noout -in ibm-confidential-computing-container-attestation.crt > contract-public-key.pub

Verify the signature of the attestation record:
```
openssl sha256 -verify contract-public-key.pub -signature se-signature.bin se-checksums.txt
 
```
Signature verification must be done on a decrypted attestation file.
You can now use the hashes from the attestation record for validation.

In case you provided a public key for encrypting the attestation record, the following script might help in decrypting the record. For information about attestation public key, see Preparing the Attestation section.

#!/bin/bash
#
# Example script to decrypt attestation document.
#
# Usage:
#   ./decrypt-attestation.sh mykey.pem [file]
#
# Token Format:
#   contract-basic.<ENC_AES_KEY_BASE64>.<ENC_MESSAGE_BASE64>


RSA_PRIV_KEY="$1"
if [ -z "$RSA_PRIV_KEY" ]; then
    echo "Usage: $0 <rsa-priv-key.pem>"
    exit 1
fi
INPUT_FILE="${2:-se-checksums.txt.enc}"
TMP_DIR="$(mktemp -d)"
#trap 'rm -r $TMP_DIR' EXIT


PASSWORD_ENC="${TMP_DIR}/password_enc"
MESSAGE_ENC="${TMP_DIR}/message_enc"


# extract encrypted AES key and encrypted message
cut -d. -f 2 "$INPUT_FILE"| base64 -d > "$PASSWORD_ENC"
cut -d. -f 3 "$INPUT_FILE"| base64 -d > "$MESSAGE_ENC"

# decrypt password
PASSWORD=$(openssl pkeyutl -decrypt -inkey "$RSA_PRIV_KEY" -in "$PASSWORD_ENC")


# decrypt message
echo -n "$PASSWORD" | openssl aes-256-cbc -d -pbkdf2 -in "$MESSAGE_ENC" -pass stdin --out se-checksums.txt

Note: In the above example, mykey.pem represents the private RSA keys generated during Preparing Attestation - Generating an RSA key pair.

IBM® Z17 attestation

Edit online

When the IBM CCCO VM is running on Z17 or later, the boot process attempts to perform Ultravisor attestation to verify that the image is running on the correct host. For more information, see Host attestation.

The attestation document

Edit online

The attestation document is available as the /var/hyperprotect/se-checksums.txt file, within the IBM CCCO VM. You can find the other related files under the same directory of the attestation document.

If user-data contains any other sections apart from workload, env, envworkloadsignature, and attestationpublickey, the SHA256 for the respective sections are generated.
If user-data is encrypted, the SHA256sum format for the encrypted part of the user data sections such as workload, env, and so on are also generated.
If any other keys apart from env, workload, envWorkloadSignature, and attestationPublicKey are present, the sha256 format of the each value is generated under the attestation records with the format sha256ofthevaluecontract: contract: otherkey.

The following information is available at the /var/hyperprotect/ directory:

/var/hyperprotect
/var/hyperprotect/
|-- se-checksums.txt
|-- se-signature.bin
|-- se-version
|-- se.hdr.bin
|-- uv_firmware.state

Checksums are the SHA256 of the message digest and can be calculated by using the following Linux command-line utility:

sha256sum <file>

The following snippet is an example of an attestation document:

For Bare Metal:

2.0.9
        Machine Type/Plant/Serial: 9175/02/C90A8
        Image age: 0 days since creation.
        Ultravisor CUID: 0x9d32448ae7096571cc2464e34572eec2
        Host HKD: HKD-9175-02C90A8.crt (Host Key Document of the LPAR)
        HKD is valid until: Sep 21 05:47:11 2027 GMT
        Encryption Certificate valid until: Jul 15 06:40:02 2027 UTC
        Attestation Certificate valid until: Jul 21 17:08:16 2027 UTC
        5e170bfe0476a0ad475509a58f8deb8aaa4c6679eaa3638d63baafab7f550715 se.hdr.bin
        b61a5ee86ec385ab8e661ded95f11a3c17535a210c6fa5dbe05ab69f2018efca uv_firmware.state
        30adc7235d5aa198ce17b544603f9d54fac0f2a5e6442bdd6cfd06073b65d5ad certificate_expiry_date.json
        3d5555eea00786dcaa0dda62e265239f2c19b47f1fddaf82d83871fca88c7922 payload.initrd.gz
        f4446eff10434a5a3e4614b3b946113fc4a0c52ea3cb3b49d91ada1f834df720 baseimage
        1bd03fe34543996d474e15bdf3aa3c5400bf6ddc1f7f8dcce04227f188359982 sbom
        bcb2d42b9097a7fb3a4275f63814ca4a7ef8f5c1ece46ea8478a072a172ad2c4 contract:workload 
        f979058d29dff0604bd13552bdc3e97ba24b5835fca1bd3b92fc5e9d5e33f3a8 contract:boot 
        7c0bae93f0c8c1afe540046458724910f5434741159a7d348105403a9bae4a53 contract:env

For Peer Pod:

26.7.1
        Machine Type/Plant/Serial: 9175/02/C90A8
        Image age: 0 days since creation.
        Encryption Certificate valid until: Jul 15 06:40:02 2027 UTC
        Attestation Certificate valid until: Jul 15 13:27:46 2027 UTC
        9bc3a98e05f35fc6d985527aae01ec9c16c9291872e7ee2856c75f035714429b se.hdr.bin
        5d295acf0841e3210f3bcd1b687d50dd1f7f1b500a7ab26044d63ffb8914583a certificate_expiry_date.json
        35f3bfe3cc11bc1be4e3032255461952b20d229a232bb40dc09ca71b47c69a8f root.tar.gz
        80895c68757debd1b6fb45b993cbca2dcf84bb3bd24790e0077a7356855a9ad4 baseimage
        65ceb8076ee34707d881a6c8f4361b9c679b4241a6b6f75419349fcebb81dc99 sbom
        efce4272685abb5927943d093f4d46bc7ac1c8df5f09e89d18c9cda0f243e8ad /dev/disk/by-label/cidata
        a38cd4cc87a9843d835b0cf09e9b9c7ca626d78d4935dc3574bc3287ce861d3c cidata/meta-data
        a466db2bc899b7455636cdcdef2cd6dd704d685e8dcf1e486d5d62715f3eee8a cidata/user-data
        e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 cidata/vendor-data
        14c1944568648fb4e738c6baa4b6feb9c57f040d61bda56174a062a54f747417 cidata/user-data.initdata
        72781cfb3e72df4c8c9b893782f07da3cf3ec166ae33dad6d1a642fe91804de0 contract:env
        34ba768ab3d559494c8968d2e3d1746337aaff050695cab73dbce0be49e58981 contract:workload

The following table describes each entry in the example attestation document:

Table 1. Entries in an attestation document

Entry	Description
`Machine Type/Plant/Serial`	`Machine Type/Plant/Serial` is the information required to obtain a Host Key Document for the secure execution VM. It reflects on which machine the secure execution VM is currently running.
`Image age`	`Image age` is the age of the CCCO VM image since its creation.
`Ultravisor CUID`	`Ultravisor CUID` is provided through Ultravisor attestation. This requires an HKD to be available.
`Host HKD`	`Host HKD` is the name of the Host Key Document file used for Ultravisor attestation. If supplied, the Host HKD field will include the description field.
`HKD is valid until`	`HKD is valid until`is valid until the expiration date of the HKD that was used for attestation.
`baseimage`	For Bare Metal: The `base image` for the IBM CCCO is an IBM internal file called `bootloader-os.tgz`. This file serves as the primary source for most of the operating system components used in the container image. It is utilized only during the image build process by the enabler.The enabler combines this base image with additional RPM packages to generate two critical artifacts: `se.hdr.bin` - the secure execution header. The encrypted Secure Execution kernel (initrd image) - which is essential for running workloads in a secure execution environment. For Peer Pod: The `baseimage` is the IBM internal QEMU Copy On Write Version 2 (QCOW2) file, which is used as the source for most of the operating system files of the IBM Confidential Computing Containers for Red Hat OpenShift Container Platform image. It is used only at image build time by the enabler process. The enabler uses this source together with other Debian packages to create the `root.tar.gz` and the encrypted secure execution kernel or 'initrd' image.
`se.hdr.bin`	The `se.hdr.bin` is part of the final secure execution enabled IBM CCCO image.
`root.tar.gz`	The `root.tar.gz` is part of the final secure execution enabled IBM Confidential Computing Containers for Red Hat OpenShift Container Platform image and contains all operating system files. It is stored on the image's first partition (boot partition) as `/boot/root.tar.gz`.
`/dev/disk/by-label/cidata`	The `/dev/disk/by-label/cidata` is a block device that is attached to the running CCCO VM that contains the cloud-init files. For more information about Cloud-Init, see cloud-init documentation.
`cidata`	All the files in `cidata` copied are from the `/dev/disk/by-label/cidata` block device.
`attestationPublicKey`	The `attestationPublicKey` is the public key that you provide which is used to encrypt the attestation document. The `attestationPublicKey` is part of the user-data file. Encrypting the attestation document is optional.
`uv_firmware.state`	The `uw_firmware.state` is a file produced by Z17 extended ultravisor attestation. It reflects `ultravisor/firmware` level states.
`boot`	The `boot` contains the encrypted details of `se.hdr.bin`.
`payload.initrd.gz`	The `payload.initrd.gz` is a mandatory section. The workload persona provides information about the container (or workload) that needs to be brought up on the IBM CCCO VM.
`workload`	`Workload` is a mandatory section. The workload persona provides information about the container (or workload) that needs to be brought up on the CCCO VM.
`env`	`env` is a mandatory section. The deployer persona works closely with IBM Cloud. The deployer then creates the `env` section of the contract. The `env` section has information that is specific to the IBM Cloud environment.
`envWorkloadSignature`	`envWorkloadSignature` is an optional section and contains the signature of the other sections of the contract.
`contract: sections`	`contract: sections` are the checksums of top level keys in the contract document.

The following are the shasum of the Bare Metal image (kernel image):

baseimage:

f4446eff10434a5a3e4614b3b946113fc4a0c52ea3cb3b49d91ada1f834df720 baseimage

payload.initrd.gz

3d5555eea00786dcaa0dda62e265239f2c19b47f1fddaf82d83871fca88c7922 payload.initrd.gz

The following are the shasum of the Peer Pod image (QCOW2):

baseimage:

80895c68757debd1b6fb45b993cbca2dcf84bb3bd24790e0077a7356855a9ad4 baseimage

root.tar.gz

35f3bfe3cc11bc1be4e3032255461952b20d229a232bb40dc09ca71b47c69a8f root.tar.gz

Calculating sha of different subsection of contract

Edit online

To calculate contract subsection of sha, the input should be similar to the one provided in the contract. If publicKey is provided in the contract, for example with attestationPublicKey, sha should be calculated on the publicKey file only, even though the input is provided in one line with \n or \r.

Decrypting the attestation document

Edit online

If user data contains a public RSA key (attribute: attestationPublicKey), then the attestation document (se-checksums.txt) is encrypted with the given key. The encryption is done by the same process as that of contract encryption. For more information, see Contract encryption. The public RSA key itself can also be encrypted like the contract.

The encrypted attestation document is then named se-checksums.txt.enc.

Verifying the IBM Confidential Computing Containers for Red Hat OpenShift Container Platform image checksum value

Edit online

You can verify the checksum of the IBM Confidential Computing Containers for Red Hat OpenShift Container Platform image to confirm that the image has not been tampered with.

For Bare Metal

Identify the worker nodes by running the following command:

oc get nodes

Example output

NAME                                            STATUS   ROLES                         AGE    VERSION
control-bm-san-mn-0.bm-storage.bm-san-mn.coco   Ready    control-plane,master,worker   7d1h   v1.33.9
control-bm-san-mn-1.bm-storage.bm-san-mn.coco   Ready    control-plane,master,worker   7d1h   v1.33.9
control-bm-san-mn-2.bm-storage.bm-san-mn.coco   Ready    control-plane,master,worker   7d1h   v1.33.9

Access each node to check the logs according to the following example:
```
oc debug node/<node-name>
```
For example:
```
oc debug node/control-bm-san-mn-1.bm-storage.bm-san-mn.coco
```
Note:
1. For Single Node OpenShift (SNO), check the single node.
2. For multi-node clusters, repeat this process on all worker nodes.
Switch to the host root filesystem by running the following command:
```
chroot /host
```

Check the Kata addon kernel service logs by running the following command:

systemctl status kata-addon-kernel.service

Example output

......
......... 
Found version file in container: /artifacts/kernel/version.json
Container image kata version: 3.25.0
Version check passed: 3.25.0 == 3.25.0
Copying kernel /image/hpcc-bm-2.0.9.img to /var/cache/kata-containers/vmlinuz.ibm-se
.........
.........
.........
Kernel checksum: 6483d788fb40e5c3db2345dc0bbf4ff3061fc62a044b096fe202c798422f0994
.........
......

Note: The CCCO container image version and the Kata RPM file version from OpenShift Container Platform (OCP) must match. Version mismatches can lead to deployment or runtime issues and should be resolved before proceeding. For more information, see Troubleshooting.

Confirm that the kernel checksum matches the expected value for the IBM Confidential Computing Containers for Red Hat OpenShift Container Platform image:
```
6483d788fb40e5c3db2345dc0bbf4ff3061fc62a044b096fe202c798422f0994
```

For Peer Pod

Search for the osc-podvm-image-creation job on your cluster by running the following command:

$ oc get po -n openshift-sandboxed-containers-operator | grep osc-podvm-image-creation

Example output:

osc-podvm-image-creation-wqc7h                 0/1     Completed   0          6h56m

Check the logs of osc-podvm-image-creation by running the following command:

$ oc logs osc-podvm-image-creation-wqc7h -n openshift-sandboxed-containers-operator

Example output:

....
....
Validating PodVM image: /image/rootfs/image/ccco-26.7.1.qcow2
Getting format of the PodVM image: /image/rootfs/image/ccco-26.7.1.qcow2
PodVM image format for /image/rootfs/image/ccco-26.7.1.qcow2: qcow2
Checksum of the PodVM image: d22e952303608d4d4856edbdce265dad9cc0dfa74fe45aee81e70f94ce684be8
/image/rootfs/image/ccco-26.7.1.qcow2
....
....

The checksum of the IBM Confidential Computing Containers for Red Hat OpenShift Container Platform image must be d22e952303608d4d4856edbdce265dad9cc0dfa74fe45aee81e70f94ce684be8.

Host attestation

Edit online

The host-attestation section contains one or more host key documents (HKD).

When the IBM CCCO VM is running on Z17 or later, the boot process attempts to perform Ultravisor attestation to verify that the image is running on the correct host. The Ultravisor attestation produces evidence that is revealed through the Attestation Document.

type: env
host-attestation:
  HKD-<MODEL>-<PLANT><SERIAL>:
    description:<Description of HKD>
    host-key-doc: <A Host Key Document encoded in base64>
  HKD-<MODEL>-<PLANT><SERIAL>:
    description: :<Description of HKD>
    host-key-doc: <A Host Key Document encoded in base64>

Note: To get MODEL, PLANT, and SERIAL, run the cat /proc/sysinfo command on the Z secure execution KVM host. If the format is not followed, the IBM CCCO VM fails to come up. You can also follow the naming convention of the downloaded HKD file without .crt. For example, HKD-3931-0271A28 for HKD-3931-0271A28.crt. For more information about downloading HKD, see Obtaining a host key document from Resource Link.

Final contract

Edit online

The sample unencrypted contract to include host-attestation feature must look like:


type: env
host-attestation:
HKD-9175-xxxxxxx:
   description: test1
   host-key-doc: <A Host Key Document encoded in base64>
HKD-9175-xxxxxxx:
   description: test2
   host-key-doc: <A Host Key Document encoded in base64>
logging:
   logRouter:
      hostname: <host name of the egress endpoint of the logRouter>
      iamApiKey: <IBM Cloud API key>
      port: 443
type: workload
confidential-containers:
   secret:
      decryptionKey: ${SECRET_DECRYPTION_KEY}
      verificationKey: ${SECRET_VERIFICATION_KEY}
type: boot
sehdr: <The base64 encrypted value of kata-se-hdr.bin>

Note: The sehdr field is only applicable for Bare Metal CCCO deployments.

Verifying host attestation

Edit online

Perform the following steps to verify the host attestation:

For Bare Metal

Retrieve the pod’s MAC address by running the following command:
```
$ oc describe pod <pod-name>
```

Access kata runtime logs on the worker node by running the following command:

journalctl -x -t kata -f

Note: The journalctl logs should be viewed immediately after the CCCO VM starts running.

The host attestation is successful, if the output is as follows:

# user-data contains HKD info
# user-data contains HKD for this machine.
# HKD description: z17 HKD
# performing ultravisor attestation
# CUID=0x9dadaf2122310f74b439f0e97a910194
# create attestation data
# encrypt attestation document...
# set hostname...
# finish root disk setup...

The host attestation is a failure, if the output is as follows:

# decrypt user-data...
2 token decrypted, 0 encrypted token ignored
# user-data contains HKD info
HPL11001E: The supplied HKD file in the contract was not correct for this machine.
# HPL11 build:0.0.0 enabler:25.6.1
sysrq: Power Off

For Peer Pod

Extract the IBM CCCO peerpod VM name by running the following command on your LPAR:

$ virsh list

Example Output

Id Name State
839 podvm-busybox-att-31338e37 running
840 podvm-busybox-17329105 running
865 podvm-busybox-contract-signing-a96b195d running
873 podvm-simple-policy-test-7d460c21 running

View the peerpod VM console log by running the following command:

Note: The console logs should be viewed immediately after the IBM CCCO peerpod VM starts running.

$ virsh console <podvm-name>

<podvm-name>: Specify the peerpod VM name for which you want to view the console. For example, podvm-busybox-17329105.

The host attestation is successful, if the output is as follows:

# user-data contains HKD info# user-data contains HKD for this machine.# HKD description: z17 HKD# performing ultravisor attestation# CUID=0x9dadaf2122310f74b439f0e97a910194# create attestation data# encrypt attestation document...# set hostname...# finish root disk setup...

The host attestation is a failure, if the output is as follows:

# decrypt user-data...
2 token decrypted, 0 encrypted token ignored
# user-data contains HKD info
HPL11001E: The supplied HKD file in the contract was not correct for this machine.
# HPL11 build:0.0.0 enabler:25.6.1
sysrq: Power Off

Understanding attestation flows

Edit online

The following diagram shows two scenarios for attestation from the point of view of the auditor to validate that the deployment is the expected one. The left side of the diagram shows the establishment of trust by the auditor who is rooted on a third-party certificate authority. Any key used is kept in a Confidential Computing Containers Crypto Service and signed in a certificate chain based on the third-party authority. The Build environment that is used by Confidential Computing Containers is running in a trusted execution environment by using IBM Secure Execution Technology.

The result is a CCCO image that is seen at the end of the diagram, which is an encrypted secure execution image. To the right of the diagram, the validation of the deployment is outlined. For this, the auditor includes into the encrypted workload contract of the IBM CCCO VM, the public key of a secret, which only the auditor has control over. Such secrets might be protected by appropriate means, HSM or just a random key. Only the Confidential Computing Containers bootloader that is executed in the trusted execution environment provided through IBM Secure Execution for Linux on IBM LinuxONE, can run the secure execution image of IBM CCCO. The bootloader contains the secret to decrypt the contract.

During boot several hashes of components and measures of code are taken and added to the attestation record. To further protect this attestation record, the record is encrypted with the public key that the auditor provided. By doing so, only the auditor is in the position to decrypt the attestation record and can validate that the workload that is deployed in the enclave is the expected and untampered version of the workload that is expected to be deployed into the IBM CCCO VM.