Attestation

Edit online

Attestation is a process that starts by default when the IBM Hyper Protect Confidential Container (CCCO) virtual machine (VM) is created. It ensures that the CCCO VM image is indeed built by IBM, and that it was not modified. This process also provides information and allows validation of any data that is provided to the CCCO VM at the time of workload deployment.

When you create a CCCO VM by using the IBM CCCO image, the image uses an initial file system that is protected by encryption and signed by IBM Secure Execution (SE). For more information, see Confidential computing with LinuxONE.

To know more about the attestation process, see this video. This video shows the attestation process of IBM Hyper Protect Virtual Servers. The IBM CCCO for Red Hat OpenShift Container Platform uses the similar approach.

The boot process creates a unique root disk encryption key to ensure protection of the root disk. To perform attestation, the IBM CCCO VM image contains an attestation-signing key and the hash of the root partition at build time. The boot process validates the root partition. If the hash of the root partition does not match, the boot process does not continue because it assumes that the image was modified before boot. The attestation signing key is a random RSA 4 K key that is signed by an IBM root key that is maintained in Hyper Protect Crypto Services. The IBM root key is signed by Digicert.

During deployment of the IBM CCCO VM, an attestation record is created. It contains hashes of the following items:

The original base image
The root partition at the moment of the first boot
The root partition at build time

The attestation record is signed by the attestation key. As an extra protection layer, you can provide a public key during deployment, against which the attestation record is encrypted. The hash of this public key is added to the attestation record to ensure that the record can be viewed only by the compliance authority, and the expected authority can be easily identified through that hash.

After a workload is brought up using IBM CCCO, you can validate the attestation records within the created pod. Workload running on IBM CCCO VM will have the records added to /var/hyperprotect directory. Perform the following procedures for further verifications:

The attestation record is signed by the attestation signing key.
The attestation signing key can be confirmed by the IBM intermediate certificate. The IBM intermediate certificate is signed by DigiCert, which is proven by the root certificate of DigiCert, thus completing the chain of trust.

The encryption and attestation certificates are signed by the IBM intermediate certificate and this has been signed by the IBM Digicert intermediate cert (which in turn is signed by DigiCert Trusted Root G4). For more information about the certificates, see DigiCert Trusted Root Authority Certificates.

Use the following procedure to validate the attestation record and hashes:

Obtain the attestation record se-checksums.txt and the signature file se-signature.bin from your IBM CCCO VM. To do so, you can implement your container to provide the attestation record and the signature file. The attestation record and the signature file are made available to your container in the /var/hyperprotect directory.
Get the IBM attestation certificate that is a part of the image TAR file.
Validate the attestation certificate by following the instructions here.

Extract the attestation public key from the attestation certificate by using the following command:

 openssl x509 -pubkey -noout -in ibm-hyper-protect-confidential-container-attestation.crt > contract-public-key.pub

Verify the signature of the attestation record:
```
openssl sha256 -verify contract-public-key.pub -signature se-signature.bin se-checksums.txt
 
```
Signature verification must be done on a decrypted attestation file.
You can now use the hashes from the attestation record for validation.

In case you provided a public key for encrypting the attestation record, the following script might help in decrypting the record. For information about attestation public key, see Preparing the Attestation section.

#!/bin/bash
#
# Example script to decrypt attestation document.
#
# Usage:
#   ./decrypt-attestation.sh <rsa-priv-key.pem> [file]
#
# Token Format:
#   hyper-protect-basic.<ENC_AES_KEY_BASE64>.<ENC_MESSAGE_BASE64>


RSA_PRIV_KEY="$1"
if [ -z "$RSA_PRIV_KEY" ]; then
    echo "Usage: $0 <rsa-priv-key.pem>"
    exit 1
fi
INPUT_FILE="${2:-se-checksums.txt.enc}"
TMP_DIR="$(mktemp -d)"
#trap 'rm -r $TMP_DIR' EXIT


PASSWORD_ENC="${TMP_DIR}/password_enc"
MESSAGE_ENC="${TMP_DIR}/message_enc"


# extract encrypted AES key and encrypted message
cut -d. -f 2 "$INPUT_FILE"| base64 -d > "$PASSWORD_ENC"
cut -d. -f 3 "$INPUT_FILE"| base64 -d > "$MESSAGE_ENC"

# decrypt password
PASSWORD=$(openssl pkeyutl -decrypt -inkey "$RSA_PRIV_KEY" -in "$PASSWORD_ENC")


# decrypt message
echo -n "$PASSWORD" | openssl aes-256-cbc -d -pbkdf2 -in "$MESSAGE_ENC" -pass stdin --out se-checksums.txt

IBM® Z17 attestation

Edit online

When the IBM CCCO VM is running on Z17 or later, the boot process attempts to perform Ultravisor attestation to verify that the image is running on the correct host. For more information, see Host attestation.

The attestation document

Edit online

The attestation document is available as the /var/hyperprotect/se-checksums.txt file, within the IBM CCCO VM. You can find the other related files under the same directory of the attestation document.

If user-data contains any other sections apart from workload, env, envworkloadsignature, and attestationpublickey, the SHA256 for the respective sections are generated.
If user-data is encrypted, the SHA256sum format for the encrypted part of the user data sections such as workload, env, and so on are also generated.
If any other keys apart from env, workload, envWorkloadSignature, and attestationPublicKey are present, the sha256 format of the each value is generated under the attestation records with the format sha256ofthevaluecontract: contract: otherkey.

The following information is available at the /var/hyperprotect/ directory:

/var/hyperprotect
/var/hyperprotect/
|-- se-checksums.txt
|-- se-signature.bin
|-- se-version
|-- se.hdr.bin
|-- uv_firmware.state

Checksums are the SHA256 of the message digest and can be calculated by using the following Linux command-line utility:

sha256sum <file>

The following snippet is an example of an attestation document:

For Bare Metal:

2.0.6
Machine Type/Plant/Serial: 9175/02/C90A8
        Image age: 0 days since creation.
        Ultravisor CUID: 0xae2c9bc563c7b60e2324e3f1a0288554
        Host HKD: HKD-9175-02C90A8.crt (Host Key Document of the LPAR)
        HKD is valid until: Sep 21 05:47:11 2027 GMT
        Encryption Certificate valid until: Mar 12 09:41:39 2027 UTC
        Attestation Certificate valid until: Apr 24 14:26:10 2027 UTC
        se.hdr.bin: e29fe5a46ac66aee1b189f25e697650b89e5c10523bc746121e74cfde7c7312b
        f5ae67ebab538ab3547ca5da9f62aafdb200e1992cdaf1de23aaa24442538011 uv_firmware.state
        c479941e5a54b932a7e63804d2ee18a85bc2ffe931d5e8dc45139e97790397b4 certificate_expiry_date.json
        5153bb5b37a8b2f5d62b0bcc3b9aa128509c698ea6fb6d7fa2b8148765d89b4a payload.initrd.gz
        05efebdc59956673705ef8e2ee98194dec56d6fc4bf22c2770acadf971b0d0b9 baseimage
        5deddb779d9f8596e0c3b8e439c244ace02a4bc24e1f453459e778e58eb9a8ce sbom
        64610164763e5ec36b61b22cd95a98fe21a77570bc634ce088ba9fcdb6248e06 contract:env
        24f51636906520eec25d7d27b89d56f060b20ef728db5ba7840c693f5a1364d6 contract:workload
        b9d5d3a4d31556460b269279cdbfeef328b082a6254d08fc71e61e5dcdceeee8 contract:boot

For Peerpod:

26.3.1
Machine Type/Plant/Serial: 9175/02/C90A8
        Image age: 4 days since creation.
        Encryption Certificate valid until: Mar 12 09:41:39 2027 UTC
        Attestation Certificate valid until: Apr 24 14:29:35 2027 UTC
        ee109778e14712035ac4388020be029b383f22765ed1e169ffb3602166f88ca5 certificate_expiry_date.json
        a842f4e6d13811e5956480db5049c8cb4c80a9010bbc859a115835ae542f3f44 root.tar.gz
        95dd965e224c47d11ff4300acc57b9a87a1f300bf67caba7d10835f12c221a11 baseimage
        4c958351b29feb1905be55f571ea1cea2c22de897230e22ff129bd669a6704e4 sbom
        c9b28b7cc47cdc03e4f7941a57a9088752c662fd6bb0856d1e084200a490b4a9 /dev/disk/by-label/cidata
        27514189ae54fb5d612d8f1cd31aa9e8a4bf7faec5cf33400f8ec7feb8a91388 cidata/meta-data
        a9267156da9a100bb25508633c9697f8c44f7e49611bb3df85989bad30cb12a1 cidata/user-data
        e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 cidata/vendor-data
        b5ba03a432373c3e482ccd69e29784c0b6ac4c94f15c3029f7eea9bf91452141 contract:env
        9cf505260eb5a5aa78b0484a5cbad1024698d6fed9624a2384d1b0d6f3c16bb1 contract:workload

The following table describes each entry in the example attestation document:

Table 1. Entries in an attestation document

Entry	Description
`Machine Type/Plant/Serial`	`Machine Type/Plant/Serial` is the information required to obtain a Host Key Document for the secure execution VM. It reflects on which machine the secure execution VM is currently running.
`Image age`	`Image age` is the age of the CCCO VM image since its creation.
`Ultravisor CUID`	`Ultravisor CUID` is provided through Ultravisor attestation. This requires an HKD to be available.
`Host HKD`	`Host HKD` is the name of the Host Key Document file used for Ultravisor attestation. If supplied, the Host HKD field will include the description field.
`HKD is valid until`	`HKD is valid until`is valid until the expiration date of the HKD that was used for attestation.
`baseimage`	For Bare Metal: The `base image` for the IBM CCCO is an IBM internal file called `bootloader-os.tgz`. This file serves as the primary source for most of the operating system components used in the container image. It is utilized only during the image build process by the enabler.The enabler combines this base image with additional RPM packages to generate two critical artifacts: `se.hdr.bin` - the secure execution header. The encrypted Secure Execution kernel (initrd image) - which is essential for running workloads in a secure execution environment. For Peerpod: The `baseimage` is the IBM internal QEMU Copy On Write Version 2 (QCOW2) file, which is used as the source for most of the operating system files of the IBM Confidential Computing Containers for Red Hat OpenShift Container Platform image. It is used only at image build time by the enabler process. The enabler uses this source together with other Debian packages to create the `root.tar.gz` and the encrypted secure execution kernel or 'initrd' image.
`se.hdr.bin`	The `se.hdr.bin` is part of the final secure execution enabled IBM CCCO image.
`root.tar.gz`	The `root.tar.gz` is part of the final secure execution enabled IBM Confidential Computing Containers for Red Hat OpenShift Container Platform image and contains all operating system files. It is stored on the image's first partition (boot partition) as `/boot/root.tar.gz`.
`/dev/disk/by-label/cidata`	The `/dev/disk/by-label/cidata` is a block device that is attached to the running IBM Hyper Protect Confidential Container VM that contains the cloud-init files. For more information about Cloud-Init, see cloud-init documentation.
`cidata`	All the files in `cidata` copied are from the `/dev/disk/by-label/cidata` block device.
`attestationPublicKey`	The `attestationPublicKey` is the public key that you provide which is used to encrypt the attestation document. The `attestationPublicKey` is part of the user-data file. Encrypting the attestation document is optional.
`uv_firmware.state`	The `uw_firmware.state` is a file produced by Z17 extended ultravisor attestation. It reflects `ultravisor/firmware` level states.
`boot`	The `boot` contains the encrypted details of `se.hdr.bin`.
`payload.initrd.gz`	The `payload.initrd.gz` is a mandatory section. The workload persona provides information about the container (or workload) that needs to be brought up on the IBM CCCO VM.
`workload`	`Workload` is a mandatory section. The workload persona provides information about the container (or workload) that needs to be brought up on the CCCO VM.
`env`	`env` is a mandatory section. The deployer persona works closely with IBM Cloud. The deployer then creates the `env` section of the contract. The `env` section has information that is specific to the IBM Cloud environment.
`envWorkloadSignature`	`envWorkloadSignature` is an optional section and contains the signature of the other sections of the contract.
`contract: sections`	`contract: sections` are the checksums of top level keys in the contract document.

The following are the shasum of the Peerpod image (QCOW2):

baseimage:

95dd965e224c47d11ff4300acc57b9a87a1f300bf67caba7d10835f12c221a11 baseimage

root.tar.gz

a842f4e6d13811e5956480db5049c8cb4c80a9010bbc859a115835ae542f3f44 root.tar.gz

The following are the shasum of the Bare Metal image (kernel image):

baseimage:

05efebdc59956673705ef8e2ee98194dec56d6fc4bf22c2770acadf971b0d0b9 baseimage

payload.initrd.gz

5153bb5b37a8b2f5d62b0bcc3b9aa128509c698ea6fb6d7fa2b8148765d89b4a payload.initrd.gz

Calculating sha of different subsection of contract

Edit online

To calculate contract subsection of sha, the input should be similar to the one provided in the contract. If publicKey is provided in the contract, for example with attestationPublicKey, sha should be calculated on the publicKey file only, even though the input is provided in one line with \n or \r.

Decrypting the attestation document

Edit online

If user data contains a public RSA key (attribute: attestationPublicKey), then the attestation document (se-checksums.txt) is encrypted with the given key. The encryption is done by the same process as that of contract encryption. For more information, see Contract encryption. The public RSA key itself can also be encrypted like the contract.

The encrypted attestation document is then named se-checksums.txt.enc.

Verifying the IBM Confidential Computing Containers for Red Hat OpenShift Container Platform image checksum value

Edit online

You can verify the checksum of the IBM Confidential Computing Containers for Red Hat OpenShift Container Platform image to confirm that the image has not been tampered with.

For Bare Metal

Identify the worker nodes by running the following command:

oc get nodes

Example output

NAME                                            STATUS   ROLES                         AGE    VERSION
control-bm-san-mn-0.bm-storage.bm-san-mn.coco   Ready    control-plane,master,worker   7d1h   v1.33.9
control-bm-san-mn-1.bm-storage.bm-san-mn.coco   Ready    control-plane,master,worker   7d1h   v1.33.9
control-bm-san-mn-2.bm-storage.bm-san-mn.coco   Ready    control-plane,master,worker   7d1h   v1.33.9

Access each node to check the logs according to the following example:
```
oc debug node/<node-name>
```
For example:
```
oc debug node/control-bm-san-mn-1.bm-storage.bm-san-mn.coco
```
Note:
1. For Single Node OpenShift (SNO), check the single node.
2. For multi-node clusters, repeat this process on all worker nodes.
Switch to the host root filesystem by running the following command:
```
chroot /host
```

Check the Kata addon kernel service logs by running the following command:

systemctl status kata-addon-kernel.service

Example output

......
......... 
Kernel checksum: 91631354fb4c521088862a99abe9ce126eb9084b1caeb4fc4b2419930d178676
.........
......

Confirm that the kernel checksum matches the expected value for the IBM Confidential Computing Containers for Red Hat OpenShift Container Platform image:
```
91631354fb4c521088862a99abe9ce126eb9084b1caeb4fc4b2419930d178676
```

For Peerpod

Search for the osc-podvm-image-creation job on your cluster by running the following command:

$ oc get po -n openshift-sandboxed-containers-operator | grep osc-podvm-image-creation

Example output:

osc-podvm-image-creation-wqc7h                 0/1     Completed   0          6h56m

Check the logs of osc-podvm-image-creation by running the following command:

$ oc logs osc-podvm-image-creation-wqc7h -n openshift-sandboxed-containers-operator

Example output:

....
....
Validating PodVM image: /image/rootfs/image/ccco-26.3.1.qcow2
Getting format of the PodVM image: /image/rootfs/image/ccco-26.3.1.qcow2
PodVM image format for /image/rootfs/image/ccco-26.3.1.qcow2: qcow2
Checksum of the PodVM image: a842f4e6d13811e5956480db5049c8cb4c80a9010bbc859a115835ae542f3f44
/image/rootfs/image/ccco-26.3.1.qcow2
....
....

The checksum of the IBM Confidential Computing Containers for Red Hat OpenShift Container Platform image must be a842f4e6d13811e5956480db5049c8cb4c80a9010bbc859a115835ae542f3f44.

Host attestation

Edit online

The host-attestation section contains one or more host key documents (HKD).

When the IBM CCCO VM is running on Z17 or later, the boot process attempts to perform Ultravisor attestation to verify that the image is running on the correct host. The Ultravisor attestation produces evidence that is revealed through the Attestation Document.

type: env
host-attestation:
  HKD-<MODEL>-<PLANT><SERIAL>:
    description:<Description of HKD>
    host-key-doc: <A Host Key Document encoded in base64>
  HKD-<MODEL>-<PLANT><SERIAL>:
    description: :<Description of HKD>
    host-key-doc: <A Host Key Document encoded in base64>

Note: To get MODEL, PLANT, and SERIAL, run the cat /proc/sysinfo command on the Z secure execution KVM host. If the format is not followed, the IBM CCCO VM fails to come up. You can also follow the naming convention of the downloaded HKD file without .crt. For example, HKD-3931-0271A28 for HKD-3931-0271A28.crt. For more information about downloading HKD, see Obtaining a host key document from Resource Link.

Final contract

Edit online

The sample unencrypted contract to include host-attestation feature must look like:


type: env
host-attestation:
HKD-9175-xxxxxxx:
   description: test1
   host-key-doc: <A Host Key Document encoded in base64>
HKD-9175-xxxxxxx:
   description: test2
   host-key-doc: <A Host Key Document encoded in base64>
logging:
   logRouter:
      hostname: <host name of the egress endpoint of the logRouter>
      iamApiKey: <IBM Cloud API key>
      port: 443
type: workload
confidential-containers:
   secret:
      decryptionKey: ${SECRET_DECRYPTION_KEY}
      verificationKey: ${SECRET_VERIFICATION_KEY}
type: boot
sehdr: <The base64 encrypted value of kata-se-hdr.bin>

Note: The sehdr field is only applicable for Bare Metal CCCO deployments.

Verifying host attestation

Edit online

Perform the following steps to verify the host attestation:

For Bare Metal

Retrieve the pod’s MAC address by running the following command:
```
$ oc describe pod <pod-name>
```

Access kata runtime logs on the worker node by running the following command:

journalctl -x -t kata -f

Note: The journalctl logs should be viewed immediately after the CCCO VM starts running.

The host attestation is successful, if the output is as follows:

# user-data contains HKD info
# user-data contains HKD for this machine.
# HKD description: z17 HKD
# performing ultravisor attestation
# CUID=0x9dadaf2122310f74b439f0e97a910194
# create attestation data
# encrypt attestation document...
# set hostname...
# finish root disk setup...

The host attestation is a failure, if the output is as follows:

# decrypt user-data...
2 token decrypted, 0 encrypted token ignored
# user-data contains HKD info
HPL11001E: The supplied HKD file in the contract was not correct for this machine.
# HPL11 build:0.0.0 enabler:25.6.1
sysrq: Power Off

For Peerpod

Extract the IBM CCCO peerpod VM name by running the following command on your LPAR:

$ virsh list

Example Output

Id Name State
839 podvm-busybox-att-31338e37 running
840 podvm-busybox-17329105 running
865 podvm-busybox-contract-signing-a96b195d running
873 podvm-simple-policy-test-7d460c21 running

View the peerpod VM console log by running the following command:

Note: The console logs should be viewed immediately after the IBM CCCO peerpod VM starts running.

$ virsh console <podvm-name>

<podvm-name>: Specify the peerpod VM name for which you want to view the console. For example, podvm-busybox-17329105.

The host attestation is successful, if the output is as follows:

# user-data contains HKD info# user-data contains HKD for this machine.# HKD description: z17 HKD# performing ultravisor attestation# CUID=0x9dadaf2122310f74b439f0e97a910194# create attestation data# encrypt attestation document...# set hostname...# finish root disk setup...

The host attestation is a failure, if the output is as follows:

# decrypt user-data...
2 token decrypted, 0 encrypted token ignored
# user-data contains HKD info
HPL11001E: The supplied HKD file in the contract was not correct for this machine.
# HPL11 build:0.0.0 enabler:25.6.1
sysrq: Power Off

Understanding attestation flows

Edit online

The following diagram shows two scenarios for attestation from the point of view of the auditor to validate that the deployment is the expected one. The left side of the diagram shows the establishment of trust by the auditor who is rooted on a third-party certificate authority. Any key used is kept in a Hyper Protect Crypto Service and signed in a certificate chain based on the third-party authority. The Build environment that is used by Hyper Protect is running in a trusted execution environment by using IBM Secure Execution Technology.

The result is a CCCO image that is seen at the end of the diagram, which is an encrypted secure execution image. To the right of the diagram, the validation of the deployment is outlined. For this, the auditor includes into the encrypted workload contract of the IBM CCCO VM, the public key of a secret, which only the auditor has control over. Such secrets might be protected by appropriate means, HSM or just a random key. Only the Hyper Protect bootloader that is executed in the trusted execution environment provided through IBM Secure Execution for Linux on IBM LinuxONE, can run the secure execution image of IBM CCCO. The bootloader contains the secret to decrypt the contract.

During boot several hashes of components and measures of code are taken and added to the attestation record. To further protect this attestation record, the record is encrypted with the public key that the auditor provided. By doing so, only the auditor is in the position to decrypt the attestation record and can validate that the workload that is deployed in the enclave is the expected and untampered version of the workload that is expected to be deployed into the IBM CCCO VM.