Verifying container image integrity

Container images for IBM Cloud Pak® for Business Automation are signed by using GPG keys. You can verify the integrity of these images by using signature validation when you pull images by using Linux® command-line tools. In Red Hat® OpenShift®, you can create a policy to enforce signature validation when your worker nodes pull images.

About this task

IBM Cloud Pak for Business Automation provides container images in the IBM® Entitled Registry that are signed following the approach from Red Hat. You can use the signature to verify that the images come from IBM when they are pulled onto the system. For more information about the signature verification, see Verifying signatures of Red Hat container images External link opens a new window or tab .

Cloud Pak for Business Automation installs a number of dependencies, such as IBM Cloud Pak foundational services External link opens a new window or tab , which has its own signature validation process.

Red Hat OpenShift can be configured to verify container image signatures upon pulling images from a specific location. See Container image signatures External link opens a new window or tab for details about Red Hat's signatures of Red Hat OpenShift images. Images for IBM Cloud Pak for Business Automation are signed by using GPG keys, which in turn are signed by a certificate chain with Digicert as its root. The signing key changed multiple times. Therefore, multiple keys must be configured in the signature policy.

The following sections can be used to verify the IBM Cloud Pak for Business Automation signature on the container images.

Creating files that contain GPG public keys to pull images
Creating a policy to require signed images
Signature validation in Linux command-line tools
Optional: Verifying key material
Cleaning up local images and caches
Scanning for vulnerabilities

Creating files that contain GPG public keys to pull images

Create a file that contains the public GPG key in the GNU Privacy Guard (GPG) format to sign container images for IBM Cloud Pak for Business Automation.

Table 1. File for the public GPG key
File name for GPG key	Date
`icp4a-pubkey-from-2024.gpg`	For images released in December 2024 and later.

Create a file icp4a-pubkey-from-2024.gpg and add the following public key in the GNU Privacy Guard (GPG) format. This key can be used to pull images from December 2024 and onwards.

Creating a policy to require signed images

A security policy can deny or allow images to be pulled, or require a trust relationship for pulling images. This can be done for entire image registries or specific parts of image registries. For more information, see Controlling what image sources can be deployed External link opens a new window or tab .

The following policy is least intrusive, as it requires images that are pulled from cp.icr.io/cp/cp4a to be signed by using any of the GPG keys listed in the keyPaths array. Images from any other container registry and other parts inside cp.icr.io are accepted without signatures.

If you can provide public keys for signature validation of all applications and their dependencies in your Red Hat OpenShift cluster, you can set the default type to reject.

Create a file and name it policy.json, then include the following content:

{
    "default": [
      {
       "type": "insecureAcceptAnything"
      }
    ],
    "transports": {
      "docker": {
        "cp.icr.io/cp/cp4a": [
          {
            "type": "signedBy",
            "keyType": "GPGKeys",
            "keyPaths": ["/etc/pki/icp4a-pubkey-from-2024.gpg"]
          }
        ]
      }
    }
}

You can use a MachineConfig object to inject the GPG public key files and the policy.json file into the nodes. The following example includes all GPG public key files inline in the URL encoded format (machine-config.yaml).

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
    labels:
    machineconfiguration.openshift.io/role: worker
    name: image-signature
spec:
    config:
    ignition:
        version: 3.2.0
    storage:
        files:
        - contents:
            source: >-
                data:,-----BEGIN%20PGP%20PUBLIC%20KEY%20BLOCK-----mQINBGcyQU0BEADAO%2Fbfaez4vdjXIuZfNFqpOpAOHzs89EhBguxGVHUdx2mtGUZo%NEZB3%2FFWVO+IlmW5clZrNSvoBDCF1oV63UDBiZUcoE1lS0HsKQJuhEKxZqBURZOB%5E12EjfVbS6xn+R3EubdepBwlNwv1rCq2un4KDhxY7phl0iwA1uWuxn4yU0LKQ%2FB%%2FJ%2F6jZL%2FecSjlzITYNsbfOwmCFeRhjmeSxuMobzWd9A%2FcLA6sPh7sDTzSmyJp%2F27%6ch3MijuxHLAdEFdR%2FoWvyh4XK04dRRQh2ecsJ2m5+EsS1DeYbFhwdgvMGtBVz6f%fDKK7BBMt%2FcsgT9wZX4BD97KDPFdAsANNC55ipYIxMKiDDrsuHJ1eD1UqaUGdTOn%MuT+AC1Rfsjcbxwl0FItfdQ0rBUFmHEw1qTScDNa+WSKflj386lPV5hWneOG58UG%u1vQuGHJchwcGYRIDvgeYC+lw%2Fq%2FjECqgeLrnJx9JVcMUmkTbDxfTDGe82mIqbiw%B38%2FboAujKCpCkvZFhP5OIylEmyCfRCZ%2F0ul8hvzS3kCjG5QSjBVcHolK75++CXb%6yYNawmQYVjPk1KuNe6Y+TVl0nsYqtSEjnknO4LRsKX2ZHxkASimFqJCjbAEQiEx%0B0mZ3RTUXR92Y%2FUy7XXtusymGleCkW+bHs2F0+9i32l9Nuxgl%2FD7g1WywARAQAB%tC9JQk0gQ2xvdWQgUGFrIGZvciBBdXRvbWF0aW9uIDxwc2lydEB1cy5pYm0uY29t%PokCOgQTAQgAJAUCZzJBTQIbDwULCQgHAgYVCgkICwIEFgIDAQIeAQUJAAAAAAAK%CRC0lqyjOeABc5p+D%2F4uVq6HmZQ7B2oTYYBoOa6xBMdnI91GQPaoGRhAwAPzQh8W%jw5fNiBJCLPVzTCIh26MJUwBWLXSZtlBDIg7zFEgXqUMqtS4qMuLTnBHyovx30hQ%zJyH6DkCCeBTqS06oq2N3comsCk+pbLlHSEMZlDR1WrZE2omi7P42ET9wQn4Px+c%iHmy07qJDoV4HyZEYAJ6LSyEXy8l0XTaqhMEerrFMKcLQGXsy0y0Al4%2FkvmtSuyw%hZaAadMsaPV+2rSz8yfyjNUDf7ZPh2whelV3VYfaVHnuy7S+RRCFqAeOtIAxOpai%++hxazlf03mgnPQH7WGSvjaz8sWtEUUMYFE0mnwusrK0DObOXd76qghkifN1IRiS%bUPvfa15U1%2FCwC0C5xFRWQfgrX2EwqUYNMiHqKRYxz8+LEC+vdT03naLzyxpAKwM%6bZ2laNJYNofQd7dYU+Co4oWrCTGfFLD%2FLYnejzWYR7Drt5Ppz1qgW1q5J1Qi7+V%s7+w4ga8n61Ta2zaDcccgdn%2FTG0IO8bWnxgwKwZS8w3wj+GFx70o4fIlH++PhL1%2F%zdI2jfH4XqvjT+mx3LNpfEXUOquIl9ikkRZMG787lqdjr1L58YF9BdZPtqs5RufU%mGzh82sCEmw9GaSJiGQb%2FfcoSIolqbLmEj4Aqy%2FhA6QSthBu58CGLOs91SevQQ%0A%0A%%0AJn52-----END%20PGP%20PUBLIC%20KEY%20BLOCK-----
            filesystem: root
            mode: 420
            path: /etc/pki/icp4a-pubkey-from-2024.gpg 
        - contents:
            source: >-
                data:,%7B%0D%0A%20%20%20%20%22default%22%3A%20%5B%0D%0A%20%20%20%20%20%20%7B%0D%0A%20%20%20%20%20%20%20%22type%22%3A%20%22insecureAcceptAnything%22%0D%0A%20%20%20%20%20%20%7D%0D%0A%20%20%20%20%5D,%0D%0A%20%20%20%20%22transports%22%3A%20%7B%0D%0A%20%20%20%20%20%20%22docker%22%3A%20%7B%0D%0A%20%20%20%20%20%20%20%20%22cp.icr.io%2Fcp%2Fcp4a%22%3A%20%5B%0D%0A%20%20%20%20%20%20%20%20%20%20%7B%0D%0A%20%20%20%20%20%20%20%20%20%20%20%20%22type%22%3A%20%22signedBy%22,%0D%0A%20%20%20%20%20%20%20%20%20%20%20%20%22keyType%22%3A%20%22GPGKeys%22,%0D%0A%20%20%20%20%20%20%20%20%20%20%20%20%22keyPaths%22%3A%20%22%2Fetc%2Fpki%2Ficp4a-pubkey-from-2024.gpg%22%5D%0D%0A%20%20%20%20%20%20%20%20%20%20%7D%0D%0A%20%20%20%20%20%20%20%20%5D%0D%0A%20%20%20%20%20%20%7D%0D%0A%20%20%20%20%7D%0D%0A%7D
            filesystem: root
            mode: 420
            path: /etc/containers/policy.json

You can decode the files into clear text by using a URL decoder, and then apply the MachineConfig resource:

oc apply -f machine-config.yaml

You can watch the rebooted worker nodes as they apply the updated policy:

oc get mcp

If you are running a starter pattern, PodDisruptionBudgets on some pods might prevent a node from restarting gracefully. You can force deletion of pods on the nodes by first identifying a node for which scheduling is disabled:

oc get node

Use the node name to delete the pods on this node:

node=...
oc delete pod --field-selector="spec.nodeName=$node" --all-namespaces

Signature validation in Linux command-line tools

When inspecting and transporting images, skopeo External link opens a new window or tab is a useful tool as it does not need a docker or a daemon. It can be easily used in continuous integration (CI) pipelines to copy images between two registries, provide credentials for secured registries, or to promote images from a development registry into production.

If you pull the images from the IBM Entitled Registry with skopeo, you can enable the container image signature verification. The icp4a-pubkey-from-2024.gpg file that you created can be used to verify the images under a certain repository and path.

In the following example, images are pulled from cp.icr.io/cp/cp4a and verified with the GPG public key that is in the icp4a-pubkey-from-2024.gpg file. All images are rejected, except the signed images in the cp4a folder.

The keyPaths array is not supported in podman and skopeo. Instead, you need to point to a specific key file by using the keyPath parameter (icp4a-pubkey-from-2024.gpg).

Create a file and name it policy-2.json, then include the following content:

{
    "default": [
      {
       "type": "reject"
      }
    ],
    "transports": {
      "docker": {
        "cp.icr.io/cp/cp4a": [
          {
            "type": "signedBy",
            "keyType": "GPGKeys",
            "keyPath": "icp4a-pubkey-from-2024.gpg"
          }
        ]
      }
    }
}

Note: If cp.icr.io is specified, all the images that are not signed under this folder are rejected. The skopeo command has the useful option

--policy
policy-2.json

, which sets a specific policy file instead of using the default /etc/containers/policy.json file.

The following sample pulls one image into the local file system and validates its signature.

skopeo --policy ./policy-2.json copy docker://cp.icr.io/cp/cp4a/ban/navigator:24.0.1-amd64 dir:./ban

When you pull images by using podman, you can provide a --signature-policy policy2.json option to point to a policy file.

Referencing the correct public key for signature validation, images can be pulled.

podman pull --signature-policy policy-2.json cp.icr.io/cp/cp4a/ban/navigator:24.0.1-amd64
Trying to pull cp.icr.io/cp/cp4a/ban/navigator:24.0.1-amd64...
Getting image source signatures
Checking if image destination supports signatures
...

Pulling images fail when the policy file references another public key.

podman pull --signature-policy policy-1.json cp.icr.io/cp/cp4a/ban/navigator:24.0.1-amd64
Trying to pull cp.icr.io/cp/cp4a/ban/navigator:24.0.1-amd64...
None of the signatures were accepted, reasons: Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"7C5CF97A39F13650", Status:gpgme.Error{err:0x7000009}, Timestamp:time.Time{wall:0x0, ext:63812637354, loc:(*time.Location)(0x55fc881421a0)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x55fc881421a0)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"7C5CF97A39F13650", Status:gpgme.Error{err:0x7000009}, Timestamp:time.Time{wall:0x0, ext:63812637429, loc:(*time.Location)(0x55fc881421a0)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x55fc881421a0)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}
Error: error pulling image "cp.icr.io/cp/cp4a/ban/navigator:24.0.1-amd64": unable to pull cp.icr.io/cp/cp4a/ban/navigator:24.0.1-amd64: unable to pull image: Source image rejected: None of the signatures were accepted, reasons: Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"7C5CF97A39F13650", Status:gpgme.Error{err:0x7000009}, Timestamp:time.Time{wall:0x0, ext:63812637354, loc:(*time.Location)(0x55fc881421a0)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x55fc881421a0)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}; Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"7C5CF97A39F13650", Status:gpgme.Error{err:0x7000009}, Timestamp:time.Time{wall:0x0, ext:63812637429, loc:(*time.Location)(0x55fc881421a0)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x55fc881421a0)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}

Verifying key material

You can verify the public keys once, and you do not need to repeat the process when you apply policies to multiple clusters.

If you want to verify the icp4a-pubkey-from-2024.gpg file:

Create a file icp4a-cert-from-2024.pem and add the following public certificate in the PEM format.

Create a file icp4a-issuer-from-2024.pem and add the following issuer public certificate in the PEM format.

Run the following commands to verify the GPG key and its issuer with the icp4a-cert-from-2024.pem and icp4a-issuer-from-2024.pem files.

Table 2. Commands to verify certificate issuance and validity for icp4a-pubkey-from-2024.gpg
Command	Description
`openssl x509 -text -in icp4a-cert-from-2024.pem`	Displays the IBM Cloud Pak for Business Automation public certificate. Verify that the "Subject" line is IBM, and the "Issuer" is www.digicert.com. The icp4a-cert-from-2024.pem certificate is signed by Digicert.
`gpg2 -v --list-packets icp4a-pubkey-from-2024.gpg`	Displays the GPG public key. Verify that pkey[0] is the same as the "Modulus" of icp4a-cert-from-2024.pem. The command validates that icp4a-pubkey-from-2024.gpg is the public key (in GPG format) for IBM Cloud Pak for Business Automation.
`openssl ocsp -no_nonce \ -issuer icp4a-issuer-from-2024.pem \ -cert icp4a-cert-from-2024.pem \ -VAfile icp4a-issuer-from-2024.pem \ -text -url http://ocsp.digicert.com`	Verifies that the certificate icp4a.pem comes from Digicert by using the Online Certificate Status Protocol (OCSP).

Cleaning up local images and caches

Before any test or use of the image signature, you must clean up the container images and the local cache, even if your image pull policy is "always". If an image is cached locally, it is not pulled again, and the image signature verification does not take place.

for node in $(oc get no -l node-role.kubernetes.io/worker --no-headers -o name); do
    oc debug $node -- chroot /host sh -c "podman images | grep -e 'cp.icr.io/cp/cp4a' | awk '{print \$3}' | xargs podman rmi -f "
done

Scanning for vulnerabilities

All the container images pass rigorous image vulnerability scans during the development lifecycle, but new vulnerabilities are discovered almost every day. It is likely that most of the vulnerabilities that you find are related to the Linux kernel or to a few embedded packages. IBM tracks these vulnerabilities and provides fixes for them in fix packs or subsequent releases. To find out more, you can contact IBM support External link opens a new window or tab .