Verifying container image security

When it comes to inspecting and transporting images, skopeo is a useful tool as it does not need a docker or a daemon. It can be used easily in continuous integration (CI) pipelines to copy images between two registries, provide credentials for secured registries, or to promote images from a development registry into production.

About this task

IBM Cloud Pak® for Business Automation provides container images in the IBM® Entitled Registry that are signed following the approach from Red Hat. You can use the signature to verify that the images come from IBM when they are pulled onto the system. For more information about the signature verification, see https://developers.redhat.com/blog/2019/10/29/verifying-signatures-of-red-hat-container-images/.

To inspect the Cloud Pak container images and to convert them if necessary, you can use the skopeo tool. Skopeo is a utility run on the command line that provides various operations with container images and registries.

The following sections can be used to verify the Cloud Pak for Business Automation signature on the container images.

Creating a public key to pull images from IBM Cloud®
Verifying a container image signature
Configuring the image source in a deployment
Cleaning up local images and caches
Scanning for vulnerabilities
Signing unsigned images and generating a GPG key pair to pull them

Creating a public key to pull images from IBM Cloud

The Cloud Pak for Business Automation container images on IBM Entitled Registry are signed by Digicert and can be verified with the following steps.

Create a file icp4a-pubkey.gpg and add the following public key in the GNU Privacy Guard (GPG) format.

Optional: If you want to verify the icp4a-pubkey.gpg file, create a file icp4a.pem and add the following public certificate in the PEM format.

Optional: If you want to verify the icp4a-pubkey.gpg file, create a file icp4a-issuer.pem and add the following issuer public certificate in the PEM format.

Optional: Run the following commands to verify the GPG key and its issuer with the icp4a.pem and icp4a-issuer.pem files.

Table 1. Commands to verify certificate issuance and validity
Command	Description
`openssl x509 -text -in icp4a.pem`	Displays the IBM Cloud Pak for Business Automation public certificate. Verify that the "Subject" line is IBM, and the "Issuer" is www.digicert.com. The icp4a.pem certificate is signed by Digicert.
`gpg2 -v --list-packets icp4a-pubkey.gpg`	Displays the GPG public key. Verify that `pkey[0]` is the same as the "Modulus" of icp4a.pem. The command validates that icp4a-pubkey.gpg is the public key (in GPG format) for IBM Cloud Pak for Business Automation.
`openssl ocsp -no_nonce \ -issuer icp4a-issuer.pem \ -cert icp4a.pem \ -VAfile icp4a-issuer.pem \ -text -url http://ocsp.digicert.com`	Verifies that the certificate icp4a.pem comes from Digicert by using the Online Certificate Status Protocol (OCSP).

Verifying a container image signature

If you pull the images from the IBM Entitled Registry with skopeo, you can enable the container image signature verification. The icp4a-pubkey.gpg file that you created can be used to verify the images under a certain repository and path.

In the following example, images are pulled from "cp.icr.io/cp/cp4a" and verified with the GPG public key that is in the "icp4a-pubkey.gpg" file. All images are rejected, except the signed images in the cp4a folder.

{
  "default": [
    {
      "type": "reject"
    }
  ],
  "transports": {
    "docker": {
      "cp.icr.io/cp/cp4a": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "icp4a-pubkey.gpg"
        }
      ]
    }
  }
}

Note: If "cp.icr.io" is specified, all of the images that are not signed under this folder are rejected. The skopeo command has the useful option "

--policy
mypolicy.json

", which sets a specific policy file instead of using the default /etc/containers/policy.json file.

Configuring the image source in a deployment

You can apply a security policy to specify the source of an image and whether to trust it even after they are deployed. For more information, see https://docs.openshift.com/container-platform/4.5/security/container_security/security-deploy.html#security-deploy-image-sources_security-deploy.

You can use a MachineConfig object to inject the needed configuration files into the nodes. The needed files are the GPG public key file and the policy.json file. The following example includes a /etc/pki/icp4a-pubkey.gpg file and a /etc/containers/policy.json file inline in the URL encoded format.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: test-image-signature
spec:
  config:
    ignition:
      version: 2.2.0
    storage:
      files:
        - contents:
            source: >-
              data:,-----BEGIN%20PGP%20PUBLIC%20KEY%20BLOCK-----%0AmQENBF0UBwABCADAk2QMLvahhx2owpDyasQo0E36I7VM%2FYdK1J9lAAgsHBJFvn%2FP%0A0PznCfnnFAw2cCj49ftYBN8orxJOSkuQE3p3hz%2Fg21C9jrsrqwoNfNbo5WRUsIxA%0A0hL9ywV8EvuOLAKdNY0ACOBHv2g82KSa%2FbZMg3bPA3Ir1jZZ3jjQrAzwAzvhp2Bo%0Av5FwU0xYqm9DwCb%2Fd4yYaEJ28jFWrwGdRnVHuoohu5PyUoQMpt4rwiU1yb3CabaN%0AEpGo4ZF2%2F07sVdhSv1ieqfQMa%2FrC6XcFALygt%2FvxzQDeSkPlxWb8wBqzq%2FsgI5Jb%0AwG6I55f3EMP67d%2FvWzA62FNbXk8DWMRLzWDHABEBAAG0BWljcDRhiQE5BBMBCAAj%0ABQJdFAcAAhsvBwsJCAcDAgEGFQgCCQoLBBYCAwECHgECF4AACgkQcjTsJWhPPpnT%0Ahgf%2FZYjAuPqiiHtJkmm1Nrw5HSCKHMsiHwUc1y5lb%2BJ%2Bxxfz7B3Jm07r%2FR82h6zI%0AZNnUTpgWzbEM7NazSAPcfL%2FKQXd3jw713hIXr2D4wzyMnvVAPJz0U%2FFUXDkZeTil%0AU04PT4T9BF5b%2BkIh560TuouY4bot2QFhuBMnKasLMMrXx1PXbySH9OHWc5bnx3J6%0Aj%2B%2Fd7TMj7MsdVgraoqmPtM68fOQUXyn0FCsx2M4kQRllGp7Wv0rIXVVh5lbT7VFs%0AOP%2B8LBBj1DEfCuffQc%2F21jO4DkC3nalWhXW5jAcAnTLlzKWsAxm%2FYJLfRYzeAOHa%0Amgv9nt%2FuDfwf0GasJw0cEeNq0A%3D%3D%0A%3D9hbX%0A-----END%20PGP%20PUBLIC%20KEY%20BLOCK-----%0A
          filesystem: root
          mode: 420
          path: /etc/pki/icp4a-pubkey.gpg
        - contents:
            source: >-
              data:,%7B%0A%20%20%20%20%22default%22%3A%20%5B%7B%0A%20%20%20%20%20%20%20%20%22type%22%3A%20%22reject%22%0A%20%20%20%20%7D%5D%2C%0A%20%20%20%20%22transports%22%3A%20%7B%0A%20%20%20%20%20%20%20%20%22docker%22%3A%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22cp.stg.icr.io%2Fcp%2Fcp4a%22%3A%20%5B%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%22type%22%3A%20%22signedBy%22%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%22keyType%22%3A%20%22GPGKeys%22%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%22keyPath%22%3A%20%22%2Fetc%2Fpki%2Ficp4a-pubkey.gpg%22%0A%20%20%20%20%20%20%20%20%20%20%20%20%7D%5D%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%7D
          filesystem: root
          mode: 420
          path: /etc/containers/policy.json

With a URL decoder, you can decode the two files into clear text.

Cleaning up local images and caches

Before any test or use of the image signature, you must clean up the container images and the local cache, even if your image pull policy is "always". If an image is cached locally, it is not pulled again and the image signature verification does not take place.

Scanning for vulnerabilities

All of the Cloud Pak container images pass rigorous image vulnerability scans during the development lifecycle, but new vulnerabilities are discovered almost every day. It is likely that most of the vulnerabilities you find are related to the Linux® kernel or to a few embedded packages. IBM tracks these vulnerabilities and provides fixes for them in fix packs or subsequent releases. To find out more, you can contact IBM support.

Signing unsigned images and generating a GPG key pair to pull them

The container images in the Passport Advantage® (PPA) packages are not signed. If you download the PPA packages, you can sign these images with your own private key, and then push them into a local or corporate registry.

Follow the OpenShift Container Platform (OCP) documentation to sign a container image, generate a GPG private key, and a public key "mysignedkey.gpg" that is generated from it. For more information, see Managing Containers: Chapter 3. Signing Container Images.

Edit the /etc/containers/policy.json file on your target OCP image registry to include the GPG public key "mysignedkey.gpg".

...
    "docker": {
      "$(oc registry info)": [
        {
          "type": "signedBy",
          "keyType": "GPGKeys",
          "keyPath": "/etc/pki/containers/mysignedkey.gpg"
        }
      ]
    }
...

Copy the GPG public key to the registry keyPath location.

sudo mv mysignedkey.gpg /etc/pki/containers/mysignedkey.gpg

Use the skopeo copy command to create a mirror of the Cloud Pak for Business Automation container image. The command needs the --remove-signatures option.