Deploy Containers From Encrypted Images in Red Hat OpenShift on IBM Cloud Clusters

As of 15 December 2020, you can now deploy containers from encrypted images in Red Hat OpenShift on IBM Cloud clusters that run version 4.4 or later.

Container images are the industry-standard artifacts for packaging, shipping and deploying applications with their dependencies. Oftentimes container images contain sensitive information (e.g., proprietary algorithms) that are not meant to be shared with the public. To protect these private images from being accessed by others, organizations established internal build pipelines that push container images into private container registries that are not accessible by unauthorized clients. This is a good practice, but no system is unbreakable; if attackers managed to break into your private container registry, they could steal your private container images to run them elsewhere, examine their contents, find application weaknesses and so on.

Container image encryption can help advance security measures by using private-public key pairs to encrypt and decrypt container image layers. Encrypted container images cannot be accessed without the appropriate decryption keys, which decreases the exposure of images to information theft.

This is only one use case for container image encryption, but there are several others.

Encrypting container images

To build an encrypted container you do not have to change anything in your existing Dockerfiles but you will need some new tools, such as buildah or skopeo . You will also need to generate the private-public key pair(s) that you are going to use for encryption.

There are multiple workflows for building, encrypting and pushing container images, including the following examples:

You can use buildah to build a container image from Dockerfile, then push it to a registry and specify an encryption key.
You can use skopeo to push an existing container to a registry and specify an encryption key.

For more detailed instructions on building, encrypting and pushing container images, I recommend reading the official documentation.

Example

This example shows the steps to build a container image using a Dockerfile, generate a private-public key pair, use the key pair to encrypt the image and push the image into a registry.

Start with a Dockerfile, such as this example of a super-sensitive container:

FROM docker/whalesay
RUN apt update && apt install fortune -y
CMD while true; do /usr/games/fortune | cowsay; sleep 10; done

Use the buildah bud command to build the image:

$ buildah bud -t us.icr.io/attila-fabian/mycontainer:latest .
...
979337e38c8e28491e14ba4d0fabaac5b49ff80380eeb2783ed1b06d14754542


$ buildah images
REPOSITORY                                             TAG      IMAGE ID       CREATED          SIZE
us.icr.io/attila-fabian/mycontainer                    latest   979337e38c8e   19 seconds ago   266 MB

Before encrypting and pushing the container, generate a private-public key pair to use for encryption:

$ openssl genrsa --out private.pem
Generating RSA private key, 2048 bit long modulus (2 primes)
...

$ openssl rsa -in private.pem -pubout -out public.pem
writing RSA key

$ ls
private.pem  public.pem

Use the buildah bud command to encrypt and push the image:

$ buildah push --encryption-key jwe:public.pem us.icr.io/attila-fabian/mycontainer:latest
Getting image source signatures
...
Writing manifest to image destination
Storing signatures

Delete the local image, and then try to pull the image with buildah bud:

$ buildah rmi us.icr.io/attila-fabian/mycontainer:latest
untagged: us.icr.io/attila-fabian/mycontainer:latest
979337e38c8e28491e14ba4d0fabaac5b49ff80380eeb2783ed1b06d14754542

$ buildah pull us.icr.io/attila-fabian/mycontainer:latest
Getting image source signatures
...
Error decrypting layer sha256:45a9fb9687e20fec30a55939219706470948ef11a4e4080a6df23e3726000470: missing private key needed for decryption
ERRO exit status 125

This error is expected. Now, try to pull the image with the private key specified:

$ buildah pull --decryption-key private.pem us.icr.io/attila-fabian/mycontainer:latest
Getting image source signatures
...
Writing manifest to image destination
Storing signatures
979337e38c8e28491e14ba4d0fabaac5b49ff80380eeb2783ed1b06d14754542

This time the image is pulled successfully.

Deploying applications from encrypted container images

Most container runtimes can handle encrypted container images. In the case of Red Hat OpenShift on IBM Cloud, we are using cri-o because it provides support for encrypted container images since version 1.17.

When using encrypted container images, you must provide the appropriate decryption keys to the container runtime. When using cri-o, this means placing the private keys under the /etc/crio/keys directory (with the default settings).

For OpenShift clusters, there are two challenges to providing decryption keys:

The decryption keys must be provided to cri-o on every worker node.
The keys should not be manually copied into worker node file systems.

To meet these challenges, you can use the IBM Cloud Image Key Synchronizer managed add-on. This cluster add-on allows you to specify decryption keys in the form of Kubernetes secret resources and takes care of synchronizing these keys onto the worker nodes.

After you enable the add-on for your OpenShift cluster, all you have to do is specify the keys as secrets and you are ready to go.

For more detailed instructions on using IBM Cloud Image Key Synchronizer, I recommend reading the official documentation.

Example using plain private keys

This example shows the steps to install the IBM Cloud Image Key Synchronizer managed add-on onto a test OpenShift cluster and create a pod using the image that was built in the previous example.

First, identify an OpenShift 4.4 or later cluster that you can use:

$ ibmcloud oc cluster ls
OK
Name                               ID                     State     Created        Workers   Location   Version                 Resource Group Name   Provider
encrypted-containers-example       bvbs8g8s0akvcjht7f7g   normal    3 weeks ago    2         Sydney     4.5.18_1523_openshift   Default               classic

Install the IBM Cloud Image Key Synchronizer managed add-on in the cluster:

$ ibmcloud oc cluster addon enable image-key-synchronizer --cluster bvbs8g8s0akvcjht7f7g
Enabling add-on image-key-synchronizer for cluster bvbs8g8s0akvcjht7f7g...
The add-on might take several minutes to deploy and become ready for use.
OK

After the add-on is deployed, a DaemonSet is created in new project called image-key-synchronizer:

$ oc get daemonset -n image-key-synchronizer
NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
addon-image-key-synchronizer   2         2         2       2            2           <none>          14s

Try to deploy the container that you built and pushed in the previous example:

$ oc apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - image: us.icr.io/attila-fabian/mycontainer:latest
    name: mycontainer
    imagePullPolicy: Always
EOF
pod/mypod created

The pod should not start, as the decryption keys haven’t been specified yet:

$ oc describe pod mypod
Name:         mypod
Namespace:    default
...
Containers:
  mycontainer:
    Image:          us.icr.io/attila-fabian/mycontainer:latest
    ...
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    ...
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
...
Events:
  Type     Reason          Age                 From                     Message
  ----     ------          ----                ----                     -------
  ...
  Warning  Failed          45s (x3 over 102s)  kubelet, 10.138.107.220  Failed to pull image "us.icr.io/attila-fabian/mycontainer:latest": rpc error: code = Unknown desc = Error decrypting layer sha256:45a9fb9687e20fec30a55939219706470948ef11a4e4080a6df23e3726000470: no suitable key unwrapper found or none of the private keys could be used for decryption
  Warning  Failed          45s (x3 over 102s)  kubelet, 10.138.107.220  Error: ErrImagePull
  Normal   BackOff         18s (x4 over 101s)  kubelet, 10.138.107.220  Back-off pulling image "us.icr.io/attila-fabian/mycontainer:latest"
  Warning  Failed          18s (x4 over 101s)  kubelet, 10.138.107.220  Error: ImagePullBackOff
  Normal   Pulling         5s (x4 over 114s)   kubelet, 10.138.107.220  Pulling image "us.icr.io/attila-fabian/mycontainer:latest"

Wonderful. Next, specify the decryption key in a secret and create the secret in the image-key-synchronizer project:

$ oc apply -f - <<EOF
apiVersion: v1
kind: Secret
type: key
metadata:
  name: mydecryptionsecret
  namespace: image-key-synchronizer
data:
  private.pem: $(cat private.pem | base64 | tr -d "\n")
EOF
secret/mydecryptionsecret created

Now, delete the pod and create it again:

$ oc delete pod mypod
pod "mypod" deleted

$ oc apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - image: us.icr.io/attila-fabian/mycontainer:latest
    name: mycontainer
    imagePullPolicy: Always
EOF
pod/mypod created

$ oc get pods
NAME    READY   STATUS    RESTARTS   AGE
mypod   1/1     Running   0          35s

$ oc logs mypod
...
 ______________________________________
/ F.S. Fitzgerald to Hemingway:        \
|                                      |
| "Ernest, the rich are different from |
| us." Hemingway:                      |
|                                      |
\ "Yes. They have more money."         /
 --------------------------------------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/
...

Perfect! This time, the encrypted container image was pulled and the pod was started successfully.

Example using IBM Key Protect for IBM Cloud-wrapped private keys

The IBM Cloud Image Key Synchronizer add-on provides integration with the IBM Key Protect for IBM Cloud service.

Key Protect can help you work with encryption keys more securely. By using Key Protect, you can wrap your private keys using a specific root key and specify the wrapped key instead of the plain private key.

In the following example, a pre-created Key Protect instance and root key are used. For more details on creating a service instance and root key, I recommend reading the official documentation.

$ ibmcloud kp keys --instance-id 832fdb1a-c3b3-exmp-847e-31f238f4767c
Retrieving keys...
OK
Key ID                                 Key Name
883cc3d3-d281-44f3-a1d5-372289491c77   test-root-key

The IBM Cloud Image Key Synchronizer requires some configuration to know how to connect to your Key Protect instance:

oc apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: keyprotect-config
  namespace: image-key-synchronizer
type: Opaque
stringData:
  config.json: |
      {
          "keyprotect-url":"https://us-south.kms.cloud.ibm.com",
          "instance-id": "832fdb1a-c3b3-exmp-847e-31f238f4767c",
          "apikey": "5ZtsYoNA0zexamleD8Av7lDSuaof-w7examleKc_MGgE"
      }
EOF
secret/keyprotect-config created

After creating the configuration, you can start specifying Key Protect-wrapped keys. The following snippet uses the ibmcloud kp key wrap command to wrap the contents of your private.pem key file using a root key under your Key Protect instance. Key Protect returns a ciphertext that is unusable without unwrapping. The configuration that was provided above allows the IBM Cloud Image Key Synchronizer to unwrap the provided secrets:

$ oc apply -f - <<EOF
apiVersion: v1
kind: Secret
type: kp-key
metadata:
  name: mywrappeddecryptionsecret
  namespace: image-key-synchronizer
data:
  private.pem: $(ibmcloud kp key wrap 883cc3d3-d281-44f3-a1d5-372289491c77 --plaintext "$(cat private.pem | base64)" --instance-id 832fdb1a-c3b3-exmp-847e-31f238f4767c -o json | jq -r .Ciphertext)
EOF
secret/mywrappeddecryptionsecret created

Now, delete test pod and recreate it again:

$ oc delete pod mypod
pod "mypod" deleted

$ oc apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - image: us.icr.io/attila-fabian/mycontainer:latest
    name: mycontainer
    imagePullPolicy: Always
EOF
pod/mypod created

$ oc get pods
NAME    READY   STATUS    RESTARTS   AGE
mypod   1/1     Running   0          55s

$ oc logs mypod
...
 __________________________________
< Someone is speaking well of you. >
 ----------------------------------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/
...

Well done! This time the encrypted image was successfully pulled using a Key Protect wrapped private key.

More information

To learn more about encrypted container images, check out the Encrypted container images for container image security at rest and Advancing container image security with encrypted container images articles.

For more information, check out the official documentation.

Contact us

If you have questions, engage our team via Slack by registering here and join the discussion in the #general channel on our public IBM Cloud Kubernetes Service Slack.

Was this article helpful?

YesNo

Attila Fábián

Software Engineer, IBM Cloud Kubernetes Service

As of 15 December 2020, you can now deploy containers from encrypted images in Red Hat OpenShift on IBM Cloud clusters that run version 4.4 or later.

Encrypting container images

Example

Deploying applications from encrypted container images

Example using plain private keys

Example using IBM Key Protect for IBM Cloud-wrapped private keys

More information

Contact us

More from Cloud

Serverless vs. microservices: Which architecture is best for your business?

Serverless use cases: How enterprises are using the technology to let developers innovate

How a US bank modernized its mainframe applications with IBM Consulting and Microsoft Azure

IBM Newsletters