As of 15 December 2020, you can now deploy containers from encrypted images in Red Hat OpenShift on IBM Cloud clusters that run version 4.4 or later.
Container images are the industry-standard artifacts for packaging, shipping and deploying applications with their dependencies. Oftentimes container images contain sensitive information (e.g., proprietary algorithms) that are not meant to be shared with the public. To protect these private images from being accessed by others, organizations established internal build pipelines that push container images into private container registries that are not accessible by unauthorized clients. This is a good practice, but no system is unbreakable; if attackers managed to break into your private container registry, they could steal your private container images to run them elsewhere, examine their contents, find application weaknesses and so on.
Container image encryption can help advance security measures by using private-public key pairs to encrypt and decrypt container image layers. Encrypted container images cannot be accessed without the appropriate decryption keys, which decreases the exposure of images to information theft.
This is only one use case for container image encryption, but there are several others.
Encrypting container images
To build an encrypted container you do not have to change anything in your existing Dockerfiles but you will need some new tools, such as buildah or skopeo. You will also need to generate the private-public key pair(s) that you are going to use for encryption.
There are multiple workflows for building, encrypting and pushing container images, including the following examples:
- You can use buildah to build a container image from Dockerfile, then push it to a registry and specify an encryption key.
- You can use skopeo to push an existing container to a registry and specify an encryption key.
For more detailed instructions on building, encrypting and pushing container images, I recommend reading the official documentation.
This example shows the steps to build a container image using a Dockerfile, generate a private-public key pair, use the key pair to encrypt the image and push the image into a registry.
Start with a Dockerfile, such as this example of a super-sensitive container:
buildah bud command to build the image:
Before encrypting and pushing the container, generate a private-public key pair to use for encryption:
buildah bud command to encrypt and push the image:
Delete the local image, and then try to pull the image with
This error is expected. Now, try to pull the image with the private key specified:
This time the image is pulled successfully.
Deploying applications from encrypted container images
Most container runtimes can handle encrypted container images. In the case of Red Hat OpenShift on IBM Cloud, we are using cri-o because it provides support for encrypted container images since version 1.17.
When using encrypted container images, you must provide the appropriate decryption keys to the container runtime. When using cri-o, this means placing the private keys under the
/etc/crio/keys directory (with the default settings).
For OpenShift clusters, there are two challenges to providing decryption keys:
- The decryption keys must be provided to cri-o on every worker node.
- The keys should not be manually copied into worker node file systems.
To meet these challenges, you can use the IBM Cloud Image Key Synchronizer managed add-on. This cluster add-on allows you to specify decryption keys in the form of Kubernetes secret resources and takes care of synchronizing these keys onto the worker nodes.
After you enable the add-on for your OpenShift cluster, all you have to do is specify the keys as secrets and you are ready to go.
For more detailed instructions on using IBM Cloud Image Key Synchronizer, I recommend reading the official documentation.
Example using plain private keys
This example shows the steps to install the IBM Cloud Image Key Synchronizer managed add-on onto a test OpenShift cluster and create a pod using the image that was built in the previous example.
First, identify an OpenShift 4.4 or later cluster that you can use:
Install the IBM Cloud Image Key Synchronizer managed add-on in the cluster:
After the add-on is deployed, a DaemonSet is created in new project called
Try to deploy the container that you built and pushed in the previous example:
The pod should not start, as the decryption keys haven't been specified yet:
Wonderful. Next, specify the decryption key in a secret and create the secret in the
Now, delete the pod and create it again:
Perfect! This time, the encrypted container image was pulled and the pod was started successfully.
Example using IBM Key Protect for IBM Cloud-wrapped private keys
Key Protect can help you work with encryption keys more securely. By using Key Protect, you can wrap your private keys using a specific root key and specify the wrapped key instead of the plain private key.
In the following example, a pre-created Key Protect instance and root key are used. For more details on creating a service instance and root key, I recommend reading the official documentation.
The IBM Cloud Image Key Synchronizer requires some configuration to know how to connect to your Key Protect instance:
After creating the configuration, you can start specifying Key Protect-wrapped keys. The following snippet uses the
ibmcloud kp key wrap command to wrap the contents of your
private.pem key file using a root key under your Key Protect instance. Key Protect returns a ciphertext that is unusable without unwrapping. The configuration that was provided above allows the IBM Cloud Image Key Synchronizer to unwrap the provided secrets:
Now, delete test pod and recreate it again:
Well done! This time the encrypted image was successfully pulled using a Key Protect wrapped private key.
To learn more about encrypted container images, check out the Encrypted container images for container image security at rest and Advancing container image security with encrypted container images articles.
For more information, check out the official documentation.