September 20, 2019 By Powell Quiring 6 min read

Learn how to use the IBM Cloud CLI and Terraform to bring an encrypted cloud-init enabled image to the IBM Cloud.  

Your company may have data-at-rest encryption requirements that include the images associated with provisioned Virtual Server Instances (VSIs). It is possible to encrypt an industry-standard Virtual Hardware Device (VHD) cloud-init image on-premises with your own data encryption key and provision an IBM VSI based on that encrypted image. The VHD encryption starts on one end—on-premises—and stays encrypted even as it is used on the other end—an IBM Cloud VSI. Hence, the name end-to-end encryption.

Migrating one virtual server instance

You can encrypt and migrate a cloud-init enabled image that you have built on-premises. My on-premises starting point is an IBM Cloud Virtual Server Instance. It may be helpful to follow these instructions as you are learning the process, or you can choose to skip my “on-premises” setup and feather in your own process.

At a high level, the steps will be as follows:

  1. Populate a running “on-premises” VSI with the software and data required.
  2. Encrypt the VHD image using an AES 512 bit key.
  3. Import the encrypted image from a bucket in IBM Cloud Object Storage.
  4. Provision a new VSI from the encrypted VSI template.

Using automation to avoid repetitive steps

It is possible to go through most of these steps using the GUI, with the exception of the steps described in 050-use-vsi-encrypter-to-encrypt-cos-image.sh below.

Going through this process for one virtual server is okay but if you want to integrate this process into a CI/CD pipeline or have several images, you may want to look into automation. 

This is exactly what I did; once you go through the steps manually, it becomes clear that all those steps in the IBM Cloud console can be automated with their command line counterparts. Equipped with IBM Cloud CLI and Terraform, I decided to write a set of shell scripts and Terraform templates to prove it.

The files can be found in the Git repository. The scripts automate all steps you would find in the docs under Using End to End (E2E) Encryption to provision an encrypted instance:

  1. Create a Cloud Object Storage (COS) instance and a bucket to store the image to capture.
  2. Key Protect instance and Key to securely wrap the 512 bit AES encryption key.
  3. Encrypt the image and copy the encrypted image to COS object.
  4. Create an encrypted image template from the encrypted COS object.
  5. Create a test VSI from the encrypted image template.

On-premises mock up:

  1. Create an on-premises VSI with Nginx installed.
  2. Capture the VSI image and wait for the image to be ready.
  3. Copy the image to COS.

Running the automation scripts

To make it easier to understand what happens, I created several scripts mapping to the high-level steps:

~/github.com/IBM-Cloud/vsi-encrypted $ ls -1a 0*
000-initialize-verify-prereqs.sh
010-prepare-cos.sh
015-create-sshkey-in-cloud.sh
020-create-onprem-vm.sh
030-capture-classic-to-cos.sh
040-create-encrypter-vm.sh
050-use-vsi-encrypter-to-encrypt-cos-image.sh
060-prepare-kp.sh
070-create-encrypted-vm-image-template.sh
080-create-test-vm.sh
090-cleanup.sh

Instructions on how to run the scripts can be found in the folder “README.” You will need the right permissions to work with VSIs in Classic Infrastructure, with images, with Cloud Object Storage, and with Key Protect. 

In addition, the scripts rely on IBM Cloud CLI, Terraform, IBM Cloud Provider for Terraform, and JQ.

Once you have configured your environment with the local.env file, you can start running the scripts—000-initialize-verify-prereqs.sh, 010-prepare-cos.sh, etc. Only call 090-cleanup.sh to remove all resources created by the previous scripts.

000-initialize-verify-prereqs.sh initializes the cli with the region and resource group configured in local.env and makes basic checks for the required IBM Cloud plugins and external tools.

010-prepare-cos.sh creates a Cloud Object Storage service and a bucket. The initial unencrypted VHD and, later, the encrypted VHD will be put into this bucket:

The VSIs, ssh keys for the VSIs, and image templates are found in the Classic section of the IBM Cloud console:

015-create-sshkey-in-cloud.sh uses Terraform to create an ssh key:

020-create-onprem-vm.sh uses Terraform to create a VSI that represents your on-premises computer. Load the boot disk with the data and programs you want to use in the cloud. I just put Nginx on mine:

030-capture-classic-to-cos.sh creates a VHD shown as an Image Template. It then copies the VHD into the COS bucket:

040-create-encrypter-vm.sh creates a VM that will be used to encrypt the VHD image.

050-use-vsi-encrypter-to-encrypt-cos-image.sh runs a script on the VM to get the VHD image from COS, then encrypts the VHD image and writes the encrypted VHD image back to the same bucket. The encrypting and image are done with the vhd-util command:

vhd-util copy -n INPUT -N OUTPUT -k dek

The dek is the file that contains the AES 512 bit encryption key. I created one using the dd command. You may have a corporate policy the defines the method used to create your key:

dd if=/dev/urandom of=dek bs=64 count=1

You will notice that the ibmcloud cos commands initialize an environment variable, IBMCLOUD_TRACE=false, to work around a bug that may have been fixed by the time you read this:

IBMCLOUD_TRACE=false ibmcloud cos get-object --bucket "$COS_BUCKET_NAME" --key $IMAGE --region $COS_REGION $IMAGE

060-prepare-kp.sh creates a Key Protect service instance, root key, and IAM authorization for block storage to access the Key Protect instance:

The VHD is encrypted at rest but it will be decrypted as it is read into memory and encrypted before writing to block storage. The next step demonstrates the use of Key Protect to wrap the data encryption key. In this step, IAM authorization is provided.

The authorization is visible in IAM:

070-create-encrypted-vm-image-template.sh creates a VSI template encrypted image from the encrypted VHD image that in cos.

A wrapped encryption key will be associated with image template. First, the binary data encryption key will be converted into ascii using base64.

Then, the ascii version will be wrapped. In the script code, notice the following:

if [ x"$(uname)" = xDarwin ]; then
    DEK_BASE64=$(base64 < dek)
else  
    DEK_BASE64=$(base64 -w 0 < dek)
fi
WRAPPED_DEK_BASE64=$(ibmcloud kp wrap $KP_KEY_ID -i $KP_GUID --plaintext $DEK_BASE64 --output json | jq -r '.Ciphertext')

DEK_BASE64 is the ascii version of the data encryption key.

WRAPPED_DEK_BASE64 is the encrypted representation of the ascii. The Key Protect service uses the root key identified as the key material to perform the wrap operation.  This clear text string does not need to be kept secure since it can only be converted back to the DEK_BASE64 string by those that have access to the key protect instance.

Use the IAM service to restrict access to the Key Protect instance keys.  

Finally, create the VSI encrypted image template:

ibmcloud sl call-api SoftLayer_Virtual_Guest_Block_Device_Template_Group createFromIcos \
  --parameters '[{
      "bootMode":"HVM",
      "operatingSystemReferenceCode":"CENTOS_7_64",
      "cloudInit":true,
      "name": "'$IMAGE_ENCRYPTED'",
      "ibmApiKey": "'$IBMCLOUD_API_KEY'",
      "isEncrypted": true,
      "uri": "cos://'$COS_REGION'/'$COS_BUCKET_NAME'/'$IMAGE_ENCRYPTED'",
      "wrappedDek":"'$WRAPPED_DEK_BASE64'",
      "crkCrn":"'$KP_CRN'"
  }]'

Notice the parameters:

  • isEncrypted: This image is encrypted.
  • uri: Location of the encrypted image in COS.
  • wrappedDek: Data encryption key converted to base64 then wrapped by Key Protect.
  • crkCrn: The Cloud Resource Name (crn) of the Key Protect Key—it identifies both the Key Protect instance and the Key ID.

Here is a sample run:

+WRAPPED_DEK_BASE64=eyJjaXBoZXJ0ZXh0IjoiVTJ1NVRJVEoxY3lZWUZ1K1B4TnhNUHRNT3dkT2QzRzY0d1NTdG1ISC9FWUZSelJHWjNBZFc5RldLTXUrN2FrSnk0SUtKSTBEVHd2MHBOdDhhcjhpa2xQbFpOeUZYdUFXajZSTUJnPT0iLCJpdiI6InQrN2FmSGpPSmljVVl4aUMiLCJ2ZXJzaW9uIjoiNC4wLjAiLCJoYW5kbGUiOiJhN2E5MDc3My1kYjExLTRhZTYtYmI0Yi1jY2EzNDYwOGIyMjMifQ==
+ ibmcloud sl call-api SoftLayer_Virtual_Guest_Block_Device_Template_Group createFromIcos --parameters '[{
      "bootMode":"HVM",
      "operatingSystemReferenceCode":"CENTOS_7_64",
      "cloudInit":true,
      "name": "pfq00-e2e-89599686-image-encrypted-0.vhd",
      "ibmApiKey": "mysecretAPIkey",
      "isEncrypted": true,
      "uri": "cos://us-south/pfq00-e2e-classic-images/pfq00-e2e-89599686-image-encrypted-0.vhd",
      "wrappedDek":"eyJjaXBoZXJ0ZXh0IjoiVTJ1NVRJVEoxY3lZWUZ1K1B4TnhNUHRNT3dkT2QzRzY0d1NTdG1ISC9FWUZSelJHWjNBZFc5RldLTXUrN2FrSnk0SUtKSTBEVHd2MHBOdDhhcjhpa2xQbFpOeUZYdUFXajZSTUJnPT0iLCJpdiI6InQrN2FmSGpPSmljVVl4aUMiLCJ2ZXJzaW9uIjoiNC4wLjAiLCJoYW5kbGUiOiJhN2E5MDc3My1kYjExLTRhZTYtYmI0Yi1jY2EzNDYwOGIyMjMifQ==",
      "crkCrn":"crn:v1:bluemix:public:kms:us-south:a/713c729476507a53135fe6793c37cc74:a9c41c7e-0053-419c-806b-78dc569c1c34:key:a7a90773-db11-4ae6-bb4b-cca34608b223"
  }]'

Notice that the Image Type confirms it is encrypted:

080-create-test-vm.sh

Finally, a VSI can be started with the encrypted image. Nothing special here—just provision the instance:

In the Storage panel, confirm it is encrypted:

090-cleanup.sh deletes everything that was created—image templates, COS instance, Key Protect instance, VSI instances—using a mix of IBM Cloud CLI and Terraform.

Conclusion

The scripts listed here handle the most simple encryption use case but give the foundation of more complex setup

Keep in mind that only instances created with a cloud-init enabled image can be encrypted with this process. Also, only image templates with a single primary boot volume (or disk) can be encrypted. Secondary disks are not supported.

If you have feedback, suggestions, or questions about this post, please reach out to me on Twitter: @powellquiring.

Was this article helpful?
YesNo

More from Cloud

IBM Cloud Virtual Servers and Intel launch new custom cloud sandbox

4 min read - A new sandbox that use IBM Cloud Virtual Servers for VPC invites customers into a nonproduction environment to test the performance of 2nd Gen and 4th Gen Intel® Xeon® processors across various applications. Addressing performance concerns in a test environment Performance testing is crucial to understanding the efficiency of complex applications inside your cloud hosting environment. Yes, even in managed enterprise environments like IBM Cloud®. Although we can deliver the latest hardware and software across global data centers designed for…

10 industries that use distributed computing

6 min read - Distributed computing is a process that uses numerous computing resources in different operating locations to mimic the processes of a single computer. Distributed computing assembles different computers, servers and computer networks to accomplish computing tasks of widely varying sizes and purposes. Distributed computing even works in the cloud. And while it’s true that distributed cloud computing and cloud computing are essentially the same in theory, in practice, they differ in their global reach, with distributed cloud computing able to extend…

How a US bank modernized its mainframe applications with IBM Consulting and Microsoft Azure

9 min read - As organizations strive to stay ahead of the curve in today's fast-paced digital landscape, mainframe application modernization has emerged as a critical component of any digital transformation strategy. In this blog, we'll discuss the example of a US bank which embarked on a journey to modernize its mainframe applications. This strategic project has helped it to transform into a more modern, flexible and agile business. In looking at the ways in which it approached the problem, you’ll gain insights into…

IBM Newsletters

Get our newsletters and topic updates that deliver the latest thought leadership and insights on emerging trends.
Subscribe now More newsletters