Automate the Encryption of a Virtual Server Image for Deployment onto Classic Infrastructure

5 min read

Learn how to use the IBM Cloud CLI and Terraform to bring an encrypted cloud-init enabled image to the IBM Cloud.  

Your company may have data-at-rest encryption requirements that include the images associated with provisioned Virtual Server Instances (VSIs). It is possible to encrypt an industry-standard Virtual Hardware Device (VHD) cloud-init image on-premises with your own data encryption key and provision an IBM VSI based on that encrypted image. The VHD encryption starts on one end—on-premises—and stays encrypted even as it is used on the other end—an IBM Cloud VSI. Hence, the name end-to-end encryption.

Migrating one virtual server instance

You can encrypt and migrate a cloud-init enabled image that you have built on-premises. My on-premises starting point is an IBM Cloud Virtual Server Instance. It may be helpful to follow these instructions as you are learning the process, or you can choose to skip my "on-premises" setup and feather in your own process.

At a high level, the steps will be as follows:

  1. Populate a running "on-premises" VSI with the software and data required.
  2. Encrypt the VHD image using an AES 512 bit key.
  3. Import the encrypted image from a bucket in IBM Cloud Object Storage.
  4. Provision a new VSI from the encrypted VSI template.
Migrating one virtual server instance

Using automation to avoid repetitive steps

It is possible to go through most of these steps using the GUI, with the exception of the steps described in 050-use-vsi-encrypter-to-encrypt-cos-image.sh below.

Going through this process for one virtual server is okay but if you want to integrate this process into a CI/CD pipeline or have several images, you may want to look into automation. 

This is exactly what I did; once you go through the steps manually, it becomes clear that all those steps in the IBM Cloud console can be automated with their command line counterparts. Equipped with IBM Cloud CLI and Terraform, I decided to write a set of shell scripts and Terraform templates to prove it.

The files can be found in the Git repository. The scripts automate all steps you would find in the docs under Using End to End (E2E) Encryption to provision an encrypted instance:

  1. Create a Cloud Object Storage (COS) instance and a bucket to store the image to capture.
  2. Key Protect instance and Key to securely wrap the 512 bit AES encryption key.
  3. Encrypt the image and copy the encrypted image to COS object.
  4. Create an encrypted image template from the encrypted COS object.
  5. Create a test VSI from the encrypted image template.

On-premises mock up:

  1. Create an on-premises VSI with Nginx installed.
  2. Capture the VSI image and wait for the image to be ready.
  3. Copy the image to COS.

Running the automation scripts

To make it easier to understand what happens, I created several scripts mapping to the high-level steps:

~/github.com/IBM-Cloud/vsi-encrypted $ ls -1a 0*
000-initialize-verify-prereqs.sh
010-prepare-cos.sh
015-create-sshkey-in-cloud.sh
020-create-onprem-vm.sh
030-capture-classic-to-cos.sh
040-create-encrypter-vm.sh
050-use-vsi-encrypter-to-encrypt-cos-image.sh
060-prepare-kp.sh
070-create-encrypted-vm-image-template.sh
080-create-test-vm.sh
090-cleanup.sh

Instructions on how to run the scripts can be found in the folder "README." You will need the right permissions to work with VSIs in Classic Infrastructure, with images, with Cloud Object Storage, and with Key Protect. 

In addition, the scripts rely on IBM Cloud CLI, Terraform, IBM Cloud Provider for Terraform, and JQ.

Once you have configured your environment with the local.env file, you can start running the scripts—000-initialize-verify-prereqs.sh, 010-prepare-cos.sh, etc. Only call 090-cleanup.sh to remove all resources created by the previous scripts.

000-initialize-verify-prereqs.sh initializes the cli with the region and resource group configured in local.env and makes basic checks for the required IBM Cloud plugins and external tools.

010-prepare-cos.sh creates a Cloud Object Storage service and a bucket. The initial unencrypted VHD and, later, the encrypted VHD will be put into this bucket:

010-prepare-cos.sh creates a Cloud Object Storage service and a bucket. The initial unencrypted VHD and, later, the encrypted VHD will be put into this bucket:

The VSIs, ssh keys for the VSIs, and image templates are found in the Classic section of the IBM Cloud console:

The VSIs, ssh keys for the VSIs, and image templates are found in the Classic section of the IBM Cloud console:
The VSIs, ssh keys for the VSIs, and image templates are found in the Classic section of the IBM Cloud console:

015-create-sshkey-in-cloud.sh uses Terraform to create an ssh key:

015-create-sshkey-in-cloud.sh uses Terraform to create an ssh key.
015-create-sshkey-in-cloud.sh uses Terraform to create an ssh key.

020-create-onprem-vm.sh uses Terraform to create a VSI that represents your on-premises computer. Load the boot disk with the data and programs you want to use in the cloud. I just put Nginx on mine:

020-create-onprem-vm.sh uses Terraform to create a VSI that represents your on-premises computer

030-capture-classic-to-cos.sh creates a VHD shown as an Image Template. It then copies the VHD into the COS bucket:

030-capture-classic-to-cos.sh creates a VHD shown as an Image Template. It then copies the VHD into the COS bucket:
030-capture-classic-to-cos.sh creates a VHD shown as an Image Template. It then copies the VHD into the COS bucket:

040-create-encrypter-vm.sh creates a VM that will be used to encrypt the VHD image.

050-use-vsi-encrypter-to-encrypt-cos-image.sh runs a script on the VM to get the VHD image from COS, then encrypts the VHD image and writes the encrypted VHD image back to the same bucket. The encrypting and image are done with the vhd-util command:

vhd-util copy -n INPUT -N OUTPUT -k dek

The dek is the file that contains the AES 512 bit encryption key. I created one using the dd command. You may have a corporate policy the defines the method used to create your key:

dd if=/dev/urandom of=dek bs=64 count=1

You will notice that the ibmcloud cos commands initialize an environment variable, IBMCLOUD_TRACE=false, to work around a bug that may have been fixed by the time you read this:

IBMCLOUD_TRACE=false ibmcloud cos get-object --bucket "$COS_BUCKET_NAME" --key $IMAGE --region $COS_REGION $IMAGE
op15

060-prepare-kp.sh creates a Key Protect service instance, root key, and IAM authorization for block storage to access the Key Protect instance:

op16

The VHD is encrypted at rest but it will be decrypted as it is read into memory and encrypted before writing to block storage. The next step demonstrates the use of Key Protect to wrap the data encryption key. In this step, IAM authorization is provided.

The authorization is visible in IAM:

op17
op181

070-create-encrypted-vm-image-template.sh creates a VSI template encrypted image from the encrypted VHD image that in cos.

A wrapped encryption key will be associated with image template. First, the binary data encryption key will be converted into ascii using base64.

Then, the ascii version will be wrapped. In the script code, notice the following:

if [ x"$(uname)" = xDarwin ]; then
    DEK_BASE64=$(base64 < dek)
else  
    DEK_BASE64=$(base64 -w 0 < dek)
fi
WRAPPED_DEK_BASE64=$(ibmcloud kp wrap $KP_KEY_ID -i $KP_GUID --plaintext $DEK_BASE64 --output json | jq -r '.Ciphertext')

DEK_BASE64 is the ascii version of the data encryption key.

WRAPPED_DEK_BASE64 is the encrypted representation of the ascii. The Key Protect service uses the root key identified as the key material to perform the wrap operation.  This clear text string does not need to be kept secure since it can only be converted back to the DEK_BASE64 string by those that have access to the key protect instance.

Use the IAM service to restrict access to the Key Protect instance keys.  

Finally, create the VSI encrypted image template:

ibmcloud sl call-api SoftLayer_Virtual_Guest_Block_Device_Template_Group createFromIcos \
  --parameters '[{
      "bootMode":"HVM",
      "operatingSystemReferenceCode":"CENTOS_7_64",
      "cloudInit":true,
      "name": "'$IMAGE_ENCRYPTED'",
      "ibmApiKey": "'$IBMCLOUD_API_KEY'",
      "isEncrypted": true,
      "uri": "cos://'$COS_REGION'/'$COS_BUCKET_NAME'/'$IMAGE_ENCRYPTED'",
      "wrappedDek":"'$WRAPPED_DEK_BASE64'",
      "crkCrn":"'$KP_CRN'"
  }]'

Notice the parameters:

  • isEncrypted: This image is encrypted.
  • uri: Location of the encrypted image in COS.
  • wrappedDek: Data encryption key converted to base64 then wrapped by Key Protect.
  • crkCrn: The Cloud Resource Name (crn) of the Key Protect Key—it identifies both the Key Protect instance and the Key ID.

Here is a sample run:

+WRAPPED_DEK_BASE64=eyJjaXBoZXJ0ZXh0IjoiVTJ1NVRJVEoxY3lZWUZ1K1B4TnhNUHRNT3dkT2QzRzY0d1NTdG1ISC9FWUZSelJHWjNBZFc5RldLTXUrN2FrSnk0SUtKSTBEVHd2MHBOdDhhcjhpa2xQbFpOeUZYdUFXajZSTUJnPT0iLCJpdiI6InQrN2FmSGpPSmljVVl4aUMiLCJ2ZXJzaW9uIjoiNC4wLjAiLCJoYW5kbGUiOiJhN2E5MDc3My1kYjExLTRhZTYtYmI0Yi1jY2EzNDYwOGIyMjMifQ==
+ ibmcloud sl call-api SoftLayer_Virtual_Guest_Block_Device_Template_Group createFromIcos --parameters '[{
      "bootMode":"HVM",
      "operatingSystemReferenceCode":"CENTOS_7_64",
      "cloudInit":true,
      "name": "pfq00-e2e-89599686-image-encrypted-0.vhd",
      "ibmApiKey": "mysecretAPIkey",
      "isEncrypted": true,
      "uri": "cos://us-south/pfq00-e2e-classic-images/pfq00-e2e-89599686-image-encrypted-0.vhd",
      "wrappedDek":"eyJjaXBoZXJ0ZXh0IjoiVTJ1NVRJVEoxY3lZWUZ1K1B4TnhNUHRNT3dkT2QzRzY0d1NTdG1ISC9FWUZSelJHWjNBZFc5RldLTXUrN2FrSnk0SUtKSTBEVHd2MHBOdDhhcjhpa2xQbFpOeUZYdUFXajZSTUJnPT0iLCJpdiI6InQrN2FmSGpPSmljVVl4aUMiLCJ2ZXJzaW9uIjoiNC4wLjAiLCJoYW5kbGUiOiJhN2E5MDc3My1kYjExLTRhZTYtYmI0Yi1jY2EzNDYwOGIyMjMifQ==",
      "crkCrn":"crn:v1:bluemix:public:kms:us-south:a/713c729476507a53135fe6793c37cc74:a9c41c7e-0053-419c-806b-78dc569c1c34:key:a7a90773-db11-4ae6-bb4b-cca34608b223"
  }]'
op22

Notice that the Image Type confirms it is encrypted:

080-create-test-vm.sh

Finally, a VSI can be started with the encrypted image. Nothing special here—just provision the instance:

op24

In the Storage panel, confirm it is encrypted:

op25

090-cleanup.sh deletes everything that was created—image templates, COS instance, Key Protect instance, VSI instances—using a mix of IBM Cloud CLI and Terraform.

Conclusion

The scripts listed here handle the most simple encryption use case but give the foundation of more complex setup

Keep in mind that only instances created with a cloud-init enabled image can be encrypted with this process. Also, only image templates with a single primary boot volume (or disk) can be encrypted. Secondary disks are not supported.

If you have feedback, suggestions, or questions about this post, please reach out to me on Twitter: @powellquiring.

Be the first to hear about news, product updates, and innovation from IBM Cloud