How to Use a Delivery Pipeline to Rotate App Credentials

3 min read

By: Henrik Loeser

A look at the process for rotating API keys and passwords for a containerized solution on IBM Cloud Kubernetes Service.

In my recent posts, I touched on updating credentials for solutions deployed on IBM Cloud Functions or using Cloud Foundry on IBM Cloud. Today, I am going to show you how to rotate API keys and passwords for a containerized solution on IBM Cloud Kubernetes Service that makes use of a delivery pipeline (DevOps). 

I am going to use the app discussed in the tutorial on how to apply end-to-end security to a cloud application

Tutorial overview 

The app discussed in the tutorial provides a solution for securely sharing files. It uses Cloud Object Storage and a NoSQL database to manage files and related metadata. Access is controlled via IBM Cloud App ID, which provides authentication and identity services. 

The code for the app and a toolchain to automatically create required services and keys as well as to deploy the app to Kubernetes are provided in the GitHub repository secure-file-storage.

ht1

Rotating credentials 

Security works best when it is easy to integrate. Therefore, I looked into adding stages to the delivery pipeline for rotating credentials. A quick assessment showed that credentials are used in three different ways:

  1. The app itself accesses Cloud Object Storage and a Cloudant database. Both credentials are made available in a single Kubernetes secret. A requirement is to keep the app online while changing the secret, (i.e., a zero-downtime update).
  2. App ID is deployed as cluster integration—the cluster ingress (network access) requires a valid authentication. The access protection needs to be kept active all the time. A zero-downtime update is required, as well.
  3. During the container deployment, the deploy process uses an image pull secret to access the IBM Cloud container registry.

For all three scenarios, the steps to rotate the access data involves renaming the existing access key, creating new credentials with the previous name, applying the new credentials to the Kubernetes secret (updating or recreating it), and eventually deactivating the old credentials by deleting the old service key. 

To handle each of the three scenarios independently, I created three scripts (look for pipeline-ROTATE_XXX_CREDENTIALS.sh).

Updating secrets

There are different ways that secrets are composed and how they are updated. You can read the source for details, but I want to point out few interesting things I learned. The first is that you can only create or delete secrets, but there is a still a neat way of updating them. It involves creating a new secret in dry run mode, then piping it to apply to replace the existing content:

kubectl create secret secret-type --from-source-type= --dry-run=true -o yaml | kubectl apply -f -

The second detail is a new kubectl rollout subcommand to restart apps with zero downtime:

kubectl rollout restart deployment-name

It is used by the script to rotate the storage credentials. The command first brings up new pods with the app that has the new credentials, then it shuts down the old (existing) pods and the older credentials are no longer used:

ht5

Pipeline integration 

The three scripts to rotate the credentials can be easily integrated into a delivery pipeline as manual stages. That way, they can access the cluster and account information that was used to build and deploy the app.

With a single click and no further configuration I am able to update passwords and API keys when it is required, either periodically or when a team member leaves or if there is a (suspected) incident:

ht6

Easy, right? Check out the updated tutorial on how to apply end to end security to a cloud application and the related GitHub repository.

Feedback

If you have feedback, suggestions, or questions about this post, please reach out to me on Twitter (@data_henrik) or LinkedIn

Be the first to hear about news, product updates, and innovation from IBM Cloud