A look at the process for rotating API keys and passwords for a containerized solution on IBM Cloud Kubernetes Service.
In my recent posts, I touched on updating credentials for solutions deployed on IBM Cloud Functions or using Cloud Foundry on IBM Cloud. Today, I am going to show you how to rotate API keys and passwords for a containerized solution on IBM Cloud Kubernetes Service that makes use of a delivery pipeline (DevOps).
I am going to use the app discussed in the tutorial on how to apply end-to-end security to a cloud application.
The app discussed in the tutorial provides a solution for securely sharing files. It uses Cloud Object Storage and a NoSQL database to manage files and related metadata. Access is controlled via IBM Cloud App ID, which provides authentication and identity services.
Security works best when it is easy to integrate. Therefore, I looked into adding stages to the delivery pipeline for rotating credentials. A quick assessment showed that credentials are used in three different ways:
- The app itself accesses Cloud Object Storage and a Cloudant database. Both credentials are made available in a single Kubernetes secret. A requirement is to keep the app online while changing the secret, (i.e., a zero-downtime update).
- App ID is deployed as cluster integration—the cluster ingress (network access) requires a valid authentication. The access protection needs to be kept active all the time. A zero-downtime update is required, as well.
- During the container deployment, the deploy process uses an image pull secret to access the IBM Cloud container registry.
For all three scenarios, the steps to rotate the access data involves renaming the existing access key, creating new credentials with the previous name, applying the new credentials to the Kubernetes secret (updating or recreating it), and eventually deactivating the old credentials by deleting the old service key.
To handle each of the three scenarios independently, I created three scripts (look for pipeline-ROTATE_XXX_CREDENTIALS.sh).
There are different ways that secrets are composed and how they are updated. You can read the source for details, but I want to point out few interesting things I learned. The first is that you can only create or delete secrets, but there is a still a neat way of updating them. It involves creating a new secret in dry run mode, then piping it to apply to replace the existing content:
kubectl create secret secret-type --from-source-type= --dry-run=true -o yaml | kubectl apply -f -
kubectl rollout restart deployment-name
It is used by the script to rotate the storage credentials. The command first brings up new pods with the app that has the new credentials, then it shuts down the old (existing) pods and the older credentials are no longer used:
The three scripts to rotate the credentials can be easily integrated into a delivery pipeline as manual stages. That way, they can access the cluster and account information that was used to build and deploy the app.
With a single click and no further configuration I am able to update passwords and API keys when it is required, either periodically or when a team member leaves or if there is a (suspected) incident: