How to Migrate Certificates from IBM Certificate Manager to IBM Cloud Secrets Manager

6 May 2022 2 min read

By:

Takeyuki Nakajo, Advanced Customer Support Engineer
Theodora Cheng, Software Developer - Armada Ingress

In this step-by-step guide, we'll show you how to migrate TLS certificates from your current IBM Certificate Manager instance to an IBM Cloud Secrets Manager instance.

An IBM Certificate Manager instance is automatically created for each cluster that you can use to manage the cluster's ingress TLS certificates.

As of 10 February 2022, the ability to manage the lifecycle of your certificates by using the Certificate Manager service is deprecated. Automatic provisioning of Certificate Manager instances in new clusters ends on 23 September 2022. Support for Certificate Manager ends on 1 December 2022, and any remaining Certificate Manager instances are set to be deleted on 31 Dec 2022. After this date, certificates are set to be written only to the cluster unless you set up an IBM Cloud Secrets Manager instance.

IBM Cloud Secrets Manager

The strategic alternative for managing certificates is IBM Cloud Secrets Manager. With Secrets Manager, you can centralize your application secrets in a single service, including the TLS certificates that you might already store and manage in Certificate Manager. Both services provide a secure repository, but with Secrets Manager, you can programmatically call the service through SDKs, manage multiple secret types and take advantage of data isolation through single-tenancy.

You need to migrate TLS certificates to Secrets Manager if your clusters were created before 23 September 2022.

Step-by-step instructions

In this step-by-step guide, I will show you how to migrate TLS certificates from your current IBM Certificate Manager instance to an IBM Cloud Secrets Manager instance.

1. Review the existing certificates/secrets in your cluster

In the IBM Cloud console, navigate to your IBM Cloud resource list.
Expand the Services row.
Look for a Certificate Manager instance that is named in the format kube-crtmgr-<cluster_ID> or kube-<cluster_ID>.
Click the instance's name. The Your certificates details page opens:

You can also verify the current default Certificate Manager instance by using the ibmcloud ks ingress instance ls command:

$ ibmcloud ks ingress instance ls --cluster c9l270id07a2ajbtnu0g
OK
Name                                Type         Is Default   Status    CRN   
kube-certmgr-c9l270id07a2ajbtnu0g   cloudcerts   true         created   crn:v1:bluemix:public:cloudcerts:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:8356fa57-b971-46c9-ac13-ae04c6d4a260::

The IBM-generated certificate for the default ingress subdomain exists in your cluster's Certificate Manager instance, which you have full control over. You can check if you have your own TLS certificates by uploading your own TLS certificates or ordering TLS certificates for your custom domains in the Certificate Manager instance.

For example, I have a cluster and an application running in the namespace project1. The ingress of the application uses the secrets (TLS certificates).

The secrets are listed below. The first three secrets in the namespace default, ibm-cert-store, kube-system are generated by IBM Cloud for the default ingress subdomain mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000.us-south.containers.appdomain.cloud. The fourth secret was copied manually into the namespace project1. The fifth secret was imported manually for the custom domain tnexample.com.

As you can see from the Certificate CRN value crn:v1:bluemix:public:cloudcerts:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:8356fa57-b971-46c9-ac13-ae04c6d4a260:..., it indicates these secrets were created from the Certificate Manager instance:

$ ibmcloud ks ingress secret ls --cluster c9l270id07a2ajbtnu0g
Name                                                 Namespace        CRN                                                                                                                                                              Expires On                 Domain                                                                                     Status    Type   


mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000   default          crn:v1:bluemix:public:cloudcerts:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:8356fa57-b971-46c9-ac13-ae04c6d4a260:certificate:314d95628f88c0e3c650521e2a0e7209   2022-07-27T04:23:39+0000   mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000.us-south.containers.appdomain.cloud     created   TLS   
mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000   ibm-cert-store   crn:v1:bluemix:public:cloudcerts:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:8356fa57-b971-46c9-ac13-ae04c6d4a260:certificate:314d95628f88c0e3c650521e2a0e7209   2022-07-27T04:23:39+0000   mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000.us-south.containers.appdomain.cloud     created   TLS   
mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000   kube-system      crn:v1:bluemix:public:cloudcerts:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:8356fa57-b971-46c9-ac13-ae04c6d4a260:certificate:314d95628f88c0e3c650521e2a0e7209   2022-07-27T04:23:39+0000   mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000.us-south.containers.appdomain.cloud     created   TLS   
mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000   project1         crn:v1:bluemix:public:cloudcerts:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:8356fa57-b971-46c9-ac13-ae04c6d4a260:certificate:314d95628f88c0e3c650521e2a0e7209   2022-07-27T04:23:39+0000   *.mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000.us-south.containers.appdomain.cloud   created   TLS   
customcertsecret                                     project1         crn:v1:bluemix:public:cloudcerts:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:8356fa57-b971-46c9-ac13-ae04c6d4a260:certificate:314d95628f88c0e3c650521e2a24d1c4   2022-07-20T01:27:23+0000   *.tnexample.com                                                                            created   TLS

You can double-check whether or not a secret is managed by IBM Cloud by running the ibmcloud ks ingress secret get command to view the details. In the output, if User Managed is marked false, the secret is managed by IBM Cloud. If it is marked true, the secret is not managed by IBM Cloud:

$ ibmcloud ks ingress secret get --cluster c9l270id07a2ajbtnu0g --name mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000 --namespace default
Name:           mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000   
Namespace:      default   
...
User Managed:   false   
...
$ ibmcloud ks ingress secret get --cluster c9l270id07a2ajbtnu0g --name mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000 --namespace project1
Name:           mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000   
Namespace:      project1   
...
User Managed:   true   
...
$ ibmcloud ks ingress secret get --cluster c9l270id07a2ajbtnu0g --name customcertsecret --namespace project1
Name:           customcertsecret   
Namespace:      project1   
...
User Managed:   true   
...

2. Create an IBM Cloud Secrets Manager instance

In order to migrate the certificates to a new Secrets Manager instance, you must have an instance or create a Secrets Manager instance and then register your instance to your cluster. You can use Secrets Manager instances across multiple clusters, and a single cluster can have more than one instance. For each cluster, you set one instance as a default where all ingress subdomain certificates are uploaded.

Create a Secrets Manager instance.
Copy the Service Instance CRN value for the Secrets Manager service instance from the Settings page:
To enable service-to-service communication, you must set up IAM credentials for Secrets Manager and create a service-to-service authorization.

Set up the IAM credentials engine

Add an IAM credentials engine configuration to configure your instance to start creating IAM credentials:

On the Secrets engines page in the Secrets Manager instance, click the IAM credentials tab.
Click Configure.
Enter an API key that has access to create and manage other API keys in your account. You will need to create a service ID that is associated with your API key that has Editor platform access on the IAM Access Groups Service and Operator platform access on the IAM Identity Service.
Click Configure:

Create an authorization between Secrets Manager and Kubernetes Service

Be sure that you have the SecretsReader service role or higher on your Secrets Manager instance:

In the console, click Manage > Access (IAM) and select Authorizations.
Click Create.
From the Source service list, select Kubernetes Service.
From the Target service list, select Secrets Manager.
Select the Manager role.
Click Authorize:

3. Register a Secrets Manager instance as the default instance to a cluster

Next, you must register the instance as the default instance to a cluster: ibmcloud ks ingress instance register --cluster <cluster_name_or_id> --crn <the new secret manager service instance_crn> --is-default

You see the new Secrets Manager instance as the default instance to the cluster. The Certificate Manager instance is also removed as the default:

192-168-1-14:~ tnakajo$ ibmcloud ks ingress instance register --cluster c9l270id07a2ajbtnu0g --crn crn:v1:bluemix:public:secrets-manager:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:64353325-db92-4433-86dd-a2855bb634fb:: --is-default
OK
192-168-1-14:~ tnakajo$ ibmcloud ks ingress instance ls --cluster c9l270id07a2ajbtnu0g
OK
Name                                Type              Is Default   Status    CRN   
kube-certmgr-c9l270id07a2ajbtnu0g   cloudcerts        false        created   crn:v1:bluemix:public:cloudcerts:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:8356fa57-b971-46c9-ac13-ae04c6d4a260::   
secretsmanager-tn                   secrets-manager   true         created   crn:v1:bluemix:public:secrets-manager:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:64353325-db92-4433-86dd-a2855bb634fb::

4. Update the IBM-managed secrets to upload them to the new default instance

When you set a new default Secrets Manager instance, any existing secrets that are managed by IBM Cloud can be updated by the ibmcloud ks nlb-dns secret regenerate command: ibmcloud ks nlb-dns secret regenerate --cluster <cluster_name_or_id> --nlb-subdomain <nlb_subdomain>

$ ibmcloud ks nlb-dns secret regenerate --cluster c9l270id07a2ajbtnu0g --nlb-subdomain mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000.us-south.containers.appdomain.cloud
Regenerating NLB subdomain certificate and secret...
Note: It might take a few minutes for your changes to be applied.
OK

After several minutes, you can see the certificate CRN is starting with crn:v1:bluemix:public:secrets-manager.... It indicates the first three IBM-managed secrets are updated with the certificates from the Secrets Manager instance:

$ ibmcloud ks ingress secret ls --cluster c9l270id07a2ajbtnu0g
OK
Name                                                 Namespace        CRN                                                                                                                                                                  Expires On                 Domain                                                                                     Status    Type   


mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000   default          crn:v1:bluemix:public:secrets-manager:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:64353325-db92-4433-86dd-a2855bb634fb:secret:2d97c1de-7b12-a4d5-201f-bd9fe226d883   2022-07-28T05:23:50+0000   mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000.us-south.containers.appdomain.cloud     created   TLS   
mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000   ibm-cert-store   crn:v1:bluemix:public:secrets-manager:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:64353325-db92-4433-86dd-a2855bb634fb:secret:2d97c1de-7b12-a4d5-201f-bd9fe226d883   2022-07-28T05:18:16+0000   mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000.us-south.containers.appdomain.cloud     created   TLS   
mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000   kube-system      crn:v1:bluemix:public:secrets-manager:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:64353325-db92-4433-86dd-a2855bb634fb:secret:2d97c1de-7b12-a4d5-201f-bd9fe226d883   2022-07-28T05:18:16+0000   mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000.us-south.containers.appdomain.cloud     created   TLS   
mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000   project1         crn:v1:bluemix:public:cloudcerts:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:8356fa57-b971-46c9-ac13-ae04c6d4a260:certificate:314d95628f88c0e3c650521e2a0e7209       2022-07-27T04:23:39+0000   *.mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000.us-south.containers.appdomain.cloud   created   TLS   
customcertsecret                                     project1         crn:v1:bluemix:public:cloudcerts:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:8356fa57-b971-46c9-ac13-ae04c6d4a260:certificate:314d95628f88c0e3c650521e2a24d1c4       2022-07-20T01:27:23+0000   *.tnexample.com                                                                           created   TLS

5. Update the user-managed secrets to upload them to the new default instance

For secrets that are not managed by IBM Cloud (e.g., the fourth secret above where User Managed was marked false), you'll need to update the secret with the certificate CRN value in the Secrets Manager instance crn:v1:bluemix:public:secrets-manager:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:64353325-db92-4433-86dd-a2855bb634fb:secret:2d97c1de-7b12-a4d5-201f-bd9fe226d883 via the ibmcloud ks ingress secret update command: ibmcloud ks ingress secret update --cluster <cluster_name_or_id> --name <secret_name> --namespace <namespace> --cert-crn <cert crn of secret manager instance>

$ ibmcloud ks ingress secret update --cluster c9l270id07a2ajbtnu0g --name mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000 --namespace project1 --cert-crn crn:v1:bluemix:public:secrets-manager:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:64353325-db92-4433-86dd-a2855bb634fb:secret:2d97c1de-7b12-a4d5-201f-bd9fe226d883
OK
192-168-1-14:~ tnakajo$ ibmcloud ks ingress secret ls --cluster c9l270id07a2ajbtnu0g
OK
Name                                                 Namespace        CRN                                                                                                                                                                  Expires On                 Domain                                                                                     Status    Type   
mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000   project1         crn:v1:bluemix:public:secrets-manager:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:64353325-db92-4433-86dd-a2855bb634fb:secret:2d97c1de-7b12-a4d5-201f-bd9fe226d883   2022-07-28T05:23:50+0000   *.mycluster-tn-5be51ad3139a99d89cdf8f97c78ef71c-0000.us-south.containers.appdomain.cloud   created   TLS

6. Migrate certificates stored with custom domains

If you have a certificate that is stored with a custom domain and is not managed by IBM Cloud, you must manually download the certificate and upload it to the new Secrets Manager instance or reorder the certificate in the new Secrets Manager instance.

To download the certificate and upload it to the new Secrets Manager instance use the following steps:

In the Certificate Manager instance dashboard, select the checkbox for the certificate that you want to download.
Click Download Certificate. Depending on what you imported into the service, you'll receive a compressed file that contains PEM files for the certificate, an associated private key and an associated intermediate certificate.
In the Secrets Manger instance dashboard, click the TLS certificates tile in the Secrets table. Then click Add.
Add a name and description to easily identify your certificate.
Select a certificate file or enter its value.
Optional: Select a private key file or enter its value. If you choose to store a private key, ensure that it matches to your certificate. The private key must be unencrypted before you can import it to the service.
Optional: Select an intermediate certificate file or enter its value.
Optional: Add labels to help you to search for similar secrets in your instance.
Click Import.

To reorder the certificate, you can follow the steps in Ordering public certificates:

Optionally, you can turn the Automatic certificate rotation on at the bottom of the certificate detail pane. It will enable you to rotate the certificate automatically 31 days before it expires.

For the fifth secret customcertsecret, for the custom domain's TLS certificate, update the secret with the new certificate_CRN in the Secrets Manager: ibmcloud ks ingress secret update --cluster <cluster_name_or_id> --name <secret_name> --namespace <namespace> --cert-crn <certificate_crn of new secret manager instance>

$ ibmcloud ks ingress secret update --cluster c9l270id07a2ajbtnu0g --name customcertsecret --namespace project1 --cert-crn crn:v1:bluemix:public:secrets-manager:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:64353325-db92-4433-86dd-a2855bb634fb:secret:570fea00-7380-2e52-cf73-bb8411f9c690
OK
192-168-1-14:~ tnakajo$ ibmcloud ks ingress secret ls --cluster c9l270id07a2ajbtnu0g
OK
Name                                                 Namespace        CRN                                                                                                                                                                  Expires On                 Domain                                                                                     Status    Type   
customcertsecret                                     project1         crn:v1:bluemix:public:secrets-manager:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:64353325-db92-4433-86dd-a2855bb634fb:secret:570fea00-7380-2e52-cf73-bb8411f9c690   2022-07-20T01:27:23+0000   *.tnexample.com                                                                            created   TLS

7. Remove the Certificate Manager instance from the cluster

Once you migrate all secrets to the Secrets Manager instance, you can unregister the Certificate Manager instance from the cluster and remove the Certificate Manager instance: ibmcloud ks ingress instance unregister --cluster <cluster_name_or_id> --name <certiticate_manager_instance_name>

$ ibmcloud ks ingress instance ls --cluster c9l270id07a2ajbtnu0g
Name                                Type              Is Default   Status    CRN   
kube-certmgr-c9l270id07a2ajbtnu0g   cloudcerts        false        created   crn:v1:bluemix:public:cloudcerts:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:8356fa57-b971-46c9-ac13-ae04c6d4a260::   
secretsmanager-tn                   secrets-manager   true         created   crn:v1:bluemix:public:secrets-manager:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:64353325-db92-4433-86dd-a2855bb634fb::   
192-168-1-14:~ tnakajo$ ibmcloud ks ingress instance unregister --cluster c9l270id07a2ajbtnu0g --name kube-certmgr-c9l270id07a2ajbtnu0g
OK
$ ibmcloud ks ingress instance ls --cluster c9l270id07a2ajbtnu0g
Name                Type              Is Default   Status    CRN   
secretsmanager-tn   secrets-manager   true         created   crn:v1:bluemix:public:secrets-manager:us-south:a/bde92b81610817c89a6b7b3f8e8813f8:64353325-db92-4433-86dd-a2855bb634fb::

Optionally, you can delete the Certificate Manager instance in the IBM Cloud Dashboard. If you leave it, the Certificate Manager instance will delete it automatically on 31 December 2022.

Learn more

To learn more about IBM Cloud Secrets Manager with IBM Cloud Kubernetes and IBM Cloud Openshift, check out the links below:

Takeyuki Nakajo

Advanced Customer Support Engineer

Theodora Cheng

Software Developer - Armada Ingress

Follow IBM Cloud

Be the first to hear about news, product updates, and innovation from IBM Cloud.

Email subscribe RSS