IBM Support

Cloud Pak for Integration - Refreshing Expired Certificates

How To


Summary

Self-signed CA certificate refresh does not automatically refresh leaf certificates resulting in unavailable services. This document describes how to refresh these certificates for each component of Cloud Pak for Integration. Section 1 covers the necessary steps to refresh the certificates, and the subsequent sections are optional dependent on which capabilities are deployed.

Objective

The self-signed CA certificate used by Common Service 3.6.3 and below, and created by the cert-manager service has a duration of 90 days. The CA certificate is refreshed by cert-manager but the leaf certificates that use the ca certificate must be manually refreshed. 
 

Steps

Necessary steps to refresh certificate: Optional steps to modify certificate duration: Subsequent steps to use the refreshed certificate:
1. To refresh the cs-ca-certificate used by Common Services
1.1. Check the expiration date:
oc describe certificate cs-ca-certificate -n ibm-common-services
1.2. If certificate refresh is required then delete the secret:
oc delete secret cs-ca-certificate-secret -n ibm-common-services
1.3. Refresh the leaf certificates based on cs-ca-certificate:
Include this step if you refreshed cs-ca-certificate in the previous step, or if it was already refreshed by cert-manager. This step forces the leaf certificates to be updated with the new ca.
mkdir secret_backup
cd secret_backup
oc get certs -o custom-columns=:spec.secretName,:spec.issuerRef.name --no-headers |egrep "cs-ca-clusterissuer|cs-ca-issuer" | while read secretName issuerName
do
oc get secret $secretName -o yaml -n ibm-common-services > secret.$secretName.yaml
oc delete secret $secretName -n ibm-common-services
done
After the certificates are refreshed, all IBM Cloud Platform Common Services pods that mount these certificates will be automatically refreshed by cert-manager.
1.4. Restart the auth-idp, auth-pap, auth-pdp pods
oc delete pod -l app=auth-idp -n ibm-common-services
oc delete pod -l app=auth-pap -n ibm-common-services
oc delete pod -l app=auth-pdp -n ibm-common-services
1.5. Refresh the ibmcloud-cluster-ca-cert
For Common Services versions before version 3.6.2, the ibmcloud-cluster-ca-cert secret must be refreshed to pick up the refreshed ca.crt. The ibm-management-ingress-operator will re-create the secret.
Run the following command to delete the secret.
oc delete secret ibmcloud-cluster-ca-cert -n ibm-common-services
1.6. Re-create the cp-console route
For IBM Cloud Platform Common Services versions before version 3.6.2, the cp-console route must be deleted and re-created to use the refreshed leaf certificate route-tls (secret route-tls-secret). The ibm-management-ingress-operator re-creates the route.
Run the following command to delete the route.
oc delete route cp-console -n ibm-common-services
1.7. Use the refreshed certificate for the components
The components of Cloud Pak for Integration will require steps to use the refreshed certificate as described by section 3 onwards.
2. To change the duration of the cs-ca-certificate used by Common Services
2.1. Update the cs-ca-certificate yaml file to add the duration and renewBefore parameters
duration: 17520h
renewBefore: 240h
In the following example, duration value is set to two years and the renewBefore value is set to 10 days.
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
 name: cs-ca-certificate
 namespace: ibm-common-services
 labels:
  app.kubernetes.io/instance: ibm-cert-manager-operator
  app.kubernetes.io/managed-by: ibm-cert-manager-operator
  app.kubernetes.io/name: cert-manager
  certmanager.k8s.io/issuer-kind: Issuer
  certmanager.k8s.io/issuer-name: cs-ss-issuer
 spec:
  commonName: cs-ca-certificate
  isCA: true
  issuerRef:
   kind: Issuer
   name: cs-ss-issuer
  secretName: cs-ca-certificate-secret
  duration: 17520h
  renewBefore: 240h
 status:
  conditions:
  - lastTransitionTime: '2020-12-08T04:20:27Z'
  message: Certificate is up to date and has not expired
  reason: Ready
  status: 'True'
  type: Ready
  notAfter: '2021-03-08T04:20:27Z'
2.2. Delete the secret to refresh the certificate
oc delete secret cs-ca-certificate-secret -n ibm-common-services
2.3. Refresh the leaf certificates based on the cs-ca-certificate

Include this step if you refreshed cs-ca-certificate in the previous step, or if it was already refreshed by cert-manager. This step forces the leaf certificates to be updated with the new ca.
mkdir secret_backup
cd secret_backup
oc get certs -o custom-columns=:spec.secretName,:spec.issuerRef.name --no-headers |egrep "cs-ca-clusterissuer|cs-ca-issuer" | while read secretName issuerName
do
oc get secret $secretName -o yaml -n ibm-common-services > secret.$secretName.yaml
oc delete secret $secretName -n ibm-common-services
done
After the certificates are refreshed, all IBM Cloud Platform Common Services pods that mount these certificates will be automatically refreshed by cert-manager.
2.4. Restart the auth-idp, auth-pap, auth-pdp pods
oc delete pod -l app=auth-idp -n ibm-common-services
oc delete pod -l app=auth-pap -n ibm-common-services
oc delete pod -l app=auth-pdp -n ibm-common-services
2.5. Refresh the ibmcloud-cluster-ca-cert
For Common Services versions before version 3.6.2, the ibmcloud-cluster-ca-cert secret must be refreshed to pick up the refreshed ca.crt. The ibm-management-ingress-operator will re-create the secret.
Run the following command to delete the secret.
oc -n ibm-common-services delete secret ibmcloud-cluster-ca-cert
2.6. Re-create the cp-console route
For IBM Cloud Platform Common Services versions before version 3.6.2, the cp-console route must be deleted and re-created to use the refreshed leaf certificate route-tls (secret route-tls-secret). The ibm-management-ingress-operator re-creates the route.
Run the following command to delete the route.
oc delete route cp-console -n ibm-common-services
2.7. Use the refreshed certificate for the components
The components of Cloud Pak for Integration will require steps to use the refreshed certificate as described by section 3 onwards.

Additional Information

3. Event Streams within Cloud Pak for Integration
The secret called `<cr_name>-ibm-es-ibmcloud-ca-cert` will need to be manually refreshed to pick up a new CA certificate.
Once deleted, the event streams operator will reconcile and recreate the secret after a minute or two. 
3.1. Switch to the namespace where Event Streams is installed

3.2. Make sure Common Services refreshed the certificate management-ingress-ibmcloud-cluster-ca-cert . If not delete it and wait for it to be recreated by Common Services again: 
oc delete secret management-ingress-ibmcloud-cluster-ca-cert
If the secret has not been created automatically, look for errors or delete operand-deployment-lifecycle-manager-* pod(s) in Common Services namespace (usually ibm-common-services).
 
3.3. Delete the secret <cr_name>-ibm-es-ibmcloud-ca-cert and wait for it to be recreated. If the secret does not appear check for errors or delete the <cr_name>-cluster-operator-* found either in the same namespace or in openshift-operators if installed cluster wide.  To verify that the secret has been updated:
kubectl describe secret <cr_name>-ibm-es-ibmcloud-ca-cert
3.4. Restart the admin pod. To restart the admin pod, delete the pod using the following command: 
oc get pod --selector app.kubernetes.io/name=admin-api -o name | xargs oc delete
4. Operations Dashboard within Cloud Pak for Integration

4.1. Switch to the names space where OD installed
4.2. Make sure Common Services refreshed the certificate management-ingress-ibmcloud-cluster-ca-cert . If not delete it and wait it will be created by Common Services again:
oc delete secret management-ingress-ibmcloud-cluster-ca-cert
If the secret has not been created automatically, look for errors or delete operand-deployment-lifecycle-manager-* pod(s) in Common Services namespace (usually ibm-common-services).
4.3. Delete the secret <cr name>-ibm-integration-operations-dashboard-com-clustrca
4.4. Delete the operator pod (it may be in a different namespace such as openshift-operators)
4.5. Wait for the secret <cr name>-ibm-integration-operations-dashboard-com-clustrca to be recreated
4.6. Delete all pods related to OD deployments :
<CR name>-ibm-integration-od-fe-*
<CR name>-ibm-integration-od-hkw-*
<CR name>-ibm-integration-od-job-*
<CR name>-ibm-integration-od-scd-*
5. Asset Repository within Cloud Pak for Integration
5.1. Switch to the namespace where the Asset Repository is installed.
5.2. Make sure Common Services refreshed the certificate management-ingress-ibmcloud-cluster-ca-cert . If not delete it and wait it will be created by Common Services again:
oc delete secret management-ingress-ibmcloud-cluster-ca-cert
5.3. Delete the API and UI pods for the Asset Repository deployment:
<CR name>-ibm-integration-asset-repository-ui-*
<CR name>-ibm-integration-asset-repository-api-*
6. API Connect and DataPower within Cloud Pak for Integration
Note that the renewal of certificates, particularly the gateway certificates below, will cause outage of Gateway API calls for a few minutes, typically up to 10 minutes.
6.1. Identify the API Connect ingress-ca certificate:
  • Identify the "API Connect" capability instance "Name" from your Platform Navigator. This is referred to as <Name> in steps below.
  • Identify the K8s namespace in which API Connect instance was deployed, also from the Platform Navigator. This is referred to as <namespace> in steps below.
  • The APIC ingress-ca certificate on the stack is called <Name>-ingress-ca and it exists in the <namespace> namespace.
6.2. Update the duration and renewBefore of the ingress-ca:
Edit certificate <Name>-ingress-ca  and add the duration and renewBefore fields under the spec field as below, then save and exit the editor:
  
$ oc -n <namespace> edit <Name>-ingress-ca
.
.
.
spec:
  duration: 87600h # 10 years
  renewBefore: 720h # 30 days	
.
.
.
6.2.1 Renew the ingress-ca:
$ oc delete secret <Name>-ingress-ca -n <namespace>
6.3. Renew the certificates issued by ingress-ca:
If the API Connect <Name>-ingress-ca certificate has been renewed (either due to expiry or manually), and the derived certificates have not been renewed automatically by the cert-manager, API Connect subsystems will not be able to communicate successfully with each-other. The following K8S secrets correspond to the certificates issued by the <Name>-ingress-issuer using <Name>-ingress-ca secret. They need to be deleted and the identified pods restarted in order to ensure API Connect subsystems are communicating correctly. Deleting the K8S secret corresponding to certificate managed by cert-manager will cause cert-manager to re-issue the certificate.
6.3.1. Analytics:
- Delete secrets
oc delete secret <Name>-a7s-ac-endpoint -n <namespace>
oc delete secret <Name>-a7s-ai-endpoint -n <namespace>
oc delete secret <Name>-a7s-cl-client -n <namespace>
oc delete secret <Name>-a7s-ing-client -n <namespace>
All required analytics subsystem pods will be restarted automatically.
6.3.2. Manager:
- Delete secrets
oc delete secret <Name>-mgmt-api-manager -n <namespace>
oc delete secret <Name>-mgmt-consumer-api -n <namespace>
oc delete secret <Name>-mgmt-platform-api -n <namespace>
oc delete secret <Name>-mgmt-admin -n <namespace>
All required management subsystem pods will be restarted automatically.
6.3.3. Portal:
- Delete secrets
oc delete secret <Name>-ptl-adm-client -n <namespace>
oc delete secret <Name>-ptl-portal-web -n <namespace>
oc delete secret <Name>-ptl-portal-director -n <namespace>
All required portal subsystem pods will be restarted automatically.
6.3.4. Gateway:
Please follow the steps to update the secret that protects gateway-peering communications within a Gateway cluster in your API Connect deployment at https://www.ibm.com/support/knowledgecenter/en/SSMNED_v10/com.ibm.apic.install.doc/replace_gwy_pwd_keys.html
7. App Connect Enterprise within Cloud Pak for Integration
IBM App Connect Enterprise does not use management-ingress-ibmcloud-cluster-ca-cert and so no action is necessary.
8. IBM MQ within Cloud Pak for Integration
8.1 Switch to the namespace where the IBM MQ queue managers are deployed.
8.2. Restart the queue manager pod to pick up the refreshed certificates.
There are different types of availability: "SingleInstance" and "MultiInstance".
8.2.1 For a single-instance queue manager, restart the queue manager pod.
For example, for a queue manager named quickstart-cp4i the pod name would look like quickstart-cp4i-ibm-mq-0. 
oc delete pod quickstart-cp4i-ibm-mq-0

8.2.2 For multi-instance queue manager, first restart the standby queue manager pod and then restart the active queue manager pod.
For example, for a queue manager named quickstart-cp4i the pod names may look like quickstart-cp4i-ibm-mq-0 and quickstart-cp4i-ibm-mq-1. You can run dspmq command to identify which pod is running as a standby queuemanager and active queue manager.

oc rsh <queuemanager-pod-name> dspmq
For example, oc rsh quickstart-cp4i-ibm-mq-0 dspmq
 QMNAME(QUICKSTART)                                         STATUS(Running as standby)

In our case the standby queue manager pod is quickstart-cp4i-ibm-mq-0 therefore we will first delete the standby pod.

oc delete pod quickstart-cp4i-ibm-mq-0
Verify if the standby queue manager pod is back and is up & running:
oc get pod --selector app.kubernetes.io/instance=quickstart-cp4i
Verify if your standby queue manager is running:
oc rsh quickstart-cp4i-ibm-mq-0 dspmq
then you can similarly restart your active queue manager pod quickstart-cp4i-ibm-mq-1
oc delete pod quickstart-cp4i-ibm-mq-1
9.1. Switch to the namespace where the Platform Navigator is installed.
9.2. Make sure Common Services refreshed the certificate management-ingress-ibmcloud-cluster-ca-cert . If not then delete it and ensure it is re-created by Common Services again:
oc delete secret management-ingress-ibmcloud-cluster-ca-cert
oc delete secret ibmcloud-cluster-ca-cert -n ibm-common-services
oc -n ibm-common-services delete pod -l app=auth-idp
Note that this is not required on Common Services 3.7 and later versions.

9.3. Get the name of the Platform Navigator pod:
oc get pod | grep ibm-integration-platform-navigator-deployment
This will return information about the Platform Navigator pod e.g.:
pn-1-ibm-integration-platform-navigator-deployment-668795fkkxbx   2/2     Running     0          5d2h
The pod name will always have the form:
<CR name>-ibm-integration-platform-navigator-deployment-<some ID>

9.4. Delete the Platform Navigator pod returned by the previous step. This will force a new Platform Navigator pod to be created:
oc delete pod <pod name>

Document Location

Worldwide

[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SS8QTD","label":"IBM Cloud Pak for Integration"},"ARM Category":[{"code":"a8m0z0000001hogAAA","label":"Common Services"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)"}]

Document Information

Modified date:
08 June 2021

UID

ibm16381380

Page Feedback