Refreshing foundational services internal certificates (IBM Cloud Pak foundational services Version 3.6.3 and earlier)

A common CA issuer is created from a self-signed CA certificate (certificate authority) and leaf certificates are created by individual foundational services from the common CA issuer. The CA certificate default duration is 90 days. You can customize this duration based on the certificate rotation schedule. Cert-manager will automatically refresh the CA certificate but the leaf certificates that are created from this CA certificate must be manually refreshed when the CA certificate is refreshed.

Complete the following steps to refresh the foundational services internal certificates.

1. Delete the secret to refresh the CA certificate.

    oc -n ibm-common-services delete secret cs-ca-certificate-secret
   

2. Refresh the leaf certificates based on the cs-ca-certificate.

   This step forces the leaf certificates to be updated with the new ca.

    mkdir secret_backup
   

    cd secret_backup
   

    oc get certs -o custom-columns=:spec.secretName,:spec.issuerRef.name --no-headers |egrep "cs-ca-clusterissuer|cs-ca-issuer" | while read secretName issuerName
    do
    oc get secret $secretName -o yaml -n ibm-common-services > secret.$secretName.yaml
    oc delete secret $secretName -n ibm-common-services
    done
   

   After the certificates are refreshed, all IBM Cloud Pak foundational services pods that mount these certificates will be automatically refreshed by cert-manager.

3. Restart the auth-idp, auth-pap, auth-pdp pods.

    oc delete pod -l app=auth-idp -n ibm-common-services
   

    oc delete pod -l app=auth-pap -n ibm-common-services
   

    oc delete pod -l app=auth-pdp -n ibm-common-services
   

4. Check and run the security-onboarding-job as-needed.

   1. Check the status of the security onboarding job.

        oc get jobs |grep security-onboarding
      

      If the job ran properly, it shows completion as 1/1. If a failure occurred, it shows 0/1.

   2. If a failure occurs, take a backup of the job and then delete the job to rerun it.

        oc get job <security-onboarding-job> -o yaml > security-onboarding-backup.yaml
      

        oc delete job security-onboarding
      

   3. Wait for the job to complete.

        oc get jobs
      

        oc get jobs
        NAME                       COMPLETIONS   DURATION   AGE
        iam-onboarding             1/1           4m46s      157m
        oidc-client-registration   1/1           6m45s      3h11m
        security-onboarding        0/1           14s        14s
      
        oc get jobs -w
        NAME                       COMPLETIONS   DURATION   AGE
        iam-onboarding             1/1           4m46s      158m
        oidc-client-registration   1/1           6m45s      3h11m
        security-onboarding        0/1           30s        30s
        security-onboarding        1/1           88s        88s
      

5. Refresh the ibmcloud-cluster-ca-cert.

   For foundational services versions before 3.6.2, the ibmcloud-cluster-ca-cert secret must be refreshed to pick up the refreshed ca.crt. The ibm-management-ingress-operator re-creates the secret.

   If you replaced the foundational services endpoint certificate by using procedure, Replacing the foundational services endpoint certificate, then do not perform this step.

   Run the following command to delete the secret.

    oc -n ibm-common-services delete secret ibmcloud-cluster-ca-cert
   

6. Re-create the cp-console route.

   For foundational services versions before 3.6.2, the cp-console route must be deleted and re-created to use the refreshed leaf certificate route-tls (secret route-tls-secret). The ibm-management-ingress-operator re-creates the route.

   Run the following command to delete the route.

    oc delete route cp-console -n ibm-common-services
   

7. (Optional) If you are using your own certificate and replaced the management-ingress certificate by using procedure, Replacing the foundational services endpoint certificate, then you must complete the following steps to update the cp-console route. The destination ca certificate is updated based on the refresh of the foundational services internal CA certificate.

   1. Obtain the updated destination CA certificate.

        oc get -o jsonpath={.data."ca.crt"} secret icp-management-ingress-tls-secret | base64 -d > dest-ca.crt
      

   2. Save the generated cert, key, and ca-cert of your certificate in the same directory.

   3. Regenerate the route spec.

         oc -n ibm-common-services create route reencrypt cp-console --service=icp-management-ingress     --cert=./tls.crt     --key=./tls.key     --ca-cert=./ca.crt     --dest-ca-cert=./dest-ca.crt     --hostname=cp-console.apps.demo.cp.fyre.ibm.com     --insecure-policy='Redirect'     --dry-run='client' -o yaml  > cp-console.yaml
      

   4. Apply the change.

        oc -n ibm-common-services  apply -f cp-console.yaml
      

8. After refresh, if the monitoring console is not accessible then restart the alertmanager-ibm-monitoring-alertmanager-0, prometheus-ibm-monitoring-prometheus-0, and grafana pods.

    oc delete pod -l app=alertmanager -n ibm-common-services
   

    oc delete pod -l app=grafana -n ibm-common-services
   

    oc delete pod -l app=prometheus -n ibm-common-services
   

Changing the duration of the CA certificate and refreshing foundational services internal certificates

Complete the following steps to change the duration of the CA certificate and then refresh the foundational services internal certificates.

1. Back up the certificate by running the following command:

    oc get certificate cs-ca-certificate -n ibm-common-services > cs-ca-certificate.yaml.bakup
   

2. Update the cs-ca-certificate yaml file and add the duration and renewBefore parameters.

   1. To update the cs-ca-certificate yaml file run the following command:

        oc edit certificate cs-ca-certificate -n ibm-common-services
      

   2. Add the duration and renewBefore parameters with the following values under the spec section in the cs-ca-certificate yaml file.

        duration: 17520h
        renewBefore: 240h
      

      In the following example, duration value is set to two years and the renewBefore value is set to 10 days.

       apiVersion: cert-manager.io/v1
       kind: Certificate
       metadata:
         name: cs-ca-certificate
         namespace: ibm-common-services
       labels:
         app.kubernetes.io/instance: ibm-cert-manager-operator
         app.kubernetes.io/managed-by: ibm-cert-manager-operator
         app.kubernetes.io/name: cert-manager
         certmanager.k8s.io/issuer-kind: Issuer
         certmanager.k8s.io/issuer-name: cs-ss-issuer
       spec:
         commonName: cs-ca-certificate
         isCA: true
         issuerRef:
            kind: Issuer
            name: cs-ss-issuer
         secretName: cs-ca-certificate-secret
           duration: 17520h
           renewBefore: 240h
       status:
         conditions:
         - lastTransitionTime: '2020-12-08T04:20:27Z'
         message: Certificate is up to date and has not expired
         reason: Ready
         status: 'True'
         type: Ready
         notAfter: '2021-03-08T04:20:27Z'
      

3. Save the cs-ca-certificate yaml file.

4. Delete the secret to refresh the CA certificate.

    oc -n ibm-common-services delete secret cs-ca-certificate-secret
   

5. Refresh the leaf certificates based on the cs-ca-certificate.

   Include this step if you refreshed cs-ca-certificate in the previous step, or if it was already refreshed by cert-manager. This step forces the leaf certificates to be updated with the new ca.

    mkdir secret_backup
   

    cd secret_backup
   

    oc get certs -o custom-columns=:spec.secretName,:spec.issuerRef.name --no-headers |egrep "cs-ca-clusterissuer|cs-ca-issuer" | while read secretName issuerName
    do
    oc get secret $secretName -o yaml -n ibm-common-services > secret.$secretName.yaml
    oc delete secret $secretName -n ibm-common-services
    done
   

   After the certificates are refreshed, all IBM Cloud Pak foundational services pods that mount these certificates will be automatically refreshed by cert-manager.

6. Restart the auth-idp, auth-pap, auth-pdp pods.

    oc delete pod -l app=auth-idp -n ibm-common-services
   

    oc delete pod -l app=auth-pap -n ibm-common-services
   

    oc delete pod -l app=auth-pdp -n ibm-common-services
   

7. Check and run the security-onboarding-job as-needed.

   1. Check the status of the security onboarding job.

        oc get jobs |grep security-onboarding
      

      If the job ran properly, it shows completion as 1/1. If a failure occurred, it shows 0/1.

   2. If a failure occurs, take a backup of the job and then delete the job to rerun it.

        oc get job <security-onboarding-job> -o yaml > security-onboarding-backup.yaml
      

        oc delete job security-onboarding
      

   3. Wait for the job to complete.

        oc get jobs
      

        oc get jobs
        NAME                       COMPLETIONS   DURATION   AGE
        iam-onboarding             1/1           4m46s      157m
        oidc-client-registration   1/1           6m45s      3h11m
        security-onboarding        0/1           14s        14s
      
        oc get jobs -w
        NAME                       COMPLETIONS   DURATION   AGE
        iam-onboarding             1/1           4m46s      158m
        oidc-client-registration   1/1           6m45s      3h11m
        security-onboarding        0/1           30s        30s
        security-onboarding        1/1           88s        88s
      

8. Refresh the ibmcloud-cluster-ca-cert.

   For foundational services versions before 3.6.2, the ibmcloud-cluster-ca-cert secret must be refreshed to pick up the refreshed ca.crt. The ibm-management-ingress-operator re-creates the secret.

   If you replaced the foundational services services endpoint certificate by using procedure, Replacing the foundational services endpoint certificate, then do not perform this step.

   Run the following command to delete the secret.

    oc -n ibm-common-services delete secret ibmcloud-cluster-ca-cert
   

9. Re-create the cp-console route.

   For foundational services versions before 3.6.2, the cp-console route must be deleted and re-created to use the refreshed leaf certificate route-tls (secret route-tls-secret). The ibm-management-ingress-operator re-creates the route.

   Run the following command to delete the route.

   oc delete route cp-console -n ibm-common-services
   

10. (Optional) If you are using your own certificate and replaced the management-ingress certificate by using procedure, Replacing the foundational services endpoint certificate, then you must complete the following steps to update the cp-console route. The destination ca certificate is updated based on the refresh of the foundational services internal CA certificate.

    1. Obtain the updated destination CA certificate.

        oc get -o jsonpath={.data."ca.crt"} secret icp-management-ingress-tls-secret | base64 -d > dest-ca.crt
       

    2. Save the cert, key, and ca-cert of your certificate in the same directory. For example,

        ls -l
        total 68
        -rw-r--r-- 1 root root  2021 Oct 19 18:17 ca.crt
        -rw-r--r-- 1 root root  1168 Oct 19 18:28 dest-ca.crt
        -rw-r--r-- 1 root root  1777 Oct 19 18:18 tls.crt
        -rw-r--r-- 1 root root  1675 Oct 19 18:17 tls.key
       

    3. Regenerate the route spec.

         oc -n ibm-common-services create route reencrypt cp-console --service=icp-management-ingress     --cert=./tls.crt     --key=./tls.key     --ca-cert=./ca.crt     --dest-ca-cert=./dest-ca.crt     --hostname=cp-console.apps.demo.cp.fyre.ibm.com     --insecure-policy='Redirect'     --dry-run='client' -o yaml  > cp-console.yaml
       

    4. Apply the change.

        oc -n ibm-common-services  apply -f cp-console.yaml
       

11. After refresh, if the monitoring console is not accessible then restart the alertmanager-ibm-monitoring-alertmanager-0, prometheus-ibm-monitoring-prometheus-0, and grafana pods.

    oc delete pod -l app=alertmanager -n ibm-common-services
    

    oc delete pod -l app=grafana -n ibm-common-services
    

    oc delete pod -l app=prometheus -n ibm-common-services
    

    For more information, see Customizing cert-manager certificates and Refreshing cert-manager certificates.