Credentials rotation

The IBM API Catalog service uses user passwords, encryption keys, certificates, and client ID and secret for securing access to the APIs. You can manually refresh these credentials when needed.

Password rotation

Passwords of only the admin, apiCatalogIBM, and apiCatalogGlobalOrgUser users can be rotated. Complete these steps to set up their password rotation:

  1. Create a secret rotation custom resource (CR) specification YAML file with the username for which you want to rotate the password. The file name used as an example in these steps is rotation.apicatalog_v1beta1_apicatalogcertkeyrotation.yaml.

      apiVersion: rotation.apicatalog.ibm.com/v1beta1
  kind: ApiCatalogCertKeyRotation
  metadata:
    name: apicatalogcertkeyrotation2
  spec:
    rotateUserPasswords:
      - "<username>"
      - "<username>"
      - "<username>"

    In the specification, the <username> value can be admin, apiCatalogIBM, or apiCatalogGlobalOrgUser. You can add more than one username.

  2. Create the secret rotation CR in the namespace where you deployed foundational services. In the following command, the default namespace ibm-common-services is used.

     oc create -f rotation.apicatalog_v1beta1_apicatalogcertkeyrotation.yaml -n ibm-common-services

    Following is a sample response:

     apicatalogcertkeyrotation.rotation.apicatalog.ibm.com/apicatalogcertkeyrotation created

  3. Verify whether the rotation job Failed or Completed.

     oc get apicatalogcertkeyrotation -n ibm-common-services

    Following is a sample output:

     NAME                         READY   STATUS      AGE
 apicatalogcertkeyrotation2   1/1     Completed   19m

    • If the password was successfully rotated, the status shows as Completed. The new password is automatically updated in the secret of that username.

    • If the rotation job fails, the status shows as Failed. To identify the password rotations that failed, first check the rotatePwordMessage of the CR by running the following command:

      oc get apicatalogcertkeyrotation <CR-name> -n ibm-common-services -o yaml`

      The output shows the usernames and the status of their password rotation. See the following sample message:

      rotatePwordMessage: Usernames that pword changed [apiCatalogIBM admin apiCatalogGlobalOrgUser],
usernames failed [notAUser]

      Then, check the ibm-apicatalog operator pod logs to view the error messages that explain the reasons of failure. For steps to view pod logs of an operator, see Gathering Operator logs Opens in a new tab.

Encryption secret rotation

Rotate the secret that is used for encrypted database fields.

Complete these steps to rotate the encryption secret:

  1. Create a secret rotation CR specification YAML file. The file name used as an example in these steps is rotation.apicatalog_v1beta1_apicatalogcertkeyrotation.yaml. If you want to, you can specify the name of the secret to contain the new encryption_secret.bin, or you can omit it so that a new encryption key is automatically generated.

      apiVersion: rotation.apicatalog.ibm.com/v1beta1
  kind: ApiCatalogCertKeyRotation
  metadata:
    name: apicatalogcertkeyrotation
  spec:
    rotateEncryptionSecret:
      rotate: true
      # encryptionSecret: <name_of_secret>

  2. Create the secret rotation CR in the namespace where you deployed foundational services. In the following command, the default namespace ibm-common-services is used.

     oc create -f rotation.apicatalog_v1beta1_apicatalogcertkeyrotation.yaml -n ibm-common-services
 apicatalogcertkeyrotation.rotation.apicatalog.ibm.com/apicatalogcertkeyrotation created

    The rotation job now starts.

  3. Verify the status of the rotation CR. After the CR is created, initially, the status might show as Pending. After some time, the status shows as Running. After the rotation job successfully completes, the status of the CR shows as Completed.

    • Verify the status of the rotation CR.

      oc get apicatalogcertkeyrotation -n ibm-common-services

      Following are sample outputs:

      NAME                        READY   STATUS    AGE
apicatalogcertkeyrotation   0/1     Pending   37s

      NAME                        READY   STATUS    AGE
apicatalogcertkeyrotation   0/1     Running   71s

    • Verify the status of the rotation job. At this stage, the status shows as Running.

      oc get pods -n ibm-common-services | grep apicatalog

      Following is a sample output:

      NAME                                                    READY   STATUS             RESTARTS   AGE
apicatalog-apicatalogcertkeyrotation-kqxmr              0/1     Running            0          15s
apicatalog-apidiscovery-56949d7d9-shf9h                 1/1     Running            0          128m
apicatalog-apim-8b48cdcf9-g4n87                         1/1     Running            4          128m
apicatalog-apim-data-populate-nh84h                     0/1     Completed          0          3d23h
.
.

      If the job successfully completes, the status of the rotation CR and rotation job shows as Completed.

    • Status of the rotation CR:

        oc get apicatalogcertkeyrotation -n ibm-common-services

      Following is a sample output:

        NAME                        READY   STATUS      AGE
  apicatalogcertkeyrotation   1/1     Completed   10m

    • Status of the rotation job pod:

        oc get pods -n ibm-common-services | grep apicatalog

      Following is a sample output:

        NAME                                                    READY   STATUS             RESTARTS   AGE
  apicatalog-apicatalogcertkeyrotation-kqxmr              0/1     Completed          0          9m6s
  apicatalog-apidiscovery-56949d7d9-shf9h                 1/1     Running            0          128m
  apicatalog-apim-8b48cdcf9-g4n87                         1/1     Running            4          128m
  apicatalog-apim-data-populate-nh84h                     0/1     Completed          0          3d23h
  .
  .

    • Status of the rotation job:

        oc get job -n ibm-common-services | grep apicatalog

      Following is a sample output:

        NAME                                   COMPLETIONS   DURATION   AGE
  apicatalog-apicatalogcertkeyrotation   1/1           17s        15m
  apicatalog-apim-data-populate          1/1           64s        4d
  apicatalog-apim-schema                 1/1           9m14s      4d
  apicatalog-lur-data-populate           1/1           7s         4d
  apicatalog-lur-schema                  1/1           8s         4d

Certificate renewal

The certificate authority (CA) certificates and end-entity certificates have a fixed duration of validity and time period before which they need to be renewed. You cannot change these time periods. Following table shows the fixed values:

Table 1. Certificate duration and renewal time
Duration RenewBefore
CA Certificate 87600 hours (10 years) 24 hours
End entity Certificates 17520 hours (2 years) 24 hours

If you want to manually renew the certificates at any point in time, complete these steps:

Note: You must renew the end-entity certificates after you renew the CA certificate.

  1. Install the Kubernetes cert-manager plug-in. For more information, see Kubectl plugin Opens in a new tab.

  2. View the list of all certificates in your cluster by using the following command. You can also see the certificate status, and the date and time of the last update of each certificate.

     oc get certs -n ibm-common-services -o wide

    Following is an example output:

     NAME                             READY   SECRET                              ISSUER                   STATUS                                          AGE   EXPIRATION
 apicatalog-api-endpoint-secret   True    apicatalog-api-endpoint-secret      ingress-issuer           Certificate is up to date and has not expired   12h   2023-03-17T20:42:32Z
 apicatalog-apicuser              True    apicatalog-apicuser                 apicatalog-ca            Certificate is up to date and has not expired   12h   2023-10-04T08:49:46Z
 apicatalog-apidiscovery          True    apicatalog-apidiscovery             apicatalog-ca            Certificate is up to date and has not expired   12h   2023-10-04T08:49:27Z
 apicatalog-ca                    True    apicatalog-ca                       selfsigning-issuer       Certificate is up to date and has not expired   12h   2031-10-02T08:16:14Z
 apicatalog-client                True    apicatalog-client                   apicatalog-ca            Certificate is up to date and has not expired   12h   2023-10-04T08:49:41Z
 apicatalog-db-replication        True    apicatalog-db-replication           apicatalog-ca            Certificate is up to date and has not expired   12h   2023-10-04T08:49:33Z
 apicatalog-db-server             True    apicatalog-db-server                apicatalog-ca            Certificate is up to date and has not expired   12h   2023-10-04T08:49:30Z
 apicatalog-ingress-ca            True    apicatalog-ingress-ca               selfsigning-issuer       Certificate is up to date and has not expired   12h   2022-01-01T20:33:17Z
 apicatalog-postgres              True    apicatalog-postgres                 apicatalog-ca            Certificate is up to date and has not expired   12h   2023-10-04T08:49:44Z
 apicatalog-server                True    apicatalog-server                   apicatalog-ca            Certificate is up to date and has not expired   12h   2023-10-04T08:49:49Z
 auth-pdp-cert                    True    auth-pdp-secret                     cs-ca-issuer             Certificate is up to date and has not expired   12h   2022-01-01T20:13:02Z
 common-web-ui-ca-cert            True    common-web-ui-cert                  cs-ca-issuer             Certificate is up to date and has not expired   12h   2022-01-01T20:13:51Z
 cs-ca-certificate                True    cs-ca-certificate-secret            cs-ss-issuer             Certificate is up to date and has not expired   13h   2023-10-03T20:00:16Z
 iam-pap-cert                     True    iam-pap-secret                      cs-ca-issuer             Certificate is up to date and has not expired   12h   2022-01-01T20:13:03Z
 icp-mongodb-client-cert          True    icp-mongodb-client-cert             mongodb-root-ca-issuer   Certificate is up to date and has not expired   12h   2023-10-03T20:15:34Z
 identity-provider-cert           True    identity-provider-secret            cs-ca-issuer             Certificate is up to date and has not expired   12h   2022-01-01T20:13:12Z
 internal-tls-certificate         True    internal-tls                        zen-tls-issuer           Certificate is up to date and has not expired   13h   2022-01-01T20:01:08Z
 internal-tls-pkcs8-certificate   True    internal-tls-pkcs8                  zen-tls-issuer           Certificate is up to date and has not expired   13h   2022-01-01T20:01:08Z
 management-ingress-cert          True    icp-management-ingress-tls-secret   cs-ca-issuer             Certificate is up to date and has not expired   12h   2022-10-03T20:13:31Z
 mongodb-root-ca-cert             True    mongodb-root-ca-cert                god-issuer               Certificate is up to date and has not expired   12h   2023-10-03T20:14:48Z
 platform-api-ca-cert             True    platform-api-secret                 cs-ca-issuer             Certificate is up to date and has not expired   12h   2022-01-01T20:15:18Z
 platform-auth-cert               True    platform-auth-secret                cs-ca-issuer             Certificate is up to date and has not expired   12h   2022-01-01T20:13:06Z
 platform-identity-management     True    platform-identity-management        cs-ca-issuer             Certificate is up to date and has not expired   12h   2022-01-01T20:13:16Z
 route-cert                       True    route-tls-secret                    cs-ca-issuer             Certificate is up to date and has not expired   12h   2022-10-03T20:13:34Z
 zen-ca-certificate               True    zen-ca-cert-secret                  zen-ss-issuer            Certificate is up to date and has not expired   13h   2023-10-03T20:05:02Z

  3. To renew the CA certificate that the IBM API Catalog service uses, run the following command:

     oc cert-manager renew apicatalog-ca -n ibm-common-services

    Some IBM API Catalog service pods restart after you renew the certificate. Wait until the apicatalogmanager CR shows the Running status.

     oc get apicatalogmanager -n ibm-common-services

    Following is a sample output:

     NAME         READY   STATUS    RECONCILED VERSION   AGE
 apicatalog   12/12   Running   1.0.0-55             12h

  4. To renew the end-entity certificates that were issued by the CA, run the following command:

     oc cert-manager renew apicatalog-apicuser apicatalog-apidiscovery apicatalog-client apicatalog-db-replication apicatalog-db-server apicatalog-postgres apicatalog-server -n ibm-common-services

  5. Restart the database pod.

    1. Get the database pod name.

      oc get pods -n ibm-common-services | grep apicatalog-db

    2. Delete the pod to restart it.

      oc delete pod <pod-name> -n ibm-common-services

      Some IBM API Catalog service pods restart after you renew the certificate. Wait until the apicatalogmanager pod shows the Running status.

      oc get apicatalogmanager -n ibm-common-services

      Following is a sample output:

      NAME         READY   STATUS    RECONCILED VERSION   AGE
apicatalog   12/12   Running   1.0.0-55             12h

Client ID and secret rotation

You can rotate only the apicatalog-juhu-cred client ID and secret.

  1. Create a secret rotation CR specification YAML file with the ID and secret of the apicatalog-juhu-cred client. The file name used as an example in these steps is rotation.apicatalog_v1beta1_apicatalogcertkeyrotation.yaml.

      apiVersion: rotation.apicatalog.ibm.com/v1beta1
  kind: ApiCatalogCertKeyRotation
  metadata:
    name: apicatalogcertkeyrotation1
  spec:
    rotateClientIDandSecret:
      secrets:
      - apicatalog-juhu-cred

  2. Create the secret rotation CR in the namespace where you deployed foundational services. In the following command, the default namespace ibm-common-services is used.

     oc create -f rotation.apicatalog_v1beta1_apicatalogcertkeyrotation.yaml -n ibm-common-services
 apicatalogcertkeyrotation.rotation.apicatalog.ibm.com/apicatalogcertkeyrotation created

  3. Verify whether the rotation job Failed or Completed.

     oc get apicatalogcertkeyrotation -n ibm-common-services

    Following is a sample output:

     NAME                         READY   STATUS      AGE
 apicatalogcertkeyrotation1   1/1     Failed      19m
 apicatalogcertkeyrotation2   1/1     Completed   19m