Renewing certificates in a two data center deployment on Kubernetes and OpenShift

Renewing TLS certificates in an API Connect two data center disaster recovery deployment requires you to copy some updated information between data centers while ensuring that the shared information is not overwritten during the remaining updates.

About this task

Begin the process by updating certificates as needed on data center 1 (DC1). Then,copy the ingress-ca from DC1 to data center 2 (DC2) before updating certificates on DC2. Finally, you will copy encryption keys for the Management and Portal subsystem from DC1 to DC2.

Procedure

Complete the following steps in the sequence shown to ensure that you do not overwrite information that was copied from DC1 while updating DC2.

  1. DC1: Renew TLS certificates as explained in Renewing TLS certificates.
  2. DC2: Complete the following steps to copy the ingress-ca from DC1:
    1. On DC1, export the ingress-ca secret to a YAML file by running the following command:
      kubectl -n namespace get secret ingress-ca -o yaml > ingress-ca.yaml
    2. Still on DC1, edit the ingress-ca.yaml file to remove all labels, annotations, creationTimestamp, resourceVersion, uid, and selfLink.
    3. Copy the ingress-ca.yaml to DC2.
    4. On DC2, apply the ingress-ca.yaml file by running the following command:
      kubectl -n namespace apply -f ingress-ca.yaml
    5. Validate that the ingress-ca on DC2 matches the ingress-ca on DC1:
      1. On DC1, run the following command to save the original ingress-ca as a PEM file:
        kubectl -n namespace get secrets ingress-ca -o yaml | grep tls.crt | awk '{print $2}' | base64 -d > /tmp/ingress.pem.dc1
      2. On DC2, run the following command to save the copied ingress-ca as a PEM file::
        kubectl -n namespace get secrets ingress-ca -o yaml | grep tls.crt | awk '{print $2}' | base64 -d > /tmp/ingress.pem.dc2
      3. Copy /tmp/ingress.pem.dc1 from DC1 and store it in the /tmp folder on DC2.
      4. On DC2, run the following command to compare the two files and verify that there are no differences:
        diff /tmp/ingress.pem.dc1 /tmp/ingress.pem.dc2 
  3. DC2: Update all external certificates that are based on the ingress-ca (see List of ingress certificates).
  4. DC2: Update all of the internal certificates (see List of intra-subsystem certificates).

    At this point, DC1 and DC2 have the same ingress-ca secrets. Next, copy the encryption keys for the Management and Portal subsystems from DC1 to DC2.

  5. Copy the encryption key for the Management subsystem to DC2 by completing the following steps:
    1. On DC1, copy the management encryption key:
      1. Get the name of the management encryption key by running the following command:
        $ kubectl -n namespace get mgmt -o yaml|grep enc 

        The response looks like the following example:

        encryptionSecret: dallas-enc-key

        Make a note of the secret name for the next step. In the example, the key's name is dallas-enc-key.

      2. Retrieve the secret from the encryption key by running the following command (substitute in the name of the key from the previous step):
        kubectl get secret dallas-enc-key -o yaml

        The output looks like the following example:

        apiVersion: v1
        data:
          encryption_secret.bin: VKBNFj7sAOizxvE1H6i+9P31oJHvWWsO+x***********************************EGe/K+x6b3D7FEWGgoyGlWBUJKB4+T21My2iR5rBTovpyLiY5g********************************************tSiRcQKegMPNBPgL829SVBCxuv3I=
        kind: Secret
        metadata:
          creationTimestamp: "2020-08-28T12:53:41Z"
          labels:
            app.kubernetes.io/instance: m1
            app.kubernetes.io/managed-by: ibm-apiconnect
            app.kubernetes.io/name: dallas-enc-key
          name: dallas-enc-key
          namespace: default
          resourceVersion: "43039"
          selfLink: /api/v1/namespaces/default/secrets/m1-enc-key
          uid: 46c92395-9cc2-4421-b2c4-48e472c0cbb1
        type: Opaque
      3. Copy the output and save it in a YAML file; for example dc1-enc-key.yaml.
      4. Edit the file and make the following changes:
        • Delete the contents of the metadata: section (retain the section).
        • In the metadata: section, add in a name: attribute, with dc1-enc-key as the value.

        The following example shows the updated file:

        apiVersion: v1
        data:
          encryption_secret.bin: VKBNFj7sAOizxvE1H6i+9P31oJHvWWsO+x***********************************EGe/K+x6b3D7FEWGgoyGlWBUJKB4+T21My2iR5rBTovpyLiY5g********************************************tSiRcQKegMPNBPgL829SVBCxuv3I= 
        kind: Secret
        metadata:
          name: dc1-enc-key
        type: Opaque
      5. Copy the updated file (dc1-enc-key.yaml) to the /tmp folder on DC2.
    2. On DC2, run the following command to create a secret from the encryption key file (dc1-enc-key.yaml) that you copied from DC1:
      $ kubectl  -n namespace create -f /tmp/dc1-enc-key.yaml

      The response looks like the following example:

      secret/dc1-enc-key created
  6. Update the encryption key for the Portal subsystem on DC1 and then copy it to DC2 as explained in Changing the secret for Portal data and system tools.