Generating custom certificates using cert-manager in a two data center deployment

You can generate custom certificates in a two data center disaster recovery deployment on Kubernetes.

1. Set KUBECONFIG for the target cluster:

   export KUBECONFIG=<path_to_cluster_config_YAML_file>

   Example path:

   /Users/user/.kube/clusters/<cluster_name>/kube-config-<cluster_name>.yaml

2. Install cert-manager version 1.5.1 or later :
   1. Download v1.16.3 from https://github.com/cert-manager/cert-manager/releases/tag/v1.16.3.
   2. Install cert-manager. Do not specify a namespace:

      kubectl apply -f cert-manager.yaml

   3. Check the status of the cert-manager pods:

      kubectl -n cert-manager get po

      Wait for cert-manager pods to enter Running 1/1 status before proceeding. There are 3 cert-manager pods in total.

3. Set up the customized external certificates:

   For quick setup, you can use the provided yaml files that are included in the helper_files that you expanded in Deploying operators and cert-manager. The yaml for the active data center (DC1) is called custom-certs-external-dc1.yaml and the yaml for the warm-standby data center (DC2) is called custom-certs-external-dc2.yaml.

   1. Open custom-certs-external-dc1.yaml and custom-certs-external-dc2.yaml in a text-editor. Find and replace each occurrence of example.com with the desired ingress subdomain for the API Connect stack.
   2. Ensure that the management CR for DC1 is ready for applying but not yet applied, make a note of the hostname of the site.
   3. In custom-certs-external-dc1.yaml find spec.dnsNames
      For example, in DC1:

      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: dc1-mgmt-replication
      spec:
        commonName: dc1-mgmt-replication
        secretName: dc1-mgmt-replication
        dnsNames:
        - <INSERT SITE HOSTNAME HERE. THIS WILL BE THE SAME VALUE AS IN THE MANAGEMENT CR spec.multiSiteHA.hosts.name for site 1>
        issuerRef:
          name: ingress-issuer
        usages:
        - "signing"
        - "key encipherment"
        duration: 17520h # 2 years
        renewBefore: 720h # 30 days
        privateKey:
          rotationPolicy: Always

   4. Replace the line that begins - <INSERT SITE HOSTNAME HERE with - hostname of the site.
   5. Ensure that the management CR for DC2 is ready for applying but not yet applied, make a note of the hostname of the site.
   6. In custom-certs-external-dc2.yaml find spec.dnsNames and replace the line that begins - <INSERT SITE HOSTNAME HERE with - hostname of the site.
   7. Ensure that the portal CR for DC1 is ready for applying but not yet applied, make a note of the hostname of the site.
   8. In custom-certs-external-dc1.yaml find spec.dnsNames
      For example, in DC1:

      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: dc1-ptl-replication
      spec:
        commonName: dc1-ptl-replication
        secretName: dc1-ptl-replication
        dnsNames:
        - <INSERT SITE HOSTNAME HERE. THIS WILL BE THE SAME VALUE AS IN THE PORTAL CR spec.multiSiteHA.hosts.name for site 1>
          issuerRef:
          name: ingress-issuer
        usages:
        - "signing"
        - "key encipherment"
        duration: 17520h # 2 years
        renewBefore: 720h # 30 days
        privateKey:
          rotationPolicy: Always

   9. Replace the line that begins - <INSERT SITE HOSTNAME HERE with - hostname of the site.
   10. Ensure that the portal CR for DC2 is ready for applying but not yet applied, make a note of the hostname of the site.
   11. In custom-certs-external-dc2.yaml find spec.dnsNames and replace the line that begins - <INSERT SITE HOSTNAME HERE with - hostname of the site.
4. Use the following steps to allow ingress-ca secrets to be the same on both data centers.
   1. On DC1 apply the file custom-certs-external-dc1.yaml:

      kubectl -n <namespace> apply -f custom-certs-external-dc1.yaml

   2. Validate that the command succeeded:

      kubectl get certificates -n <namespace>

   3. Export ingress-ca secret as a yaml from DC1:

      kubectl -n <namespace> get secret ingress-ca -o yaml > ingress-ca.yaml

   4. Edit the ingress-ca.yaml file to remove all labels, annotations, creationTimestamp, resourceVersion, uid, and selfLink.
   5. Copy the ingress-ca.yaml from DC1 to DC2 and apply that file on DC2:

      kubectl -n <namespace> apply -f ingress-ca.yaml
      

   6. On DC2 apply the file custom-certs-external-dc2.yaml:

      kubectl -n <namespace> apply -f custom-certs-external-dc2.yaml
      

   7. Use the following commands to test that they are the same, on DC1 run:

      kubectl -n <namespace> get secrets ingress-ca -o yaml | grep tls.crt | awk '{print $2}' | base64 -d > /tmp/ingress.pem.dc1

   8. On DC2 run:

      kubectl -n <namespace> get secrets ingress-ca -o yaml | grep tls.crt | awk '{print $2}' | base64 -d > /tmp/ingress.pem.dc2

   9. To see the differences run:

      diff /tmp/ingress.pem.dc1 /tmp/ingress.pem.dc2 

      The files should be the same.
   10. To ensure that the certificates are working correctly and that they are using the ingress-ca secret. First, get the portal-admin-client crt file:

       kubectl -n <namespace> get secrets portal-admin-client -o yaml | grep tls.crt | awk '{print $2}' | base64 -d > /tmp/admin-client.crt

   11. Test that it is working by using OpenSSL:

       openssl verify -verbose -CAfile /tmp/ingress.pem.dc1 /tmp/admin-client.crt

       If it is working, you should see:

       /tmp/admin-client.crt: OK

5. Update the API Connect subsystem CR .yaml files to use the custom certificates. To do this, continue with Configuring custom certificates before installation.