Renewing certificates with cert-manager

Use Kubernetes cert-manager to renew the issuers, CA certificates, and derived certificates that it manages for your API Connect deployment.

1. VMware only: Connect to the virtual machine as the API Connect administrator by completing the following steps:
   1. Run the following command to connect as the API Connect administrator, replacing ip_address with the appropriate IP address:

      ssh ip_address -l apicadm

   2. When prompted, select Yes to continue connecting.

   3. When you are connected, run the following command to receive the necessary permissions for working directly on the appliance:

      sudo -i

2. To renew a CA certificate or an issuer (for example, ingress-ca, management_CR-ca, analytics_CR-ca, or portal_CR-ca), complete the following steps:
   1. Run the following command to list of all the certificate names, with the status and the date and time of the last update for each certificate:

      kubectl get certificate -o wide

   2. Run the following command to renew the CA certificate:

      kubectl get certificate certificate_name -o=jsonpath='{.spec.secretName}' | xargs kubectl delete secret

      where certificate_name is the name of the CA certificate that you want to renew, and will be one of the following CA certificates: ingress-ca, management_CR-ca, analytics_CR-ca, or portal_CR-ca.

      Tip: For the management_CR-ca, you can optionally create a special CR (Custom Resource) to renew the CA certificate and its dependent certificates, then restart the dependent pods. For instructions, see Using a ManagementSecretRotation CR to update the management CA, later in this topic.
   3. Proceed to step 3 and renew each of the end-entity certificates that were issued by the cert-manager issuer based on the CA certificate.
3. Renew an end-entity certificate by running the following command:

   kubectl get certificate certificate_name -o=jsonpath='{.spec.secretName}' | xargs kubectl delete secret

   where certificate_name is the name of the certificate that you want to renew.

   Refer to the following topics to see the list of certificates that need renewing based on the CA certificate that you renewed in step 1:

   • List of internal certificates
   • List of external certificates

   Tip: To renew all the end-entity certificates of the ingress-ca issuer, you can use this one line command:

   kubectl get secrets -n <namespace> -o custom-columns='NAME:.metadata.name,ISSUER:.metadata.annotations.cert-manager\.io/issuer-name' --no-headers=true | grep ingress-issuer | awk '{ print $1 }' | xargs kubectl delete secret -n <namespace>

   This command only works for ingress-ca issued certificates, as the other issuers might have secret names that are the same as the issuer name.
4. For each updated certificate, check that the pods that are affected by that certificate are restarted. If they do not restart automatically, restart them manually by deleting the pod.
   To see which pods are affected by each certificate, see the following topics:

   • List of external certificates
   • List of internal certificates

   Then, see the following topics for instructions for restarting pods:

   • Management subsystem: restarting pods
   • Gateway subsystem: restarting pods
   • Analytics subsystem: restarting endpoints and pods
   • Portal subsystem: restarting pods for a two data center configuration or Cloud Pak for Integration

     If you are using a single data center configuration with a stand-alone API Connect deployment, the Portal subsystem pods pick up the new certs and restart any internal processes automatically. Changing the portal-server or portal-client certificates will be a zero downtime operation. However, changing the portal-ca certificate will cause downtime because all portal-db pods need to restart MySQL to allow for the certificate to update. The pods will not restart, but they will all go non-ready and then ready again, which will take a few minutes.

   Note: Due to a known issue, when the ingress-ca is renewed, it is possible that Analytics in the Developer Portal might stop working. If this happens, complete the following steps to ensure that certificate changes take effect:

   1. Log in to the Cloud Manager user interface.
   2. In the navigation list, click Topology.
   3. Edit the Analytics service.
   4. On the Analytics page, edit the Summary field to force a change; for example, by adding a space to the end of a sentence.
   5. Click Save.
5. If you have subsystems in different namespaces, you need to update the renewed CA secrets to be the same those namespaces. The following example shows how to do this for the ingress-ca where the gateway is in a different namespace:
   1. Export the management namespace ingress-ca secret as a yaml:

      kubectl -n <management namespace> get secret ingress-ca -o yaml > ingress-ca.yaml

   2. Edit the ingress-ca.yaml file to remove all annotations, labels, creationTimestamp, managedFields, manager, operation, time, resourceVersion, selfLink, and uid. Update the namespace field to that of your gateway namespace.
   3. Apply the updated the ingress-ca.yaml in the namespace of the other subsystem:

      kubectl -n <gateway namespace> apply -f ingress-ca.yaml
      

   4. Use the following commands to confirm that the ingress-ca certificates are the same, in both namespaces run:

      kubectl -n <management namespace> get secrets ingress-ca -o yaml | grep tls.crt | grep -v 'f:tls' | awk '{print $2}' | base64 -d > /tmp/ingress.pem.dc1

      kubectl -n <gateway namespace> get secrets ingress-ca -o yaml | grep tls.crt | grep -v 'f:tls' | awk '{print $2}' | base64 -d > /tmp/ingress.pem.dc2

   5. To see the differences run:

      diff /tmp/ingress.pem.dc1 /tmp/ingress.pem.dc2 

      The files should be the same.
   6. Repeat steps 3 and 4 to renew the end-entity certificates in the other namespace.