Installing cert-manager and certificates in a two data center deployment

Additional instructions explaining how to install a cert-manager and certificates in a two data center disaster recovery deployment on Kubernetes.

Before you begin

Before you install a cert-manager, you should know your strategy for using certificates with API Connect. Review the certificate requirements in Deployment requirements.

You should also have already completed Obtaining product files.

About this task

You must create certificates and keys for API Manager and the Developer Portal on both data centers, dc1 and dc2, and ensure that they match on both.

Use these instructions to install the supplied cert-manager and ingress-issuer-v1.yaml.

Note that ingress-issuer-v1.yaml is supplied by cert-manager.

Procedure

Install cert-manager.
Cert-manager adds convenience to the generation and management of API Connect certificates. For more information about cert-manager, see Key Concepts: Cert-manager, Issuers, and Secrets.
1. Obtain the certificate manager.
  API Connect v10 uses cert-manager v1.12.13 of cert-manager, which is a native Kubernetes certificate management controller.
  
  You can obtain cert-manager v1.12.13 from the API Connect v10 distribution helper_files.zip archive, or download it from https://github.com/cert-manager/cert-manager/releases/tag/v1.12.13
2. Apply the CR:
```
kubectl apply -f cert-manager-1.12.13.yaml
```
  Do not specify a custom namespace.
  
  See https://cert-manager.io/docs/release-notes/release-notes-1.5/.
3. Wait for cert-manager pods to enter Running 1/1 status before proceeding. To check the status:
```
kubectl get po -n cert-manager 
```
  There are 3 cert-manager pods in total.
Use the following steps to allow ingress-ca secrets to be the same on both data centers.
1. On DC1 apply the file ingress-issuer-v1-dc1.yaml:
```
kubectl -n <namespace> apply -f ingress-issuer-v1-dc1.yaml
```
2. Validate that the command succeeded:
```
kubectl get certificates -n <namespace>
```
3. Export ingress-ca secret as a yaml from DC1:
```
kubectl -n <namespace> get secret ingress-ca -o yaml > ingress-ca.yaml
```
4. Edit the ingress-ca.yaml file to remove all annotations, labels, creationTimestamp, managedFields, manager, operation, time, resourceVersion, selfLink, and uid. Also, if you are using a different namespace in DC2, then update the namespace field.
5. Copy the ingress-ca.yaml from DC1 to DC2 and apply that file on DC2:
```
kubectl -n <namespace> apply -f ingress-ca.yaml
```
6. On DC2 apply the file ingress-issuer-v1-dc2.yaml:
```
kubectl -n <namespace> apply -f ingress-issuer-v1-dc2.yaml
```
7. Use the following commands to test that they are the same, on DC1 run:
```
kubectl -n <namespace> get secrets ingress-ca -o yaml | grep tls.crt | grep -v 'f:tls' | awk '{print $2}' | base64 -d > /tmp/ingress.pem.dc1
```
8. On DC2 run:
```
kubectl -n <namespace> get secrets ingress-ca -o yaml | grep tls.crt | grep -v 'f:tls' | awk '{print $2}' | base64 -d > /tmp/ingress.pem.dc2
```
9. To see the differences run:
```
diff /tmp/ingress.pem.dc1 /tmp/ingress.pem.dc2 
```
  The files should be the same.
10. On DC2, to ensure that the certificates are working correctly and that they are using the ingress-ca secret. First, get the portal-admin-client crt file:
```
kubectl -n <namespace> get secrets portal-admin-client -o yaml | grep tls.crt | awk '{print $2}' | base64 -d > /tmp/admin-client.crt
```
11. Test that it is working by using OpenSSL:
```
openssl verify -verbose -CAfile /tmp/ingress.pem.dc1 /tmp/admin-client.crt
```
  If it is working, you should see:
```
/tmp/admin-client.crt: OK
```
Continue with Deploying operators and cert-manager.