Multiple CertificateRequest objects block Certificate objects from becoming ready

The duplicate CertificateRequests in the cert manager block the Certificates to be in the ready status.

Symptoms

• Certificate objects never become ready even when the Issuer object is ready.

• The following error message is displayed in the cert-manager-controller pod:

   "error"="multiple CertificateRequests were found for the...
  

Cause

• The cluster contains multiple cert-manager-controller pods.

• The cert-manager-controller pod restarts several times.

Resolving the problem

Resolving multiple cert-manager-controller pods issue

To resolve the multiple cert-manager-controller pods issue, see Problem when you install two different cert-managers.

Resolving cert-manager-controller pods restart issue

Complete the following steps to resolve the restart issue of the cert-manager-controller pods:

1. Find the root cause of the issue. Some possible causes for the issue are as follows:

   • Too many secrets on the cluster. For example, ten of thousands of secrets on the cluster.
   • Out of Memory (OOM) issues for cert-manager-controller pod.
   • Insufficient CPU or memory allocation for cert-manager-controller pod.

   Note: If the possible causes for the issue are not found on the cluster, contact IBM Support.

2. Complete the following steps to fix the restart issue:

   1. Increase the CPU or memory limit for the cert-manager-controller pod.
   2. Delete the duplicate secrets.

   Note: It is possible that some service on the cluster constantly generates new secrets incorrectly. If the root cause for the issue is not found, contact IBM Support.

3. Scale down the ibm-cert-manager-operator pod to 0 temporarily.

   oc scale --replicas=0 deployment ibm-cert-manager-operator
   

4. Scale down the cert-manager-controller pod to 0 temporarily.

   oc scale --replicas=0 deployment cert-manager-controller
   

5. Delete the duplicate CertificateRequests that are shown in the error logs of the cert-manager-controller pod.

   1. Find the CertificateRequests for a particular Certificate that is shown in the error logs.

      oc get certificaterequest -n <namespace> | grep <certificate name from logs>
      

   2. Delete all CertificateRequests for a particular Certificate.

      oc get certificaterequest -n <namespace> | grep <certificate name from logs> | xargs oc delete -n <namespace> certificaterequest
      

      Note: The deletion process takes some time if there are too many CertificateRequests. Ensure that you delete the CertificateRequests for all Certificates that are shown in the error logs of the cert-manager-controller pod.

6. Scale up the cert-manager-controller pod to 1

   oc scale --replicas=1 deployment cert-manager-controller
   

7. Scale up the ibm-cert-manager-operator pod to 1.

   oc scale --replicas=1 deployment ibm-cert-manager-operator