Problem when you install two different cert-managers
The cert-manager that is installed by foundational services is based on the CNCF cert-manager . This was previously known as the Jetstack cert-manager.
Certificates never become ready
In the cert-manager-controller pod, there are error messages that indicate there is more than one CertificateRequest for one or more Certificates. Note that cert-manager will create a CertificateRequest object whenever a Certificate object is created, but there should not be more than one per Certificate. If there are, you will see an error message.
Multiple cert-managers are installed on one cluster. To check if there are multiple cert-managers on the cluster, run the following:
oc get pods -A | grep cert-manager
If you have foundational services cert-manager installed, then the output should resemble the following:
ibm-common-services cert-manager-cainjector-xxx-xxx ibm-common-services cert-manager-controller-xxx-xxx ibm-common-services cert-manager-webhook-xxx-xxx ibm-common-services ibm-cert-manager-operator-xxx-xxx
If you have the CNCF cert-manager installed, then the output should resemble the following:
cert-manager cert-manager-cainjector-xxx-xxx cert-manager cert-manager-xxx-xxx cert-manager cert-manager-webhook-xxx-xxx
If you see both sets of pods, then there are multiple cert-manager instances installed.
The CNCF cert-manager has a limitation where only one instance of it can properly run on a cluster. If there is more than one, then there can be unexpected behavior, such as Certificates never becoming ready.
Because foundational services installs a cert-manager that is based off of CNCF cert-manager, it has the same limitation.
Resolving the problem
Depending on the situation, there are several methods to resolve the problem, but all of them involve uninstalling one of the cert-managers.
Before attempting any method, create a backup of the Issuers, ClusterIssuers, and Certificate objects. For example:
oc get -A -o yaml issuers > issuers.yaml
None of the methods should remove any of these objects, but if an incident occurs and the Custom Resource Definition (CRD) for these objects are unintentionally removed, then those objects will also be removed. Do not remove any CRDs.
Method 1: Uninstall foundational services cert-manager operands
- Foundational services version 3.19 or later
If you want to use the CNCF cert-manager to manage certificates, either because it was installed on the cluster first or due to a requirement, then follow this method:
Follow the instructions in Control installation of Certificate manager operands
Delete the CertManager object by running the following command:
oc delete certmanagers.operator.ibm.com default
a. Optional: If the deletion gets stuck, you can force delete by editing the yaml and set the value of the finalizer to
oc edit certmanagers.operator.ibm.com default
ibm-cert-manager-operator pod. To get the pod name, run the following command:
oc get pods -n ibm-common-services -l "app.kubernetes.io/managed-by=ibm-cert-manager-operator"
Verify that the foundational services cert-manager pods are no longer running (except for
oc get pods -n ibm-common-services | grep cert-manager ibm-cert-manager-operator-xxxx
Method 2: Uninstall CNCF cert-manager
This method will vary depending on how the CNCF cert-manager was installed. The most important thing is that the CRDs are NOT removed. There are generally three ways that the CNCF cert-manager could have been installed:
Uninstalling kubectl apply
The CNCF instructions for uninstalling via the YAML file is simply to delete the YAML file you used in order to install it in the first place. However, this also removes the CRDs, which must be avoided. Hence there are two options:
Edit the YAML file first, and remove the CustomResourceDefinition YAMLs inside of it. After removing the CustomResourceDefinition YAMLs from the file, delete the file by running:
oc delete -f <file.yaml>
foundational services provides a YAML file you can use to delete cert-manager, which already has the CustomResourceDefinition YAMLs removed from it:
oc delete -f https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/cert-manager.yaml
Follow the CNCF instructions. However, skip the part where the CustomResourceDefinitions (CRDs) are deleted.
If using the Red Hat OpenShift Container Platform console:
- Navigate to Installed Operators
- Find the cert-manager operator (NOT the IBM Cert Manager)
- Click the three dots on the right and click "Uninstall operator"
If using the CLI:
Get the subscription:
oc get sub -n openshift-operators | grep cert-manager
Delete the subscription:
oc delete sub -n openshift-operators cert-manager
Get the CSV:
oc get csv -n openshift-operators | grep cert-manager
Delete the CSV:
oc delete csv -n openshift-operators cert-manager.v1.x.x
Note that the names of the subscriptions and CSVs may vary depending on how you installed it.
Identifying and deleting duplicate certificate requests for uninstalled cert-manager
Since there are two cert-managers in the environment, there could be duplicate certificate requests for each certificate — one generated by each cert-manager. You must identify the duplicate certificate requests for the uninstalled cert-manager and delete them.
- Run the following command then identify the
CertificateRequestsrequested by the uninstalled cert-manager (visible in the
REQUESTORcolumn of the output):
oc get certificaterequests -A
- Once you identify the
CertificateRequestsrequested by the uninstalled cert-manager, delete the
oc delete CertificateRequests <name of the CertificateRequests to be deleted> -n <namespace>
Example: If you decide to uninstall CNCF cert-manager, you can clear up all the
CertificateRequests requested by CNCF cert-manager-controller (that is, non ibm-cert-manager-controller). You can distinguish this from
the REQUESTOR column of the result and delete the
The following output shows the CertificateRequests generated from IBM cert-manager and third party cert-manger (CNCF cert-manager):
NAMESPACE NAME APPROVED DENIED READY ISSUER REQUESTOR AGE namespace1 auth-pdp-cert-dq62v True True cs-ca-issuer system:serviceaccount:ibm-common-services:ibm-cert-manager-controller 109d namespace1 auth-pdp-cert-svjr9 True True cs-ca-issuer system:serviceaccount:ibm-common-services:ibm-cert-manager-controller 169d namespace1 auth-pdp-cert-tsm2t True True cs-ca-issuer system:serviceaccount:ibm-common-services:ibm-cert-manager-controller 49d namespace1 common-web-ui-ca-cert-cn8rd True True cs-ca-issuer system:serviceaccount:ibm-common-services:ibm-cert-manager-controller 109d namespace1 ibm-bts-cert-79z5p True True ibm-bts-ca-issuer system:serviceaccount:openshift-operators:cert-manager 20d namespace1 iaf-system-automationui-aui-zen-cert-26p8b True True iaf-system-automationui-aui-zen-issuer system:serviceaccount:openshift-operators:cert-manager 2d12h namespace1 iaf-system-automationui-aui-zen-cert-29sh2 True True iaf-system-automationui-aui-zen-issuer system:serviceaccount:openshift-operators:cert-manager 2d12h namespace1 iaf-system-automationui-aui-zen-cert-2cxfw True True iaf-system-automationui-aui-zen-issuer system:serviceaccount:openshift-operators:cert-manager 2d12h