Cert Manager fails to call webhook

Symptoms

The following three cases show when the Cert Manager might fail to call webhook.

Case 1

• IBM Cert Manager Operator is installed in the cluster but IBM Common Service Operator did not install cs-ca-certificate, cs-ca-issuer, and cs-ss-issuer resources.

• Cert Manager Controller pod shows the following error messages:

   2023-04-28T13:40:42.299Z ERROR controller.certificate-controller failed to create v1 Certificate {"name": "ibm-monitoring-certs", "namespace": "ibm-common-services", "Request.Namespace": "ibm-common-services", "Request.Name": "ibm-monitoring-certs", "error": "Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.openshift-cert-manager.svc:443/mutate?timeout=10s\": service "cert-manager-webhook" not found"}
  

Case 2

• After upgrading foundational services to a new version, you are not able to create Issuers and Certificates, and the following error is displayed in the logs:

   ERROR   controller.certmanager-controller       Reconciler error        {"name": "default", "namespace": "", "error": "Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": Post \"https://cert-manager-webhook.ibm-common-services.svc:443/mutate?timeout=10s\": x509: certificate signed by unknown authority"}
  

Case 3

• When you have installed OpenShift Cert Manager or CNCF Cert Manager in the cluster before, uninstall them, then try to install IBM Cert Manager, IBM Cert Manager operator pod is running but cert-manager-controller, cert-manager-cainjector, cert-manager-webhook pod are not deployed.

• IBM Cert Manager Operator pod shows the following error messages:

   2023-05-05T20:22:29.090Z INFO controller_certmanager Failed to create Issuer {"name:": "smoke-check-issuer", "namespace:": "cs-control"}
   2023-05-05T20:22:29.090Z INFO controller_certmanager Checking if error is from webhook
   2023-05-05T20:22:29.090Z INFO controller_certmanager Auto-detection found error with calling cert-manager-webhook, verify your open source cert-manager installation, and then restart this pod
  

Cause

After upgrade or reinstallation, cert-manager does not refresh the mutating or validating of webhook configurations.

Resolving the problem

You must remove the webhook configuration manually.

✗ oc get mutatingwebhookconfigurations | grep cert-manager
cert-manager-webhook                                                1          13d

✗ oc get validatingwebhookconfigurations | grep cert-manager
cert-manager-webhook                                             1          13d

Deleting the webhook configurations and restarting the cert-manager-webhook pod resolves this issue. If the cert-manager-webhook pod does not exist, restart the ibm-cert-manager-operator pod.