After upgrade or reinstallation, cert-manager does not refresh the mutating or validating of webhook configurations, causing failure to create Issuers and Certificates. 

Symptom

The following three cases show when the Cert Manager might fail to call webhook.

Case 1

• IBM Cert Manager Operator is installed in the cluster, but IBM Common Service Operator did not install cs-ca-certificate, cs-ca-issuer, and cs-ss-issuer resources.

• Cert Manager Controller pod shows the following error messages:

  2023-04-28T13:40:42.299Z ERROR controller.certificate-controller failed to create v1 Certificate {"name": "ibm-monitoring-certs", 
  "namespace": "ibm-common-services", "Request.Namespace": "ibm-common-services", "Request.Name": "ibm-monitoring-certs", 
  "error": "Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: 
  Post "https://cert-manager-webhook.openshift-cert-manager.svc:443/mutate?timeout=10s\": service "cert-manager-webhook" not found"}
  

Case 2

• After upgrading foundational services to a new version, you are not able to create Issuers and Certificates, and the following error is displayed in the logs:

  ERROR   controller.certmanager-controller Reconciler error {"name": "default", "namespace": "", "error": "Internal error occurred: failed calling webhook
  "webhook.cert-manager.io\": Post \"https://cert-manager-webhook.ibm-common-services.svc:443/mutate?timeout=10s\": x509: certificate signed by unknown authority"}
  

Case 3

• When you have installed IBM foundational service Cert Manager or CNCF Cert Manager in the cluster before, uninstall them, then try to install IBM Cert Manager, IBM Cert Manager operator pod is running but cert-manager-controller, cert-manager-cainjector, cert-manager-webhook pod are not deployed.

• IBM Cert Manager Operator pod shows the following error messages:

  2023-05-05T20:22:29.090Z INFO controller_certmanager Failed to create Issuer {"name:": "smoke-check-issuer", "namespace:": "cs-control"}
  2023-05-05T20:22:29.090Z INFO controller_certmanager Checking if error is from webhook
  2023-05-05T20:22:29.090Z INFO controller_certmanager Auto-detection found error with calling cert-manager-webhook, verify your open source cert-manager installation, and then restart this pod
  

Case 4

• When you have multiple cert-manager installed and delete one of the cert-manager, depending on the sequence of original installation, the mutatingwebhookconfiguration/cert-manager-webhook namespace can be pointing to the wrong namespace, resulting in the following error is displayed in the logs:

  
  E0828 12:48:33.340398 1 controller.go:163] cert-manager/controller/certificates-issuing "msg"="re-queuing item due to error processing" 
  "error"="Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: 
  Post \"https://cert-manager-webhook.ibm-common-services.svc:443/mutate?timeout=10s\": service \"cert-manager-webhook\" not found" "key"="xxxx/internal-tls-pkcs12-certificate"