Cannot create pod because of MutatingAdmissionWebhook failure

The MutatingAdmissionWebhook admission plug-in failed to complete mutation in 13 seconds.


You usually see this issue during upgrade of IBM Cloud Pak foundational services in your cluster.

When this issue happens, the replicaset-controllers cannot generate their pods. You can verify this behavior by deleting the pod.

  1. Get the pod names.

     oc get pods -n <pod-namespace>
  2. Delete a pod in the cluster.

     oc delete pod <pod-name> -n <pod-namespace>
  3. Verify whether the pod is created.

     oc get pods -n <pod-namespace>

    You see that the pod is not re-created.

  4. Get the replicaset of the pod.

     oc get replicaset -n <pod-namespace>
  5. Check the events in the replicaset.

     oc describe replicaset <pod-replicaset> -n <pod-namespace>

    Following is a sample event:

       Type     Reason        Age                 From                   Message
       ----     ------        ----                ----                   -------
       Warning  FailedCreate  37m (x20 over 47m)  replicaset-controller  Error creating: Internal error occurred: admission plug-in "MutatingAdmissionWebhook" failed to complete mutation in 13s


The MutatingAdmissionWebhook doesn't work in the cluster.

Resolving the problem

Delete the failed MutatingAdmissionWebhook.

  1. Get the kube-apiserver pod name. The kube-apiserver pod is in the openshift-kube-apiserver namespace in the OpenShift cluster.

     oc get pod -n openshift-kube-apiserver
  2. Identify the webhook server that has failed. Check the kube-apiserver logs to identify the failed webhook server.

     oc logs <kube-apiserver-pod> -n openshift-kube-apiserver -c <kube-apiserver-container>

    Following is a sample from the log:

       W0501 11:12:28.735594       1 dispatcher.go:168] Failed calling webhook, failing open failed calling webhook "": Post https://platform-identity-management.kube-system.svc:443/identity/api/v1/users/validateandmutate?timeout=30s: context canceled

    The log shows that the webhook has failed.

  3. Get the MutatingAdmissionWebhook information.

     oc get MutatingWebhookConfiguration
  4. Delete the failed webhook.

     oc delete MutatingWebhookConfiguration <webhook-name>
  5. Check whether the foundational services webhook is causing the webhook server to fail.

     oc get pod -n ibm-common-services | grep ibm-common-service-webhook

    If the ibm-common-service-webhook pod shows errors, delete the pod to re-create it.

     oc delete pod <ibm-common-service-webhook pod name> -n ibm-common-services