preload_data.sh fails due to cert manager
Symptom
When running the preload_data.sh script, it is possible that the cert manager is not ready at runtime, causing problems when creating the dummy mongo that is used in migrating data from the original namespace to the target namespace.
This might manifest through the following message being output repeatedly in the script logs:
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0 0/1 Init:0/2 0 5m59s
Or like this:
# oc get pod -n cpfs-data
NAME READY STATUS RESTARTS AGE
icp-mongodb-0 0/1 Init:0/2 0 12h
And when describing the mongo pod:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 25m (x390 over 12h) kubelet (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[ca], unattached volumes=[configdir install ca mongodbdir keydir kube-api-access-t6bhn config tmp-mongodb init]: timed out waiting for the condition
Warning FailedMount 5m19s (x322 over 12h) kubelet MountVolume.SetUp failed for volume "ca" : secret "mongodb-root-ca-cert" not found
Looking at the logs, the cert manager resources were not created properly:
# Deploying a temporary mongodb in cpfs-data
-----------------------------------------------------------------------
configmap/icp-mongodb-install created
configmap/icp-mongodb-init created
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.ibm-cert-manager.svc:443/mutate?timeout=10s": no endpoints available for service "cert-manager-webhook"
configmap/ibm-cpp-config created
secret/icp-mongodb-admin created
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.ibm-cert-manager.svc:443/mutate?timeout=10s": no endpoints available for service "cert-manager-webhook"
configmap/icp-mongodb created
secret/icp-mongodb-keyfile created
secret/icp-mongodb-metrics created
serviceaccount/ibm-mongodb-operand created
service/mongodb created
service/icp-mongodb created
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.ibm-cert-manager.svc:443/mutate?timeout=10s": no endpoints available for service "cert-manager-webhook"
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.ibm-cert-manager.svc:443/mutate?timeout=10s": no endpoints available for service "cert-manager-webhook"
configmap/namespace-scope created
Compared to a successful run of preload_data.sh:
# Deploying a temporary mongodb in cpfs-data
-----------------------------------------------------------------------
configmap/icp-mongodb-install created
configmap/icp-mongodb-init created
issuer.cert-manager.io/god-issuer created
configmap/ibm-cpp-config created
secret/icp-mongodb-admin unchanged
certificate.cert-manager.io/icp-mongodb-client-cert created
configmap/icp-mongodb created
secret/icp-mongodb-keyfile created
secret/icp-mongodb-metrics created
serviceaccount/ibm-mongodb-operand created
service/mongodb created
service/icp-mongodb created
certificate.cert-manager.io/mongodb-root-ca-cert created
issuer.cert-manager.io/mongodb-root-ca-issuer created
configmap/namespace-scope created
Resolution
First, wait for cert manager to come all the way ready. After all of the cert manager pods are running, this should be enough to determine whether the service is ready. You can check with the following command:
[root@ibm.com]# oc get pods -A | grep cert
ibm-cert-manager cert-manager-cainjector-64f697c766-hzt7n 1/1 Running 0 14m
ibm-cert-manager cert-manager-controller-648fb9cd58-wpffm 1/1 Running 0 14m
ibm-cert-manager cert-manager-webhook-549f89964b-nm2mt 1/1 Running 0 14m
ibm-cert-manager ibm-cert-manager-operator-5847b6ddf9-k9gvf 1/1 Running 0 14m
When you see that the cainjector, controller, and webhook pods are all running, this should be enough to determine whether cert manager is running. You can verify by attempting to create a test issuer or certificate. Using the following yaml resource:
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: test-issuer
namespace: <namespace>
spec:
selfSigned: {}
You can run oc apply -f <test issuer file name> and verify its successful creation. This resource can be safely deleted after.
After preload_data.sh fails, the dummy mongo must be cleaned up in the target namespace before attempting to run the script again. The following resources need to be deleted:
oc delete statefulset icp-mongodb --ignore-not-found -n <target namespace>
oc delete service icp-mongodb --ignore-not-found -n <target namespace>
oc delete issuer god-issuer --ignore-not-found -n <target namespace>
oc delete cm ibm-cpp-config --ignore-not-found -n <target namespace>
oc delete certificate icp-mongodb-client-cert --ignore-not-found -n <target namespace>
oc delete cm icp-mongodb --ignore-not-found -n <target namespace>
oc delete cm icp-mongodb-init --ignore-not-found -n <target namespace>
oc delete cm icp-mongodb-install --ignore-not-found -n <target namespace>
oc delete secret icp-mongodb-keyfile --ignore-not-found -n <target namespace>
oc delete secret icp-mongodb-metrics --ignore-not-found -n <target namespace>
oc delete sa ibm-mongodb-operand --ignore-not-found -n <target namespace>
oc delete service mongodb --ignore-not-found -n <target namespace>
oc delete certificate mongodb-root-ca-cert --ignore-not-found -n <target namespace>
oc delete issuer mongodb-root-ca-issuer --ignore-not-found -n <target namespace>
oc delete cm namespace-scope --ignore-not-found -n <target namespace>
oc delete pod icp-mongodb-0 --ignore-not-found -n <target namespace>
After the resources are deleted, the preload_data.sh script can be run again.
Note: The preceding part about cleaning up the dummy mongo is applicable for any preload data issue where the script does not complete.