secret-watcher and security-onboarding pods fail to start
During IBM Cloud Pak foundational services installation, the
security-onboarding pods fail to start.
The identity management (IM) pods are not ready.
To get the status of
security-onboarding pods, run the following command:
oc get pod -A | grep -e secret-watcher -e security-onboarding
After you run the command, you might get the following output. The
STATUS in the sample output shows that the
secret-watcher pod is running successfully and the
security-onboarding pod is completed.
NAMESPACE NAME READY STATUS RESTARTS AGE <your-foundational-services-namespace> secret-watcher-5c79cb8969-7w4nw 1/1 Running 0 19h <your-foundational-services-namespace> security-onboarding-8kh8l 0/1 Completed 0 20h
For an instance, the
STATUS of the pods can also display the error:
NAMESPACE NAME READY STATUS RESTARTS AGE <your-foundational-services-namespace> secret-watcher-cccd54d9f-vdnn2 0/1 CreateContainerConfigError 0 20m <your-foundational-services-namespace> security-onboarding-btlhh 0/1 CreateContainerConfigError 0 20m
If you observe the error in the pod's status, you can run the following command to know the cause of the
CreateContainerConfigError error. The following command display the list of events for the
oc get event --namespace <your-foundational-services-namespace> --field-selector involvedObject.name=security-onboarding-xxxx
After you run the command, you might get the following status that indicates that the "icp-serviceid-apikey-secret" key is missing due to which the pod is showing
CreateContainerConfigError error. It indicates the issue with
108m Warning Failed pod/security-onboarding-8kh8l Error: secret "icp-serviceid-apikey-secret" not found
Resolving the problem
iam-onboarding job to resolve the issue.
Note: From foundational services version 3.21, the
iam-onboarding job restarts automatically. However, if the restart process of the
iam-onboarding job takes longer time to complete, see iam-onboarding job is in progress state for longer time.
Log in to your boot node by running the
- Get the
oc get job iam-onboarding
- If the
iam-onboardingdoesn't exist, restart the IM operator.
oc delete po -l name=ibm-iam-operator
- If the
iam-onboardingjob exists, restart the iam-onboarding job.
oc get job iam-onboarding -o json | jq 'del(.spec.selector)' | jq 'del(.spec.template.metadata.labels)' | kubectl replace --force -f -
- Get the