secret-watcher and security-onboarding pods fail to start

During IBM Cloud Pak foundational services installation, the secret-watcher and security-onboarding pods fail to start.

Causes

The identity management (IM) pods are not ready.

To get the status of secret-watcher and security-onboarding pods, run the following command:

oc get pod -A | grep -e secret-watcher -e security-onboarding

After you run the command, you might get the following output. The STATUS in the sample output shows that the secret-watcher pod is running successfully and the security-onboarding pod is completed.

NAMESPACE                     NAME                              READY  STATUS     RESTARTS  AGE
<your-foundational-services-namespace>           secret-watcher-5c79cb8969-7w4nw   1/1    Running    0         19h
<your-foundational-services-namespace>           security-onboarding-8kh8l         0/1    Completed  0         20h

For an instance, the STATUS of the pods can also display the error:

NAMESPACE             NAME                             READY    STATUS                       RESTARTS    AGE
<your-foundational-services-namespace>   secret-watcher-cccd54d9f-vdnn2   0/1      CreateContainerConfigError   0           20m
<your-foundational-services-namespace>   security-onboarding-btlhh        0/1      CreateContainerConfigError   0           20m

If you observe the error in the pod's status, you can run the following command to know the cause of the CreateContainerConfigError error. The following command display the list of events for the security-onboarding pod:

oc get event --namespace <your-foundational-services-namespace> --field-selector involvedObject.name=security-onboarding-xxxx

After you run the command, you might get the following status that indicates that the "icp-serviceid-apikey-secret" key is missing due to which the pod is showing CreateContainerConfigError error. It indicates the issue with the iam-onboarding job.

108m    Warning  Failed      pod/security-onboarding-8kh8l  Error: secret "icp-serviceid-apikey-secret" not found

Resolving the problem

Restart the iam-onboarding job to resolve the issue.

1. Log in to your boot node by running the oc login command.

2. Restart the iam-onboarding job.

   • Get the iam-onboarding job.

   oc get job iam-onboarding
   

   • If the iam-onboarding doesn't exist, restart the IM operator.

   oc delete po -l name=ibm-iam-operator
   

   • If the iam-onboarding job exists, restart the iam-onboarding job.

   oc get job iam-onboarding -o json | jq 'del(.spec.selector)' | jq 'del(.spec.template.metadata.labels)' | kubectl replace --force -f -