Secret watcher pod shows CrashLoopBackOff status

The secret-watcher pod keeps crashing and shows a CrashLoopBackOff status.

Cause

There might be many Kubernetes secrets that are configured in the cluster. The current memory limit that is configured in the secret-watcher pod is not sufficient to manage the Kubernetes secrets. Kubernetes killed the pod due to less memory.

Resolving the problem

To resolve the issue, increase the resource memory limit to a value that is sufficient for the pod to manage all the Kubernetes secrets.

  1. Log in to your boot node with the oc login command.

  2. Check the number of Kubernetes secrets that are in the cluster.

    oc get secrets --all-namespaces | wc -l
    
  3. Edit the CommonService custom resource.

    oc -n <your-foundational-services-namespace> edit CommonService common-service
    
  4. Add the following piece of configuration under the spec.services section. Add the required memory limit in the resources.limits.memory section.

     spec:
       services:
       - name: ibm-im-operator
         spec:
           secretwatcher:
             resources:
               limits:
                 memory: 220Mi
    
  5. Save and exit the custom resource.

    Wait for some time and then check the secret-watcher pod status. The status must show as 1/1 Running and the updated memory must show in the pod description.

  6. Get the secret-watcher pod status and name.

    oc -n <your-foundational-services-namespace> get pods | grep secret-watcher
    
  7. Check the pod description.

    oc -n <your-foundational-services-namespace> describe pod <secret-watcher-pod-name>