Troubleshooting operator

• operator pod is not successfully deployed
• operator pod shows container restarts

operator pod is not successfully deployed

No operator pod exists in the ibm-spectrum-scale-operator namespace.

• Verify that all worker nodes in the Red Hat OpenShift Container Platform cluster are in a Ready state. Nodes that are NotReady might be preventing the operator pod from being scheduled.

   oc get nodes
  

   # oc get nodes
   NAME                  STATUS      ROLES    AGE   VERSION
   master0.example.com   Ready       master   65d   v1.18.3+6c42de8
   master1.example.com   Ready       master   65d   v1.18.3+6c42de8
   master2.example.com   Ready       master   65d   v1.18.3+6c42de8
   worker0.example.com   NotReady    worker   65d   v1.18.3+6c42de8
   worker1.example.com   NotReady    worker   65d   v1.18.3+6c42de8
   worker2.example.com   NotReady    worker   65d   v1.18.3+6c42de8
  

• Inspect the operator namespace and look for details that might point to problems.

  • Check the operator deployment: oc get deployment -n ibm-spectrum-scale-operator

    Describe the deployment for more details:

     oc describe deployment ibm-spectrum-scale-controller-manager -n ibm-spectrum-scale-operator
    

  • Check the operator replicaset: oc get replicasets -n ibm-spectrum-scale-operator

    Describe the replicaset for more details:

     oc describe replicaset $(oc get replicasets -n ibm-spectrum-scale-operator \
     -ojson | jq -r .items[0].metadata.name) -n ibm-spectrum-scale-operator
    

operator pod shows container restarts

Kubernetes keeps the logs of the current container and the previous container.

Look at the previous container logs and look for any issues that might be causing the container to restart.

oc logs -p $(oc get pods -n ibm-spectrum-scale-operator -ojson | \
jq -r .items[0].metadata.name) -n ibm-spectrum-scale-operator