Operator installation or upgrade fails with exceeded progress deadline error

The IBM Cloud Pak foundational services operator ClusterServiceVersion (CSV) status shows Failed and the ibm-common-service-operator deployment shows exceeded its progress deadline error.

Symptom

The foundational services operator CSV status shows as Failed. On the Red Hat® OpenShift® Container Platform cluster console, you see an error message similar to the following message:

install failed: deployment ibm-common-service-operator not ready before timeout: deployment "ibm-common-service-operator" exceeded its progress deadline

• When you check the operator pod by using the command-line interface (CLI), you see that the pod is 0/1 Running.

    oc get pods -n <operator-namespace> | grep ibm-common-service-operator
  

  Following is a sample output:

    ibm-common-service-operator-67f4bf777f-299db                      0/1     Running     1 (117s ago)   14m
  

• When you check the operator pod logs by using the CLI, you see an error message similar to the following message:

    oc logs <ibm-common-service-operator-xxxx> -n <operator-namespace>
  

  Following is a sample output:

    E0418 13:39:02.224359 1 main.go:149] Failed to verify cluster type unable to get: cpd-instance/ibm-cpp-config because of unknown namespace for the cache
  

Cause

The environment variable is not updated in operator CSV or propagated from operator CSV to its deployment.

The expected value of environment variable WATCH_NAMESPACE is properly defined in the namespace-scope configmap. However, the WATCH_NAMESPACE value that is in the operator CSV and deployment is still from olm.targetNamespaces annotation.

Root Cause 1

NamespaceScope operator fails to update the WATCH_NAMESPACE environment variable in operator CSV.

Root Cause 2

Operator Lifecycle Manager (OLM) fails to propagate the environment variable value from operator CSV to its deployment.

Resolution

Complete the following steps to resolve the issue:

1. Get the ibm-common-service-operator CSV name.

    oc get csv -n <operator-namespace> | grep ibm-common-service-operator | awk '{print $1}'
   

   See the following sample output:

    ibm-common-service-operator.v4.1.0
   

2. Identify the value of environment variable WATCH_NAMESPACE in the ibm-common-service-operator CSV.

    oc get csv <ibm-common-service-operator-CSV> -n <operator-namespace> -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env[?(@.name=="WATCH_NAMESPACE")].valueFrom}'
   

   • If the command returns the following sample output, continue to step 3.

       {"fieldRef":{"fieldPath":"metadata.annotations['olm.targetNamespaces']"}}
     

   • If the command returns the following sample output, you can skip step 3, 4, and 5.

       {"configMapKeyRef":{"key":"namespaces","name":"namespace-scope","optional":true}}
     

3. Identify the number of ibm-common-service-operator CSVs.

    oc get csv -n <operator-namespace> | grep ibm-common-service-operator
   

   See the following sample output:

    ibm-common-service-operator.v4.1.0            IBM Cloud Pak foundational services    4.1.0                                                   Installing
   

4. If there is more than one ibm-common-service-operator CSV, delete all old versioned ibm-common-service-operator CSVs.

    oc delete csv <old-ibm-common-serivce-operator-csv> -n <operator-namespace>
   

5. Verify that the WATCH_NAMESPACE environment variable value in the newer versioned CSV is obtained from the namespace-scope configmap.

    oc get csv <new-ibm-common-service-operator-CSV> -n <operator-namespace> -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env[?(@.name=="WATCH_NAMESPACE")].valueFrom}'
   
    {"configMapKeyRef":{"key":"namespaces","name":"namespace-scope","optional":true}}
   

6. Delete the ibm-common-service-operator deployment.

    oc delete deployment ibm-common-service-operator -n <operator-namespace>
   

7. Wait for OLM to re-create the new ibm-common-service-operator deployment.

    oc get deployment ibm-common-service-operator -n <operator-namespace>
   

8. Verify that the WATCH_NAMESPACE value in the newly generated deployment is obtained from the namespace-scope configmap.

   oc get deployment ibm-common-service-operator -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="WATCH_NAMESPACE")].valueFrom}' -n <operator-namespace>
   
   {"configMapKeyRef":{"key":"namespaces","name":"namespace-scope","optional":true}}#