Install strategy fails after upgrading OpenShift to 4.15.x

When you upgrade an OpenShift cluster to OpenShift version 4.15.x, the IBM Cloud Pak foundational services operator fails.

Symptom

After upgrading an OpenShift cluster to OpenShift version 4.15.x via the OpenShift console, the foundational services operator CSV fails with the following message:

install strategy failed: rolebindings.rbac.authorization.k8s.io "ibm-common-service-operator-service-auth-reader"`.

Cause

The issue is caused by the Operator Lifecycle Manager's (OLM) failure to appropriately set labels on the resources that are specified in the CSV during upgrade. Consequently, OLM fails to detect and manage these resources effectively, which results in installation failure for the operators.

Resolving the problem

To resolve this issue, complete the following steps:

  1. Use the following command to identify the affected resources by examining the failing ClusterServiceVersion (CSV):

    oc describe csv <the name of the foundational services CSV that fails> -n <operator-namespace> | tail -n 10

    Find the following message that identifies the affected resources in the output:

    message: 'install strategy failed: rolebindings.rbac.authorization.k8s.io "ibm-common-service-operator-service-auth-reader"

  2. Run the following command to delete the affected resource:

    oc delete rolebinding -n kube-system ibm-common-service-operator-service-auth-reader

  3. Check whether the resource is redeployed with the appropriate labels that indicate OLM management:

    $ oc get rolebinding -n kube-system ibm-common-service-operator-service-auth-reader -oyaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    olm.managed: "true"
    olm.owner: ibm-common-service-operator.v4.6.2
    olm.owner.kind: ClusterServiceVersion
    olm.owner.namespace: cs-operators
  operators.coreos.com/ibm-common-service-operator.cs-operators: ""
  ...

  4. Check the CSV again and verify the Deployment:

After you delete the affected resource, continuously review the CSV, until the CSV no longer reports any resources as already exists. The installation process is fully complete when the deployment becomes ready.

   $ oc get csv <failed ibm-common-service-operator-CSV> -n cs-operators -oyaml | yq '.status.conditions[].message'

   ...
   install strategy completed with no errors

For more information about how to resolve the issue, see After upgrading OpenShift to 4.15.x, install strategy failed: rolebindings.rbac.authorization.k8s.io already exists in Red Hat documentation. }