Guardium Data Security Center upgrade becomes stuck because CSV does not update

Symptoms

After upgrading Guardium Data Security Center Version 3.3.x (for example, from 3.3.3 to 3.3.4), the CSV does not update (for example, it does not update from CSV 3.3.3 to CSV 3.3.4).

The Guardium Data Security Center operator log includes this error:

{"level":"info","ts":1715419159.915022,"logger":"logging_event_handler","msg":"[playbook task start]","name":"ispchz72s4pxldinqk76dfz8","namespace":"is","gvk":"gi.ds.isc.ibm.com/v1, Kind=TenantMinisnif","event_type":"playbook_on_task_start","job":"6302081700133434776","EventData.Name":"gi-global-defaults : Test if version is supported in general"}
--------------------------- Ansible Task StdOut -------------------------------
TASK [Test if version is supported in general] ********************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Desired version: \"3.3.4\" is not supported. Operator supports Fresh Install at 3.3.3."}
-------------------------------------------------------------------------------
{"level":"error","ts":1715419160.0816762,"logger":"logging_event_handler","msg":"","name":"ispchz72s4pxldinqk76dfz8","namespace":"is","gvk":"gi.ds.isc.ibm.com/v1, Kind=TenantMinisnif","event_type":"runner_on_failed","job":"6302081700133434776","EventData.Task":"Test if version is supported in general","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/gi-global-defaults/tasks/main.yml:43","error":"[playbook task failed]","stacktrace":"github.com/operator-framework/operator-sdk/internal/ansible/events.loggingEventHandler.Handle\n\toperator-sdk/internal/ansible/events/log_events.go:111"}
----- Ansible Task Status Event StdOut (gi.ds.isc.ibm.com/v1, Kind=TenantMinisnif, ispchz72s4pxldinqk76dfz8/is) -----
PLAY RECAP *********************************************************************
localhost : ok=8 changed=0 unreachable=0 failed=1 skipped=7 rescued=0 ignored=0

Resolving the problem

  1. Determine the name of the job:
    oc project openshift-marketplace
    oc get job -oyaml | less

    In the output, locate guardium-insights-operator-bundle, get the job name, and delete it:

    oc delete job <name>
  2. Issue this command:
    oc project openshift-operator-lifecycle-manager
    and then restart all the pods in this project by running:
    oc delete $(oc get po -oname)
  3. Determine the name of the Guardium Data Security Center installplan:
    oc project sysqa
    oc get ip

    And then delete the installplan:

    oc delete ip <Guardium Insights installplan name>
  4. Determine the name of the Guardium Data Security Center operator deployment:
    oc get deployment | grep guardium-insights-operator

    And then delete the deployment:

    oc delete deployment <Guardium Insights operator deployment name>
  5. Save the existing subscription:
    oc get sub ibm-guardium-insights-operator-subscription -oyaml > SaveLocation
    oc delete sub ibm-guardium-insights-operator-subscription
    vi SaveLocation

    And then edit the subscription so that it looks like this:

    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: ibm-guardium-insights-operator-subscription
    spec:
      channel: v3.3
      installPlanApproval: Automatic
      name: ibm-guardium-insights-operator
      source: ibm-guardium-insights-operator-catalog
      sourceNamespace: openshift-marketplace

    And then apply the subscription:

    oc apply SaveLocation