Rolling back Netcool Operations Insight on Red Hat OpenShift

Use these instructions to roll back from 1.6.7 of Netcool® Operations Insight® to a previously deployed 1.6.6, using the Red Hat® OpenShift® Operator Lifecycle Manager (OLM) user interface (UI), or the command line.

Before you begin

If you are attempting to roll back a failed upgrade, then the redis pods may become stuck. If this occurs, then manually restart the redis pods with the following command oc delete pod redis*.

Procedure

  1. If Netcool Operations Insight 1.6.6 was upgraded to version 1.6.7 using airgap, then, before you rollback from version 1.6.7 to 1.6.6, you must set the image registry back to the 1.6.6 Docker image repository. Otherwise the cem-operator and asm-operator pods will fail with ImagePullError errors.
    1. Edit the noi-operator deployment, and find the key-value pair for OPERATOR_REPO. The value of this is set to the 1.6.7 airgap target registry. Replace this value with the Netcool Operations Insight 1.6.6 image registry where the Netcool Operations Insight 1.6.6 Passport Advantage® (PPA) package is uploaded, for example image-registry.openshift-image-registry.svc:5000/<namespace>.
      oc edit deploy noi-operator
    2. If the Netcool Operations Insight 1.6.6 image registry is authenticated and requires a pull secret, then edit the noi-operator serviceaccount and add this secret in the imagePullSecrets section.
      oc edit serviceaccount noi-operator
  2. Rollback can be performed from the command line and from the OLM UI. The process of rolling back involves downgrading the operator.
    1. To rollback from the command line, use: oc edit noi and change the version back to version 1.6.6. To rollback from the OLM UI, go to Operators > Installed Operators > NOI and then select the Cloud Deployment tab if your deployment is only on Red Hat OpenShift, or the Hybrid Deployment tab if you have a hybrid deployment that is on Red Hat OpenShift and on-premises. Select Edit NOI and then the YAML tab. Change the version back to version 1.6.6 and save the changes.
    2. Downgrade the operator to 1.6.6 by running the following commands.
      oc get csv | grep noi
      The noi.v1.7.0 CSV is returned.
      oc delete csv noi.v1.7.0
      Get the subscription.
      oc get subscription
      The noi subscription is returned.
      NAME   PACKAGE   SOURCE              CHANNEL
      noi    noi       noi-catalogsource   v1.11
      
      Delete the subscription.
      oc delete subscription noi
    3. Reinstall the operator. Go to Operators > Operator Hub. select IBM® Cloud Pak for Watson™ AIOps Event Manager and click Install. Verify that the operator restarts.
  3. Verify that the cron job is recreated by running the oc get cronjob command.
    Example output:
    NAME                                           SCHEDULE        SUSPEND   ACTIVE   LAST SCHEDULE   AGE
    noi-curator-pattern-metrics                0 0 * * *       False     0        <none>          22m
    noi-ibm-hdm-analytics-dev-policy-cleanup   0 0/12 * * *    False     0        <none>          22m
    noi-register-cnea-mgmt-artifact            1,*/5 * * * *   False     0        2m24s           22m
  4. Obtain the metrics deployment by running the following command:
    oc get deployment | grep -i metric
  5. Delete the minio stateful set by running the following command:
    oc get sts | grep <release name>-ibm-minio
    oc delete sts <minio sts name>