Rolling back a hybrid deployment

Edit online

For on-premises components of your hybrid deployment, the Hybrid Integration Kit does not support rollback. You must uninstall the latest versions of your on-premises components and reinstall the older versions of the components. For IBM® Netcool® Operations Insight® on Red Hat® OpenShift®, learn how to roll back from version 1.6.13 to version 1.6.12 or version 1.6.11. Roll back with the Red Hat OpenShift Container Platform Operator Lifecycle Manager (OLM) user interface (UI), or the command line.

Rolling back to version 1.6.12 or 1.6.11

Roll back to version 1.6.12 or 1.6.11 from the command line or from the OLM UI.

Scale down the inventory service.

Find the inventory pod and note its replication level.

oc get pods --field-selector=status.phase=Running --no-headers=true --output=custom-columns=CNAME:.metadata.ownerReferences[0].name | grep ${release}-topology-inventory | uniq --count

Scale down the inventory pod.

oc scale deployment --replicas=0 ${release}-topology-inventory

Clear the inventory database.

Find and store the inventory database administrator username.

oc get secret $release-topology-postgres-admin -o jsonpath='{.data.username}' | base64 -d

Find and store the inventory database user password.

oc get secret $release-topology-postgres-admin -o jsonpath='{.data.password}' | base64 -d

Find and store the inventory database name.

oc get asmformation $release-topology -o jsonpath='{.spec.helmValues.global.postgres.dbname}'

Find and store the inventory schema name.

oc get asmformation $release-topology -o jsonpath='{.spec.helmValues.global.postgres.schema}'

Run the following command to determine which is the primary pod in the database cluster.
```
oc get cluster -o custom-columns=:.status.currentPrimary
```
Clear the inventory database data. Replace the variables in the following example with the values that you gathered in previous steps.
```
oc exec -it <database_cluster_primary_pod> -- /bin/bash -c 'psql --host localhost --username <database_user> --dbname <database_name> --command "DROP SCHEMA IF EXISTS <database_schema> CASCADE;"'
```
You are prompted to enter the inventory database username and password.

To avoid issues with CouchDB or Redis pods after roll back, complete the following steps.
If your deployment has more than one CouchDB replica, for example a production size deployment, scale the CouchDB statefulset to zero.
```
oc scale sts <release-name>-couchdb --replicas=0
```
Scale the Redis statefulset to zero.
```
oc scale sts <release-name>-ibm-redis-server --replicas=0
```
Roll back to version 1.6.12 or 1.6.11.
- Command line: To roll back from the command line, use the oc edit noihybrid command and change the version back to version 1.6.12 or 1.6.11.Save changes to the noihybrid instance. The pods restart.
- OLM UI: To roll back from the OLM UI, go to Operators > Installed Operators > IBM Cloud Pak for AIOps Event Manager. Then, select the NOI tab. Select your deployment and then select the YAML tab. Change the version back to version 1.6.12 or 1.6.11 and save the changes.
The inventory service scales up automatically.

If you rolled back to version 1.6.12, locate the ea-datalayer deployment.

oc get deploy | grep
    ea-datalayer

Delete the ea-datalayer deployment.

oc delete deploy $(oc get deploy | grep ea-datalayer | awk '{ print $1 }'

(Optional) If you rolled back to version 1.6.11, restart the cem-operator pod to ensure that it runs the correct image digests for version 1.6.11.
```
oc delete <release_name>-cem-operator
```
Rebroadcast data to the inventory service. If data in inventory is out of sync with data in the Cassandra database, resynchronize it by calling the rebroadcast API of the topology service. This triggers the rebroadcast of all known resources on Kafka, and the inventory service will then index those resources in PostgreSQL. Call the rebroadcast crawler by calling the rebroadcast API of the Topology service, specifying a tenantId:
```
https://master_fqdn/1.0/topology/swagger#!/Crawlers/rebroadcastTopology
```
The rebroadcast crawler can also be run from the Data Administration Routines UI. Log in to the Administration > Topology Configuration UI and select the Data administration routines tile. For more information about running routines, see Running data administration routines in the IBM Agile Service Manager documentation.

What to do next

After a roll-back, decommission the workloads that were deployed as part of version 1.6.13.

Delete any deployments belonging to aidl (ea-datalayer).

oc delete deploy  -l app.kubernetes.io/\component=aidl

Delete the AIDL curator cronjob.

oc delete cronjob -l app.kubernetes.io/\component=aidl-curator

Delete the elasticmigration job, so that it can be re-attempted when you want to upgrade.
```
oc delete job     -l app.kubernetes.io/\component=elasticmigration
```