Upgrading IBM Netcool Operations Insight on Red Hat OpenShift with the oc ibm-pak plug-in and Container Application Software for Enterprises (CASE)

Edit online

Use these instructions to upgrade an existing Netcool® Operations Insight® deployment from version 1.6.12 or version 1.6.11 to 1.6.13, by using the Red Hat® OpenShift® Container Platform Operator Lifecycle Manager (OLM) user interface (UI) and Container Application Software for Enterprises (CASE). Before you upgrade, you must back up your deployment.

Before you begin

  • Note: Upgrading from OwnNamespace mode to SingleNamespace mode is not supported. The SingleNamespace mode is supported only for cloud and hybrid deployments that are installed with the oc-ibm_pak plug-in and portable compute or portable storage devices..
  • Ensure that you complete all the steps in Preparing your cluster. Most of these steps were completed as part of your previous Netcool Operations Insight deployment.
  • Ensure that you have an adequately sized cluster. For more information, see Sizing for a Netcool Operations Insight on Red Hat OpenShift deployment.
  • Configure persistent storage for your deployment. Only version 1.6.12 or version 1.6.11 deployments with persistence enabled are supported for upgrade to version 1.6.13.
  • Note: Before you upgrade to version 1.6.13, if present, remove the noi-root-ca secret by running the following command:
    oc delete secret noi-root-ca
  • Before you upgrade, save a backup copy of the cloud native analytics gateway configmap: ea-noi-layer-eanoigateway. For more information, see Preserving cloud native analytics gateway configmap customizations on upgrade.

All the necessary images for version 1.6.13 are either in the freely accessible operator repository (icr.io/cpopen), or in the IBM Entitled Registry (cp.icr.io) for which you need an entitlement key.

To upgrade from version 1.6.12 or 1.6.11 to version 1.6.13, complete the following steps.

Procedure

Set environment variables

  1. Run the following commands to set environment variables. 
    export CASE_NAME=$CASE_NAME
export CASE_VERSION=$CASE_VERSION
export TARGET_NAMESPACE=$TARGET_NAMESPACE
    Example:
    export CASE_NAME=ibm-netcool-prod
export CASE_VERSION=1.13.0
export TARGET_NAMESPACE=noi

Get the Netcool Operations Insight CASE.

  1. Download and install version 1.11.2 or later of IBM Catalog Management Plug-in for IBM Cloud® Paks from the IBM/ibm-pak-plugin external icon. Version 1.10.0 or earlier is also supported. Versions 1.11.0 and 1.11.1 are not supported. Extract the binary file by entering the following command: 
    tar -xf oc-ibm_pak-linux-amd64.tar.gz
  2. Run the following command to move the file to the /usr/local/bin directory. 
    mv oc-ibm_pak-linux-amd64 /usr/local/bin/oc-ibm_pak
    Note: If you are installing as a nonroot user, you must use sudo.
  3. Confirm that the oc ibm-pak is installed by running the following command: 
    oc ibm-pak --help

    Expected result: The plug-in usage is displayed.

  4. Download the Netcool Operations Insight CASE bundle (ibm-netcool-prod) to your Red Hat OpenShift Container Platform cluster. Run the following command. 
    oc ibm-pak get $CASE_NAME --version $CASE_VERSION
    Note: If you want to install the previous 1.6.12 version, specify --version 1.12.0 in the oc ibm-pak get command.
  5. Check that the CASE repository URL is pointing to the default https://github.com/IBM/cloud-pak/raw/master/repo/case/ location by running the oc ibm-pak config command.

    Example output:

    Repository Config

Name                        CASE Repo URL
----                        -------------
IBM Cloud-Pak Github Repo * https://github.com/IBM/cloud-pak/raw/master/repo/case/
    If the repository is not pointing to the default location (asterisk indicates default URL), then run the following command.
    oc ibm-pak config repo 'IBM Cloud-Pak Github Repo' --enable
    If the URL is not displayed, then add the repository by running the following command.
    oc ibm-pak config repo 'IBM Cloud-Pak Github Repo' --url https://github.com/IBM/cloud-pak/raw/master/repo/case/

Install the Netcool Operations Insight catalog and upgrade the operator.

  1. Install the catalog source and set the recursive flag and input directory. 
    oc ibm-pak launch \
$CASE_NAME \
--version $CASE_VERSION \
--namespace openshift-marketplace \
--inventory noiOperatorSetup \
--action install-catalog \
--args "--recursive --inputDir $HOME/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION"
  2. Upgrade the IBM Netcool Operations Insight operator by using CASE. Run the following command. 
    export CASE_INVENTORY_SETUP=noiOperatorSetup
oc ibm-pak launch \
$CASE_NAME \
--version $CASE_VERSION \
--namespace $NAMESPACE \
--inventory $CASE_INVENTORY_SETUP \
--action install-operator --args "--recursive --inputDir $IBMPAK_HOME/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION"
  3. Note: Complete this step if you are upgrading from version 1.6.11 to version 1.6.13.
    To avoid issues with CouchDB or Redis pods after upgrade, complete the following steps.
    If your deployment has more than one CouchDB replica, for example a production size deployment, scale the CouchDB statefulset to zero.
    oc scale sts <release-name>-couchdb --replicas=0
    Scale the Redis statefulset to zero.
    oc scale sts <release-name>-ibm-redis-server --replicas=0
  4. Upgrade the Netcool Operations Insight instance by going to the OLM UI. Go to Operators > Installed Operators and select your project. Then select IBM Cloud Pak for AIOps Event Manager.
  5. Note: Complete this step if you are upgrading from version 1.6.11.
    Go to the All instances tab and select your instance. Edit the YAML.
    • Update the spec.version value (from 1.6.11) to spec.version: 1.6.13.
    Note: To avoid an imagepullbackoff error after an upgrade, edit the cemformation. For more information, see Migration job does not complete.
  6. Note: Complete this step if you are upgrading from version 1.6.12.
    Go to the All instances tab and select your instance. Edit the YAML.
    1. Update the spec.version value (from 1.6.12) to spec.version: 1.6.13.
    2. Add the following settings for the helmValuesCEM section.
      spec:
  helmValuesCEM:
    commonimages.brokers.image.digest: sha256:d40731e355dd6a28cef6224f788b5456c94df7556eba1babc51ce5e9ac2784da
    commonimages.brokers.image.tag: 1.15.0-20240307T210317Z-multi-arch-L-KBDX-JKT9JU
    commonimages.cemusers.image.digest: sha256:8c6a4f119270029982cac2206aaa433118d3d0f04738bbf220abf73c2de21798
    commonimages.cemusers.image.tag: 1.15.0-20240307T210317Z-multi-arch-L-KBDX-JKT9JU
    commonimages.channelservices.image.digest: sha256:3b64430b32787ae03ec88b409e8fd720d619b214ff2c4c1947bf26300139ee1a
    commonimages.channelservices.image.tag: 1.15.0-20240307T210317Z-multi-arch-L-KBDX-JKT9JU
    commonimages.eventanalyticsui.image.digest: sha256:bee1274a8fe0d3015425bdd13c09d9faec26885f343da6720e17f7a5c37b27eb
    commonimages.eventanalyticsui.image.tag: 1.15.0-20240307T210322Z-multi-arch-L-KBDX-JKT9JU
    commonimages.eventpreprocessor.image.digest: sha256:0066b6d6b28b1c70552c86e4c13aa752f865608fba368057e724d8ce98820b68
    commonimages.eventpreprocessor.image.tag: 1.15.0-20240307T210317Z-multi-arch-L-KBDX-JKT9JU
    commonimages.incidentprocessor.image.digest: sha256:4ef66e6932b362c56d2a7fbe5e789a364ef8ef960c8100f3bf4a09fe11a58bb0
    commonimages.incidentprocessor.image.tag: 1.15.0-20240307T210317Z-multi-arch-L-KBDX-JKT9JU
    commonimages.notificationprocessor.image.digest: sha256:d875654db5b60fd9727bddca4c4da8539d2fb20c872fc83496e2acfeaa2d28e2
    commonimages.notificationprocessor.image.tag: 1.15.0-20240307T210317Z-multi-arch-L-KBDX-JKT9JU
    commonimages.integrationcontroller.image.digest: sha256:7f82bad08d6c51fff865f74551466027e36bc39cbd7bbed9481a134881063165
    commonimages.integrationcontroller.image.tag: 1.15.0-20240307T210317Z-multi-arch-L-KBDX-JKT9JU
    commonimages.normalizer.image.digest: sha256:af550c0e3cd2d2c239556c9175198e9f5a4561999f059841c79a98c838e3801b
    commonimages.normalizer.image.tag:   1.15.0-20240307T210317Z-multi-arch-L-KBDX-JKT9JU
    commonimages.rba.rbs.image.digest: sha256:2e82106c4be36dddc03132d8e74eec11cc5de1624d3bf73a3a12d2cf29cbe5d8
    commonimages.rba.rbs.image.tag: 1.34.0-20240305124329-ubi8-minimal-L-KBDX-JKT9JU
    commonimages.rba.as.image.digest: sha256:937c4ac77690670e1eee409ae514d7c880a66b7e70b1cf1c55091e64cdd6e506
    commonimages.rba.as.image.tag: 1.34.0-20240301170623-ubi8-minimal-L-KBDX-JKT9JU
      Wait for all pods, except for the CEM pods, to restart and all jobs to complete. Then, edit the YAML file and remove the helmValuesCEM section. The CEM pods will restart.
    Note: To avoid an imagepullbackoff error after an upgrade, edit the cemformation. For more information, see Migration job does not complete.
  7. Select Save.

What to do next

If the {{ .Release.Name }}-spark-shared-state PVC is included after the upgrade, you can delete it. This PVC is not used in Netcool Operations Insight 1.6.13 and onward.
  1. Check whether a shared spark PVC exists:
    oc get pvc | grep 'spark'
  2. Delete the {{ .Release.Name }}-spark-shared-state PVC resource if it is included:
    oc delete pvc {{ .Release.Name }}-spark-shared-state
    Where {{ .Release.Name }} is the release name of the PVC resource.

After you upgrade your cloud deployment, update the cloud native analytics gateway configmap. For more information, see Preserving cloud native analytics gateway configmap customizations on upgrade.

To enable or disable an observer, use the oc patch command, as in following example:
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/netDisco", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/aaionap", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/alm", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/ansibleawx", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/appdynamics", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/aws", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/azure", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/bigcloudfabric", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/bigfixinventory", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/cienablueplanet", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/ciscoaci", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/contrail", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/dns", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/docker", "value": 'true' }]'	
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/dynatrace", "value": 'true' }]'		
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/file", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/gitlab", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/googlecloud", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/hpnfvd", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/ibmcloud", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/itnm", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/jenkins", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/junipercso", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/kubernetes", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/newrelic", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/openstack", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/rancher", "value": 'true' }]'	
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/rest", "value": 'true' }]'						
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/sdconap", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/servicenow", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/sevone", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/taddm", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/viptela", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/vmvcenter", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/vmwarensx", "value": 'true' }]'
oc patch noi $noi_instance_name -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/topology/observers/zabbix", "value": 'true' }]'

If you don't want to roll back to a previous version, remove the nasm-elasticsearch and common-datarouting pods from your full cloud 1.6.13 deployment. For more information, see Postupgrade task.