Offline upgrade of IBM Cloud Pak for AIOps

Overview

This procedure is for deployments of IBM Cloud Pak for AIOps on Red Hat® OpenShift® Container Platform. This procedure can be used on an offline deployment of IBM Cloud Pak for AIOps 4.7.0 or later, and can still be used if the deployment has had hotfixes applied.

You cannot use these instructions to upgrade deployments of IBM Cloud Pak for AIOps 4.6.1 or earlier. For more information, see Upgrade paths.

If you have an online deployment of IBM Cloud Pak for AIOps on Red Hat OpenShift, follow the instructions in Upgrading IBM Cloud Pak for AIOps (online).

If you have a deployment of IBM Cloud Pak for AIOps on Linux®, follow the instructions in Offline upgrade of IBM Cloud Pak for AIOps on Linux.

Important: If IBM® Netcool® Operations Insight® is deployed on the same cluster as IBM Cloud Pak for AIOps, ensure that Netcool Operations Insight is at version 1.6.13 or later before you upgrade IBM Cloud Pak for AIOps. Failure to do so may result in a broken Netcool Operations Insight deployment, as the IBM Cloud Pak for AIOps upgrade process updates a shared component that is used by both applications.

Before you begin

Notes:

• Ensure that you are on a version of Red Hat OpenShift that your current and target versions of IBM Cloud Pak for AIOps both support. If you already have a qualifying version of Red Hat OpenShift but you want to upgrade it, then complete the IBM Cloud Pak for AIOps upgrade first. For more information, see Guidance for upgrades that require an Red Hat OpenShift upgrade.
• Ensure that you are logged in to your Red Hat® OpenShift® Container Platform cluster with oc login for any steps that use the OpenShift command-line interface (CLI).
• Red Hat OpenShift Container Platform requires a user with cluster-admin privileges for the following operations:
  • Mirror images
  • Update the catalog
  • Update the operator subscription

Warnings:

• Custom patches, labels, and manual adjustments to IBM Cloud Pak for AIOps resources are lost when IBM Cloud Pak for AIOps is upgraded, and must be manually reapplied after upgrade. For more information, see Manual adjustments are not persisted.
• The upgrade cannot be removed or rolled back.
• If you previously increased the size of a PVC directly, then you must follow the correct procedure that is supplied in Resizing storage to ensure that the size is updated by the operator. Failure to do so before upgrading IBM Cloud Pak for AIOps causes the operator to attempt to restore a lower default value for the PVC, and causes an error in your IBM Cloud Pak for AIOps deployment.

Database migration during upgrade

The IBM Cloud Pak for AIOps 4.7.x to v4.8 upgrade process migrates the Elasticsearch database to an OpenSearch database.

Extra resources are needed during the upgrade, and are shown in the following table. These resources can be released after the upgrade is complete.

If IBM Sales representatives and Business Partners supplied you with a custom profile, then you must calculate the additional resource requirements as follows:

1. Run the following commands to view your custom profile ConfigMap.

   export PROJECT_CP4AIOPS=<project>
   oc get configmap -n "${PROJECT_CP4AIOPS}" $(oc get installation -n "${PROJECT_CP4AIOPS}" -o jsonpath='{.items[0].status.customProfileConfigmap}') -o yaml
   

   Where <project> is the namespace (project) that IBM Cloud Pak for AIOps is deployed in.

2. Locate the section for elasticsearch in the output, and calculate the additional resources that are needed for the duration of upgrade.

   The additional vCPU needed is replicas multiplied by requests.cpu, and the additional memory needed is replicas multiplied bu requests.memory.

   Example output for an elasticsearch section, where upgrade will need 15 additional vCPU (5 x 3) and 40960 Mi memory (5 x 8192).

   elasticsearch:
        replicas: 5
        resources:
          requests:
            cpu: 3000m
            memory: 8192Mi
          limits:
            cpu: 6000m
            memory: 8192Mi
   

   
   

The migration causes approximately 10 minutes of downtime for the following IBM Cloud Pak for AIOps components: log integrations, ticket integrations, log anomaly model training, and the Insights Dashboard. If alert archiving to Elasticsearch is enabled on your deployment, then the downtime for the migration might be extended depending on the quantity of alerts stored.

During upgrade, you can run the following command to monitor the status of the migration:

oc describe installation -n ${PROJECT_CP4AIOPS}

Example output for a migration in progress:

Conditions:
    Last Transition Time:            2024-10-29T16:58:55Z
    Message:                         Foreground Elasticsearch migration in progress
    Observed Generation:             2
    Reason:                          ForegroundMigrationInProgress
    Status:                          True
    Type:                            MigratingElastic

When migration is complete, Message has a value of Elasticsearch migration completed.

Note: Some integrations might be marked as Not Running while the migration is in progress. This is expected behavior, and the integrations must not be altered until the migration is complete.

After all services resume function, historic log data is migrated in the background. Historic log data is not accessible for training new models while this migration occurs. The duration of this background migration depends on the quantity of historic log data stored.

Upgrade procedure

Follow these steps to upgrade your offline IBM Cloud Pak for AIOps deployment.

1. Ensure cluster readiness
2. Download CASE files
3. Mirror images
4. Update the catalog
5. Update the operator subscription
6. Verify the deployment
7. Post upgrade actions

1. Ensure cluster readiness

Recommended: Take a backup before upgrading. For more information, see Backup and restore.

1. Ensure that your cluster still meets all of the prerequisites for an air-gapped deployment.

   Ensure that your cluster has the additional resources that are required for upgrade, as outlined in Database migration during upgrade.

   Review the steps for your installation approach:

   • Bastion host: Prerequisites
   • Portable device: Prerequisites

2. Ensure that the IBM Catalog Management Plug-in for IBM Cloud Pak® (ibm-pak-plugin) is at the required level.

   Important: ibm-pak-plugin v1.9 and lower cannot process the IBM Cloud Pak for AIOps 4.8.0 catalog. You must ensure that you have ibm-pak-plugin v1.10 or higher installed.

   1. Check which version of ibm-pak-plugin you have installed.

      Run the following command on your bastion host, portable compute device, or connected compute device if you are using a portable storage device.

      oc ibm-pak --version
      

      Example output:

      oc ibm-pak --version
      v1.13.0
      

      
      

   2. If the ibm-pak-plugin version is lower than v1.10.0, then you must download and install the most recent version.

      Follow the steps for your installation approach:

      • Bastion host: Install the IBM Catalog Management Plug-in for IBM Cloud Pak®.

      • Portable device: Install the IBM Catalog Management Plug-in for IBM Cloud Pak®.

3. Download scripts.

   1. Download the prerequisite checker script and copy it to your air-gapped environment.

      For more information about the script, including how to download and run it, see github.com/IBM .

   2. Download the IBM Cloud Pak for AIOps uninstall script, and copy it to your air-gapped environment.

      For more information about the script, including how to download and run it, see github.com/IBM .

   3. (Optional) Download the status checker script, and copy it to your air-gapped environment.

      For more information about the script, including how to download and run it, see github.com/IBM. The status checker script can be used in step 9. Verify the deployment to give information about the status of your deployment. The use of this script is optional, as status can be found directly from the ibm-aiops-orchestrator custom resource.

4. If you still have waiops_var.sh from when you installed IBM Cloud Pak for AIOps, then run the following command from the directory that the script is in, to set the environment variables that are used later.

   . ./waiops_var.sh
   

   If you do not have waiops_var.sh, then run the following commands to set the environment variables that you need for upgrade.

   export PROJECT_CP4AIOPS=<project>
   export INSTALL_MODE_NAMESPACE=<install_namespace>
   export CATALOG_SRC_CP4AIOPS=ibm-aiops-catalog
   

   Where

   • <project> is the namespace (project) that your IBM Cloud Pak for AIOps subscription is deployed in.
   • <install_namespace> is ${PROJECT_CP4AIOPS} if your deployment is namespace scoped, or openshift-operators if your deployment has a cluster-wide scope.

5. Run the IBM Cloud Pak for AIOps prerequisite checker script.

   Run the prerequisite checker script to ensure that your Red Hat OpenShift Container Platform cluster is correctly set up for an IBM Cloud Pak for AIOps upgrade.

   Download the prerequisite checker script from github.com/IBM , and run it with the following command:

   ./prereq.sh -n ${PROJECT_CP4AIOPS} --ignore-allocated
   

   Important: If you are installing on a multi-zone cluster, then also specify the -m flag to assess whether there are sufficient resources to withstand a zone outage.

   Note: The prerequisite checker script might show inadequate resources in the Resource Summary because the script does not account for resources already being in use by the upgrading deployment. This can be ignored, as can the following message: [ FAIL ] Small or Large Profile Install Resources.

   Example output:

   # ./prereq.sh -n cp4aiops --ignore-allocated
   
   [INFO] Starting IBM Cloud Pak for AIOps prerequisite checker v4.8...
   
   CLI: oc
   
   [INFO] =================================Platform Version Check=================================
   [INFO] Checking Platform Type....
   [INFO] You are using Openshift Container Platform
   [INFO] OCP version 4.16.7 is compatible but only nodes with AMD64 architectures are supported at this time. 
   [INFO] =================================Platform Version Check=================================
   
   [INFO] =================================Storage Provider=================================
   [INFO] Checking storage providers
   [INFO] No IBM Storage Fusion Found... Skipping configuration check.
   
   [INFO] No Portworx StorageClusters found with "Running" or "Online" status. Skipping configuration check for Portworx.
   [INFO] Openshift Data Foundation found.
   [INFO] No IBM Cloud Storage found... Skipping configuration check for IBM Cloud Storage Check.
   
   Checking Openshift Data Foundation Configuration...
   Verifying if Red Hat Openshift Data Foundation pods are in "Running" or "Completed" status
   [INFO] Pods in openshift-storage project are "Running" or "Completed"
   [INFO] ocs-storagecluster-ceph-rbd exists.
   [INFO] ocs-storagecluster-cephfs exists.
   [INFO] No warnings or failures found when checking for Storage Providers.
   [INFO] =================================Storage Provider=================================
   
   [INFO] =================================Cert Manager Check=================================
   [INFO] Checking for Cert Manager operator
   
   [INFO] Successfully functioning cert-manager found.
   
   CLUSTERSERVICEVERSION             NAMESPACE
   ibm-cert-manager-operator.v4.2.8  ibm-cert-manager
   
   [INFO] =================================Cert Manager Check=================================
   
   [INFO] =================================Licensing Service Operator Check=================================
   [INFO] Checking for Licensing Service operator
   
   [INFO] Successfully functioning licensing service operator found.
   
   CLUSTERSERVICEVERSION          NAMESPACE
   ibm-licensing-operator.v4.2.8  ibm-licensing
   
   [INFO] =================================Licensing Service Operator Check=================================
   
   [INFO] =================================Starter or Production Install Resources=================================
   [INFO] Checking for cluster resources
   
   [INFO] ==================================Resource Summary=====================================================
   [INFO]                                                     Nodes     |     vCPU       |  Memory(GB)
   [INFO] Starter (Non-HA) Base (available/required)       [  9 / 3 ]   [  144 / 47 ]    [  289 / 123 ]
   [INFO]     (+ Log Anomaly Detection & Ticket Analysis)  [  9 / 3 ]   [  144 / 55 ]    [  289 / 136 ]
   
   [INFO] Production (HA) Base (available/required)        [  9 / 6 ]   [  144 / 136 ]   [  289 / 310 ]
   [INFO]     (+ Log Anomaly Detection & Ticket Analysis)  [  9 / 6 ]   [  144 / 162 ]   [  289 / 368 ]
   [INFO] ==================================Resource Summary=====================================================
   [INFO] Cluster currently has resources available to create a Starter (Non-HA) install of Cloud Pak for AIOps
   
   [INFO] =================================Prerequisite Checker Tool Summary=================================
         [  PASS  ] Platform Version Check 
         [  PASS  ] Storage Provider
         [  PASS  ] Starter (Non-HA) Base Install Resources
         [  FAIL  ] Production (HA) Base Install Resources
         [  PASS  ] Cert Manager Operator Installed
         [  PASS  ] Licensing Service Operator Installed
   [INFO] =================================Prerequisite Checker Tool Summary=================================
   

   
   

6. Delete any evicted connector-orchestrator pods.

   1. Run the following command to check if there are any evicted connector-orchestrator pods.

      oc get pods -n ${PROJECT_CP4AIOPS} | grep connector-orchestrator
      

   2. Cleanup any evicted connector-orchestrator pods.

      If the previous command returned any pods with a STATUS of Evicted, then run the following command to delete each of them.

      oc delete pod -n ${PROJECT_CP4AIOPS} <connector_orchestrator>
      

      Where <connector_orchestrator> is a pod returned in the previous step.

2. Download CASE files

On your OpenShift cluster, rerun step 2 of the air-gap installation procedure Download CASE files to download the latest CASE files.

Follow the steps for your installation approach:

• Bastion host: Download the CASE

• Portable device: Download the CASE

3. Mirror images

Rerun step 3 of the air-gap installation procedure to mirror the updated images to the offline registry.

Follow the steps for your installation approach:

• Bastion host: Mirror images

• Portable device: Mirror images

4. Update the catalog

1. Rerun step 5.2 of the air-gap installation procedure Create the catalog source to update your catalog source.

   Follow the steps for your installation approach:

   • Bastion host: Create the catalog source

   • Portable device: Create the catalog source

2. From IBM Cloud Pak for AIOps 4.8.0, IBM Cloud Pak® foundational services Cert Manager and License Service are deployed using their own independent catalogs, and a new catalog called ibm-aiops-catalog is used for the remaining IBM Cloud Pak for AIOps operators. You must migrate your subscriptions to use the new catalogs.

   Run the following script from the command line:

   # New catalog name to migrate to
   target="ibm-aiops-catalog" 
   
   # Previous catalog names to migrate from 
   catalogs="""aiops-analytics-operator-catalog
   aiops-lifecycle-operator-catalog
   cloud-native-postgresql-catalog
   ibm-aiops-appconnect-catalog
   ibm-aiops-ir-core-operators
   ibm-aiops-opencontent-flink-catalog
   ibm-asm-catalog ibm-cp-waiops-catalog
   ibm-cp-waiops-elastic-catalog
   ibm-cp-waiops-flink-catalog
   ibm-aiopsedge-case
   ibm-redis-cp-operator-catalog
   ibm-secure-tunnel-operator-catalog
   ibm-watson-ai-manager-catalog
   ibm-watson-aiops-ui-catalog
   opencloud-operators""" 
   
   echo "Getting all subscriptions..."
   all_subs=$(oc get subscriptions.operators.coreos.com --no-headers -n ${PROJECT_CP4AIOPS})
   
   for catalog in $catalogs; do
     subs=$(echo "$all_subs" | awk -v c=$catalog '$3 == c {print $1}' -)
     if [[ -n $subs ]]; then
       echo "Updating catalog source for subscription $subs from $catalog to $target"
       oc patch subscriptions.operators.coreos.com $subs -n ${PROJECT_CP4AIOPS} --type=json --patch='[{"op": "replace", "path": "/spec/source", "value": '$target'}]'
     fi
   
     oc delete catalogsource $catalog -n openshift-marketplace
   done
   

5. Update the operator subscription

1. Run the following command to update the IBM Cloud Pak for AIOps subscription to use the new channel, v4.8.

   oc patch subscription.operators.coreos.com ibm-aiops-orchestrator -n ${INSTALL_MODE_NAMESPACE} --type=json -p='[{'op': 'replace', 'path': '/spec/channel', 'value': 'v4.8'}]'
   

2. If you are installing in AllNamespaces mode, run the following command to refresh the connectors' secret:

   oc delete secret cp4waiops-connectors-deploy-cert-secret -n "${PROJECT_CP4AIOPS}"
   

   For more information about installation modes, see Operator installation mode.

6. Verify the deployment

oc get installations.orchestrator.aiops.ibm.com -n ${PROJECT_CP4AIOPS}

NAME           PHASE     LICENSE    STORAGECLASS   STORAGECLASSLARGEBLOCK   AGE
ibm-cp-aiops   Updating  Accepted   rook-cephfs    rook-ceph-block          3m

oc get installation.orchestrator.aiops.ibm.com -o yaml -n ${PROJECT_CP4AIOPS} | grep 'Not Ready'

lifecycleservice: Not Ready
zenservice: Not Ready

oc get <component> -o yaml -n ${PROJECT_CP4AIOPS}

oc get csv -l operators.coreos.com/ibm-aiops-orchestrator.${INSTALL_MODE_NAMESPACE} -n ${INSTALL_MODE_NAMESPACE}

oc get csv -l operators.coreos.com/ibm-aiops-orchestrator.cp4aiops -n cp4aiops
NAME                           DISPLAY                  VERSION  REPLACES                       PHASE
ibm-aiops-orchestrator.v4.8.0  IBM Cloud Pak for AIOps  4.8.0    ibm-aiops-orchestrator.v4.7.1  Succeeded

7. Post upgrade actions

1. Delete the persistent volumes that were used for storing data in Elasticsearch, the database that was migrated from.

   Run the following command:

   oc delete pvc -l app.kubernetes.io/managed-by=ibm-elastic-operator -n ${PROJECT_CP4AIOPS}
   

2. If you previously set up backup or restore on your deployment, then you must follow the instructions in Upgrading IBM Cloud Pak for AIOps backup and restore artifacts.

3. If the EXPIRY_SECONDS environment variable was set for configuring log anomaly alerts, the environment variable was not retained in the upgrade. After the upgrade is completed, set the environment variable again. For more information about setting the variable, see Configuring expiry time for log anomaly alerts.

4. If you have a metric integration configured that stops working after upgrade, then you must follow the instructions in After upgrade, a metric integration goes into a failed state.