Online upgrade to IBM Concert Operate (GitOps method)

If you used GitOps to install a production deployment of IBM Cloud Pak for AIOps on a Red Hat OpenShift cluster, then you can use GitOps to upgrade your deployment by a maximum of one release at a time, and you can also upgrade directly from the final release of one version to the first release of the next version. For more information, see Upgrade paths.

• Ensure that you are on a version of Red Hat OpenShift that your current and target versions of both support. If you already have a qualifying version of Red Hat OpenShift but you want to upgrade it, then complete the IBM Cloud Pak for AIOps upgrade first. For more information, see Guidance for upgrades that require an Red Hat OpenShift upgrade.
• Backup and restore with IBM Fusion is not supported on Red Hat OpenShift 4.22 and later.
• Ensure that you are logged in to your Red Hat OpenShift cluster with oc login for any steps that use the Red Hat OpenShift command-line interface (CLI).
• The upgrade cannot be removed or rolled back.

Upgrade considerations

The following table gives an indication of the duration of the v5.1.0 upgrade relative to previous upgrades, and whether more resources are needed.

This upgrade includes major component updates that extend the upgrade duration beyond typical releases. The upgrade includes a major version upgrade of IBM Cloud Pak foundational services and a switch to a new Postgres operator, both of which require more time to complete.

Important: Ensure that your cluster has sufficient capacity before you upgrade. If your cluster does not have sufficient resources, the upgrade might fail or cause a degraded performance. For more information about the resource requirements for upgrades to 5.1.0, see

• Persistent storage sizing Significantly more disk space is needed, primarily for the addition of JanusGraph tablespaces for the Topology database.
• Hardware requirements - IBM Concert Operate More vCPU and memory are required for production deployments.

Follow these steps to upgrade your deployment.

1. Ensure cluster readiness
2. Suspend automatic synchronization
3. Update the target revision
4. Synchronize applications
5. Verify the deployment
6. (Optional) Re-enable automatic synchronization
7. Post upgrade actions

1. Ensure that your cluster still meets all of the prerequisites for deployment. For more information, see Planning.

2. Export an environment variable for your namespace.

   export PROJECT_CP4AIOPS=<namespace>

   Where <namespace> is the project (namespace) that your IBM Cloud Pak for AIOps installation is deployed in.

3. Verify whether your environment is correctly configured for the upgrade.

   1. Download and install v5.1.0 of the aiopsctl command line interface tool.

      AIOPSCTL_TAR=<aiopsctl_tar>
      AIOPSCTL_INSTALL_URL="https://github.com/IBM/aiopsctl/releases/download/v5.1.0/${AIOPSCTL_TAR}"
      curl -LO "${AIOPSCTL_INSTALL_URL}"
      tar xf "${AIOPSCTL_TAR}"
      mv aiopsctl /usr/local/bin/aiopsctl
      

      Where <aiopsctl_tar> is the operating system specific file that you require from the following set: aiopsctl-linux_s390x.tar.gz, aiopsctl-linux_arm64.tar.gz,aiopsctl-linux_amd64.tar.gz, aiopsctl-darwin_amd64.tar.gz, aiopsctl-darwin_arm64.tar.gz.

   2. Run the following command to run the precheck:

      aiopsctl server precheck -n ${PROJECT_CP4AIOPS}

      • If you have a multi-zone cluster, then also specify the -m flag. This flag enables extra checks to help ensure that the cluster has sufficient resources to withstand a zone outage, and that the zones are well balanced for memory and CPU. For example, aiopsctl server precheck -n cp4aiops -m.
      • If you are using hybrid storage, then also specify the --hybrid-storage flag. This flag enables extra checks to help ensure that sufficient local storage is configured. For example, aiopsctl server precheck -n cp4aiops --hybrid-storage
      Example output:

      # aiopsctl server precheck -n concert-operate
      o- [12 Jun 26 15:10 IST] Running precheck tool
        o- [12 Jun 26 15:10 IST] Checking hardware resources...
        Total Node Count (Available Schedulable / Required): 6/6
        Production (HA) Base CPU (vCPU): 93 / 146
        Production (HA) Base Memory (GB): 191 / 336
      
        Production (HA) Extended CPU (vCPU): 93 / 173
        Production (HA) Extended Memory (GB): 191 / 396
      
        Total Node Count (Available Schedulable / Required): 6/3
        Starter (Non-HA) Base CPU (vCPU): 93 / 47
        Starter (Non-HA) Base Memory (GB): 191 / 123
      
        Starter (Non-HA) Extended CPU (vCPU): 93 / 55
        Starter (Non-HA) Extended Memory (GB): 191 / 136
      
        You have enough resources for 1 instance(s) of small Base install
        You have enough resources for 1 instance(s) of small Extended install
      
        minimum hardware requirements met for starter size, but not production size
      
        o- [12 Jun 26 15:10 IST] Checking storage...
        Required StorageClasses found for provider Red Hat OpenShift Data Foundation: ocs-storagecluster-cephfs, ocs-storagecluster-ceph-rbd
        Checking if PVC can bind to supported storage class
      
        Verifying storage provider functionality: elapsed time [2s], estimated time [1m0s]
        Verifying storage provider functionality: elapsed time [5s], estimated time [1m0s]
        Storage check passed
      
        o- [12 Jun 26 15:10 IST] Checking OCP Version...
        Cluster meets OCP version requirements
      
        o- [12 Jun 26 15:10 IST] Checking if Cert Manager is present...
        Certificate CustomResourceDefinition Found
      
        o- [12 Jun 26 15:10 IST] Checking if certs will expire within 4 days...
      
        o- [12 Jun 26 15:10 IST] Precheck Summary Results
        Check                          Result
        Meets OCP Version Requirement  Passed
        Cert Mgr Operator Exists       Passed
        Certificates Valid             Passed
        Meets Hardware Requirements    Passed
        No Storage Issues              Passed

      Note:

      • The "You have enough resources for <...>" statements denote the number of instances that the cluster can support, and include any existing instances. The number that is given is for the number of base instances or extended instances. It does not mean that the cluster can support the stated number of base instances and the stated number of extended instances.
      • The tool might show inadequate resources in the Resource Summary because the script does not account for resources already being in use by the upgrading deployment. This can be ignored, as can the following message: [ FAIL ] Small or Large Profile Install Resources.

      The aiopsctl tool checks for expired and soon to expire certificates. If any certificates are identified as expired or expiring soon, they must be renewed. After the certificates are renewed, run aiopsctl precheck again to check the certificates. If you are using IBM Cloud Pak foundational services Cert Manager, you can use the cert-manager command line tool to renew the necessary certificates. Otherwise, follow the instructions for your chosen certificate manager.

4. Delete any evicted connector-orchestrator pods.

   1. Run the following command to check if there are any evicted connector-orchestrator pods.

      oc get pods -n ${PROJECT_CP4AIOPS} | grep connector-orchestrator
      

   2. Cleanup any evicted connector-orchestrator pods.

      If the previous command returned any pods with a STATUS of Evicted, then run the following command to delete each of them.

      oc delete pod -n ${PROJECT_CP4AIOPS} <connector_orchestrator>
      

      Where <connector_orchestrator> is a pod returned in the previous step.

If automatic synchronization is enabled for the Argo CD applications, then disable it.

1. In the Argo CD user interface (UI), select the Red Hat OpenShift application.
2. Click Application Settings for all of the IBM Cloud Pak for AIOps Argo CD applications
3. Disable Auto-Sync to prevent automatic reconciliation during the upgrade process.

1. Remove any stale PostgreSQL binding secret references. The commands are conditional and have no effect if the field is already absent.

   oc get issueresolutioncore aiops -n "${PROJECT_CP4AIOPS}" \
    -o jsonpath='{.spec.postgresql.bindingSecret}' | grep -q . && \
   oc patch issueresolutioncore aiops -n "${PROJECT_CP4AIOPS}" --type=json \
    -p '[{"op":"remove","path":"/spec/postgresql/bindingSecret"}]'
   
   oc get asm aiops-topology -n "${PROJECT_CP4AIOPS}" \
    -o jsonpath='{.spec.helmValuesASM.global\.postgres\.bindingSecret}' | grep -q . && \
   oc patch asm aiops-topology -n "${PROJECT_CP4AIOPS}" --type=json \
    -p '[{"op":"remove","path":"/spec/helmValuesASM/global.postgres.bindingSecret"}]'

2. Edit the Argo CD custom health check application manifest to update the spec.source.targetRevision to 5.1.0, and then save and apply your changes.

   Example excerpt (truncated):

   cat <<EOF | oc apply -f -
   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: argocd
     namespace: openshift-gitops
     labels:
       app.kubernetes.io/instance: argocd
     annotations:
       argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
   spec:
     destination:
       namespace: cp4aiops
       server: 'https://kubernetes.default.svc'
     source:
       repoURL: https://github.com/IBM/cp4aiops-gitops
       path: argocd
       targetRevision: 5.1.0
   <...>
   

3. Edit the Argo CD IBM Cert Manager application manifest to update the spec.source.targetRevision to 5.1.0, and then save and apply your changes.

   cat <<EOF | oc apply -f -
   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: ibm-cert-manager
     namespace: openshift-gitops
     labels:
       app.kubernetes.io/instance: ibm-cert-manager
     annotations:
       argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
   spec:
     destination:
       namespace: ibm-cert-manager
       server: 'https://kubernetes.default.svc'
     source:
       repoURL: https://github.com/IBM/cp4aiops-gitops
       path: cert-manager
       targetRevision: 5.1.0
   

4. Edit the Argo CD IBM Cloud Pak for AIOps application manifest to update the spec.source.targetRevision to 5.1.0, and then save and apply your changes.

   Example excerpt (truncated):

   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: ibm-cp4aiops
     namespace: openshift-gitops
     labels:
       app.kubernetes.io/instance: ibm-cp4aiops
     annotations:
       argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
   spec:
     destination:
       namespace: cp4aiops
       server: 'https://kubernetes.default.svc'
     source:
       repoURL: https://github.com/IBM/cp4aiops-gitops
       path: cp4aiops
       targetRevision: 5.1.0
   <...>
   

Argo CD detects the change to the application manifests, and marks the applications as OutOfSync.

1. In the Argo CD UI, review the diff to verify the changes that are to be applied when the IBM Cloud Pak for AIOps application is synchronized.

2. Click SYNC in the Argo CD UI to apply the changes and initiate the upgrade process.

   Initiate synchronization of the Argo CD applications in the following order: argocd, ibm-cert-manager, ibm-cp4aiops.

When all components are successfully deployed, the final state for each application in ArgoCD is Healthy and Synced. It takes around 60-90 minutes for the upgrade to complete (subject to the speed with which images can be pulled). You can monitor the ArgoCD app sync to see how the upgrade is progressing.

If you want to monitor the rollout of components more closely, use the following procedures.

If you disabled automatic synchronization, then you can update the sync policy to Automated in the Argo CD App Details view for the IBM Concert Operate application. In this mode, changes to the installation manifests in the source Git repository are automatically applied to the cluster. Any values that are defined by the valuesObject section are preserved and continue to override the values file.

1. If you previously set up backup or restore on your deployment, then you must follow the instructions in Upgrading backup and restore artifacts.

2. If the EXPIRY_SECONDS environment variable was set for configuring log anomaly alerts, the environment variable was not retained in the upgrade. After the upgrade is completed, set the environment variable again. For more information about setting the variable, see Configuring expiry time for log anomaly alerts.