Upgrading an Red Hat OpenShift cluster that is hosting IBM Cloud Pak for AIOps
When planned correctly and completed within supported version ranges, upgrades of Red Hat OpenShift Container Platform usually complete without disruption to IBM Cloud Pak for AIOps.
Overview
IBM Cloud Pak for AIOps has in-cluster high-availability, and uses PodDisruptionBudgets (PDBs) that are designed to allow Red Hat OpenShift upgrades to proceed without manual intervention. In typical production deployments, if Red Hat OpenShift reports that a node cannot be drained due to a PDB, this is usually because more processing and reconciliation time is needed or because of a problem with application health. Starter non-HA deployments of IBM Cloud Pak for AIOps have components that might use PDBs that permit zero disruptions, and manual intervention might be required to allow upgrades to progress.
A Red Hat OpenShift cluster that is running IBM Cloud Pak for AIOps takes longer to upgrade than an empty Red Hat OpenShift cluster due to the additional work required to evict and reschedule active workloads. This behavior is expected.
Pre-upgrade procedure
Before you start your Red Hat OpenShift upgrade, use the following steps to help ensure that the process completes with minimal to no downtime for IBM Cloud Pak for AIOps.
1. Check version compatibility
2. Check adequate compute resource headroom
3. Check adequate storage headroom
When you are ready to upgrade Red Hat OpenShift, use the instructions in the [Red Hat OpenShift documentation .
1. Check version compatibility
Check that your current IBM Cloud Pak for AIOps version is compatible with your target Red Hat OpenShift version.
Each Red Hat OpenShift release includes updates to the underlying Kubernetes API and Red Hat OpenShift services. To avoid compatibility issues and problems, your currently installed version of IBM Cloud Pak for AIOps must support the version of Red Hat OpenShift that you are upgrading to. For more information about the versions of Red Hat OpenShift that are supported by IBM Cloud Pak for AIOps, see Supported platforms: Red Hat OpenShift Container Platform.
If your currently installed version of IBM Cloud Pak for AIOps does not support the version of Red Hat OpenShift that you are upgrading to, then upgrade IBM Cloud Pak for AIOps to a version that does. You must upgrade IBM Cloud Pak for AIOps before you upgrade Red Hat OpenShift. For more information about upgrading Red Hat OpenShift and IBM Cloud Pak for AIOps, see Guidance for upgrades that require an Red Hat OpenShift upgrade.
2. Check adequate compute resource headroom
As part of the Red Hat OpenShift upgrade, the control plane nodes are updated. IBM Cloud Pak for AIOps operators use the Kubernetes API services that are hosted on the control plane nodes to manage the application workloads. If a problem occurs during this update, it might affect IBM Cloud Pak for AIOps's lifecycle management, but this is not expected.
- Cordoned (no new pods scheduled)
- Drained (running pods are evicted)
- Upgraded and restarted
- Returned to service
You must adhere to the cluster sizing guidance to help ensure that adequate cluster resources are available for workloads to run on without interruption while nodes are updated. For more information about sizing and hardware requirements, see Hardware requirements.
3. Check adequate storage headroom
The storage layer often needs to be upgraded alongside Red Hat OpenShift to maintain compliance with its own Red Hat OpenShift version compatibility requirements - especially if you are using Red Hat OpenShift Data Foundation.
If a problem occurs in the storage layer, IBM Cloud Pak for AIOps is impacted because of its dependency on the storage.
When each node that hosts storage is drained and reloaded during the Red Hat OpenShift upgrade, IBM Cloud Pak for AIOps temporarily runs in a degraded state because it does not have its full complement of storage nodes. If the storage is already at or near capacity, the node drainage can push the storage layer into a critical state where writes to it are disabled and IBM Cloud Pak for AIOps is impacted. To help avoid this occurring, ensure that adequate storage is available for the upgrade - ideally a storage utilization of 60% or less, and definitely less than 74%.
For more information about sizing and hardware requirements, see Storage requirements.
4. Verify application health
Verify the health of IBM Cloud Pak for AIOps components and confirm that no workloads are already degraded.
mustgather healthcheck tool to verify that your IBM Cloud Pak for AIOps deployment is in a healthy state before you commence your Red Hat OpenShift upgrade. For more information, see the following topics: