IBM Cloud Pak for Watson AIOps Hot Fix for Log4j CVE-2021-44228, CVE-2021-45046, CVE-2021-45105

Content

 This package contains a shell script that applies a hotfix to a deployed and configured instance of IBM Cloud Pak® for Watson AIOps AIManager v3.2.0.  Use the following instructions to apply the hotfix.  An instance of AI Manager is deployed if you completed a starter or production installation as described in IBM Documentation and the following components are configured:

• A Red Hat® OpenShift® Container Platform (OCP) cluster is configured to use IBM Cloud Pak for Watson AIOps AI Manager.
• The IBM Cloud Pak for Watson AIOps AI Manager Operator is installed to the OCP cluster, in either All Namespaces or Single Namespace mode.
• An AI Manager instance, an installation.orchestrator.aiops.ibm.com custom resource definition (CRD), is created in a project on the OCP cluster.

For more information about the installation procedure, see:

• Starter installation
• Production installation

A hotfix is a temporary fix that is intended to modify a configured AI Manager instance until a fix pack containing the permanent fix is available to replace the hotfix. Each new AI Manager instance must be modified with the hotfix, even if that AI Manager instance was created after the hotfix was applied to another AI Manager instance on the same OCP cluster.

Several modifications to the AI Manager instance are necessary. A shell script named apply-fix.sh is provided here to perform the modifications. Follow these directions to apply this hotfix successfully.

Prerequisites

To apply this hotfix, the same software requirements for installing an AI Manager instance still apply.  In addition, if you installed AI Manager in an air-gapped (offline) environment, ensure that your cluster still meets all of the prerequisites that are listed in the documentation, regardless of your chosen installation method. This is required in order for your mirroring environment to work properly. For more information, see the prerequisites for your installation method:

• Preparing to install (online)
• Preparing to install (offline)
  • Preparing to install using a Bastion host
  • Preparing to install using a portable compute device
  • Preparing to install using a portable storage device

The hotfix is applied using a client computer that is connected to the target OCP cluster by using the oc command. Run the following steps on the client computer before running the apply-fix.sh shell script:

1. Ensure that the oc command is available. Use the following Red Hat documentation to install the client: Getting started with the OpenShift CLI.
2. Ensure that you are logged in to your Red Hat OpenShift cluster with oc login.
3. Switch to the project containing your AI Manager instance with oc project.

To prevent lost or duplicate processing of log source data, you must disable data flows from log sources before you apply this patch. To disable data flows, use the following steps for each connector type that is enabled and contains a toggle data flow field:

1. Go to the IBM Cloud Pak for Watson AIOps administrator home page and click each connection type one by one, such as "ServiceNow" or "Elk."
2. This takes you to the view page where you can see all of your connections for each integration type.
3. On this page, click each connection individually. Go to the last step of each connection, disable the data flow toggle, and press "Save". Take note of each connection that you disable, as you will need to re-enable them after the hotfix is applied. You can skip any connector that does not have the data flow toggle in the form, such as Kubernetes.

Preparing to apply the hotfix in an air-gapped (offline) environment

Complete the following steps only if you installed AI Manager in an air-gapped (offline) environment. If you installed in an online environment, skip these steps and proceed to the next section, Installation.

For an air-gapped (offline) environment, there are four main steps to prepare your environment for running the script to apply the hotfix:

1. On your cluster, rerun the commands that are listed in step 2 of the preparing to documentation to download the latest IBM Automation Foundation (IAF) and Bedrock versions. Make sure that you follow the steps for your installation approach (Bastion Host, Portable Storage, or Portable Compute). For more information, see the documentation for your installation method:

   • Download the CASE files for an install using a bastion host
   • Download the CASE files for an install using a portable compute device
   • Download the CASE files for an install using a portable storage device

If you have an offline cache from a previous install, you have two options to ensure that the latest hotfix CASEs are saved using cloudctl case save. You can either:

• Delete the existing CASE tgz files from your offline cache, or
• Run cloudctl case save using a different offline cache directory and then copy the contents of that new directory into your existing offline cache.

Once you have the latest files in your offline cache, you can continue the airgap process using that offline cache directory.

2. Next, rerun the commands that are listed in step 3 of the preparing to install documentation to mirror images to the portable registry. Make sure to follow the steps for your installation approach (Bastion Host, Portable Storage, or Portable Compute). For more information, see the documentation for your installation method:

   • Mirroring images for an install using a bastion host
   • Mirroring images for an install using a portable compute device
   • Mirroring images for an install using a portable storage device

3. Mirror the updated images published as part of the hotfix. You can use this sample script to mirror these updated images iteratively:

   • Use Podman to login to the src_registry (cp.icr.io) and dest_registry (Your mirror registry)

   • Download the attached hotfix-image-mirror-list.txt and then run the following sample script:

#!/bin/bash
# This script copies the images listed in the file 'hotfix-image-mirror-list.txt' from IBM registry to your target registry

src_registry=cp.icr.io
dest_registry=<your_target_registry>

FILENAME="hotfix-image-mirror-list.txt"
IMAGELIST=$(cat $FILENAME)
for IMAGE in $IMAGELIST
do
    # skopeo copy from source to destination registry:
    skopeo copy --all docker://$src_registry/$IMAGE docker://$dest_registry/$IMAGE --src-tls-verify=false --dest-tls-verify=false
done

4. Run the commands to install the catalog source again, as listed in step 5.1 of the preparing to install documentation. Make sure to follow the steps for your installation approach (Bastion Host, Portable Storage, or Portable Compute). For more information, see the documentation for your installation method:

   • Create the catalog sources for an install using a bastion host
   • Create the catalog sources for an install using a portable compute device
   • Create the catalog sources for an install using a portable storage device

You can now run the hotfix patch script apply-fix.sh as described the next section, Installation.

Installation

The apply-fix.sh shell script can be run from the command prompt as follows:

./apply-fix.sh

The script prompts you to confirm data flows described in the prerequisites section have been disabled before the script moves forward. To skip confirmation, you can run the script as follows:

./apply-fix.sh -s

The shell script confirms that the prerequisites are met, and that the current project contains an AI Manager instance. The shell script then runs a series of oc commands to apply the hotfix to the AI Manager instance. You will notice that several pods restart while the script is running.

If the script runs successfully to completion, you see the following message:

Hotfix was applied successfully

If the script fails with an error, then correct the error as described in the error message. Then, rerun the apply-fix.sh shell script. The shell script can be used multiple times on the same AI Manager instance. If necessary, contact IBM Support for help with resolving the error condition.

Postinstallation

After the hotfix is applied successfully, you must re-enable the data flows that you previously disabled.

1. Go to the IBM Cloud Pak for Watson AIOps administrator home page and click each connection type one by one, such as "ServiceNow" or "Elk."
2. This takes you to the view page where you can see all of your connections for each integration type.
3. On this page, click each connection that you previously disabled. Go to the last step of each connection, enable the data flow toggle, and press "Save".

Removal

After an AI Manager instance is modified with the hotfix, the hotfix cannot be removed. To obtain an unmodified instance, the AI Manager instance must be deleted and a new AI Manager instance must be created in its place.