Installing IBM Cloud Pak for AIOps in an air-gapped environment (offline) using a portable device

If your cluster is not connected to the internet, you can deploy an installation of IBM Cloud Pak for AIOps on your Red Hat OpenShift Container Platform cluster by using a portable compute device such as a laptop, or a portable storage device such as a USB device.

In this scenario, your air-gapped (offline) environment has a target registry, and a Red Hat OpenShift cluster on which IBM Cloud Pak for AIOps is to be installed. IBM Cloud Pak for AIOps images are mirrored from the internet to a file system on a portable compute device or a portable storage device. The portable device is then disconnected from the internet and connected in the offline environment, where the images are loaded to the target registry. IBM Cloud Pak for AIOps can then be installed in the offline environment by using the target registry.

Before you begin

If IBM Sales representatives and Business Partners supplied you with a custom profile ConfigMap to customize your deployment, then you must follow their instructions to apply it during installation. The custom profile cannot be applied after installation, and attempting to do so can break your IBM Cloud Pak for AIOps deployment. For more information about custom sizing, see Custom sizing.

The following procedure is based on an Red Hat OpenShift 4.21 environment and includes links for that version. If your environment uses a different supported version of Red Hat OpenShift, ensure that you follow the Red Hat OpenShift documentation for that version.

Important: If you use Instana to monitor IBM Cloud Pak for AIOps, take the following actions to avoid operational issues and installation and upgrade failures:

1. Set up the mirroring environment

Prerequisites

Allow access to the following sites and ports:

Site Description
icr.io

cp.icr.io

dd0.icr.io

dd2.icr.io

dd4.icr.io

dd6.icr.io

Allow access to these hosts on port 443 to enable access to the IBM Cloud Container Registry, CASE OCI artifact, and IBM Cloud Pak foundational services catalog source.
dd1-icr.ibm-zh.com

dd3-icr.ibm-zh.com

dd5-icr.ibm-zh.com

dd7-icr.ibm-zh.com

If you are located in China, also allow access to these hosts on port 443.
github.com Github houses CASE files, IBM Cloud Pak tools and scripts.
redhat.com Red Hat OpenShift registries that are required for Red Hat OpenShift, and for Red Hat OpenShift upgrades.

You must be able to download content from GitHub. If you are not able to, verify that your network or proxy settings permit access to GitHub's file server domain and if needed contact your network administrator to allow it.

1.1 Download documentation, scripts and tools for offline access

Download the following documentation, scripts and tools that you might need to access during your IBM Cloud Pak for AIOps installation, and copy them to your air-gapped environment.

  1. IBM Cloud Pak for AIOps 4.13.0 scripts and tools

    1. Download the aiopsctl command line tool for use in steps 5.4 Evaluate storage performance and 5.8 Verify cluster readiness. For more information about downloading this script, see the first step of Evaluate storage performance.

    2. (Optional) The status checker script can be used in step 5.10 Install IBM Cloud Pak for AIOps to give information about the status of your deployment. The use of this script is optional, as status can be found directly from the ibm-aiops-orchestrator custom resource. This script can be downloaded from github.com/IBMOpens in a new tab .

    3. An uninstall script can be downloaded from github.com/IBMOpens in a new tab .

  2. IBM Cloud Pak for AIOps 4.13.0 documentation

    Download the Cloud Pak for AIOps 4.13.0 PDF (this documentation) so that you can access it offline.

  3. Red Hat OpenShift documentation

    The Red Hat OpenShift documentation can be downloaded for offline access from Red HatOpens in a new tab . The Security and compliance, Installing, CLI Tools, Images, and Operators sections are referenced by this documentation.

1.2 Install and configure Red Hat OpenShift

IBM Cloud Pak for AIOps requires Red Hat OpenShift to be installed and running on your target cluster. You must have administrative access to your Red Hat OpenShift cluster.

For more information about the supported versions of Red Hat OpenShift, see Supported Red Hat OpenShift Container Platform versions.

  1. Before installing Red Hat OpenShift, work with your system administrator to verify that the nodes that are intended for the installation have their system clocks synchronized with an NTP server, or are at least manually set to be within a few seconds of one another. If you are installing on a cloud platform, this is usually already configured.

  2. Install Red Hat OpenShift by using the instructions in the Red Hat OpenShift documentation Opens in a new tab . Information about installing a cluster in a restricted network is given in Mirroring images for a disconnected installation Opens in a new tab .

  3. Install the Red Hat OpenShift command line interface (oc) on your cluster's boot node and run oc login. For more information, see the instructions in Getting started with the Red Hat OpenShift CLIOpens in a new tab .

  4. To function properly, distributed platforms and applications such as Red Hat OpenShift and IBM Cloud Pak for AIOps require the system clocks of all of their nodes to be highly synchronized with one another. Discrepancies between the clocks can cause IBM Cloud Pak for AIOps to experience operational issues. All Red Hat OpenShift nodes in the cluster must have access to an NTP server to synchronize their clocks. For more information, see the Red Hat OpenShift documentation Opens in a new tab .

  5. Optionally configure a custom certificate for IBM Cloud Pak for AIOps to use. You can use either of the following methods:

    • Configure a custom certificate for the Red Hat OpenShift cluster. Follow the instructions in the Red Hat OpenShift documentation Replacing the default ingress certificate.
    • If you would like to use a custom certificate for the IBM Cloud Pak for AIOps console only, then after installation is complete follow the instructions in Using a custom certificate.

1.3 Set up a target registry

You must have a local Docker type production-grade registry available in the air-gapped environment to store the IBM Cloud Pak for AIOps images in. The registry must meet the following requirements:

  • supports Docker Manifest V2, Schema 2.
  • supports multi-architecture images.
  • is accessible from the Red Hat OpenShift cluster nodes.
  • allows path separators in the image name.
  • you have the username and password for a user who can read from and write to the registry.
  • must have 175 GB of storage to hold all of the software that is to be transferred to the target registry.

If you do not already have a suitable production-grade registry available, then you must install and configure one. For more information, see About the mirror registry Opens in a new tab in the Red Hat OpenShift documentation.

Do not use the Red Hat OpenShift image registry as your target registry. The Red Hat OpenShift registry does not support multi-architecture images or path separators in the image name.

1.4 Prepare a host

Prepare a portable compute device, or if you are using a portable storage device then a connected compute device, as follows.

You must be able to connect your host to the internet. Your host must be on a Linux x86_64 or Mac platform with any operating system that the IBM Cloud Pak CLI and the Red Hat OpenShift CLI support. If you are on a Windows® platform, you must run the actions in a Linux x86_64 VM or from a Windows Subsystem for Linux® (WSL) terminal.

Your portable device and any intermediary devices must have 175 GB of storage to hold all the software that is to be transferred to the local target registry.

Complete the following steps on your host.

  1. Install Podman.

    To install Podman, see the Podman Installation InstructionsOpens in a new tab .

    Note: Docker is not shipped or supported for Red Hat Enterprise Linux (RHEL) 8. The Podman container engine replaced docker as the preferred, maintained, and supported container runtime of choice for Red Hat Enterprise Linux 8 systems. For more information, see Running containers without DockerOpens in a new tab in the Red Hat documentation.
  2. Install the Red Hat OpenShift CLI tool, oc.

    oc is required for Red Hat OpenShift management. For more information, see Getting started with the Red Hat OpenShift CLIOpens in a new tab in the Red Hat OpenShift documentation.

1.5 Install the IBM Catalog Management Plug-in for IBM Cloud Pak

The IBM Catalog Management Plug-in for IBM Cloud Pak (ibm-pak-plugin) is used for the deployment of IBM Cloud Paks in a disconnected environment. It simplifies the process for discovering required IBM product images and uses standard tools for registry and cluster access. The ibm-pak-plugin also extends the Red Hat OpenShift CLI (oc) capability to streamline the process of delivering installation images to the IBM Cloud Pak in an air-gapped environment.

  1. Download and install ibm-pak-plugin version 1.21.2 for your host operating system from github.com/IBMOpens in a new tab .

  2. Run the following command to extract the files.

    tar -xf oc-ibm_pak-linux-amd64.tar.gz
    
  3. Run the following command to move the file to the /usr/local/bin directory.

    mv oc-ibm_pak-linux-amd64 /usr/local/bin/oc-ibm_pak
    
    Note: If you are installing as a non-root user, you must use sudo.
  4. Confirm that ibm-pak-plugin is installed by running the following command.

    oc ibm-pak --help
    

    Expected result: The ibm-pak-plugin usage is displayed.

2. Download the CASE

Set environment variables on the portable device, and connect it to the internet so that you can download the IBM Cloud Pak for AIOps CASE files.

Note: Save a copy of your environment variable values to a file that you can use as a reference when you are completing your air-gapped installation tasks.
  1. Create the following environment variables.

    export CASE_NAME=ibm-cp-waiops
    export CASE_VERSION=1.19.0
    export CASE_INVENTORY_SETUP=cpwaiopsSetup
    export TARGET_REGISTRY_HOST=<IP_or_FQDN_of_target_registry>
    export TARGET_REGISTRY_PORT=<port_number_of_target_registry>
    export TARGET_REPOSITORY=''
    export TARGET_REGISTRY=$TARGET_REGISTRY_HOST:$TARGET_REGISTRY_PORT
    if [ -n "$TARGET_REPOSITORY" ]; then export TARGET_REGISTRY=$TARGET_REGISTRY_HOST:$TARGET_REGISTRY_PORT/$TARGET_REPOSITORY; fi
    export TARGET_REGISTRY_USER=<username>
    export TARGET_REGISTRY_PASSWORD=<password>
    

    The target registry is the registry where the IBM Cloud Pak for AIOps images are mirrored to, and accessed from by the Red Hat OpenShift cluster, as setup in 1.3 Set up a target registry

    If you are installing multiple offline instances of IBM Cloud Pak for AIOps and they use the same registry, then you must mirror the images for each instance to different repositories in the registry. Set the value of TARGET_REPOSITORY to a distinct value when you install each instance, for example cp4aiops and cp4aiops2. For more information about deploying multiple offline instances of IBM Cloud Pak for AIOps, see Deploying multiple instances on a single cluster.

    If your portable device must connect to the internet through a proxy, then also set the following environment variables.

    export https_proxy=http://proxy-server-hostname:port
    export http_proxy=http://proxy-server-hostname:port
    
  2. Connect your portable device to the internet and disconnect it from the air-gapped environment.

  3. Download the IBM Cloud Pak for AIOps installer and image inventory to your portable device.

    oc ibm-pak get ${CASE_NAME} --version ${CASE_VERSION}
    

    The CASE is downloaded to ~/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION. The logs files are available at ~/.ibm-pak/logs/oc-ibm_pak.log.

    If the download fails, you can try to increase the default values of the HTTP timeout and maximum HTTP retry attempts as required in the IBMPAK_HTTP_TIMEOUT and IBMPAK_HTTP_RETRY flags.

    Note: If you do not specify the CASE version, then the latest CASE is downloaded. The root directory that is used by the ibm-pak-plugin is ~/.ibm-pak. If required, the root directory can be configured by setting the IBMPAK_HOME environment variable.

3. Mirror images

Complete the following steps to mirror the IBM Cloud Pak for AIOps and IBM Cloud Pak foundational services Cert Manager images from the internet to the file system on your portable device, and then from the file system to the target registry in the air-gapped environment.

3.1. Generate mirror manifests

Run the following command to generate mirror manifests to be used when mirroring the images to the target registry.

oc ibm-pak generate mirror-manifests ${CASE_NAME} file://cpfs --version ${CASE_VERSION} --final-registry ${TARGET_REGISTRY}/cpfs
Argument Description
file://cpfs Specifies the path extension where images are mirrored to. Images are mirrored to $IMAGE_PATH/cpfs when the oc image mirror command is run with images-mapping-to-filesystem.txt. For more information, see Mirror the images to the file system.
$TARGET_REGISTRY/cpfs This argument generates an image-mapping file that is used by the oc image mirror commands to mirror images to the TARGET_REGISTRY at namespace cpfs. For example, if the CASE you are installing needs the image, quay.io/opencloudio/ibm-events-kafka-2.6.0@sha256:10d422dddd29ff19c87066fc6510eee05f5fa4ff608b87a06e898b3b6a3a13c7, then its final URL in your target registry will be $TARGET_REGISTRY/cpfs/opencloudio/ibm-events-kafka-2.6.0. Note the new namespace of cpfs in the URL. The namespace path can be multi level if your target registry supports it.

The command generates the following files at ~/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION:

  • images-mapping-to-filesystem.txt
  • images-mapping-from-filesystem.txt
  • image-content-source-policy.yaml or image-digest-mirror-set.yaml, depending on whether your Red Hat OpenShift cluster mirror configuration is set up for ImageContentSourcePolicy (ICSP) or ImageDigestMirrorSet (IDMS).

If you see the following warning, it can be safely ignored because the the files mentioned are not applied.

Catalog source collision detected for cloud-native-postgresql-catalog in /Users/tmg/.ibm-pak/data/mirror/ibm-cp-waiops/1.15.0/catalog-sources.yaml. Please remove incorrect version from that file before applying to the cluster 

3.2. Authenticate with the IBM Entitled Registry

Log in to the IBM Entitled Registry to generate an authentication file containing the IBM Entitled Registry credentials, and then create an environment variable that has the location of the authentication file. This file is used later to enable the oc image mirror command to pull the images from the IBM Entitled Registry.

  1. Get the authentication credentials for the IBM Entitled Registry.

    1. To obtain the entitlement key that is assigned to your IBMid, log in to MyIBM Container Software Library Opens in a new tab with the IBMid and password details that are associated with the entitled software.

    2. In the Entitlement keys section, select Copy key to copy the entitlement key.

  2. Run the following command to create an environment variable that contains your entitlement key.

    export ENTITLED_REGISTRY_PASSWORD=<key>

    Where <key> is the entitlement key that you copied in the previous step.

  3. Store the authentication credentials for the IBM Entitled Registry.

    Run the following commands:

    podman login cp.icr.io -u cp -p ${ENTITLED_REGISTRY_PASSWORD}
    export REGISTRY_AUTH_FILE=${XDG_RUNTIME_DIR}/containers/auth.json
    unset ENTITLED_REGISTRY_PASSWORD
    

    The authentication file is usually at ${XDG_RUNTIME_DIR}/containers/auth.json. For more information, see the Options section in the Podman documentationOpens in a new tab .

3.3. Mirror the images to the file system

Complete these steps to mirror the images from the internet to a file system on your portable device.

  1. Create an environment variable to store the location of the file system where the images are to be stored.

    export IMAGE_PATH=<image-path>
    

    Where <image-path> is the directory where you want the images to be stored.

  2. Run the following command to mirror the images from the IBM Entitled Registry to the file system.

    nohup oc image mirror \
    -f ~/.ibm-pak/data/mirror/${CASE_NAME}/${CASE_VERSION}/images-mapping-to-filesystem.txt \
    -a ${REGISTRY_AUTH_FILE} \
    --filter-by-os '.*' \
    --insecure \
    --skip-multiple-scopes \
    --dir "${IMAGE_PATH}" \
    --max-per-registry=1 > my-mirror-progress.txt 2>&1 &
    

    The UNIX® command nohup is used to ensure that the mirroring process continues even if there is a loss of network connection, and redirection of output to a file provides improved monitoring and error visibility.

    Run the following command if you want to see the progress of the mirroring:

    tail -f my-mirror-progress.txt

    If an error occurs during mirroring, the mirror command can be rerun.

3.4 Setup the file system in the air-gapped environment

  1. Copy files to the air-gapped environment (portable storage device only)

    If you are using a portable storage device, you must copy the files from the portable storage device to a local compute device in the air-gapped environment that has access to the target registry. If you are using a portable compute device, then these items are already present, and you can proceed to the next step.

    Copy the following items to your local compute device:

    • the file system located at $IMAGE_PATH, which you specified earlier
    • ~/.ibm-pak directory
  2. Disconnect the device that has your file system, (the portable compute device or the local compute device) from the internet and connect it to the air-gapped environment.

  3. Ensure that environment variables are set on the device in the air-gapped environment that has access to the target registry.

    If you are using a portable storage device, then set the following environment variables on your local compute device within the air-gapped environment.

    If you are using a portable compute device that you have restarted since mirroring the images, then your environment variables will have been lost and you will need to set the following environment variables on your portable compute device again.

    export CASE_NAME=ibm-cp-waiops
    export CASE_VERSION=1.19.0
    export CASE_INVENTORY_SETUP=cpwaiopsSetup
    export TARGET_REGISTRY_HOST=<IP_or_FQDN_of_target_registry>
    export TARGET_REGISTRY_PORT=<port_number_of_target_registry>
    export TARGET_REPOSITORY=''
    export TARGET_REGISTRY=$TARGET_REGISTRY_HOST:$TARGET_REGISTRY_PORT
    if [ -n "$TARGET_REPOSITORY" ]; then export TARGET_REGISTRY=$TARGET_REGISTRY_HOST:$TARGET_REGISTRY_PORT/$TARGET_REPOSITORY; fi
    export TARGET_REGISTRY_USER=<username>
    export TARGET_REGISTRY_PASSWORD=<password>
    export IMAGE_PATH=<image_path>
    

    If you are installing multiple offline instances of IBM Cloud Pak for AIOps and they use the same registry, then you must mirror the images for each instance to different repositories in the registry. Set the value of TARGET_REPOSITORY to a distinct value when you install each instance, for example cp4aiops and cp4aiops2. For more information about deploying multiple offline instances of IBM Cloud Pak for AIOps, see Deploying multiple instances on a single cluster.

3.5 Authenticate with the target registry

Authenticate with the target registry in the air-gapped environment that you are mirroring the images into.

Run the following commands:

podman login ${TARGET_REGISTRY} -u ${TARGET_REGISTRY_USER} -p ${TARGET_REGISTRY_PASSWORD}
export REGISTRY_AUTH_FILE=${XDG_RUNTIME_DIR}/containers/auth.json

The authentication file is usually at ${XDG_RUNTIME_DIR}/containers/auth.json. For more information, see the Options section in the Podman documentationOpens in a new tab .

3.6 Mirror the images to the target registry from the file system

Complete the steps in this section on the device that has your file system (the portable compute device or the local compute device) to copy the images from the file system to the $TARGET_REGISTRY. Your device with the file system must be connected to both the target registry and the Red Hat OpenShift cluster.

Run the following command to copy the images referenced in the images-mapping-from-filesystem.txt from the $IMAGE_PATH file system to the final target registry.

nohup oc image mirror \
-f ~/.ibm-pak/data/mirror/${CASE_NAME}/${CASE_VERSION}/images-mapping-from-filesystem.txt \
--from-dir "${IMAGE_PATH}" \
-a ${REGISTRY_AUTH_FILE} \
--filter-by-os '.*' \
--insecure  \
--skip-multiple-scopes \
--max-per-registry=1 > my-mirror-progress2.txt 2>&1 &

The UNIX command nohup is used to ensure that the mirroring process continues even if there is a loss of network connection, and redirection of output to a file provides improved monitoring and error visibility.

Run the following command if you want to see the progress of the mirroring:

tail -f my-mirror-progress2.txt

If an error occurs during mirroring, the mirror command can be rerun.

3.7 Configure the cluster

  1. Log in to your Red Hat OpenShift cluster.

    You can identify your specific oc login command by clicking the user menu in the upper left of the Red Hat OpenShift console, and then clicking Copy Login Command.

    Example:

    oc login <server> -u <cluster username> -p <cluster pass>
  2. Update the global image pull secret for your Red Hat OpenShift cluster.

    Follow the steps in the Red Hat OpenShift documentation topic Updating the global cluster pull secret Opens in a new tab .

    These steps enable your cluster to have authentication credentials in place to pull images from your TARGET_REGISTRY as specified in the image-content-source-policy.yaml or image-digest-mirror-set.yaml, which you will apply to your cluster in the next step.

  3. Configure image mirroring using either ImageContentSourcePolicy (ICSP) or ImageDigestMirrorSet (IDMS).

    Select one of the following two options:

    • Option A: IDMS - Use for new installations of IBM Cloud Pak for AIOps, and for the upgrade of IBM Cloud Pak for AIOps deployments on Red Hat OpenShift clusters that use IDMS.

    • Option B: ICSP - Use for the upgrade of IBM Cloud Pak for AIOps deployments on Red Hat OpenShift clusters that use ICSP.

    Red Hat OpenShift have deprecated the use of ICSP for repository mirroring in v4.14 and higher. ICSP is still supported by v4.14-4.19, but support may be removed in future Red Hat OpenShift releases.

    Option A: IDMS

    1. Create the ImageDigestMirrorSet.

      oc apply -f  ~/.ibm-pak/data/mirror/${CASE_NAME}/${CASE_VERSION}/image-digest-mirror-set.yaml
    2. Verify that the ImageDigestMirrorSet resource is created.

      oc get imagedigestmirrorset

    Option B: ICSP

    1. Create the ImageContentSourcePolicy.

      oc apply -f  ~/.ibm-pak/data/mirror/${CASE_NAME}/${CASE_VERSION}/image-content-source-policy.yaml
    2. Verify that the ImageContentSourcePolicy resource is created.

      oc get imagecontentsourcepolicy
  4. Verify your cluster node status.

    oc get MachineConfigPool -w
    
    Important: After the application of the global image pull secret and either ImageContentSourcePolicy or ImageDigestMirrorSet, the configuration of your nodes will be updated sequentially. Wait until all the MachineConfigPools are updated before proceeding to the next step.
  5. (Optional) If you use an insecure registry, you must add the target registry to the cluster's insecureRegistries list.

    oc patch image.config.openshift.io/cluster --type=merge \
    -p '{"spec":{"registrySources":{"insecureRegistries":["'${TARGET_REGISTRY}'"]}}}'
    

4. Configure storage

The storage configuration must satisfy your sizing requirements. Two storage classes are needed for installing IBM Cloud Pak for AIOps. For more information, see Storage.

Note: Storage classes and storage providers cannot be changed after you install IBM Cloud Pak for AIOps. OADP backup and restore requires that a ReadWriteMany (RWX) storage class must be provided. If OADP backup and restore is not needed, a ReadWriteOnce (RWO) storage class can be provided as the RWX-storage-class-name in the installation instance CR YAML file. This configuration cannot be changed after IBM Cloud Pak for AIOps is installed.

5. Install IBM Cloud Pak for AIOps

Now that the images are mirrored to your air-gapped environment, you can deploy IBM Cloud Pak for AIOps to that environment. To install IBM Cloud Pak for AIOps, complete the following steps.

5.1 Create environment variables

Create a shell script that defines the environment variables that will be used to provide installation parameters for your deployment. Use the following codeblock as a template, replacing the brackets < ... > with values for your environment. Name the script waiops_var_<project>.sh, where <project> is the value that you used for PROJECT_CP4AIOPS. For example, waiops_var_cp4aiops.sh.

If you are deploying more than one instance of IBM Cloud Pak for AIOps in the same cluster, then each instance must have a different value for PROJECT_CP4AIOPS. If you already have a deployment of IBM Cloud Pak for AIOps in the namespace cp4aiops, then export PROJECT_CP4AIOPS with a different value, such as cp4aiops2. For more information, see Deploying multiple instances on a single cluster.

#============================================================================================================
# Cloud Pak for AIOps installation variables
#============================================================================================================

export CP4AIOPS_NAME=ibm-cp-aiops
export CP4AIOPS_SIZE=large # Set to small if you only require a starter non-production deployment.
export SECURE_TUNNEL=false # Set to `true` to install Secure Tunnel, otherwise set to `false`.
export PROJECT_CP4AIOPS=cp4aiops
export ACCEPT_LICENSE=false # Set to `true` to agree to the license terms, otherwise install will fail.
export CATALOG_SRC_CP4AIOPS=ibm-aiops-catalog

# -----------------------------------------------------------------------------------------------------------
# Incremental adoption - set your deployment type.
# Set to `true` to install an extended deployment with log anomaly detection and ticket analysis capabilities
# Set to `false` to install a base deployment without log anomaly detection and ticket analysis capabilities
# -----------------------------------------------------------------------------------------------------------
export LOG_ANOMALY=false 

# -----------------------------------------------------------------------------------------------------------
#  Persistent storage
# -----------------------------------------------------------------------------------------------------------
export STG_CLASS=<RWX-storage-class-name>
export STG_CLASS_BLOCK=<RWO-storage-class-name>

# -------------------------------------------------------------------------------------------------------
# Local storage
#
# If you are not using hybrid storage, do not edit the environment variables in this section.
#
# If you are using hybrid storage, set the environment variables in this section to the LVM storage classes
# that you configured for Postgres and Kafka.
# 
# For example:
# export IR_CORE_POSTGRES_LOCAL_STORAGE_CLASS=lvms-vg-ir-core-postgres-1
# export TOPOLOGY_POSTGRES_LOCAL_STORAGE_CLASS=lvms-vg-topology-postgres-1
# export KAFKA_LOCAL_STORAGE_CLASS=lvms-vg-kafka-1
#-------------------------------------------------------------------------------------------------------
export IR_CORE_POSTGRES_LOCAL_STORAGE_CLASS=''
export TOPOLOGY_POSTGRES_LOCAL_STORAGE_CLASS=''
export KAFKA_LOCAL_STORAGE_CLASS=''

# -------------------------------------------------------------------------------------------------------
# Your customer details
# -------------------------------------------------------------------------------------------------------
export CUSTOMER_NAME=<your company name>
export CUSTOMER_ICN=<your IBM Customer Number>
export CUSTOMER_ENVIRONMENT=<Set to `trial`, `poc`, or `production`>

# -------------------------------------------------------------------------------------------------------
# `OwnNamespace` installation mode:  leave INSTALL_MODE_NAMESPACE as it is.
# `AllNamespaces` installation mode: change to export INSTALL_MODE_NAMESPACE=openshift-operators
# -------------------------------------------------------------------------------------------------------
export INSTALL_MODE_NAMESPACE=${PROJECT_CP4AIOPS}

# -------------------------------------------------------------------------------------------------------
# Topology resource group terminology
# Specify `application` or `service` as the terminology to be used for collections of topology resource
# groups. The default is `application`.
# -------------------------------------------------------------------------------------------------------
export TOPOLOGY_TERMINOLOGY=application

If you need help with deciding on the values to set for these environment variables, see the following topics.

You can update your deployment type after installation. For more information, see Updating the deployment type.

You can set a different value for $PROJECT_CP4AIOPS and $CP4AIOPS_NAME if you want. However, you must not use the default, kube-system, kube-public, openshift-node, openshift-infra, or openshift projects (namespaces) for $PROJECT_CP4AIOPS. This is because IBM Cloud Pak for AIOps uses Security Context Constraints (SCC), and SCCs cannot be assigned to pods created in one of the default Red Hat OpenShift projects (namespaces).

Run the following command to source your script and set the environment variables:

. ./waiops_var_<project>.sh

Where <project> is the value that you used for PROJECT_CP4AIOPS.

5.2 Create a custom project (namespace)

  1. Run the following command to create a project (namespace) to deploy IBM Cloud Pak for AIOps into.

    oc create namespace ${PROJECT_CP4AIOPS}
    
  2. Add a node-selector annotation to the IBM Cloud Pak for AIOps namespace.

    The annotation ensures that on a multi-architecture Red Hat OpenShift cluster, IBM Cloud Pak for AIOps workloads are only scheduled on nodes that have an architecture that IBM Cloud Pak for AIOps supports.

    Failure to do so might result in the scheduling and subsequent failure of IBM Cloud Pak for AIOps workloads on Red Hat OpenShift nodes that have a nonsupported architecture. For more information about supported architectures, see Supported platforms.

    Run one of the following commands.

    If you want to use amd64 architecture:

    oc annotate namespace "${PROJECT_CP4AIOPS}" openshift.io/node-selector="kubernetes.io/arch=amd64"
    

    If you want to use s390x architecture:

    oc annotate namespace "${PROJECT_CP4AIOPS}" openshift.io/node-selector="kubernetes.io/arch=s390x"
    

5.3 Create the catalog sources

  1. Run the following command to create the catalog sources for IBM Cloud Pak for AIOps and IBM Cloud Pak foundational services Cert Manager.

    cat << EOF | oc apply -f -
    apiVersion: operators.coreos.com/v1alpha1
    kind: CatalogSource
    metadata:
      name: ${CATALOG_SRC_CP4AIOPS}
      namespace: ${INSTALL_MODE_NAMESPACE}
    spec:
      displayName: ${CATALOG_SRC_CP4AIOPS}
      publisher: IBM Content
      sourceType: grpc
      image: icr.io/cpopen/ibm-aiops-catalog@sha256:294adebdcbfb1dec82d598b4b8439c40ea51e308548207a537ff69bfdca75701
      grpcPodConfig:
        securityContextConfig: restricted
    ---
    apiVersion: operators.coreos.com/v1alpha1
    kind: CatalogSource
    metadata:
      name: ibm-cert-manager-catalog
      namespace: openshift-marketplace
    spec:
      displayName: ibm-cert-manager
      publisher: IBM
      sourceType: grpc
      image: icr.io/cpopen/ibm-cert-manager-operator-catalog@sha256:97610d00d5b46b4de8d3c98233591cc554b7211d96f3d30ed935a84f076e3b65
    EOF
    
  2. Verify that the ibm-aiops-catalog and ibm-cert-manager-catalog CatalogSource objects are in the output that is returned by the following command:

    oc get CatalogSources -n openshift-marketplace
    oc get CatalogSource -n ${INSTALL_MODE_NAMESPACE}
    

    Example output:

    oc get CatalogSources -n openshift-marketplace
    NAME                     DISPLAY                      TYPE   PUBLISHER   AGE
    ibm-cert-manager-catalog ibm-cert-manager             grpc   IBM         2m 
    
    oc get CatalogSource -n cp4aiops
    NAME                     DISPLAY                      TYPE   PUBLISHER   AGE
    ibm-aiops-catalog        ibm-aiops-catalog            grpc   IBM         2m 
    

5.4 Evaluate storage performance

Use the following procedure to evaluate whether your storage performance is sufficient to withstand the demands of a production deployment of IBM Cloud Pak for AIOps.

  1. Download and install the aiopsctl command line interface tool if you haven't already done so.

    AIOPSCTL_TAR=<aiopsctl_tar>
    AIOPSCTL_INSTALL_URL="https://github.com/IBM/aiopsctl/releases/download/v4.13.0/${AIOPSCTL_TAR}"
    curl -LO "${AIOPSCTL_INSTALL_URL}"
    tar xf "${AIOPSCTL_TAR}"
    mv aiopsctl /usr/local/bin/aiopsctl
    

    Where <aiopsctl_tar> is the operating system specific file that you require from the following set: aiopsctl-linux_s390x.tar.gz, aiopsctl-linux_arm64.tar.gz,aiopsctl-linux_amd64.tar.gz, aiopsctl-darwin_amd64.tar.gz, aiopsctl-darwin_arm64.tar.gz.

  2. Evaluate distributed network storage.

    Run the following command:

    aiopsctl benchmark storage --namespace=${PROJECT_CP4AIOPS} --storage-class=${STG_CLASS_BLOCK}

    The tool selects a node in your cluster and benchmarks the performance of the node's storage. The process takes around 8 minutes to run.

    If you think that the storage performance between your nodes varies significantly, then you can use the --node <node_name> argument to pass in the name of the node that you want the tool to run on.

  3. Verify that your benchmarking results meet or exceed the required metrics.

    The following table identifies the storage performance metrics that must be achieved to support a deployment of IBM Cloud Pak for AIOps. If your deployment is custom-sized to support higher rates than the default production rates listed in Processing abilities, then your storage performance must exceed these metrics.

    Metric Read Write
    Minimum sequential IOPS (higher is better, lower is worse) 5000 5000
    Minimum sequential bandwidth (higher is better, lower is worse) 20 Mi/sec 20 Mi/sec
    Maximum average sequential latency (lower is better, higher is worse) 500 usec 1000 usec

    These metrics are not applicable if you are using native storage in a public cloud provider such as Amazon Elastic Block Store (EBS) or IBM Cloud Block Storage. Performance will be adequate if you selected the native storage classes documented for your cloud platform.

  4. Evaluate local storage.

    If your IBM Cloud Pak for AIOps deployment does not use hybrid storage then skip this step.

    If your IBM Cloud Pak for AIOps deployment is using hybrid storage, then evaluate the storage performance for Postgres and Kafka, by passing in the Postgres and Kafka storage classes, and verifying that the results meet or exceed the required metrics in the previous step.

    For example:

    aiopsctl benchmark storage --namespace=${PROJECT_CP4AIOPS} --storage-class=lvms-vg-kafka-1
    aiopsctl benchmark storage --namespace=${PROJECT_CP4AIOPS} --storage-class=lvms-vg-ir-core-postgres-1
    aiopsctl benchmark storage --namespace=${PROJECT_CP4AIOPS} --storage-class=lvms-vg-topology-postgres-1
    

5.5 Configure usage data collection

To help the development of IBM Cloud Pak for AIOps, daily aggregated usage data is collected to analyse how IBM Cloud Pak for AIOps is used. The collection of usage data is enabled by default, but can be disabled. Usage data is collected by the cp4waiops-metricsprocessor pod, and stored in the cp4waiops-metricsprocessor pod's logs. This usage data is sent to IBM when MustGather output is sent to IBM, as the MustGather includes the output from the cp4waiops-metricsprocessor pod's logs. The usage data is then sent to and stored in IBM controlled GDPR-compliant systems. The usage data that is collected is numeric, or is about the deployment type and platform. It does not include email addresses, passwords, or specific details. Only the following data is collected:

  • Current number of applications
  • Current number of alerts (all severities aggregated)
  • Current number of incidents (all priorities aggregated)
  • Current number of policies (includes predefined and user created)
  • Current number of runbooks run since installation
  • Current number of integrations of each type (For example ServiceNow, Instana, Falcon Logscale)
  • Secure tunnel enablement: whether connection (which controls whether you can create a secure tunnel) is enabled in the Installation custom resource
  • Deployment type: base deployment or extended deployment
  • Deployment platform: Red Hat OpenShift Container Platform or Linux

This usage data is sent to IBM only if the MustGather tool is run and sent to IBM. The MustGather includes the output from the cp4waiops-metricsprocessor pod's logs.

Configuring the collection of usage data

If you do not want to disable the collection of usage data, run the following command to configure the usage data with your customer details.

oc create secret generic aiops-metrics-processor -n ${PROJECT_CP4AIOPS} --from-literal=customerName=${CUSTOMER_NAME} --from-literal=customerICN=${CUSTOMER_ICN} --from-literal=environment=${CUSTOMER_ENVIRONMENT}

Usage data without your customer details is still collected even if you do not create this secret. If you do not want any usage data collected, then you must run the command given in Disabling the collection of usage data.

Disabling the collection of usage data

If you want to disable the collection of usage data, run the following command.

oc create secret generic aiops-metrics-processor -n ${PROJECT_CP4AIOPS} --from-literal=customerName=${CUSTOMER_NAME} --from-literal=customerICN=${CUSTOMER_ICN} --from-literal=environment=${CUSTOMER_ENVIRONMENT} --from-literal=enableCollection=false 

You can update your usage data collection preferences after installation. For more information, see Updating usage data collection preferences.

5.6 Install Cert Manager

Skip this step if you already have a certificate manager installed on the Red Hat OpenShift cluster that you are installing IBM Cloud Pak for AIOps on. If you do not have a certificate manager then you must install one.

The IBM Cloud Pak foundational services Cert Manager is recommended. For more information about IBM Cloud Pak foundational services Cert Manager hardware requirements, see IBM Certificate Manager (cert-manager) hardware requirements Opens in a new tab in the IBM Cloud Pak foundational services documentation.

The Red Hat OpenShift Cert Manager is also supported. For more information, see cert-manager Operator for Red Hat OpenShift in the Red Hat OpenShift documentation.

The IBM Cloud Pak foundational services Cert Manager can be installed with the following steps.

  1. Run the following command to create the resource definitions that you need:

    cat << EOF | oc apply -f -
    apiVersion: v1
    kind: Namespace
    metadata:
      name: ibm-cert-manager
    ---
    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      name: ibm-cert-manager-operator-group
      namespace: ibm-cert-manager
    ---
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: ibm-cert-manager-operator
      namespace: ibm-cert-manager
    spec:
      channel: v4.2
      installPlanApproval: Automatic
      name: ibm-cert-manager-operator
      source: ibm-cert-manager-catalog
      sourceNamespace: openshift-marketplace
    EOF
    
  2. Run the following command to ensure that the IBM Cloud Pak foundational services Cert Manager pods have a STATUS of Running before proceeding to the next step.

    oc -n ibm-cert-manager get pods
    

    Example output for a successful IBM Cloud Pak foundational services Cert Manager installation:

    NAME                                        READY   STATUS    RESTARTS   AGE
    cert-manager-cainjector-674854c49d-vstq4    1/1     Running   0          8d
    cert-manager-controller-646d4bd6fd-zwmqm    1/1     Running   0          8d
    cert-manager-webhook-8598787c8-s4lkt        1/1     Running   0          8d
    ibm-cert-manager-operator-c96957695-dkxnm   1/1     Running   0          8d
    

5.7 Verify cluster readiness

Run the following command to verify whether your environment is correctly set up for an IBM Cloud Pak for AIOps installation.

aiopsctl server precheck -n ${PROJECT_CP4AIOPS} 
  • If you have a multi-zone cluster, then also specify the -m flag. This flag enables extra checks to help ensure that the cluster has sufficient resources to withstand a zone outage, and that the zones are well balanced for memory and CPU. For example, aiopsctl server precheck -n cp4aiops -m.
  • If you are using hybrid storage, then also specify the --hybrid-storage flag. This flag enables extra checks to help ensure that sufficient local storage is configured. For example, aiopsctl server precheck -n cp4aiops --hybrid-storage
Example output:
# aiopsctl server precheck -n cp4aiops
o- [25 Mar 26 16:08 GMT] Running precheck tool
o- [25 Mar 26 16:08 GMT] Checking hardware resources...
Total Node Count (Available Schedulable / Required): 17/6
Production (HA) Base CPU (vCPU): 264 / 143
Production (HA) Base Memory (GB): 539 / 331

Production (HA) Extended CPU (vCPU): 264 / 170
Production (HA) Extended Memory (GB): 539 / 391

You have enough resources for 1 instance(s) of large Base install
You have enough resources for 1 instance(s) of large Extended install

Total Node Count (Available Schedulable / Required): 17/3
Starter (Non-HA) Base CPU (vCPU): 264 / 47
Starter (Non-HA) Base Memory (GB): 539 / 123

Starter (Non-HA) Extended CPU (vCPU): 264 / 55
Starter (Non-HA) Extended Memory (GB): 539 / 136

You have enough resources for 5 instance(s) of small Base install
You have enough resources for 4 instance(s) of small Extended install

minimum requirements met for both starter and production

o- [25 Mar 26 16:08 GMT] Checking storage...
Required StorageClasses found for provider Red Hat OpenShift Data Foundation: ocs-storagecluster-cephfs, ocs-storagecluster-ceph-rbd
Checking if PVC can bind to supported storage class

[Attempt 1 of 14] PVC aiops-prereq-storage-test-pvc-ocs-storagecluster-cephfs is in phase: Pending
PVC aiops-prereq-storage-test-pvc-ocs-storagecluster-cephfs is successfully bound to storage class ocs-storagecluster-cephfs

[Attempt 1 of 14] PVC aiops-prereq-storage-test-pvc-ocs-storagecluster-ceph-rbd is in phase: Pending
PVC aiops-prereq-storage-test-pvc-ocs-storagecluster-ceph-rbd is successfully bound to storage class ocs-storagecluster-ceph-rbd

Storage check passed

o- [25 Mar 26 16:08 GMT] Checking OCP Version...
[WARN] Heterogeneous architecture clusters are not being supported at this time
Cluster meets OCP version requirements

o- [25 Mar 26 16:08 GMT] Checking if Cert Manager is present...
Certificate CustomResourceDefinition Found

o- [25 Mar 26 16:08 GMT] Checking if certs will expire within 4 days...

o- [25 Mar 26 16:08 GMT] Precheck Summary Results
Check                          Result
Meets Hardware Requirements    Passed
No Storage Issues              Passed
Meets OCP Version Requirement  Passed
Cert Mgr Operator Exists       Passed
Certificates Valid             Passed
  • The "You have enough resources for <...>" statements denote the number of instances that the cluster can support, and include any existing instances. The number that is given is for the number of base instances or extended instances. It does not mean that the cluster can support the stated number of base instances and the stated number of extended instances.
  • If you are not using IBM Cloud Pak foundational services Cert Manager, then ignore any errors that are returned by the Cert Manager check.

5.8 Install the operator

For more information about installing operators, see Adding Operators Opens in a new tab to a cluster in the Red Hat OpenShift documentation.

For more information about the operators which are installed with IBM Cloud Pak for AIOps, see Operator Details.

  1. Create an OperatorGroup.

    Skip this step if you are installing using the 'All Namespaces' installation mode. Check that you set INSTALL_MODE_NAMESPACE correctly in step 5.1, and proceed to the next step, Install the IBM Cloud Pak for AIOps operator.

    If you are installing using the 'OwnNamespace' installation mode, then you must create an operator group in your custom project (namespace), or the IBM Cloud Pak for AIOps operator will not install. There might be an operator group for managing a namespace for given APIs. If there is an operator group for the namespace, do not create a second one.

    Create the Operator group by running the following command:

    cat << EOF | oc apply -f -
    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      name: cp4aiops-operator-group
      namespace: ${PROJECT_CP4AIOPS}
    spec:
      targetNamespaces:
        - "${PROJECT_CP4AIOPS}"
    EOF
    
  2. Install the IBM Cloud Pak for AIOps operator.

    Run the following command.

    cat << EOF | oc apply -f -
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: ibm-aiops-orchestrator
      namespace: $INSTALL_MODE_NAMESPACE
    spec:
      channel: v4.13
      installPlanApproval: Automatic
      name: ibm-aiops-orchestrator
      source: ${CATALOG_SRC_CP4AIOPS}
      sourceNamespace: ${INSTALL_MODE_NAMESPACE}
    EOF
    

    installPlanApproval must not be changed to Manual. Manual approval, which requires the manual review and approval of the generated InstallPlans, is not supported. Incorrect timing or ordering of manual approvals of InstallPlans can result in a failed installation.

  3. After a few minutes, the IBM Cloud Pak for AIOps operator is installed. Verify that the all of the components have a state of Succeeded by running the following command:

    oc get csv -n ${INSTALL_MODE_NAMESPACE} | egrep "ibm-aiops-orchestrator"
    

    Example output:

    $ oc get csv -n ${INSTALL_MODE_NAMESPACE} | egrep "ibm-aiops-orchestrator"
    ibm-aiops-orchestrator.v4.13.0         IBM Cloud Pak for AIOps               4.13.0   Succeeded
    

5.9 Install IBM Cloud Pak for AIOps

Create an instance of the IBM Cloud Pak for AIOps custom resource. A maximum of one IBM Cloud Pak for AIOps custom resource is allowed per namespace.

  1. Use one of the following YAML codeblocks to create an instance of the IBM Cloud Pak for AIOps custom resource, depending on whether you are installing on a multi-zone cluster.

    Installing on a non multi-zone cluster:

    cat << EOF | oc apply -f -
    apiVersion: orchestrator.aiops.ibm.com/v1alpha1
    kind: Installation
    metadata:
      name: ${CP4AIOPS_NAME}
      namespace: ${PROJECT_CP4AIOPS}
    spec:
      size: ${CP4AIOPS_SIZE}
      storage:
        aiops-ir-core-postgres:
          storageClass: ${IR_CORE_POSTGRES_LOCAL_STORAGE_CLASS}
        aiops-ir-core-postgres-wal:
          storageClass: ${IR_CORE_POSTGRES_LOCAL_STORAGE_CLASS}
        aiops-topology-postgres:
          storageClass: ${TOPOLOGY_POSTGRES_LOCAL_STORAGE_CLASS}
        aiops-topology-postgres-wal:
          storageClass: ${TOPOLOGY_POSTGRES_LOCAL_STORAGE_CLASS}
        data-iaf-system-kafka:
          storageClass: ${KAFKA_LOCAL_STORAGE_CLASS}
      storageClass: ${STG_CLASS}
      storageClassLargeBlock: ${STG_CLASS_BLOCK}
      imagePullSecret:
      topologyModel: ${TOPOLOGY_TERMINOLOGY}
      license:
        accept: ${ACCEPT_LICENSE}
      pakModules:
      - name: aiopsFoundation
        enabled: true
      - name: applicationManager
        enabled: true
      - name: aiManager
        enabled: true
      - name: connection
        enabled: ${SECURE_TUNNEL}
      - name: logAnomalyDetection
        enabled: ${LOG_ANOMALY}
    EOF
    

    Installing on a multi-zone cluster:

    cat << EOF | oc apply -f -
    apiVersion: orchestrator.aiops.ibm.com/v1alpha1
    kind: Installation
    metadata:
      name: ${CP4AIOPS_NAME}
      namespace: ${PROJECT_CP4AIOPS}
    spec:
      size: ${CP4AIOPS_SIZE}
      storage:
        aiops-ir-core-postgres:
          storageClass: ${IR_CORE_POSTGRES_LOCAL_STORAGE_CLASS}
        aiops-ir-core-postgres-wal:
          storageClass: ${IR_CORE_POSTGRES_LOCAL_STORAGE_CLASS}
        aiops-topology-postgres:
          storageClass: ${TOPOLOGY_POSTGRES_LOCAL_STORAGE_CLASS}
        aiops-topology-postgres-wal:
          storageClass: ${TOPOLOGY_POSTGRES_LOCAL_STORAGE_CLASS}
        data-iaf-system-kafka:
          storageClass: ${KAFKA_LOCAL_STORAGE_CLASS}
      storageClass: ${STG_CLASS}
      storageClassLargeBlock: ${STG_CLASS_BLOCK}
      imagePullSecret:
      topologyModel: ${TOPOLOGY_TERMINOLOGY}
      license:
        accept: ${ACCEPT_LICENSE}
      pakModules:
      - name: aiopsFoundation
        enabled: true
      - name: applicationManager
        enabled: true
      - name: aiManager
        enabled: true
      - name: connection
        enabled: ${SECURE_TUNNEL}
      - name: logAnomalyDetection
        enabled: ${LOG_ANOMALY}
      zones:
      - name: <zone_name1>
      - name: <zone_name2>
      - name: <zone_name3>
    EOF
    

    Where <zone_name1>, <zone_name2>, and <zone_name3> exactly match the zone labels that you applied to each of your nodes in step 1a of Installing IBM Cloud Pak for AIOps on a multi-zone architecture (multi-zone HA).

    The pakModules aiopsFoundation, applicationManager, and aiManager must be enabled as in the preceding YAML. Do not change these values to false.

  2. Verify your installation.

    Run the following command to check that the PHASE of your installation is Updating.

    oc get installations.orchestrator.aiops.ibm.com -n ${PROJECT_CP4AIOPS}
    Example output:
    NAME                  PHASE     LICENSE    STORAGECLASS   STORAGECLASSLARGEBLOCK   AGE
    ibm-cp-aiops          Updating  Accepted   rook-cephfs    rook-ceph-block          3m
    

    It takes around 60-90 minutes for the installation to complete (subject to the speed with which images can be pulled). When installation is complete and successful, the PHASE of your installation changes to Running. If your installation phase does not change to Running, then use the following command to find out which components are not ready:

    oc get installation.orchestrator.aiops.ibm.com -o yaml -n ${PROJECT_CP4AIOPS} | grep 'Not Ready'
    
    Example output:
    lifecycleservice: Not Ready
    zenservice: Not Ready
    

    To see details about why a component is Not Ready run the following command, where <component> is the component that is not ready, for example zenservice.

    oc get <component> -o yaml -n ${PROJECT_CP4AIOPS}

    (Optional) If you downloaded the status checker script earlier in step 1.1 Download documentation and scripts for offline access, then you can also run this script to see information about the status of your deployment.

If the installation fails, or is not complete and is not progressing, then see Troubleshooting installation and upgrade and Known Issues to help you identify any installation problems.

5.10 Set the Postgres operator replica count

Set the replica count for the Postgres operator to 2 to help ensure high availability.

  1. Run the following command:

    oc patch csv cloud-native-postgresql.v1.25.5 --type json -p '[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/replicas", "value": 2}]' -n ${PROJECT_CP4AIOPS}

    Example output:

    clusterserviceversion.operators.coreos.com/cloud-native-postgresql.v1.25.5 patched
  2. Run the following command to verify that there are 2 Postgres replicas:
    oc wait deployment/postgresql-operator-controller-manager-1-25-5 --for=jsonpath='{.status.readyReplicas}'=2 -n ${PROJECT_CP4AIOPS}
    After a few seconds, you will see the following output if the patch command was successful:
    deployment.apps/postgresql-operator-controller-manager-1-25-5 condition met
    If the command is not successful, the following output is displayed after 30 seconds. Contact IBM Support.
    error: timed out waiting for the condition on deployments/postgresql-operator-controller-manager-1-25-5

5.11 Verify local storage

If your IBM Cloud Pak for AIOps deployment does not use hybrid storage then skip this step.

If your IBM Cloud Pak for AIOps deployment uses hybrid storage, verify that local storage has been correctly created for Postgres and Kafka.

  1. Verify IR Core Postgres local storage.

    1. Run the following command to verify that IR Core Postgres pods are scheduled on the nodes that you configured for Postgres local storage.

      oc get pod -l "k8s.enterprisedb.io/cluster=aiops-ir-core-postgres" -o wide -n ${PROJECT_CP4AIOPS}
      
      Example output, where Postgres pods are scheduled on nodes named worker9, worker10 and worker11.
      NAME                       READY   STATUS    RESTARTS   AGE   IP             NODE                   NOMINATED NODE   READINESS GATES
      aiops-ir-core-postgres-1   1/1     Running   0          25m   10.254.68.38   worker11.example.com   <none>           <none>
      aiops-ir-core-postgres-2   1/1     Running   0          23m   10.254.60.30   worker9.example.com    <none>           <none>
      aiops-ir-core-postgres-3   1/1     Running   0          22m   10.254.36.87   worker10.example.com   <none>           <none>
      

      Keep a copy of this output that maps PVCs to nodes so that you have this information available if you need to restore your deployment. Add this information to the file that you created for step 2.4 of Hybrid storage.

    2. Run the following command to verify that logical volumes have been created for IR Core Postgres.

      oc get pvc -l "k8s.enterprisedb.io/cluster=aiops-ir-core-postgres" -o wide -n ${PROJECT_CP4AIOPS}
      

      Example output:

      NAME                          STATUS  VOLUME                                     CAPACITY  ACCESS MODES  STORAGECLASS                VOLUMEATTRIBUTESCLASS  AGE  VOLUMEMODE
      aiops-ir-core-postgres-1      Bound   pvc-d4e70a18-d791-4d6f-9c52-af8235417265   50Gi      RWO           lvms-vg-ir-core-postgres-1  <unset>                38m  Filesystem
      aiops-ir-core-postgres-1-wal  Bound   pvc-271f246d-8726-4823-9ab7-26fd0622f50b   10Gi      RWO           lvms-vg-ir-core-postgres-1  <unset>                38m  Filesystem
      aiops-ir-core-postgres-2      Bound   pvc-f3e0c1f1-7e18-4d60-895e-623b9e9b4cc1   50Gi      RWO           lvms-vg-ir-core-postgres-1  <unset>                39m  Filesystem
      aiops-ir-core-postgres-2-wal  Bound   pvc-362e47cd-4124-4f95-995f-55e216b804b4   10Gi      RWO           lvms-vg-ir-core-postgres-1  <unset>                39m  Filesystem
      aiops-ir-core-postgres-3      Bound   pvc-b08d6313-d262-4e9a-baf7-09b6fdbacf1c   50Gi      RWO           lvms-vg-ir-core-postgres-1  <unset>                39m  Filesystem
      aiops-ir-core-postgres-3-wal  Bound   pvc-6bbb154e-1845-44b5-a92c-00f22b41e1c5   10Gi      RWO           lvms-vg-ir-core-postgres-1  <unset>                39m  Filesystem
      
  2. Verify Topology Postgres local storage.

    1. Run the following command to verify that the Topology Postgres pods are scheduled on the nodes that you configured for Postgres local storage.
      oc get pod -l "k8s.enterprisedb.io/cluster=aiops-topology-postgres" -o wide -n ${PROJECT_CP4AIOPS}
      Example output, where Postgres pods are scheduled on nodes named worker9, worker10 and worker11.
      NAME                        READY   STATUS    RESTARTS   AGE   IP             NODE                   NOMINATED NODE   READINESS GATES
      aiops-topology-postgres-1   1/1     Running   0          83m   10.254.52.70   worker12.example.com   <none>           <none>
      aiops-topology-postgres-2   1/1     Running   0          93m   10.254.56.20   worker13.example.com   <none>           <none>
      aiops-topology-postgres-3   1/1     Running   0          86m   10.254.12.52   worker14.example.com   <none>           <none>
      

      Keep a copy of this output that maps PVCs to nodes so that you have this information available if you need to restore your deployment. Add this information to the file that you created for step 2.4 of Hybrid storage.

    2. Run the following command to verify that logical volumes have been created for topology Postgres.

      oc get pvc -l "k8s.enterprisedb.io/cluster=aiops-topology-postgres" -o wide -n ${PROJECT_CP4AIOPS}
      

      Example output:

      NAME                           STATUS  VOLUME                                    CAPACITY  ACCESS MODES  STORAGECLASS                 VOLUMEATTRIBUTESCLASS  AGE  VOLUMEMODE
      aiops-topology-postgres-1      Bound   pvc-b2b29420-a0b8-469a-99e4-94b359487417  50Gi      RWO           lvms-vg-topology-postgres-1  <unset>                95m  Filesystem
      aiops-topology-postgres-1-wal  Bound   pvc-6bbb154e-1845-44b5-a92c-00f22b41e1c5  10Gi      RWO           lvms-vg-topology-postgres-1  <unset>                95m  Filesystem
      aiops-topology-postgres-2      Bound   pvc-3bb7fca3-f722-4bba-9ed9-ec2d46279223  50Gi      RWO           lvms-vg-topology-postgres-1  <unset>                95m  Filesystem
      aiops-topology-postgres-2-wal  Bound   pvc-5cbbe8e9-7472-4e70-826b-89e22bea4250  10Gi      RWO           lvms-vg-topology-postgres-1  <unset>                95m  Filesystem
      aiops-topology-postgres-3      Bound   pvc-d6de7642-dc24-42b9-bd87-11d7b882cfc6  50Gi      RWO           lvms-vg-topology-postgres-1  <unset>                95m  Filesystem
      aiops-topology-postgres-3-wal  Bound   pvc-5a76bd34-c570-4118-9316-33398cb83a16  10Gi      RWO           lvms-vg-topology-postgres-1  <unset>                95m  Filesystem
      
  3. Verify Kafka local storage.

    1. Run the following command to verify that Kafka pods are scheduled on the nodes that you configured for Kafka local storage.
      oc get pod -l ibmevents.ibm.com/name=iaf-system-kafka -o wide -n ${PROJECT_CP4AIOPS}
      Example output, where Kafka pods are scheduled on nodes named worker6, worker7 and worker8.
      NAME                 READY   STATUS    RESTARTS   AGE    IP             NODE                  NOMINATED NODE   READINESS GATES
      iaf-system-kafka-0   1/1     Running   0          118m   10.123.24.14   worker6.example.com   <none>           <none>
      iaf-system-kafka-1   1/1     Running   0          118m   10.123.28.12   worker7.example.com   <none>           <none>
      iaf-system-kafka-2   1/1     Running   0          118m   10.123.56.27   worker8.example.com   <none>           <none>
      

      Keep a copy of this output that maps PVCs to nodes so that you have this information available if you need to restore your deployment. Add this information to the file that you created for step 2.4 of Hybrid storage.

    2. Run the following command to verify that logical volumes have been created for Kafka.

      oc get pvc -l ibmevents.ibm.com/name=iaf-system-kafka -n ${PROJECT_CP4AIOPS}
      

      Example output:

      NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
      data-iaf-system-kafka-0   Bound    pvc-fc36a8c8-7b73-473b-86a4-d4686228d521   100Gi      RWO            lvms-vg-kafka-1   124m
      data-iaf-system-kafka-1   Bound    pvc-2f653468-95a0-4d5b-b698-64023e25f643   100Gi      RWO            lvms-vg-kafka-1   124m
      data-iaf-system-kafka-2   Bound    pvc-c477e794-c825-4b58-af68-8a40b290126d   100Gi      RWO            lvms-vg-kafka-1   124m
      

5.12 Create an EgressFirewall

There is no egress firewall policy defined when you install IBM Cloud Pak for AIOps, so outgoing traffic from workload pods to the internal and external network is unrestricted.

To create a more secure environment, use the following steps.

  1. Create an EgressFirewall on your Red Hat OpenShift cluster to limit egress from the IBM Cloud Pak for AIOps project (namespace).

    For more information about creating an EgressFirewall, see Configuring an egress firewall for a projectOpens in a new tab .

    There must be only one EgressFirewall per project/namespace.

  2. Configure exceptions to the EgressFirewall.

    Edit your EgressFirewall to add exceptions for the following IBM Cloud Pak for AIOps components that have egress dependencies, otherwise these IBM Cloud Pak for AIOps components fail when they attempt egress.

    1. Allow egress to any external services, such as the following integrations:
      • Kubernetes
      • GitHub
      • Microsoft® Teams
      • ServiceNow
      • Slack
      • VMware® vCenter
    2. Configure your EgressFirewall to allow traffic for your GitHub, Kubernetes, ServiceNow, and VMware vCenter integrations.

      Edit your EgressFirewall to allow or deny egress, as in the following example:
      kind: EgressFirewall
      metadata:
        name: default
      spec:
        egress:
        - type: Allow
          to:
            cidrSelector: <1.2.3.0/24>
        - type: Allow
          to:
            dnsName: <www.github.com>
        - type: Allow
          to:
            dnsName: <www.developer.kubernetes.com>
        - type: Allow
          to:
            dnsName: <www.developer.servicenow.com>
        - type: Allow
          to:
            dnsName: <www.developer.vcenter.com>
        - type: Deny
          to:
            cidrSelector: <0.0.0.0/0>
      
      Substitute values for `dnsName` and `cidrSelector` that are the DNS names and addresses of your GitHub, Kubernetes, ServiceNow, or VMware vCenter sources.

5.13 Access the Cloud Pak for AIOps console

After you successfully install IBM Cloud Pak for AIOps, get the URL for accessing the Cloud Pak for AIOps console.

  1. Use the following command to get the URL to access the Cloud Pak for AIOps console:

    oc get route -n ${PROJECT_CP4AIOPS} cpd -o jsonpath='{.spec.host}'
    The following output is a sample output:
    cpd-cp4aiops.apps.mycluster.mydomain

    Based on the sample output, your console URL would be https://cpd-cp4aiops.apps.mycluster.mydomain

  2. Enter the URL in your browser to open the Cloud Pak for AIOps console. Log in with your username and password.

Find the IBM Cloud Pak for AIOps console username and password

The default username to access the Cloud Pak for AIOps console is admin. You can check the default username and their password with the following commands.

This information is for the IBM provided credentials (admin only) authentication type.

  1. Find the default username.

    oc -n ${PROJECT_CP4AIOPS} get secret platform-auth-idp-credentials -o jsonpath='{.data.admin_username}' | base64 -d && echo
  2. Get the password for the admin username.

    oc -n ${PROJECT_CP4AIOPS} get secret platform-auth-idp-credentials -o jsonpath='{.data.admin_password}' | base64 -d
    The following extract shows a sample output:
    EwK9dj9fwPZHyHTyu9TyIgh9klZSzVsA

    Based on the sample output, your password would be EwK9dj9fwPZHyHTyu9TyIgh9klZSzVsA.

    You can change this default password at any time. For more information, see Changing the cluster administrator password.

What to do next