Installing IBM Cloud Pak for AIOps in an air-gapped environment (offline) using a portable device
If your cluster is not connected to the internet, you can deploy an installation of IBM Cloud Pak for AIOps on your Red Hat OpenShift Container Platform cluster by using a portable compute device such as a laptop, or a portable storage device such as a USB device.
In this scenario, your air-gapped (offline) environment has a target registry, and a Red Hat OpenShift cluster on which IBM Cloud Pak for AIOps is to be installed. IBM Cloud Pak for AIOps images are mirrored from the internet to a file system on a portable compute device or a portable storage device. The portable device is then disconnected from the internet and connected in the offline environment, where the images are loaded to the target registry. IBM Cloud Pak for AIOps can then be installed in the offline environment by using the target registry.
Before you begin
- You must know whether you are deploying a base deployment or a extended deployment of IBM Cloud Pak for AIOps. For more information, see Incremental adoption.
- After you have installed IBM Cloud Pak for AIOps, you cannot switch between a multi-zone and non multi-zone environment.
- Review the Planning section. Your environment must meet the system requirements.
- Ensure that you are logged in to your Red Hat OpenShift cluster with
oc loginfor any steps that use the Red Hat OpenShift command-line interface (CLI). - The following commands must be run as a root user or by a user with sudo access:
yum install podmanandpodman login. The rest of the procedure can be run as a non-root user. - If you require details about the permissions that the IBM Cloud Pak for AIOps operators need, see Permissions (IBM Cloud Pak for AIOps).
- A user with
cluster-adminprivileges is needed for the following operations:
If IBM Sales representatives and Business Partners supplied you with a custom profile ConfigMap to customize your deployment, then you must follow their instructions to apply it during installation. The custom profile cannot be applied after installation, and attempting to do so can break your IBM Cloud Pak for AIOps deployment. For more information about custom sizing, see Custom sizing.
The following procedure is based on an Red Hat OpenShift 4.21 environment and includes links for that version. If your environment uses a different supported version of Red Hat OpenShift, ensure that you follow the Red Hat OpenShift documentation for that version.
- Ensure that Instana AutoTrace is disabled for the IBM Cloud Pak for AIOps namespace. For more information, see Instana AutoTrace causes pod eviction and prevents install and upgrade.
- Increase the ephemeral storage for the Flink pods. For more information, see flink-kubernetes-operator ephemeral storage eviction when monitoring with Instana.
Installation procedure
1. Set up the mirroring environment
Prerequisites
Allow access to the following sites and ports:
| Site | Description |
|---|---|
icr.io
|
Allow access to these hosts on port 443 to enable access to the IBM Cloud Container Registry, CASE OCI artifact, and IBM Cloud Pak foundational services catalog source. |
dd1-icr.ibm-zh.com
|
If you are located in China, also allow access to these hosts on port 443. |
github.com
|
Github houses CASE files, IBM Cloud Pak tools and scripts. |
redhat.com
|
Red Hat OpenShift registries that are required for Red Hat OpenShift, and for Red Hat OpenShift upgrades. |
You must be able to download content from GitHub. If you are not able to, verify that your network or proxy settings permit access to GitHub's file server domain and if needed contact your network administrator to allow it.
1.1 Download documentation, scripts and tools for offline access
Download the following documentation, scripts and tools that you might need to access during your IBM Cloud Pak for AIOps installation, and copy them to your air-gapped environment.
-
IBM Cloud Pak for AIOps 4.13.0 scripts and tools
-
Download the
aiopsctlcommand line tool for use in steps 5.4 Evaluate storage performance and 5.8 Verify cluster readiness. For more information about downloading this script, see the first step of Evaluate storage performance. -
(Optional) The status checker script can be used in step 5.10 Install IBM Cloud Pak for AIOps to give information about the status of your deployment. The use of this script is optional, as status can be found directly from the
ibm-aiops-orchestratorcustom resource. This script can be downloaded from github.com/IBM.
-
An uninstall script can be downloaded from github.com/IBM
.
-
-
IBM Cloud Pak for AIOps 4.13.0 documentation
Download the Cloud Pak for AIOps 4.13.0 PDF (this documentation) so that you can access it offline.
-
Red Hat OpenShift documentation
The Red Hat OpenShift documentation can be downloaded for offline access from Red Hat
. The
Security and compliance,Installing,CLI Tools,Images, andOperatorssections are referenced by this documentation.
1.2 Install and configure Red Hat OpenShift
IBM Cloud Pak for AIOps requires Red Hat OpenShift to be installed and running on your target cluster. You must have administrative access to your Red Hat OpenShift cluster.
For more information about the supported versions of Red Hat OpenShift, see Supported Red Hat OpenShift Container Platform versions.
-
Before installing Red Hat OpenShift, work with your system administrator to verify that the nodes that are intended for the installation have their system clocks synchronized with an NTP server, or are at least manually set to be within a few seconds of one another. If you are installing on a cloud platform, this is usually already configured.
-
Install Red Hat OpenShift by using the instructions in the Red Hat OpenShift documentation
. Information about installing a cluster in a restricted network is given in Mirroring images for a disconnected installation
.
-
Install the Red Hat OpenShift command line interface (
oc) on your cluster's boot node and runoc login. For more information, see the instructions in Getting started with the Red Hat OpenShift CLI.
-
To function properly, distributed platforms and applications such as Red Hat OpenShift and IBM Cloud Pak for AIOps require the system clocks of all of their nodes to be highly synchronized with one another. Discrepancies between the clocks can cause IBM Cloud Pak for AIOps to experience operational issues. All Red Hat OpenShift nodes in the cluster must have access to an NTP server to synchronize their clocks. For more information, see the Red Hat OpenShift documentation
.
-
Optionally configure a custom certificate for IBM Cloud Pak for AIOps to use. You can use either of the following methods:
- Configure a custom certificate for the Red Hat OpenShift cluster. Follow the instructions in the Red Hat OpenShift documentation Replacing the default ingress certificate.
- If you would like to use a custom certificate for the IBM Cloud Pak for AIOps console only, then after installation is complete follow the instructions in Using a custom certificate.
1.3 Set up a target registry
You must have a local Docker type production-grade registry available in the air-gapped environment to store the IBM Cloud Pak for AIOps images in. The registry must meet the following requirements:
- supports Docker Manifest V2, Schema 2.
- supports multi-architecture images.
- is accessible from the Red Hat OpenShift cluster nodes.
- allows path separators in the image name.
- you have the username and password for a user who can read from and write to the registry.
- must have 175 GB of storage to hold all of the software that is to be transferred to the target registry.
If you do not already have a suitable production-grade registry available, then you must install and configure one. For more information, see About the mirror registry
in the Red Hat OpenShift documentation.
Do not use the Red Hat OpenShift image registry as your target registry. The Red Hat OpenShift registry does not support multi-architecture images or path separators in the image name.
1.4 Prepare a host
Prepare a portable compute device, or if you are using a portable storage device then a connected compute device, as follows.
You must be able to connect your host to the internet. Your host must be on a Linux x86_64 or Mac platform with any operating system that the IBM Cloud Pak CLI and the Red Hat OpenShift CLI support. If you are on a Windows® platform, you must run the actions in a Linux x86_64 VM or from a Windows Subsystem for Linux® (WSL) terminal.
Your portable device and any intermediary devices must have 175 GB of storage to hold all the software that is to be transferred to the local target registry.
Complete the following steps on your host.
-
Install Podman.
To install Podman, see the Podman Installation Instructions
.
Note: Docker is not shipped or supported for Red Hat Enterprise Linux (RHEL) 8. The Podman container engine replaced docker as the preferred, maintained, and supported container runtime of choice for Red Hat Enterprise Linux 8 systems. For more information, see Running containers without Dockerin the Red Hat documentation.
-
Install the Red Hat OpenShift CLI tool,
oc.ocis required for Red Hat OpenShift management. For more information, see Getting started with the Red Hat OpenShift CLIin the Red Hat OpenShift documentation.
1.5 Install the IBM Catalog Management Plug-in for IBM Cloud Pak
The IBM Catalog Management Plug-in for IBM Cloud Pak (ibm-pak-plugin) is used for the deployment of IBM Cloud Paks in a disconnected environment. It simplifies the process for discovering required IBM product images and uses standard tools for registry and cluster access. The ibm-pak-plugin also extends the Red Hat OpenShift CLI (oc) capability to streamline the process of delivering installation images to the IBM Cloud Pak in an air-gapped environment.
-
Download and install
ibm-pak-pluginversion1.21.2for your host operating system from github.com/IBM.
-
Run the following command to extract the files.
tar -xf oc-ibm_pak-linux-amd64.tar.gz -
Run the following command to move the file to the
/usr/local/bindirectory.mv oc-ibm_pak-linux-amd64 /usr/local/bin/oc-ibm_pakNote: If you are installing as a non-root user, you must use sudo. -
Confirm that
ibm-pak-pluginis installed by running the following command.oc ibm-pak --helpExpected result: The
ibm-pak-pluginusage is displayed.
2. Download the CASE
Set environment variables on the portable device, and connect it to the internet so that you can download the IBM Cloud Pak for AIOps CASE files.
-
Create the following environment variables.
export CASE_NAME=ibm-cp-waiops export CASE_VERSION=1.19.0 export CASE_INVENTORY_SETUP=cpwaiopsSetup export TARGET_REGISTRY_HOST=<IP_or_FQDN_of_target_registry> export TARGET_REGISTRY_PORT=<port_number_of_target_registry> export TARGET_REPOSITORY='' export TARGET_REGISTRY=$TARGET_REGISTRY_HOST:$TARGET_REGISTRY_PORT if [ -n "$TARGET_REPOSITORY" ]; then export TARGET_REGISTRY=$TARGET_REGISTRY_HOST:$TARGET_REGISTRY_PORT/$TARGET_REPOSITORY; fi export TARGET_REGISTRY_USER=<username> export TARGET_REGISTRY_PASSWORD=<password>The target registry is the registry where the IBM Cloud Pak for AIOps images are mirrored to, and accessed from by the Red Hat OpenShift cluster, as setup in 1.3 Set up a target registry
If you are installing multiple offline instances of IBM Cloud Pak for AIOps and they use the same registry, then you must mirror the images for each instance to different repositories in the registry. Set the value of TARGET_REPOSITORY to a distinct value when you install each instance, for example
cp4aiopsandcp4aiops2. For more information about deploying multiple offline instances of IBM Cloud Pak for AIOps, see Deploying multiple instances on a single cluster.If your portable device must connect to the internet through a proxy, then also set the following environment variables.
export https_proxy=http://proxy-server-hostname:port export http_proxy=http://proxy-server-hostname:port -
Connect your portable device to the internet and disconnect it from the air-gapped environment.
-
Download the IBM Cloud Pak for AIOps installer and image inventory to your portable device.
oc ibm-pak get ${CASE_NAME} --version ${CASE_VERSION}The CASE is downloaded to
~/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION. The logs files are available at ~/.ibm-pak/logs/oc-ibm_pak.log.If the download fails, you can try to increase the default values of the HTTP timeout and maximum HTTP retry attempts as required in the
IBMPAK_HTTP_TIMEOUTandIBMPAK_HTTP_RETRYflags.Note: If you do not specify the CASE version, then the latest CASE is downloaded. The root directory that is used by theibm-pak-pluginis~/.ibm-pak. If required, the root directory can be configured by setting theIBMPAK_HOMEenvironment variable.
3. Mirror images
Complete the following steps to mirror the IBM Cloud Pak for AIOps and IBM Cloud Pak foundational services Cert Manager images from the internet to the file system on your portable device, and then from the file system to the target registry in the air-gapped environment.
3.1. Generate mirror manifests
Run the following command to generate mirror manifests to be used when mirroring the images to the target registry.
oc ibm-pak generate mirror-manifests ${CASE_NAME} file://cpfs --version ${CASE_VERSION} --final-registry ${TARGET_REGISTRY}/cpfs
| Argument | Description |
|---|---|
file://cpfs
|
Specifies the path extension where images are mirrored to. Images are mirrored to $IMAGE_PATH/cpfs when the oc image mirror command is run with images-mapping-to-filesystem.txt. For more information, see Mirror the images to the file system. |
$TARGET_REGISTRY/cpfs
|
This argument generates an image-mapping file that is used by the oc image mirror commands to mirror images to the TARGET_REGISTRY at namespace cpfs. For example, if the CASE you are installing needs the image, quay.io/opencloudio/ibm-events-kafka-2.6.0@sha256:10d422dddd29ff19c87066fc6510eee05f5fa4ff608b87a06e898b3b6a3a13c7, then its final URL in your target registry will be $TARGET_REGISTRY/cpfs/opencloudio/ibm-events-kafka-2.6.0. Note the new namespace of cpfs in the URL. The namespace path can be multi level if your target registry supports it. |
The command generates the following files at ~/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION:
- images-mapping-to-filesystem.txt
- images-mapping-from-filesystem.txt
-
image-content-source-policy.yamlorimage-digest-mirror-set.yaml, depending on whether your Red Hat OpenShift cluster mirror configuration is set up for ImageContentSourcePolicy (ICSP) or ImageDigestMirrorSet (IDMS).
If you see the following warning, it can be safely ignored because the the files mentioned are not applied.
Catalog source collision detected for cloud-native-postgresql-catalog in /Users/tmg/.ibm-pak/data/mirror/ibm-cp-waiops/1.15.0/catalog-sources.yaml. Please remove incorrect version from that file before applying to the cluster
3.2. Authenticate with the IBM Entitled Registry
Log in to the IBM Entitled Registry to generate an authentication file containing the IBM Entitled Registry credentials, and then create an environment variable that has the location of the authentication file. This file is used later to enable the oc image mirror command to pull the images from the IBM Entitled Registry.
-
Get the authentication credentials for the IBM Entitled Registry.
-
To obtain the entitlement key that is assigned to your IBMid, log in to MyIBM Container Software Library
with the IBMid and password details that are associated with the entitled software.
-
In the Entitlement keys section, select Copy key to copy the entitlement key.
-
-
Run the following command to create an environment variable that contains your entitlement key.
export ENTITLED_REGISTRY_PASSWORD=<key>Where
<key>is the entitlement key that you copied in the previous step. -
Store the authentication credentials for the IBM Entitled Registry.
Run the following commands:
podman login cp.icr.io -u cp -p ${ENTITLED_REGISTRY_PASSWORD} export REGISTRY_AUTH_FILE=${XDG_RUNTIME_DIR}/containers/auth.json unset ENTITLED_REGISTRY_PASSWORDThe authentication file is usually at
${XDG_RUNTIME_DIR}/containers/auth.json. For more information, see the Options section in the Podman documentation.
3.3. Mirror the images to the file system
Complete these steps to mirror the images from the internet to a file system on your portable device.
-
Create an environment variable to store the location of the file system where the images are to be stored.
export IMAGE_PATH=<image-path>Where
<image-path>is the directory where you want the images to be stored. -
Run the following command to mirror the images from the IBM Entitled Registry to the file system.
nohup oc image mirror \ -f ~/.ibm-pak/data/mirror/${CASE_NAME}/${CASE_VERSION}/images-mapping-to-filesystem.txt \ -a ${REGISTRY_AUTH_FILE} \ --filter-by-os '.*' \ --insecure \ --skip-multiple-scopes \ --dir "${IMAGE_PATH}" \ --max-per-registry=1 > my-mirror-progress.txt 2>&1 &The UNIX® command
nohupis used to ensure that the mirroring process continues even if there is a loss of network connection, and redirection of output to a file provides improved monitoring and error visibility.Run the following command if you want to see the progress of the mirroring:
tail -f my-mirror-progress.txtIf an error occurs during mirroring, the mirror command can be rerun.
3.4 Setup the file system in the air-gapped environment
-
Copy files to the air-gapped environment (portable storage device only)
If you are using a portable storage device, you must copy the files from the portable storage device to a local compute device in the air-gapped environment that has access to the target registry. If you are using a portable compute device, then these items are already present, and you can proceed to the next step.
Copy the following items to your local compute device:
- the file system located at $IMAGE_PATH, which you specified earlier
-
~/.ibm-pakdirectory
-
Disconnect the device that has your file system, (the portable compute device or the local compute device) from the internet and connect it to the air-gapped environment.
-
Ensure that environment variables are set on the device in the air-gapped environment that has access to the target registry.
If you are using a portable storage device, then set the following environment variables on your local compute device within the air-gapped environment.
If you are using a portable compute device that you have restarted since mirroring the images, then your environment variables will have been lost and you will need to set the following environment variables on your portable compute device again.
export CASE_NAME=ibm-cp-waiops export CASE_VERSION=1.19.0 export CASE_INVENTORY_SETUP=cpwaiopsSetup export TARGET_REGISTRY_HOST=<IP_or_FQDN_of_target_registry> export TARGET_REGISTRY_PORT=<port_number_of_target_registry> export TARGET_REPOSITORY='' export TARGET_REGISTRY=$TARGET_REGISTRY_HOST:$TARGET_REGISTRY_PORT if [ -n "$TARGET_REPOSITORY" ]; then export TARGET_REGISTRY=$TARGET_REGISTRY_HOST:$TARGET_REGISTRY_PORT/$TARGET_REPOSITORY; fi export TARGET_REGISTRY_USER=<username> export TARGET_REGISTRY_PASSWORD=<password> export IMAGE_PATH=<image_path>If you are installing multiple offline instances of IBM Cloud Pak for AIOps and they use the same registry, then you must mirror the images for each instance to different repositories in the registry. Set the value of TARGET_REPOSITORY to a distinct value when you install each instance, for example
cp4aiopsandcp4aiops2. For more information about deploying multiple offline instances of IBM Cloud Pak for AIOps, see Deploying multiple instances on a single cluster.
3.5 Authenticate with the target registry
Authenticate with the target registry in the air-gapped environment that you are mirroring the images into.
Run the following commands:
podman login ${TARGET_REGISTRY} -u ${TARGET_REGISTRY_USER} -p ${TARGET_REGISTRY_PASSWORD}
export REGISTRY_AUTH_FILE=${XDG_RUNTIME_DIR}/containers/auth.json
The authentication file is usually at ${XDG_RUNTIME_DIR}/containers/auth.json. For more information, see the Options section in the Podman documentation
.
3.6 Mirror the images to the target registry from the file system
Complete the steps in this section on the device that has your file system (the portable compute device or the local compute device) to copy the images from the file system to the $TARGET_REGISTRY. Your device with the file system must be connected to both the target registry and the Red Hat OpenShift cluster.
Run the following command to copy the images referenced in the images-mapping-from-filesystem.txt from the $IMAGE_PATH file system to the final target registry.
nohup oc image mirror \
-f ~/.ibm-pak/data/mirror/${CASE_NAME}/${CASE_VERSION}/images-mapping-from-filesystem.txt \
--from-dir "${IMAGE_PATH}" \
-a ${REGISTRY_AUTH_FILE} \
--filter-by-os '.*' \
--insecure \
--skip-multiple-scopes \
--max-per-registry=1 > my-mirror-progress2.txt 2>&1 &
The UNIX command nohup is used to ensure that the mirroring process continues even if there is a loss of network connection, and redirection of output to a file provides improved monitoring and error visibility.
Run the following command if you want to see the progress of the mirroring:
tail -f my-mirror-progress2.txt
If an error occurs during mirroring, the mirror command can be rerun.
3.7 Configure the cluster
-
Log in to your Red Hat OpenShift cluster.
You can identify your specific
oc logincommand by clicking the user menu in the upper left of the Red Hat OpenShift console, and then clicking Copy Login Command.Example:
oc login <server> -u <cluster username> -p <cluster pass> -
Update the global image pull secret for your Red Hat OpenShift cluster.
Follow the steps in the Red Hat OpenShift documentation topic Updating the global cluster pull secret
.
These steps enable your cluster to have authentication credentials in place to pull images from your TARGET_REGISTRY as specified in the
image-content-source-policy.yamlorimage-digest-mirror-set.yaml, which you will apply to your cluster in the next step. -
Configure image mirroring using either
ImageContentSourcePolicy(ICSP) orImageDigestMirrorSet(IDMS).Select one of the following two options:
-
Option A: IDMS - Use for new installations of IBM Cloud Pak for AIOps, and for the upgrade of IBM Cloud Pak for AIOps deployments on Red Hat OpenShift clusters that use IDMS.
-
Option B: ICSP - Use for the upgrade of IBM Cloud Pak for AIOps deployments on Red Hat OpenShift clusters that use ICSP.
Red Hat OpenShift have deprecated the use of ICSP for repository mirroring in v4.14 and higher. ICSP is still supported by v4.14-4.19, but support may be removed in future Red Hat OpenShift releases.
Option A: IDMS
-
Create the
ImageDigestMirrorSet.oc apply -f ~/.ibm-pak/data/mirror/${CASE_NAME}/${CASE_VERSION}/image-digest-mirror-set.yaml -
Verify that the
ImageDigestMirrorSetresource is created.oc get imagedigestmirrorset
Option B: ICSP
-
Create the
ImageContentSourcePolicy.oc apply -f ~/.ibm-pak/data/mirror/${CASE_NAME}/${CASE_VERSION}/image-content-source-policy.yaml -
Verify that the
ImageContentSourcePolicyresource is created.oc get imagecontentsourcepolicy
-
-
Verify your cluster node status.
oc get MachineConfigPool -wImportant: After the application of the global image pull secret and eitherImageContentSourcePolicyorImageDigestMirrorSet, the configuration of your nodes will be updated sequentially. Wait until all theMachineConfigPoolsare updated before proceeding to the next step. -
(Optional) If you use an insecure registry, you must add the target registry to the cluster's
insecureRegistrieslist.oc patch image.config.openshift.io/cluster --type=merge \ -p '{"spec":{"registrySources":{"insecureRegistries":["'${TARGET_REGISTRY}'"]}}}'
4. Configure storage
The storage configuration must satisfy your sizing requirements. Two storage classes are needed for installing IBM Cloud Pak for AIOps. For more information, see Storage.
RWX-storage-class-name in the installation instance CR YAML file. This configuration cannot be changed after IBM Cloud Pak for AIOps is installed.5. Install IBM Cloud Pak for AIOps
Now that the images are mirrored to your air-gapped environment, you can deploy IBM Cloud Pak for AIOps to that environment. To install IBM Cloud Pak for AIOps, complete the following steps.
- 5.1 Create environment variables
- 5.2 Create a custom project
- 5.3 Create the catalog source
- 5.4 Evaluate storage performance
- 5.5 Configure usage data collection
- 5.6 Install Cert Manager
- 5.7 Verify cluster readiness
- 5.8 Install the operator
- 5.9 Install IBM Cloud Pak for AIOps
- 5.10 Set the Postgres operator replica count
- 5.11 Verify local storage
- 5.12 Create an EgressFirewall
- 5.13 Access the Cloud Pak for AIOps console
5.1 Create environment variables
Create a shell script that defines the environment variables that will be used to provide installation parameters for your deployment. Use the following codeblock as a template, replacing the brackets < ... > with values for your environment. Name the script waiops_var_<project>.sh, where <project> is the value that you used for PROJECT_CP4AIOPS. For example, waiops_var_cp4aiops.sh.
If you are deploying more than one instance of IBM Cloud Pak for AIOps in the same cluster, then each instance must have a different value for PROJECT_CP4AIOPS. If you already have a deployment of IBM Cloud Pak for AIOps in the namespace cp4aiops, then export PROJECT_CP4AIOPS with a different value, such as cp4aiops2. For more information, see Deploying multiple instances on a single cluster.
#============================================================================================================
# Cloud Pak for AIOps installation variables
#============================================================================================================
export CP4AIOPS_NAME=ibm-cp-aiops
export CP4AIOPS_SIZE=large # Set to small if you only require a starter non-production deployment.
export SECURE_TUNNEL=false # Set to `true` to install Secure Tunnel, otherwise set to `false`.
export PROJECT_CP4AIOPS=cp4aiops
export ACCEPT_LICENSE=false # Set to `true` to agree to the license terms, otherwise install will fail.
export CATALOG_SRC_CP4AIOPS=ibm-aiops-catalog
# -----------------------------------------------------------------------------------------------------------
# Incremental adoption - set your deployment type.
# Set to `true` to install an extended deployment with log anomaly detection and ticket analysis capabilities
# Set to `false` to install a base deployment without log anomaly detection and ticket analysis capabilities
# -----------------------------------------------------------------------------------------------------------
export LOG_ANOMALY=false
# -----------------------------------------------------------------------------------------------------------
# Persistent storage
# -----------------------------------------------------------------------------------------------------------
export STG_CLASS=<RWX-storage-class-name>
export STG_CLASS_BLOCK=<RWO-storage-class-name>
# -------------------------------------------------------------------------------------------------------
# Local storage
#
# If you are not using hybrid storage, do not edit the environment variables in this section.
#
# If you are using hybrid storage, set the environment variables in this section to the LVM storage classes
# that you configured for Postgres and Kafka.
#
# For example:
# export IR_CORE_POSTGRES_LOCAL_STORAGE_CLASS=lvms-vg-ir-core-postgres-1
# export TOPOLOGY_POSTGRES_LOCAL_STORAGE_CLASS=lvms-vg-topology-postgres-1
# export KAFKA_LOCAL_STORAGE_CLASS=lvms-vg-kafka-1
#-------------------------------------------------------------------------------------------------------
export IR_CORE_POSTGRES_LOCAL_STORAGE_CLASS=''
export TOPOLOGY_POSTGRES_LOCAL_STORAGE_CLASS=''
export KAFKA_LOCAL_STORAGE_CLASS=''
# -------------------------------------------------------------------------------------------------------
# Your customer details
# -------------------------------------------------------------------------------------------------------
export CUSTOMER_NAME=<your company name>
export CUSTOMER_ICN=<your IBM Customer Number>
export CUSTOMER_ENVIRONMENT=<Set to `trial`, `poc`, or `production`>
# -------------------------------------------------------------------------------------------------------
# `OwnNamespace` installation mode: leave INSTALL_MODE_NAMESPACE as it is.
# `AllNamespaces` installation mode: change to export INSTALL_MODE_NAMESPACE=openshift-operators
# -------------------------------------------------------------------------------------------------------
export INSTALL_MODE_NAMESPACE=${PROJECT_CP4AIOPS}
# -------------------------------------------------------------------------------------------------------
# Topology resource group terminology
# Specify `application` or `service` as the terminology to be used for collections of topology resource
# groups. The default is `application`.
# -------------------------------------------------------------------------------------------------------
export TOPOLOGY_TERMINOLOGY=application
If you need help with deciding on the values to set for these environment variables, see the following topics.
- LOG_ANOMALY: Incremental adoption
- STG_CLASS and STG_CLASS_BLOCK: Storage class summary table
- CP4AIOPS_SIZE: Sizing
- SECURE_TUNNEL: Secure Tunnel
- INSTALL_MODE_NAMESPACE: Operator installation mode
- KAFKA_LOCAL_STORAGE_CLASS, IR_CORE_POSTGRES_LOCAL_STORAGE_CLASS and TOPOLOGY_POSTGRES_LOCAL_STORAGE_CLASS: Hybrid storage
You can update your deployment type after installation. For more information, see Updating the deployment type.
You can set a different value for $PROJECT_CP4AIOPS and $CP4AIOPS_NAME if you want. However, you must not use the default, kube-system, kube-public, openshift-node, openshift-infra, or openshift projects (namespaces) for $PROJECT_CP4AIOPS. This is because IBM Cloud Pak for AIOps uses Security Context Constraints (SCC), and SCCs cannot be assigned to pods created in one of the default Red Hat OpenShift projects (namespaces).
Run the following command to source your script and set the environment variables:
. ./waiops_var_<project>.sh
Where <project> is the value that you used for PROJECT_CP4AIOPS.
5.2 Create a custom project (namespace)
-
Run the following command to create a project (namespace) to deploy IBM Cloud Pak for AIOps into.
oc create namespace ${PROJECT_CP4AIOPS} -
Add a node-selector annotation to the IBM Cloud Pak for AIOps namespace.
The annotation ensures that on a multi-architecture Red Hat OpenShift cluster, IBM Cloud Pak for AIOps workloads are only scheduled on nodes that have an architecture that IBM Cloud Pak for AIOps supports.
Failure to do so might result in the scheduling and subsequent failure of IBM Cloud Pak for AIOps workloads on Red Hat OpenShift nodes that have a nonsupported architecture. For more information about supported architectures, see Supported platforms.
Run one of the following commands.
If you want to use amd64 architecture:
oc annotate namespace "${PROJECT_CP4AIOPS}" openshift.io/node-selector="kubernetes.io/arch=amd64"If you want to use s390x architecture:
oc annotate namespace "${PROJECT_CP4AIOPS}" openshift.io/node-selector="kubernetes.io/arch=s390x"
5.3 Create the catalog sources
-
Run the following command to create the catalog sources for IBM Cloud Pak for AIOps and IBM Cloud Pak foundational services Cert Manager.
cat << EOF | oc apply -f - apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: ${CATALOG_SRC_CP4AIOPS} namespace: ${INSTALL_MODE_NAMESPACE} spec: displayName: ${CATALOG_SRC_CP4AIOPS} publisher: IBM Content sourceType: grpc image: icr.io/cpopen/ibm-aiops-catalog@sha256:294adebdcbfb1dec82d598b4b8439c40ea51e308548207a537ff69bfdca75701 grpcPodConfig: securityContextConfig: restricted --- apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: ibm-cert-manager-catalog namespace: openshift-marketplace spec: displayName: ibm-cert-manager publisher: IBM sourceType: grpc image: icr.io/cpopen/ibm-cert-manager-operator-catalog@sha256:97610d00d5b46b4de8d3c98233591cc554b7211d96f3d30ed935a84f076e3b65 EOF -
Verify that the ibm-aiops-catalog and ibm-cert-manager-catalog
CatalogSourceobjects are in the output that is returned by the following command:oc get CatalogSources -n openshift-marketplace oc get CatalogSource -n ${INSTALL_MODE_NAMESPACE}Example output:
oc get CatalogSources -n openshift-marketplace NAME DISPLAY TYPE PUBLISHER AGE ibm-cert-manager-catalog ibm-cert-manager grpc IBM 2m oc get CatalogSource -n cp4aiops NAME DISPLAY TYPE PUBLISHER AGE ibm-aiops-catalog ibm-aiops-catalog grpc IBM 2m
5.4 Evaluate storage performance
Use the following procedure to evaluate whether your storage performance is sufficient to withstand the demands of a production deployment of IBM Cloud Pak for AIOps.
-
Download and install the
aiopsctlcommand line interface tool if you haven't already done so.AIOPSCTL_TAR=<aiopsctl_tar> AIOPSCTL_INSTALL_URL="https://github.com/IBM/aiopsctl/releases/download/v4.13.0/${AIOPSCTL_TAR}" curl -LO "${AIOPSCTL_INSTALL_URL}" tar xf "${AIOPSCTL_TAR}" mv aiopsctl /usr/local/bin/aiopsctlWhere
<aiopsctl_tar>is the operating system specific file that you require from the following set:aiopsctl-linux_s390x.tar.gz,aiopsctl-linux_arm64.tar.gz,aiopsctl-linux_amd64.tar.gz,aiopsctl-darwin_amd64.tar.gz,aiopsctl-darwin_arm64.tar.gz. -
Evaluate distributed network storage.
Run the following command:
aiopsctl benchmark storage --namespace=${PROJECT_CP4AIOPS} --storage-class=${STG_CLASS_BLOCK}The tool selects a node in your cluster and benchmarks the performance of the node's storage. The process takes around 8 minutes to run.
If you think that the storage performance between your nodes varies significantly, then you can use the
--node <node_name>argument to pass in the name of the node that you want the tool to run on. -
Verify that your benchmarking results meet or exceed the required metrics.
The following table identifies the storage performance metrics that must be achieved to support a deployment of IBM Cloud Pak for AIOps. If your deployment is custom-sized to support higher rates than the default production rates listed in Processing abilities, then your storage performance must exceed these metrics.
Metric Read Write Minimum sequential IOPS (higher is better, lower is worse) 5000 5000 Minimum sequential bandwidth (higher is better, lower is worse) 20 Mi/sec 20 Mi/sec Maximum average sequential latency (lower is better, higher is worse) 500 usec 1000 usec These metrics are not applicable if you are using native storage in a public cloud provider such as Amazon Elastic Block Store (EBS) or IBM Cloud Block Storage. Performance will be adequate if you selected the native storage classes documented for your cloud platform.
-
Evaluate local storage.
If your IBM Cloud Pak for AIOps deployment does not use hybrid storage then skip this step.
If your IBM Cloud Pak for AIOps deployment is using hybrid storage, then evaluate the storage performance for Postgres and Kafka, by passing in the Postgres and Kafka storage classes, and verifying that the results meet or exceed the required metrics in the previous step.
For example:
aiopsctl benchmark storage --namespace=${PROJECT_CP4AIOPS} --storage-class=lvms-vg-kafka-1 aiopsctl benchmark storage --namespace=${PROJECT_CP4AIOPS} --storage-class=lvms-vg-ir-core-postgres-1 aiopsctl benchmark storage --namespace=${PROJECT_CP4AIOPS} --storage-class=lvms-vg-topology-postgres-1
5.5 Configure usage data collection
To help the development of IBM Cloud Pak for AIOps, daily aggregated usage data is collected to analyse how IBM Cloud Pak for AIOps is used. The collection of usage data is enabled by default, but can be disabled. Usage data is collected by the cp4waiops-metricsprocessor pod, and stored in the cp4waiops-metricsprocessor pod's logs. This usage data is sent to IBM when MustGather output is sent to IBM, as the MustGather includes the output from the cp4waiops-metricsprocessor pod's logs. The usage data is then sent to and stored in IBM controlled GDPR-compliant systems. The usage data that is collected is numeric, or is about the deployment type and platform. It does not include email addresses, passwords, or specific details. Only the following data is collected:
- Current number of applications
- Current number of alerts (all severities aggregated)
- Current number of incidents (all priorities aggregated)
- Current number of policies (includes predefined and user created)
- Current number of runbooks run since installation
- Current number of integrations of each type (For example ServiceNow, Instana, Falcon Logscale)
- Secure tunnel enablement: whether connection (which controls whether you can create a secure tunnel) is enabled in the Installation custom resource
- Deployment type: base deployment or extended deployment
- Deployment platform: Red Hat OpenShift Container Platform or Linux
This usage data is sent to IBM only if the MustGather tool is run and sent to IBM. The MustGather includes the output from the cp4waiops-metricsprocessor pod's logs.
Configuring the collection of usage data
If you do not want to disable the collection of usage data, run the following command to configure the usage data with your customer details.
oc create secret generic aiops-metrics-processor -n ${PROJECT_CP4AIOPS} --from-literal=customerName=${CUSTOMER_NAME} --from-literal=customerICN=${CUSTOMER_ICN} --from-literal=environment=${CUSTOMER_ENVIRONMENT}
Usage data without your customer details is still collected even if you do not create this secret. If you do not want any usage data collected, then you must run the command given in Disabling the collection of usage data.
Disabling the collection of usage data
If you want to disable the collection of usage data, run the following command.
oc create secret generic aiops-metrics-processor -n ${PROJECT_CP4AIOPS} --from-literal=customerName=${CUSTOMER_NAME} --from-literal=customerICN=${CUSTOMER_ICN} --from-literal=environment=${CUSTOMER_ENVIRONMENT} --from-literal=enableCollection=false
You can update your usage data collection preferences after installation. For more information, see Updating usage data collection preferences.
5.6 Install Cert Manager
Skip this step if you already have a certificate manager installed on the Red Hat OpenShift cluster that you are installing IBM Cloud Pak for AIOps on. If you do not have a certificate manager then you must install one.
The IBM Cloud Pak foundational services Cert Manager is recommended. For more information about IBM Cloud Pak foundational services Cert Manager hardware requirements, see IBM Certificate Manager (cert-manager) hardware requirements
in the IBM Cloud Pak foundational services documentation.
The Red Hat OpenShift Cert Manager is also supported. For more information, see cert-manager Operator for Red Hat OpenShift in the Red Hat OpenShift documentation.
The IBM Cloud Pak foundational services Cert Manager can be installed with the following steps.
-
Run the following command to create the resource definitions that you need:
cat << EOF | oc apply -f - apiVersion: v1 kind: Namespace metadata: name: ibm-cert-manager --- apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: ibm-cert-manager-operator-group namespace: ibm-cert-manager --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: ibm-cert-manager-operator namespace: ibm-cert-manager spec: channel: v4.2 installPlanApproval: Automatic name: ibm-cert-manager-operator source: ibm-cert-manager-catalog sourceNamespace: openshift-marketplace EOF -
Run the following command to ensure that the IBM Cloud Pak foundational services Cert Manager pods have a STATUS of Running before proceeding to the next step.
oc -n ibm-cert-manager get podsExample output for a successful IBM Cloud Pak foundational services Cert Manager installation:
NAME READY STATUS RESTARTS AGE cert-manager-cainjector-674854c49d-vstq4 1/1 Running 0 8d cert-manager-controller-646d4bd6fd-zwmqm 1/1 Running 0 8d cert-manager-webhook-8598787c8-s4lkt 1/1 Running 0 8d ibm-cert-manager-operator-c96957695-dkxnm 1/1 Running 0 8d
5.7 Verify cluster readiness
Run the following command to verify whether your environment is correctly set up for an IBM Cloud Pak for AIOps installation.
aiopsctl server precheck -n ${PROJECT_CP4AIOPS}
- If you have a multi-zone cluster, then also specify the
-mflag. This flag enables extra checks to help ensure that the cluster has sufficient resources to withstand a zone outage, and that the zones are well balanced for memory and CPU. For example,aiopsctl server precheck -n cp4aiops -m. - If you are using hybrid storage, then also specify the
--hybrid-storageflag. This flag enables extra checks to help ensure that sufficient local storage is configured. For example,aiopsctl server precheck -n cp4aiops --hybrid-storage
# aiopsctl server precheck -n cp4aiops
o- [25 Mar 26 16:08 GMT] Running precheck tool
o- [25 Mar 26 16:08 GMT] Checking hardware resources...
Total Node Count (Available Schedulable / Required): 17/6
Production (HA) Base CPU (vCPU): 264 / 143
Production (HA) Base Memory (GB): 539 / 331
Production (HA) Extended CPU (vCPU): 264 / 170
Production (HA) Extended Memory (GB): 539 / 391
You have enough resources for 1 instance(s) of large Base install
You have enough resources for 1 instance(s) of large Extended install
Total Node Count (Available Schedulable / Required): 17/3
Starter (Non-HA) Base CPU (vCPU): 264 / 47
Starter (Non-HA) Base Memory (GB): 539 / 123
Starter (Non-HA) Extended CPU (vCPU): 264 / 55
Starter (Non-HA) Extended Memory (GB): 539 / 136
You have enough resources for 5 instance(s) of small Base install
You have enough resources for 4 instance(s) of small Extended install
minimum requirements met for both starter and production
o- [25 Mar 26 16:08 GMT] Checking storage...
Required StorageClasses found for provider Red Hat OpenShift Data Foundation: ocs-storagecluster-cephfs, ocs-storagecluster-ceph-rbd
Checking if PVC can bind to supported storage class
[Attempt 1 of 14] PVC aiops-prereq-storage-test-pvc-ocs-storagecluster-cephfs is in phase: Pending
PVC aiops-prereq-storage-test-pvc-ocs-storagecluster-cephfs is successfully bound to storage class ocs-storagecluster-cephfs
[Attempt 1 of 14] PVC aiops-prereq-storage-test-pvc-ocs-storagecluster-ceph-rbd is in phase: Pending
PVC aiops-prereq-storage-test-pvc-ocs-storagecluster-ceph-rbd is successfully bound to storage class ocs-storagecluster-ceph-rbd
Storage check passed
o- [25 Mar 26 16:08 GMT] Checking OCP Version...
[WARN] Heterogeneous architecture clusters are not being supported at this time
Cluster meets OCP version requirements
o- [25 Mar 26 16:08 GMT] Checking if Cert Manager is present...
Certificate CustomResourceDefinition Found
o- [25 Mar 26 16:08 GMT] Checking if certs will expire within 4 days...
o- [25 Mar 26 16:08 GMT] Precheck Summary Results
Check Result
Meets Hardware Requirements Passed
No Storage Issues Passed
Meets OCP Version Requirement Passed
Cert Mgr Operator Exists Passed
Certificates Valid Passed
- The
"You have enough resources for <...>"statements denote the number of instances that the cluster can support, and include any existing instances. The number that is given is for the number of base instances or extended instances. It does not mean that the cluster can support the stated number of base instances and the stated number of extended instances. - If you are not using IBM Cloud Pak foundational services Cert Manager, then ignore any errors that are returned by the Cert Manager check.
5.8 Install the operator
For more information about installing operators, see Adding Operators
to a cluster in the Red Hat OpenShift documentation.
For more information about the operators which are installed with IBM Cloud Pak for AIOps, see Operator Details.
-
Create an OperatorGroup.
Skip this step if you are installing using the 'All Namespaces' installation mode. Check that you set
INSTALL_MODE_NAMESPACEcorrectly in step 5.1, and proceed to the next step, Install the IBM Cloud Pak for AIOps operator.If you are installing using the 'OwnNamespace' installation mode, then you must create an operator group in your custom project (namespace), or the IBM Cloud Pak for AIOps operator will not install. There might be an operator group for managing a namespace for given APIs. If there is an operator group for the namespace, do not create a second one.
Create the Operator group by running the following command:
cat << EOF | oc apply -f - apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: cp4aiops-operator-group namespace: ${PROJECT_CP4AIOPS} spec: targetNamespaces: - "${PROJECT_CP4AIOPS}" EOF -
Install the IBM Cloud Pak for AIOps operator.
Run the following command.
cat << EOF | oc apply -f - apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: ibm-aiops-orchestrator namespace: $INSTALL_MODE_NAMESPACE spec: channel: v4.13 installPlanApproval: Automatic name: ibm-aiops-orchestrator source: ${CATALOG_SRC_CP4AIOPS} sourceNamespace: ${INSTALL_MODE_NAMESPACE} EOFinstallPlanApproval must not be changed to Manual. Manual approval, which requires the manual review and approval of the generated InstallPlans, is not supported. Incorrect timing or ordering of manual approvals of InstallPlans can result in a failed installation.
-
After a few minutes, the IBM Cloud Pak for AIOps operator is installed. Verify that the all of the components have a state of
Succeededby running the following command:oc get csv -n ${INSTALL_MODE_NAMESPACE} | egrep "ibm-aiops-orchestrator"Example output:
$ oc get csv -n ${INSTALL_MODE_NAMESPACE} | egrep "ibm-aiops-orchestrator" ibm-aiops-orchestrator.v4.13.0 IBM Cloud Pak for AIOps 4.13.0 Succeeded
5.9 Install IBM Cloud Pak for AIOps
Create an instance of the IBM Cloud Pak for AIOps custom resource. A maximum of one IBM Cloud Pak for AIOps custom resource is allowed per namespace.
-
Use one of the following YAML codeblocks to create an instance of the IBM Cloud Pak for AIOps custom resource, depending on whether you are installing on a multi-zone cluster.
Installing on a non multi-zone cluster:
cat << EOF | oc apply -f - apiVersion: orchestrator.aiops.ibm.com/v1alpha1 kind: Installation metadata: name: ${CP4AIOPS_NAME} namespace: ${PROJECT_CP4AIOPS} spec: size: ${CP4AIOPS_SIZE} storage: aiops-ir-core-postgres: storageClass: ${IR_CORE_POSTGRES_LOCAL_STORAGE_CLASS} aiops-ir-core-postgres-wal: storageClass: ${IR_CORE_POSTGRES_LOCAL_STORAGE_CLASS} aiops-topology-postgres: storageClass: ${TOPOLOGY_POSTGRES_LOCAL_STORAGE_CLASS} aiops-topology-postgres-wal: storageClass: ${TOPOLOGY_POSTGRES_LOCAL_STORAGE_CLASS} data-iaf-system-kafka: storageClass: ${KAFKA_LOCAL_STORAGE_CLASS} storageClass: ${STG_CLASS} storageClassLargeBlock: ${STG_CLASS_BLOCK} imagePullSecret: topologyModel: ${TOPOLOGY_TERMINOLOGY} license: accept: ${ACCEPT_LICENSE} pakModules: - name: aiopsFoundation enabled: true - name: applicationManager enabled: true - name: aiManager enabled: true - name: connection enabled: ${SECURE_TUNNEL} - name: logAnomalyDetection enabled: ${LOG_ANOMALY} EOFInstalling on a multi-zone cluster:
cat << EOF | oc apply -f - apiVersion: orchestrator.aiops.ibm.com/v1alpha1 kind: Installation metadata: name: ${CP4AIOPS_NAME} namespace: ${PROJECT_CP4AIOPS} spec: size: ${CP4AIOPS_SIZE} storage: aiops-ir-core-postgres: storageClass: ${IR_CORE_POSTGRES_LOCAL_STORAGE_CLASS} aiops-ir-core-postgres-wal: storageClass: ${IR_CORE_POSTGRES_LOCAL_STORAGE_CLASS} aiops-topology-postgres: storageClass: ${TOPOLOGY_POSTGRES_LOCAL_STORAGE_CLASS} aiops-topology-postgres-wal: storageClass: ${TOPOLOGY_POSTGRES_LOCAL_STORAGE_CLASS} data-iaf-system-kafka: storageClass: ${KAFKA_LOCAL_STORAGE_CLASS} storageClass: ${STG_CLASS} storageClassLargeBlock: ${STG_CLASS_BLOCK} imagePullSecret: topologyModel: ${TOPOLOGY_TERMINOLOGY} license: accept: ${ACCEPT_LICENSE} pakModules: - name: aiopsFoundation enabled: true - name: applicationManager enabled: true - name: aiManager enabled: true - name: connection enabled: ${SECURE_TUNNEL} - name: logAnomalyDetection enabled: ${LOG_ANOMALY} zones: - name: <zone_name1> - name: <zone_name2> - name: <zone_name3> EOFWhere
<zone_name1>,<zone_name2>, and<zone_name3>exactly match the zone labels that you applied to each of your nodes in step 1a of Installing IBM Cloud Pak for AIOps on a multi-zone architecture (multi-zone HA).The pakModules
aiopsFoundation,applicationManager, andaiManagermust be enabled as in the preceding YAML. Do not change these values to false. -
Verify your installation.
Run the following command to check that the PHASE of your installation is
Updating.oc get installations.orchestrator.aiops.ibm.com -n ${PROJECT_CP4AIOPS}Example output:NAME PHASE LICENSE STORAGECLASS STORAGECLASSLARGEBLOCK AGE ibm-cp-aiops Updating Accepted rook-cephfs rook-ceph-block 3mIt takes around 60-90 minutes for the installation to complete (subject to the speed with which images can be pulled). When installation is complete and successful, the PHASE of your installation changes to
Running. If your installation phase does not change toRunning, then use the following command to find out which components are not ready:oc get installation.orchestrator.aiops.ibm.com -o yaml -n ${PROJECT_CP4AIOPS} | grep 'Not Ready'Example output:lifecycleservice: Not Ready zenservice: Not ReadyTo see details about why a component is
Not Readyrun the following command, where<component>is the component that is not ready, for examplezenservice.oc get <component> -o yaml -n ${PROJECT_CP4AIOPS}(Optional) If you downloaded the status checker script earlier in step 1.1 Download documentation and scripts for offline access, then you can also run this script to see information about the status of your deployment.
If the installation fails, or is not complete and is not progressing, then see Troubleshooting installation and upgrade and Known Issues to help you identify any installation problems.
5.10 Set the Postgres operator replica count
Set the replica count for the Postgres operator to 2 to help ensure high availability.
-
Run the following command:
oc patch csv cloud-native-postgresql.v1.25.5 --type json -p '[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/replicas", "value": 2}]' -n ${PROJECT_CP4AIOPS}Example output:
clusterserviceversion.operators.coreos.com/cloud-native-postgresql.v1.25.5 patched - Run the following command to verify that there are 2 Postgres replicas:
oc wait deployment/postgresql-operator-controller-manager-1-25-5 --for=jsonpath='{.status.readyReplicas}'=2 -n ${PROJECT_CP4AIOPS}After a few seconds, you will see the following output if the patch command was successful:deployment.apps/postgresql-operator-controller-manager-1-25-5 condition metIf the command is not successful, the following output is displayed after 30 seconds. Contact IBM Support.error: timed out waiting for the condition on deployments/postgresql-operator-controller-manager-1-25-5
5.11 Verify local storage
If your IBM Cloud Pak for AIOps deployment does not use hybrid storage then skip this step.
If your IBM Cloud Pak for AIOps deployment uses hybrid storage, verify that local storage has been correctly created for Postgres and Kafka.
-
Verify IR Core Postgres local storage.
-
Run the following command to verify that IR Core Postgres pods are scheduled on the nodes that you configured for Postgres local storage.
oc get pod -l "k8s.enterprisedb.io/cluster=aiops-ir-core-postgres" -o wide -n ${PROJECT_CP4AIOPS}Example output, where Postgres pods are scheduled on nodes named worker9, worker10 and worker11.NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES aiops-ir-core-postgres-1 1/1 Running 0 25m 10.254.68.38 worker11.example.com <none> <none> aiops-ir-core-postgres-2 1/1 Running 0 23m 10.254.60.30 worker9.example.com <none> <none> aiops-ir-core-postgres-3 1/1 Running 0 22m 10.254.36.87 worker10.example.com <none> <none>Keep a copy of this output that maps PVCs to nodes so that you have this information available if you need to restore your deployment. Add this information to the file that you created for step 2.4 of Hybrid storage.
-
Run the following command to verify that logical volumes have been created for IR Core Postgres.
oc get pvc -l "k8s.enterprisedb.io/cluster=aiops-ir-core-postgres" -o wide -n ${PROJECT_CP4AIOPS}Example output:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE VOLUMEMODE aiops-ir-core-postgres-1 Bound pvc-d4e70a18-d791-4d6f-9c52-af8235417265 50Gi RWO lvms-vg-ir-core-postgres-1 <unset> 38m Filesystem aiops-ir-core-postgres-1-wal Bound pvc-271f246d-8726-4823-9ab7-26fd0622f50b 10Gi RWO lvms-vg-ir-core-postgres-1 <unset> 38m Filesystem aiops-ir-core-postgres-2 Bound pvc-f3e0c1f1-7e18-4d60-895e-623b9e9b4cc1 50Gi RWO lvms-vg-ir-core-postgres-1 <unset> 39m Filesystem aiops-ir-core-postgres-2-wal Bound pvc-362e47cd-4124-4f95-995f-55e216b804b4 10Gi RWO lvms-vg-ir-core-postgres-1 <unset> 39m Filesystem aiops-ir-core-postgres-3 Bound pvc-b08d6313-d262-4e9a-baf7-09b6fdbacf1c 50Gi RWO lvms-vg-ir-core-postgres-1 <unset> 39m Filesystem aiops-ir-core-postgres-3-wal Bound pvc-6bbb154e-1845-44b5-a92c-00f22b41e1c5 10Gi RWO lvms-vg-ir-core-postgres-1 <unset> 39m Filesystem
-
-
Verify Topology Postgres local storage.
-
Run the following command to verify that the Topology Postgres pods are scheduled on the nodes that you configured for Postgres local storage.
oc get pod -l "k8s.enterprisedb.io/cluster=aiops-topology-postgres" -o wide -n ${PROJECT_CP4AIOPS}Example output, where Postgres pods are scheduled on nodes named worker9, worker10 and worker11.NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES aiops-topology-postgres-1 1/1 Running 0 83m 10.254.52.70 worker12.example.com <none> <none> aiops-topology-postgres-2 1/1 Running 0 93m 10.254.56.20 worker13.example.com <none> <none> aiops-topology-postgres-3 1/1 Running 0 86m 10.254.12.52 worker14.example.com <none> <none>Keep a copy of this output that maps PVCs to nodes so that you have this information available if you need to restore your deployment. Add this information to the file that you created for step 2.4 of Hybrid storage.
-
Run the following command to verify that logical volumes have been created for topology Postgres.
oc get pvc -l "k8s.enterprisedb.io/cluster=aiops-topology-postgres" -o wide -n ${PROJECT_CP4AIOPS}Example output:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE VOLUMEMODE aiops-topology-postgres-1 Bound pvc-b2b29420-a0b8-469a-99e4-94b359487417 50Gi RWO lvms-vg-topology-postgres-1 <unset> 95m Filesystem aiops-topology-postgres-1-wal Bound pvc-6bbb154e-1845-44b5-a92c-00f22b41e1c5 10Gi RWO lvms-vg-topology-postgres-1 <unset> 95m Filesystem aiops-topology-postgres-2 Bound pvc-3bb7fca3-f722-4bba-9ed9-ec2d46279223 50Gi RWO lvms-vg-topology-postgres-1 <unset> 95m Filesystem aiops-topology-postgres-2-wal Bound pvc-5cbbe8e9-7472-4e70-826b-89e22bea4250 10Gi RWO lvms-vg-topology-postgres-1 <unset> 95m Filesystem aiops-topology-postgres-3 Bound pvc-d6de7642-dc24-42b9-bd87-11d7b882cfc6 50Gi RWO lvms-vg-topology-postgres-1 <unset> 95m Filesystem aiops-topology-postgres-3-wal Bound pvc-5a76bd34-c570-4118-9316-33398cb83a16 10Gi RWO lvms-vg-topology-postgres-1 <unset> 95m Filesystem
-
-
Verify Kafka local storage.
-
Run the following command to verify that Kafka pods are scheduled on the nodes that you configured for Kafka local storage.
oc get pod -l ibmevents.ibm.com/name=iaf-system-kafka -o wide -n ${PROJECT_CP4AIOPS}Example output, where Kafka pods are scheduled on nodes named worker6, worker7 and worker8.NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES iaf-system-kafka-0 1/1 Running 0 118m 10.123.24.14 worker6.example.com <none> <none> iaf-system-kafka-1 1/1 Running 0 118m 10.123.28.12 worker7.example.com <none> <none> iaf-system-kafka-2 1/1 Running 0 118m 10.123.56.27 worker8.example.com <none> <none>Keep a copy of this output that maps PVCs to nodes so that you have this information available if you need to restore your deployment. Add this information to the file that you created for step 2.4 of Hybrid storage.
-
Run the following command to verify that logical volumes have been created for Kafka.
oc get pvc -l ibmevents.ibm.com/name=iaf-system-kafka -n ${PROJECT_CP4AIOPS}Example output:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-iaf-system-kafka-0 Bound pvc-fc36a8c8-7b73-473b-86a4-d4686228d521 100Gi RWO lvms-vg-kafka-1 124m data-iaf-system-kafka-1 Bound pvc-2f653468-95a0-4d5b-b698-64023e25f643 100Gi RWO lvms-vg-kafka-1 124m data-iaf-system-kafka-2 Bound pvc-c477e794-c825-4b58-af68-8a40b290126d 100Gi RWO lvms-vg-kafka-1 124m
-
5.12 Create an EgressFirewall
There is no egress firewall policy defined when you install IBM Cloud Pak for AIOps, so outgoing traffic from workload pods to the internal and external network is unrestricted.
To create a more secure environment, use the following steps.
-
Create an
EgressFirewallon your Red Hat OpenShift cluster to limit egress from the IBM Cloud Pak for AIOps project (namespace).For more information about creating an
EgressFirewall, see Configuring an egress firewall for a project.
There must be only one
EgressFirewallper project/namespace. -
Configure exceptions to the
EgressFirewall.Edit your
EgressFirewallto add exceptions for the following IBM Cloud Pak for AIOps components that have egress dependencies, otherwise these IBM Cloud Pak for AIOps components fail when they attempt egress.- Allow egress to any external services, such as the following integrations:
- Kubernetes
- GitHub
- Microsoft® Teams
- ServiceNow
- Slack
- VMware® vCenter
-
Configure your
EgressFirewallto allow traffic for your GitHub, Kubernetes, ServiceNow, and VMware vCenter integrations.Edit yourSubstitute values for `dnsName` and `cidrSelector` that are the DNS names and addresses of your GitHub, Kubernetes, ServiceNow, or VMware vCenter sources.EgressFirewallto allow or deny egress, as in the following example:kind: EgressFirewall metadata: name: default spec: egress: - type: Allow to: cidrSelector: <1.2.3.0/24> - type: Allow to: dnsName: <www.github.com> - type: Allow to: dnsName: <www.developer.kubernetes.com> - type: Allow to: dnsName: <www.developer.servicenow.com> - type: Allow to: dnsName: <www.developer.vcenter.com> - type: Deny to: cidrSelector: <0.0.0.0/0>
- Allow egress to any external services, such as the following integrations:
5.13 Access the Cloud Pak for AIOps console
After you successfully install IBM Cloud Pak for AIOps, get the URL for accessing the Cloud Pak for AIOps console.
-
Use the following command to get the URL to access the Cloud Pak for AIOps console:
oc get route -n ${PROJECT_CP4AIOPS} cpd -o jsonpath='{.spec.host}'The following output is a sample output:cpd-cp4aiops.apps.mycluster.mydomainBased on the sample output, your console URL would be
https://cpd-cp4aiops.apps.mycluster.mydomain -
Enter the URL in your browser to open the Cloud Pak for AIOps console. Log in with your username and password.
Find the IBM Cloud Pak for AIOps console username and password
The default username to access the Cloud Pak for AIOps console is admin. You can check the default username and their password with the following commands.
This information is for the IBM provided credentials (admin only) authentication type.
-
Find the default username.
oc -n ${PROJECT_CP4AIOPS} get secret platform-auth-idp-credentials -o jsonpath='{.data.admin_username}' | base64 -d && echo -
Get the password for the
adminusername.oc -n ${PROJECT_CP4AIOPS} get secret platform-auth-idp-credentials -o jsonpath='{.data.admin_password}' | base64 -dThe following extract shows a sample output:EwK9dj9fwPZHyHTyu9TyIgh9klZSzVsABased on the sample output, your password would be
EwK9dj9fwPZHyHTyu9TyIgh9klZSzVsA.You can change this default password at any time. For more information, see Changing the cluster administrator password.
What to do next
- Define integrations and applications with Defining.
- You can integrate with IBM Cognos Analytics. For more information, see Integrating IBM Cognos Analytics with IBM Cloud Pak for AIOps.
- If yo- If you have an existing on-premises IBM Tivoli Netcool/OMNIbus deployment, then you can connect it to IBM Cloud Pak for AIOps through an integration. For more information, see Creating IBM Tivoli Netcool/OMNIbus integrations.
- If you have an existing on-premises IBM Tivoli Netcool/Impact deployment, then you can connect it to IBM Cloud Pak for AIOps through an integration. For more information, see Creating IBM Tivoli Netcool/Impact integrations.
- Familiarize yourself with backup and restore procedures. It is recommended that you take regular backups of your IBM Cloud Pak for AIOps deployment. For more information, see Backup and restore.
- For more information about health checks and monitoring, see Health checks and monitoring. It is recommended that you implement self-monitoring checks and self-protection to improve the stability of your deployment. For more information, see Configuring and enabling OpenShift Container Platform monitoring.