Online production installation of IBM Cloud Pak for AIOps (console method)

Before you begin

• You must know whether you are deploying a base deployment or a extended deployment of IBM Cloud Pak for AIOps. For more information, see Incremental adoption.
• Review the Planning section.
• Online installations of IBM Cloud Pak for AIOps can be run entirely as a non-root user, and do not require that user to have sudo access.
• Some steps must still be run with the command line. Ensure that you are logged in to your Red Hat OpenShift cluster with oc login for any steps that use the Red Hat OpenShift command-line interface (CLI).
• The display names of some Red Hat OpenShift console components, such as window titles and push buttons, vary between Red Hat OpenShift versions. The following instructions are based on Red Hat OpenShift version 4.14 console components.
• If you require details about the permissions that the IBM Cloud Pak for AIOps operators need, see Permissions (IBM Cloud Pak for AIOps).
• A user with cluster-admin privileges is needed for the following operations:
  • Create a custom project
  • Create the catalog sources
  • Install Cert Manager
  • Install the License Service
  • Install the operator

Important: If IBM Sales representatives and Business Partners supplied you with a custom profile ConfigMap to customize your deployment, then you must follow their instructions to apply it during installation. The custom profile cannot be applied after installation, and attempting to do so can break your IBM Cloud Pak for AIOps deployment. For more information about custom sizing, see Custom sizing.

Installation procedure

Follow these steps to install IBM Cloud Pak for AIOps.

1. Install and configure Red Hat OpenShift
2. Configure storage
3. Create a custom project (namespace)
4. Create the entitlement key secret
5. Configure usage data collection
6. Create the catalog sources
7. Install Cert Manager
8. Install the License Service
9. Verify cluster readiness
10. Install the operator
11. Install IBM Cloud Pak for AIOps
12. Verify your installation
13. Create an EgressFirewall to restrict outgoing traffic
14. Access the IBM Cloud Pak for AIOps console

Prerequisites

Allow access to the following sites and ports:

For more information, see Configuring your firewall for OpenShift Container Platform.

1. Install and configure Red Hat OpenShift

IBM Cloud Pak for AIOps requires Red Hat OpenShift to be installed and running. You must have administrative access to your Red Hat OpenShift cluster.

For more information about supported versions of Red Hat OpenShift, see Supported Red Hat OpenShift Container Platform versions.

1. Install Red Hat OpenShift by using the instructions in the Red Hat OpenShift documentation .

2. Install the Red Hat OpenShift command line interface (oc) on your cluster's boot node and run oc login. For more information, see the instructions in Getting started with the Red Hat OpenShift CLI .

3. Ensure that the clocks on your Red Hat OpenShift cluster are synchronized. Each Red Hat OpenShift node in the cluster must have access to an NTP server. Red Hat OpenShift nodes use NTP to synchronize their clocks. IBM Cloud Pak for AIOps runs on Red Hat OpenShift and also has this requirement. Discrepancies between the clocks on the Red Hat OpenShift nodes can cause IBM Cloud Pak for AIOps to experience operational issues. See the Red Hat OpenShift documentation for information about how to use a MachineConfig custom resource to configure chrony to connect to your NTP servers.

4. Optionally configure a custom certificate for IBM Cloud Pak for AIOps to use. You can use either of the following methods:

   • Configure a custom certificate for the Red Hat OpenShift cluster. Follow the instructions in the Red Hat OpenShift documentation Replacing the default ingress certificate. Then, deploy the signing CA certificate into the cluster by following the instructions in the Red Hat OpenShift documentation Replacing the CA Bundle certificate.
   • If you would like to use a custom certificate for IBM Cloud Pak for AIOps only, then after installation is complete follow the instructions in Using a custom certificate.

2. Configure storage

The storage configuration must satisfy your sizing requirements. For more information about the storage classes that are needed for installing IBM Cloud Pak for AIOps, see Storage.

Note: Storage classes and storage providers cannot be changed after you install IBM Cloud Pak for AIOps. OADP backup and restore requires that a ReadWriteMany (RWX) storage class must be provided. If OADP backup and restore is not needed, a ReadWriteOnce (RWO) storage class can be provided as the RWX-storage-class-name in the installation instance CR YAML file. This configuration cannot be changed after IBM Cloud Pak for AIOps is installed.

3. Create a custom project (namespace)

Create a project (namespace) to deploy IBM Cloud Pak for AIOps into.

A project is a Kubernetes namespace. You must create a custom project (namespace) and not use the default, kube-system, kube-public, openshift-node, openshift-infra, or openshift projects (namespaces). This is because IBM Cloud Pak for AIOps uses Security Context Constraints (SCC), and SCCs cannot be assigned to pods created in one of the default Red Hat OpenShift projects (namespaces).

1. From your Red Hat OpenShift console, click Home > Projects.

2. Select Create Project, specify the Name of the project that you want to create, for example cp4aiops and click Create.

4. Create the entitlement key secret

Complete the following steps to create a registry secret to enable your deployment to pull the IBM Cloud Pak for AIOps images from the IBM® Entitled Registry.

1. Obtain the entitlement key that is assigned to your IBMid. Log in to MyIBM Container Software Library with the IBMid and password details that are associated with the entitled software.

2. In the Active entitlement keys section, select Copy to copy the entitlement key to the clipboard.

3. From your Red Hat OpenShift console, click Workloads > Secrets.

4. From the Project menu, select the project that you created earlier in Create a custom project (namespace).

5. Click the Create button, and select Image pull secret from the menu. The Create image pull secret form is displayed. Enter the following values and then click Create.

   • Secret name: ibm-entitlement-key
   • Authentication type: Image registry credentials
   • Registry server address: cp.icr.io
   • Username: cp
   • Password: use the entitlement key that you copied in step 2.

5. Configure usage data collection

To help the development of IBM Cloud Pak for AIOps, daily aggregated usage data is collected to analyse how IBM Cloud Pak for AIOps is used. The usage data is collected by the cp4waiops-metricsprocessor pod, and is sent to and stored in IBM controlled GDPR-compliant systems. The collection of usage data is enabled by default, but can be disabled. For transparency, the cp4waiops-metricsprocessor pod's logs contain all the information that is collected. The usage data that is collected is numeric, or is about the deployment type and platform. It does not include email addresses, passwords, or specific details. Only the following data is collected:

• Current number of applications
• Current number of alerts (all severities aggregated)
• Current number of incidents (all priorities aggregated)
• Current number of policies (includes predefined and user created)
• Current number of runbooks run since installation
• Current number of integrations of each type (For example ServiceNow, Instana, Falcon Logscale)
• Secure tunnel enablement: whether connection (which controls whether you can create a secure tunnel) is enabled in the Installation custom resource
• Deployment type: base deployment or extended deployment
• Deployment platform: Red Hat® OpenShift® Container Platform or Linux®

Use the following steps to configure or disable usage data collection.

1. From your Red Hat OpenShift console, click Workloads > Secrets.

2. From the Project menu, select the project that you created earlier in Create a custom project (namespace).

3. Click the Create button, and select Key/value secret from the menu. The Create key/value secret form is displayed. Enter the following values and then click Create.

   • Secret name: aiops-metrics-processor
   • Add the following Key/Value pairs:
     • customerName: your company name
     • customerICN: your IBM Customer Number (ICN)
     • environment: trial for testing, poc for proof of concept, or production for production environments.
   • If you want to disable usage data collection, also add the following key/value pair: enableCollection: false

4. If you have a firewall enabled, ensure that outbound traffic to https://api.segment.io is allowed.

Important: Usage data without your customer details is still collected even if you do not create this secret. If you do not want any usage data collected, then you must create this secret with enableCollection set to false.

You can update your usage data collection preferences after installation. For more information, see Updating usage data collection preferences.

6. Create the catalog sources

1. Run the following command to create the catalog sources for IBM Cloud Pak for AIOps and IBM Cloud Pak® foundational services Cert Manager and License Service in the openshift-marketplace namespace.

   Log in to your Red Hat OpenShift cluster's console. Click the plus icon in the upper right to open the Import YAML dialog box, paste in the following YAML, and then click Create.

   apiVersion: operators.coreos.com/v1alpha1
   kind: CatalogSource
   metadata:
     name: ibm-aiops-catalog
     namespace: openshift-marketplace
   spec:
     displayName: ibm-aiops-catalog
     publisher: IBM Content
     sourceType: grpc
     image: icr.io/cpopen/ibm-aiops-catalog@sha256:3d0054b251b8dd9ce698c175003b4c5fd6f3c84b5e0184806dec124e64b74ada
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: CatalogSource
   metadata:
     name: ibm-cert-manager-catalog
     namespace: openshift-marketplace
   spec:
     displayName: ibm-cert-manager
     publisher: IBM
     sourceType: grpc
     image: icr.io/cpopen/ibm-cert-manager-operator-catalog
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: CatalogSource
   metadata:
     name: ibm-licensing-catalog
     namespace: openshift-marketplace
   spec:
     displayName: IBM License Service Catalog
     publisher: IBM
     sourceType: grpc
     image: icr.io/cpopen/ibm-licensing-catalog
   

2. Go to Administration > Cluster Settings. Under Configuration > OperatorHub > Sources, verify that the ibm-aiops-catalog, ibm-cert-manager-catalog and ibm-licensing-catalog CatalogSource objects are present.

7. Install Cert Manager

Skip this step if you already have a certificate manager installed on the Red Hat OpenShift cluster that you are installing IBM Cloud Pak for AIOps on. If you do not have a certificate manager then you must install one. The IBM Cloud Pak® foundational services Cert Manager is recommended, and can be installed with the following steps.

For more information about IBM Cloud Pak® foundational services Cert Manager hardware requirements, see IBM Certificate Manager (cert-manager) hardware requirements in the IBM Cloud Pak foundational services documentation.

1. Log in to your Red Hat OpenShift cluster's console.

2. Click Operators > OperatorHub. The OperatorHub page is displayed.

3. In the All Items field, enter IBM Cert Manager. The IBM Cert Manager operator is displayed.

4. Click the IBM Cert Manager tile. The IBM Cert Manager window is displayed.

5. Click Install. You see the Install Operator page.

6. Set the Update Channel to the v4.2 version. If the Channel v4.2 version is not available, click other IBM Cert Manager tile from OperatorHub to install the correct version.

7. Set Installation Mode to All namespaces on the cluster (default).

8. Set Installed Namespace to ibm-cert-manager(Operator recommended).

9. Set Update approval to Automatic.

10. Click Install.

8. Install the License Service

Skip this step if the IBM Cloud Pak® foundational services License Service is already installed on the Red Hat OpenShift cluster that you are installing IBM Cloud Pak for AIOps on. If you do not know whether the License Service is already installed, then see Verifying if License Service is already installed on the cluster in the IBM Cloud Pak foundational services documentation.

IBM Cloud Pak for AIOps requires the installation of the IBM Cloud Pak foundational services License Service. You must install the IBM Cloud Pak foundational services License Service on the Red Hat OpenShift cluster that you are installing IBM Cloud Pak for AIOps on.

Follow the instructions in Installing the License Service with OpenShift console in the IBM Cloud Pak foundational services documentation, from step 2 Create the ibm-licensing namespace onwards.

9. Verify cluster readiness

Run the prerequisite checker script to verify whether your Red Hat OpenShift cluster is correctly set up for a IBM Cloud Pak for AIOps installation.

Download the prerequisite checker script from github.com/IBM , and run it with the following command:

./prereq.sh -n <project>

Where <project> is the project that your IBM Cloud Pak for AIOps installation is deployed in.

Important: If you are installing on a multi-zone cluster, then also specify the -m flag to assess whether there are sufficient resources to withstand a zone outage.

Example output:

# ./prereq.sh -n cp4aiops
[INFO] Starting IBM Cloud Pak for AIOps prerequisite checker v4.8...

CLI: oc

[INFO] =================================Platform Version Check=================================
[INFO] Checking Platform Type....
[INFO] You are using Openshift Container Platform
[INFO] OCP version 4.16.27 is compatible but only nodes with x86_64 (amd64) architectures are supported at this time. 
[INFO] =================================Platform Version Check=================================

[INFO] =================================Storage Provider=================================
[INFO] Checking storage providers
[INFO] No IBM Storage Fusion Found... Skipping configuration check.
[INFO] No IBM Storage Fusion HCI System... Skipping configuration check.

[INFO] No Portworx StorageClusters found with "Running" or "Online" status. Skipping configuration check for Portworx.
[INFO] Openshift Data Foundation found.
[INFO] No IBM Cloud Storage found... Skipping configuration check for IBM Cloud Storage Check.

Checking Openshift Data Foundation Configuration...
Verifying if Red Hat Openshift Data Foundation pods are in "Running" or "Completed" status
[INFO] Pods in openshift-storage project are "Running" or "Completed"
[WARNING] ocs-storagecluster-ceph-rbd does not exist. 
[INFO] One of more warnings found when checking for Storage Providers.
[INFO] =================================Storage Provider=================================

[INFO] =================================Cert Manager Check=================================
[INFO] Checking for Cert Manager operator

[INFO] Successfully functioning cert-manager found.

CLUSTERSERVICEVERSION              NAMESPACE
ibm-cert-manager-operator.v4.2.11  ibm-cert-manager

[INFO] =================================Cert Manager Check=================================

[INFO] =================================Licensing Service Operator Check=================================
[INFO] Checking for Licensing Service operator

[INFO] Successfully functioning licensing service operator found.

CLUSTERSERVICEVERSION           NAMESPACE
ibm-licensing-operator.v4.2.11  ibm-licensing

[INFO] =================================Licensing Service Operator Check=================================

[INFO] =================================Starter or Production Install Resources=================================
[INFO] Checking for cluster resources

[INFO] ==================================Resource Summary=====================================================
[INFO]                                                        Nodes            vCPU            Memory(GB)
[INFO] Starter (Non-HA) Base (available/required)           [  9 / 3 ]     [  84 / 47 ]     [  110 / 123 ]
[INFO]     (+ Log Anomaly Detection & Ticket Analysis)      [  9 / 3 ]     [  84 / 55 ]     [  110 / 136 ]

[INFO] Production (HA) Base (available/required)            [  9 / 6 ]     [  84 / 136 ]    [  110 / 310 ]
[INFO]     (+ Log Anomaly Detection & Ticket Analysis)      [  9 / 6 ]     [  84 / 162 ]    [  110 / 368 ]
[INFO] ==================================Resource Summary=====================================================
[ERROR] Cluster does not have required resources available to install Cloud Pak for AIOps.

[INFO] =================================Starter or Production Install Resources=================================


[INFO] =================================Prerequisite Checker Tool Summary=================================
      [  PASS  ] Platform Version Check 
      [  WARNING  ] Storage Provider
      [  FAIL  ] Starter (Non-HA) Base Install Resources
      [  FAIL  ] Production (HA) Base Install Resources
      [  PASS  ] Cert Manager Operator Installed
      [  PASS  ] Licensing Service Operator Installed
[INFO] =================================Prerequisite Checker Tool Summary=================================

10. Install the operator

For more information about installing operators, see Adding Operators to a cluster in the Red Hat OpenShift documentation.

For more information about the operators which are installed with IBM Cloud Pak for AIOps, see Operator Details.

Note: During the initial installation of Cloud Pak for AIOps, Kubernetes jobs might fail and re-run. If a job succeeds on the second or third attempt, there can be one or two pods in Error state and one pod in the Completed state. If a job fails repeatedly, the attempt is abandoned, and the logs from failed pods can be used to determine the cause of the failure. When you determine the cause for the failure, you can delete the job, and the operator can recreate it to reattempt the operations.

1. Log in to your Red Hat OpenShift cluster's console.

2. Click Operators > OperatorHub. The OperatorHub page is displayed.

3. In the All Items field, enter IBM Cloud Pak for AIOps. The IBM Cloud Pak for AIOps operator is displayed.

4. Click the IBM Cloud Pak for AIOps tile with the CatalogSource tag ibm-aiops-catalog. The IBM Cloud Pak for AIOps window is displayed.

5. Click Install. The Install Operator page is displayed.

6. Enter the following values:

   • Set Update channel to v4.8.
   • Installation mode - For more information about installation modes, see Operator installation mode.
   • Installed Namespace - If you are using the OwnNamespace installation mode (a specific namespace), then set this field to be the project (namespace) in which to install the operator, such as cp4aiops. If you are using the AllNamespaces installation mode, then set this field to openshift-operators.
   • Set Update approval to Automatic. This must not be changed to Manual. Manual approval, which requires the manual review and approval of the generated InstallPlans, is not supported. Incorrect timing or ordering of manual approvals of InstallPlans can result in a failed installation.

7. Click Install and wait for the IBM Cloud Pak for AIOps operator to install.

8. Verify that the IBM Cloud Pak for AIOps operator is successfully installed.

   Navigate to Operators > Installed Operators, and select your project from the Projects dropdown. IBM Cloud Pak for AIOps and its dependant operators in the project are listed with a Status of Succeeded.

11. Install IBM Cloud Pak for AIOps

Create an instance of the IBM Cloud Pak for AIOps custom resource. A maximum of one IBM Cloud Pak for AIOps custom resource is allowed per cluster.

1. From your Red Hat OpenShift console, click Operators > Installed Operators.

2. From the Project dropdown menu, select the project that you want to create the IBM Cloud Pak for AIOps instance in. Use the project that you created earlier in Create a custom project (namespace).

   Note: You cannot use the default, kube-system, kube-public, openshift-node, openshift-infra, or openshift projects. This is because IBM Cloud Pak for AIOps uses Security Context Constraints (SCC), and SCCs cannot be assigned to pods created in one of the default Red Hat OpenShift projects.

3. Select IBM Cloud Pak for AIOps operator, then click the IBM Cloud Pak for AIOps tab.

4. Click Create Installation. The default Form View is displayed.

   Warning: The pakModules aiopsFoundation, applicationManager, and aiManager must be enabled. Do not change these values to false.

   Enter the following values:

   • Name: Specify the name that you want your IBM Cloud Pak for AIOps instance to be called, for example ibm-cp-aiops.

   • license: Expand the license section and read the agreement. Toggle the License Acceptance switch to true to accept the license.

   • File Storage Class and Large Block Storage Class are the storage classes that you want to use, as detailed in the following table. For more information about storage, see Storage.

   • Enable Secure Tunnel: Set to true if you want to install Secure Tunnel. For more information about Secure Tunnel, see Secure Tunnel.

   • Image Pull Secret: Select the ibm-entitlement-key secret that you created in the step Create the entitlement key secret.

   • Resource Overrides ConfigMap Do not edit this field unless you have been supplied with a custom ConfigMap by an IBM Sales representative (or Business Partner).

   • Size: Select the size that you require for your IBM Cloud Pak for AIOps installation.

   • Topology resource group terminology: Specify application or service as the terminology to be used for collections of topology resource groups. The default is application.

   • Zones: If you are installing on a multi-zone cluster, then add the names of the zones that you are using. The zone names must exactly match the zone labels that you applied to each of your nodes in step 1a of Installing IBM Cloud Pak for AIOps on a multi-zone architecture (multi-zone HA).

     Note: To confirm that you have the storage classes for your chosen storage provider as shown in the table, run oc get sc.

5. Configure your deployment to be a base deployment or an extended deployment.

   For more information about these deployment types, see Incremental adoption.

   Your deployment defaults to a base deployment which does not have log anomaly detection and ticket analysis capabilites. If you want a base deployment, then proceed to the next step.

   If you want an extended deployment with log anomaly detection and ticket analysis capabilites, then switch to the YAML view and set the value of enabled to true.

   Example excerpt:

   spec:
     pakModules:
     - enabled: true
       name: logAnomalyDetection
   

   
   

   You can update your deployment type after installation. For more information, see Updating the deployment type.

6. Click Create to create a custom resource that is an instance of IBM Cloud Pak for AIOps.

12. Verify your installation

1. After a few minutes, use the following steps to check the status of your installation. Click Operators > Installed Operators.

2. From the Project list, select the project (namespace) that IBM Cloud Pak for AIOps is deployed in.

3. Select IBM Cloud Pak for AIOps and then click the IBM Cloud Pak for AIOps tab.

4. Under Installations, look for the entry with the name that you specified for your IBM Cloud Pak for AIOps instance, and verify that it has a Status of Phase: Updating. It takes around 60-90 minutes for the installation to complete (subject to the speed with which images can be pulled). When installation is complete and successful, the Status changes to Phase: Running.

5. (Optional) If you want to see more detail about the status of the installation's components, select the entry with the name that you specified for your IBM Cloud Pak for AIOps instance, and then switch to the YAML view. Scroll down to the Status section near the end of the YAML. A component's installation is complete and successful when the component has a value of Ready.

   Example YAML:

   status:
     size: small
     customProfileConfigmap: aiops-custom-size-profile
     customProfileValidationStatus: >-
       Custom profile configmap not found, continue installation process without customization
     storageclasslargeblock: rook-ceph-rbd
     componentstatus:
       issueresolutioncore: Ready
       kafka: Ready
       aiopsanalyticsorchestrator: Ready
       aiopsedge: Ready
       tunnel: Ready
       lifecycleservice: Ready
       zenservice: Ready
       flinkcluster: Ready
       cluster: Ready
       elasticsearchcluster: Ready
       aiopsui: Ready
       redissentinel: Ready
       <...>
   

   
   

(Optional) You can also download and run a status checker script to see information about the status of your deployment. For more information about how to download and run the script, see github.com/IBM.

If the installation fails, or is not complete and is not progressing, then see Troubleshooting installation and upgrade and Known Issues to help you identify any installation problems

13. Create an EgressFirewall

There is no egress firewall policy defined when you install IBM Cloud Pak for AIOps, so outgoing traffic from workload pods to the internal and external network is unrestricted.

To create a more secure environment, use the following steps.

1. Create an EgressFirewall on your Red Hat OpenShift cluster to limit egress from the IBM Cloud Pak for AIOps project (namespace).

   For more information about creating an EgressFirewall, see Configuring an egress firewall for a project.

   Note: You can only have one EgressFirewall per project/namespace.

2. Configure exceptions to the EgressFirewall.

   Edit your EgressFirewall to add exceptions for the following IBM Cloud Pak for AIOps components that have egress dependencies, otherwise these IBM Cloud Pak for AIOps components fail when they attempt egress.

   1. Allow egress to any external services, such as the following integrations:
      • Kubernetes
      • GitHub
      • Microsoft® Teams
      • ServiceNow
      • Slack
      • VMware® vCenter

   2. Configure your EgressFirewall to allow traffic for your GitHub, Kubernetes, ServiceNow, and VMware vCenter integrations.

      Edit your EgressFirewall to allow or deny egress, as in the following example:

      kind: EgressFirewall
      metadata:
        name: default
      spec:
        egress:
        - type: Allow
          to:
            cidrSelector: <1.2.3.0/24>
        - type: Allow
          to:
            dnsName: <www.github.com>
        - type: Allow
          to:
            dnsName: <www.developer.kubernetes.com>
        - type: Allow
          to:
            dnsName: <www.developer.servicenow.com>
        - type: Allow
          to:
            dnsName: <www.developer.vcenter.com>
        - type: Deny
          to:
            cidrSelector: <0.0.0.0/0>
      

      Where the values you enter for dnsName and cidrSelector are the DNS names and addresses of your GitHub, Kubernetes, ServiceNow, or VMware vCenter sources.

14. Access the IBM Cloud Pak for AIOps UI

After you successfully install IBM Cloud Pak for AIOps, you can use the IBM Cloud Pak Administration panel to manage the underlying deployment, or use the IBM Cloud Pak for AIOps console to use IBM Cloud Pak for AIOps.

What to do next

• Define integrations and applications with Defining.
• You can integrate with IBM Cognos® Analytics. For more information, see Integrating IBM Cognos Analytics with IBM Cloud Pak for AIOps.
• If you have an existing on-premises IBM Tivoli Netcool/OMNIbus deployment, then you can connect it to IBM Cloud Pak for AIOps through an integration. For more information, see Creating IBM Tivoli Netcool/OMNIbus integrations.
• If you have an existing on-premises IBM Tivoli Netcool/Impact deployment, then you can connect it to IBM Cloud Pak for AIOps through an integration. For more information, see Creating IBM Tivoli Netcool/Impact integrations.
• Familiarize yourself with backup and restore procedures. It is recommended that you take regular backups of your IBM Cloud Pak for AIOps deployment. For more information, see Backup and restore.
• For more information about health checks and monitoring, see Health checks and monitoring.