Online production installation of IBM Cloud Pak for Watson AIOps (console method)

Before you begin

• Review the Planning section.
• Online installations of IBM Cloud Pak for Watson AIOps can be run entirely as a nonroot user, and do not require that user to have sudo access.
• Some steps must still be run with the command line. Ensure that you are logged in to your Red Hat OpenShift cluster with oc login for any steps that use the Red Hat OpenShift command-line interface (CLI).
• The display names of some Red Hat OpenShift console components, such as window titles and push buttons, vary between Red Hat OpenShift versions. The following instructions are based on Red Hat OpenShift version 4.12 console components.
• If you require details about the permissions that the IBM Cloud Pak for Watson AIOps operators need, see Permissions (IBM Cloud Pak for Watson AIOps).
• Red Hat OpenShift requires a user with admin privileges to Create a custom project, and a user with cluster-admin privileges for the following operations:
  • Create the catalog source
  • Install the operator

Installation procedure

Follow these steps to install IBM Cloud Pak for Watson AIOps.

1. Install and configure Red Hat OpenShift
2. Configure storage
3. Create a custom project (namespace)
4. Create the entitlement key secret
5. Configure usage data collection
6. Create the catalog source
7. Verify cluster readiness
8. Install the operator
9. Create the custom resource
10. (Optional) Create an EgressNetworkPolicy to restrict outgoing traffic
11. Create a network policy for log anomaly detection
12. Access the IBM Cloud Pak Automation console

1. Install and configure Red Hat OpenShift

IBM Cloud Pak for Watson AIOps requires Red Hat OpenShift to be installed and running. You must have administrative access to your Red Hat OpenShift cluster.

For more information about supported versions of Red Hat OpenShift, see Supported Red Hat OpenShift Container Platform versions. The hardware architecture that you install IBM Cloud Pak for Watson AIOps on must be AMD64.

1. Install Red Hat OpenShift by using the instructions in the Red Hat OpenShift documentation .

2. Install the Red Hat OpenShift command line interface (oc) on your cluster's boot node and run oc login. For more information, see the instructions in Getting started with the Red Hat OpenShift CLI .

3. Optionally configure a custom certificate for IBM Cloud Pak for Watson AIOps to use. You can use either of the following methods:

   • Configure a custom certificate for the Red Hat OpenShift cluster. Follow the instructions in the Red Hat OpenShift documentation Replacing the default ingress certificate. Then, deploy the signing CA certificate into the cluster by following the instructions in the Red Hat OpenShift documentation Replacing the CA Bundle certificate.
   • If you would like to use a custom certificate for IBM Cloud Pak for Watson AIOps only, then after installation is complete follow the instructions in Using a custom certificate.

2. Configure storage

The storage configuration must satisfy your sizing requirements. For more information about the storage classes that are needed for installing IBM Cloud Pak for Watson AIOps, see Storage.

3. Create a custom project (namespace)

Create a project (namespace) to deploy IBM Cloud Pak for Watson AIOps into.

A project is a Kubernetes namespace. You must create a custom project (namespace) and not use the default, kube-system, kube-public, openshift-node, openshift-infra, or openshift projects (namespaces). This is because IBM Cloud Pak for Watson AIOps uses Security Context Constraints (SCC), and SCCs cannot be assigned to pods created in one of the default Red Hat OpenShift projects (namespaces).

1. From your Red Hat OpenShift console, click Home > Projects.

2. Select Create Project, specify the Name of the project that you want to create, for example cp4waiops and click Create.

4. Create the entitlement key secret

Complete the following steps to create a docker-registry secret to enable your deployment to pull the IBM Cloud Pak for Watson AIOps images from the IBM® Entitled Registry.

1. Obtain the entitlement key that is assigned to your IBMid. Log in to MyIBM Container Software Library with the IBMid and password details that are associated with the entitled software.

2. In the Active entitlement keys section, select Copy to copy the entitlement key to the clipboard.

3. From your Red Hat OpenShift console, click Workloads > Secrets.

4. From the Project menu, select the project that you created earlier in Create a custom project (namespace).

5. Click the Create button, and select Image Pull Secret from the menu. The Create image pull secret form is displayed. Enter the following values and then click Create.

   • Secret name: ibm-entitlement-key
   • Authentication type: Image registry credentials
   • Registry server address: cp.icr.io
   • Username: cp
   • Password: use the entitlement key that you copied in step 2.

5. Configure usage data collection

To help the development of IBM Cloud Pak for Watson AIOps, daily aggregated usage data is collected to analyse how IBM Cloud Pak for Watson AIOps is used. The usage data is collected by the cp4waiops-metricsprocessor pod, and is sent to and stored in IBM controlled GDPR-compliant systems. The collection of usage data is enabled by default, but can be disabled. For transparency, the cp4waiops-metricsprocessor pod's logs contain all the information that is collected. The usage data that is collected is all numeric, and does not include email addresses, passwords, or specific details. Only the following data is collected:

• Current number of applications
• Current number of alerts (all severities aggregated)
• Current number of incidents (all priorities aggregated)
• Current number of policies (includes predefined and user created)
• Current number of runbooks run since installation
• Current number of connectors of each type (For example ServiceNow, Instana, Falcon Logscale)

Use the following steps to configure or disable usage data collection.

1. From your Red Hat OpenShift console, click Workloads > Secrets.

2. From the Project menu, select the project that you created earlier in Create a custom project (namespace).

3. Click the Create button, and select Key/value secret from the menu. The Create key/value secret form is displayed. Enter the following values and then click Create.

   • Secret name: aiops-metrics-processor
   • Add the following Key/Value pairs:
     • customerName: your company name
     • customerICN: your IBM Customer Number (ICN)
     • environment: trial for testing, poc for proof of concept, or production for production environments.
   • If you want to disable usage data collection, also add the following key/value pair: enableCollection: false

4. If you have a firewall enabled, ensure that outbound traffic to https://api.segment.io is allowed.

Important: Usage data without your customer details is still collected even if you do not create this secret. If you do not want any usage data collected, then you must create this secret with enableCollection set to false.

You can update your usage data collection preferences after installation. For more information, see Updating usage data collection preferences.

6. Create the catalog source

Add the IBM Cloud Pak for Watson AIOps catalog source to your Red Hat OpenShift cluster.

After installation, the ibm-operator-catalog CatalogSource object determines whether the upgrade of your IBM Cloud Pak for Watson AIOps deployment is initiated automatically when a new patch becomes available. The ibm-operator-catalog CatalogSource object can be configured to automatically poll for and retrieve a newer catalog by enabling the polling attribute, spec.updateStrategy.registryPoll. If a newer catalog for a patch is found and retrieved, then an automatic upgrade of your IBM Cloud Pak for Watson AIOps deployment is initiated. For more information, see Controlling upgrade.

Note: ibm-operator-catalog also contains the catalogs for other IBM Cloud Paks®. If multiple IBM Cloud Paks are installed on your cluster, then the polling attribute is configured for all of them.

1. Log in to your Red Hat OpenShift cluster's console.

2. Add the IBM Operators CatalogSource.

   If you want to enable the automatic initiation of patch upgrades, click the plus icon in the upper right corner to open the Import YAML dialog box, paste in the following YAML, and then click Create.

    apiVersion: operators.coreos.com/v1alpha1
    kind: CatalogSource
    metadata:
      name: ibm-operator-catalog
      namespace: openshift-marketplace
    spec:
      displayName: ibm-operator-catalog
      publisher: IBM Content
      sourceType: grpc
      image: icr.io/cpopen/ibm-operator-catalog:latest
      updateStrategy:
        registryPoll:
          interval: 45m
   

   If you want to disable the automatic initiation of patch upgrades, then use the following steps that do not configure catalog polling, and which also fix the image to the current image digest only. This ensures that the latest image is not pulled if a node reload or other issue causes the ibm-operator-catalog CatalogSource pods to restart.

   1. Click the plus icon in the upper right to open the Import YAML dialog box, paste in the following YAML, and then click Create.

      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      metadata:
        name: ibm-operator-catalog
        namespace: openshift-marketplace
      spec:
        displayName: ibm-operator-catalog
        publisher: IBM Content
        sourceType: grpc
        image: icr.io/cpopen/ibm-operator-catalog:latest
      

   2. Find the current image digest.

      1. Go to Home > Projects, and select openshift-marketplace.

      2. Go to Workloads > Pods (on the left menu), and then search for ibm-operator-catalog.

      3. Click the returned ibm-operator-catalog-<...> pod.

      4. Click YAML to switch to the YAML view.

      5. Search for imageID in the YAML, and copy down the value of spec.containerStatuses.imageID. The value is in a format similar to the following example:

         icr.io/cpopen/ibm-operator-catalog@sha256:<...>
         

         
         

   3. Go to Administration > Cluster Settings. Under Configuration > OperatorHub > Sources, scroll down and click ibm-operator-catalog.

   4. Click YAML to switch to the YAML view.

   5. Set the value of spec.image to the value of the current image digest that you found in step 2, instead of to icr.io/cpopen/ibm-operator-catalog:latest.

   You can disable or re-enable automatic patch upgrade after installation if you change your mind. For more information, see Configuring automatic patch upgrades.

3. Go to Administration > Cluster Settings. Under Configuration > OperatorHub > Sources, verify that the ibm-operator-catalog CatalogSource object is present.

7. Verify cluster readiness

Run the prerequisite checker script to verify whether your Red Hat OpenShift cluster is correctly set up for a IBM Cloud Pak for Watson AIOps installation. For more information about the script, and how to download and run it, see github.com/IBM .

• Expand to see example output

  
  # ./prereq.sh 
  Using project "cp4waiops" on server "https://myserver.mycluster.mydomain:6443".
  Starting IBM Cloud Pak for Watson AIOps AI Manager prerequisite checker v4.1...
  [INFO] =================================Openshift Container Platform Version Check=================================
  [INFO] Checking OCP Version. Compatible Versions of OCP are v4.10.46+ and v4.12.x
  [INFO] OCP version 4.12.18 is compaitble
  [INFO] =================================Openshift Container Platform Version Check=================================
  [INFO] =================================Entitlement Pull Secret=================================
  [INFO] Checking whether the Entitlement secret or Global pull secret is configured correctly.
  [INFO] Checking if the job 'cp4waiops-entitlement-key-test-job' already exists.
  [INFO] The job with name 'cp4waiops-entitlement-key-test-job' was not found, so moving ahead and creating it.
  [INFO] Creating the job 'cp4waiops-entitlement-key-test-job'
  job.batch/cp4waiops-entitlement-key-test-job created
  [INFO] Verifying if the job 'cp4waiops-entitlement-key-test-job' completed successfully..
  [INFO] SUCCESS! Entitlement secret is configured correctly.
  job.batch "cp4waiops-entitlement-key-test-job" deleted
  [INFO] =================================Entitlement Pull Secret=================================
  [INFO] =================================Storage Provider=================================
  [INFO] Checking storage providers
  [INFO] No IBM Storage Fusion Found... Skipping configuration check.
  [INFO] No Portworx StorageClusters found with "Running" or "Online" status. Skipping configuration check for Portworx.
  [INFO] Openshift Data Foundation found.
  [INFO] No IBM Cloud Storage found... Skipping configuration check for IBM Cloud Storage Check.
  Checking Openshift Data Foundation Configuration...
  Verifying if Red Hat Openshift Data Foundation pods are in "Running" or "Completed" status
  [INFO] Pods in openshift-storage project are "Running" or "Completed"
  [INFO] ocs-storagecluster-ceph-rbd exists.
  [INFO] ocs-storagecluster-cephfs exists.
  [INFO] No warnings or failures found when checking for Storage Providers.
  [INFO] =================================Storage Provider=================================
  [INFO] =================================Small or Large Profile Install Resources=================================
  [INFO] Checking for cluster resources
  [INFO] ==================================Resource Summary=====================================================
  [INFO]                                       Nodes   |      vCPU      |     Memory(GB)
  [INFO] Small profile(available/required)  [  14 / 3 ]   [  255 / 62 ]       [  479 / 140 ]
  [INFO] Large profile(available/required)  [  14 / 10 ]   [  255 / 162 ]       [  479 / 360 ]
  [INFO] ==================================Resource Summary=====================================================
  [INFO] Cluster currently has resources available to create a large profile of Cloud Pak for Watson AIOps AI Manager
  [INFO] =================================Small or Large Profile Install Resources=================================
  [INFO] =================================Prerequisite Checker Tool Summary=================================
  [  PASS  ] Openshift Container Platform Version Check
  [  PASS  ] Entitlement Pull Secret
  [  PASS  ] Storage Provider
  [  PASS  ] Small or Large Profile Install Resources
  [INFO] =================================Prerequisite Checker Tool Summary=================================
  

8. Install the operator

For more information about operators, see Adding Operators to a cluster in the Red Hat OpenShift documentation.

1. Log in to your Red Hat OpenShift cluster's console.

2. Click Operators > OperatorHub. The OperatorHub page is displayed.

3. In the All Items field, enter IBM Cloud Pak for Watson AIOps AI Manager. The IBM Cloud Pak for Watson AIOps AI Manager operator is displayed.

4. Click the IBM Cloud Pak for Watson AIOps AI Manager tile. The IBM Cloud Pak for Watson AIOps AI Manager window is displayed.

5. Click Install. The Install Operator page is displayed.

6. Enter the following values:

   • Set Update channel to v4.1.
   • Installation mode - For more information about installation modes, see Operator installation mode.
   • Installed Namespace - If you are using the OwnNamespace installation mode (a specific namespace), then set this field to be the project (namespace) in which to install the operator, such as cp4waiops. If you are using the AllNamespaces installation mode, then set this field to openshift-operators.
   • Set Update approval to Automatic.

   Warning: Update approval must not be changed to Manual. Manual approval, which requires the manual review and approval of the generated InstallPlans, is not supported. Incorrect timing or ordering of manual approvals of InstallPlans can result in a failed installation.

7. Click Install and wait for the IBM Cloud Pak for Watson AIOps AI Manager operator to install.

8. Verify that the IBM Cloud Pak for Watson AIOps AI Manager operator is successfully installed.

   Navigate to Operators > Installed Operators, and select your project from the Projects dropdown. IBM Cloud Pak for Watson AIOps AI Manager and its dependant operators in the project are listed with a status of Succeeded.

   Important: A dependency may not yet be present when this command is run, and cause IBM Elastic to be Failed. This dependency is resolved when the custom resource is created and rollout continues. No user intervention or delay is required.

9. Create the custom resource

Create an instance of the IBM Cloud Pak for Watson AIOps custom resource. A maximum of one IBM Cloud Pak for Watson AIOps custom resource is allowed per cluster.

1. From your Red Hat OpenShift console, click Operators > Installed Operators.

2. From the Project dropdown menu, select the project that you want to create the IBM Cloud Pak for Watson AIOps instance in. Use the project that you created earlier in Create a custom project (namespace).

   Note: You cannot use the default, kube-system, kube-public, openshift-node, openshift-infra, or openshift projects. This is because IBM Cloud Pak for Watson AIOps uses Security Context Constraints (SCC), and SCCs cannot be assigned to pods created in one of the default Red Hat OpenShift projects.

3. Select IBM Cloud Pak for Watson AIOps AI Manager operator, then click the IBM Cloud Pak for Watson AIOps AI Manager tab.

4. Click Create Installation. The default Form View is displayed.

   Warning: The pakModules aiopsFoundation, applicationManager, and aiManager must be enabled. Do not change these values to false.

   Enter the following values:

   • Name: Specify the name that you want your IBM Cloud Pak for Watson AIOps instance to be called.

   • License: Expand the License section and read the agreement. Toggle the License Acceptance switch to True to accept the license.

   • File Storage Class and Large Block Storage Class are the storage classes that you want to use, as detailed in the following table. For more information about storage, see Storage.

   • Enable Secure Tunnel: Set to true if you want to install Secure Tunnel. For more information about Secure Tunnel, see Secure Tunnel.

   • Image Pull Secret: Select the ibm-entitlement-key secret that you created in the step Create the entitlement key secret.

   • Resource Overrides ConfigMap Do not edit this field unless you have been supplied with a custom ConfigMap by an IBM Sales representative.

   • Size: Select the size that you require for your IBM Cloud Pak for Watson AIOps installation.

   • Topology resource group terminology: Specify application or service as the terminology to be used for collections of topology resource groups. The default is application.

     Note: To confirm that you have the storage classes for your chosen storage provider as shown in the table, run oc get sc.

5. Click Create to create a custom resource that is an instance of IBM Cloud Pak for Watson AIOps.

6. After a few minutes, use the following steps to check the status of your installation. Click Operators > Installed Operators.

7. From the Project list, select the project (namespace) that IBM Cloud Pak for Watson AIOps is deployed in.

8. Select IBM Cloud Pak for Watson AIOps AI Manager and then click the IBM Cloud Pak for Watson AIOps AI Manager tab.

9. Under Installations, look for the entry with the name that you specified for your IBM Cloud Pak for Watson AIOps instance, and verify that it has a Status of Phase: Updating. It takes around 60-90 minutes for the installation to complete (subject to the speed with which images can be pulled). When installation is complete and successful, the Status changes to Phase: Running.

   (Optional): If you want to see more detail about the status of the installation's components, select the entry with the name that you specified for your IBM Cloud Pak for Watson AIOps instance, and then switch to the YAML view. Scroll down to the Status section near the end of the YAML. A component's installation is complete and successful when the component has a value of Ready.

   Example YAML:

   status:
     size: small
     customProfileConfigmap: aiops-custom-size-profile
     customProfileValidationStatus: >-
       Custom profile configmap not found, continue installation process without customization
     storageclasslargeblock: rook-ceph-rbd
     componentstatus:
       issueresolutioncore: Ready
       kafka: Ready
       aiopsanalyticsorchestrator: Ready
       aiopsedge: Ready
       tunnel: Ready
       lifecycleservice: Ready
       zenservice: Ready
       vaultaccess: Ready
       vaultdeploy: Ready
       flinkcluster: Ready
       cluster: Ready
       elasticsearch: Ready
       kong: Ready
       aiopsui: Ready
       redissentinel: Ready
       <...>
   

(Optional) You can also download and run a status checker script to see information about the status of your deployment. For more information about how to download and run the script, see github.com/IBM.

If the installation fails, or is not complete and is not progressing, then see Troubleshooting installation and upgrade and Known Issues to help you identify any installation problems

10. (Optional) Create an EgressNetworkPolicy

There is no egress firewall policy defined when you install IBM Cloud Pak for Watson AIOps, so outgoing traffic from workload pods to the internal and external network is unrestricted.

If you require a more secure environment, then use the following steps.

1. Create an EgressNetworkPolicy on your Red Hat OpenShift cluster to limit egress from the IBM Cloud Pak for Watson AIOps project (namespace).

   For more information about creating an EgressNetworkPolicy, see Configuring an egress firewall for a project.

   Note: You can only have one EgressNetworkPolicy per project/namespace.

2. Configure exceptions to the EgressNetworkPolicy.

   Edit your EgressNetworkPolicy to add exceptions for the following IBM Cloud Pak for Watson AIOps components that have egress dependencies, otherwise these IBM Cloud Pak for Watson AIOps components fail when they attempt egress.

   1. Allow egress to any external services, such as the following connections:
      • Kubernetes
      • GitHub
      • Microsoft® Teams
      • ServiceNow
      • Slack
      • VMware® vCenter

   2. Configure your EgressNetworkPolicy to allow traffic for your GitHub, Kubernetes, ServiceNow, and VMware vCenter connections.

      Edit your EgressNetworkPolicy to allow or deny egress, as in the following example:

      kind: EgressNetworkPolicy
      metadata:
        name: default
      spec:
        egress:
        - type: Allow
          to:
            cidrSelector: <1.2.3.0/24>
        - type: Allow
          to:
            dnsName: <www.github.com>
        - type: Allow
          to:
            dnsName: <www.developer.kubernetes.com>
        - type: Allow
          to:
            dnsName: <www.developer.servicenow.com>
        - type: Allow
          to:
            dnsName: <www.developer.vcenter.com>
        - type: Deny
          to:
            cidrSelector: <0.0.0.0/0>
      

      Where the values you enter for dnsName and cidrSelector are the DNS names and addresses of your GitHub, Kubernetes, ServiceNow, or VMware vCenter sources.

11. Create a network policy for log anomaly detection

If you plan to use log anomaly, run the following commands before you create log connections. Replace the AIOPS_NAMESPACE value with the name of the project in which Cloud Pak for Watson AIOps is installed.

AIOPS_NAMESPACE="cp4waiops"
cat << EOF | oc apply -n $AIOPS_NAMESPACE -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  labels:
    app: flink
    cluster: cp4waiops-eventprocessor-eve-29ee-ep
    component: taskmanager
  name: cp4waiops-eventprocessor-eve-29ee-ep-tm-patch
spec:
  egress:
  - {}
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: flink
          cluster: cp4waiops-eventprocessor-eve-29ee-ep
          component: taskmanager
    - podSelector:
        matchLabels:
          app: flink
          cluster: cp4waiops-eventprocessor-eve-29ee-ep
          component: jobmanager
  - ports:
    - port: 9248
      protocol: TCP
    - port: 6122
      protocol: TCP
    - port: 6121
      protocol: TCP
  podSelector:
    matchLabels:
      app: flink
      cluster: cp4waiops-eventprocessor-eve-29ee-ep
      component: taskmanager
  policyTypes:
  - Ingress
  - Egress
EOF

cat << EOF | oc apply -n $AIOPS_NAMESPACE -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  labels:
    app: flink
    cluster: cp4waiops-eventprocessor-eve-29ee-ep
    component: jobmanager
  name: cp4waiops-eventprocessor-eve-29ee-ep-jm-patch
spec:
  egress:
  - {}
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: flink
          cluster: cp4waiops-eventprocessor-eve-29ee-ep
          component: taskmanager
    - podSelector:
        matchLabels:
          app: flink
          cluster: cp4waiops-eventprocessor-eve-29ee-ep
          component: jobmanager
  - ports:
    - port: 8081
      protocol: TCP
    - port: 6123
      protocol: TCP
    - port: 6125
      protocol: TCP
    - port: 8080
      protocol: TCP
    - port: 6124
      protocol: TCP
    - port: 9249
      protocol: TCP
  podSelector:
    matchLabels:
      app: flink
      cluster: cp4waiops-eventprocessor-eve-29ee-ep
      component: jobmanager
  policyTypes:
  - Ingress
  - Egress
EOF

12. Access the IBM Cloud Pak for Watson AIOps UI

After you successfully install IBM Cloud Pak for Watson AIOps, you can use the IBM Cloud Pak Administration panel to manage the underlying deployment, or use the IBM Cloud Pak Automation console to use IBM Cloud Pak for Watson AIOps. You can log in to the console by using OpenShift authentication or IBM provided credentials.

What to do next

• Define integrations and applications with Defining.
• If you have an existing on-premises IBM Tivoli Netcool/OMNIbus deployment, then you can connect IBM Cloud Pak for Watson AIOps to it with the Netcool connector. For more information, see Creating IBM Tivoli Netcool/OMNIbus connections.
• If you have an existing on-premises IBM Tivoli Netcool/Impact deployment, then you can connect IBM Cloud Pak for Watson AIOps to it with the IBM Tivoli Netcool/Impact connector. For more information, see Creating IBM Tivoli Netcool/Impact connections
• Familiarize yourself with backup and restore procedures. It is recommended that you take regular backups of your IBM Cloud Pak for Watson AIOps deployment. For more information, see Backup and restore.