Offline installation of IBM Cloud Pak for AIOps on Linux (bastion host)

Overview

If your Linux® cluster does not have internet access, you can still install IBM Cloud Pak® for AIOps by using an offline or air-gapped deployment. In this scenario, your air-gapped (offline) environment has a target registry, and a Linux cluster on which IBM Cloud Pak for AIOps is to be installed. The bastion host has access to the internet and to the air-gapped environment. IBM Cloud Pak for AIOps images are mirrored from the IBM Entitled Registry through the bastion server to the target registry in the offline environment. IBM Cloud Pak for AIOps can then be installed in the offline environment by using the target registry.

You can install a production deployment of IBM Cloud Pak for AIOps on a Linux platform, without installing Red Hat® OpenShift® Container Platform.

The deployment is configured to collect usage data. If you want to disable usage data collection, then you can do this post-install. For more information, see Updating usage data collection preferences.

Limitations

• Starter-size deployments are not supported.
• The following features and capabilities are only available for deployments of IBM Cloud Pak for AIOps on Red Hat OpenShift, and are not available for deployments of IBM Cloud Pak for AIOps on Linux:
  • FIPS support
  • Secure tunnel
  • Multi-zone HADR
  • IBM Cognos® Analytics integration
• The aiopsctl tool is supported only on x86_64 (amd64) architecture.

Before you begin

Ensure that you meet the following prerequisites:

• Your cluster meets all of the requirements that are detailed in Planning an installation of IBM Cloud Pak for AIOps on Linux. The Linux cluster must be reserved for the sole use of IBM Cloud Pak for AIOps, and you must have a minimum of three control planes nodes.
• Local storage is configured in accordance with the instructions in Configuring local volumes.
• You have the credentials for the root user. The root user must be used to install IBM Cloud Pak for AIOps.
• The bastion host and worker nodes have network connectivity to the control plane nodes.
• You have a load balancer configured in accordance with the details in Load balancing.

Important: If IBM Sales representatives and Business Partners supplied you with a custom profile ConfigMap to customize your deployment, then you must follow their instructions to apply it during installation. The custom profile cannot be applied after installation, and attempting to do so can break your IBM Cloud Pak for AIOps deployment. For more information about custom sizing, see Custom sizing.

Installation Procedure

The installation procedure requires you to download the aiopsctl tool, which installs IBM Cloud Pak for AIOps and its prerequisites on your cluster.

1. Setup the mirroring environment
2. Retrieve your entitlement key
3. Optionally configure a custom certificate
4. Create environment variables
5. Mirror images
6. Install the aiopsctl tool and register cluster nodes
7. Install IBM Cloud Pak for AIOps
8. Verify your installation
9. Access the Cloud Pak for AIOps console

1. Setup the mirroring environment

2. Retrieve your entitlement key

Obtain the IBM entitlement key that is assigned to your IBMid. The key is needed to pull the IBM Cloud Pak for AIOps images from the IBM® Entitled Registry.

1. Log in to MyIBM Container Software Library with the IBMid and password details that are associated with the entitled software.

2. In the Entitlement key section, select Copy to copy the entitlement key to the clipboard.

3. Optionally configure a custom certificate

If you want to use your own custom certificate for IBM Cloud Pak for AIOps instead of the default cluster certificate, then use the following steps to create a certificate and key that you can supply as parameters at installation time.

1. Ensure that you have the following three PEM-encoded X.509 certificate files:

   • caintermediate.pem: The intermediate certificate that issued your server certificate.
   • aiops.pem: An IBM Cloud Pak for AIOps certificate, which includes the two fully qualified domain names (FQDNs) for aiops-cpd and cp-console-aiops in the Subject Alternative Name (SAN) list.
   • aiops.key.pem: A key file for the signed certificate in aiops.pem

   Tip: You can create the FQDN strings to use for aiops-cpd and cp-console-aiops by prepending cp-console-aiops and aiops-cpd to your load balancer's host name. For example, if your load balancer host name is loadbalancerhost.acme.com, then the FQDN strings are cp-console-aiops.loadbalancerhost.acme.com and aiops-cpd.loadbalancerhost.acme.com.

2. Concatenate the server and intermediate certificates into one file called aiops-certificate-chain.pem.

   cat aiops.pem caintermediate.pem > aiops-certificate-chain.pem 
   

Note: If you do not install IBM Cloud Pak for AIOps with a custom certificate, you can switch to using a custom certificate after installation. For more information, see Using a custom certificate (IBM Cloud Pak for AIOps on Linux).

4. Create environment variables

Create and then source a shell script that is named aiops_var.sh, which defines the environment variables that are used to provide installation parameters for your deployment. Use the following codeblock as a template, replacing the brackets < ... > with values for your environment. It is important that you keep this file.

For more information about choosing a deployment type, see Incremental adoption. Subject to further hardware requirements, you can update the deployment type post-install. For more information, see Updating the deployment type.

Important: If you used an alternative path for APP_STORAGE_PATH or PLATFORM_STORAGE_PATH when you set up your local storage in Configuring local volumes, then you must change the values of these environment variables accordingly.

#================================================================================================================================
# IBM Cloud Pak for AIOps installation variables (Linux)
#================================================================================================================================
export TARGET_USER="root"
export ACCEPT_LICENSE=false # Set to true to agree to the license terms.

# -------------------------------------------------------------------------------------------------------------------------------
# IBM Entitled Registry
# -------------------------------------------------------------------------------------------------------------------------------
export IBM_ENTITLEMENT_KEY=<ibm-entitlement-key> # Set to the entitlement key retrieved in previous step.

# --------------------------------------------------------------------------------------------------------------------------------
# Hostnames
# `<load_balancer_hostname>` - the hostname of your load balancer
# `<control_plane_node_n>` - the FQDN or IP address of each control plane node. For example, "control_plane_node_1.example.com"
# `<worker-n>` - the FQDN or IP address of each worker node. For example, "worker_1.example.com"
# --------------------------------------------------------------------------------------------------------------------------------
export LOAD_BALANCER_HOST="<load_balancer_hostname>"
export CONTROL_PLANE_NODE="<control_plane_node_1>"
export ADDITIONAL_CONTROL_PLANE_NODES=(
  "<control_plane_node_2>"
  "<control_plane_node_3>"
)
export WORKER_NODES=(
  "<worker_1>"
  "<worker_2>"
  "<worker_3>"
  "<worker_4>"
  "<worker_5>"
  "<worker_6>"
  "<worker_7>"
)

# -----------------------------------------------------------------------------------------------------------
# Incremental adoption - set your deployment type.
# Set to `extended` to install an extended deployment with log anomaly detection and ticket analysis capabilities
# Set to `base` to install a base deployment without log anomaly detection and ticket analysis capabilities
# -----------------------------------------------------------------------------------------------------------
export DEPLOY_TYPE="base"

# -------------------------------------------------------------------------------------------------------------------------------
# Storage
# -------------------------------------------------------------------------------------------------------------------------------
export APP_STORAGE_PATH="/var/lib/aiops/storage"
export PLATFORM_STORAGE_PATH="/var/lib/aiops/platform"

# -------------------------------------------------------------------------------------------------------------------------------
# Variables for offline installation
# If you need to specify a custom path for your target registry, then export TARGET_REGISTRY with a custom path. For example:
# export TARGET_REGISTRY=$TARGET_REGISTRY_HOST:$TARGET_REGISTRY_PORT/my/custom/path
# -------------------------------------------------------------------------------------------------------------------------------
export TARGET_REGISTRY_HOST=<IP_or_FQDN_of_target_registry>
export TARGET_REGISTRY_PORT=<port_number_of_target_registry>
export TARGET_REGISTRY=$TARGET_REGISTRY_HOST:$TARGET_REGISTRY_PORT
export TARGET_REGISTRY_USER=<your-registry-username>
export TARGET_REGISTRY_PASSWORD=<your-registry-password>
export SKIP_TLS=false # Set to true if your target registry does not provide a signed TLS certificate

Run the following command to source your script and set the environment variables:

. ./aiops_var.sh

5. Mirror images

1. Connect your bastion host to the internet and disconnect it from the air-gapped environment.

2. Run the following commands to download the aiopsctl tool and its prerequisites.

   AIOPSCTL_TAR="aiopsctl-linux_amd64.tar.gz"
   AIOPSCTL_INSTALL_URL="https://github.com/IBM/aiopsctl/releases/download/v4.8.1/${AIOPSCTL_TAR}"
   curl -LO "${AIOPSCTL_INSTALL_URL}"
   tar xf "${AIOPSCTL_TAR}"
   mv aiopsctl /usr/local/bin/aiopsctl
   
   aiopsctl bastion login cp.icr.io -u cp -p ${IBM_ENTITLEMENT_KEY}
   aiopsctl bastion login ${TARGET_REGISTRY} -u ${TARGET_REGISTRY_USER} -p ${TARGET_REGISTRY_PASSWORD} --insecure-skip-tls-verify="${SKIP_TLS}"
   unset IBM_ENTITLEMENT_KEY
   
   aiopsctl bastion mirror-images --registry ${TARGET_REGISTRY}
   

   You can use the following command to monitor the mirroring process:

   tail -f /root/.aiopsctl/logs/aiopsctl.log
   

3. Disconnect the bastion host from the internet, and connect it to the air-gapped environment.

6. Install the aiopsctl tool and register cluster nodes

1. Run the following commands to install aiopsctl on your offline cluster, and then register the control plane nodes and worker nodes.

   scp /usr/local/bin/aiopsctl ${TARGET_USER}@${CONTROL_PLANE_NODE}:/usr/local/bin/aiopsctl
   
   if [ "$(command -v "oc")"  ]; then
     scp "$(command -v "oc")" ${TARGET_USER}@${CONTROL_PLANE_NODE}:/usr/local/bin/oc
   else
     scp ~/.aiopsctl/mirror/oc ${TARGET_USER}@${CONTROL_PLANE_NODE}:/usr/local/bin/oc
   fi   
   
   echo "Installing main control plane node ${CONTROL_PLANE_NODE}"
   
   ssh ${TARGET_USER}@${CONTROL_PLANE_NODE} aiopsctl cluster node up --accept-license=${ACCEPT_LICENSE} --role=control-plane --registry="${TARGET_REGISTRY}" --registry-user="${TARGET_REGISTRY_USER}" --registry-token="${TARGET_REGISTRY_PASSWORD}" --offline --insecure-skip-tls-verify="${SKIP_TLS}" --app-storage "${APP_STORAGE_PATH}" --platform-storage "${PLATFORM_STORAGE_PATH}" --load-balancer-host="${LOAD_BALANCER_HOST}"
   
   # Configure kubeconfig for oc CLI
   ssh ${TARGET_USER}@${CONTROL_PLANE_NODE} "mkdir -p ~/.kube && ln -s /etc/rancher/k3s/k3s.yaml ~/.kube/config"
   
   K3S_TOKEN=$(ssh ${TARGET_USER}@${CONTROL_PLANE_NODE} aiopsctl cluster node info --token-only)
   K3S_HOST=$(ssh ${TARGET_USER}@${CONTROL_PLANE_NODE} aiopsctl cluster node info --server-url-only)
   
   echo "Installing additional control plane nodes"
   for CP_NODE in "${ADDITIONAL_CONTROL_PLANE_NODES[@]}"; do
     ssh ${TARGET_USER}@${CONTROL_PLANE_NODE} scp /usr/local/bin/aiopsctl ${TARGET_USER}@${CP_NODE}:/usr/local/bin/aiopsctl
     ssh ${TARGET_USER}@${CONTROL_PLANE_NODE} ssh ${TARGET_USER}@${CP_NODE} aiopsctl cluster node up --accept-license=${ACCEPT_LICENSE} --role=control-plane --server-url="${K3S_HOST}" --token="${K3S_TOKEN}" --registry="${TARGET_REGISTRY}" --registry-user="${TARGET_REGISTRY_USER}" --registry-token="${TARGET_REGISTRY_PASSWORD}" --offline --app-storage "${APP_STORAGE_PATH}" --platform-storage "${PLATFORM_STORAGE_PATH}" --load-balancer-host="${LOAD_BALANCER_HOST}"
   done
   
   echo "Installing worker nodes"
   K3S_LB_HOST="https://${LOAD_BALANCER_HOST}:6443"
   for WORKER_NODE in "${WORKER_NODES[@]}"
   do
     ssh ${TARGET_USER}@${CONTROL_PLANE_NODE} scp /usr/local/bin/aiopsctl ${TARGET_USER}@${WORKER_NODE}:/usr/local/bin/aiopsctl
     ssh ${TARGET_USER}@${CONTROL_PLANE_NODE} ssh ${TARGET_USER}@${WORKER_NODE} aiopsctl cluster node up --accept-license=${ACCEPT_LICENSE} --role=worker --server-url="${K3S_LB_HOST}" --token="${K3S_TOKEN}" --registry="${TARGET_REGISTRY}" --registry-user="${TARGET_REGISTRY_USER}" --registry-token="${TARGET_REGISTRY_PASSWORD}" --app-storage "${APP_STORAGE_PATH}" --offline --insecure-skip-tls-verify="${SKIP_TLS}"
   done
   

   Note: The preceding commands install the Red Hat OpenShift CLI (oc) if it is not already installed. If it is already installed, then you must ensure that it is at version 4.14 or higher.

2. Run the following command to see information about your cluster:

   ssh ${TARGET_USER}@${CONTROL_PLANE_NODE} aiopsctl cluster node info
   

   Example output:

   # ssh ${TARGET_USER}@${CONTROL_PLANE_NODE} aiopsctl cluster node info
   Node Details
   Role: Control plane
   Server URL: https://test-server.acme.com:6443
   Token: ***** (Use --show-secrets to reveal token)
   

   
   

3. Check for network connectivity issues.

   VMWare vSphere virtual machines (VMs) with Red Hat® Enterprise Linux® can have connectivity issues. If you are not using vSphere VMs for your IBM Cloud Pak for AIOps deployment, then skip this step. Otherwise, run the following lookup command:

   nslookup kubernetes.default.svc.cluster.local 10.43.0.10
   

   If the lookup is successful, then proceed to the next step. If the lookup fails, then run the following command on each of your VMs to rectify connectivity problems:

   ethtool -K flannel.1 tx-checksum-ip-generic off
   

   For more information, see Installation on Linux fails with network connectivity errors when using aiopsctl to install on vSphere VMs.

7. Install IBM Cloud Pak for AIOps

Run the aiopsctl tool to install IBM Cloud Pak for AIOps.

If you did not configure a custom certificate in step 3 then run the following command:

ssh ${TARGET_USER}@${CONTROL_PLANE_NODE} aiopsctl server up --load-balancer-host="${LOAD_BALANCER_HOST}" --mode "${DEPLOY_TYPE}"

If you configured a custom certificate in step 3 then run the following command:

ssh ${TARGET_USER}@${CONTROL_PLANE_NODE} aiopsctl server up --load-balancer-host="${LOAD_BALANCER_HOST}" --mode "${DEPLOY_TYPE}" --certificate-file aiops-certificate-chain.pem --key-file aiops.key.pem

8. Verify your installation

The installation takes one to two hours to complete. If the installation is unsuccessful, an error message is displayed and a nonzero exit code is returned.

Run the following command to check the status of the components of your IBM Cloud Pak for AIOps installation:

aiopsctl status

Example output for a healthy installation:

$ aiopsctl status
o- [12 Aug 24 08:40 PDT] Getting cluster status
Control Plane Node(s):
    test-server-1.acme.com Ready
    test-server-2.acme.com Ready
    test-server-3.acme.com Ready

Worker Node(s):
    test-agent-1.acme.com Ready
    test-agent-2.acme.com Ready
    test-agent-3.acme.com Ready
    test-agent-4.acme.com Ready
    test-agent-5.acme.com Ready
    test-agent-6.acme.com Ready
    test-agent-7.acme.com Ready

o- [12 Aug 24 08:40 PDT] Checking AIOps installation status

  15 Ready Components
    cluster
    aimanager
    lifecycletrigger
    aiopsanalyticsorchestrator
    baseui
    elasticsearchcluster
    aiopsedge
    rediscp
    asm
    zenservice
    aiopsui
    commonservice
    kafka
    issueresolutioncore
    lifecycleservice

  AIOps installation healthy

If the installation fails, or is not complete and is not progressing, then see Troubleshooting installation and upgrade and Known Issues to help you identify any installation problems.

9. Access the Cloud Pak for AIOps console

Run the following command on a control plane node to see the URL, username, and password for your IBM Cloud Pak for AIOps deployment.

aiopsctl server info --show-secrets

Example output:

aiopsctl server info --show-secrets

Cluster Access Details
URL:      aiops-cpd.test-server.acme.com
Username: cpadmin
Password: abcdefghijklmno

Important: Ensure that your environment's Domain Name System (DNS) is correctly configured to resolve the hosts for accessing the Cloud Pak for AIOps console. For more information, see DNS requirements.

What to do next

Important: Any commands that use oc must be run from a control plane node.

• Define integrations and applications with Defining.
• If you have an existing on-premises IBM Tivoli Netcool/OMNIbus deployment, then you can connect it to IBM Cloud Pak for AIOps through an integration. For more information, see Creating IBM Tivoli Netcool/OMNIbus integrations.
• If you have an existing on-premises IBM Tivoli Netcool/Impact deployment, then you can connect it to IBM Cloud Pak for AIOps through an integration. For more information, see Creating IBM Tivoli Netcool/Impact integrations.

To uninstall your IBM Cloud Pak for AIOps on Linux deployment, follow the instructions in Uninstalling a deployment of IBM Cloud Pak for AIOps on Linux.