Installing cloud native Netcool Operations Insight components in an air-gapped environment (offline) with cloudctl and a portable storage device

If your cluster is not connected to the internet, you can deploy a production install of IBM® Netcool® Operations Insight® on Red Hat® OpenShift® on your cluster by using a portable storage device.

Note: Support for the cloudctl installation method is deprecated in this release. You can install IBM Netcool Operations Insight on Red Hat OpenShift with the oc ibm-pak plug-in. For more information, see Installing IBM Netcool Operations Insight on Red Hat OpenShift in an air-gapped environment (offline) with the oc ibm-pak plug-in.

Before you begin

Review the Planning section. Your environment must meet the system requirements.

If you want to install Netcool Operations Insight on Red Hat OpenShift components in an air-gapped environment as a nonroot user, review the information in the Install commands that require root or sudo access topic.

The operator images for Netcool Operations Insight on Red Hat OpenShift are in the freely accessible operator repository (icr.io/cpopen), and the operand images are in the IBM Entitled Registry (cp.icr.io), for which you require an entitlement key. The CASE bundle is available from IBM cloudPaks. Ensure that you have access to icr.io:443 for the IBM Entitled Registry and Netcool Operations Insight on Red Hat OpenShift catalog source.

For more information about the OLM, see https://access.redhat.com/documentation/en-us/openshift_container_platform/4.10/html/operators/understanding-operators#operator-lifecycle-manager-olm external link.

Ensure that you have access to quay.io:443 for the Netcool Operations Insight on Red Hat OpenShift catalog and images. The quay.io/openshift-release-dev/ocp-v4.0-art-dev image is required and must be present in the cluster to install the NOI operator.

Important: The following procedure is based on a Red Hat OpenShift 4.10 environment and includes links for that version. If your environment uses a different supported version of Red Hat OpenShift, ensure that you follow the Red Hat OpenShift documentation for that version.

Prepare a host

Complete the following steps on your host:
  1. Install OpenSSL 1.11.1 or later. For more information, see Downloads external link in the OpenSSL documentation.
  2. Install Docker or Podman.
    • To install Docker (for example, on Red Hat Enterprise Linux®), run the following commands:
      yum check-update
      yum install docker
      Note: Docker is not shipped or supported by Red Hat for Red Hat Enterprise Linux (RHEL) 8. The podman container engine replaced docker as the preferred, maintained, and supported container runtime of choice for Red Hat Enterprise Linux 8 systems. For more information, see Running containers without Docker external link in the Red Hat documentation.
    • To install Podman, see Podman Installation Instructions external link in the Podman documentation.
  3. Install httpd-tools.
    yum install httpd-tools
  4. Download and install IBM® Cloud Pak CLI (cloudctl) version 3.20.0 of cloudctl-linux-amd64.tar.gz from the IBM/cloud-pak-cli external link GitHub repository. Version 3.20.0 is the maximum supported version for installing Event Manager.
    1. Extract the binary file by entering the following command:
      tar -xf cloudctl-linux-amd64.tar.gz
    2. Run the following command to modify the file:
      chmod 755 cloudctl-linux-amd64
    3. Run the following command to move the file to the /usr/local/bin directory:
      mv cloudctl-linux-amd64 /usr/local/bin/cloudctl
      Note: You can confirm that cloudctl is installed by entering the cloudctl --help command.
    The cloudctl usage is displayed.
  5. Install the oc Red Hat OpenShift CLI tool. For more information, see Install the OpenShift CLI.
  6. Install the skopeo CLI 1.1.9 or later. For more information, see install.md external link in the skopeo repo external link.
  7. Create a directory that serves as the offline store. The following directory is an example directory, which is used in subsequent steps.
    mkdir $HOME/offline
    Note: This offline store must be persistent to avoid transferring data more than once. The persistence also helps to run the mirroring process multiple times or on a schedule.
The portable device must be configured such that it can independently connect to the internet and to the air-gapped network.

Download the case bundle

  1. Set the following variables:
    export CASE_NAME=ibm-netcool-prod
    export CASE_VERSION=1.7.0
    export CASE_ARCHIVE=$CASE_NAME-$CASE_VERSION.tgz
    export CASE_INVENTORY_SETUP=noiOperatorSetup
    export OFFLINEDIR=$HOME/offline
    export OFFLINEDIR_ARCHIVE=offline.tgz
    export CASE_REPO_PATH=https://github.com/IBM/cloud-pak/raw/master/repo/case
    export CASE_LOCAL_PATH=$OFFLINEDIR/$CASE_ARCHIVE
    export PORTABLE_DOCKER_REGISTRY_HOST=localhost
    export PORTABLE_DOCKER_REGISTRY_PORT=443
    export PORTABLE_DOCKER_REGISTRY=$PORTABLE_DOCKER_REGISTRY_HOST:$PORTABLE_DOCKER_REGISTRY_PORT
    export PORTABLE_DOCKER_REGISTRY_USER=localuser
    export PORTABLE_DOCKER_REGISTRY_PASSWORD=l0calPassword!
    export PORTABLE_DOCKER_REGISTRY_PATH=$OFFLINEDIR/imageregistry
    export IBM_ENTITLEMENT_KEY=
  2. Note: If you want to install the previous 1.6.6 version, specify --version 1.6.0 in the cloudctl case save command. For more information, see the 1.6.6 documentation: https://www.ibm.com/docs/en/noi/1.6.6?topic=installing
    Download the Netcool Operations Insight CASE bundle (ibm-netcool-prod), into a local directory on your portable storage device.
    cloudctl case save \
      --case $CASE_NAME \
      --version $CASE_VERSION \
      --outputdir $OFFLINEDIR \
      --repo https://github.com/IBM/cloud-pak/raw/master/repo/case
  3. Verify that the Netcool Operations Insight CASE bundle, images.csv, and charts.csv are successfully downloaded by running the following command:
    find $OFFLINEDIR -type f

Setting up the portable docker registry

Run a Docker registry service from your connected device (localhost) by completing the following steps:
  1. Open your CLI and go to your $HOME directory. Initialize the Docker type registry by running the following command:
    cloudctl case launch \
      --case $CASE_LOCAL_PATH \
      --inventory $CASE_INVENTORY_SETUP \
      --action init-registry \
      --args "--registry $PORTABLE_DOCKER_REGISTRY_HOST --port $PORTABLE_DOCKER_REGISTRY_PORT --user $PORTABLE_DOCKER_REGISTRY_USER --pass $PORTABLE_DOCKER_REGISTRY_PASSWORD --dir $PORTABLE_DOCKER_REGISTRY_PATH"
  2. Start the Docker type registry by running the following command:
    cloudctl case launch \
      --case $CASE_LOCAL_PATH \
      --inventory $CASE_INVENTORY_SETUP \
      --action start-registry \
      --args "--registry $PORTABLE_DOCKER_REGISTRY_HOST --port $PORTABLE_DOCKER_REGISTRY_PORT --user $PORTABLE_DOCKER_REGISTRY_USER --pass $PORTABLE_DOCKER_REGISTRY_PASSWORD --dir $PORTABLE_DOCKER_REGISTRY_PATH"
Your host is now configured and you are ready to mirror your images.

Mirroring images to the portable docker registry

  1. Set up access to the IBM Entitled Registry, cp.icr.io, which you pull images from. Run the following commands on your portable storage device.
    cloudctl case launch \
        --case $CASE_LOCAL_PATH \
        --inventory noiOperatorSetup \
        --action configure-creds-airgap \
        --args "--registry cp.icr.io --user cp --pass $IBM_ENTITLEMENT_KEY"
    cloudctl case launch \
        --case $CASE_LOCAL_PATH \
        --inventory noiOperatorSetup \
        --action configure-creds-airgap \
        --args "--registry $PORTABLE_DOCKER_REGISTRY --user $PORTABLE_DOCKER_REGISTRY_USER --pass $PORTABLE_DOCKER_REGISTRY_PASSWORD"
  2. Before mirroring images, set CLOUDCTL_CASE_USE_CATALOG_DIGEST by running the following command:
    export CLOUDCTL_CASE_USE_CATALOG_DIGEST=1
  3. Mirror the images from IBM Entitled registry to the portable docker registry by running the following commands:
    cloudctl case launch \
        --case $CASE_LOCAL_PATH \
        --inventory noiOperatorSetup \
        --action mirror-images \
        --args "--registry $PORTABLE_DOCKER_REGISTRY --inputDir $OFFLINEDIR --chunks 14"

Copying the offline data to the portable storage device

  1. Save the registry image.
    • For a Podman registry, run the following command to save the image:
      podman save -o $PORTABLE_DOCKER_REGISTRY_PATH/registry-image.tar docker.io/library/registry:<tag>
      Where <tag> is the registry tag, such as 2.8.1. To display the tag for a Podman registry, run the podman images command, as in the following example:
      podman images
      REPOSITORY                  TAG         IMAGE ID      CREATED      SIZE
      docker.io/library/registry >>> 2.8.1  <<<<     773dbf02e42e  3 weeks ago  24.6 MB
    • For a Docker registry, run the following command to save the image:
      docker save -o $PORTABLE_DOCKER_REGISTRY_PATH/registry-image.tar docker.io/library/registry:<tag>
      Where <tag> is the registry tag, such as 2.8.1. To display the tag for a Docker registry, run the docker images command, as in the following example:
      docker images
      Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
      REPOSITORY                  TAG         IMAGE ID      CREATED      SIZE
      docker.io/library/registry >>> 2.8.1  <<<<       773dbf02e42e  3 weeks ago  24.6 MB
  2. Copy the saved offline data for a portable storage device. Connect the portable storage device, such as a USB drive or external HDD, to the external host (connected compute device). Archive the offline data for transfer by running the following command:
    tar -cvzf $OFFLINEDIR_ARCHIVE -C $OFFLINEDIR .
    Copy the compressed file to the portable storage device. Physically transfer your portable storage device from the workstation that has a public internet connection to the workstation that has no internet connectivity (your air-gapped environment).

Setting up the local docker registry

Run the following steps on the server with access to Red Hat OpenShift and your local docker registry.
  1. Set up the following environment variables:
    export CASE_NAME=ibm-netcool-prod
    export CASE_VERSION=1.7.0
    export CASE_ARCHIVE=$CASE_NAME-$CASE_VERSION.tgz
    export CASE_INVENTORY_SETUP=noiOperatorSetup
    export OFFLINEDIR=$HOME/offline
    export OFFLINEDIR_ARCHIVE=offline.tgz
    export CASE_REPO_PATH=https://github.com/IBM/cloud-pak/raw/master/repo/case
    export CASE_LOCAL_PATH=$OFFLINEDIR/$CASE_ARCHIVE
    export PORTABLE_DOCKER_REGISTRY_HOST=localhost
    export PORTABLE_DOCKER_REGISTRY_PORT=443
    export PORTABLE_DOCKER_REGISTRY=$PORTABLE_DOCKER_REGISTRY_HOST:$PORTABLE_DOCKER_REGISTRY_PORT
    export PORTABLE_DOCKER_REGISTRY_USER=localuser
    export PORTABLE_DOCKER_REGISTRY_PASSWORD=l0calPassword!
    export PORTABLE_DOCKER_REGISTRY_PATH=$OFFLINEDIR/imageregistry
    export LOCAL_DOCKER_REGISTRY_HOST=<IP_or_FQDN_of_local_docker_registry>
    export LOCAL_DOCKER_REGISTRY_PORT=<port_number_of_local_docker_registry>
    export LOCAL_DOCKER_REGISTRY=$LOCAL_DOCKER_REGISTRY_HOST:$LOCAL_DOCKER_REGISTRY_PORT
    export LOCAL_DOCKER_REGISTRY_USER=<username>
    export LOCAL_DOCKER_REGISTRY_PASSWORD=<password>
    export TARGET_NAMESPACE=<namespace-for-noi>
  2. Extract the transferred offline data from the portable storage device:
    mkdir -p $OFFLINEDIR 
    tar -xvf $OFFLINEDIR_ARCHIVE -C $OFFLINEDIR
    
  3. Set up the local docker registry. Run the local docker registry as a container. The registry then points to the docker file system directory that was transferred from the external host:
    cloudctl case launch \
      --case $CASE_LOCAL_PATH \
      --inventory $CASE_INVENTORY_SETUP \
      --action start-registry \
      --args "--registry $PORTABLE_DOCKER_REGISTRY_HOST --port $PORTABLE_DOCKER_REGISTRY_PORT --user $PORTABLE_DOCKER_REGISTRY_USER --pass $PORTABLE_DOCKER_REGISTRY_PASSWORD --dir $PORTABLE_DOCKER_REGISTRY_PATH"
  4. Configure an authentication secret for the local docker registry.
    cloudctl case launch \
      --case $CASE_LOCAL_PATH \
      --inventory $CASE_INVENTORY_SETUP \
      --action configure-creds-airgap \
      --args "--registry $PORTABLE_DOCKER_REGISTRY --user $PORTABLE_DOCKER_REGISTRY_USER --pass $PORTABLE_DOCKER_REGISTRY_PASSWORD"
  5. Store credentials of the registry that is going to serve images to the cluster or workloads:
    cloudctl case launch \
      --case $CASE_LOCAL_PATH \
      --inventory $CASE_INVENTORY_SETUP \
      --action configure-creds-airgap \
      --args "--registry $LOCAL_DOCKER_REGISTRY --user $LOCAL_DOCKER_REGISTRY_USER --pass $LOCAL_DOCKER_REGISTRY_PASSWORD"
  6. Before mirroring images, set CLOUDCTL_CASE_USE_CATALOG_DIGEST by running the following command:
    export CLOUDCTL_CASE_USE_CATALOG_DIGEST=1
  7. Mirror the images from the portable registry to the target registry on your cluster:
    cloudctl case launch \
      --case $CASE_LOCAL_PATH \
      --inventory $CASE_INVENTORY_SETUP \
      --action mirror-images \
      --args "--fromRegistry $PORTABLE_DOCKER_REGISTRY --registry $LOCAL_DOCKER_REGISTRY --inputDir $OFFLINEDIR --chunks 14"

Configure Red Hat OpenShift cluster for airgap.

Warning:
  • Cluster resources must adjust to the new pull secret, which can temporarily limit the usability of the cluster. Authorization credentials are stored in $HOME/.airgap/secrets and /tmp/airgap* to support this action.
Configure your Red Hat OpenShift cluster for airgap. This step can take 90+ minutes, if nodes are recycled.

Insecure registry

If your offline registry is insecure, complete the following steps for registry access:
  • Login as kubeadmin with the following command:
    oc login -u kubeadmin -p <password> <server's REST API URL>
  • Run the following command on your bastion server to create a global image pull secret for the target registry, and create an ImageContentSourcePolicy.
    oc patch image.config.openshift.io/cluster --type=merge -p '{"spec":{"registrySources":{"insecureRegistries":["'${LOCAL_DOCKER_REGISTRY}'"]}}}'
  • Verify that the ImageContentSourcePolicy is created successfully, by running the oc get MachineConfigPool command. Confirm that the value in the UPDATED column is set to true, as in the following example:
    oc get MachineConfigPool
    NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
    master   rendered-master-cd77bc65ebc2442ad4336bca7b61a6be   True      False      False      3              3                   3                     0                      25h
    worker   rendered-worker-6418363c7fa863367edecf08eca421e4   True      False      False      4              4                   4                     0                      25h

Secure registry

If your offline registry is secure, complete the following steps for registry access. You can add certificate authorities (CA) to the cluster for use when pushing and pulling images. You must have cluster administrator privileges. You must have access to the public certificates of the registry, usually a hostname/ca.crt file located in the /etc/docker/certs.d/ directory.
  • Login as kubeadmin with the following command:
    oc login -u kubeadmin -p <password> <server's REST API URL>
  • Create a configmap in the openshift-config namespace that contains the trusted certificates for the registries that use self-signed certificates. For each CA file, ensure that the key in the configmap is the hostname of the registry, in the hostname[..port] format:
    oc create configmap registry-cas -n openshift-config
    --from-file=$TARGET_REGISTRY_HOST..$TARGET_REGISTRY_PORT=/etc/docker/certs.d/myregistry.corp.com:$TARGET_REGISTRY_PORT/ca.crt
  • Update the cluster image configuration:
    oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"registry-cas"}}}' --type=merge
  • Run the following commands for your secure or insecure registry:
    cloudctl case launch \
      --case $OFFLINEDIR/$CASE_ARCHIVE \
      --namespace $TARGET_NAMESPACE \
      --inventory noiOperatorSetup \
      --action configure-cluster-airgap \
      --args "--registry $LOCAL_DOCKER_REGISTRY --inputDir $OFFLINEDIR --user $LOCAL_DOCKER_REGISTRY_USER --pass $LOCAL_DOCKER_REGISTRY_PASSWORD" 
  • Monitor the machineconfigpool pod to verify that the update is completely pushed to all nodes. Ensure that the UPDATED column displays True for master and worker nodes, as in the following example:
    oc get machineconfigpool
    NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
    master   rendered-master-f4b89d6ca38d7b11a8a2decaf59ba6dd   True      False      False      3              3                   3                     0                      13h
    worker   rendered-worker-75afdaafd9615c58a165b118fef04750   True      False      False      4              4                   4                     0                      13h

Install the Netcool Operations Insight catalog and operator.

  1. Install the catalog by using CASE:
    Run the following commands
    oc create secret docker-registry target-registry-secret \
        --docker-server=$LOCAL_DOCKER_REGISTRY \
        --docker-username=$LOCAL_DOCKER_REGISTRY_USER \
        --docker-password=$LOCAL_DOCKER_REGISTRY_PASSWORD \
        --namespace=$TARGET_NAMESPACE
    Set the project (namespace) to install the catalog:
    export CATALOG_NAMESPACE=openshift-marketplace
    You can set the project to a custom namespace.
    Create the catalog source:
    cloudctl case launch \
      --case $CASE_LOCAL_PATH \
      --inventory $CASE_INVENTORY_SETUP \
      --action install-catalog \
      --namespace $CATALOG_NAMESPACE \
      --args "--registry $LOCAL_DOCKER_REGISTRY/cpopen"
  2. Verify the Netcool Operations Insight catalog source. From the Red Hat OpenShift OLM UI, go to Administration > Cluster Settings, and then select the OperatorHub configuration resource under the Global Configuration tab. Verify that the noi-operator-catalog catalog source is present.
  3. Install the Netcool Operations Insight operator by using CASE. Run the following command:
    cloudctl case launch \
        --case $CASE_LOCAL_PATH \
        --namespace $TARGET_NAMESPACE \
        --inventory $CASE_INVENTORY_SETUP \
        --action install-operator \
        --args "--secret target-registry-secret"
    Where:
    • $TARGET_NAMESPACE is the custom namespace to be used for your deployment.
    • target-registry-secret is the secret for accessing the target registry that you created previously.
  4. Verify the status of IBM Cloud Pak for Watson™ AIOps Event Manager. From the Red Hat OpenShift OLM UI, go to Operators > Installed Operators, and verify that the status of IBM Cloud Pak for Watson AIOps Event Manager is Succeeded.

Create a Netcool Operations Insight instance.

  1. From the Red Hat OpenShift OLM UI, go to Operators > Installed Operators, and select IBM Cloud Pak for Watson AIOps Event Manager. Under Provided APIs > NOIHybrid, select the project that you want to install NOIHybrid into. Next, select Create Instance.
  2. From the Red Hat OpenShift OLM UI, use the YAML view or the Form view to configure the properties for the Netcool Operations Insight deployment. For more information about configurable properties for a cloud-only deployment, see Hybrid operator properties.
    Note: Ensure that the name of the Netcool Operations Insight instance does not exceed 10 characters.
  3. Edit the Netcool Operations Insight properties to provide access to the target registry. Set spec.entitlementSecret to the target registry secret.
  4. Select Create.
  5. Under the All Instances tab, a Netcool Operations Insight instance appears. To monitor the status of the installation, see Monitoring cloud installation progress.
    Note:
    • Changing an existing deployment from a Trial deployment type to a Production deployment type is not supported.
    • Changing an instance's deployment parameters in the Form view is not supported post deployment.
    • If you update custom secrets in the OLM console, the crypto key is corrupted and the command to encrypt passwords does not work. Update custom secrets only with the CLI. For more information about storing a certificate as a secret, see https://www.ibm.com/docs/en/SS9LQB_1.1.16/LoadingData/t_asm_obs_configuringsecurity.html external link

What to do next

  • To enable or disable a feature or observer after installation edit the Netcool Operations Insight instance by running the command:
    oc edit noi <noi-instance-name>
    Where <noi-instance-name> is the name of the deployment that you want to change.

    You can then select to enable or disable the feature or observer. When you disable features postinstallation , the resource is not automatically deleted. To find out whether the feature is deleted, you must check the asm-operator log.