Installing cloud native Netcool Operations Insight components in an air-gapped environment (offline) with cloudctl and a bastion host

Follow these instructions to deploy an installation of cloud native Netcool® Operations Insight® components for a hybrid deployment on an offline environment, using Container Application Software for Enterprises (CASE) and the Red Hat® OpenShift® Operator Lifecycle Manager (OLM)

Create a target registry to store all the images locally

1. Install and start a production grade Docker 2 compatible registry, such as Quay Enterprise, JFrog Artifactory, or Docker Registry.
   The target registry must be accessible by the Red Hat OpenShift cluster and the bastion host. The Red Hat OpenShift internal registry is not supported.
2. Set up the following variables:

   export CASE_NAME=ibm-netcool-prod
   export CASE_VERSION=1.6.0
   export CASE_ARCHIVE=$CASE_NAME-$CASE_VERSION.tgz
   export OFFLINEDIR=$HOME/offline
   export CASE_LOCAL_PATH=$OFFLINEDIR/$CASE_ARCHIVE
   export CASE_REPO_PATH=https://github.com/IBM/cloud-pak/raw/master/repo/case
   export LOCAL_DOCKER_REGISTRY_HOST=<IP_or_FQDN_of_local_docker_registry>
   export LOCAL_DOCKER_REGISTRY_PORT=<port_number_of_local_docker_registry>
   export LOCAL_DOCKER_REGISTRY=$LOCAL_DOCKER_REGISTRY_HOST:$LOCAL_DOCKER_REGISTRY_PORT
   export LOCAL_DOCKER_REGISTRY_USER=<user>
   export LOCAL_DOCKER_REGISTRY_PASSWORD=<password>
   export TARGET_NAMESPACE=<Event Manager namespace>
   export IBM_ENTITLEMENT_KEY=<your entitlement key>

3. You must have a username and password configured for your target registry. Create a secret for access to the target registry
   Run the following command on your Red Hat OpenShift cluster.
   Create a $TARGET_NAMESPACE namespace:

   oc create namespace $TARGET_NAMESPACE

   Create a secret:

   oc create secret docker-registry target-registry-secret \
       --docker-server=$LOCAL_DOCKER_REGISTRY \
       --docker-username=$LOCAL_DOCKER_REGISTRY_USER \
       --docker-password=$LOCAL_DOCKER_REGISTRY_PASSWORD \
       --namespace=$TARGET_NAMESPACE

   Where:

   • target-registry-secret is the name of the secret that you are creating. Suggested value is target-registry-secret.
   • $LOCAL_DOCKER_REGISTRY is the target registry that you created.
   • $LOCAL_DOCKER_REGISTRY_USER and $LOCAL_DOCKER_REGISTRY_PASSWORD are the credentials to access your target registry.
   • $TARGET_NAMESPACE is the namespace that you want to deploy Netcool Operations Insight in.
4. Install the skopeo CLI 1.1.9 or later. For more information, see install.md in the skopeo repo .
5. Install Docker version 1.38 or later, and start the docker daemon. For more information, see https://docs.docker.com/install/.

Prepare the bastion host

6. Verify the bastion server's access.
   Log on to the bastion server and verify that it has access to the following components:

   • public internet - to download the Netcool Operations Insight CASE and images from the source registries.
   • target registry - where the images are mirrored.
   • target Red Hat OpenShift cluster - to install the Netcool Operations Insight operator.
7. Download and install the following items onto the bastion server.
   • cloudctl- Download and install IBM® Cloud Pak CLI (cloudctl) version 3.17.0 from the cloud-pak-cli GitHub repository .

   cloudctl verifies the integrity of the Netcool Operations Insight CASE's digital signature by default. If you want to verify the cloudctl binary, follow the instructions in https://github.com/IBM/cloud-pak-cli#check-certificatekey-validity. Extract the cloudctl binary, give it executable permissions, and ensure that it is in your PATH.
   • oc - Download and install the Red Hat OpenShift CLI (oc) 4.8.x or later. For more information, see Getting started with the OpenShift CLI in the Red Hat OpenShift documentation.

Download CASE bundle onto the bastion server

8. Note: If you want to install the previous 1.6.5 version, specify --version 1.5.0 in the cloudctl case save command. For more information, see the 1.6.5 documentation: https://www.ibm.com/docs/en/noi/1.6.5?topic=installing
   Download the Netcool Operations Insight CASE bundle (ibm-netcool-prod), into a local directory on your bastion server.

   mkdir $OFFLINEDIR

   cloudctl case save \
     --case $CASE_NAME \
     --outputdir $OFFLINEDIR \
     --repo $CASE_REPO_PATH \
     --version $CASE_VERSION
   

   1. Filter out the unwanted digests from the images.csv file. Remove ppc64le and s390x architecture images and opencontent-redis-5 images with version tag=2.0.13 from the images.csv file.
9. Verify that the Netcool Operations Insight CASE bundle, images.csv, and charts.csv are successfully downloaded on your bastion server, with the following command:

   find $OFFLINEDIR -type f

Configure bastion server authentication

10. Set up access to the IBM Entitled Registry, cp.icr.io, which you pull images from.
    Run the following command on your bastion server:

    cloudctl case launch \
        --case $CASE_LOCAL_PATH \
        --namespace $TARGET_NAMESPACE \
        --inventory noiOperatorSetup \
        --action configure-creds-airgap \
        --args "--registry cp.icr.io --user cp --pass $IBM_ENTITLEMENT_KEY"

    Where

    • $TARGET_NAMESPACE is the custom namespace that you want to deploy Netcool Operations Insight into.
    • $IBM_ENTITLEMENT_KEY is your IBM Entitled Registry entitlement key, as found when you prepared your cluster.

Mirror images from CASE to the target registry in the airgap environment

11. Before mirroring images, set CLOUDCTL_CASE_USE_CATALOG_DIGEST by running the command:

    export CLOUDCTL_CASE_USE_CATALOG_DIGEST=1

12. Mirror images from CASE to the target registry. This can take up to 2 hours.
    Run the following command for each input directory on your bastion server:

    cloudctl case launch \
        --case $CASE_LOCAL_PATH \
        --namespace $TARGET_NAMESPACE \
        --inventory noiOperatorSetup \
        --action mirror-images \
        --args "--registry $LOCAL_DOCKER_REGISTRY --inputDir $OFFLINEDIR --chunks 14"

    The images that are listed in the downloaded CASE, (images.csv), are copied to the target registry in the airgap environment.

Configure Red Hat OpenShift Cluster for airgap

13. Warning:

    • Cluster resources must adjust to the new pull secret, which can temporarily limit the usability of the cluster. Authorization credentials are stored in $HOME/.airgap/secrets and /tmp/airgap* to support this action.
    • Applying ImageContentSourcePolicy causes cluster nodes to recycle for Red Hat OpenShift versions 4.7 or earlier.
    • Applying ImageContentSourcePolicy causes cluster nodes to temporarily have a STATUS SchedulingDisabled status for Red Hat OpenShift versions 4.10 or later.
    Configure your Red Hat OpenShift cluster for airgap. This step can take 90+ minutes, if nodes are recycled.
    Note: If you already set up access to the target registry, skip this step.

    Insecure registry: If your offline registry is insecure, complete the following steps for registry access:

    • Login as kubeadmin with the following command:

      oc login -u kubeadmin -p <password> <server's REST API URL>

    • Run the following command on your bastion server to create a global image pull secret for the target registry, and create an ImageContentSourcePolicy.

      oc patch image.config.openshift.io/cluster --type=merge -p '{"spec":{"registrySources":{"insecureRegistries":["'${LOCAL_DOCKER_REGISTRY}'"]}}}'

    Secure registry: If your offline registry is secure, complete the following steps for registry access. You can add certificate authorities (CA) to the cluster for use when pushing and pulling images. You must have cluster administrator privileges. You must have access to the public certificates of the registry, usually a hostname/ca.crt file located in the /etc/docker/certs.d/ directory.

    • Login as kubeadmin with the following command:

      oc login -u kubeadmin -p <password> <server's REST API URL>

    • Create a configmap in the openshift-config namespace containing the trusted certificates for the registries that use self-signed certificates. For each CA file, ensure that the key in the configmap is the hostname of the registry, in the hostname[..port] format. To create the certificate authorities registry configmap, run the following command from your local registry server:

      oc create configmap registry-cas -n openshift-config
      --from-file=$TARGET_REGISTRY_HOST..$TARGET_REGISTRY_PORT=/etc/docker/certs.d/myregistry.corp.com:$TARGET_REGISTRY_PORT/ca.crt

    • Update the cluster image configuration:

      oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"registry-cas"}}}' --type=merge

    Run the following commands for your secure or insecure registry:

    cloudctl case launch \
      --case $OFFLINEDIR/$CASE_ARCHIVE \
      --namespace $TARGET_NAMESPACE \
      --inventory noiOperatorSetup \
      --action configure-cluster-airgap \
      --args "--registry $LOCAL_DOCKER_REGISTRY --inputDir $OFFLINEDIR --user $LOCAL_DOCKER_REGISTRY_USER --pass $LOCAL_DOCKER_REGISTRY_PASSWORD" 

    Monitor the machineconfigpool pod to verify that the update is completely pushed to all nodes. Ensure that the UPDATED column displays True for master and worker nodes, as in the following example:

    oc get machineconfigpool
    NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
    master   rendered-master-f4b89d6ca38d7b11a8a2decaf59ba6dd   True      False      False      3              3                   3                     0                      13h
    worker   rendered-worker-75afdaafd9615c58a165b118fef04750   True      False      False      4              4                   4                     0                      13h

Install the Netcool Operations Insight Catalog and Operator

14. Install the catalog by using CASE. Run the following command on your bastion server:
    Set the project (namespace) to install the catalog:

    export CATALOG_NAMESPACE=openshift-marketplace

    Create the catalog source:

    cloudctl case launch \
      --case $CASE_LOCAL_PATH \
      --inventory noiOperatorSetup \
      --action install-catalog \
      --namespace $CATALOG_NAMESPACE \
      --args "--registry $LOCAL_DOCKER_REGISTRY/cpopen"

    Note: Verify that the image in the catalog source does not reference the $LOCAL_DOCKER_REGISTRY/noi-operator-catalog:latest image. If the image references noi-operator-catalog:latest, set export CLOUDCTL_CASE_USE_CATALOG_DIGEST=1 and re-run the cloudctl case command to re-create the catalog source.
15. Verify the catalog source.
    From the Red Hat OpenShift OLM UI, go to Administration > Cluster Settings. Under Global Configuration > OperatorHub > Sources, verify that the ibm-noi-catalog CatalogSource object is present.
16. Install the Netcool Operations Insight operator by using CASE. Run the following command on your bastion server:

    cloudctl case launch \
      --case $CASE_LOCAL_PATH \
      --inventory noiOperatorSetup \
      --action install-operator \
      --namespace $TARGET_NAMESPACE \
      --args "--registry $LOCAL_DOCKER_REGISTRY"

17. Verify the status of IBM Cloud Pak for Watson™ AIOps Event Manager.
    From the Red Hat OpenShift OLM UI, go to Operators > Installed Operators, and verify that the status of IBM Cloud Pak for Watson AIOps Event Manager is Succeeded.

Create a cloud native Netcool Operations Insight components instance for a hybrid deployment

18. From the Red Hat OpenShift OLM UI, navigate to Operators > Installed Operators, and select IBM Cloud Pak for Watson AIOps Event Manager. Under Provided APIs > NOIHybrid, select Create Instance.
19. From the Red Hat OpenShift OLM UI, use the YAML or the Form view to configure the properties for the cloud native Netcool Operations Insight components deployment. For more information about configurable properties for a hybrid deployment, see Hybrid operator properties.
20. Edit the Netcool Operations Insight properties to provide access to the target registry.
    1. Update spec.advanced.imagePullRepository so that it points to the target registry that you created.
    2. Set spec.entitlementSecret to the target registry secret.
21. Select the Create button.
22. Under the All Instances tab, a Netcool Operations Insight hybrid instance appears.
    Navigate to Operators > Installed Operators and check that the status of your Netcool Operations Insight instance is Phase: OK. Click on Netcool Operations Insight > All Instances to check it. This means that IBM Cloud Pak for Watson AIOps Event Manager has started and is now in the process of starting up the various pods.
    To monitor the status of the installation, see Monitoring cloud installation progress.
    Note:

    • Changing an existing deployment from a Trial deployment type to a Production deployment type is not supported.
    • Changing an instance's deployment parameters in the Form view is not supported post deployment.
    • If you update custom secrets in the OLM console, the crypto key is corrupted and the command to encrypt passwords does not work. Update custom secrets only with the CLI. For more information about storing a certificate as a secret, see https://www.ibm.com/docs/en/SS9LQB_1.1.15/LoadingData/t_asm_obs_configuringsecurity.html

oc edit noihybrid noi-instance-name