Online production installation of IBM Concert Operate (GitOps method)

GitOps can be used to deploy IBM Concert Operate from an IBM Git repository that contains the installation manifests. The deployment process is automated and managed through Argo CD, providing a uniform application delivery that can be repeated across test and production systems.

Red Hat OpenShift includes the Red Hat OpenShift GitOps operator, which automates the full lifecycle management of Argo CD and its components. For more information, see About Red Hat OpenShift GitOps in the Red Hat OpenShift documentation.

Argo CD applications are created and registered with the Argo CD server to define the required state of Kubernetes resources for a deployment of IBM Concert Operate. Each Argo CD application specifies the source repository that contains its manifests, and the target Red Hat OpenShift cluster where those resources are to be deployed. To deploy IBM Concert Operate, the Argo CD server synchronizes the applications to the target cluster, ensuring that the defined resources are applied and maintained in the required state.

For more information about upgrading your IBM Concert Operate with GitOps, see Online upgrade of IBM Concert Operate (GitOps method).

• Review the Planning section, and ensure that you meet the supported platform, hardware, and storage requirements.
• You must know whether you are deploying a base deployment or an extended deployment of IBM Concert Operate. For more information, see Incremental adoption.
• After you have installed IBM Concert Operate, you cannot switch between a multi-zone and non multi-zone environment.
• Ensure that you are logged in to your Red Hat OpenShift cluster with oc login for any steps that use the Red Hat OpenShift command-line interface (CLI).
• If you require details about the permissions that the IBM Concert Operate operators need, see Permissions (IBM Concert Operate).
• A user with cluster-admin privileges is needed for the step Verify cluster readiness - aiopsctl server precheck. A user with cluster-admin privileges is not required for any other steps if you grant sufficient permissions for Argo CD. For more information, see Configure GitOps privileges.

If IBM Sales representatives and Business Partners supplied you with a custom profile ConfigMap to customize your deployment, then you must follow their instructions to apply it during installation. The custom profile cannot be applied after installation, and attempting to do so can break your IBM Concert Operate deployment. For more information about custom sizing, see Custom sizing.

(Optional) Forking and customizing the IBM Concert Operate GitOps repositories

If required, you can host and customize the IBM Concert Operate GitOps repositories in your own Git systems. To customize your IBM Concert Operate deployment by using your own Git repository, use the following steps.

1. Fork the IBM Concert Operate GitOps repository to your own GitHub account, or extract and host the packaged IBM Concert Operate release Helm chart templates in your own GitHub repository. If you use a forked repository or a different branch, then you must update the values of the Repository URL and Revision parameters in your ArgoCD applications to match your repository and branch.

2. Define more template files as needed. For example, you can include a custom sizing ConfigMap for IBM Concert Operate. Template files are automatically detected and deployed by the Argo CD application during synchronization.

3. Add bespoke environment-specific values files, such as for staging, production, or other deployment targets, and include any custom Installation attributes as needed. Then, configure the Argo CD application to reference your bespoke values file during deployment in step 15 Create Argo CD IBM Concert Operate application.

Follow these steps to use GitOps to install IBM Concert Operate.

1. Install and configure Red Hat OpenShift
2. Configure storage
3. Retrieve your entitlement key
4. Create environment variables
5. Create a custom project (namespace)
6. Evaluate storage performance
7. Create the entitlement key secret
8. Configure usage data collection
9. Install Red Hat OpenShift GitOps
10. Configure GitOps privileges
11. Create an Argo CD custom health check application
12. Create an Argo CD IBM Cert Manager application
13. Verify cluster readiness
14. Create an Argo CD IBM Concert Operate application
15. Synchronize applications
16. Verify your installation
17. Verify local storage
18. Create an EgressFirewall
19. Access the Concert Operate console

Allow access to the following sites and ports:

You must be able to download content from GitHub. If you are not able to, verify that your network or proxy settings permit access to GitHub's file server domain and if needed contact your network administrator to allow it.

IBM Concert Operate requires Red Hat OpenShift to be installed and running. You must have administrative access to your Red Hat OpenShift cluster.

For more information about the supported versions of Red Hat OpenShift, see Supported Red Hat OpenShift Container Platform versions.

1. Before installing Red Hat OpenShift, work with your system administrator to verify that the nodes that are intended for the installation have their system clocks synchronized with an NTP server, or are at least manually set to be within a few seconds of one another. If you are installing on a cloud platform, this is usually already configured.

2. Install Red Hat OpenShift by using the instructions in the Red Hat OpenShift documentation .

3. Install the Red Hat OpenShift command line interface (oc) on your cluster's boot node and run oc login. For more information, see the instructions in Getting started with the Red Hat OpenShift CLI .

4. To function properly, distributed platforms and applications such as Red Hat OpenShift and IBM Concert Operate require the system clocks of all of their nodes to be highly synchronized with one another. Discrepancies between the clocks can cause IBM Concert Operate to experience operational issues. All Red Hat OpenShift nodes in the cluster must have access to an NTP server to synchronize their clocks. For more information, see the Red Hat OpenShift documentation .

5. Optionally configure a custom certificate for IBM Concert Operate to use. You can use either of the following methods:

   • Configure a custom certificate for the Red Hat OpenShift cluster. Follow the instructions in the Red Hat OpenShift documentation Replacing the default ingress certificate.
   • If you would like to use a custom certificate for the IBM Concert Operate console only, then after installation is complete follow the instructions in Using a custom certificate.

Configure persistent storage for IBM Concert Operate that satisfies your sizing requirements. For more information about storage, see Storage.

Obtain the IBM entitlement key that is assigned to your IBMid, which is needed to pull the IBM Concert Operate images from the IBM® Entitled Registry.

1. Log in to MyIBM Container Software Library with the IBMid and password details that are associated with the entitled software.

2. In the Entitlement key section, select Copy to copy the entitlement key to the clipboard. You will need to use this value to set the environment variable $IBM_ENTITLEMENT_KEY in the next step.

Create a shell script that defines the environment variables that will be used to provide installation parameters for your deployment. Use the following codeblock as a template, replacing the brackets < ... > with values for your environment. Name the script waiops_var_<project>.sh, where <project> is the value that you used for PROJECT_CP4AIOPS. For example, waiops_var_cp4aiops.sh.

If you are deploying more than one instance of IBM Concert Operate in the same cluster, then each instance must have a different value for PROJECT_CP4AIOPS. If you already have a deployment of IBM Concert Operate in the namespace cp4aiops, then export PROJECT_CP4AIOPS with a different value, such as cp4aiops2. For more information, see Deploying multiple instances on a single cluster.

If you need help with deciding on the values to set for STG_CLASS and STG_CLASS_BLOCK, see Storage class summary table.

#============================================================================================================
# Cloud Pak for AIOps installation variables for deployment by GitOps
#
# GITOPS_NAMESPACE - set to the namespace in which the Argo CD instance is deployed.
# SOURCE_REPOSITORY - if you are using your own forked Git repository or a different branch, then update the
# value of the `SOURCE_REPOSITORY` environment variables to match your repository and branch.
#============================================================================================================
export GITOPS_NAMESPACE=openshift-gitops
export GITOPS_INSTANCE=openshift-gitops
export GITOPS_SERVICEACCOUNT=openshift-gitops-argocd-application-controller
export PROJECT_CP4AIOPS=cp4aiops
export ACCEPT_LICENSE=false # Set to `true` to agree to the license terms, otherwise install will fail.
export SOURCE_REPOSITORY=https://github.com/IBM/cp4aiops-gitops
export TARGET_REVISION=5.1.5

# -------------------------------------------------------------------------------------------------------
# Storage
#-------------------------------------------------------------------------------------------------------
export STG_CLASS=<RWX-storage-class-name>
export STG_CLASS_BLOCK=<RWO-storage-class-name>

# -------------------------------------------------------------------------------------------------------
# Your customer details
# -------------------------------------------------------------------------------------------------------
export CUSTOMER_NAME=<your company name>
export CUSTOMER_ICN=<your IBM Customer Number>
export CUSTOMER_ENVIRONMENT=<Set to `trial`, `poc`, or `production`>

# -------------------------------------------------------------------------------------------------------
# IBM Entitled Registry
# -------------------------------------------------------------------------------------------------------
export IBM_ENTITLEMENT_KEY=<IBM-entitlement-key> # Set to the entitlement key retrieved in previous step.

Run the following command to source your script and set the environment variables:

. ./waiops_var_<project>.sh

Where <project> is the value that you used for PROJECT_CP4AIOPS.

You can set a different value for $PROJECT_CP4AIOPS if you want. However, you must not use the default, kube-system, kube-public, openshift-node, openshift-infra, or openshift projects (namespaces) for $PROJECT_CP4AIOPS. This is because IBM Concert Operate uses Security Context Constraints (SCC), and SCCs cannot be assigned to pods created in one of the default Red Hat OpenShift projects (namespaces).

1. Run the following command to create a project (namespace) to deploy IBM Concert Operate into.

   oc create namespace "${PROJECT_CP4AIOPS}"
   

2. Add a node-selector annotation to the IBM Concert Operate namespace.

   The annotation ensures that on a multi-architecture Red Hat OpenShift cluster, IBM Concert Operate workloads are only scheduled on nodes that have an architecture that IBM Concert Operate supports.

   Failure to do so might result in the scheduling and subsequent failure of IBM Concert Operate workloads on Red Hat OpenShift nodes that have a nonsupported architecture. For more information about supported architectures, see Supported platforms.

   Run one of the following commands.

   If you want to use amd64 architecture:

   oc annotate namespace "${PROJECT_CP4AIOPS}" openshift.io/node-selector="kubernetes.io/arch=amd64"
   

   If you want to use s390x architecture:

   oc annotate namespace "${PROJECT_CP4AIOPS}" openshift.io/node-selector="kubernetes.io/arch=s390x"
   

Use the following procedure to evaluate whether your storage performance is sufficient to withstand the demands of a production deployment of IBM Concert Operate.

1. Download and install the aiopsctl command line interface tool.

   AIOPSCTL_TAR=<aiopsctl_tar>
   AIOPSCTL_INSTALL_URL="https://github.com/IBM/aiopsctl/releases/download/v5.1.5/${AIOPSCTL_TAR}"
   curl -LO "${AIOPSCTL_INSTALL_URL}"
   tar xf "${AIOPSCTL_TAR}"
   mv aiopsctl /usr/local/bin/aiopsctl
   

   Where <aiopsctl_tar> is the operating system specific file that you require from the following set: aiopsctl-linux_s390x.tar.gz, aiopsctl-linux_arm64.tar.gz,aiopsctl-linux_amd64.tar.gz, aiopsctl-darwin_amd64.tar.gz, aiopsctl-darwin_arm64.tar.gz.

2. Evaluate distributed network storage.

   Run the following command:

   aiopsctl benchmark storage --namespace=${PROJECT_CP4AIOPS} --storage-class=${STG_CLASS_BLOCK}
   

   The tool selects a node in your cluster and benchmarks the performance of the node's storage. The process takes around 8 minutes to run.

   If you think that the storage performance between your nodes varies significantly, then you can use the --node <node_name> argument to pass in the name of the node that you want the tool to run on.

3. Verify that your benchmarking results meet or exceed the required metrics.

   The following table identifies the storage performance metrics that must be achieved to support a deployment of IBM Concert Operate. If your deployment is custom-sized to support higher rates than the default production rates listed in Processing abilities, then your storage performance must exceed these metrics.

   Metric	Read	Write
   Minimum sequential IOPS (higher is better, lower is worse)	5000	5000
   Minimum sequential bandwidth (higher is better, lower is worse)	20 Mi/sec	20 Mi/sec
   Maximum average sequential latency (lower is better, higher is worse)	500 usec	1000 usec

   These metrics are not applicable if you are using native storage in a public cloud provider such as Amazon Elastic Block Store (EBS) or IBM Cloud Block Storage. Performance will be adequate if you selected the native storage classes documented for your cloud platform.

4. Evaluate local storage.

   If your IBM Concert Operate deployment does not use hybrid storage then skip this step.

   If your IBM Concert Operate deployment is using hybrid storage, then evaluate the storage performance for Postgres and Kafka, by passing in the Postgres and Kafka storage classes, and verifying that the results meet or exceed the required metrics in the previous step.

   For example:

   aiopsctl benchmark storage --namespace=${PROJECT_CP4AIOPS} --storage-class=lvms-vg-kafka-1
   aiopsctl benchmark storage --namespace=${PROJECT_CP4AIOPS} --storage-class=lvms-vg-ir-core-postgres-1
   aiopsctl benchmark storage --namespace=${PROJECT_CP4AIOPS} --storage-class=lvms-vg-topology-postgres-1
   

Run the following command to create an image pull secret called ibm-entitlement-key to enable your deployment to pull the IBM Concert Operate images from the IBM Entitled Registry.

oc create secret docker-registry ibm-entitlement-key --docker-username=cp --docker-password="${IBM_ENTITLEMENT_KEY}" --docker-server=cp.icr.io --namespace=${PROJECT_CP4AIOPS} 

To help the development of IBM Concert Operate, daily aggregated usage data is collected to analyze how IBM Concert Operate is used. The usage data is collected by the cp4waiops-metricsprocessor pod, and is sent to and stored in IBM controlled GDPR-compliant systems. The collection of usage data is enabled by default, but can be disabled. For transparency, the cp4waiops-metricsprocessor pod's logs contain all the information that is collected. The usage data that is collected is numeric, or is about the deployment type and platform. It does not include email addresses, passwords, or specific details. Only the following data is collected:

• Current number of applications
• Current number of alerts (all severities aggregated)
• Current number of incidents (all priorities aggregated)
• Current number of policies (includes predefined and user created)
• Current number of runbooks run since installation
• Current number of integrations of each type (For example ServiceNow, Instana, Falcon Logscale)
• Secure tunnel enablement: whether connection (which controls whether you can create a secure tunnel) is enabled in the Installation custom resource
• Deployment type: base deployment or extended deployment
• Deployment platform: Red Hat OpenShift Container Platform or Linux

1. Install Red Hat OpenShift GitOps (Argo CD) on your Red Hat OpenShift cluster.

   For more information, see Installing OpenShift GitOps in the Red Hat OpenShift GitOps documentation.

2. Log in to Argo CD

   1. From your Red Hat OpenShift console, click the menu on the upper right, and select Cluster Argo CD.

      

   2. Log in with OpenShift.

      The Argo CD user interface (UI) is displayed. Click LOG IN VIA OPENSHIFT.

      

The service account that is used by the GitOps Application Controller requires elevated privileges to manage specific resources during the IBM Concert Operate installation.

The extent of these privilege escalations depends on whether the Red Hat OpenShift GitOps Argo CD instance is deployed in namespace-scoped or cluster-scoped mode. For more information, see Argo CD instance scopes in the Red Hat OpenShift GitOps documentation.

1. Create a role (gitops-cp4aiops-role).

   Run the command in one of the following options to grant the additional permissions that are required by the GitOps Application Controller. A cluster administrator must carefully review and validate these permissions before they are granted.

   Option 1: Deploy the GitOps Argo CD instance in namespace-scoped mode:

   cat <<EOF | oc apply -f -
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: gitops-cp4aiops-role
     namespace: ${PROJECT_CP4AIOPS}
   rules:
     - apiGroups: ["operators.coreos.com"]
       resources: ["operatorgroups", "catalogsources"]
       verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
     - apiGroups: ["orchestrator.aiops.ibm.com"]
       resources: ["installations"]
       verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
   EOF
   

   Option 2: Deploy the GitOps Argo CD instance in cluster-scoped mode:

   cat <<EOF | oc apply -f -
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: gitops-cp4aiops-role
     namespace: ${PROJECT_CP4AIOPS}
   rules:
     - apiGroups: ["orchestrator.aiops.ibm.com"]
       resources: ["installations"]
       verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
   EOF
   

2. Create a role binding (gitops-cp4aiops-rolebinding).

   Run the following command to create gitops-cp4aiops-rolebinding, and to link gitops-cp4aiops-role to the GitOps Application Controller service account.

   cat <<EOF | oc create -f -
   kind: RoleBinding
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     name: gitops-cp4aiops-rolebinding
     namespace: ${PROJECT_CP4AIOPS}
   subjects:
   - kind: ServiceAccount
     name: ${GITOPS_SERVICEACCOUNT}
     namespace: ${GITOPS_NAMESPACE}
   roleRef:
     kind: Role
     name: gitops-cp4aiops-role
   EOF
   

Run the following command to create an Argo CD application that has custom health check configurations for the IBM Concert Operate resources.

cat <<EOF | oc apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd
  namespace: ${GITOPS_NAMESPACE}
  labels:
    app.kubernetes.io/instance: argocd
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  destination:
    namespace: ${GITOPS_NAMESPACE}
    server: 'https://kubernetes.default.svc'
  source:
    repoURL: ${SOURCE_REPOSITORY}
    path: argocd
    targetRevision: ${TARGET_REVISION}
    helm:
      valuesObject:
        gitops:
          instance: ${GITOPS_INSTANCE}
          namespace: ${GITOPS_NAMESPACE}
  syncPolicy:
    retry:
      limit: 10
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 1m
  project: default
EOF

Skip this step if a certificate manager is already installed on the Red Hat OpenShift cluster that you are installing IBM Concert Operate on. If you do not have a certificate manager, then you must install one.

The IBM Cloud Pak foundational services Cert Manager is recommended. For more information about IBM Cloud Pak foundational services Cert Manager hardware requirements, see IBM Certificate Manager (cert-manager) hardware requirements in the IBM Cloud Pak foundational services documentation.

The Red Hat OpenShift Cert Manager is also supported. For more information, see cert-manager Operator for Red Hat OpenShiftin the Red Hat OpenShift documentation.

Run the following command to create an Argo CD application to deploy IBM Cloud Pak foundational services Cert Manager.

cat <<EOF | oc apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ibm-cert-manager
  namespace: ${GITOPS_NAMESPACE}
  labels:
    app.kubernetes.io/instance: ibm-cert-manager
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  destination:
    namespace: ibm-cert-manager
    server: 'https://kubernetes.default.svc'
  source:
    repoURL: ${SOURCE_REPOSITORY}
    path: cert-manager
    targetRevision: ${TARGET_REVISION}
  syncPolicy:
    retry:
      limit: 10
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 1m
  project: default
EOF

Run the following command to verify whether your environment is correctly set up for an IBM Concert Operate installation.

aiopsctl server precheck -n ${PROJECT_CP4AIOPS} 

• If you have a multi-zone cluster, then also specify the -m flag. This flag enables extra checks to help ensure that the cluster has sufficient resources to withstand a zone outage, and that the zones are well balanced for memory and CPU. For example, aiopsctl server precheck -n cp4aiops -m.
• If you are using hybrid storage, then also specify the --hybrid-storage flag. This flag enables extra checks to help ensure that sufficient local storage is configured. For example, aiopsctl server precheck -n cp4aiops --hybrid-storage

# aiopsctl server precheck -n concert-operate
o- [12 Jun 26 15:10 IST] Running precheck tool
  o- [12 Jun 26 15:10 IST] Checking hardware resources...
  Total Node Count (Available Schedulable / Required): 6/6
  Production (HA) Base CPU (vCPU): 93 / 146
  Production (HA) Base Memory (GB): 191 / 336

  Production (HA) Extended CPU (vCPU): 93 / 173
  Production (HA) Extended Memory (GB): 191 / 396

  Total Node Count (Available Schedulable / Required): 6/3
  Starter (Non-HA) Base CPU (vCPU): 93 / 47
  Starter (Non-HA) Base Memory (GB): 191 / 123

  Starter (Non-HA) Extended CPU (vCPU): 93 / 55
  Starter (Non-HA) Extended Memory (GB): 191 / 136

  You have enough resources for 1 instance(s) of small Base install
  You have enough resources for 1 instance(s) of small Extended install

  minimum hardware requirements met for starter size, but not production size

  o- [12 Jun 26 15:10 IST] Checking storage...
  Required StorageClasses found for provider Red Hat OpenShift Data Foundation: ocs-storagecluster-cephfs, ocs-storagecluster-ceph-rbd
  Checking if PVC can bind to supported storage class

  Verifying storage provider functionality: elapsed time [2s], estimated time [1m0s]
  Verifying storage provider functionality: elapsed time [5s], estimated time [1m0s]
  Storage check passed

  o- [12 Jun 26 15:10 IST] Checking OCP Version...
  Cluster meets OCP version requirements

  o- [12 Jun 26 15:10 IST] Checking if Cert Manager is present...
  Certificate CustomResourceDefinition Found

  o- [12 Jun 26 15:10 IST] Checking if certs will expire within 4 days...

  o- [12 Jun 26 15:10 IST] Precheck Summary Results
  Check                          Result
  Meets OCP Version Requirement  Passed
  Cert Mgr Operator Exists       Passed
  Certificates Valid             Passed
  Meets Hardware Requirements    Passed
  No Storage Issues              Passed

1. Select a values file from the Helm chart templates that will be supplied to your Installation custom resource in the last step of this section.

   Select one of the three values YAML files that best fits your deployment requirements:

   • values.yaml: Standard high availability (HA) installation.
   • values-multi-zone.yaml: Multi-Zone HA installation. For more information about multi-zone HA, see Installing IBM Concert Operate on multi-zone HA.
   • values-hybrid-storage.yaml: Standard HA deployment with a hybrid storage configuration. For more information, see Hybrid storage.

   The values files have default values for the storage classes and some other attributes that you can configure and override in the valuesObject section of the Installation custom resource in a later step.

2. Configure the gitops.namespaceScoped attribute.

   Skip this step if your Argo CD instance is deployed in cluster-scoped mode.

   The attribute gitops.namespaceScoped is defaulted to false in all of the values files. If you deployed your Argo CD instance in namespace-scoped mode, then you must edit this attribute in your values file to be true.

   When gitops.namespaceScoped is set to true, the namespace is excluded from GitOps management. This is necessary because a namespace-scoped Argo CD instance cannot manage cluster-scoped resources such as namespaces, custom resource definitions (CRDs), or ClusterRoles.

3. Export an environment variable for your chosen values file.

   export VALUES_FILE=<values_file>
   

   Where <values_file> is your chosen values file in the source repository. For example, export VALUES_FILE=values.yaml

4. Create an Argo CD Application to deploy IBM Concert Operate.

   Read the whole of this step and consider whether you want to make edits to override any attribute values in the valuesObject section in the following codeblock:

   cat <<EOF | oc apply -f -
   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: ibm-cp4aiops
     namespace: ${GITOPS_NAMESPACE}
     labels:
       app.kubernetes.io/instance: ibm-cp4aiops
     annotations:
       argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
   spec:
     destination:
       namespace: cp4aiops
       server: 'https://kubernetes.default.svc'
     source:
       repoURL: ${SOURCE_REPOSITORY}
       path: cp4aiops
       targetRevision: ${TARGET_REVISION}
       helm:
         valueFiles:
           - ${VALUES_FILE}
         valuesObject:
           installation:
             spec:
               license:
                 accept: ${ACCEPT_LICENSE}
               storageClass: ${STG_CLASS}
               storageClassLargeBlock: ${STG_CLASS_BLOCK}
               imagePullSecret: ibm-entitlement-key
     syncPolicy:
         retry:
           limit: 10
           backoff:
             duration: 5s
             factor: 2
             maxDuration: 1m
     project: default
   EOF
   

   The following attributes are already overridden in the valuesObject section with the values that you set in environment variables earlier: license.accept, storageClass, storageClassLargeBlock, and imagePullSecret. You can choose to make the following edits to override the default configuration:

   1. If you are deploying on a multi-zone architecture with a bespoke values file or with values-multi-zone.yaml, then you must add the following to the end of the valuesObject.installation.spec section:

      zones:
       - name: <zone_name1>
       - name: <zone_name2>
       - name: <zone_name3>
      

      Where <zone_name1>, <zone_name2>, and <zone_name3> exactly match the zone labels that you applied to each of your nodes in step 1a of Installing IBM Concert Operate on a multi-zone architecture (multi-zone HA).

   2. Optionally enable pakModules.

      • Enable connection if you want to use the secure tunnel feature. For more information, see Secure Tunnel.

      • Enable logAnomalyDetection for an extended deployment of IBM Concert Operate. For more information, see Incremental adoption.

      If you want to enable either or both of these features, add the following to the end of the valuesObject.installation.spec section:

      pakModules:
      - name: aiopsFoundation
        enabled: true
      - name: applicationManager
        enabled: true
      - name: aiManager
        enabled: true
      - name: connection
        enabled: <secure_tunnel>
      - name: logAnomalyDetection
        enabled: <extended_deployment>
      

      Where:

      • <secure_tunnel> is true if you want to enable the Secure Tunnel feature, otherwise false.
      • <extended_deployment> is true if you want an extended deployment of IBM Concert Operate, otherwise false.

      The values for pakModules aiopsFoundation, applicationManager, and aiManager must not be overridden, and must be set to true.

   Run the command in the codeblock when you have finished making any override edits to the valuesObject section that you require.

The three Argo CD applications that you created in the previous steps appear in the Argo CD Application UI.

Example Argo CD Application UI:

Initiate synchronization of the Argo CD applications in the following order:

1. Click SYNC on the argocd tile.
2. Click SYNC on the ibm-cert-manager tile.
3. Click SYNC on the ibm-cp4aiops tile.

The ibm-cp4aiops application enters a Progressing state, and remains in that state until the installation is fully completed on the cluster. When all components are successfully deployed, the final state for each application is Healthy and Synced.

(Optional) Automatic synchronization

You can optionally configure automatic synchronization after the initial synchronization is complete by updating the sync policy to Automated in the Argo CD App Details view for the IBM Concert Operate application. In this mode, changes to the installation manifests in the source Git repository are automatically applied to the cluster. Any values that are defined by the valuesObject section are preserved and continue to override the values file. You must disable automatic synchronization before uninstalling IBM Concert Operate. For more information, see Uninstalling.

Run the following command to check that the PHASE of your installation is Updating.

oc get installations.orchestrator.aiops.ibm.com -n ${PROJECT_CP4AIOPS}

NAME                  PHASE     LICENSE    STORAGECLASS   STORAGECLASSLARGEBLOCK   AGE
ibm-cp-aiops          Updating  Accepted   rook-cephfs    rook-ceph-block          3m

It takes around 60-90 minutes for the installation to complete (subject to the speed with which images can be pulled). When installation is complete and successful, the PHASE of your installation changes to Running. If your installation phase does not change to Running, then use the following command to find out which components are not ready:

oc get installation.orchestrator.aiops.ibm.com -o yaml -n ${PROJECT_CP4AIOPS} | grep 'Not Ready'

lifecycleservice: Not Ready
zenservice: Not Ready

To see details about why a component is Not Ready run the following command, where <component> is the component that is not ready, for example zenservice.

oc get <component> -o yaml -n ${PROJECT_CP4AIOPS}

(Optional) You can also download and run a status checker script to see information about the status of your deployment. For more information about how to download and run the script, see github.com/IBM .

If the installation fails, or is not complete and is not progressing, then see Troubleshooting installation and upgrade and Known Issues to help you identify any installation problems.

If your IBM Concert Operate deployment does not use hybrid storage then skip this step.

If your IBM Concert Operate deployment uses hybrid storage, verify that local storage has been correctly created for Postgres and Kafka.

1. Verify IR Core Postgres local storage.

   1. Run the following command to verify that IR Core Postgres pods are scheduled on the nodes that you configured for Postgres local storage.

      oc get pod -l "pg.ibm.com/cluster=aiops-ir-core-postgres" -o wide -n ${PROJECT_CP4AIOPS}
      

      Example output, where Postgres pods are scheduled on nodes named worker9, worker10 and worker11.

      NAME                       READY   STATUS    RESTARTS   AGE   IP             NODE                   NOMINATED NODE   READINESS GATES
      aiops-ir-core-postgres-1   1/1     Running   0          25m   10.254.68.38   worker11.example.com   <none>           <none>
      aiops-ir-core-postgres-2   1/1     Running   0          23m   10.254.60.30   worker9.example.com    <none>           <none>
      aiops-ir-core-postgres-3   1/1     Running   0          22m   10.254.36.87   worker10.example.com   <none>           <none>
      

      Keep a copy of this output that maps PVCs to nodes so that you have this information available if you need to restore your deployment. Add this information to the file that you created for step 2.4 of Hybrid storage.

   2. Run the following command to verify that logical volumes have been created for IR Core Postgres.

      oc get pvc -l "pg.ibm.com/cluster=aiops-ir-core-postgres" -o wide -n ${PROJECT_CP4AIOPS}
      

      Example output:

      NAME                          STATUS  VOLUME                                     CAPACITY  ACCESS MODES  STORAGECLASS                VOLUMEATTRIBUTESCLASS  AGE  VOLUMEMODE
      aiops-ir-core-postgres-1      Bound   pvc-d4e70a18-d791-4d6f-9c52-af8235417265   50Gi      RWO           lvms-vg-ir-core-postgres-1  <unset>                38m  Filesystem
      aiops-ir-core-postgres-1-wal  Bound   pvc-271f246d-8726-4823-9ab7-26fd0622f50b   10Gi      RWO           lvms-vg-ir-core-postgres-1  <unset>                38m  Filesystem
      aiops-ir-core-postgres-2      Bound   pvc-f3e0c1f1-7e18-4d60-895e-623b9e9b4cc1   50Gi      RWO           lvms-vg-ir-core-postgres-1  <unset>                39m  Filesystem
      aiops-ir-core-postgres-2-wal  Bound   pvc-362e47cd-4124-4f95-995f-55e216b804b4   10Gi      RWO           lvms-vg-ir-core-postgres-1  <unset>                39m  Filesystem
      aiops-ir-core-postgres-3      Bound   pvc-b08d6313-d262-4e9a-baf7-09b6fdbacf1c   50Gi      RWO           lvms-vg-ir-core-postgres-1  <unset>                39m  Filesystem
      aiops-ir-core-postgres-3-wal  Bound   pvc-6bbb154e-1845-44b5-a92c-00f22b41e1c5   10Gi      RWO           lvms-vg-ir-core-postgres-1  <unset>                39m  Filesystem
      

2. Verify Topology Postgres local storage.

   1. Run the following command to verify that the Topology Postgres pods are scheduled on the nodes that you configured for Postgres local storage.

      oc get pod -l "pg.ibm.com/cluster=aiops-topology-postgres" -o wide -n ${PROJECT_CP4AIOPS}
      

      Example output, where Postgres pods are scheduled on nodes named worker9, worker10 and worker11.

      NAME                        READY   STATUS    RESTARTS   AGE   IP             NODE                   NOMINATED NODE   READINESS GATES
      aiops-topology-postgres-1   1/1     Running   0          83m   10.254.52.70   worker12.example.com   <none>           <none>
      aiops-topology-postgres-2   1/1     Running   0          93m   10.254.56.20   worker13.example.com   <none>           <none>
      aiops-topology-postgres-3   1/1     Running   0          86m   10.254.12.52   worker14.example.com   <none>           <none>
      

      Keep a copy of this output that maps PVCs to nodes so that you have this information available if you need to restore your deployment. Add this information to the file that you created for step 2.4 of Hybrid storage.

   2. Run the following command to verify that logical volumes have been created for topology Postgres.

      oc get pvc -l "pg.ibm.com/cluster=aiops-topology-postgres" -o wide -n ${PROJECT_CP4AIOPS}
      

      Example output:

      NAME                           STATUS  VOLUME                                    CAPACITY  ACCESS MODES  STORAGECLASS                 VOLUMEATTRIBUTESCLASS  AGE  VOLUMEMODE
      aiops-topology-postgres-1      Bound   pvc-b2b29420-a0b8-469a-99e4-94b359487417  50Gi      RWO           lvms-vg-topology-postgres-1  <unset>                95m  Filesystem
      aiops-topology-postgres-1-wal  Bound   pvc-6bbb154e-1845-44b5-a92c-00f22b41e1c5  10Gi      RWO           lvms-vg-topology-postgres-1  <unset>                95m  Filesystem
      aiops-topology-postgres-2      Bound   pvc-3bb7fca3-f722-4bba-9ed9-ec2d46279223  50Gi      RWO           lvms-vg-topology-postgres-1  <unset>                95m  Filesystem
      aiops-topology-postgres-2-wal  Bound   pvc-5cbbe8e9-7472-4e70-826b-89e22bea4250  10Gi      RWO           lvms-vg-topology-postgres-1  <unset>                95m  Filesystem
      aiops-topology-postgres-3      Bound   pvc-d6de7642-dc24-42b9-bd87-11d7b882cfc6  50Gi      RWO           lvms-vg-topology-postgres-1  <unset>                95m  Filesystem
      aiops-topology-postgres-3-wal  Bound   pvc-5a76bd34-c570-4118-9316-33398cb83a16  10Gi      RWO           lvms-vg-topology-postgres-1  <unset>                95m  Filesystem
      

3. Verify Kafka local storage.

   1. Run the following command to verify that Kafka pods are scheduled on the nodes that you configured for Kafka local storage.

      oc get pod -l ibmevents.ibm.com/name=iaf-system-kafka -o wide -n ${PROJECT_CP4AIOPS}
      

      Example output, where Kafka pods are scheduled on nodes named worker6, worker7 and worker8.

      NAME                 READY   STATUS    RESTARTS   AGE    IP             NODE                  NOMINATED NODE   READINESS GATES
      iaf-system-kafka-0   1/1     Running   0          118m   10.123.24.14   worker6.example.com   <none>           <none>
      iaf-system-kafka-1   1/1     Running   0          118m   10.123.28.12   worker7.example.com   <none>           <none>
      iaf-system-kafka-2   1/1     Running   0          118m   10.123.56.27   worker8.example.com   <none>           <none>
      

      Keep a copy of this output that maps PVCs to nodes so that you have this information available if you need to restore your deployment. Add this information to the file that you created for step 2.4 of Hybrid storage.

   2. Run the following command to verify that logical volumes have been created for Kafka.

      oc get pvc -l ibmevents.ibm.com/name=iaf-system-kafka -n ${PROJECT_CP4AIOPS}
      

      Example output:

      NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
      data-iaf-system-kafka-0   Bound    pvc-fc36a8c8-7b73-473b-86a4-d4686228d521   100Gi      RWO            lvms-vg-kafka-1   124m
      data-iaf-system-kafka-1   Bound    pvc-2f653468-95a0-4d5b-b698-64023e25f643   100Gi      RWO            lvms-vg-kafka-1   124m
      data-iaf-system-kafka-2   Bound    pvc-c477e794-c825-4b58-af68-8a40b290126d   100Gi      RWO            lvms-vg-kafka-1   124m
      

There is no egress firewall policy defined when you install IBM Concert Operate, so outgoing traffic from workload pods to the internal and external network is unrestricted.

To create a more secure environment, use the following steps.

1. Create an EgressFirewall on your Red Hat OpenShift cluster to limit egress from the IBM Concert Operate project (namespace).

   For more information about creating an EgressFirewall, see Configuring an egress firewall for a project .

   You can have only one EgressFirewall per project/namespace.

2. Configure exceptions to the EgressFirewall.

   Edit your EgressFirewall to add exceptions for the following IBM Concert Operate components that have egress dependencies, otherwise these components fail when they attempt egress.

   1. Allow egress to any external services, such as the following integrations:
      • Kubernetes
      • GitHub
      • Microsoft® Teams
      • ServiceNow
      • Slack
      • VMware® vCenter

   2. Configure your EgressFirewall to allow traffic for your GitHub, Kubernetes, ServiceNow, and VMware vCenter integrations.

      Edit your EgressFirewall to allow or deny egress, as in the following example. Substitute values for dnsName and cidrSelector that are the DNS names and addresses of your GitHub, Kubernetes, ServiceNow, or VMware vCenter sources.

      kind: EgressFirewall
      metadata:
        name: default
      spec:
        egress:
        - type: Allow
          to:
            cidrSelector: <1.2.3.0/24>
        - type: Allow
          to:
            dnsName: <www.github.com>
        - type: Allow
          to:
            dnsName: <www.developer.kubernetes.com>
        - type: Allow
          to:
            dnsName: <www.developer.servicenow.com>
        - type: Allow
          to:
            dnsName: <www.developer.vcenter.com>
        - type: Deny
          to:
            cidrSelector: <0.0.0.0/0>
      

After you successfully install IBM Concert Operate, get the URL for accessing the Concert Operate console.

1. Use the following command to get the URL to access the Concert Operate console:

   oc get route -n ${PROJECT_CP4AIOPS} cpd -o jsonpath='{.spec.host}'
   

   The following output is a sample output:

   cpd-cp4aiops.apps.mycluster.mydomain
   

   Based on the sample output, your console URL would be https://cpd-cp4aiops.apps.mycluster.mydomain

2. Enter the URL in your browser to open the Concert Operate console. Log in with your username and password.

• Define integrations and applications with Defining.
• You can integrate with IBM Cognos Analytics. For more information, see Integrating IBM Cognos Analytics with IBM Concert Operate.
• If you have an existing on-premises IBM Tivoli Netcool/OMNIbus deployment, then you can connect it to IBM Concert Operate through an integration. For more information, see Creating IBM Tivoli Netcool/OMNIbus integrations.
• If you have an existing on-premises IBM Tivoli Netcool/Impact deployment, then you can connect it to IBM Concert Operate through an integration. For more information, see Creating IBM Tivoli Netcool/Impact integrations.
• Familiarize yourself with backup and restore procedures. It is recommended that you take regular backups of your IBM Concert Operate deployment. For more information, see Backup and restore.
• For more information about health checks and monitoring, see Health checks and monitoring. It is recommended that you implement self-monitoring checks and self-protection to improve the stability of your deployment. For more information, see Configuring and enabling OpenShift Container Platform monitoring.