Multi-region cold standby disaster recovery: Restoring IBM Cloud Pak for AIOps

Use IBM Fusion to restore your IBM Cloud Pak for AIOps deployment from backup.

Restore procedure

IBM Fusion recovers an IBM Cloud Pak for AIOps instance to a working state on a target cluster, along with its data. Your target cluster can be the hub cluster, or a different Red Hat OpenShift cluster (a spoke).

Ensure that you meet the prerequisites in Before you begin. Note that user interface customizations are not restored.

This procedure can still be used if there are multiple instances of IBM Cloud Pak for AIOps on a single cluster.

Follow the steps for Restoring to the hub cluster or Restoring to a spoke cluster.

Restoring to the hub cluster

If you are restoring an IBM Cloud Pak for AIOps instance to the same Red Hat OpenShift cluster that you created your backup on, then you must completely uninstall your IBM Cloud Pak for AIOps instance and remove the instance's namespace. The Red Hat OpenShift cluster that you restore on must not have any residual IBM Cloud Pak for AIOps artifacts in the namespace that you are restoring your instance into. For more information, see Uninstalling IBM Cloud Pak for AIOps.

  1. Get the entitlement key.

    Follow the instructions in Obtaining the entitlement key Opens in a new tab in the IBM Fusion documentation.

  2. Create a global pull secret.

    Important: Skip this step if you have already created the global pull secret for another instance of IBM Cloud Pak for AIOps on the cluster.

    Follow the instructions in Creating image pull secret Opens in a new tab in the IBM Fusion documentation.

  3. Install Cert Manager

    Skip this step if a certificate manager is already installed on the Red Hat OpenShift cluster that you are installing IBM Cloud Pak for AIOps on. If you do not have a certificate manager, then you must install one. The IBM Cloud Pak® foundational services Cert Manager is recommended, and can be installed with the following steps.

    For more information about IBM Cloud Pak® foundational services Cert Manager hardware requirements, see IBM Certificate Manager (cert-manager) hardware requirements Opens in a new tab in the IBM Cloud Pak foundational services documentation.

    1. Run the following command to create the catalog source for IBM Cloud Pak® foundational services Cert Manager.

      cat << EOF | oc apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-cert-manager-catalog
  namespace: openshift-marketplace
spec:
  displayName: ibm-cert-manager
  publisher: IBM
  sourceType: grpc
  image: icr.io/cpopen/ibm-cert-manager-operator-catalog
EOF

    2. Run the following command to create the resource definitions that you need:

      cat << EOF | oc apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: ibm-cert-manager
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: ibm-cert-manager-operator-group
  namespace: ibm-cert-manager
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: ibm-cert-manager-operator
  namespace: ibm-cert-manager
spec:
  channel: v4.2
  installPlanApproval: Automatic
  name: ibm-cert-manager-operator
  source: ibm-cert-manager-catalog
  sourceNamespace: openshift-marketplace
EOF

    3. Run the following command to ensure that the IBM Cloud Pak® foundational services Cert Manager pods have a STATUS of Running before proceeding to the next step.

      oc -n ibm-cert-manager get pods

      Example output for a successful IBM Cloud Pak® foundational services Cert Manager installation:

      NAME                                        READY   STATUS    RESTARTS   AGE
cert-manager-cainjector-674854c49d-vstq4    1/1     Running   0          8d
cert-manager-controller-646d4bd6fd-zwmqm    1/1     Running   0          8d
cert-manager-webhook-8598787c8-s4lkt        1/1     Running   0          8d
ibm-cert-manager-operator-c96957695-dkxnm   1/1     Running   0          8d

  4. Install the License Service

    Skip this step if the IBM Cloud Pak® foundational services License Service is already installed on the Red Hat OpenShift cluster that you are installing IBM Cloud Pak for AIOps on.

    IBM Cloud Pak for AIOps requires the installation of the IBM Cloud Pak foundational services License Service. You must install the IBM Cloud Pak foundational services License Service on the Red Hat OpenShift cluster that you are installing IBM Cloud Pak for AIOps on.

    1. Run the following command to create the catalog source for IBM Cloud Pak® foundational services License Service.

      cat << EOF | oc apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-licensing-catalog
  namespace: openshift-marketplace
spec:
  displayName: IBM License Service Catalog
  publisher: IBM
  sourceType: grpc
  image: icr.io/cpopen/ibm-licensing-catalog
EOF

    2. Run the following command to create the resource definitions that you need:

      cat << EOF | oc apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: ibm-licensing
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: ibm-licensing-operator-group
  namespace: ibm-licensing
spec:
  targetNamespaces:
  - ibm-licensing
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: ibm-licensing-operator-app
  namespace: ibm-licensing
spec:
  channel: v4.2
  installPlanApproval: Automatic
  name: ibm-licensing-operator-app
  source: ibm-licensing-catalog
  sourceNamespace: openshift-marketplace
EOF

    3. Run the following command to ensure that the IBM Cloud Pak® foundational services License Server pods have a STATUS of Running before proceeding to the next step.

      oc -n ibm-licensing get pods

      Example output for a successful IBM Cloud Pak® foundational services License Service installation:

      NAME                                              READY   STATUS    RESTARTS   AGE
ibm-licensing-operator-db4cd746c-xzmlf            1/1     Running   0          8d
ibm-licensing-service-instance-596b99588f-76cc5   1/1     Running   0          8d

    For more information about the IBM Cloud Pak® foundational services License Service, see License Service Opens in a new tab in the IBM Cloud Pak foundational services documentation.

  5. Update the IBM Fusion Backup & Restore transaction manager cluster roles.

    Run the following command:

    oc get clusterrole transaction-manager-ibm-backup-restore -o json | jq '.rules += [{"verbs":["get","list"],"apiGroups":["orchestrator.aiops.ibm.com"],"resources":["installations"]},{"verbs":["get","list"],"apiGroups":["ai-manager.watson-aiops.ibm.com"],"resources":["aimanagers"]},{"verbs":["get","list"],"apiGroups":["lifecycle.ir.aiops.ibm.com"],"resources":["lifecycleservices"]},{"verbs":["get","list"],"apiGroups":["operators.coreos.com"],"resources":["catalogsources"]}]' | oc apply -f -

  6. Follow the instructions in Restoring the application Opens in a new tab in the IBM Fusion documentation.

    Select Same cluster, and then for Project destination select Use the same project the application is already using. Accept the default values for the other fields.

    Note: You might notice that the aiops-configmaps resource group shows as failed under the Restore sequence section of the IBM Cloud Pak for AIOps restore recipe but the overall restore is successful and works as intended.

  7. Delete any duplicate observer jobs on the restored cluster.

    1. Log in to IBM Cloud Pak for AIOps console.

    2. Expand the navigation menu (four horizontal bars), then click Define > Integrations.

    3. For any duplicate jobs, click the options menu (three vertical dots) and click Delete.

  8. From the IBM Cloud Pak for AIOps console, select and edit each incident policy that utilises a ticket integration, and manually reassign the relevant ticketing system (Github/ServiceNow).

Restoring to a spoke cluster

If you are restoring an IBM Cloud Pak for AIOps instance to a different Red Hat OpenShift cluster, ensure that your target cluster meets the prerequisites for an IBM Cloud Pak for AIOps deployment, and then install IBM Fusion and the IBM Fusion backup agent on it.

  1. Spoke cluster prerequisites
  • You must have the same storage provider and storage classes available for your target spoke cluster as you did for your IBM Cloud Pak for AIOps deployment on the hub cluster.
  • The target spoke cluster must have the same deployment platform and Red Hat OpenShift version as the source hub cluster. Review the Planning section and ensure that the environment that you are restoring to meets the system requirements.
  • The Red Hat OpenShift cluster that you restore on must not have any residual IBM Cloud Pak for AIOps artifacts in the namespace that you are restoring your instance into. If you need to uninstall IBM Cloud Pak for AIOps, see the instructions in Uninstalling IBM Cloud Pak for AIOps.

  1. Install IBM Fusion on the target spoke cluster

    1. Get the entitlement key.

      Follow the instructions in Obtaining the entitlement key Opens in a new tab in the IBM Fusion documentation.

    2. Create a global pull secret.

      Important: Skip this step if you have already created the global pull secret for another instance of IBM Cloud Pak for AIOps on the cluster.

      Follow the instructions in Creating image pull secret Opens in a new tab in the IBM Fusion documentation.

    3. Install IBM Fusion.

      You must install IBM Fusion v2.8.x or v2.9.0.

      Follow the Installing instructions for your Red Hat OpenShift deployment type in Deploying IBM Fusion Opens in a new tab in the IBM Fusion documentation. Accept the default values.

  2. Install Cert Manager

    Skip this step if you already have a certificate manager installed on the Red Hat OpenShift cluster that you are installing IBM Cloud Pak for AIOps on. If you do not have a certificate manager then you must install one. The IBM Cloud Pak® foundational services Cert Manager is recommended, and can be installed with the following steps.

    For more information about IBM Cloud Pak® foundational services Cert Manager, see Certificate Management Opens in a new tab in the IBM Cloud Pak foundational services documentation.

    1. Run the following command to create the catalog source for IBM Cloud Pak® foundational services Cert Manager.

      cat << EOF | oc apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-cert-manager-catalog
  namespace: openshift-marketplace
spec:
  displayName: ibm-cert-manager
  publisher: IBM
  sourceType: grpc
  image: icr.io/cpopen/ibm-cert-manager-operator-catalog
EOF

    2. Run the following command to create the resource definitions that you need:

      cat << EOF | oc apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: ibm-cert-manager
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: ibm-cert-manager-operator-group
  namespace: ibm-cert-manager
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: ibm-cert-manager-operator
  namespace: ibm-cert-manager
spec:
  channel: v4.2
  installPlanApproval: Automatic
  name: ibm-cert-manager-operator
  source: ibm-cert-manager-catalog
  sourceNamespace: openshift-marketplace
EOF

    3. Run the following command to ensure that the IBM Cloud Pak® foundational services Cert Manager pods have a STATUS of Running before proceeding to the next step.

      oc -n ibm-cert-manager get pods

      Example output for a successful IBM Cloud Pak® foundational services Cert Manager installation:

      NAME                                        READY   STATUS    RESTARTS   AGE
cert-manager-cainjector-674854c49d-vstq4    1/1     Running   0          8d
cert-manager-controller-646d4bd6fd-zwmqm    1/1     Running   0          8d
cert-manager-webhook-8598787c8-s4lkt        1/1     Running   0          8d
ibm-cert-manager-operator-c96957695-dkxnm   1/1     Running   0          8d

  3. Install the License Service

    Skip this step if the IBM Cloud Pak® foundational services License Service is already installed on the Red Hat OpenShift cluster that you are installing IBM Cloud Pak for AIOps on.

    IBM Cloud Pak for AIOps requires the installation of the IBM Cloud Pak foundational services License Service. You must install the IBM Cloud Pak foundational services License Service on the Red Hat OpenShift cluster that you are installing IBM Cloud Pak for AIOps on.

    1. Run the following command to create the catalog source for IBM Cloud Pak® foundational services License Service.

      cat << EOF | oc apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-licensing-catalog
  namespace: openshift-marketplace
spec:
  displayName: IBM License Service Catalog
  publisher: IBM
  sourceType: grpc
  image: icr.io/cpopen/ibm-licensing-catalog
EOF

    2. Run the following command to create the resource definitions that you need:

      cat << EOF | oc apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: ibm-licensing
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: ibm-licensing-operator-group
  namespace: ibm-licensing
spec:
  targetNamespaces:
  - ibm-licensing
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: ibm-licensing-operator-app
  namespace: ibm-licensing
spec:
  channel: v4.2
  installPlanApproval: Automatic
  name: ibm-licensing-operator-app
  source: ibm-licensing-catalog
  sourceNamespace: openshift-marketplace
EOF

    3. Run the following command to ensure that the IBM Cloud Pak® foundational services License Server pods have a STATUS of Running before proceeding to the next step.

      oc -n ibm-licensing get pods

      Example output for a successful IBM Cloud Pak® foundational services License Service installation:

      NAME                                              READY   STATUS    RESTARTS   AGE
ibm-licensing-operator-db4cd746c-xzmlf            1/1     Running   0          8d
ibm-licensing-service-instance-596b99588f-76cc5   1/1     Running   0          8d

    For more information about the IBM Cloud Pak® foundational services License Service, see License Service Opens in a new tab in the IBM Cloud Pak foundational services documentation.

  4. Get connection YAML from the hub cluster.

    On the hub cluster, follow the Use the Fusion UI instructions in Connect clusters Opens in a new tab in the IBM Fusion documentation.

    When you have copied the connection snippet, return to these steps. The connection snippet is used in the next step to connect the hub and spoke clusters.

  5. Install the Backup & Restore Agent on the spoke cluster, and connect the spoke cluster to the hub cluster.

    On the spoke cluster, follow the procedure in Backup & Restore spoke Opens in a new tab in the IBM Fusion documentation.

    For the Hub Connection Snippet field, use the connection snippet that you copied in the previous step. The storage classes must be the same storage classes that the IBM Cloud Pak for AIOps deployment on the hub cluster used.

  6. On the spoke cluster, update the IBM Fusion Backup & Restore transaction manager cluster roles.

    Run the following command:

    oc get clusterrole transaction-manager-ibm-backup-restore -o json | jq '.rules += [{"verbs":["get","list"],"apiGroups":["orchestrator.aiops.ibm.com"],"resources":["installations"]},{"verbs":["get","list"],"apiGroups":["ai-manager.watson-aiops.ibm.com"],"resources":["aimanagers"]},{"verbs":["get","list"],"apiGroups":["lifecycle.ir.aiops.ibm.com"],"resources":["lifecycleservices"]},{"verbs":["get","list"],"apiGroups":["operators.coreos.com"],"resources":["catalogsources"]}]' | oc apply -f -

  7. Follow the instructions in Restoring the application Opens in a new tab in the IBM Fusion documentation, and wait for the restore to complete.

    Select New cluster, and then accept the default values for the other fields.

    Note: You might notice that the aiops-configmaps resource group shows as failed under the Restore sequence section of the IBM Cloud Pak for AIOps restore recipe but the overall restore is successful and works as intended.

  8. Redeploy remote integrations.

    Re-run the bootstrap procedure for any remote connections that are configured.

    1. Log in to IBM Cloud Pak for AIOps console.

    2. Expand the navigation menu (four horizontal bars), then click Define > Integrations.

    3. For each remote integration, click the integration type on the Manage integrations tab of the Integrations page, then:

      1. Click the Download link in the Remote deployment script column for the integration.

      2. Click Download as a sh-file to download the deployment script as a bootstrap.sh file.

      3. From the Red Hat OpenShift CLI, switch to the target project (namespace).

        oc project <namespace>

        Where <namespace> is the namespace (project) that your IBM Cloud Pak for AIOps installation is deployed in, for example cp4aiops.

      4. Run the bootstrap.sh deployment script.

  9. If you have secure tunnel connections, then re-create them. For more information, see Create the secure tunnel connection on your restored cluster.

  10. Delete any duplicate observer jobs on the restored cluster.

    1. Log in to IBM Cloud Pak for AIOps console.

    2. Expand the navigation menu (four horizontal bars), then click Define > Integrations.

    3. For any duplicate jobs, click the options menu (three vertical dots) and click Delete.

  11. From the IBM Cloud Pak for AIOps console, select and edit each incident policy that utilises a ticket integration, and manually reassign the relevant ticketing system (Github/ServiceNow).

What to do next

The restore can take up to a few hours to run. You are notified on the IBM Fusion user interface (UI) when the restore is complete. You can also monitor the progress of the restore from Backup & restore > Jobs in the IBM Fusion UI.

Important: If you restored to a spoke cluster, the URL and password for accessing the IBM Cloud Pak for AIOps console are different from the URL and password that you used on the hub cluster. Follow the steps in Access the Cloud Pak for AIOps console to find the new URL and password.

(Optional) After the restore is complete, you can verify the status of your deployment by using the steps in Verify your installation (CLI) or Verify your installation (console).

To view the IBM Fusion logs, or to see the resources that were backed up or restored, see Troubleshooting multi-region cold standby disaster recovery for IBM Cloud Pak for AIOps.