Multi-region cold standby disaster recovery: Backing up IBM Cloud Pak for AIOps

Backup procedure

Ensure that you meet the prerequisites in Before you begin. Cluster administrator permissions are needed to install IBM Fusion, and to manage the IBM Fusion Backup and Restore service.

This procedure can still be used if there are multiple instances of IBM Cloud Pak for AIOps on a single cluster.

Use the following instructions to install IBM Fusion on the same Red Hat OpenShift cluster as IBM Cloud Pak for AIOps, and then create a backup of IBM Cloud Pak for AIOps.

Ensure that you have a VolumeSnapshotClass
Install IBM Fusion
Deploy the IBM Fusion Backup & Restore service
Configure the IBM Fusion Backup & Restore service for IBM Cloud Pak for AIOps

1. Ensure that you have a VolumeSnapshotClass

To enable IBM Fusion backups, you must have a VolumeSnapshotClass defined for one of the storage classes that host IBM Cloud Pak for AIOps. For more information, see Prerequisites Opens in a new tab in the IBM Fusion documentation.

Run the following command to check whether a VolumeSnapshotClass is defined already:

oc get volumesnapshotclass

If the preceding command does not return anything, then you do not have a VolumeSnapshotClass defined. Refer to your storage provider's documentation to create one.

2. Install IBM Fusion

You must install IBM Fusion v2.8.x, v2.9.x, 2.10.x, 2.11.0, or 2.12.x. If your IBM Cloud Pak for AIOps deployment uses hybrid storage, then IBM Fusion must be v2.9.x or higher, or backup and restore will fail.

Follow the Installing instructions for your Red Hat OpenShift deployment type in Deploying IBM Fusion Opens in a new tab in the IBM Fusion documentation. Accept the default values.

As you are installing IBM Fusion on the same Red Hat OpenShift cluster as IBM Cloud Pak for AIOps, you already have an entitlement pull secret configured.

3. Deploy the IBM Fusion Backup & Restore service

Install the IBM Fusion Backup & Restore service.

Follow the instructions in Backup & Restore hub in the IBM Fusion documentation.

The ibm-backup-restore project (namespace) is created, and the backup and restore service is installed in that project.
Configure the backup storage location for IBM Fusion to store the IBM Cloud Pak for AIOps backups in.

The backup storage location must be external object storage, and can be any S3 compatible storage or Microsoft® Azure Blob Storage.

Follow the instructions in Adding a backup storage location in the IBM Fusion documentation.
Create a backup policy.

The backup policy specifies the backup location, frequency, and retention period of the backup. Specify the backup storage location that you created in the previous step as the backup location.

Follow the instructions in Creating backup policy in the IBM Fusion documentation.
Select the IBM Cloud Pak for AIOps application that you want to backup, and assign the backup policy to it.

Follow the instructions in Set up policies from Backed up applications page in the IBM Fusion documentation.

Before you select Assign in the user interface, you must deselect Run backup now to prevent the backup from running immediately.
Find the name of your policy assignment.

Your policy assignment name is the IBM Cloud Pak for AIOps installation namespace concatenated with the backup policy name. Run the following command to find it, and then make a note of it:
```
oc get fpa -n ibm-spectrum-fusion-ns
```

4. Configure the IBM Fusion Backup & Restore service for IBM Cloud Pak for AIOps

Run the following steps on the cluster where IBM Cloud Pak for AIOps is running.

Update the IBM Fusion Backup & Restore transaction manager cluster roles.

Run the following command:

oc get clusterrole transaction-manager-ibm-backup-restore -o json | jq '.rules += [{"verbs":["get","list"],"apiGroups":["orchestrator.aiops.ibm.com"],"resources":["installations"]},{"verbs":["get","list"],"apiGroups":["ai-manager.watson-aiops.ibm.com"],"resources":["aimanagers"]},{"verbs":["get","list"],"apiGroups":["lifecycle.ir.aiops.ibm.com"],"resources":["lifecycleservices"]},{"verbs":["get","list"],"apiGroups":["operators.coreos.com"],"resources":["catalogsources"]}]' | oc apply -f -

Run the following commands to set the environment variables that you need.
```
export PROJECT_CP4AIOPS=<namespace>
export CUSTOM_PROFILE=<custom_profile>
export POLICY_ASSIGNMENT_NAME=<policy_assignment_name>
```
Where:
- <namespace> is the namespace (project) that your IBM Cloud Pak for AIOps installation is deployed in, for example cp4aiops.
- <custom_profile> is blank unless IBM Sales representatives and Business Partners used the custom sizing tool to supply you with a custom profile ConfigMap to customize your deployment. If you were supplied with a custom profile ConfigMap, then this is the name of your custom profile ConfigMap, usually aiops-custom-size-profile.
- <policy_assignment_name> is the policy assignment name that you found in the previous step.
Find your installation name and set an environment variable for it.
1. Find the name of your IBM Cloud Pak for AIOps installation.
```
oc get installation.orchestrator.aiops.ibm.com -n ${PROJECT_CP4AIOPS}
```
  Example output, where ibm-cp-aiops is the name of the IBM Cloud Pak for AIOps installation:
```
NAME          PHASE    LICENSE   STORAGECLASS STORAGECLASSLARGEBLOCK AGE
ibm-cp-aiops  Running  Accepted  rook-cephfs  rook-ceph-rbd          34h
```
2. Export an environment variable with the name of your installation.
```
export INSTALLATION_NAME=<name>
```
  Where <name> is the name returned by the previous command, for example ibm-cp-aiops.

Run the following commands to add labels to IBM Cloud Pak for AIOps resources so that they are correctly identified.

oc -n $PROJECT_CP4AIOPS label secret aiops-ir-core-postgres-secret manual=aiops-ir-core-postgres-secret
oc -n $PROJECT_CP4AIOPS label secret aiops-ir-analytics-postgres-secret manual=aiops-ir-analytics-postgres-secret
oc -n $PROJECT_CP4AIOPS label secret aiops-topology-postgres-secret manual=aiops-topology-postgres-secret
oc -n $PROJECT_CP4AIOPS label secret aiops-orchestrator-postgres-secret manual=aiops-orchestrator-postgres-secret
oc -n $PROJECT_CP4AIOPS label secret ${INSTALLATION_NAME}-redis-secret manual=${INSTALLATION_NAME}-redis-secret
oc -n $PROJECT_CP4AIOPS label secret ibm-zen-secret-jwks manual=ibm-zen-secret-jwks
oc -n $PROJECT_CP4AIOPS label secret ibm-zen-secret-jwt manual=ibm-zen-secret-jwt
oc -n $PROJECT_CP4AIOPS label secret ibm-zen-secret-jwt-private manual=ibm-zen-secret-jwt-private
oc -n $PROJECT_CP4AIOPS label secret aiopsedge-config-encryption-history manual=aiopsedge-config-encryption-history
oc -n $PROJECT_CP4AIOPS label secret platform-auth-ldaps-ca-cert manual=platform-auth-ldaps-ca-cert
oc -n $PROJECT_CP4AIOPS label secret ibm-entitlement-key manual=ibm-entitlement-key-secret
oc -n $PROJECT_CP4AIOPS label subscriptions.operators.coreos.com ibm-aiops-orchestrator manual=ibm-aiops-orchestrator

Download the IBM Cloud Pak for AIOps backup and restore IBM Fusion recipe.

The IBM Fusion recipe defines the sequence of actions and checks that IBM Fusion must use for the backup and restore of IBM Cloud Pak for AIOps resources and PVCs.

Download the recipe aiops-backup-restore.yaml, from github.com/IBM .
Adjust the recipe if you have a non-default installation name.

If your installation name is not ibm-cp-aiops, then edit the recipe that you downloaded, and alter all occurences of ibm-cp-aiops to be your installation name.
Adjust the recipe for custom sizing.

IBM Sales representatives and Business Partners may have used a custom sizing tool to supply you with a custom profile to customize your IBM Cloud Pak for AIOps deployment at installation time.

If IBM Sales representatives and Business Partners did not supply you with a custom profile, then skip this step.

If IBM Sales representatives and Business Partners supplied you with a custom profile, then add a label to the custom profile ConfigMap in your IBM Cloud Pak for AIOps deployment. Run the following command:
```
oc label cm ${CUSTOM_PROFILE} cm=aiops-custom-size-profile
```
Create the IBM Cloud Pak for AIOps recipe custom resource.

Run the following command to create the recipe custom resource.
```
oc apply -f aiops-backup-restore.yaml
```

Configure your policy assignment to use the IBM Cloud Pak for AIOps back and restore recipe.

Patch the policy assignment with the details of the IBM Cloud Pak for AIOps back and restore recipe.

oc -n ibm-spectrum-fusion-ns patch policyassignment ${POLICY_ASSIGNMENT_NAME} --type merge  -p '{"spec":{"recipe":{"name":"aiops-backup-restore-recipe", "namespace":"ibm-spectrum-fusion-ns"}}}'

If you are using IBM Fusion v2.9.1, restart transaction manager.

If your backups remain in a Pending state and then timeout, run the following command to restart the transaction-manager pod in the ibm-backup-restore namespace:
```
oc delete pod -l app.kubernetes.io/component=transaction-manager -n ibm-backup-restore
```
Set storage PVCs for the backup.

This step must be run if you are using IBM Fusion 2.10, or restore from backup will fail. This step is not needed if you are using IBM Fusion 2.8.x, v2.9.x, or v2.10.1 or later.

If you are using IBM Fusion 2.10, run the following command to set the storage PVCs for the backup.
```
oc edit cm guardian-configmap -n ibm-backup-restore 
```
Add the following line in the data section and save your changes:
```
cephRbdStorageProvisioner: random.csi.ceph.com
```
For example:
```
data:
  logLevel:INFO
  cephRbdStorageProvisioner: random.csi.ceph.com
  app: guardian
```

What to do next

Backups occur according to the schedule that you set in your policy. If you want to run a backup immediately, follow the instructions in Running on demand backups Opens in a new tab .

You can monitor the progress of the backup from Backup & restore > Jobs in the IBM Fusion user interface.

To view the IBM Fusion logs, or to see the resources that were backed up, see Troubleshooting multi-region cold standby disaster recovery for IBM Cloud Pak for AIOps.