IBM Software Hub online backup and restore to the same cluster with the OADP utility

1.1 Creating environment variables

Create the following environment variables so that you can copy commands from the documentation and run them without making any changes.

1.2 Checking the version of OADP utility components

1.3 Optional: Estimating how much storage to allocate for backups

You can estimate the amount of storage that you need to allocate for backups.

To use this feature, you must install the cpdbr-agent in the Red Hat OpenShift cluster. The cpdbr-agent deploys the node agents to the cluster. The node agents must be run in privileged mode.

1. Log in to Red Hat OpenShift Container Platform as a cluster administrator:

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. Install the cpdbr-agent by running the following command:

   cpd-cli oadp install --component=cpdbr-agent --namespace=${OADP_PROJECT} --cpd-namespace=${PROJECT_CPD_INST_OPERANDS}

3. Export the following environment variable:

   export CPDBR_ENABLE_FEATURES=volume-util

4. Estimate how much storage you need to allocate to a backup by running the following command:

   cpd-cli oadp du-pv

1.4 Removing MongoDB-related ConfigMaps

oc delete cm zen-cs-aux-br-cm

oc delete cm zen-cs-aux-ckpt-cm

oc delete cm zen-cs-aux-qu-cm

oc delete cm zen-cs2-aux-ckpt-cm

1.5 Checking the primary instance of every PostgreSQL cluster is in sync with its replicas

The replicas for Cloud Native PostgreSQL and EDB Postgres clusters occasionally get out of sync with the primary node. To check whether this problem exists and to fix the problem, see the troubleshooting topic PostgreSQL cluster replicas get out of sync.

1.6 Updating the Common core services ConfigMap

5.1.0 You might need to update the cpd-ccs-maint-aux-ckpt-cm ConfigMap before you create a backup. Do the following steps:

1. Check if any common core services download images pod is in a Running state:

   oc get po -l icpdsupport/addOnId=ccs,icpdsupport/module=ccs-common,app=download-images -n ${PROJECT_CPD_INST_OPERANDS}

2. If the output of the command shows one or more pods in a Running state, edit the managed-resources section in the cpd-ccs-maint-aux-ckpt-cm ConfigMap to ignore the pod:

     aux-meta:
       managed-resources:
         - resource-kind: pod
           labels: icpdsupport/addOnId=ccs,icpdsupport/module=ccs-common,app=download-images

1.7 Preparing IBM Knowledge Catalog

If large metadata enrichment jobs are running while an online backup operation is triggered, the Db2 pre-backup hooks might fail because the database cannot be put into a write-suspended state. It is recommended to have minimal enrichment workload while the online backup is scheduled.

1.8 Labeling the IBM Match 360 ConfigMap

1.9 Checking the status of installed services

2.1 Setting the mode in which to create backups

You can run the IBM Software Hub OADP backup and restore utility in Kubernetes mode or in REST mode.

By default, the IBM Software Hub OADP backup and restore utility runs in Kubernetes mode. In this mode, you must log in to your Red Hat OpenShift cluster and you must have Kubernetes cluster administrator privileges to use the utility.

If you installed the IBM Software Hub OADP backup REST service, you can run the utility in REST mode to create backups. In REST mode, the utility runs as a REST client that communicates to a REST server. The REST service is configured to work with a specific IBM Software Hub instance. You do not have to log in to the cluster, and IBM Software Hub users with the Administrator role can run backup and checkpoint commands on their own IBM Software Hub instances, based on the specified control plane and any tethered projects.

Running the utility in REST mode is useful when you are generally creating backups only, or when backups take a long time to complete. For backups that take a long time to complete, running the utility in REST mode avoids the problem of the Red Hat OpenShift user session token expiring before the backup process completes. If the session token expires, you must log back in to the cluster and reset the utility.

1. Log in to Red Hat OpenShift Container Platform as a cluster administrator:

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. To create backups in REST mode, run the following command:

   cpd-cli oadp client config set runtime-mode=rest-client

3. To change the IBM Software Hub OADP backup and restore utility back to the Kubernetes mode, run the following command:

   cpd-cli oadp client config set runtime-mode=

Related topic: Unable to run an online backup or restore operation

2.2 Backing up the scheduling service

If the IBM Software Hub scheduling service is installed, create a backup of the service.

Backups that are created in IBM Cloud Pak for Data 5.0 cannot be restored in IBM Software Hub 5.1.0. You must take new backups in 5.1.0.

Check the Known issues and limitations for IBM Software Hub page for any workarounds that you might need to do before you create a backup.

1. If you are running the backup and restore utility in Kubernetes mode, log in to Red Hat OpenShift Container Platform as a cluster administrator:

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. Configure the OADP client to set the IBM Software Hub project to the scheduling service project:

   cpd-cli oadp client config set cpd-namespace=${PROJECT_SCHEDULING_SERVICE}

3. Configure the OADP client to set the OADP project to the project where the OADP operator is installed:

   cpd-cli oadp client config set namespace=${OADP_PROJECT}

4. Run service backup prechecks:

   IBM Software Hub 5.1.0

   cpd-cli oadp backup precheck \
   --include-namespaces=${PROJECT_SCHEDULING_SERVICE} \
   --exclude-checks ValidVolumeSnapshotClass \
   --hook-kind=checkpoint \
   --log-level=debug \
   --verbose

   IBM Software Hub 5.1.1 and later

   cpd-cli oadp backup precheck \
   --backup-type singleton \
   --include-namespaces=${PROJECT_SCHEDULING_SERVICE} \
   --exclude-checks ValidVolumeSnapshotClass \
   --hook-kind=checkpoint \
   --log-level=debug \
   --verbose

5. Back up the IBM Software Hub scheduling service:

   IBM Software Hub 5.1.0

   cpd-cli oadp backup create ${PROJECT_SCHEDULING_SERVICE}-online \
   --include-namespaces=${PROJECT_SCHEDULING_SERVICE} \
   --include-resources='operatorgroups,configmaps,catalogsources.operators.coreos.com,subscriptions.operators.coreos.com,customresourcedefinitions.apiextensions.k8s.io,scheduling.scheduler.spectrumcomputing.ibm.com' \
   --prehooks=false \
   --posthooks=false \
   --with-checkpoint \
   --log-level=debug \
   --verbose \
   --hook-kind=checkpoint \
   --selector 'icpdsupport/ignore-on-nd-backup notin (true)'

   IBM Software Hub 5.1.1 and later

   cpd-cli oadp backup create ${PROJECT_SCHEDULING_SERVICE}-online \
   --backup-type singleton \
   --include-namespaces=${PROJECT_SCHEDULING_SERVICE} \
   --include-resources='operatorgroups,configmaps,catalogsources.operators.coreos.com,subscriptions.operators.coreos.com,customresourcedefinitions.apiextensions.k8s.io,scheduling.scheduler.spectrumcomputing.ibm.com' \
   --prehooks=false \
   --posthooks=false \
   --with-checkpoint \
   --log-level=debug \
   --verbose \
   --hook-kind=checkpoint \
   --selector 'icpdsupport/ignore-on-nd-backup notin (true)'

6. Validate the backup:

   IBM Software Hub 5.1.0

   cpd-cli oadp backup validate \
   --include-namespaces=${PROJECT_SCHEDULING_SERVICE} \
   --backup-names ${PROJECT_SCHEDULING_SERVICE}-online \
   --log-level trace \
   --verbose \
   --hook-kind checkpoint

   IBM Software Hub 5.1.1 and later

   cpd-cli oadp backup validate \
   --backup-type singleton \
   --include-namespaces=${PROJECT_SCHEDULING_SERVICE} \
   --backup-names ${PROJECT_SCHEDULING_SERVICE}-online \
   --log-level trace \
   --verbose \
   --hook-kind checkpoint

2.3 Backing up an IBM Software Hub instance

Create an online backup of each IBM Software Hub instance, or tenant, in your environment by doing the following steps.

Check the Known issues and limitations for IBM Software Hub page for any workarounds that you might need to do before you create a backup.

1. If you are running the backup and restore utility in Kubernetes mode, log in to Red Hat OpenShift Container Platform as a cluster administrator:

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. 5.1.0 Ensure that the expected EDB Postgres replica PVCs are included in the backup:

   oc label pvc,pods -l k8s.enterprisedb.io/cluster,velero.io/exclude-from-backup=true velero.io/exclude-from-backup- -n ${PROJECT_CPD_INST_OPERANDS}

3. Create a backup:

   cpd-cli oadp tenant-backup create ${TENANT_BACKUP_NAME} \
   --tenant-operator-namespace ${PROJECT_CPD_INST_OPERATORS} \
   --log-level=debug \
   --verbose

4. 5.1.0-5.1.2If watsonx Orchestrate is installed, back up App Connect resources.
   1. Download the backup-script.zip file that is needed to back up App Connect.

      For details on downloading and extracting the contents of this file, see Backing up and restoring your IBM App Connect resources and persistent volumes on Red Hat OpenShift.

   2. Open the backup-script.sh file in an editor.
   3. Change spec.storageLocation to the object storage where you are storing your IBM Software Hub backups.
   4. If you logged out of the Red Hat OpenShift Container Platform, log back in as a cluster administrator.
   5. Run the following command, specifying a name for the backup:

      ./backup-script.sh ${PROJECT_IBM_APP_CONNECT} ${APP_CONNECT_BACKUP_NAME}

5. Confirm that the tenant backup was created and has a Completed status:

   cpd-cli oadp tenant-backup list

6. To view the detailed status of the backup, run the following command:

   cpd-cli oadp tenant-backup status ${TENANT_BACKUP_NAME} \
   --details

   The command shows the following sub-backups:

   Backup	Description
   cpd-tenant-xxx	Backup that contains Kubernetes resources.
   cpd-tenant-vol-yyy	Backup that contains volume data.

   Tip: If you need more information, listed in the status details are sub-backups (of type group). You can view more information about these sub-backups by running the following command:

   cpd-cli oadp backup status <SUB_BACKUP_NAME> \
   --details

7. To view logs of the tenant backup and all sub-backups, run the following command:

   cpd-cli oadp tenant-backup log ${TENANT_BACKUP_NAME}

4.1 Optional: Restoring the scheduling service

If the IBM Software Hub scheduling service is installed on the cluster, you can restore it if you are experiencing problems with the service.

Check the Known issues and limitations for IBM Software Hub page for any workarounds that you might need to do before you restore a backup.

1. Log in to Red Hat OpenShift Container Platform as a cluster administrator:

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. Configure the OADP client to set the IBM Software Hub project to the scheduling service project:

   cpd-cli oadp client config set cpd-namespace=${PROJECT_SCHEDULING_SERVICE}

3. Restore an online backup:

   cpd-cli oadp restore create ${PROJECT_SCHEDULING_SERVICE}-{restore} \
   --from-backup=${PROJECT_SCHEDULING_SERVICE}-online \
   --include-resources='operatorgroups,configmaps,catalogsources.operators.coreos.com,subscriptions.operators.coreos.com,customresourcedefinitions.apiextensions.k8s.io,scheduling.scheduler.spectrumcomputing.ibm.com' \
   --include-cluster-resources=true \
   --skip-hooks \
   --log-level=debug \
   --verbose

4.2 Restoring an IBM Software Hub instance

Restore an IBM Software Hub instance by doing the following steps.

1. Log in to Red Hat OpenShift Container Platform as a cluster administrator:

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. 5.1.0-5.1.2If watsonx Orchestrate is installed, restore App Connect resources.
   1. Log in to Red Hat OpenShift Container Platform as an instance administrator.

      ${OC_LOGIN}

      Remember: OC_LOGIN is an alias for the oc login command.
   2. Download the restore-script.zip file that is needed to restore App Connect.

      For details on downloading and extracting the contents of this file, see Backing up and restoring your IBM App Connect resources and persistent volumes on Red Hat OpenShift.

   3. Extract the restore-script.sh file.
   4. Restore App Connect resources:

      ./restore-script.sh <appconnect_backup_name>

3. Restore IBM Software Hub:

   cpd-cli oadp tenant-restore create ${TENANT_BACKUP_NAME}-restore \
   --from-tenant-backup ${TENANT_BACKUP_NAME} \
   --verbose \
   --log-level=debug &> ${TENANT_BACKUP_NAME}-restore.log&

4. Get the status of the installed components:
   1. Log the cpd-cli in to the Red Hat OpenShift Container Platform cluster:

      ${CPDM_OC_LOGIN}

      Remember: CPDM_OC_LOGIN is an alias for the cpd-cli manage login-to-ocp command.
   2. Run the appropriate command for your environment:

      Installations with tethered projects

      cpd-cli manage get-cr-status \
      --cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} \
      --tethered_instance_ns=${PROJECT_CPD_INSTANCE_TETHERED_LIST}

      Installations without tethered projects

      cpd-cli manage get-cr-status \
      --cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS}

   3. Ensure that the status of all of the services is Completed or Succeeded.
5. To view a list of restores, run the following command:

   cpd-cli oadp tenant-restore list

6. To view the detailed status of the restore, run the following command:

   cpd-cli oadp tenant-restore status ${TENANT_BACKUP_NAME}-restore \
   --details

   The command shows a varying number of sub-restores in the following form:

   cpd-tenant-r-xxx

   Tip: If you need more information, listed in the status details are sub-restores (of type group). You can view more information about these sub-restores by running the following command:

   cpd-cli oadp restore status <SUB_RESTORE_NAME> \
   --details

7. To view logs of the tenant restore, run the following command:

   cpd-cli oadp tenant-restore log ${TENANT_BACKUP_NAME}-restore

5.1 Applying cluster HTTP proxy settings or other RSI patches to the control plane

cpd-cli manage apply-rsi-patches --cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} -vvv

5.2 Patching Cognos Analytics instances

5.3 Restarting Data Replication replications

5.4 Restarting IBM Knowledge Catalog metadata import jobs

5.5 Restarting IBM Knowledge Catalog metadata enrichment jobs

After IBM Software Hub is restored, running metadata enrichment jobs might not complete successfully. Such jobs must be manually restarted.

5.6 Rerunning IBM Knowledge Catalog lineage data import jobs

If a lineage data import job is running at the same time that an online backup is taken, the job is in a Complete state when the backup is restored. However, users cannot see lineage data in the catalog. Rerun the lineage import job.

5.7 Restarting IBM Knowledge Catalog lineage pods

5.8 Resetting status of ongoing Watson OpenScale model evaluations

When a Watson OpenScale instance is restored, some of its features, such as scheduled or on-demand model evaluations, might not function properly. You must reset the status of ongoing model evaluations. For details, see Resetting status of ongoing Watson OpenScale model evaluations.

5.9 Retraining existing watsonx Assistant skills

After restoring the watsonx Assistant backup, it is necessary to retrain the existing skills. This involves modifying a skill, to trigger training. The training process for a skill typically requires less than 10 minutes to complete. For more information, see the Retraining your backend model section in the IBM Cloud documentation.

5.10 Restoring services that do not support online backup and restore