IBM Software Hub online backup and restore to the same cluster with the OADP utility

1.1 Checking the version of OADP utility components

1.2 Optional: Estimating how much storage to allocate for backups

You can estimate the amount of storage that you need to allocate for backups.

To use this feature, you must install the cpdbr-agent in the Red Hat OpenShift cluster. The cpdbr-agent deploys the node agents to the cluster. The node agents must be run in privileged mode.

1. Log in to Red Hat OpenShift Container Platform as a cluster administrator or tenant administrator:

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. Install the cpdbr-agent by running the following command:

   cpd-cli oadp install --component=cpdbr-agent --namespace=${OADP_PROJECT} --cpd-namespace=${PROJECT_CPD_INST_OPERANDS}

3. Export the following environment variable:

   export CPDBR_ENABLE_FEATURES=volume-util

4. Estimate how much storage you need to allocate to a backup by running the following command:

   cpd-cli oadp du-pv \
   --namespace=${OADP_PROJECT} \
   --cpd-namespace=${PROJECT_CPD_INST_OPERANDS}

1.3 Checking the primary instance of every PostgreSQL cluster is in sync with its replicas

The replicas for Cloud Native PostgreSQL and EDB Postgres clusters occasionally get out of sync with the primary node. To check whether this problem exists and to fix the problem, see the troubleshooting topic PostgreSQL cluster replicas get out of sync.

1.4 Preparing IBM Knowledge Catalog

If large metadata enrichment jobs are running while an online backup operation is triggered, the Db2 pre-backup hooks might fail because the database cannot be put into a write-suspended state. It is recommended to have minimal enrichment workload while the online backup is scheduled.

1.5 Labeling external NFS volumes

1.6 Excluding external volumes from backups

1.7 Checking the status of installed services

2.1 Backing up the scheduling service

If the IBM Software Hub scheduling service is installed, create a backup of the service.

Backups that are created in IBM Cloud Pak® for Data 5.0 cannot be restored in IBM Software Hub 5.10.0. You must take new backups in 5.10.0.

Check the Known issues and limitations for IBM Software Hub page for any workarounds that you might need to do before you create a backup.

1. Run the backup and restore utility in Kubernetes mode, and log in to Red Hat OpenShift Container Platform as a cluster administrator:

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. Configure the OADP client to set the IBM Software Hub project to the scheduling service project:

   cpd-cli oadp client config set cpd-namespace=${PROJECT_SCHEDULING_SERVICE}

3. Configure the OADP client to set the OADP project to the project where the OADP operator is installed:

   cpd-cli oadp client config set namespace=${OADP_PROJECT}

4. Run service backup prechecks:

   cpd-cli oadp backup precheck \
   --backup-type singleton \
   --include-namespaces=${PROJECT_SCHEDULING_SERVICE} \
   --log-level=debug \
   --verbose \
   --hook-kind checkpoint

5. Back up the IBM Software Hub scheduling service:

   cpd-cli oadp backup create \
   --backup-type singleton ${PROJECT_SCHEDULING_SERVICE}-online \
   --include-namespaces ${PROJECT_SCHEDULING_SERVICE} \
   --include-resources='clusterrole,clusterrolebinding,role,rolebinding,serviceaccount,deployment,configmap,customresourcedefinitions.apiextensions.k8s.io,scheduling.scheduler.spectrumcomputing.ibm.com,secret' \
   --prehooks=false \
   --posthooks=false \
   --with-checkpoint \
   --log-level=debug \
   --verbose \
   --hook-kind checkpoint \
   --selector 'icpdsupport/addOnId in (scheduling, bros),icpdsupport/ignore-on-nd-backup notin (true)' \
   --include-cluster-resources=true
   

6. Validate the backup:

   cpd-cli oadp backup validate \
   --backup-type singleton \
   --include-namespaces=${PROJECT_SCHEDULING_SERVICE} \
   --backup-names ${PROJECT_SCHEDULING_SERVICE}-online \
   --log-level trace \
   --verbose \
   --hook-kind checkpoint
   

2.2 Backing up an IBM Software Hub instance

Create an online backup of each IBM Software Hub instance, or tenant, in your environment by doing the following steps.

Check the Known issues and limitations for IBM Software Hub page for any workarounds that you might need to do before you create a backup.

1. If you are running the backup and restore utility in Kubernetes mode, and log in to Red Hat OpenShift Container Platform as a cluster administrator or tenant administrator:

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. Create a backup:

   cpd-cli oadp tenant-backup create ${BACKUP_NAME} \
   --tenant-operator-namespace ${PROJECT_CPD_INST_OPERATORS} \
   --log-level=debug \
   --verbose &> ${BACKUP_NAME}.log&

3. Use the cpd-cli oadp wait command to monitor the status of the tenant-backup create until it completes.

   CR_NAME=tenantbackups
   TIMEOUT=2h
   cpd-cli oadp wait \
   --br-operator-namespace ${BR_OPERATOR_NS} \
   --cr ${CR_NAME} \
   --name ${BACKUP_NAME} \
   --runtime-mode operator \
   --timeout ${TIMEOUT}

   For a list of all the custom resources that you can wait for, see oadp wait.

4. Confirm that the tenant backup was created and has a Completed status:

   cpd-cli oadp tenant-backup list

5. To view the detailed status of the backup, run the following command:

   cpd-cli oadp tenant-backup status ${BACKUP_NAME} \
   --details

   The command shows the following sub-backups:

   Backup	Description
   cpd-tenant-xxx	Backup that contains Kubernetes resources.
   cpd-tenant-vol-yyy	Backup that contains volume data.

   Tip: If you need more information, listed in the status details are sub-backups (of type group). You can view more information about these sub-backups by running the following command:

   cpd-cli oadp backup status <SUB_BACKUP_NAME> \
   --details

6. To view logs of the tenant backup and all sub-backups, run the following command:

   cpd-cli oadp tenant-backup log ${BACKUP_NAME}

5.1 Restoring the scheduling service

If the IBM Software Hub scheduling service was installed on the cluster at the time that you backed up IBM Software Hub, then you must install or restore the service on the cluster before you restore the IBM Software Hub instance.

You can also restore the scheduling service if you are experiencing problems with the service.

Check the Known issues and limitations for IBM Software Hub page for any workarounds that you might need to do before you restore a backup.

1. Log in to Red Hat OpenShift Container Platform as a cluster administrator or tenant administrator:

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. Configure the OADP client to set the IBM Software Hub project to the scheduling service project:

   cpd-cli oadp client config set cpd-namespace=${PROJECT_SCHEDULING_SERVICE}

3. Configure the OADP client to set the OADP project to the project where the OADP operator is installed:

   cpd-cli oadp client config set namespace=${OADP_PROJECT}

4. Restore an online backup.

   cpd-cli oadp restore create ${PROJECT_SCHEDULING_SERVICE}-restore \
   --from-backup=${PROJECT_SCHEDULING_SERVICE}-online \
   --include-resources='clusterrole,clusterrolebinding,role,rolebinding,serviceaccount,deployment,configmap,customresourcedefinitions.apiextensions.k8s.io,scheduling.scheduler.spectrumcomputing.ibm.com,secret' \
   --include-cluster-resources=true \
   --skip-hooks \
   --log-level=debug \
   --verbose

5.2 Restoring an IBM Software Hub instance

Restore an IBM Software Hub instance by doing the following steps. The steps depend on whether you want to restore the backup to the same namespace or a different namespace.

Restore to the same namespace

1. Log in to Red Hat OpenShift Container Platform as a cluster administrator or tenant administrator:

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. Restore IBM Software Hub to the original project:

   cpd-cli oadp tenant-restore create ${TENANT_BACKUP_NAME}-restore \
   --from-tenant-backup ${TENANT_BACKUP_NAME} \
   --verbose \
   --log-level=debug &> ${TENANT_BACKUP_NAME}-restore.log&

Restore to a different namespace

Tech preview This is a technology preview and is not supported for use in production environments.

1. Source the environment variables used to map the source and target namespaces. The mapping is used later in the restore commands. For instructions, see Setting up installation environment variables

   Make sure you have the following environment variables defined:

   • PROJECT_CPD_INST_OPERATORS
   • PROJECT_CPD_INST_OPERATORS_NEW
   • PROJECT_CPD_INST_OPERANDS
   • PROJECT_CPD_INST_OPERANDS_NEW
   • RESTORE_PROJECT_MAPPING
     • This environment variable must include ${PROJECT_INST_BR_SVC}:${PROJECT_INST_BR_SVC_NEW}

   If tenant projects are part of the backup, add those project mappings to RESTORE_PROJECT_MAPPING.

2. Log in to Red Hat OpenShift Container Platform as a cluster administrator:

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
3. Use the following command to create a YAML file that contains the definitions for clusterrolebindings, clusterroles, and webhook configurations.

   cpd-cli oadp restore cluster-scoped-resources \
   --namespace-mappings=${RESTORE_PROJECT_MAPPING} \
   --from-tenant-backup ${BACKUP_NAME} \
   --verbose \
   --log-level=debug \
   -f cluster_scoped_resources_${BACKUP_NAME}.yaml

4. Use the following command to create the resources.

   oc create -f cluster_scoped_resources_${BACKUP_NAME}.yaml

5. Use the following command to generate a YAML file that contains the cluster-scoped resources needed to restore the backup to different namespace.

   cpd-cli oadp tenant-restore create ${RESTORE_NAME} \
     --from-tenant-backup=${BACKUP_NAME} \
     --log-level=trace \
     --namespace-mappings=${RESTORE_PROJECT_MAPPING} \
     --override-group-attributes excludedResourceTypes:clusterrolebindings,clusterroles,mutatingwebhookconfigurations,validatingwebhookconfigurations \
     --verbose &> ${RESTORE_NAME}.log&

5.3 Monitor the progress of the restore

6.1 Applying cluster HTTP proxy settings or other RSI patches to the control plane

cpd-cli manage apply-rsi-patches --cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} -vvv

6.2 Patching Cognos Analytics instances

6.3 Patching Watson OpenScale database

If a Db2 or Db2 Warehouse database in the cluster is used as data mart database in Watson OpenScale, the port must be updated. Because the Db2 database port might be different after restoring, update the port in the Watson OpenScale instance database to the correct value. You can update the port through the Watson OpenScale user interface or API.

6.4 Resetting status of ongoing Watson OpenScale model evaluations

When a Watson OpenScale instance is restored, some of its features, such as scheduled or on-demand model evaluations, might not function properly. You must reset the status of ongoing model evaluations. For details, see Resetting status of ongoing Watson OpenScale model evaluations.

6.5 Restarting Data Replication replications

6.6 Restarting IBM Knowledge Catalog metadata import jobs

6.7 Restarting IBM Knowledge Catalog metadata enrichment jobs

After IBM Software Hub is restored, running metadata enrichment jobs might not complete successfully. Such jobs must be manually restarted.

6.8 Rerunning IBM Knowledge Catalog lineage data import jobs

If a lineage data import job is running at the same time that an online backup is taken, the job is in a Complete state when the backup is restored. However, users cannot see lineage data in the catalog. Rerun the lineage import job.

6.9 Restarting IBM Knowledge Catalog lineage pods

6.10 Retraining existing watsonx Assistant skills

After restoring the watsonx Assistant backup, it is necessary to retrain the existing skills. This involves modifying a skill, to trigger training. The training process for a skill typically requires less than 10 minutes to complete. For more information, see the Retraining your backend model section in the IBM Cloud documentation.

6.11 Restoring services that do not support online backup and restore