Offline backup and restore to a different cluster with the IBM Software Hub OADP utility

2.1 Setting the mode in which to create backups

You can run the IBM Software Hub OADP backup and restore utility in Kubernetes mode or in REST mode.

By default, the IBM Software Hub OADP backup and restore utility runs in Kubernetes mode. In this mode, you must log in to your Red Hat OpenShift cluster and you must have Kubernetes cluster administrator privileges to use the utility.

If you installed the IBM Software Hub OADP backup REST service, you can run the utility in REST mode to create backups. In REST mode, the utility runs as a REST client that communicates to a REST server. The REST service is configured to work with a specific IBM Software Hub instance. You do not have to log in to the cluster, and IBM Software Hub users with the Administrator role can run backup and checkpoint commands on their own IBM Software Hub instances, based on the specified control plane and any tethered projects.

Running the utility in REST mode is useful when you are generally creating backups only, or when backups take a long time to complete. For backups that take a long time to complete, running the utility in REST mode avoids the problem of the Red Hat OpenShift user session token expiring before the backup process completes. If the session token expires, you must log back in to the cluster and reset the utility.

1. Log in to Red Hat OpenShift Container Platform as a cluster administrator:

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. To create backups in REST mode, run the following command:

   cpd-cli oadp client config set runtime-mode=rest-client

3. To change the IBM Software Hub OADP backup and restore utility back to the Kubernetes mode, run the following command:

   cpd-cli oadp client config set runtime-mode=

Related topic: Unable to run an online backup or restore operation

2.2 Backing up the scheduling service

If the IBM Software Hub scheduling service is installed, create a backup of the service.

Backups that are created in IBM Cloud Pak for Data 5.0 cannot be restored in IBM Software Hub 5.1.0. You must take new backups in 5.1.0.

Check the Known issues and limitations for IBM Software Hub page for any workarounds that you might need to do before you create a backup.

1. If you are running the backup and restore utility in Kubernetes mode, log in to Red Hat OpenShift Container Platform as a cluster administrator:

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. Configure the OADP client to set the IBM Software Hub project to the scheduling service project:

   cpd-cli oadp client config set cpd-namespace=${PROJECT_SCHEDULING_SERVICE}

3. Configure the OADP client to set the OADP project to the project where the OADP operator is installed:

   cpd-cli oadp client config set namespace=${OADP_PROJECT}

4. Run service backup prechecks:

   IBM Software Hub 5.1.0

   cpd-cli oadp backup precheck \
   --include-namespaces=${PROJECT_SCHEDULING_SERVICE} \
   --log-level=debug \
   --verbose \
   --hook-kind=br

   IBM Software Hub 5.1.1 and later

   cpd-cli oadp backup precheck \
   --backup-type singleton \
   --include-namespaces=${PROJECT_SCHEDULING_SERVICE} \
   --log-level=debug \
   --verbose \
   --hook-kind=br

5. Back up the IBM Software Hub scheduling service:

   The cluster pulls images from the IBM Entitled Registry

   IBM Software Hub 5.1.0

   cpd-cli oadp backup create ${PROJECT_SCHEDULING_SERVICE}-offline \
   --include-namespaces ${PROJECT_SCHEDULING_SERVICE} \
   --include-resources='operatorgroups,configmaps,catalogsources.operators.coreos.com,subscriptions.operators.coreos.com,customresourcedefinitions.apiextensions.k8s.io,scheduling.scheduler.spectrumcomputing.ibm.com' \
   --prehooks=true \
   --posthooks=true \
   --log-level=debug \
   --verbose \
   --hook-kind=br \
   --selector 'velero.io/exclude-from-backup notin (true)' \
   --image-prefix=registry.redhat.io/ubi9

   IBM Software Hub 5.1.1 and later

   cpd-cli oadp backup create ${PROJECT_SCHEDULING_SERVICE}-offline \
   --backup-type singleton \
   --include-namespaces ${PROJECT_SCHEDULING_SERVICE} \
   --include-resources='operatorgroups,configmaps,catalogsources.operators.coreos.com,subscriptions.operators.coreos.com,customresourcedefinitions.apiextensions.k8s.io,scheduling.scheduler.spectrumcomputing.ibm.com' \
   --prehooks=true \
   --posthooks=true \
   --log-level=debug \
   --verbose \
   --hook-kind=br \
   --selector 'velero.io/exclude-from-backup notin (true)' \
   --image-prefix=registry.redhat.io/ubi9

   The cluster pulls images from a private container registry

   IBM Software Hub 5.1.0

   cpd-cli oadp backup create ${PROJECT_SCHEDULING_SERVICE}-offline \
   --include-namespaces ${PROJECT_SCHEDULING_SERVICE} \
   --include-resources='operatorgroups,configmaps,catalogsources.operators.coreos.com,subscriptions.operators.coreos.com,customresourcedefinitions.apiextensions.k8s.io,scheduling.scheduler.spectrumcomputing.ibm.com' \
   --prehooks=true \
   --posthooks=true \
   --log-level=debug \
   --verbose \
   --hook-kind=br \
   --selector 'velero.io/exclude-from-backup notin (true)' \
   --image-prefix=PRIVATE_REGISTRY_LOCATION/ubi9

   IBM Software Hub 5.1.1 and later

   cpd-cli oadp backup create ${PROJECT_SCHEDULING_SERVICE}-offline \
   --backup-type singleton \
   --include-namespaces ${PROJECT_SCHEDULING_SERVICE} \
   --include-resources='operatorgroups,configmaps,catalogsources.operators.coreos.com,subscriptions.operators.coreos.com,customresourcedefinitions.apiextensions.k8s.io,scheduling.scheduler.spectrumcomputing.ibm.com' \
   --prehooks=true \
   --posthooks=true \
   --log-level=debug \
   --verbose \
   --hook-kind=br \
   --selector 'velero.io/exclude-from-backup notin (true)' \
   --image-prefix=PRIVATE_REGISTRY_LOCATION/ubi9

6. Validate the backup:

   IBM Software Hub 5.1.0

   cpd-cli oadp backup validate \
   --include-namespaces=${PROJECT_SCHEDULING_SERVICE} \
   --backup-names ${PROJECT_SCHEDULING_SERVICE}-offline \
   --log-level trace \
   --verbose \
   --hook-kind=br

   IBM Software Hub 5.1.1 and later

   cpd-cli oadp backup validate \
   --backup-type singleton \
   --include-namespaces=${PROJECT_SCHEDULING_SERVICE} \
   --backup-names ${PROJECT_SCHEDULING_SERVICE}-offline \
   --log-level trace \
   --verbose \
   --hook-kind=br

2.3 Backing up an IBM Software Hub instance

Create an offline backup of each IBM Software Hub instance, or tenant, in your environment by doing the following steps.

Check the Known issues and limitations for IBM Software Hub page for any workarounds that you might need to do before you create a backup.

1. If you are running the backup and restore utility in Kubernetes mode, log in to Red Hat OpenShift Container Platform as a cluster administrator:

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. 5.1.0 Ensure that the expected EDB Postgres replica PVCs are included in the backup:

   oc label pvc,pods -l k8s.enterprisedb.io/cluster,velero.io/exclude-from-backup=true velero.io/exclude-from-backup- -n ${PROJECT_CPD_INST_OPERANDS}

3. Create a backup by running one of the following commands.

   The cluster pulls images from the IBM Entitled Registry

   cpd-cli oadp tenant-backup create ${TENANT_OFFLINE_BACKUP_NAME} \
   --namespace ${OADP_PROJECT} \
   --vol-mnt-pod-mem-request=1Gi \
   --vol-mnt-pod-mem-limit=4Gi \
   --tenant-operator-namespace ${PROJECT_CPD_INST_OPERATORS} \
   --mode offline \
   --image-prefix=registry.redhat.io/ubi9 \
   --log-level=debug \
   --verbose &> ${TENANT_OFFLINE_BACKUP_NAME}.log&

   The cluster pulls images from a private container registry

   cpd-cli oadp tenant-backup create ${TENANT_OFFLINE_BACKUP_NAME} \
   --namespace ${OADP_PROJECT} \
   --vol-mnt-pod-mem-request=1Gi \
   --vol-mnt-pod-mem-limit=4Gi \
   --tenant-operator-namespace ${PROJECT_CPD_INST_OPERATORS} \
   --mode offline \
   --image-prefix=PRIVATE_REGISTRY_LOCATION/ubi9 \
   --log-level=debug \
   --verbose &> ${TENANT_OFFLINE_BACKUP_NAME}.log&

   Note: If the backup fails during the volume backup stage, try increasing the --vol-mnt-pod-mem-limit option. You might need to increase this option when you have terabytes of data.
4. Confirm that the tenant backup was created and has a Completed status:

   cpd-cli oadp tenant-backup list

5. To view the detailed status of the backup, run the following command:

   cpd-cli oadp tenant-backup status ${TENANT_BACKUP_NAME} \
   --details

   The command shows the following sub-backups:

   Backup	Description
   cpd-tenant-xxx	Backup that contains Kubernetes resources.
   cpd-tenant-vol-yyy	Backup that contains volume data.

   Tip: If you need more information, listed in the status details are sub-backups (of type group). You can view more information about these sub-backups by running the following command:

   cpd-cli oadp backup status <SUB_BACKUP_NAME> \
   --details

6. To view logs of the tenant backup and all sub-backups, run the following command:

   cpd-cli oadp tenant-backup log ${TENANT_BACKUP_NAME}

2.4 Doing post-backup tasks

For some services, you must do additional tasks after you create an offline backup.

1. 5.1.2 and later If RStudio Server Runtimes is installed, remove the RStudio Server Runtimes service from maintenance mode:

   oc patch -n ${PROJECT_CPD_INST_OPERANDS} rstudioaddon rstudio-cr --type=merge -p '{"spec": {"ignoreForMaintenance":false}}'
   oc -n ${PROJECT_CPD_INST_OPERANDS} get rstudio -w

2. 5.1.0 If Data Refinery is installed, restart the service:
   1. Log in to Red Hat OpenShift Container Platform as a cluster administrator.

      ${OC_LOGIN}

      Remember: OC_LOGIN is an alias for the oc login command.
   2. Run the following command.

      The value of <number_of_replicas> depends on the scaleConfig setting when Data Refinery was installed (1 for small, 3 for medium, and 4 for large).

      oc scale --replicas=<number_of_replicas> deploy wdp-shaper wdp-dataprep

3. 5.1.0 If watsonx.ai is installed, manually scale up three deployments.
   1. Log in to Red Hat OpenShift Container Platform as a cluster administrator.

      ${OC_LOGIN}

      Remember: OC_LOGIN is an alias for the oc login command.
   2. Wait for watsonxai-cr to reach the Completed state:

      oc get watsonxai -n ${PROJECT_CPD_INST_OPERANDS}

      Check that the command returns output such as in the following example:

      NAME           VERSION   RECONCILED   STATUS      AGE
      watsonxai-cr   9.1.0     9.1.0        Completed 4d5h

   3. Scale up the following deployment:

      oc scale deploy caikit-runtime-stack-operator -n ${PROJECT_CPD_INST_OPERATORS} --replicas=1

3.1 Preparing the target cluster

Prepare the target cluster that you want to use to restore IBM Software Hub.

1. Make sure that the target cluster meets the following requirements:
   • The target cluster has the same storage classes as the source cluster.
   • For environments that use a private container registry, such as air-gapped environments, the target cluster has the same image content source policy as the source cluster. For details on configuring the image content source policy, see Configuring an image content source policy for IBM Software Hub software images.
   • The target cluster must be able to pull software images. For details, see Updating the global image pull secret for IBM Software Hub.
   • The deployment environment of the target cluster is the same as the source cluster.
     • The target cluster uses the same hardware architecture as the source cluster. For example, x86-64.
     • The target cluster is on the same OpenShift version as the source cluster.
     • The target cluster allows for the same node configuration as the source cluster. For example, if the source cluster uses a custom KubeletConfig, the target cluster must allow the same custom KubeletConfig.
     • Moving between IBM Cloud and non-IBM Cloud deployment environments is not supported.
2. If you are using node labels as the method for identifying nodes in the cluster, re-create the labels on the target cluster.
   Best practice: Use node labels instead of node lists when you are restoring a IBM Software Hub deployment to a different cluster, especially if you plan to enforce node pinning. Node labels enable node pinning with minimal disruption. To learn more, see Passing node information to IBM Software Hub.
3. If your cluster pulls images from a private container registry or if your cluster is in a restricted network, push images that the OADP backup and restore utility needs to the private container registry so that users can run the restore commands against the cluster.

   For details, see 2. Moving images for backup and restore to a private container registry.

4. Install the components that the OADP backup and restore utility uses.
   Tip: You need to use the same configuration information that you specified in the source cluster. For example, when you install OADP, use the same credentials and DataProtectionApplication configuration that was specified on the source cluster.
   1. Log in to Red Hat OpenShift Container Platform as a cluster administrator.

      ${OC_LOGIN}

      Remember: OC_LOGIN is an alias for the oc login command.
   2. Create the environment variables that the utility needs so that you can copy commands from the documentation and run them without making any changes.
   3. Create the ${OADP_PROJECT} project where you want to install the OADP operator.
   4. Annotate the ${OADP_PROJECT} project so that Restic pods can be scheduled on all nodes.

      oc annotate namespace ${OADP_PROJECT} openshift.io/node-selector=""

   5. Install the cpdbr-tenant service role-based access controls (RBACs).
      Note: Run the cpdbr installation command in the IBM Software Hub operators project even though the project does not yet exist in the target cluster. Do not manually create the project on the target cluster. The project is created during the IBM Software Hub restore process.

      The cluster pulls images from the IBM Entitled Registry

      cpd-cli oadp install \
      --component=cpdbr-tenant \
      --namespace ${OADP_PROJECT} \
      --tenant-operator-namespace ${PROJECT_CPD_INST_OPERATORS} \
      --rbac-only \
      --log-level=debug \
      --verbose

      The cluster pulls images from a private container registry

      cpd-cli oadp install \
      --component=cpdbr-tenant \
      --namespace ${OADP_PROJECT} \
      --tenant-operator-namespace ${PROJECT_CPD_INST_OPERATORS} \
      --cpdbr-hooks-image-prefix=${PRIVATE_REGISTRY}/cpdbr-oadp:${VERSION} \
      --cpfs-image-prefix=${PRIVATE_REGISTRY} \
      --rbac-only \
      --log-level=debug \
      --verbose

   6. Install the Red Hat OADP operator.
      • To install operators in a cluster with Internet access, see the Red Hat documentation Adding Operators to a cluster.
      • To install operators in a cluster that is in a restricted network, see the Red Hat documentation Using Operator Lifecycle Manager on restricted networks.
   7. Create a secret in the ${OADP_PROJECT} project with the credentials of the S3-compatible object store that you are using to store the backups.

      Credentials must use alphanumeric characters and cannot contain special characters like the number sign (#).

      1. Create a file named credentials-velero that contains the credentials for the object store:

         cat << EOF > credentials-velero
         [default]
         aws_access_key_id=${ACCESS_KEY_ID}
         aws_secret_access_key=${SECRET_ACCESS_KEY}
         EOF

      2. Create the secret.

         The name of the secret must be cloud-credentials.

         oc create secret generic cloud-credentials \
         --namespace ${OADP_PROJECT} \
         --from-file cloud=./credentials-velero

   8. Create the DataProtectionApplication (DPA) custom resource, and specify a name for the instance.
      Tip: You can create the DPA custom resource manually or by using the cpd-cli oadp dpa create command. However, if you use this command, you might need to edit the custom resource afterward to add options that are not available with the command. This step shows you how to manually create the custom resource.
      You might need to change some values.

      • spec.configuration.restic.memory specifies the Restic memory limit. You might need to increase the Restic memory limit if Restic volume backups fail or hang on a large volume, indicated by Restic pod containers restarting due to an OOMKilled Kubernetes error.
      • If the object store is Amazon S3, you can omit s3ForcePathStyle.
      • For object stores with a self-signed certificate, add backupLocations.velero.objectStorage.caCert and specify the base64 encoded certificate string as the value. For more information, see Use Self-Signed Certificate.
      Important:

      • spec.configuration.nodeAgent.timeout specifies the Restic timeout. The default is 1 hour. You might need to increase the Restic timeout if Restic backup or restore fails, indicated by pod volume timeout errors in the Velero log.
      • If only Restic backups are needed, under spec.configuration.velero.defaultPlugins, remove csi.
      • The object storage information (backupLocations.velero.objectStorage) in the source and target cluster DPA configurations must be identical.

      Recommended DPA configuration

      The following example shows the recommended DPA configuration.

      cat << EOF | oc apply -f -
      apiVersion: oadp.openshift.io/v1alpha1
      kind: DataProtectionApplication
      metadata:
        name: dpa-sample
      spec:
        configuration:
          velero:
            customPlugins:
            - image: ${CPDBR_VELERO_PLUGIN_IMAGE_LOCATION}
              name: cpdbr-velero-plugin
            defaultPlugins:
            - aws
            - openshift
            - csi
            podConfig:
              resourceAllocations:
                limits:
                  cpu: "${VELERO_POD_CPU_LIMIT}"
                  memory: 4Gi
                requests:
                  cpu: 500m
                  memory: 256Mi
            resourceTimeout: 60m
          nodeAgent:
            enable: true
            uploaderType: restic
            timeout: 72h
            podConfig:
              resourceAllocations:
                limits:
                  cpu: "${NODE_AGENT_POD_CPU_LIMIT}"
                  memory: 32Gi
                requests:
                  cpu: 500m
                  memory: 256Mi
              tolerations:
              - key: icp4data
                operator: Exists
                effect: NoSchedule
        backupImages: false
        backupLocations:
          - velero:
              provider: aws
              default: true
              objectStorage:
                bucket: ${BUCKET_NAME}
                prefix: ${BUCKET_PREFIX}
              config:
                region: ${REGION}
                s3ForcePathStyle: "true"
                s3Url: ${S3_URL}
              credential:
                name: cloud-credentials
                key: cloud
      EOF

      5.1.0-5.1.2 DPA configuration if watsonx™ Orchestrate is installed

      If your IBM Software Hub deployment includes watsonx Orchestrate, add the appcon-plugin to the DPA configuration. To obtain the link to the appcon-plugin image, see Backing up and restoring your IBM App Connect resources and persistent volumes on Red Hat OpenShift.

      cat << EOF | oc apply -f -
      apiVersion: oadp.openshift.io/v1alpha1
      kind: DataProtectionApplication
      metadata:
        name: dpa-sample
      spec:
        configuration:
          velero:
            customPlugins:
            - image: ${CPDBR_VELERO_PLUGIN_IMAGE_LOCATION}
              name: cpdbr-velero-plugin
            - image: '<appcon-plugin-image-link>'
              name: appcon-plugin
            defaultPlugins:
            - aws
            - openshift
            - csi
            podConfig:
              resourceAllocations:
                limits:
                  cpu: "${VELERO_POD_CPU_LIMIT}"
                  memory: 4Gi
                requests:
                  cpu: 500m
                  memory: 256Mi
            resourceTimeout: 60m
          nodeAgent:
            enable: true
            uploaderType: restic
            timeout: 72h
            podConfig:
              resourceAllocations:
                limits:
                  cpu: "${NODE_AGENT_POD_CPU_LIMIT}"
                  memory: 32Gi
                requests:
                  cpu: 500m
                  memory: 256Mi
              tolerations:
              - key: icp4data
                operator: Exists
                effect: NoSchedule
        backupImages: false
        backupLocations:
          - velero:
              provider: aws
              default: true
              objectStorage:
                bucket: ${BUCKET_NAME}
                prefix: ${BUCKET_PREFIX}
              config:
                region: ${REGION}
                s3ForcePathStyle: "true"
                s3Url: ${S3_URL}
              credential:
                name: cloud-credentials
                key: cloud
      EOF

   9. After you create the DPA, do the following checks.
      1. Check that the velero pods are running in the ${OADP_PROJECT} project.

         oc get po -n ${OADP_PROJECT}

         The node-agent daemonset creates one node-agent pod for each worker node. For example:

         NAME                                                    READY   STATUS    RESTARTS   AGE
         openshift-adp-controller-manager-678f6998bf-fnv8p       2/2     Running   0          55m
         node-agent-455wd                                        1/1     Running   0          49m
         node-agent-5g4n8                                        1/1     Running   0          49m
         node-agent-6z9v2                                        1/1     Running   0          49m
         node-agent-722x8                                        1/1     Running   0          49m
         node-agent-c8qh4                                        1/1     Running   0          49m
         node-agent-lcqqg                                        1/1     Running   0          49m
         node-agent-v6gbj                                        1/1     Running   0          49m
         node-agent-xb9j8                                        1/1     Running   0          49m
         node-agent-zjngp                                        1/1     Running   0          49m
         velero-7d847d5bb7-zm6vd                                 1/1     Running   0          49m

      2. Verify that the backup storage location PHASE is Available.

         cpd-cli oadp backup-location list

         Example output:

         NAME           PROVIDER    BUCKET             PREFIX              PHASE        LAST VALIDATED      ACCESS MODE
         dpa-sample-1   aws         ${BUCKET_NAME}     ${BUCKET_PREFIX}    Available    <timestamp>

5. Install the jq JSON command-line utility.
6. Configure the IBM Software Hub OADP backup and restore utility.
7. Install Certificate manager and the IBM License Service.

   For details, see Installing shared cluster components for IBM Software Hub.

   Note: You must install the same version of Certificate manager and the IBM License Service that is installed on the source cluster.
8. If IBM Knowledge Catalog Premium or IBM Knowledge Catalog Standard is installed, install Red Hat OpenShift AI.

3.2 Cleaning up the target cluster after a previous restore

If you previously restored a IBM Software Hub backup or a previous restore attempt was unsuccessful, delete the IBM Software Hub instance projects (namespaces) in the target cluster before you try another restore.

Resources in the IBM Software Hub instance are watched and managed by operators and controllers that run in other projects. To prevent corruption or out of sync operators and resources when you delete a IBM Software Hub instance, Kubernetes resources that have finalizers specified in metadata must be located, and those finalizers must be deleted before you can delete the IBM Software Hub instance.

1. Log in to Red Hat OpenShift Container Platform as an instance administrator.

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. Download the cpd-pre-restore-cleanup.sh script from https://github.com/IBM/cpd-cli/tree/master/cpdops/5.1.3.
3. If the tenant operator project exists and has the common-service NamespaceScope custom resource that identifies all the tenant projects, run the following command:

   ./cpd-pre-restore-cleanup.sh --tenant-operator-namespace="${PROJECT_CPD_INST_OPERATORS}"

4. If the tenant operator project does not exist or specific IBM Software Hub projects need to be deleted, run the following command.

   If the common-service NamespaceScope custom resource is not available and additional projects, such as tethered projects, need to be deleted, modify the list of comma-separated projects in the --additional-namespaces option as necessary.

   ./cpd-pre-restore-cleanup.sh --additional-namespaces="${PROJECT_CPD_INST_OPERATORS},${PROJECT_CPD_INST_OPERANDS}"

5. If the IBM Software Hub scheduling service was installed, uninstall it.

   For details, see Uninstalling the scheduling service.

4.1 Restoring the scheduling service

If the IBM Software Hub scheduling service is installed on the source cluster, restore the service on the target cluster by doing the following steps.

Check the Known issues and limitations for IBM Software Hub page for any workarounds that you might need to do before you restore a backup.

1. Log in to Red Hat OpenShift Container Platform as a cluster administrator:

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. Restore an offline backup by running one of the following commands.

   The cluster pulls images from the IBM Entitled Registry

   cpd-cli oadp restore create ${PROJECT_SCHEDULING_SERVICE}-restore \
   --from-backup=${PROJECT_SCHEDULING_SERVICE}-offline \
   --include-resources='operatorgroups,configmaps,catalogsources.operators.coreos.com,subscriptions.operators.coreos.com,customresourcedefinitions.apiextensions.k8s.io,scheduling.scheduler.spectrumcomputing.ibm.com' \
   --include-cluster-resources=true \
   --skip-hooks \
   --log-level=debug \
   --verbose \
   --image-prefix=registry.redhat.io/ubi9

   The cluster pulls images from a private container registry

   cpd-cli oadp restore create ${PROJECT_SCHEDULING_SERVICE}-restore \
   --from-backup=${PROJECT_SCHEDULING_SERVICE}-offline \
   --include-resources='operatorgroups,configmaps,catalogsources.operators.coreos.com,subscriptions.operators.coreos.com,customresourcedefinitions.apiextensions.k8s.io,scheduling.scheduler.spectrumcomputing.ibm.com' \
   --include-cluster-resources=true \
   --skip-hooks \
   --log-level=debug \
   --verbose \
   --image-prefix=${PRIVATE_REGISTRY_LOCATION}

4.2 Restoring an IBM Software Hub instance

Restore an IBM Software Hub instance to a different cluster by doing the following steps.

Check the Known issues and limitations for IBM Software Hub page for any workarounds that you might need to do before you restore a backup.

1. Log in to Red Hat OpenShift Container Platform as a cluster administrator:

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. Restore IBM Software Hub by running one of the following commands.

   The cluster pulls images from the IBM Entitled Registry

   cpd-cli oadp tenant-restore create ${TENANT_OFFLINE_BACKUP_NAME}-restore \
   --from-tenant-backup ${TENANT_OFFLINE_BACKUP_NAME} \
   --image-prefix=registry.redhat.io/ubi9 \
   --verbose \
   --log-level=debug &> ${TENANT_OFFLINE_BACKUP_NAME}-restore.log&

   The cluster pulls images from a private container registry

   cpd-cli oadp tenant-restore create ${TENANT_OFFLINE_BACKUP_NAME}-restore \
   --from-tenant-backup ${TENANT_OFFLINE_BACKUP_NAME} \
   --image-prefix=${PRIVATE_REGISTRY_LOCATION}/ubi9 \
   --verbose \
   --log-level=debug &> ${TENANT_OFFLINE_BACKUP_NAME}-restore.log&

3. Get the status of the installed components:
   1. Log the cpd-cli in to the Red Hat OpenShift Container Platform cluster:

      ${CPDM_OC_LOGIN}

      Remember: CPDM_OC_LOGIN is an alias for the cpd-cli manage login-to-ocp command.
   2. Run the appropriate command for your environment:

      Installations with tethered projects

      cpd-cli manage get-cr-status \
      --cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} \
      --tethered_instance_ns=${PROJECT_CPD_INSTANCE_TETHERED_LIST}

      Installations without tethered projects

      cpd-cli manage get-cr-status \
      --cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS}

   3. Ensure that the status of all of the services is Completed or Succeeded.
4. To view a list of restores, run the following command:

   cpd-cli oadp tenant-restore list

5. To view the detailed status of the restore, run the following command:

   cpd-cli oadp tenant-restore status ${TENANT_BACKUP_NAME}-restore \
   --details

   The command shows a varying number of sub-restores in the following form:

   cpd-tenant-r-xxx

   Tip: If you need more information, listed in the status details are sub-restores (of type group). You can view more information about these sub-restores by running the following command:

   cpd-cli oadp restore status <SUB_RESTORE_NAME> \
   --details

6. To view logs of the tenant restore, run the following command:

   cpd-cli oadp tenant-restore log ${TENANT_BACKUP_NAME}-restore