Restoring an online backup to the same cluster with the Cloud Pak for Data OADP backup and restore utility

You can restore an IBM® Cloud Pak for Data instance from an online backup on the same cluster with the OADP backup and restore utility.

Before you begin

Restoring Cloud Pak for Data from an online backup has the following requirements:

Cloud Pak for Data is deployed on Red Hat® OpenShift® Data Foundation or IBM Spectrum® Scale storage.
All services are installed at the same Cloud Pak for Data release.
Online backup and restore of a deployment that is running service versions from different Cloud Pak for Data releases is not supported.
The Cloud Pak for Data control plane is installed in a single project (namespace).
The Cloud Pak for Data instance is installed in zero or more tethered projects.
IBM Cloud Pak® foundational services operators are installed and operational in the ibm-common-services project.
Cloud Pak for Data platform operator and service operators are installed and operational in the ibm-common-services project or in another project that is separate from Cloud Pak for Data instance projects.
The Cloud Pak for Data and IBM Cloud Pak foundational services operators are not watching the Cloud Pak for Data instance projects (the cpd-operators NamespaceScope resource in the Cloud Pak for Data operator project does not include the Cloud Pak for Data instance projects).
The OADP backup and restore utility is installed and configured.

About this task

Cloud Pak for Data services that do not support online backup and restore might not be functional after a restore is completed. In these cases, the services must be cleaned up or deleted and re-installed. For more information, see Post-restore tasks after restoring a Cloud Pak for Data online backup.

You cannot restore a backup into a different project of the Cloud Pak for Data instance.

Restoring a Cloud Pak for Data instance from an online backup on the same cluster involves the following steps:

Delete the instance of Cloud Pak for Data on the cluster.
Restore the instance of Cloud Pak for Data on the cluster.

For more information about the OADP backup and restore utility, including a list of commands that you can run, see the cpd-cli oadp reference documentation.

Permissions that you need for this task: Log on as a user with cluster administrator rights.

Best practice: You can run the commands in this task exactly as written if you set up environment variables. For instructions, see Setting up installation environment variables.

Ensure that you source the environment variables before you run the commands in this task.

Deleting an instance of Cloud Pak for Data on the source cluster

Before you restore a backup, delete the Cloud Pak for Data instance on the source cluster.

Note: Resources in the Cloud Pak for Data instance are watched and managed by operators and controllers that run in other projects. More steps are needed to delete the Cloud Pak for Data instance to prevent corruption or out of sync operators and resources. These steps involve locating Kubernetes resources that have finalizers specified in metadata, and removing those finalizers before deleting the Cloud Pak for Data instance.

If the Cloud Pak for Data instance is integrated with the IAM service, delete clients in the Cloud Pak for Data instance.
1. Log in to Red Hat OpenShift Container Platform as a user with sufficient permissions to complete the task:
```
oc login ${OCP_URL}
```
2. If the Cloud Pak for Data instance was configured with iamintegration: true, delete clients in Cloud Pak for Data projects by running the following command:
```
oc delete client -n ${PROJECT_CPD_INSTANCE} --all
```
  Note: If the IBM Cloud Pak foundational services IAM OIDC is not watching the project where the Cloud Pak for Data instance is installed, you must do a manual cleanup by running the following commands:
```
# Get Client name
oc get client -n ${PROJECT_CPD_INSTANCE}
```
```
# Delete metadata.finalizers from Client
oc patch client <client_name> -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge
```
```
# Delete Clients
oc delete client -n ${PROJECT_CPD_INSTANCE} --all
```
```
# cpd-oidcclient-secret Secret
oc delete secret cpd-oidcclient-secret -n ${PROJECT_CPD_INSTANCE}
```

Delete service-specific finalizers from service custom resources (CRs), and delete the service CRs in Cloud Pak for Data instance projects.

Note: The following commands are only examples of deployed services.

# Get CCS CR, delete finalizers and delete CR
oc get ccs -n ${PROJECT_CPD_INSTANCE}
oc patch ccs <ccs_name> -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge
oc delete ccs -n ${PROJECT_CPD_INSTANCE} --all

# Get DataStage CR, delete finalizers and delete CR
oc get PXRuntime -n ${PROJECT_CPD_INSTANCE}
oc patch PXRuntime <PXRuntime_name> -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge
oc delete PXRuntime -n ${PROJECT_CPD_INSTANCE} --all
oc get DataStage -n ${PROJECT_CPD_INSTANCE}
oc patch DataStage <DataStage_name> -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge
oc delete DataStage -n ${PROJECT_CPD_INSTANCE} --all
    
# Get IIS CR, delete finalizers and delete CR
oc get iis  -n ${PROJECT_CPD_INSTANCE}
oc patch iis <iis_cr_name>  -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge
oc delete iis -n ${PROJECT_CPD_INSTANCE} --all

# Get wKC CR, delete finalizers and delete CR
oc get wkc -n ${PROJECT_CPD_INSTANCE}
oc patch wkc <wkc_cr_name>  -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge
oc delete wkc -n ${PROJECT_CPD_INSTANCE} --all

# Get UG CR, delete finalizers and delete CR
oc get ug ug-cr -n ${PROJECT_CPD_INSTANCE}
oc patch ug ug-cr -n ${PROJECT_CPD_INSTANCE}  -p '{"metadata":{"finalizers":[]}}' --type=merge
oc delete ug -n ${PROJECT_CPD_INSTANCE} --all

# Get Db2assservice CR, delete finalizers and delete CR
oc get Db2aaserviceService -n ${PROJECT_CPD_INSTANCE}
oc patch  Db2aaserviceService <db2asservice_cr_name> -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge
oc delete Db2aaserviceService  -n ${PROJECT_CPD_INSTANCE} --all

Delete finalizers from Zen-Service Operand Request in the Cloud Pak for Data instance projects.
1. Get the Zen-Service Operand Request name, which typically is zen-service:
```
oc get operandrequest -n ${PROJECT_CPD_INSTANCE}
```
2. Delete the finalizers:
```
oc patch operandrequest <zen_service_operandrequest_name> -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge
```
Delete OperandRequests in Cloud Pak for Data instance projects:
```
oc delete operandrequests -n ${PROJECT_CPD_INSTANCE} --all
```
Delete ZenServices in Cloud Pak for Data instance projects:
```
oc delete zenservice -n ${PROJECT_CPD_INSTANCE} --all
```

Delete finalizers from the admin RoleBinding in Cloud Pak for Data instance projects:

oc patch rolebinding admin -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge

Specialized installations only: Remove Cloud Pak for Data instance projects from cpd-operators NamespaceScope custom resource in the cpd-operators project by doing one of the following choices.

Remove the Cloud Pak for Data instance projects by editing the cpd-operators NamespaceScope resource:

oc edit namespacescope cpd-operators -n ${PROJECT_CPD_OPS}
# Remove ${PROJECT_CPD_OPS} Namespace from namespaceMembers field

Retrieve the existing projects in the namespaceMembers list and then remove the projects that comprise the Cloud Pak for Data instance:

# Retrieve existing namespaceMembers field
oc get namespacescope cpd-operators -n ${PROJECT_CPD_OPS} -o jsonpath={.spec.namespaceMembers}

# Remove ${PROJECT_CPD_INSTANCE} Namespace from namespaceMembers field
oc patch namespacescope cpd-operators -n ${PROJECT_CPD_OPS} -p $'{"spec":{"namespaceMembers":["<remaining-namespaces"]}}' --type=merge

Delete the Cloud Pak for Data instance projects, and remove any remaining finalizers.
1. To delete the Cloud Pak for Data instance project, run the following command:
```
oc delete project ${PROJECT_CPD_INSTANCE}
```
2. Confirm the project delete and check for status messages of type NamespaceFinalizersRemaining:
```
oc get project ${PROJECT_CPD_INSTANCE} -o jsonpath="{.status}"
```
3. If there are finalizers remaining, repeat these substeps to locate resources and delete finalizers.
Note: Even though the command returns quickly, the project deletion can take several minutes to fully complete. Continue to watch the final oc get project command to wait for successful completion.

Restoring the instance of Cloud Pak for Data on the same cluster

After you delete Cloud Pak for Data instance on the source cluster, restore the backup.

Check that the backup is available and that it completed with no errors:

cpd-cli oadp backup ls
NAME           STATUS      ERRORS  WARNINGS  CREATED    EXPIRES  STORAGE LOCATION  SELECTOR
<backup_name>  Completed     0       0      <timestamp>   358d       default        <none>

Restore volume data from the online backup, providing the volume backup ID and specifying a restore ID.
```
cpd-cli oadp restore create <chkpt-restore-id1> \
--from-backup=<ckpt-backup-id1> \
--skip-hooks=true \
--log-level=debug \
--verbose
```
Note: If you are restoring Db2® or Watson™ Knowledge Catalog, add --scale-wait-timeout 15m to ensure that the restore command completes successfully.

Restore Kubernetes resources from the online backup, excluding resources that create pods (such as deployments, StatefulSets, ReplicaSets, and CronJobs).

Provide the Kubernetes resources backup ID and specify the restore ID.

cpd-cli oadp restore create <chkpt-restore-id2> \
--from-backup=<ckpt-backup-id2> \
--exclude-resources='clients,ImageTag,deploy,rs,dc,rc,sts,ds,cj,jobs,controllerrevisions,po,opreq' \
--include-cluster-resources=true \
--skip-hooks \
--log-level=debug \
--verbose

Restore Kubernetes resources from the online backup, including resources that create pods.
Provide the Kubernetes resources backup ID and specify the restore ID.
```
cpd-cli oadp restore create <chkpt-restore-id3> \
--from-backup=<ckpt-backup-id2> \
--include-resources='deploy,rs,dc,rc,sts,ds,cj,jobs,controllerrevisions' \
--preworkloadhooks=true \
--posthooks=true \
--log-level=debug \
--verbose
```
Note: If you are restoring Db2 or Watson Knowledge Catalog, add --scale-wait-timeout 15m to ensure that the restore command completes successfully.
If Watson Assistant or Watson Discovery is installed on the cluster, clean up the NGINX configuration.
4.5.0 4.5.1 4.5.3 This step is required only if you are running IBM Cloud Pak for Data 4.5.0, 4.5.1, or 4.5.3.

Do the following substeps in a different terminal window.
1. Locate a usermgmt pod.
```
oc get pods | grep usermgmt
```
2. Clean up the NGINX configuration.
  In the following command, replace <USERMGMT-POD-NAME> with the value that was obtained in the previous step.
```
oc exec <USERMGMT-POD-NAME> -- rm /user-home/_global_/nginx-conf.d/{filename}-gw-addon.conf  /user-home/_global_/nginx-conf.d/{filename}-gw-instance.conf
```
  where {filename} is the appropriate file name, such as discovery-gw-addon.conf and wa-gw-instance.conf.

Restore operators watch of Cloud Pak for Data instance projects.

Provide the Kubernetes resources backup ID and specify the restore ID.

cpd-cli oadp restore create <chkpt-restore-id4> \
--from-backup=<ckpt-backup-id2> \
--include-resources='opreq' \
--skip-hooks \
--log-level=debug \
--verbose

In specialized installations (IBM Cloud Pak foundational services operators and the IBM Cloud Pak for Data platform operator and service operators are installed in different projects), update the cpd-operators NamespaceScope resource in the Cloud Pak for Data operator project to watch the project where you are restoring the instance.
In Cloud Pak for Data projects, reconcile Cloud Pak for Data and IBM Cloud Pak foundational services operators and Cloud Pak for Data and IBM Cloud Pak foundational services custom resources.
To check the status of a restore, run the following command:
```
cpd-cli oadp restore status <restore_name> \
--details
```
To view a list of existing restores, run the following command:
```
cpd-cli oadp restore ls
```
To view restore logs, run the following command:
```
cpd-cli oadp restore logs <restore_name>
```
Optional: To delete a restore for cleanup purposes, run the following command:
```
cpd-cli oadp restore delete <restore_name>
```