Restoring the Cloud Pak for Data file system from a local repository or object store

You can do an offline restore of the persistent volumes that are associated with your IBM® Cloud Pak for Data project from a local repository or an S3 or S3 compatible object store.

Before you begin

Cloud Pak for Data provides the cpd-cli backup-restore command-line interface for backing up and restoring the file system. Before you run any cpd-cli backup-restore commands, ensure that:

You completed the steps in Installing the Cloud Pak for Data file system backup and restore service.
You must have created a backup of the volumes.
If you are restoring Watson™ Studio on NFS, there are pre-restore steps that you must do.

About this task

During the restore process, write operations in application workloads are suspended (quiesced). The quiesce command calls hooks provided by Cloud Pak for Data services to do the quiesce. Quiesce hooks that are provided by Cloud Pak for Data services offer optimizations or other enhancements compared to scaling down all resources in the project (namespace). Services might be quiesced and unquiesced in a certain order, or services might be suspended without having to bring down pods to reduce the time it takes to bring down applications and bring them back up.

You can restore volumes in two ways:

Manually scale down resources, restore volumes, and then manually scale up resources.
Automatically scale down resources, restore volumes, and automatically scale up resources with a single command.

Tip: It is a good idea to manually scale down application Kubernetes resources before you do a restore, so that you can find out whether a pod's services cannot scale down or up correctly. You can then do the restore after you fix any problems that were found.

The commands in the following section use an example Cloud Pak for Data project named zen.

For a list of all possible options, enter the command ./cpd-cli backup-restore -h.

Procedure

If you haven't already done so, initialize cpd-cli backup-restore.

cpd-cli backup-restore is initialized before you create a backup.

Note: If your Docker image registry is different than what is shown in the following examples, change the appropriate options.

OpenShift® 4.5 example:

# Initialize the cpdbr first with pvc name and s3 storage.  Note that the bucket must exist.
$ cpd-cli backup-restore init --namespace zen --pvc-name cpdbr-pvc --image-prefix=image-registry.openshift-image-registry.svc:5000/$NAMESPACE \
     --provider=s3 --s3-endpoint="s3 endpoint" --s3-bucket=cpdbr --s3-prefix=zen/

OpenShift 3.11 example:

# Initialize the cpdbr first with pvc name and s3 storage.  Note that the bucket must exist.
$ cpd-cli backup-restore init -n zen --pvc-name cpdbr-pvc --image-prefix=docker-registry.default.svc:5000/$NAMESPACE \
     --provider=s3 --s3-endpoint="s3 endpoint" --s3-bucket=cpdbr --s3-prefix=zen/

To restore volumes by manually scaling down resources, restoring volumes, and manually scaling up resources, do the following steps.
1. Manually scale down application Kubernetes resources.
```
cpd-cli backup-restore quiesce -n zen
```
  If you want to ignore quiesce hooks, include the --ignore-hooks option.
2. If you are restoring IBM Streams, delete the Streams application and build resources by running the following commands.
```
oc delete pod -l streams-builder=active
oc delete pod -l streams-app=active
```
3. Check for completed jobs and pods by running the volume restore command with the --dry-run option, specifying a restore name identifier.
  The --dry-run option reports jobs or pods that are still attached to the PVCs to be restored.
  
  Note: The restore name identifier must consist of lower case alphanumeric characters or the hyphen (-), and must start and end with an alphanumeric character. The underscore character (_) is not supported.
```
cpd-cli backup-restore volume-restore create --from-backup backup_name -n zen --dry-run <restore_name_identifier>
```
4. If the dry run reports completed or failed jobs, or pods, that reference PVCs, delete them.
  
  Tip: Consider saving the job/pod yaml before you manually delete them, or include the --cleanup-completed-resources option in the restore step.
5. Run the restore command with the --skip-quiesce option.
```
cpd-cli backup-restore volume-restore create --from-backup backup_name -n zen <restore_name_identifier> --skip-quiesce=true
```
  Note: With certain storage providers, Kubernetes resources must be scaled down to unmount the PVCs before you create the restore. In such scenarios, the volume-restore create command with the --skip-quiesce option can fail if pods are running with mounted PVCs. If this problem occurs, use the quiesce command with the --ignore-hooks option to scale down the resources, and rerun the volume-restore create command with the --skip-quiesce option. You can then scale up the Kubernetes resources after the restore by using the unquiesce command with the --ignore-hooks option.
6. Manually scale up application Kubernetes resources.
```
cpd-cli backup-restore unquiesce -n zen
```
  If you want to ignore quiesce hooks, include the --ignore-hooks option.
To automatically scale down resources, restore volumes, and automatically scale up resources, do the following steps.

Note: With auto-quiesce and auto-unquiesce, resources that are associated with PVCs are scaled down and up. Quiesce and unquiesce hooks are not invoked.
1. If you are restoring IBM Streams, delete the Streams application and build resources by running the following commands.
```
oc delete pod -l streams-builder=active
oc delete pod -l streams-app=active
```
2. Run the following volume restore command, specifying a restore name identifier.
  Note: The restore name identifier must consist of lower case alphanumeric characters or the hyphen (-), and must start and end with an alphanumeric character. The underscore character (_) is not supported.
```
cpd-cli backup-restore volume-restore create --from-backup backup_name -n zen <restore_name_identifier>
```
3. If the restore fails because there are completed or failed jobs, or pods, that reference PVCs, delete them, and rerun the restore command.
  
  Tip: Consider saving the job/pod yaml before you manually delete them, or include the --cleanup-completed-resources option in the restore command.
4. If the restore does not automatically scale up resources because of a previous failure, manually scale up resources.
```
cpd-cli backup-restore unquiesce -n zen --ignore-hooks
```

To check the status of a restore job, run the following command:

cpd-cli backup-restore volume-restore status -n zen Restore_name_identifier

To view a list of existing volume restores, run the following command:
```
cpd-cli backup-restore volume-restore list -n zen
```

To get the logs of a volume restore, run the following command:

cpd-cli backup-restore volume-restore logs -n zen Restore_name_identifier

Optional: After the volume restore is complete, clean up cpd-cli backup-restore (delete the cpd-cli backup-restore deployment and other metadata) by running the following command:
```
cpd-cli backup-restore reset -n zen --force
```