Restoring an online backup to the same cluster with the Cloud Pak for Data OADP backup and restore utility

1. Check for problems that will prevent a successful restore:

   cpd-cli oadp restore precheck \
   --backup-names=<backup_name1,backup_name2>

   This command checks for problems such as:

   • The OADP project does not exist.
   • A Velero instance is not running or its pod is not healthy.
   • The OADP backup storage location custom resource is not in the Available state.
   • The specified backup name does not exist.
   • The backup did not complete successfully or has errors or warnings.
   • Projects that are associated with the backup exist.

   For the complete list of checks that the command performs, see oadp restore precheck.

2. Restore Cloud Pak for Data Persistent Volume Claims (PVCs) and volume data, providing the volume backup name and specifying a restore name:

   cpd-cli oadp restore create <restore_name1> \
   --from-backup=<volume_backup_name> \
   --skip-hooks \
   --log-level=debug \
   --verbose

   Tip: If you are restoring Db2® or Watson Knowledge Catalog, add --scale-wait-timeout 15m to ensure that the restore command completes successfully.

Steps to restore the operator

3. Restore Kubernetes resource definitions, projects, OperatorGroups, and permissions, providing the Kubernetes resources backup name and specifying a restore name:

   cpd-cli oadp restore create <restore_name2> \
   --from-backup=<kub_resource_backup> \
   --include-resources='namespaces,operatorgroups,roles,rolebindings,serviceaccounts,securitycontextconstraints.security.openshift.io' \
   --include-cluster-resources=true \
   --skip-hooks \
   --log-level=debug \
   --verbose

4. Restore Kubernetes CommonService custom resources, providing the Kubernetes resources backup name and specifying a restore name:

   cpd-cli oadp restore create <restore_name3> \
   --from-backup=<kub_resource_backup> \
   --include-resources='namespacescopes,commonservices' \
   --include-cluster-resources=true \
   --skip-hooks \
   --log-level=debug \
   --verbose
   

   Note: If this command results in a PartiallyFailed restore status with an error such as in the following example, the error indicates that you are attempting the restore on a cluster where the Cloud Pak for Data operators visibility to the Cloud Pak for Data operand projects is not yet completed.

   Errors:
     Velero:     <none>
     Cluster:    <none>
     Namespaces:
       cpd-instance:  error restoring commonservices.operator.ibm.com/cpd-instance/common-service: admission webhook "ibm-common-service-validating-webhook.operator.ibm.com" denied the request: Services Namespace: cpd-instance is not allowed to be configured in namespace cpd-instance
   

5. Restore the Kubernetes cpd-operators configmap resource, providing the Kubernetes resources backup name and specifying a restore name:

   cpd-cli oadp restore create <restore_name4> \
   --from-backup=<kub_resource_backup> \
   --include-resources='configmaps' \
   --selector 'app=cpd-operators-backup' \
   --skip-hooks \
   --log-level=debug \
   --verbose

6. Restore (recreate) operators:

   ./cpd-operators.sh restore \
   --foundation-namespace ${PROJECT_CPD_INST_OPERATORS} \
   --operators-namespace ${PROJECT_CPD_INST_OPERATORS}

   Note: IBM Cloud Pak foundational services operators are installed in the same namespace as Cloud Pak for Data operators.

   This step restores the Operator CatalogSource, Subscription, OperandRequest, and NamespaceScope custom resources, which must be validated before you can continue with the restore process.

7. Validate operator subscriptions and visibility.
   1. Query the status of the custom resources in the Cloud Pak for Data operator and operand projects:

      oc get sub -n ${PROJECT_CPD_INST_OPERATORS} -o jsonpath="{range .items[*]}{.metadata.name}{' - currentCSV: '}{.status.currentCSV}{' - installedCSV: '}{.status.installedCSV}{'\n'}{end}"
      oc get operandrequest -A

   2. Validate the restore of the Cloud Pak for Data operators visibility:

      oc get nss -n ${PROJECT_CPD_INST_OPERATORS} common-service -o jsonpath='{.status.validatedMembers} {"\n"}' 

      The output of this command must be an array that contains the ${PROJECT_CPD_INST_OPERATORS} and all operand projects.

Steps for restoring operands

8. Restore the remaining Kubernetes custom resources, providing the Kubernetes resources backup name and specifying a restore name:
   Tip: Many of these Kubernetes resources were created in the restore (recreate) operators step.

   cpd-cli oadp restore create <restore_name5> \
   --from-backup=<kub_resource_backup> \
   --include-resources='secrets,configmaps,certificates.cert-manager.io,certificates.certmanager.k8s.io,issuers.cert-manager.io,issuers.certmanager.k8s.io,clusters.postgresql.k8s.enterprisedb.io' \
   --include-cluster-resources=true \
   --skip-hooks \
   --log-level=debug \
   --verbose

   This step restores the clusters.postgresql.k8s.enterprisedb.io custom resources in the Cloud Pak for Data instance project while the Cloud Pak for Data operators have visibility to the Cloud Pak for Data operand projects. Consequently, EDB Postgres clusters are recovered from the restored fenced off PVC and volume. EDB Postgres cluster recovery must reach a healthy state before proceeding with the later step on isolating operators from Cloud Pak for Data instance projects.

9. Isolate operators from Cloud Pak for Data instance projects:

   ./cpd-operators.sh isolate-namespacescope \
   --foundation-namespace ${PROJECT_CPD_INST_OPERATORS} \
   --operators-namespace ${PROJECT_CPD_INST_OPERATORS} 

10. Restore remaining Kubernetes custom resources except for resources that generate pods, providing the Kubernetes resources backup name and specifying a restore name:

    cpd-cli oadp restore create <restore_name6> \
    --from-backup=<kub_resource_backup> \
    --exclude-resources='persistentvolumeclaims,operatorgroups,roles,rolebindings,serviceaccounts,secrets,configmaps,certificates.cert-manager.io,certificates.certmanager.k8s.io,issuers.cert-manager.io,issuers.certmanager.k8s.io,namespacescopes,commonservices,clusters.postgresql.k8s.enterprisedb.io,deploy,rs,dc,rc,sts,ds,cj,jobs,controllerrevisions,securitycontextconstraints.security.openshift.io' \
    --include-cluster-resources=true \
    --skip-hooks \
    --log-level=debug \
    --verbose

11. Restore Kubernetes resources that generate pods, providing the Kubernetes resources backup name and specifying a restore name:

    cpd-cli oadp restore create <restore_name7> \
    --from-backup=<kub_resource_backup>  \
    --include-resources='namespaces,deploy,rs,dc,rc,sts,ds,cj,jobs,controllerrevisions' \
    --preworkloadhooks=true \
    --posthooks=true \
    --log-level=debug \
    --verbose

    Note: This command outputs messages to check the workload status. However, some workloads will not be ready until you do the next step.
12. Restore operators visibility to Cloud Pak for Data instance projects:

    ./cpd-operators.sh restore-namespacescope \
    --foundation-namespace ${PROJECT_CPD_INST_OPERATORS} \
    --operators-namespace ${PROJECT_CPD_INST_OPERATORS} 

Steps to manage restores

13. To check the status of a restore, run the following command:

    cpd-cli oadp restore status <restore_name> \
    --details

14. To view a list of existing restores, run the following command:

    cpd-cli oadp restore list

15. To view restore logs, run the following command:

    cpd-cli oadp restore logs <restore_name>

16. To delete a restore for cleanup purposes, run the following command:

    cpd-cli oadp restore delete <restore_name>