Creating an offline backup of a Cloud Pak for Data instance and restoring it on the same cluster

You can create an offline backup of an IBM Cloud Pak® for Data instance and restore it on the same cluster with the Cloud Pak for Data OpenShift® APIs for Data Protection (OADP) backup and restore utility.

A Cloud Pak for Data instance is comprised of one or more Red Hat® OpenShift Container Platform projects (Kubernetes namespaces). For example, your Cloud Pak for Data instance can be either:
  • A single project in which the control plane and services are installed.
  • A central project in which the control plane is installed and one or more tethered projects

If your deployment consists of multiple instances of Cloud Pak for Data on the same cluster, you can back up and restore each instance separately.

To back up a Cloud Pak for Data instance and restore it on the same cluster, complete the following high-level steps:

  1. Back up the instance of Cloud Pak for Data on the source cluster.
  2. Delete the instance of Cloud Pak for Data on the source cluster.
  3. Restore the instance of Cloud Pak for Data on the same cluster.

Before you begin

Ensure that the following operators are installed on the cluster:
  • IBM Cloud Pak foundational services operators
  • IBM Cloud Pak for Data platform operator and service operators
In the following sections, some steps must be done only when you performed a specialized installation. In a specialized installation, the IBM Cloud Pak foundational services operators and the IBM Cloud Pak for Data platform operator and service operators are installed in different projects. For example:
  • The IBM Cloud Pak foundational services operators are typically installed in ibm-common-services.
  • The IBM Cloud Pak for Data platform operator and service operators are installed in a different project, such as cpd-operators.
Permissions you need for this task
You must log on as a user with cluster administrator rights.
Best practice: You can run the commands in this task exactly as written if you set up environment variables. For instructions, see Setting up installation environment variables.

Ensure that you source the environment variables before you run the commands in this task.

Backing up an instance of Cloud Pak for Data on the source cluster

To back up an instance of Cloud Pak for Data, do the following steps:

  1. Complete backup prerequisite tasks.
  2. Create the backup by running the following command.

    If your instance is comprised of a central project and one or more tethered projects, replace the ${PROJECT_CPD_INSTANCE} environment variable with a comma separated list of projects. For example: --include-namespaces=cpd-instance,cpd-instance-tether1,cpd-instance-tether2.

    cpd-cli oadp backup create <instance_backup_name> \
    --include-namespaces=${PROJECT_CPD_INSTANCE} \
    --exclude-resources='Event,Event.events.k8s.io' \
    --default-volumes-to-restic \
    --cleanup-completed-resources \
    --log-level=debug \
    --verbose
  3. Optional: In case you later need to restore them, back up the operators.
    Tip: The steps to restore the operators in the same cluster are the same as restoring them on a different cluster.

Deleting an instance of Cloud Pak for Data on the source cluster

Before you restore a backup, delete the Cloud Pak for Data instance on the source cluster.

Note: Resources in the Cloud Pak for Data instance are watched and managed by operators and controllers that run in other projects. More steps are needed to delete the Cloud Pak for Data instance to prevent corruption or out of sync operators and resources. These steps involve locating Kubernetes resources that have finalizers specified in metadata, and removing those finalizers before deleting the Cloud Pak for Data instance.
  1. If the Cloud Pak for Data instance is integrated with the IAM service, delete clients in the Cloud Pak for Data instance.
    1. Log in to Red Hat OpenShift Container Platform as a user with sufficient permissions to complete the task:
      oc login ${OCP_URL}
    2. If the Cloud Pak for Data instance was configured with iamintegration: true, delete clients in Cloud Pak for Data projects by running the following command:
      oc delete client -n ${PROJECT_CPD_INSTANCE} --all
      Note: If the IBM Cloud Pak foundational services IAM OIDC is not watching the project where the Cloud Pak for Data instance is installed, you must do a manual cleanup by running the following commands:
      # Get Client name
      oc get client -n ${PROJECT_CPD_INSTANCE}
      # Delete metadata.finalizers from Client
      oc patch client <client_name> -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge
      # Delete Clients
      oc delete client -n ${PROJECT_CPD_INSTANCE} --all
      # cpd-oidcclient-secret Secret
      oc delete secret cpd-oidcclient-secret -n ${PROJECT_CPD_INSTANCE}
  2. Delete service-specific finalizers from service custom resources (CRs), and delete the service CRs in Cloud Pak for Data instance projects.
    Note: The following commands are only examples of deployed services. To get an inventory of all deployed services, including their status, see Check the status of installed services.
    # Get CCS CR, delete finalizers and delete CR
    oc get ccs -n ${PROJECT_CPD_INSTANCE}
    oc patch ccs <ccs_name> -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge
    oc delete ccs -n ${PROJECT_CPD_INSTANCE} --all
    
    # Get DataStage CR, delete finalizers and delete CR
    oc get PXRuntime -n ${PROJECT_CPD_INSTANCE}
    oc patch PXRuntime <PXRuntime_name> -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge
    oc delete PXRuntime -n ${PROJECT_CPD_INSTANCE} --all
    oc get DataStage -n ${PROJECT_CPD_INSTANCE}
    oc patch DataStage <DataStage_name> -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge
    oc delete DataStage -n ${PROJECT_CPD_INSTANCE} --all
        
    # Get IIS CR, delete finalizers and delete CR
    oc get iis  -n ${PROJECT_CPD_INSTANCE}
    oc patch iis <iis_cr_name>  -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge
    oc delete iis -n ${PROJECT_CPD_INSTANCE} --all
    
    # Get wKC CR, delete finalizers and delete CR
    oc get wkc -n ${PROJECT_CPD_INSTANCE}
    oc patch wkc <wkc_cr_name>  -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge
    oc delete wkc -n ${PROJECT_CPD_INSTANCE} --all
    
    # Get UG CR, delete finalizers and delete CR
    oc get ug ug-cr -n ${PROJECT_CPD_INSTANCE}
    oc patch ug ug-cr -n ${PROJECT_CPD_INSTANCE}  -p '{"metadata":{"finalizers":[]}}' --type=merge
    oc delete ug -n ${PROJECT_CPD_INSTANCE} --all
    
    # Get Db2assservice CR, delete finalizers and delete CR
    oc get Db2aaserviceService -n ${PROJECT_CPD_INSTANCE}
    oc patch  Db2aaserviceService <db2asservice_cr_name> -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge
    oc delete Db2aaserviceService  -n ${PROJECT_CPD_INSTANCE} --all
  3. Delete finalizers from Zen-Service OperandRequest in the Cloud Pak for Data instance projects.
    1. Get the Zen-Service OperandRequest name, which typically is zen-service:
      oc get operandrequest -n ${PROJECT_CPD_INSTANCE}
    2. Delete the finalizers:
      oc patch operandrequest <zen_service_operandrequest_name> -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge
  4. Delete OperandRequests in Cloud Pak for Data instance projects:
    oc delete operandrequests -n ${PROJECT_CPD_INSTANCE} --all
  5. Delete ZenServices in Cloud Pak for Data instance projects:
    oc delete zenservice -n ${PROJECT_CPD_INSTANCE} --all
  6. Delete finalizers from the admin RoleBinding in Cloud Pak for Data instance projects:
    oc patch rolebinding admin -n ${PROJECT_CPD_INSTANCE} -p '{"metadata":{"finalizers":[]}}' --type=merge
  7. Specialized installations only: Remove Cloud Pak for Data instance projects from the nss-cpd-operators NamespaceScope custom resource in the cpd-operators project by doing one of the following choices.
    • Remove the Cloud Pak for Data instance projects by editing the nss-cpd-operators NamespaceScope resource:
      oc edit namespacescope nss-cpd-operators -n ${PROJECT_CPD_OPS}
      # Remove ${PROJECT_CPD_OPS} Namespace from namespaceMembers field
    • Retrieve the existing projects in the namespaceMembers list and then remove the projects that comprise the Cloud Pak for Data instance:
      # Retrieve existing namespaceMembers field
      oc get namespacescope nss-cpd-operators -n ${PROJECT_CPD_OPS} -o jsonpath={.spec.namespaceMembers}
      # Remove ${PROJECT_CPD_INSTANCE} Namespace from namespaceMembers field
      oc patch namespacescope nss-cpd-operators -n ${PROJECT_CPD_OPS} -p $'{"spec":{"namespaceMembers":["<remaining-namespaces"]}}' --type=merge
  8. Delete the Cloud Pak for Data instance projects, and remove any remaining finalizers.
    1. To delete the Cloud Pak for Data instance project, run the following command:
      oc delete project ${PROJECT_CPD_INSTANCE}
    2. Confirm the project delete and check for status messages of type NamespaceFinalizersRemaining:
      oc get project ${PROJECT_CPD_INSTANCE} -o jsonpath="{.status}"
    3. If there are finalizers remaining, repeat these substeps to locate resources and delete finalizers.
    Note: Even though the command returns quickly, the project deletion can take several minutes to fully complete. Continue to watch the final oc get project command to wait for successful completion.

Restoring the instance of Cloud Pak for Data on the same cluster

After you delete Cloud Pak for Data instance on the source cluster, restore the backup.

Note: Do not attempt to restore a project that is in a terminating state. If a project is in a terminating state, do step 8 in Deleting an instance of Cloud Pak for Data on the source cluster to identify and remove all remaining finalizers.
  1. Log in to Red Hat OpenShift Container Platform as a user with sufficient permissions to complete the task:
    oc login ${OCP_URL}
  2. Specialized installations only: Enable the IBM Cloud Pak for Data platform operator to watch the project where you are restoring the instance of Cloud Pak for Data.

    Update the nss-cpd-operators NamespaceScope resource in the Cloud Pak for Data operator project to watch the project where you are restoring the instance.

    Tip: To retrieve the existing projects in the namespaceMembers list, run the following command. For example, if Cloud Pak for Data operators are in the cpd-operators project, run:
    oc get namespacescope nss-cpd-operators -n cpd-operators -o jsonpath={.spec.namespaceMembers}

    Edit the namespaceMembers list to add the project where you are restoring the instance. For example, if you are restoring the instance to the cpd-instance project, add that project to the list. If your instance is comprised of a central project and one or more tethered projects, you must add all of those projects to the list. For example, cpd-instance-tether1, cpd-instance-tether2.

    cat <<EOF |oc apply -f -
    apiVersion: operator.ibm.com/v1
    kind: NamespaceScope
    metadata:
      name: nss-cpd-operators
      namespace: ${PROJECT_CPD_OPS}         
    spec:
      csvInjector:                    # This setting is required for some services. Do not delete this line if you specified it when you created operator subscriptions. 
        enable: true                  # This setting is required for some services. Do not delete this line if you specified it when you created operator subscriptions. 
      namespaceMembers:
      - ${PROJECT_CPD_OPS}                 
      - ${PROJECT_CPD_INSTANCE}
    EOF
  3. To view a list of existing backups, run the following command:
    cpd-cli oadp backup ls
  4. Restore the Cloud Pak for Data instance.
    1. Restore the ZenService resource:
      cpd-cli oadp restore create <instance_backup_name>-zenservice-restore \
      --from-backup=<instance_backup_name> \
      --include-resources='namespaces,zenservices,secrets,certificates.cert-manager.io,certificates.certmanager.k8s.io,issuers.cert-manager.io,issuers.certmanager.k8s.io' \
      --skip-hooks \
      --log-level=debug \
      --verbose
    2. Restore all other backed up resources:
      cpd-cli oadp restore create <instance_backup_name>-restore \
      --from-backup=<instance_backup_name> \
      --exclude-resources='clients,ImageTag' \
      --include-cluster-resources=true \
      --log-level=debug \
      --verbose
      Note: If you are restoring Db2® or Watson™ Knowledge Catalog, add --scale-wait-timeout 15m to ensure that the restore command completes successfully.