Backing up and restoring Db2 Warehouse separately from Cloud Pak for Data

You can create an offline backup of a Db2® Warehouse instance separately from Cloud Pak for Data and restore it to the same cluster with the Cloud Pak for Data OpenShift® APIs for Data Protection (OADP) backup and restore utility.

Back up and restore a Db2 Warehouse instance separately from Cloud Pak for Data when Db2 Warehouse backups are very large, or you want to back up Db2 Warehouse more frequently than Cloud Pak for Data.

Backing up and restoring Db2 Warehouse and Cloud Pak for Data separately involves the following tasks:

  1. Prepare to backup Cloud Pak for Data and Db2 Warehouse.
  2. Back up Cloud Pak for Data, excluding Db2 Warehouse.
  3. Back up Db2 Warehouse.
  4. Restore Cloud Pak for Data and Db2 Warehouse.
Best practice: You can run the commands in this task exactly as written if you set up environment variables. For instructions, see Setting up installation environment variables.

Ensure that you source the environment variables before you run the commands in this task.

Preparing for the backup

To prepare for the backup, check the state of nodes, pods, and custom resources. Then prepare to exclude Db2 Warehouse from a Cloud Pak for Data backup by labeling PVCs, PVs, snapshots, and pod template in StatefulSets.

  1. Check that all nodes are in the Ready state:
    oc get nodes
  2. Check that all Db2 Warehouse pods are running in 1/1:
    oc get pod | grep c-db2wh
  3. Check the status of the Db2 Warehouse service.
    1. Identify the db2ucluster instance:
      oc get db2ucluster | grep db2wh

      Keep track of the db2ucluster ID, as it is used in some of the following steps.

    2. Open a remote shell to the container.
      oc rsh c-db2wh-<xxxxxxxxxxxxx>-db2u-0
    3. Switch to the database instance owner db2inst1:
      su - db2inst1
    4. Check that all partitions are running:
      rah 'ps -fea' 2>&1 | grep db2sysc | grep -v grep
    5. Connect to the database:
      db2 connect to mydb
    6. Check if you can select from a table by running a simple query:
      db2 select name from sysibm.systables
  4. Check that the status of custom resources is Completed or Ready.
    1. Check that the status of the IBM Cloud Pak® for Data control plane custom resource is Completed state:
      oc describe ZenService lite-cr | grep "Zen Status"

      If the status is Failed and the user-home-pvc PVC was increased in size, run the following commands:

      oc project ${PROJECT_CPFS_OPS}
oc get pod | grep zen
oc rsh ibm-zen-operator-xxxxxxxx-xxxx
vi 4.4.3/roles/0010-infra/tasks/main.yml

      Then comment with # the sections Create user-home-pvc PVC and Wait until the user-home-pvc PVC is bound.

    2. Check that the STATE of the db2ucluster custom resources are in a Ready state:
      oc get db2ucluster
      Note: If any of the custom resources are in InMaintenance state, and the db2ucluster is working (for example, you can connect to the databases and they are not in write-suspend mode), delete the following line in the yaml file:
      db2u.databases.ibm.com/maintenance-pause-reconcile: 'true'
    3. Check that the status of custom resources in the Cloud Pak for Data project is Completed:
      cpd-cli manage get-cr-status \
--cpd_instance_ns=${PROJECT_CPD_INSTANCE}
  5. Label Db2 Warehouse PVCs, PVs, and snapshots to exclude them from the Cloud Pak for Data backup.
    1. Label PVCs:
      oc label pvc `oc get pvc --show-labels | grep db2wh | awk '{print $1}'` velero.io/exclude-from-backup=true
    2. Label PVs:
      oc label pv `oc get pv --show-labels | grep db2wh | awk '{print $1}'` velero.io/exclude-from-backup=true
    3. Label snapshots:
      oc label volumesnapshots `oc get volumesnapshots  --show-labels | grep db2wh | awk '{print $1}'` velero.io/exclude-from-backup=true
  6. Label pod templates in StatefulSets.
    1. Identify StatefulSets:
      oc get sts -l app=${DB2UCLUSTER}
    2. Take note of original replicas, and scale down db2ucluster:
      oc scale sts c-${DB2UCLUSTER}-db2u --replicas=0
    3. Review the pod status:
      watch "oc get pod |grep  c-db2wh-<xxxxxxxxxxx>"
    4. Take note of original replicas, and scale down etcd:
      oc scale sts c-${DB2UCLUSTER}-etcd --replicas=0
    5. Review the pod status:
      watch "oc get pod |grep  c-db2wh-<xxxxxxxxxxx>"
    6. Take note of original replicas, and scale down tools:
      oc scale deploy c-${DB2UCLUSTER}-tools --replicas=0
    7. Review the pod status:
      watch "oc get pod |grep  c-db2wh-<xxxxxxxxxxx>"
    8. When Db2 Warehouse is scaled down, label the Pod Templates in StatefulSets and Deployments:
      oc patch StatefulSet c-db2wh-xxxxxxxxxxxxx-db2u --patch '{"spec": {"template": {"metadata": {"labels": {"velero.io/exclude-from-backup": "true"}}}}}'
oc patch StatefulSet c-db2wh-xxxxxxxxxxxxx-etcd --patch '{"spec": {"template": {"metadata": {"labels": {"velero.io/exclude-from-backup": "true"}}}}}'
oc patch deploy c-db2wh-xxxxxxxxxxx-tools --patch '{"spec": {"template": {"metadata": {"labels": {"velero.io/exclude-from-backup": "true"}}}}}'

      oc patch StatefulSet c-db2wh-xxxxxxxxxxxxx-etcd --patch '{"spec": {"template": {"metadata": {"labels": {"velero.io/exclude-from-backup": "true"}}}}}'
      oc patch deploy c-db2wh-xxxxxxxxxxx-tools --patch '{"spec": {"template": {"metadata": {"labels": {"velero.io/exclude-from-backup": "true"}}}}}'
    9. Scale up to original replicas:
      oc scale deploy c-${DB2UCLUSTER}-tools --replicas=1
      oc scale sts c-${DB2UCLUSTER}-etcd --replicas=<original-number-of-replicas>
      oc scale sts c-${DB2UCLUSTER}-db2u --replicas=<original-number-of-replicas>
    10. Review the pod status:
      watch "oc get pod |grep  c-db2wh-<xxxxxxxxxxx>"
    11. Repeat step 3 to check the status of the Db2 Warehouse service.

Backing up Cloud Pak for Data

Back up Cloud Pak for Data with Db2 Warehouse excluded by doing the following steps.

  1. Quiesce the IBM Cloud Pak for Data control plane.
    cpd-cli oadp backup prehooks \
--include-namespaces=${PROJECT_CPD_INSTANCE}
  2. Create the backup.
    cpd-cli oadp backup create <cpd_backup_name> \
--include-namespaces cpd \
--exclude-resources='Event,Event.events.k8s.io' \
--default-volumes-to-restic \
--cleanup-completed-resources \
--log-level=debug \
--verbose \
--skip-hooks \
--snapshot-volumes=false

    In an environment that uses a private container registry, such as an air-gapped cluster, additionally specify the appropriate --image-prefix.

    The cluster can pull images from the IBM Entitled Registry
    Restriction: This option is available only if the cluster can connect to the internet.
    cpd-cli oadp backup create <cpd_backup_name> \
--include-namespaces cpd \
--exclude-resources='Event,Event.events.k8s.io' \
--default-volumes-to-restic \
--cleanup-completed-resources \
--log-level=debug \
--verbose \
--skip-hooks \
--image-prefix=registry.redhat.io/ubi8 \
--snapshot-volumes=false
    The cluster pulls images from a private container registry
    Restriction: This option is available only if an administrator moved the backup and restore utility images to a private container registry.
    cpd-cli oadp backup create <cpd_backup_name> \
--include-namespaces cpd \
--exclude-resources='Event,Event.events.k8s.io' \
--default-volumes-to-restic \
--cleanup-completed-resources \
--log-level=debug \
--verbose \
--skip-hooks \
--image-prefix=${PRIVATE_REGISTRY_LOCATION} \
--snapshot-volumes=false
  3. To review the status of the backup, run the following command.
    cpd-cli oadp backup status <cpd_backup_name> \
--details
  4. To review the backup log, run the following command.
    cpd-cli oadp backup logs <cpd_backup_name>
  5. If the backup failed, delete it, and then restart the backup.
    cpd-cli oadp backup delete <cpd_backup_name>
  6. Unquiesce the IBM Cloud Pak for Data control plane.
    ./cpd-cli oadp backup posthooks \
--include-namespaces=${PROJECT_CPD_INSTANCE}
    Note: If the wkc-job-unquiesce job is taking a long time to complete, review the pod logs. If the wkf-workflow service pod is in 0/1, restart it. Then the wkc-job-unquiesce job continues.

Backing up Db2 Warehouse

Back up Db2 Warehouse by creating a Container Storage Interface (CSI) volume snapshot.

  1. Run the Db2 Warehouse pre-backup hook that invokes Db2 write-suspend.
    1. Identify the db2ucluster:
      oc get db2ucluster | grep db2wh

      The command returns DB2UCLUSTER=db2wh-<xxxxxxxxxxxxxxxx>

    2. Identify the catalog node:
      CATALOG_POD=$(oc get po -l name=dashmpp-head-0,app=${DB2UCLUSTER} --no-headers | awk '{print $1}')
    3. Suspend the Db2 Warehouse database on the pod that was identified in the previous step:
      oc rsh ${CATALOG_POD} bash
      su - db2inst1
      cd /db2u/scripts
      ./manage_snapshots.sh --action suspend
  2. Apply Db2 Warehouse instance-specific resource label to Db2 instance PVCs and resources (meta and backup PVCs are not currently labeled).
    INSTANCE_ID=$(echo ${DB2UCLUSTER} | cut -d "-" -f2)
    oc label pvc c-${DB2UCLUSTER}-meta app=${DB2UCLUSTER} component=db2wh db2u/cpdbr=db2u formation_id=${DB2UCLUSTER} cpdsupport/addOnId=db2wh icpdsupport/app=${DB2UCLUSTER} icpdsupport/createdBy=1000330999 icpdsupport/podSelector=db2u-log icpdsupport/serviceInstanceId=${INSTANCE_ID} role=db type=engine
    oc label pvc c-${DB2UCLUSTER}-backup app=${DB2UCLUSTER} component=db2wh db2u/cpdbr=db2u formation_id=${DB2UCLUSTER} icpdsupport/addOnId=db2wh icpdsupport/app=${DB2UCLUSTER} icpdsupport/createdBy=1000330999 icpdsupport/podSelector=db2u-log icpdsupport/serviceInstanceId=${INSTANCE_ID} role=db type=engine
  3. Create the backup.
    cpd-cli oadp backup create <db2wh_backup_name> \
--include-namespaces=${PROJECT_CPD_INSTANCE} \
--skip-hooks=true \
--snapshot-volumes \
--include-resources='ns,pvc,pv,volumesnapshot,volumesnapshotcontent' \
--selector="app=${DB2UCLUSTER}"
  4. Resume the Db2 Warehouse database on the pod (same pod as in step 1):
    oc rsh ${CATALOG_POD} bash
    su - db2inst1
    cd /db2u/scripts
    ./manage_snapshots.sh --action resume
  5. Check the status of Db2 Warehouse.

Restoring Cloud Pak for Data and Db2 Warehouse

Restore Cloud Pak for Data and Db2 Warehouse by doing the following steps.

  1. Note down the following values in the original Cloud Pak for Data project (namespace) yaml:
    • openshift.io/sa.scc.supplemental-groups: <xxxxxxxxxx/xxxxx>
    • openshift.io/sa.scc.uid-range: <xxxxxxxxxx/xxxxx>
    • openshift.io/requester: <requester_id>
    • openshift.io/sa.scc.mcs: '<xx>:<xxx>,<xxx>'
  2. Delete the Cloud Pak for Data instance.
  3. Restore the Db2 Warehouse volumes:
    cpd-cli oadp restore create <db2wh_restore_name> \
--from-backup=<db2wh_backup_name> \
--skip-hooks=true
  4. Check that the restore is completed:
    cpd-cli oadp restore list
  5. Change the project uid.
    1. Edit the Cloud Pak for Data project:
      oc edit namespace ${PROJECT_CPD_INSTANCE}
    2. Change the following values to the original values from step 1:
      • openshift.io/sa.scc.supplemental-groups: <xxxxxxxxxx/xxxxx>
      • openshift.io/sa.scc.uid-range: <xxxxxxxxxx/xxxxx>
      • openshift.io/requester: <requester_id>
      • openshift.io/sa.scc.mcs: '<xx>:<xxx>,<xxx>'
      Important: If you do not change these values, Cloud Pak for Data pods won't successfully initialize.
  6. Repeat the previous step in the ${PROJECT_CPD_OPS} project.
  7. Restore Cloud Pak for Data control plane services:
    cpd-cli oadp restore create <cpd_restore_name> \
--from-backup=<cpd_backup_name> \
--skip-hooks=true \
--include-resources='namespaces,zenservices,secrets,certificates.cert-manager.io,certificates.certmanager.k8s.io,issuers.cert-manager.io,issuers.certmanager.k8s.io' \
--log-level=debug \
--verbose
  8. View the status of the restore:
    cpd-cli oadp restore status <cpd_restore_name> \
--details
  9. Restore Cloud Pak for Data services:
    cpd-cli oadp restore create <cpd_restore_name> \
--from-backup=<cpd_backup_name> \
--exclude-resources='ImageTag,clients' \
--include-cluster-resources=true \
--scale-wait-timeout 30m \
--log-level=debug \
--verbose

    In an environment that uses a private container registry, such as an air-gapped cluster, additionally specify the appropriate --image-prefix.

    The cluster can pull images from the IBM Entitled Registry
    Restriction: This option is available only if the cluster can connect to the internet.
    cpd-cli oadp restore create <cpd_restore_name> \
--from-backup=<cpd_backup_name> \
--exclude-resources='ImageTag,clients' \
--include-cluster-resources=true \
--image-prefix=registry.redhat.io/ubi8 \
--scale-wait-timeout 30m \
--log-level=debug \
--verbose
    The cluster pulls images from a private container registry
    Restriction: This option is available only if an administrator moved the backup and restore utility images to a private container registry.
    cpd-cli oadp restore create <cpd_restore_name> \
--from-backup=<cpd_backup_name> \
--exclude-resources='ImageTag,clients' \
--include-cluster-resources=true \
--image-prefix=${PRIVATE_REGISTRY_LOCATION} \
--scale-wait-timeout 30m \
--log-level=debug \
--verbose
  10. View the status of the restore:
    cpd-cli oadp restore status <cpd_restore_name> \
--details