IBM Cloud Pak foundational services backup and restore with the IBM Spectrum Fusion

You can use the Spectrum Fusion to backup and restore the foundational services data to the cloud storage and network-attached storage (NAS) devices. The downtime is reduced and maximizes your recovery time objective (RTO).

You need to complete the following procedures to backup and restore with the IBM Spectrum Fusion:

• Prerequisites
• Installing the IBM Spectrum Fusion
• Installing the Backup and Restore service
• Preparing cluster for backup
• Creating IBM Spectrum Fusion custom resource
• Creating a backup
  • Verifying successful backup
• Creating a restore
  • Verifying successful restore

Prerequisites

Complete the following steps:

1. Ensure that your cluster runs on Red Hat OpenShift version 4.12, 4.14, or 4.15.

2. Install the Red Hat OpenShift Data Foundation or IBM Spectrum Scale storage class in your cluster. For more information, see Storage options.

3. Ensure that the VolumeSnapshotClass is created for your cluster. The VolumeSnapshotClass is created automatically when you install the Red Hat OpenShift Data Foundation.

4. Ensure that the pull-secret and entitlement key are created with the required access for your cluster. If the secrets are not created, see Creating image pull secret for IBM Cloud® based installation.

5. Set up any Amazon S3-compatible storage. For example, you can create a bucket in IBM Cloud Object Storage. For more information, see IBM Cloud Object Storage .

6. Add the hash-based message authentication code (HMAC) when you add a service credential to the bucket. For more information, see Service credentials . From the Cloud Object Storage navigation menu, gather the following information:

   • access key id, which can be found on the Service credentials page that is associated with the bucket.
   • secret access key, which can be found on the Service credentials page that is associated with the bucket.
   • bucket name, which can be found on the Buckets page.
   • bucket region, which can be found on the Buckets page.
   • root directory name, which is the path in the bucket where you want to store the backups.

   • s3 URL, which is the endpoint URL of the bucket.

     • Note: The endpoint URL must start with http:// or https://.

Installing the IBM Spectrum Fusion

Install the IBM Spectrum Fusion version 2.8.0 in your cluster. For more information, see Installing IBM Storage Fusion on IBM Cloud.

Installing the Backup and Restore service

To install the Backup and Restore service, complete the following steps:

1. Login to the OpenShift Container Platform console with the username and password.

2. Go to Networking > Routes and select the ibm-spectrum-fusion-ns project. If you installed the IBM Spectrum Fusion version 2.8.0 in the custom namespace, select the namespace where you deployed the IBM Spectrum Fusion in the Routes section.

3. Go to Services page in the IBM Storage Fusion user interface.

4. Install the Backup and Restore (hub) in the backup and source cluster. For more information, see Backup and Restore hub.

5. Install the Backup and Restore Agent (spoke) in the restore and target cluster. For more information, see Backup and Restore spoke.

6. Connect the Backup and Restore clusters. For more information, see Establishing the connection between hub and spoke.

Creating IBM Spectrum Fusion custom resource

To create the IBM Spectrum Fusion custom resource, complete the following steps:

1. Download the Application, PolicyAssignment, Policy resources.

   wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/application.yaml
   wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/policy_assignment.yaml
   wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/policy.yaml
   

2. Create the backup location. For more information, see Adding a backup storage location.

   You can use the values of your Amazon S3 compatible storage.

3. Edit the Application custom resource to add the namespaces in the includedNamespaces parameter that need to be included in the backup. For more information, see Applications.

   apiVersion: application.isf.ibm.com/v1alpha1
   kind: Application
   metadata:
     name: cs-application
     namespace: ibm-spectrum-fusion-ns
   spec:
     enableDR: false
     includedNamespaces:
       - <operator-namespace>
       - <service-namespace
       - <tenant-namespace-1>
       - <tenant-namespace-2>
       - <cert manager namespace>
       - <licensing namespace>
       - openshift-marketplace    
       - openshift-config
       - kube-public
   

   Ensure that you specify the namespaces that need to be backed up and restored for a tenant to run properly. You can back up and restore the resources in the entire cluster at once with the Application custom resource.

   If you installed the IBM Spectrum Fusion version 2.8.0 in the custom namespace, replace ibm-spectrum-fusion-ns in the metadata.namespace parameter with the namespace where you deployed the IBM Spectrum Fusion.

4. Edit the Backup Policy to schedule a backup. For more information, see Managing backup policy.

   apiVersion: data-protection.isf.ibm.com/v1alpha1
   kind: BackupPolicy
   metadata:
     name: cs-backup-policy
   spec:
     backupStorageLocation: <storage_location>
     provider: isf-backup-restore
     retention:
       number: 5
       unit: days
     schedule:
       cron: '00 0  * * * '
       timezone: America/New_York
   

   You can update the schedule.cron parameter to schedule the automatic backups in your cluster. Go to the backup location created in step 2 to obtain the backups of your cluster.

5. Download the recipe to find the resources that needs to be backed up and restored and the workflows that determine the sequence of events. You can download the recipe for the single or multiple namespaces based on your requirements.

   • For single namespace, run the following command:

     wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/recipes/4.7-example-recipe-single-ns.yaml
     

   • For multiple namespaces, run the following command:

     wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/recipes/4.7-example-recipe-multi-ns.yaml
     

   Update the following variables in the Recipe custom resource based on your Application requirements:

   • <service namespace>
   • <operator namespace>
   • <comma delimited (no spaces) list of Cloud Pak workload namespaces that use this foundational services instance>
   • <foundational services version>
   • <spec.size value from commonservice custom resource>
   • <install mode (manual or automatic)>
   • <zenservice name>

   Ensure to enter the correct parameters for the setup_tenant.sh script and check for the setup-tenant-command hook definition to update the parameter values. For more information, see Installing foundational services by using a script.

   For more information on the parameters in the Recipe custom resource, see Creating Recipe.

6. Edit the PolicyAssignment to link a recipe to an application and backup policy.

   apiVersion: data-protection.isf.ibm.com/v1alpha1
   kind: PolicyAssignment
   metadata:
     name: cs-policy-assignment
     namespace: ibm-spectrum-fusion-ns
   spec:
     application: cs-application
     backupPolicy: cs-backup-policy
     runNow: false
     recipe:
       apiVersion: spp-data-protection.isf.ibm.com/v1alpha1
       name: cs-recipe
       namespace: ibm-spectrum-fusion-ns
   

Preparing cluster for backup

1. Add labels to the downloaded resources with one of the following methods:

   • If you use single namespace, see Add labels to resources for a single instance of foundational services.

   • If you use multiple namespaces, see Add labels to resources for multiple instances of foundational services.

   You need to create a tethered role and tethered rolebdinding for each instance.

2. If you use SOD topology or multiple namespaces in your foundational services instance, complete the following steps:

   1. Download the resources.

      wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/cpfs-util-resources/cpfs-util-deployment.yaml
      wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/cpfs-util-resources/cpfs-util-role.yaml
      wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/cpfs-util-resources/cpfs-util-rolebinding.yaml
      wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/cpfs-util-resources/cpfs-util-services-role.yaml
      wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/cpfs-util-resources/cpfs-util-services-rolebinding.yaml
      wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/cpfs-util-resources/cpfs-util-sa.yaml
      wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/cpfs-util-resources/setup-tenant-job.yaml
      wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/cpfs-util-resources/setup-tenant-job-configmap.yaml
      wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/cpfs-util-resources/setup-tenant-job-pvc.yaml
      wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/cpfs-util-resources/setup-tenant-job-role.yaml
      wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/cpfs-util-resources/setup-tenant-job-rolebinding.yaml
      wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/cpfs-util-resources/setup-tenant-job-sa.yaml
      wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/cpfs-util-resources/setup-tenant-job-serv-tethered-role.yaml
      wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/spectrum-fusion/cpfs-util-resources/setup-tenant-job-serv-tethered-rolebinding.yaml
      

   2. Set the operator namespace, services or tethered namespace, and storage class name.

      export OPERATOR_NAMESPACE=<operator namespace>
      export TARGET_NAMESPACE=<service or tethered namespace>
      export STORAGE_CLASS=<storage class>
      

   3. Edit the resources to replace the variables in the downloaded files.

      sed -i -E "s/<operator namespace>/$OPERATOR_NAMESPACE/" cpfs-util-deployment.yaml
         sed -i -E "s/<operator namespace>/$OPERATOR_NAMESPACE/" cpfs-util-role.yaml
         sed -i -E "s/<operator namespace>/$OPERATOR_NAMESPACE/" cpfs-util-rolebinding.yaml
         sed -i -E "s/<services namespace>/$TARGET_NAMESPACE/" cpfs-util-services-role.yaml
         sed -i -E "s/<operator namespace>/$OPERATOR_NAMESPACE/" cpfs-util-services-rolebinding.yaml
         sed -i -E "s/<services namespace>/$TARGET_NAMESPACE/" cpfs-util-services-rolebinding.yaml
         sed -i -E "s/<operator namespace>/$OPERATOR_NAMESPACE/" cpfs-util-sa.yaml
      
         sed -i -E "s/<operator namespace>/$OPERATOR_NAMESPACE/" setup-tenant-job-configmap.yaml
         sed -i -E "s/<operator namespace>/$OPERATOR_NAMESPACE/" setup-tenant-job-pvc.yaml
         sed -i -E "s/<storage class>/$STORAGE_CLASS/" setup-tenant-job-pvc.yaml
         sed -i -E "s/<operator namespace>/$OPERATOR_NAMESPACE/" setup-tenant-job-role.yaml
         sed -i -E "s/<operator namespace>/$OPERATOR_NAMESPACE/" setup-tenant-job-rolebinding.yaml
         sed -i -E "s/<operator namespace>/$OPERATOR_NAMESPACE/" setup-tenant-job-sa.yaml
         sed -i -E "s/<services or tethered namespace>/$TARGET_NAMESPACE/" setup-tenant-job-serv-tethered-role.yaml
         sed -i -E "s/<services or tethered namespace>/$TARGET_NAMESPACE/" setup-tenant-job-serv-tethered-rolebinding.yaml
         sed -i -E "s/<operator namespace>/$OPERATOR_NAMESPACE/" setup-tenant-job-serv-tethered-rolebinding.yaml
         sed -i -E "s/<operator namespace>/$OPERATOR_NAMESPACE/" setup-tenant-job.yaml
      

   4. Apply the resources.

      oc apply -f cpfs-util-deployment.yaml -f cpfs-util-pvc.yaml -f cpfs-util-pvc.yaml -f cpfs-util-role.yaml -f cpfs-util-rolebinding.yaml -f cpfs-util-sa.yaml -f cpfs-util-br-configmap.yaml -f cpfs-util-serv-tethered-role.yaml -f cpfs-util-serv-tethered-rolebinding.yaml -f cpfs-util-serv-tethered-rolebinding.yaml -f setup-tenant-job-configmap.yaml -f setup-tenant-job-pvc.yaml -f setup-tenant-job-role.yaml -f setup-tenant-job-rolebinding.yaml -f setup-tenant-job-sa.yaml -f setup-tenant-job-serv-tethered-role.yaml -f setup-tenant-job-serv-tethered-rolebinding.yaml -f setup-tenant-job.yaml -f cpfs-util-services-role.yaml -f cpfs-util-services-rolebinding.yaml
      

      If you have multiple tethered namespaces, create the cpfs-util-serv-tethered-role.yaml and cpfs-util-serv-tethered-rolebinding.yaml resources in each tethered namespaces.

Creating a backup

To create a backup, complete the following steps:

1. Login to the Spectrum Fusion UI.
2. Go to Backup & Restore > Backed up Applications.
3. Select the ellipsis overflow menu of an application record.
4. Click the Backup now option.

5. Go Backup & Restore > Jobs and select the running job in the Jobs page to monitor the backup job status.

   The clipboard icon at the upper right of the Summary box displays logs from the Spectrum Fusion process. For more information, see Jobs.

Verifying successful backup

To verify that the backup is successful, complete the following steps:

1. Ensure that the job is completed in the Spectrum Fusion UI.

2. Verify the successful completion of common-service-db.

   1. Find cs-db-backup-<pod hash> in the services namespace (if applicable) on the Backup cluster with one of the following methods:

      • To find cs-db-backup-<pod hash> with the OpenShift Container Platform console, Go to pod and select Terminal.

      • To find cs-db-backup-<pod hash> with the CLI, run the following command:

        oc exec -it cs-db-backup-<pod hash> bash
        

   2. Navigate to the backup log file.

      cd cs-db/cs-db-backup/logs/
      

   3. Get the output from the log file.

      Cat backup_log_<date time value>.log
      

   The common-service-db pod verification is successfully completed.

3. Verify the successful completion of zen-metastore-db.

   1. Find zen5-backup-<pod hash> in the services namespace (if applicable) on the Backup cluster with one of the following methods:

      • To find zen5-backup-<pod hash> with the OpenShift Container Platform console, Go to pod and select Terminal.

      • To find zen5-backup-<pod hash> with the CLI, run the following command:

        oc exec -it zen5-backup-<pod hash> bash
        

        Note: If zenservice is not available immediately after the restore, you need to wait for the zenservice .status.Progress to be 100%. To check the progress, run the following command:

        oc get zenservice -A -w -o yaml | grep Progress:
        

   2. Navigate to the backup log file.

      cd zen5/zen-backup/logs/
      

   3. Get the output from the log file.

      Cat backup_log_<date time value>.log
      

   The zen-metastore-db pod verification is successfully completed.

4. If the verification is not successfully completed, resolve the issues and perform backup again.

Creating a restore

To create a restore, complete the following steps:

1. Login to the Spectrum Fusion UI.
2. Go to Backup & Restore > Backed up Applications.
3. Select the ellipsis overflow menu of an application record.
4. Click the Restore option.
5. Select the restore destination. For the New Cluster, select the target restore cluster that is linked to the Spectrum Fusion hub cluster as a spoke.
6. Select the backup that needs to be restored and click Next.
7. Ensure that you select the Include missing etc resources option.
8. Select the required subset of PVCs or all available PVCs and click Restore.
9. Confirm restore.

10. Go Backup & Restore > Jobs > Restore and select the running job in the Jobs page to monitor the backup job status.

    The clipboard icon at the upper right of the Summary box displays logs from the Spectrum Fusion process. For more information, see Jobs.

11. If you use a custom route for the restored zenservice and you are restoring to a new cluster, update the value of zenCustomRoute in the zenservice CR to reflect the new hostname and re-trigger the iam-config job. Run the following commands:

    oc -n <zenservice namespace> patch zenservice  <zenservice name>  --type='merge' -p '{"spec":{"zenCustomRoute":{"route_host":"<updated route>"}}}'
    oc -n  <zenservice namespace> patch zenservice <zenservice name>  --type='merge' -p '{"spec":{"reconcile":true}}'
    oc get job -n  <zenservice namespace> iam-config-job -o json | jq 'del(.spec.selector)' | jq 'del(.spec.template.metadata.labels)' | oc replace --force -f -
    

Verifying successful restore

To verify that the restore is successful, complete the following steps:

1. Review the restored cluster for the restored services and resources. If applicable, you can login to cp-console and cpd console.

2. If you use multiple namespaces, check that the setup tenant command completed successfully. If not, the recipe is failed in the restore cluster. Check the logs in the cpfs-util.

3. Ensure that the job is completed in the Spectrum Fusion UI.

4. Verify the successful completion of common-service-db.

   1. Find cs-db-backup-<pod hash> in the services namespace (if applicable) on the restore cluster with one of the following methods:

      • To find cs-db-backup-<pod hash> with the OpenShift Container Platform console, Go to pod and select Terminal.

      • To find cs-db-backup-<pod hash> with the CLI, run the following command:

        oc exec -it cs-db-backup-<pod hash> bash
        

   2. Navigate to the restore log file.

      cd cs-db/cs-db-backup/logs/
      

   3. Get the output from the log file.

      Cat backup_log_<date time value>.log
      

   The common-service-db pod verification is successfully completed.

5. Verify the successful completion of zen-metastore-db.

   1. Find zen5-backup-<pod hash> in the services namespace (if applicable) on the Backup cluster with one of the following methods:

      • To find zen5-backup-<pod hash> with the OpenShift Container Platform console, Go to pod and select Terminal.

      • To find zen5-backup-<pod hash> with the CLI, run the following command:

        oc exec -it zen5-backup-<pod hash> bash
        

      Note: If zenservice is not available immediately after the restore, you need to wait for the zenservice .status.Progress to be 100%. To check the progress, run the following command:

      oc get zenservice -A -w -o yaml | grep Progress:
      

   2. Navigate to the restore log file.

      cd zen5/zen-backup/logs/
      

   3. Get the output from the log file.

      Cat backup_log_<date time value>.log
      

   The common-service-db pod verification is successfully completed.

   If the restore is not successful, complete the following steps:

   1. Navigate to the root directory.

      cd ~
      

   2. Run the following command:

      ./cs-db/br_cs-db.sh restore <service namespace>
      

      Replace <service namespace> with the deployment namespace if you are not using a service namespace.

   3. Verify that the script is completed successfully.