Backing up and restoring data on different clusters

Learn how to back up the WebSphere Automation data from one cluster and restore it to another cluster. In this procedure, which can be used for offline disaster recovery, you use Red Hat® OpenShift® APIs for Data Protection (OADP) with IBM Cloud® Object Storage to back up data in a cloud storage that another cluster can access later.

Before you begin

  1. Set up a backup storage with IBM Cloud Object Storage.

    To integrate your backup PVC with IBM Cloud Object Storage, you need to make sure that you have an IBM Cloud account with an instance of IBM Cloud Object Storage. Follow these steps:

    1. Create an instance of IBM Cloud Object Storage in IBM Cloud. This step requires an IBM Cloud account.
    2. Create a bucket in IBM Cloud Object Storage.
      1. Log in to IBM Cloud, and in the menu, click Resource list.

      2. If you do not have an instance of IBM Cloud Object Storage, create one by accessing the Catalog in the dashboard.
      3. Locate your IBM Cloud Object Storage instance. In the menu, click Instances > Create bucket > Create a Custom Bucket, then click Create.
      4. Click Customize your bucket. Specify a unique bucket name. Select the type of Resiliency, Location, and type of Storage class, then click Create bucket.Creating a custom bucket in IBM Cloud Object Storage
      5. Select the bucket that you created and click Permissions. In the Access policies panel, select Service ID for Policy Type and for Role for this bucket, select Writer. Click Create access policy. If you do not have a Service ID, click the Create service ID in the Access policies panel to create one. Setting bucket access policies in IBM Cloud Object Storage
      6. To establish a connection with IBM Cloud Object Storage, a bucket endpoint name is needed. Click Configuration, then click Endpoints. Scroll to the Endpoints panel and locate the endpoint name that suits your requirements. Copy the endpoint name to a location where you can retrieve it later.Setting an endpoint name for the bucket in IBM Cloud Object Storage
      7. Authentication with IBM Cloud Object Storage requires an access key and secret key. From the IBM Cloud Navigation menu, click Resource list and click your Cloud Object Storage instance under Storage. Click the Service credentials tab. If you have not created a service credential yet, do so by using the New Credential tab and enabling the Include HMAC Credential toggle switch. Save the cos_hmac_keys block from your credential, which contains the access key and secret key for later use.
  2. Install the Red Hat OpenShift API for Data Protection (OADP) operator and create a DataProtectionApplication custom resource.

    To implement data protection for your applications across both source and target clusters, follow these steps:

    1. Install the OADP Operator in the openshift-adp project from the Red Hat OpenShift OperatorHub.
    2. Create a cloud-credentials secret within the openshift-adp project.

      1. Create a cloud-credentials secret within the openshift-adp project by using credentials sourced from the IBM Cloud Object Storage bucket. This secret facilitates secure access to the storage resources needed for backup and restoration operations.
      2. To create the cloud-credentials secret, go to the Red Hat OpenShift web console and access Workloads > Secrets from the menu.

        Click Create within the Secrets window and select Key/value secret from the drop down list.

        Specify the required details including Secret name, Key, and corresponding cloud access credentials.
        
        Specify the following values in the Create key/value secret panel:
        
        Secret name: cloud-credentials 
        Key: cloud 
        Value: [Enter the access key and secret key that you created previously.]
        
        [default]
        aws_access_key_id=<KEY_ID_VALUE>
        aws_secret_access_key=<SECRET_ACCESS_KEY_VALUE>
        Creating the cloud-credentials secret in the Red Hat OpenShift web console
        Note: Do not use quotation marks or brackets in the key values.
      3. To finalize the creation of the cloud-credentials secret, click Create.

    3. Create DataProtectionApplication custom resource.
      1. Access the Red Hat OpenShift web console and go to Operators > Installed Operators from the menu.

      2. Select the OADP Operator and proceed to configure the DataProtectionApplication resource.

      3. Click Create DataProtectionApplication and opt to configure via the YAML view.

      4. Update the fields with the provided specifications for the DataProtectionApplication custom resource.

        For spec.backupLocations.objectStorage.bucket, use the bucket name you added in a previous step. For spec.backupLocations.config.s3Url, use the bucket endpoint desired, that you added in a previous step.

        
        kind: DataProtectionApplication
        apiVersion: oadp.openshift.io/v1alpha1
        metadata:
          name: velero-dpa
          namespace: openshift-adp
        spec:
          backupLocations:
            - velero:
                config:
                  profile: default
                  region: ams03
                  s3Url: https://s3.ams03.cloud-object-storage.appdomain.cloud
                  insecureSkipTLSVerify: "true"
                  s3ForcePathStyle: "true"
                credential:
                  key: cloud
                  name: cloud-credentials
                default: true
                objectStorage:
                  bucket: wsabucket
                  prefix: velero
                provider: aws
          configuration:
            restic:
              enable: true
            velero:
              defaultPlugins:
                - openshift
                - csi
                - aws
                - kubevirt
              featureFlags:
                - EnableCSI
          snapshotLocations:
            - velero:
                config:
                  profile: default
                  region: ams03
                provider: aws
        
        Note: If you enable your cluster to run pods on the primary nodes, use the following YAML section for spec.configuration.
        
         spec:
            ...
            configuration:
                nodeAgent:
                  enable: true
                  uploaderType: restic
            velero:
            ...
        
        Note: The object storage that is used is s3-compatible storage that is provided by the IBM Cloud Object Storage instance. The custom s3Url capability from the aws velero plugin is used to access the IBM Cloud Object Storage endpoint in Velero.
      5. To finalize the creation process, click Create.

      6. From the menu, select All instances. Verify that the status of DataProtectionApplication is Condition: Reconciled and the status of BackupStorageLocation is Phase: Available.

    4. Update VolumeSnapshotClass Deletion Policy.
      When the deletionPolicy attribute of the VolumeSnapshotClass is set to Delete, the associated storage snapshot is removed concurrently with the VolumeSnapshotContent object. To maintain the integrity of volume snapshots within the storage system during the Velero backup lifecycle, consider updating the deletion policy to Retain. This adjustment makes sure that the preservation of volume snapshots, even in scenarios such as namespace loss during a disaster event.

      The OADP/Velero CSI plug-in autonomously selects the appropriate VolumeSnapshotClass within the cluster, matching both the driver name and the value of velero.io/csi-volumesnapshot-class, which must be set to true.

      To update the VolumeSnapshotClass deletion policy:
      1. In the Red Hat OpenShift web console, goto Storage > VolumeSnapshotClasses.
      2. Click csi-cephfsplugin-snapclass, and switch to the YAML tab.

      3. Update the deletionPolicy values and add relevant labels in the metadata section with the following values:

        
        apiVersion: snapshot.storage.k8s.io/v1
        deletionPolicy: Retain
        driver: rook-ceph.cephfs.csi.ceph.com
        kind: VolumeSnapshotClass
        metadata:
          labels:
            velero.io/csi-volumesnapshot-class: 'true'
        ......        
        
  3. Verify the default storage class.
    1. To verify the default storage class configuration, in the Red Hat OpenShift web console menu, go to Storage > StorageClasses.

    2. Confirm that rook-cephfs is designated as the default storage class for both the source and target clusters.

    3. Or run the following command to verify the default storage class.
      oc get StorageClasses
      The output resembles the following:
      
          NAME                    PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
          rook-ceph-block         rook-ceph.rbd.csi.ceph.com      Delete          Immediate           true                   18d
          rook-cephfs (default)   rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   18d
      
  4. Download and install the IBM Catalog Management Plug-in for IBM Cloud Paks.

    To download and install IBM Catalog Management Plug-in for IBM Cloud Paks, see Backing up and restoring data.

    Ensure that you download and install the plug-in on both the source and target clusters. This configuration enables the integration of the plug-in into your infrastructure and facilitates efficient management and backup processes across your environment.

Steps

  1. Back up the WebSphere Automation data to a PVC using the ibm-pak launch backup command
  2. Back up the PVC from a source cluster to IBM Cloud Storage using OADP Application Backup
  3. Restore the PVC from IBM Cloud Object Storage to a target cluster using OADP Application Restore
  4. Restore the WebSphere Automation data in the PVC to the target cluster using the ibm-pak launch restore command
  5. Replace the usage metering API key

Back up the WebSphere Automation data to a PVC using the ibm-pak launch backup command

To back up the WebSphere Automation data to a PVC by using the ibm-pak launch backup command, run the following steps:

  1. Set the environment variables for the IBM WebSphere Automation case.

    export CASE_NAME=ibm-websphere-automation
    export CASE_VERSION=1.8.1
  2. Run the ibm-pak launch command to start the backup process.
    oc ibm-pak launch $CASE_NAME \
        --version $CASE_VERSION \
        --inventory automationOperator \
        --action backup \
        --namespace instance_namespace

    By default, the data is stored in a PVC named wsa-backup. For more information on backing up data, see Backing up and restoring data.

  3. Ensure the backup is created successfully by verifying its presence in the backup list. Initiate a job to list available backups by using the following command.
    oc ibm-pak launch $CASE_NAME \
      --version $CASE_VERSION \
      --inventory automationOperator \
      --action restore \
      --namespace instance_namespace

    Run the oc log command as indicated in the restore command output to list the available backups and make sure that the backup you created is listed with its timestamp.

Back up the PVC from a source cluster to IBM Cloud Storage using OADP Application Backup

To run the backup of a PVC from a source cluster to IBM Cloud Object Storage using OADP (Red Hat OpenShift API for Data Protection) Application Backup, do the following steps.

  1. Add a label to the was-backup PVC.

    In the Red Hat OpenShift web console, access the menu > Storage > Persistent Volume Claims. Locate and click the PVC named was-backup within the project where WebSphere Automation is installed. In the Metadata section, add the following label.

    labels:
      wsa.ibm.com: wsa
  2. Create a pod that references the PVC. Generate a YAML file named pod.yaml with the following content. Use the namespace where you deployed the WebSphere Automation instances.
    
    apiVersion: vl 
    kind: Pod
    metadata:
         name: dummy-pod 
         namespace: wasautomation
         labels:
              wsa.ibm.com: wsa
    spec:
         containers: 
              - name: dummy-container 
                image: nginx:latest
                volumeMounts: 
                   - name: data-volume
                     mountPath: / data
         volumes: 
              - name: data-volume 
                persistentVolumeClaim:
                   claimName: wsa-backup
    

    Run the following command to create the pod.

    oc apply -f node.yaml
  3. Backup the pod that is created and the was-backup PVC to IBM Cloud Storage. Create a YAML file named backup-oadp.yaml with the following content.
    
    apiVersion: velero.io/v1
    kind: Backup
    metadata:
       name: my-backup
       namespace: openshift-adp
    spec:
       defaultVolumesToRestic: true
       storageLocation: velero-dpa-1
       includeClusterResources: true
       includedNamespaces:
       - wasautomation
       orLabelSelectors:
       - matchExpressions:
          - key: wsa.ibm.com
            operator: In
            values:
            - wsa
    

    Run the following command to perform the backup operation with OADP.

    oc apply -f backup-oadp.yaml
  4. Validate the completion of the backup operation.

    In the Red Hat OpenShift web console, access the menu > Operator > Installed Operators, within the openshift-adp project. Choose the OADP Operator > All Instances tab to confirm the backup process.

    Validating the completion of the backup process in the Red Hat OpenShift web console

Restore the PVC from IBM Cloud Object Storage to a target cluster using OADP Application Restore

To restore the PVC from IBM Cloud Object Storage to a designated target cluster, employing OADP (Red Hat OpenShift API for Data Protection) Application Restore, do the following steps.

  1. Set up the WebSphere Automation operator on the target cluster, making sure parity in versions between the source and target clusters. Then, deploy the WebSphere Automation instances on the target cluster, mirroring the configuration of the source cluster, including project names.
  2. Install the OADP operator by following the procedures outlined in the Install the Red Hat OpenShift API for Data Protection (OADP) operator and Create a DataProtectionApplication custom resource section that is provided in the preceding prerequisites.
  3. Run the restoration of the PVC, along with the associated pod, from IBM Cloud Storage to the target cluster. Create a YAML file named restore-oadp.yaml with the following content.
    
    apiVersion: velero.io/v1 
    kind: Restore
    metadata:
       name: my-restore 
       namespace: openshift-adp
    spec:
       backupName: my-backup
       excludedResources :
       - nodes
       - events
       - events.events.k8s.io
       - backups.velero.io
       - restores.velero.io
       - resticrepositories.velero.io
       hooks: {}
       includedNamespaces:
       - '*'
       labelSelector:
          matchLabels:
             wsa.ibm.com: wsa
    

    Run the following command to initiate the restore operation with OADP.

    oc apply -f restore-oadp.yaml
  4. Confirm the successful completion of the restore operation.

    In the Red Hat OpenShift web console, access the menu > Operator > Installed Operators, within the openshift-adp project. Choose the OADP Operator > All Instances tab to validate the restoration process.

    Validating the completion of the restoration process in the Red Hat OpenShift web console
  5. Verify the restoration of the wsa-backup PVC within the project where the WebSphere Automation instances were deployed.

    Run the following command.

    oc get pvc wsa-backup

    The expected output is as follows.

    
    NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    wsa-backup   Bound    pvc-ef135cd1-3da3-4a43-8e7b-081140a84a36   10Gi       RWO            rook-cephfs    10d
  6. You can delete the dummy node. It is no longer needed.

Restore the WebSphere Automation data in the PVC to the target cluster using the ibm-pak launch restore command

  1. Determine the timestamp of the backup performed on the source cluster from which recovery is intended. First, run the following commands to list available backups.
    1. Set the environment variables.
      export CASE_NAME=ibm-websphere-automation
      export CASE_VERSION=1.8.1
    2. Run the ibm-pak launch command for restoration.
      oc ibm-pak launch $CASE_NAME \
      --version $CASE_VERSION \
      --inventory automationOperator \
      --action restore \
      --namespace instance_namespace
  2. Run the oc log command as indicated in the restore command output to confirm the presence of the same backup timestamp from the source cluster.
  3. Use the ibm-pak launch restore command to restore WebSphere Automation data from the wsa-backup PVC to the target cluster.
    • To initiate the restoration from the backup on the source cluster, run the restore action with the --timestamp argument.
    • Replace <timestamp> with the timestamp of the backup directory intended for restoration.
    oc ibm-pak launch $CASE_NAME \
    --version $CASE_VERSION \
    --inventory automationOperator \
    --action restore \
    --namespace $WSA_INSTANCE_NAMESPACE \
    --args "--timestamp <timestamp>"
    For more information on restoring data, see Backing up and restoring data
  4. Validate that your WebSphere Automation data is restored successfully by opening the WebSphere Automation UI.

Replace the usage metering API key

  1. For more information on how to replace the usage metering API key, see Revoking or replacing the usage metering API key.
  2. Upon replacing the usage metering API key, it is necessary to update your WebSphere Application Server and WebSphere Application Server Liberty configurations to incorporate the new API key.
  3. View the server logs to validate that the servers are registered successfully.