Application DR

Steps to setup an Application DR, enroll an application, failover applications, and disable applications.

Step 1-setup Application DR

You can setup Application DR anytime after you set up the Metro-DR and before you enable DR for an application.
  1. Set up Metro-DR. For the procedure to setup, see Setting up Metro-DR. After Site 1, Site 2, and Tiebreaker are installed, proceed with the next steps.
  2. Do the following steps on both Site 1 and Site 2 of Metro-DR:
    1. If OADP (version used by Backup & Restore) operator does not exist, then create a namespace and install it.
      1. Go to OperatorHub > Operator Installation.
      2. Find OADP, the Red Hat® distribution of Velero.
      3. In the Install Operator window, select the default namespace openshift-adp. You can also install it in another namespace.
      4. Click Install.
    2. Create DataProtectionApplication instance to start the velero operator.
      
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  labels:
    app.kubernetes.io/component: velero
  name: velero
  namespace: openshift-adp
spec:
  backupImages: false
  configuration:
    restic:
      enable: false
    velero:
      defaultPlugins:
        - openshift
        - aws
      noDefaultBackupLocation: true
      podConfig:
        resourceAllocations:
          limits:
            cpu: '1'
            ephemeral_storage: 25Mi
            memory: 1Gi
          requests:
            cpu: 100m
            ephemeral_storage: 25Mi
            memory: 256Mi
  podDnsConfig: {}
  3. Create secrets in the Velero operator namespace to access S3 storage on both the sites of Metro-DR. The S3 credentials in this secret are in the following Amazon Web Services format:
    
[default]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
  4. Access details in the ibm-spectrum-fusion-ns (or your Fusion namespace) Velero secret:
    • For Velero secret to access Site 1 S3 storage, see isf-metrodr-minio-site1 secret.
    • For Velero secret to access Site 2 S3 storage, see isf-metrodr-minio-site2 secret.
    Example cloud-credentials-site1 secret content to access the S3 storage of Metro-DR sites:
    • Metro-DR Site 1.
      
Key: cloud
Value:
[default]
aws_access_key_id=minio
aws_secret_access_key=04ie0oO7x46g0i5
    • Metro-DR Site 2:
      
Key: cloud
Value:
[default]
aws_access_key_id=minio
aws_secret_access_key=84ie0oO9x46g0i8
  5. Enable kubeObjectProtection in ramen-dr-cluster-operator-config map of ibm-spectrum-fusion-ns namespace:
    1. To enable kubeObjectProtection, set disabled to false and specify the Velero installation namespace.
      
kubeObjectProtection:
      disabled: false
      veleroNamespaceName: openshift-adp
    2. In S3StoreProfiles section, for every S3StoreProfile, specify the Velero secret details for the corresponding site, including key name and secret name:

      Example for Site 1: 

      
veleroNamespaceSecretKeyRef:
        key: cloud
        name: cloud-credentials-site1
      Example for Site 2: 
      
veleroNamespaceSecretKeyRef:
        key: cloud
        name: cloud-credentials-site2
    3. Retrieve caCertificates:

      For every site, use ca-bundle.crt key in the default-ingress-cert ConfigMap of openshift-config-managed namespace to retrieve certificates for the corresponding site and encode them by using the Base64 encoder.

    4. Configure ramen-dr-cluster-operator-config ConfigMap by using the following example:
      kind: ConfigMap
apiVersion: v1
metadata:
  name: ramen-dr-cluster-operator-config
  namespace: ibm-spectrum-fusion-ns
data:
  ramen_manager_config.yaml: |
    apiVersion: ramendr.openshift.io/v1alpha1
    drClusterOperator: {}
    health:
      healthProbeBindAddress: :8081
    kind: RamenConfig
    kubeObjectProtection:
      disabled: false
      veleroNamespaceName: openshift-adp
    leaderElection:
      leaderElect: true
      leaseDuration: 0s
      renewDeadline: 0s
      resourceLock: ""
      resourceName: dr-cluster.ramendr.openshift.io
      resourceNamespace: ""
      retryPeriod: 0s
    metrics:
      bindAddress: 127.0.0.1:9289
    ramenControllerType: dr-cluster
    s3StoreProfiles:
    - s3Bucket: isf-minio-site1
      s3CompatibleEndpoint: https://isf-minio-ibm-spectrum-fusion-ns.apps.rackag2.mydomain.com
      s3ProfileName: site1
      s3Region: site1
      s3SecretRef:
        name: isf-minio-site1
        namespace: ibm-spectrum-fusion-ns
      veleroNamespaceSecretKeyRef:
        key: cloud
        name: cloud-credentials-site1
      caCertificates: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURiRENDQWxTZ0F3SUJBZ0lIWjc3OHV4V3QrREFOQmdrcWhraUc5dzBCQVFzRkFEQW1NU1F3SWdZRFZRUUQKREJ0cGJtZHlaWE56TFc5d1pYSmhkRzl5UURFMk9ESXpNVGs1TlRBd0hoY05Nak13TkRJME1EY3dOVFE1V2hjTgpNalV3TkRJek1EY3dOVFV3V2pBbU1TUXdJZ1lEVlFRRERCc3FMbUZ3Y0hNdWNtRmphMkZuTWk1dGVXUnZiV0ZwCmJpNWpiMjB3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQ3BiaWI5dURTTTVjTk4KdzdKVGRSWGppd2JNdUlWRGhXbmhIMU9oejA4SEY4ck9McEplVEpiYXZJKzBPdjdnaHcwb2p3MWVoaW9adktraApVdWg2R1dyWUtiTk5DSm0xVkhrSWt6ZDhXYlpDcklkdTFXcCtnN21zaHpkcTBYZEtNOXVkblB3aG9VNURtV2RzCmNuRjU0bnl0bkhoOXJVM0ZNU1hHUGhyU3J0cnVEQ01EUkQzQVJ6UmlOL09VNVlyZHFtYk1teHM3RUJtZmRJY1QKWGlFQkoxb09TTC85enkzNnZiSFFkckptc2hPUEN6VG1LWkJ6MklVL3NvR2NraVBrT2tXL1gvRERUemZNTEJNWApab3NxWWRpNWxjOTR0ek9wUjNMOGU1bVhrR2k4cnNHamJpcURmTHpMZFFtN1pmT1BqYWdsQjE1M0JvdnRMM1hjCmhKQzJiWGJsQWdNQkFBR2pnWjR3Z1pzd0RnWURWUjBQQVFIL0JBUURBZ1dnTUJNR0ExVWRKUVFNTUFvR0NDc0cKQVFVRkJ3TUJNQXdHQTFVZEV3RUIvd1FDTUFBd0hRWURWUjBPQkJZRUZJVy9UdjdtY1BBS1JVWlVJNlRDMmVMdwpaZWhTTUI4R0ExVWRJd1FZTUJhQUZDNmpGaXJiZzdVeXdUeEc2UTFDWHMyMWVwVjJNQ1lHQTFVZEVRUWZNQjJDCkd5b3VZWEJ3Y3k1eVlXTnJZV2N5TG0xNVpHOXRZV2x1TG1OdmJUQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUEKdENDUGZkUWs2Ui9XeGZ3VC9SWGdlOFczWlFwNUJjL3A1eEJ0MG1QODcwbVBFc0ZmRGhBVVpiYStZeFJNWWp0SApOblBhemlKQkdmUVpFamRoblVGb0lVVjNYd0VENUQrSzZtVCtxVWgyNUNNazBuc1YveEZ0eUZuTlYyWElFcSthCmtjclowZTcvVVRpcVJsR3lYelgwSUpFYnlsYWxrL1lHb1pqMG10SDZ6aEVNVjNaczV1UVlJYmNJSmh0N1Z0UHkKR2lJQmdxaHBhS2NPU2VJWUpaUzlyVDE0NEdIL2lvc2xQTTNpUFJDQkxmUVNWSnJqMGRndVpKN0czQS9Vc3BEbAozOFp5QnNFZXZ1bXgxTDY5M0xoOElsN2NzcnBTVVBFZG8zc3ZZcXdtWVBlQzdVZTN3RGEyaCtjQkEyMW5CYW5zCllLT1ErYTV5eXFUR0MyKzBoVnpiaHc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlERERDQ0FmU2dBd0lCQWdJQkFUQU5CZ2txaGtpRzl3MEJBUXNGQURBbU1TUXdJZ1lEVlFRRERCdHBibWR5ClpYTnpMVzl3WlhKaGRHOXlRREUyT0RJek1UazVOVEF3SGhjTk1qTXdOREkwTURjd05UUTVXaGNOTWpVd05ESXoKTURjd05UVXdXakFtTVNRd0lnWURWUVFEREJ0cGJtZHlaWE56TFc5d1pYSmhkRzl5UURFMk9ESXpNVGs1TlRBdwpnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFEc1VhME9IOG96TFBtS3VxTVhSV1h4CnNVWFgyN0tuY2pqdnMzNFc5b2V0aGp2ZWh5U0NLMHJvYkpDQmt2enpHelpRaXRTT2NmUUNnV1BUQ2NIVWl5YVMKNWNKUUt1bGFSbjBYUTFrVld4MVQxbFo2NXZHcmliVnhYblhTUTVhVzR3T0w1bVJNT1hyVnkyZ0ZLK2NHMzZ2RgprQ2VBMmZXcXZONi8ycnZ3aEFpU2VYTHpOc3dEWHk1Q3puOGZiajl3K2Q0YWE2VFhUSG96VWZlQ21aaCtpK3MyCjlpQ0MzSzhjSGtSdTN4K0xwZldWK2MzNGd3c0JCbGtJYU1xWkpNY094WGhNemJ3dHJtRGJOeHpLc3ZUeXJ2bVoKcnhiWHVEblpyMExWZ3hvSS9qNjhvUEdreWpRdVQxd0VEcEdtZ1JHczhVdFRNaVdrcG1QY2VPTmVrTzVDbmltbgpBZ01CQUFHalJUQkRNQTRHQTFVZER3RUIvd1FFQXdJQ3BEQVNCZ05WSFJNQkFmOEVDREFHQVFIL0FnRUFNQjBHCkExVWREZ1FXQkJRdW94WXEyNE8xTXNFOFJ1a05RbDdOdFhxVmRqQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUEKMjRJN3JvYmJTZmtXdXZGVjZQNGNEeXdoQ0xWKzZjWHQzdnVLWnFCWmxjSlB4ODFpNnl3YktyTjZidXNQRWkxQwpDWkIzdVpJcTBrZUhVRGFoQndwM1BYVWlhcG9LMHRRNC9CcWpBd21DejZXVHc2VmVRRDlaZWErcFpINWlaNHZDClBIV1VkSHdId0ZUOUZWRkVxNWxRa3RUYVdqclRLdTE1YnpHT3RpNTE3RFkyaHNrMDJFcVJXdk1DUzZTeWhNb3IKbDN5MlNiMTFnVWMyZ1czaU1TSFo4ZzkzcHVIZGZRUXVEMElqbXFSMWp4ZHc1K05tcFlka3N1M1ArbGE4aW1lawpxa1U3eUp4UzhnRzZ1NmFrdXk0YU9YakhwVUFtSTVOTG0xdW9wNXdiUTlGV2RVZkNxQmd0cmxRblhhaVNXQ3lBCmJXTkxjR0V1a3RnZzUrcGVVVjJzMHc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0t
    - s3Bucket: isf-minio-site2
      s3CompatibleEndpoint: https://isf-minio-ibm-spectrum-fusion-ns.apps.rackag3.mydomain.com
      s3ProfileName: site2
      s3Region: site2
      s3SecretRef:
        name: isf-minio-site1
        namespace: ibm-spectrum-fusion-ns
      veleroNamespaceSecretKeyRef:
        key: cloud
        name: cloud-credentials-site2
      caCertificates: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURiVENDQWxXZ0F3SUJBZ0lJVTc3MFdhck5pdVl3RFFZSktvWklodmNOQVFFTEJRQXdKakVrTUNJR0ExVUUKQXd3YmFXNW5jbVZ6Y3kxdmNHVnlZWFJ2Y2tBeE5qZ3lORGc0TXpFeE1CNFhEVEl6TURReU5qQTFOVEUxTVZvWApEVEkxTURReU5UQTFOVEUxTWxvd0pqRWtNQ0lHQTFVRUF3d2JLaTVoY0hCekxuSmhZMnRoWnpNdWJYbGtiMjFoCmFXNHVZMjl0TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUFxN1JYaXA2WW04UlkKYVJEYVhZTGdZaWpTVjNaOEx6S3UvWDVnakR4NUtqd2tBUW5IeWZJazBpOWVrSnRMR2RabWgzbEFCZi9jRS9YNAoyUzlZLzFiRUhPN3J2cHpLVHhZajVtQ1BSRXRLT3BBY1B1bEY3SjJqcnBYODRMYzJJdmNtZHB1blR0TVhUVHBtCkxFS0JVdWMyVUZkOHdrcUpKV3pRRVlhVFpueDZxVVVYOEM2SkRYVzlNM1B6THJIOHBza2xxS0ZvWk9yaEtzNnYKYjhOeUJZMjd1eTNaZWFhS21KU2l2K1R4M2RURGR1amxwNTdoUnd3Y004WnNRdy93MjB2cTQ2S2JPcWZ2YmpmZApIZ0pEbHV3WjhBQzZGWld2OCtwRVk4YTlaVDVtUE1MTnZ5OUZnS2NRMzczdjZEd2VTSGZudWxsZjJVYldBTkJLCjJLOTNTTUwyc3dJREFRQUJvNEdlTUlHYk1BNEdBMVVkRHdFQi93UUVBd0lGb0RBVEJnTlZIU1VFRERBS0JnZ3IKQmdFRkJRY0RBVEFNQmdOVkhSTUJBZjhFQWpBQU1CMEdBMVVkRGdRV0JCUTVCelVrVkdic3BVUEU2cjhrSkZrZApxV01lTGpBZkJnTlZIU01FR0RBV2dCUW5nOHErVlFMSnRzUWtEbTk1S0x1RTJYYTRrVEFtQmdOVkhSRUVIekFkCmdoc3FMbUZ3Y0hNdWNtRmphMkZuTXk1dGVXUnZiV0ZwYmk1amIyMHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUIKQUN3NmFVTkRNY0VtaVBMUWZWcTEzY0laZHZVaWE3Y252Q3hLNWR1WDdReVZKOU5jZ21QMUo0WDJ6Vmh0enJMawptNnAybjhCcHF0WXZGSDBSRkxJUHpCVTd6eVN0N0szUDJ5clhHM3laV1N6aXVIYURPdUQ5YXBmTVY3QnlrOVdYCk8xQnVxeU0ydzA3NDBPajU3bndybTNKd3pBK0dsVStVc0RTaSs0RlRuT0lBTHF6R05FWG5IMy80eHN5WVgwejUKNDNpNXdpbUJPYVE0ZWE4K1hlalNOeGJBVzRTNUF3VGlTQnVHaWNSN1RrR0NaNGo3WDZ2eEF0dWZ3Q2pnNElxZAo3RGNWcW11WEdxREhZTFBtVVhKMWVBaXYrRmRCTkQ3dE8xVWhQdXdMdEZxWlAyTERiNHpZSkF5Tnd4NHZPYjJGCnBNZ3dzVmNFZXdkbklmaGduM1hLZGZNPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlERERDQ0FmU2dBd0lCQWdJQkFUQU5CZ2txaGtpRzl3MEJBUXNGQURBbU1TUXdJZ1lEVlFRRERCdHBibWR5ClpYTnpMVzl3WlhKaGRHOXlRREUyT0RJME9EZ3pNVEV3SGhjTk1qTXdOREkyTURVMU1UVXdXaGNOTWpVd05ESTEKTURVMU1UVXhXakFtTVNRd0lnWURWUVFEREJ0cGJtZHlaWE56TFc5d1pYSmhkRzl5UURFMk9ESTBPRGd6TVRFdwpnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFEWFp3SFVoTTJ0Z2pZMG5LOEJGZkFtCllMZkFTOHAwaEdKZ3B3SmtCZFllTUUyTlg2Q1RSeFU0RmZWMVRaZm8vQld3K3dpcHpzY1hzWE9OMUEzWDczVUIKcnhUOXM4OFJTZDJtWlI3VllPVThqbVNPZXR2N1NOeUlqYWtic3hJcmJJanRJOHVpZlpQNGlJT2VlMmFUaWJSYQorWEY4MURuLzdCNldwdmxrYUpxa0ZXekQ3Q2pQeFdUd1NqRjJRZllyOVVYYldYZWI2SnMwaHNwbGRGNDhzQ2pqClQrRXE4VnNxRmdjNFJaQWxjRFVMZEE3VUs3OE1YTnI2NkU4Z2NqdlhtaS9RdjJvZ3dHZGczeHZHTlNDUGw5YW0KQzFkSGJ4Uk9vSkZjNkx6THpiZ3BqeVJHbnRJblNTbCtSRmsvTk13cUUrci8rQkhpWXFpOTlReWxUR2lZY1JQbgpBZ01CQUFHalJUQkRNQTRHQTFVZER3RUIvd1FFQXdJQ3BEQVNCZ05WSFJNQkFmOEVDREFHQVFIL0FnRUFNQjBHCkExVWREZ1FXQkJRbmc4cStWUUxKdHNRa0RtOTVLTHVFMlhhNGtUQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUEKelRoOHhsS3FneWlPU2V4NS9ZblFob0ZycHN5c0x0UGJmMkE2QUVCSy9acGVORlc1T3VmZm5QU25EUEtqdGtORwp5UTlUOFJuS2FrVTRKK0NldGNlUTdKM0l4R3ZzdjNqVGZ3ZXlVMjEra1R2TTJhaGZXVmt3NTZEZlRUZmU4dm5sCnQrcVpqRnBmV2p4NnU1Nnp5S0t1NXcwTm12WllXSUdDOFFUWlJNOE9VRmhudjJFeCtnT2NRb2dPcEtNdXdnVWkKemEzWVRxUC93aVJ2Q2l6ZURXUmsxUmdnR0xVWkcvRGUxZUYxcVp0U0xaZHlFelQ3N3hxMEdCbVpXQjh6NktISAp1eW5VS0ZBa0tZSDJObTAydmlmdVN1MEU2TVN4a2lsVm5qM1ZJQUgrVnJTNU15OHh6MkJzcGdYVnVSSmtnL1lICnFmR1Ywb3VYenZCczV2SHJUZFdNdlE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0t
    volSync:
      disabled: true
    webhook:
      port: 9443
    5. Go to Workloads > Pods in the ibm-spectrum-fusion-ns namespace (or Fusion namespace) and then delete the pod with the prefix ramen-dr-cluster-operator- to reflect the previous Ramen configuration changes.

Step 2-enroll an application

  1. Create VolumeReplicationGroup (VRG) in the namespace of the Application with content similar to the following example:
    Note: You can also do the enrollment from the IBM Storage Fusion user interface.
    apiVersion: ramendr.openshift.io/v1alpha1
kind: VolumeReplicationGroup
metadata:
  name: shioramen
  namespace: shioramen
spec:
  kubeObjectProtection: {}
  pvcSelector: {}
  replicationState: primary
  s3Profiles:
    - site2
    - site1
  sync: {}
  volSync:
    disabled: true
    Important: Make sure that all the required parameters are defined in spec during the VRG creation. If you edit VRG parameters later on, it can lead to an inconsistent behavior. If you still want to edit, recreate the VRG.
  2. If the Application is already enrolled by using the UI, then use the following command to patch the existing VRG:
    kubectl patch -n <namespace> vrg/<namespace> --type json -p'[{"op":"add", "path":"/spec/kubeObjectProtection", "value": {}}]'
  3. The VRG status must show ClusterDataProtected as true. If you see AnnotationFailed reason, then go to the PVC and remove the following annotation from it: 
    volumereplicationgroups.ramendr.openshift.io/vr-archived: archiveV1-0

Step 3-failover applications

  1. Failover applications. For steps to failover applications, see Failover applications from site 1 to site 2.
  2. After the persistent volumes are recovered, recover the application. To enable Kubernetes resource protection and recover, update the VolumeReplicationGroup in the namespace of the application.

    Example: 

    kubectl patch -nmy-ns vrg/my-ns --type json -p'[{"op":"add", "path":"/spec/kubeObjectProtection", "value": {}}]'
  3. Failover must be successful for the Application, and the VRG must reflect the same in the status ClusterDataProtected as true. In case the condition is false and you can see the AnnotationFailed reason, then go to the PVC and remove the following annotation from it:
    volumereplicationgroups.ramendr.openshift.io/vr-archived: archiveV1-0

Step 4-disable Application DR

For each application that has DR enabled, disable Metro-DR for that application to clean it up, including deletion of its VolumeReplicationGroup.
  1. Disable DR for applications from the IBM Storage Fusion HCI System. For the procedure to disable, see Disable disaster recovery point in Backing up your applications topic.
  2. Update Ramen ConfigMap and restart the Ramen controller:
    
kubectl -nibm-spectrum-fusion-ns patch cm/ramen-dr-cluster-operator-config --type json -p[{\"op\":add\,\"path\":/data/ramen_manager_config.yaml\,\"value\":\""$(kubectl -nibm-spectrum-fusion-ns get cm/ramen-dr-cluster-operator-config -ojsonpath='{.data.ramen_manager_config\.yaml}'|sed -n '/^kubeObjectProtection:/{:1;n;/^ /b1};p;$!s/$/\\n/'|tr -d '\n')"\"}]
sleep 120
kubectl -nibm-spectrum-fusion-ns rollout restart deploy/ramen-dr-cluster-operator;kubectl -nibm-spectrum-fusion-ns rollout status -w deploy/ramen-dr-cluster-operator
    Note: Update Ramen ConfigMap, wait for two minutes, and then restart the Ramen controller.