Switching to passive hub cluster

Switch to the passive hub cluster in a Metro DR setup by restoring backups and verifying that managed clusters and DRPolicies are successfully imported when the active hub is down or unreachable.

1. Restore the backups on the passive hub cluster. For information, see Operations > Business continuity > Backup and restore > Restoring a backup in Red Hat Advanced Cluster Management for Kubernetes product documentation.
   Important: Recovering a failed hub to its passive instance only restores applications and their DR protected state to its last scheduled backup. Any application that was DR protected after the last scheduled backup would need to be protected again on the new hub.
2. During the restore procedure, to avoid eviction of resources when ManifestWorks are not regenerated correctly, you can enlarge the AppliedManifestWork eviction grace period.
   1. Verify that the restore is complete.

      oc -n <restore-namespace> wait restore <restore-name> --for=jsonpath='{.status.phase}'=Finished --timeout=120s

   2. After the restore is completed, on the hub cluster, check for existing global KlusterletConfig
      • If global KlusterletConfig exists then edit and set the value for appliedManifestWorkEvictionGracePeriod parameter to a larger value. For example, 24 hours or more.
      • If global KlusterletConfig does not exist, then create the Klusterletconfig using the following yaml:

        apiVersion: config.open-cluster-management.io/v1alpha1
        kind: KlusterletConfig
        metadata:
          name: global
        spec:
          appliedManifestWorkEvictionGracePeriod: "24h"

        The configuration is propagated to all the managed clusters automatically.

3. Verify that the Primary and Seconday managed clusters are successfully imported into the RHACM console and they are accessible. If any of the managed clusters are down or unreachable then they are not successfully imported.
4. Wait until DRPolicy validation succeeds.
5. Verify that the DRPolicy is created successfully. Run this command on the Hub cluster for each of the DRPolicy resources created, where <drpolicy_name> is replaced with a unique name.

   oc get drpolicy <drpolicy_name> -o jsonpath='{.status.conditions[].reason}{"\n"}'

   Example output:

   Succeeded

6. Refresh the RHACM console to make the DR monitoring dashboard tab accessible if it was enabled on the Active hub cluster.
7. After all components are recovered, edit the global KlusterletConfig on the new hub and remove the parameter appliedManifestWorkEvictionGracePeriod and its value.
8. If only the active hub cluster is down, restore the hub by performing hub recovery, and restoring the backups on the passive hub. If the managed clusters are still accessible, no further action is required.
9. If the primary managed cluster is down, along with the active hub cluster, you need to fail over the workloads from the primary managed cluster to the secondary managed cluster. For failover instructions, based on your workload type, see Subscription-based application failover between managed clusters or ApplicationSet-based application failover between managed clusters.
10. Verify that the failover is successful. When the Primary managed cluster is down, then the PROGRESSION status for the workload would be in Cleaning Up phase until the down managed cluster is back online and successfully imported into the RHACM console.
    On the passive hub cluster, run the following command to check the PROGRESSION status.

    oc get drpc -o wide -A

    NAMESPACE      NAME                              AGE   PREFERREDCLUSTER   FAILOVERCLUSTER   DESIREDSTATE   CURRENTSTATE   PROGRESSION   START TIME             DURATION       PEER READY
    busybox        busybox-cephfs-placement-1-drpc   42m   ocp4perf1                                           Deployed       Completed     2024-04-18T09:03:05Z   1.044537443s   True