Configuring Regional-DR

Configure regional disaster recovery (Regional-DR) capability in the IBM Fusion HCI with Fusion Data Foundation to replicate and recover applications and virtual machines (VMs) across clusters.

Before you begin

If an external Red Hat® Advanced Cluster Management (RHACM) hub cluster is already available, you can use it for the Regional-DR configuration. If you don't have it, install the RHACM operator to create a hub cluster outside the IBM Fusion HCI sites.

For instructions, see the Install guide in the RHACM documentation set.

If hosted control plane (HCP) clusters deployed on IBM Fusion HCI sites are upgraded from a previous version and the RHACM operator is already installed on an HCP cluster, migrate the RHACM operator from the HCP cluster to a new cluster. For assistance with migration, contact IBM support.
It is recommended that the VMs should use the ocs-storagecluster-ceph-rbd-virtualization storage class.
Ensure that the storage class with the same name exists on the secondary-managed cluster.
Ensure that the version of the HCP clusters is 4.18 or higher.
Ensure that the OpenShift® Container Platform version is 4.18.19 or higher to enable Regional-DR for CephFS applications deployed using the ocs-storagecluster-cephfs storage class.

Procedure

Deploy primary and secondary IBM Fusion HCI sites and configure them to have the network reachability between them.
Each site should have one IBM Fusion HCI. If only one IBM Fusion HCI site is present, install an IBM Fusion HCI in another site by following the instructions mentioned in Installing IBM Fusion HCI.
Important: The IBM Fusion HCI version number should be the same across both sites.
Open the following ports to ensure that the pods are reachable from other clusters:
Note: In provider mode, host networking is used for MONs and OSD.
- Ceph mon v1: 6789
- Ceph mon v2: 3300
- Ceph OSD: 6800–7300
- Ceph Manager: 9283
Optional: Set up an SSL connection between clusters deployed on IBM Fusion HCI sites.
For instructions, see Establishing an SSL connection between clusters.
Note: If a signed and valid set of certificates has been used for the deployment of base or HCP clusters in both IBM Fusion HCI sites, skip this step.
Install the Fusion Data Foundation in newly created clusters.
- For base clusters:
  Install the Fusion Data Foundation in provider mode by following the instructions mentioned in Installing Fusion Data Foundation.
- For HCP clusters:
  Install the Fusion Data Foundation client by following the instructions mentioned in Installing Fusion Data Foundation on a Hosted Control Plane cluster.
Important: The Fusion Data Foundation version number must be the same on base and HCP clusters deployed across IBM Fusion HCI sites.
Import the clusters of primary and secondary IBM Fusion HCI sites to the RHACM hub cluster.
- For base clusters:
  Import base clusters to the RHACM hub by following the instructions mentioned in Discovering multicluster engine operator hosted clusters in Red Hat Advanced Cluster Management of the RHACM documentation.
- For HCP clusters:
  Discover HCP clusters from the multicluster engine (MCE) operator by following the instructions mentioned in Discovering hosted clusters of the RHACM documentation.
  
  Discovered HCP clusters can be automatically imported into the RHACM hub if the spec.importAsManagedCluster field is set to true. You need to set this field in the individual discovered cluster's custom resource (CR). For instructions, see Configuring settings for automatic import of the RHACM documentation.
Connect the manged clusters and service networks by using the Submariner add-on of the RHACM.
Verify that the clusters have non-overlapping Classless Inter-Domain Routings (CIDRs) and cluster private networks. For overlapping CIDRs, enable Globalnet during the Submariner add-on installation to connect clusters.

Run the following command on each managed cluster to verify that the clusters have overlapping CIDRs or non-overlapping CIDRs:
```
oc get networks.config.openshift.io cluster -o json | jq .spec
```
The following example is for clusters with overlapping CIDRs, so the Globalnet needs to be enabled to connect clusters.

Example output for primary-managed cluster:
```
{ 
  "clusterNetwork": [ 
    { 
      "cidr": "10.5.0.0/16", 
      "hostPrefix": 23 
    } 
  ], 
  "externalIP": { 
    "policy": {} 
  }, 
  "networkType": "OVNKubernetes", 
  "serviceNetwork": [ 
    "10.15.0.0/16" 
  ] 
}
```
Example output for secondary-managed cluster:
```
{ 
  "clusterNetwork": [ 
    { 
      "cidr": "10.5.0.0/16", 
      "hostPrefix": 23 
    } 
  ], 
  "externalIP": { 
    "policy": {} 
  }, 
  "networkType": "OVNKubernetes", 
  "serviceNetwork": [ 
    "10.15.0.0/16" 
  ] 
}
```
Additionally, if Submariner and Fusion Data Foundation are already installed on the managed clusters, use the Red Hat OpenShift Data Foundation command-line interface (CLI) tool to get additional information about the managed clusters. This information determines the need for enabling Globalnet during Submariner installation based on the cluster service and private networks. Download the Red Hat OpenShift Data Foundation CLI tool from the Red Hat Customer Portal.

Note: Download the latest Red Hat OpenShift Data Foundation CLI package from the Red Hat Customer Portal. No additional license is required to use the CLI tool. The latest CLI package works with earlier and later versions of Fusion Data Foundation, though newer CLI versions can include additional capabilities.

Run the following command on one of the two managed clusters, where PeerManagedClusterName(ClusterID) is the name of the peer Fusion Data Foundation cluster:
```
odf get dr-prereq <PeerManagedClusterName(ClusterID)>
```
If Submariner is installed without Globalnet on managed clusters with overlapping services, the following output appears:
```
Info: Submariner is installed.
Info: Globalnet is required.
Info: Globalnet is not enabled.
```
Note: This requires Submariner to be uninstalled and then reinstalled with Globalnet enabled.

If Submariner is installed with Globalnet on managed clusters with overlapping services, the following output appears:
```
Info: Submariner is installed.
Info: Globalnet is required.
Info: Globalnet is enabled.
```
If Submariner is installed without Globalnet on managed clusters with non-overlapping services, the following output appears:
```
Info: Submariner is installed.
Info: Globalnet is not required.
Info: Globalnet is not enabled.
```
If Submariner is installed with Globalnet on managed clusters with non-overlapping services, the following output appears:
```
Info: Submariner is installed.
Info: Globalnet is not required.
Info: Globalnet is enabled.
```
For more information on Submariner add-ons and Globalnet configuration, see Submariner multicluster networking and service discovery of the RHACM documentation.
Install the OpenShift Multicluster Orchestrator operator on the hub cluster.
For instructions, see Installing Multicluster Orchestrator operator.
To connect clusters with providerAPIServerServiceType: ClusterIP by using the Submariner add-on do one of the following methods:
If MetalLB is configured and the IP range is provided, do method 1. If MetalLB is not configured and Globalnet was enabled when setting up Submariner, do method 2.
Note: If a proxy is configured on the clusters, ensure that the No_Proxy list is configured based on the method that you use.
- If you use method 1 or method 2, add the api-server-exported-address to the No_Proxy list.
- If you do not use either method, add the service network of the peer managed cluster to the No_Proxy list.
- Method 1:
  If MetalLB is configured and the ocs-provider-server-load-balancer service has assigned IP addresses, add the following annotation (key-value pair) to the StorageCluster resource on both managed clusters:
  - Key: ocs.openshift.io/api-server-exported-address
  - Value: Endpoint of the <LoadBalancer ip>:50051.
  If the ocs-provider-server-load-balancer service is in the Pending state and has no assigned IP addresses, assign IP addresses to the LoadBalancer service. If no IP addresses are available, use method 2.
  To fetch the IP address of the ocs-provider-server-load-balancer service, run the following command:
```
oc get svc -n openshift-storage -o wide | grep provide
```
  Example output:
```
ocs-provider-server-load-balancer   LoadBalancer   172.30.213.146   172.200.88.109   50051:30506/TCP   6d   app=ocsProviderApiServer
```
  Once the annotation is added, the odf-info configMap will detect this endpoint, which the MCO will use to create the StorageClusterPeer.
  For example:
```
ocs.openshift.io/api-server-exported-address: '172.200.88.109:50051'
```
- Method 2:
  If you do not want to use MetalLB and no IP addresses are available, create a service export and use Submariner for cross-cluster communication between the two clusters. Do the following steps:
  1. Create a ServiceExport for the ocs-provider-server service.
```
apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceExport
metadata:
  name: ocs-provider-server
  namespace: openshift-storage
```
  2. Add the following annotation (key-value pair) to the StorageCluster resource on both managed clusters:
    
    Key: ocs.openshift.io/api-server-exported-address
    
    Value: <managerClusterName>.ocs-provider-server.openshift-storage.svc.clusterset.local:50051
    managedClusterName refers to the name used during cluster import into the hub cluster. For example, if rackm03 was the name provided, then:
    ocs.openshift.io/api-server-exported-address:rackm03.ocs-provider-server.openshift-storage.svc.clusterset.local:50051
    Command example to add the annotation:
    oc -n openshift-storage annotate storagecluster ocs-storagecluster 'ocs.openshift.io/api-server-exported-address="173.100.89.108:50051"'
    Where,
    
    openshift-storage: The namespace where Fusion Data Foundation is installed.
    
    ocs-storagecluster: The storage cluster name.
    
    key: ocs.openshift.io/api-server-exported-address
    
    value: "173.100.89.108:50051"
    Note: Either RHACM, MCE, or Submariner brings the ServiceExport CRD. So, you can create the ServiceExport after Submariner is installed for the clusters that do not have MCE installed.
    
    Once the annotation is added, the odf-info configMap will detect this endpoint, which the MCO will use to create the StorageClusterPeer.
Check whether a KlusterletAddonConfig resource already exists for both managed clusters on the RHACM hub cluster.
Run the following command on the RHACM hub cluster to verify that KlusterletAddonConfig resources exist for both managed clusters:
```
oc get klusterletaddonconfig -A
```
If the KlusterletAddonConfig resources for both managed clusters are listed in the output, proceed to the next step. Otherwise, create the missing resource by applying the following YAML configuration:

For example:
```
apiVersion: agent.open-cluster-management.io/v1
kind: KlusterletAddonConfig
metadata:
  name: <rack name>
  namespace: <rack name>
spec:
  clusterName: <rack name>
  clusterNamespace: <rack name>
  clusterLabels:
    name: <rack name>
    cloud: auto-detect
    vendor: auto-detect
  applicationManager:
    enabled: true
  policyController:
    enabled: true
  searchCollector:
    enabled: true
  certPolicyController:
    enabled: true
```
Remember: In the YAML configuration, provide the same cluster name that you get in the output of the following command:
```
oc get managedcluster
```
Create a disaster recovery policy (DRPolicy) for the managed clusters.
For instructions, see Creating a disaster recovery policy on hub cluster.
Optional: Create a VolumeReplicationClass (VRC) for the DRPolicy that is configured with a schedule other than the default replication interval of 5 minutes (such as 10 or 15 minutes).
For instructions, see Creating a VRC.
Verify whether the Regional-DR is successfully configured.
For instructions, see Verifying Regional-DR configuration.

What to do next

Validate Regional-DR with applications by following the instructions mentioned in Validating Regional-DR with applications.