Setting Up a Multi-Region Cluster in HashiCorp Nomad

Problem

Introduction:

A multi-region cluster in HashiCorp Nomad allows you to distribute workload scheduling and management across multiple regions, providing fault tolerance and improved availability for your applications. This knowledge base article provides a step-by-step guide on setting up a multi-region cluster in Nomad.

Prerequisites:

Before proceeding with the setup, ensure you have the following prerequisites:

1. Nomad: Install and configure the Nomad server and client nodes in each region.
2. Network Connectivity: Establish network connectivity between the nodes across regions, either through VPNs or direct network connections.

Procedure:

Step 1: Configure Nomad Server Nodes:

• On each Nomad server node, modify the server.hcl configuration file.
• Specify the region name using the region parameter, e.g., region = "india".
• Configure server addresses using the bind_addr parameter, e.g., bind_addr = "0.0.0.0".
• Set the bootstrap_expect parameter to the expected number of servers, e.g., bootstrap_expect = 3.
• Optionally, add any additional server configuration options as needed.

Below is the sample configuration for your reference -

name = "nomad-server"
region = "india"
datacenter = "dc1"
data_dir = "/opt/nomad"
bind_addr = "0.0.0.0"
server {
  enabled = true
  bootstrap_expect = 1
  server_join {
    retry_join = ["<ip-address of your server>"]
  }
}

Step 2: Configure Nomad Client Nodes:

• On each Nomad client node, modify the client.hcl configuration file.
• Specify the region name using the region parameter, e.g., region = "india".
• Configure server addresses using the servers parameter, providing the addresses of the Nomad servers in the respective region.
• Optionally, add any additional client configuration options as needed.

Below is the sample configuration for Nomad Client -

name = "nomad-client"
region = "india"
datacenter = "dc1"
data_dir = "/opt/nomad"
bind_addr = "0.0.0.0"
client {
  enabled = true
  server_join {
    retry_join = ["<ip-address of your nomad server>"]
  }
}

Step 3: Start Nomad Server and Client Nodes:

• Start the Nomad server nodes first, ensuring they are up and running. You can check the same using the below commands -

-> systemctl status nomad
-> nomad server members

• Start the Nomad client nodes, ensuring they can communicate with their respective server nodes. You can run the below command to check whether your client node is ready and federated with the Nomad server -

nomad node status

• Repeat the above steps of starting the Nomad server and client for each region.

Step 4: Configure the Authoritative Region:

The authoritative region is used to provide a single source of truth for ACL Policies, ACL Tokens, and Namespaces in multi-region, federated Nomad clusters. When used, all servers should have this configuration parameter set and non-authoritative regions will have the data replicated to them from the authoritative region.

You need to decide which region will be your authoritative region among multiple regions in your Nomad cluster. Here, I am considering "India" as an authoritative region.

• For making region = "india"as an authoritative region, please append the authoritative_region = "india"parameter in server.hclfile on the India region server node. Please see the below example -

name = "nomad-server"
region = "india"
datacenter = "dc1"
data_dir = "/opt/nomad"
bind_addr = "0.0.0.0"
server {
  enabled = true
  authoritative_region = "india"
  bootstrap_expect = 1
  server_join {
    retry_join = ["<ip-address of your server>"]
  }
}

• Restart the Nomad server agents to reflect the changes. You can use the below command for the same - 

systemctl restart nomad

Step 5: Configure the Nomad server of each region to connect and federate with the Authoritative Region:

• On the Non-Authoritative regions, please append the below two parameters in the Nomad server configuration so that these regions can federate with the authoritative region.
  • authoritative_region = <authoritative region name>
  • <ip of authoritative region:serf_wan_port>

Below is a sample example for your reference - 

name = "nomad-server"
region = "emea"
datacenter = "dc2"
data_dir = "/opt/nomad"
bind_addr = "0.0.0.0"
server {
  enabled = true
  bootstrap_expect = 1
  authoritative_region = "india"
  server_join {
    retry_join = ["<ip-address of same region>", "<ip of authoritative region:serf_wan_port"]
  }
}

• Restart the nomad server agent by using the below command - 

systemctl restart nomad

Step 6: Verify Cluster Communication:

1. Use the nomad server members command to verify that all nomad servers are correctly connected and communicating within the cluster.
2. Confirm that servers from all regions are listed in the output. Below is the sample output for your reference - 

Name               Address  Port Status Leader Raft Version Build     Datacenter Region
nomad-server.emea  *.*.*.*  4648 alive  true   3            1.5.3+ent dc2        emea
nomad-server.india *.*.*.*  4648 alive  true   3            1.5.3+ent dc1        india

Conclusion:

By following the steps outlined in this knowledge base article, you can successfully set up a multi-region cluster in HashiCorp Nomad. This cluster configuration allows for workload distribution, fault tolerance, and improved availability across multiple regions, enhancing the resilience of your applications. For more advanced configuration options and in-depth details, refer to the official Nomad documentation.

Reference:

Multi-Region Federation - https://developer.hashicorp.com/nomad/tutorials/manage-clusters/federation

Multi-Region Deployments - https://developer.hashicorp.com/nomad/tutorials/manage-clusters/multiregion-deployments