Best Practices for AWS Cloud auto-join

Problem

Introduction

Within the Vault configuration file, the retry_join stanza defines a set of connection details for another node in the cluster, which is used to help nodes locate a leader in order to join a cluster. When an auto_join value is provided, Vault will automatically attempt to discover and resolve potential Raft leader addresses using go-discover for a cloud auto_join configuration. This article will help describe best practices in configuring auto_join with AWS.

Best Practices

• Review Vault architecture to declare the appropriate auto_join_scheme
  • http or https
• The auto_join feature works on the go-discover library, which will search the nodes based on their IP and cloud tags.
• Ensure that all nodes and potential nodes are able to communicate with each other over the API and cluster ports
  • Default ports 8200 and 8201
• Confirmation of connectivity between nodes can be done by running the following curl call
  

curl -kv http:// ip:8200, 8201

Below is an example configuration for the auto_join stanza using the http scheme:

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-1"
  retry_join {
    auto_join = "provider=aws addr_type=public_v4 tag_key=auto_join tag_value=vault-raft-cluster region=us-east-1"
    auto_join_scheme = "http"
  }
}

• Ensure that the AWS Security Groups and Network Access Control Lists in use allow traffic between the tagged EC2 instances on the API and cluster ports.

Verify discovery of nodes from the Vault operational logs:

[INFO] core: [DEBUG] discover-aws: Found ip addresses: []"
[INFO] core: [DEBUG] discover-aws: Found 0 reservations"
"Sep 29 16:28:19 vault: 2021-09-29T16:28:19.307Z [INFO] core: [INFO] discover-aws: Filter instances with auto_join=vault-raft-cluster"

You can also leverage the AWS CLI to verify if the tagged instances are getting discovered. This will list all EC2 instances with the tag defined:

aws ec2 describe-instances --filters "Name=tag:auto_join,Values=vault-raft-cluster" --region ap-southeast-1 | jq '.Reservations[].Instances[].PublicIpAddress'

Keep in mind that https cloud auto-join you will need to ensure that the range of IP addresses is added to the SAN list of the SSL certificate which is used in the architecture. Also ensure that DNS resolution is happening properly amongst all the nodes and FQDNs.

Additional Information

• Raft Deployment Tutorial: https://learn.hashicorp.com/tutorials/vault/raft-deployment-guide?in=vault/raft#raft-configuration
• Raft Auto-Join Tutorial: https://learn.hashicorp.com/tutorials/vault/raft-storage-aws?in=vault/raft#cloud-auto-join
• Go Discover for Cloud Providers: https://github.com/hashicorp/go-discover