Deploying HashiCorp Vault on IBM z/OS Container Extensions

This comprehensive guide provides step-by-step instructions for deploying a secure, highly-available 3-node HashiCorp Vault Enterprise cluster on IBM z/OS Container Extensions (zCX).

Prerequisites

Complete the following requirements before starting your deployment:

Review the Sealing and unsealing documentation to understand Vault's security model.
Note: Your deployment will remain inaccessible until properly unsealed.
Docker CLI installation: Install the Docker CLI to manage containers during deployment.
Access Permissions: Confirm you have sufficient privileges to:
- Deploy and manage Docker containers.
- Execute vault operator commands for cluster management.
- Configure and deploy load balancer settings.
- Prepare the following certificates for secure communication between Vault clients, HAProxy, and cluster nodes. You need to have CA Signed certificates.
- vault.pem - Server's TLS certificate for inter-node communication. The certificate's Common Name (CN) should match the zCX host IP or hostname.
- vault.key - Private key for Vault encryption operations.
ca.pem - Certificate Authority for mutual TLS validation, contains the full CA certificate chain, the root CA certificate plus all intermediate CA certificates to establish trust.
- Vault CLI installation (optional): Install the Vault CLI for local command execution. Alternatively, use API calls via command line.

Procedure

Create persistent Docker volumes to preserve Vault and HAProxy configurations across cluster nodes.
1. Create Vault configuration volume:
```
docker volume create vault-config
```
2. Create HAProxy configuration volume:
```
docker volume create haproxy-config
```
3. Verify volume creation:
```
docker volume ls | grep config
```
Create a local directory structure to organize deployment files and maintain version control.
Set up the following directory hierarchy in local system:
```
vault-deploy/
  └── vault.env
  └── vault.Dockerfile
  └── vault.compose.yml
  └── local-config/
      ├── certs/
      └── hcl/
proxy-deploy/
  └── local-config/
  └── proxy.compose.yml
shared-deploy/
  └── cluster-env.yml
```
The subsequent steps assume the following file structure:
1. Shared Docker Compose files in shared-deploy/.
2. Vault Docker Compose and Dockerfile in vault-deploy/.
3. Vault license file (vault-license.hclic) in vault-deploy/local-config/.
4. TLS certificates in vault-deploy/local-config/certs/.
5. Vault node configuration files in vault-deploy/local-config/hcl/.
6. Load balancer Docker Compose and Dockerfile in proxy-deploy/.
7. Load balancer configuration files in proxy-deploy/local-config/.

Create a centralized environment file at vault-deploy/vault.env to define common Vault environment variables:


# Mount path for vault-config volume
VAULT_CONFIG="/vault/config"

# Path to license file
VAULT_LICENSE_PATH="${VAULT_CONFIG}/vault-license.hclic"

# Custom variable to select node-specific config file
NODE_IDX=""

Create a Docker Compose file at shared-deploy/cluster-env.yml to define the cluster environment, including persistent volumes and a shared network for container communication:

Template:


networks:
  <resource_name>:
    name: <custom_network_name>
    driver: bridge
    ipam:
      config:
        - subnet: "<custom_subnet>"
          ip_range: "<IP_range_for_nodes>"
          gateway: "<gateway_IP>"

volumes:
  vault-config:
    external: true
    name: vault-config
  haproxy-config:
    external: true
    name: haproxy-config

Example configuration:


networks:
  vault_cluster:
    name: vault-network
    driver: bridge
    ipam:
      config:
        - subnet: "192.168.42.128/25"
          ip_range: "192.168.42.128/25"
          gateway: "192.168.42.129"

volumes:
  vault-config:
    external: true
    name: vault-config
  haproxy-config:
    external: true
    name: haproxy-config

Create a base Docker Compose file at vault-deploy/vault.compose.yml to define the build process for your Vault nodes:

Template:


name: <project_name>
services:
  <service_name>:
    volumes:
      - vault-config:/vault/config
    env_file:
      - path: <relative_path_to_env_settings>
        required: true
    networks:
      - <cluster_network_name>
    build:

include:
  - path: <path_to_cluster_env_file>

Example configuration:


name: zcx
services:
  vault-base:
    volumes:
      - vault-config:/vault/config
    env_file:
      - path: ./vault.env
        required: true
    networks:
      - vault_cluster
    build:

include:
  - path: ../shared-deploy/cluster-env.yml

Create a Docker file at vault-deploy/vault.Dockerfile that performs the following operations:

Copies configuration files and certificates to persistent storage.
Creates necessary directories for Vault data and plugins.
Sets appropriate ownership and permissions.
Starts Vault with node-specific configuration using environment variables.

Example configuration:


# syntax=docker/dockerfile:1
FROM hashicorp/vault-enterprise:<version_tag>

# Copy configuration and license files
COPY local-config/hcl/vault-*.hcl     /vault/config/hcl/
COPY local-config/vault-license.hclic /vault/config/vault-license.hclic

# Copy TLS certificates
COPY local-config/certs/vault.pem /vault/config/certs/vault.pem
COPY local-config/certs/vault.key /vault/config/certs/vault.key
COPY local-config/certs/ca.pem    /vault/config/certs/ca.pem

# Create required directories
RUN mkdir -p /vault/data/ /vault/plugins/ /vault/plugins/tmp

# Set Vault ownership for all Vault-related files
RUN chown -R vault:vault /vault

# Configure certificate file permissions
RUN chown root:vault /vault/config/certs/vault.key && \
    chmod 0644 /vault/config/certs/vault.pem && \
    chmod 0644 /vault/config/certs/ca.pem && \
    chmod 0640 /vault/config/certs/vault.key

# Start Vault with node-specific configuration
CMD vault server -config=${VAULT_CONFIG}/hcl/vault-${NODE_IDX}.hcl

Update vault.compose.yml to configure the base build and define individual node configurations.

Update vault.compose.yml to reference the Docker file:


name: zcx
services:
  vault-base:
    volumes:
      - vault-config:/vault/config
    env_file:
      - path: ./vault.env
        required: true
    networks:
      - vault_cluster
    build:
      context: .
      dockerfile: vault.Dockerfile

include:
  - path: ../shared-deploy/cluster-env.yml

Extend the base service to create configurations for each cluster node:

Set the VAULT_ADDR and NODE_IDX environment variables to customize each node's behavior:


name: zcx
services:
  vault-base:
    # ... (base configuration from above)

  vault-1:
    extends:
      service: vault-base
    container_name: zcx-vault-1
    hostname: zcx-vault-1
    environment:
      VAULT_ADDR: https://zcx-vault-1:8200
      NODE_IDX:   "1"

  vault-2:
    extends:
      service: vault-base
    container_name: zcx-vault-2
    hostname: zcx-vault-2
    environment:
      VAULT_ADDR: https://zcx-vault-2:8200
      NODE_IDX:   "2"

  vault-3:
    extends:
      service: vault-base
    container_name: zcx-vault-3
    hostname: zcx-vault-3
    environment:
      VAULT_ADDR: https://zcx-vault-3:8200
      NODE_IDX:   "3"

include:
  - path: ../shared-deploy/cluster-env.yml

Create individual HCL configuration files (vault-N.hcl) for each Vault node using the following template.


ui = true
disable_mlock = true
license_path = "<path_to_license_file_on_cluster>"

# Configure the non-loopback interface
api_addr     = "https://<container_name>:8200"
cluster_addr = "https://<container_name>:8201"
cluster_name = "<custom_name_for_cluster>"

# Listener configuration
listener "tcp" {
  address            = "[::]:8200"
  tls_disable        = "false"
  tls_cert_file      = "<path_to_cert_files>/vault.pem"
  tls_key_file       = "<path_to_cert_files>/vault.key"
  tls_client_ca_file = "<path_to_cert_files>/ca.pem"
}

# Plugin configuration
plugin_directory = "<path_to_plugin_dir_on_container>"
plugin_tmpdir    = "<path_to_tmp_dir_on_container>"

# Integrated storage with Raft
storage "raft" {
  path    = "/vault/data"
  node_id = "<container_name>"

  # Rejoin configuration for cluster node N+1
  retry_join {
    leader_api_addr    = "https://<alt_container_name_1>:8200"
    tls_disable        = "false"
    tls_cert_file      = "<path_to_cert_files>/vault.pem"
    tls_key_file       = "<path_to_cert_files>/vault.key"
    tls_client_ca_file = "<path_to_cert_files>/ca.pem"
  }

  # Rejoin configuration for cluster node N+2
  retry_join {
    leader_api_addr    = "https://<alt_container_name_2>:8200"
    tls_disable        = "false"
    tls_cert_file      = "<path_to_cert_files>/vault.pem"
    tls_key_file       = "<path_to_cert_files>/vault.key"
    tls_client_ca_file = "<path_to_cert_files>/ca.pem"
  }
}

Important configuration notes:

Set tls_disable to false to enforce TLS communication within the cluster.
Configure UI, license path, and plugin directory settings during initial setup.
Define plugin locations early to avoid cluster restarts later, even if not immediately using custom plugins.

Example configuration (vault-1.hcl):


ui = true
disable_mlock = true
license_path = "/vault/config/license.hclic"

# Configure the non-loopback interface
api_addr     = "https://zcx-vault-1:8200"
cluster_addr = "https://zcx-vault-1:8201"
cluster_name = "zcx-vault"

# Plugin configuration
plugin_directory = "/vault/plugins/"
plugin_tmpdir    = "/vault/plugins/tmp"

# Listener configuration
listener "tcp" {
  address            = "[::]:8200"
  tls_disable        = "false"
  tls_cert_file      = "/vault/config/certs/vault.pem"
  tls_key_file       = "/vault/config/certs/vault.key"
  tls_client_ca_file = "/vault/config/certs/ca.pem"
}

# Integrated storage with Raft
storage "raft" {
  path    = "/vault/data"
  node_id = "zcx-vault-1"

  # Rejoin configuration for vault-2
  retry_join {
    leader_api_addr    = "https://zcx-vault-2:8200"
    tls_disable        = "false"
    tls_cert_file      = "/vault/config/certs/vault.pem"
    tls_key_file       = "/vault/config/certs/vault.key"
    tls_client_ca_file = "/vault/config/certs/ca.pem"
  }

  # Rejoin configuration for vault-3
  retry_join {
    leader_api_addr    = "https://zcx-vault-3:8200"
    tls_disable        = "false"
    tls_cert_file      = "/vault/config/certs/vault.pem"
    tls_key_file       = "/vault/config/certs/vault.key"
    tls_client_ca_file = "/vault/config/certs/ca.pem"
  }
}

Initialize Vault by starting a single node first, then bring up and unseal the remaining nodes.

Start the first Vault service:


docker compose \
  -f vault-deploy/vault.compose.yml up vault-1 --build --detach

Initialize Vault and save the output to a secure location:
```
docker exec -it zcx-vault-1 \
  vault operator init -format=json > /secure/location/vault-init.json
```
Important: Store vault-init.json securely. It contains the root token and unseal keys required for cluster operations.

Unseal the first Vault node using the generated unseal keys:


for token in $(
    cat /secure/location/vault-init.json | \
    jq -r '.unseal_keys_b64[0:3] | join(" ")'
  ) ; do
    docker exec -it zcx-vault-1 vault operator unseal $token
  done

Check the seal status to confirm successful initialization:


docker exec -it zcx-vault-1 vault status

Expected output:


Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.21.2+ent
Build Date              2026-01-06T16:58:57Z
Storage Type            raft
Cluster Name            zcx-vault
Cluster ID              520515c4-887f-4ace-b267-8625cfd5fb43
HA Enabled              true
HA Cluster              https://zcx-vault-1:8201
HA Mode                 active
Active Since            2026-02-04T02:51:25.540535695Z
Raft Committed Index    9945
Raft Applied Index      9945

Start vault-2:


docker compose \
  -f vault-deploy/vault.compose.yml up vault-2 --build --detach

Unseal vault-2:


for token in $(
    cat /secure/location/vault-init.json | \
    jq -r '.unseal_keys_b64[0:3] | join(" ")'
  ) ; do
    docker exec -it zcx-vault-2 vault operator unseal $token
  done

Repeat for vault-3 and any additional nodes.

Create a Docker Compose file at proxy-deploy/proxy.compose.yml for the load balancer using the same network configuration as the Vault cluster:


name: haproxy
services:
  vault-lb:
    image: ibmz-hc-registry.ngrok.dev/haproxy:3.2
    volumes:
      - haproxy-config:/usr/local/etc/haproxy
    networks:
      - vault_cluster
    ports:
      - "8300:8200"
      - "8404:8404"

include:
  - path: ../shared-deploy/cluster-env.yml

Create an HAProxy configuration file at proxy-deploy/local-config/haproxy.cfg that defines load balancing behavior for your Vault cluster.

This configuration enables:

TLS termination for external clients.
Health checks for Vault nodes.
Layer-4 TCP forwarding with round-robin distribution.


global
  maxconn 4096
  log stdout format raw local0

defaults
  mode tcp                     # TCP mode for Layer-4 forwarding
  timeout connect 5s
  timeout client 1m
  timeout server 1m
  log global

# Stats Page (HTTPS)
frontend stats
  bind *:8404 ssl crt /usr/local/etc/haproxy/haproxy.pem
  mode http
  stats enable
  stats uri /stats
  stats refresh 10s
  stats admin if TRUE

# Vault API Frontend (TLS Passthrough)
frontend vault_api
  bind *:8200                 # Listen on Vault API port
  mode tcp                    # Layer-4 forwarding to preserve TLS
  default_backend vault_nodes

# Vault Nodes Backend
backend vault_nodes
  mode tcp
  balance roundrobin          # Simple load distribution among nodes
  option tcp-check            # Health check at TCP level
  server node1 zcx-vault-1:8200 check
  server node2 zcx-vault-2:8200 check
  server node3 zcx-vault-3:8200 check

Deploy the load balancer.

Copy the HAProxy configuration to the persistent volume using a temporary container:


docker run \
  --name haproxy-tmp \
  -v haproxy-config:/usr/local/etc/haproxy \
  alpine sh -c "sleep 1" \
  && docker cp \
    proxy-deploy/local-config/haproxy.cfg \
    haproxy-tmp:/usr/local/etc/haproxy/haproxy.cfg \
  && docker rm -f haproxy-tmp

Deploy the HAProxy load balancer:


docker compose \
  -f proxy-deploy/proxy.compose.yml up vault-lb --build --detach

Verify the deployment.

Get the IP address of the first Vault node:


docker inspect -f \
  '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' zcx-vault-1

Set environment variables for direct access:


export VAULT_ADDR=https://<vault-1-IP>:8200
export VAULT_TOKEN=<root_token_from_vault_init.json>

List all cluster nodes to confirm that the nodes are operational:

Using CLI:


vault operator raft list-peers

Expected output:


Node           Address             State       Voter
----           -------             -----       -----
zcx-vault-1    zcx-vault-1:8201    leader      true
zcx-vault-2    zcx-vault-2:8201    follower    true
zcx-vault-3    zcx-vault-3:8201    follower    true

Using API:


curl \
  --request GET \
  --header "X-Vault-Token: ${VAULT_TOKEN}" \
  ${VAULT_ADDR}/v1/sys/storage/raft/configuration | jq .data.config.servers

Expected output:


[
  {
    "node_id": "zcx-vault-1",
    "address": "zcx-vault-1:8201",
    "leader": true,
    "protocol_version": "3",
    "voter": true
  },
  {
    "node_id": "zcx-vault-2",
    "address": "zcx-vault-2:8201",
    "leader": false,
    "protocol_version": "3",
    "voter": true
  },
  {
    "node_id": "zcx-vault-3",
    "address": "zcx-vault-3:8201",
    "leader": false,
    "protocol_version": "3",
    "voter": true
  }
]

Test HAProxy health by opening the stats page in your browser:
```
https://<proxy_url>:<proxy_port>/stats
```

Switch to using the load balancer for all Vault operations:


export VAULT_PROXY_ADDR=https://<proxy_url>:<proxy_port>
unset VAULT_ADDR

Confirm the load balancer correctly routes traffic to the Vault cluster:

Using CLI:


vault read -format json sys/storage/raft/configuration | jq .data

Expected output:


{
  "config": {
    "index": 0,
    "servers": [
      {
        "address": "zcx-vault-1:8201",
        "leader": true,
        "node_id": "zcx-vault-1",
        "protocol_version": "3",
        "voter": true
      },
      {
        "address": "zcx-vault-2:8201",
        "leader": false,
        "node_id": "zcx-vault-2",
        "protocol_version": "3",
        "voter": true
      },
      {
        "address": "zcx-vault-3:8201",
        "leader": false,
        "node_id": "zcx-vault-3",
        "protocol_version": "3",
        "voter": true
      }
    ]
  }
}

Using API:


curl \
  --request GET \
  --header "X-Vault-Token: ${VAULT_TOKEN}" \
  ${VAULT_PROXY_ADDR}/v1/sys/storage/raft/configuration | jq .data

Expected output:


{
  "config": {
    "servers": [
      {
        "node_id": "zcx-vault-1",
        "address": "zcx-vault-1:8201",
        "leader": true,
        "protocol_version": "3",
        "voter": true
      },
      {
        "node_id": "zcx-vault-2",
        "address": "zcx-vault-2:8201",
        "leader": false,
        "protocol_version": "3",
        "voter": true
      },
      {
        "node_id": "zcx-vault-3",
        "address": "zcx-vault-3:8201",
        "leader": false,
        "protocol_version": "3",
        "voter": true
      }
    ],
    "index": 0
  }
}

Additional resources

Category	Resources
HashiCorp Vault documentation - Configuration	Vault configuration parameters - Complete reference for all Vault configuration options. Integrated storage (raft) - Detailed Raft storage backend configuration. Listener configuration - TCP listener and TLS settings.
HashiCorp Vault documentation - Operations	Seal/unseal concepts - Understanding Vault's sealing mechanism. CLI command: operator init - Initialize a new Vault server. CLI command: operator unseal - Unseal a Vault server. CLI command: operator raft - Manage Raft cluster operations.
HashiCorp Vault documentation - High availability	High availability concepts - Understanding Vault HA architecture. Production hardening - Best practices for production deployments.
IBM z/OS resources	IBM z/OS container extensions (zCX) - Official zCX documentation. Docker on zCX - Docker implementation on z/OS.
HAProxy documentation	HAProxy configuration manual - Complete HAProxy configuration reference. HAProxy best practices - Recommended configuration patterns.