IBM Support

Upgrading vault nodes without using automated upgrade feature of autopilot

Troubleshooting


Problem

Problem statement

Vault Enterprise provides automated version upgrades with the autopilot feature when using Integrated Storage. This feature, which was introduced in v1.11.0, allows you to start new Vault nodes alongside the older version ones and automatically switch to the new nodes after they reach quorum. But there might be cases or scenarios where adding new nodes is not desired or possible and customers opt to upgrade Vault using their own strategy like AMI upgrade or upgrading vault in the same VM. When upgrading Vault this way, customer will lose quorum causing downtime for the cluster until the active node of the cluster is upgraded.

For example, when testing with 3 nodes cluster & upgrading the vault from 1.11.0 to 1.11.3, after one standby node from the cluster is upgraded from 1.11.0 to 1.11.3 & then you check the autopilot state like shown here we can see that standby node node_vaults0 came as non-voter

$ vault operator raft autopilot state
#Output:-
  #Upgrade Info:
    #Status: await-new-voters
    #Target Version: 1.11.3
    #Target Version Voters:
    #Target Version Non-Voters: node_vaults0
    #Other Version Voters: node_vaults1, node_vaults2
    #Other Version Non-Voters:

When disable_upgrade_migration flag is false which is set by default, when you add a new node to the cluster, it gets added as a non-voter and the status of autopilot upgrade changes to await-new-voters. Since the new nodes or upgraded vault nodes in your case will come back to the cluster as non-voter , you lose quorum until you upgrade the leader. Even though you will lose quorum in the process, you should have a functioning cluster after you upgrade all the nodes.

You will see logs similar to this in the last Active Vault Node when the quorum is lost

Sep 15 10:04:52 vaults1 vault[3943]: 2022-09-15T10:04:52.284Z [WARN] storage.raft: failed to contact quorum of nodes, stepping down
Sep 15 10:04:52 vaults1 vault[3943]: 2022-09-15T10:04:52.111Z [DEBUG] core.cluster-listener: creating rpc dialer: address=192.168.50.42:8201 alpn=raft_storage_v1 host=raft-023bf9ba-536f-a7ee-14f1-16339682d2b0
Sep 15 10:04:52 vaults1 vault[3943]: 2022-09-15T10:04:52.112Z [ERROR] storage.raft: failed to heartbeat to: peer=192.168.50.42:8201 error="dial tcp 192.168.50.42:8201: connect: connection refused"
Sep 15 10:04:52 vaults1 vault[3943]: 2022-09-15T10:04:52.284Z [WARN] storage.raft: failed to contact: server-id=node_vaults2 time=2.524339216s
Sep 15 10:04:52 vaults1 vault[3943]: 2022-09-15T10:04:52.284Z [WARN] storage.raft: failed to contact quorum of nodes, stepping down
Sep 15 10:04:52 vaults1 vault[3943]: 2022-09-15T10:04:52.284Z [INFO] storage.raft: entering follower state: follower="Node at 192.168.50.41:8201 [Follower]" leader-address= leader-id=
Sep 15 10:04:52 vaults1 vault[3943]: 2022-09-15T10:04:52.285Z [INFO] storage.raft: aborting pipeline replication: peer="{Voter node_vaults0 192.168.50.40:8201}"
Sep 15 10:04:52 vaults1 vault[3943]: 2022-09-15T10:04:52.285Z [WARN] core: leadership lost, stopping active operation
Sep 15 10:04:52 vaults1 vault[3943]: 2022-09-15T10:04:52.285Z [ERROR] replication.index.perf: failed to persist checkpoint: error="failed to persist: leadership lost while committing log"
Sep 15 10:04:52 vaults1 vault[3943]: 2022-09-15T10:04:52.285Z [INFO] core: pre-seal teardown starting
Sep 15 10:04:52 vaults1 vault[3943]: 2022-09-15T10:04:52.285Z [TRACE] core.snapshotmgr: shutting down automatic snapshots
Sep 15 10:04:52 vaults1 vault[3943]: 2022-09-15T10:04:52.285Z [DEBUG] storage.raft.autopilot: state update routine is now stopped
Sep 15 10:04:52 vaults1 vault[3943]: 2022-09-15T10:04:52.286Z [DEBUG] storage.raft.autopilot: autopilot is now stopped
Sep 15 10:04:52 vaults1 vault[3943]: 2022-09-15T10:04:52.286Z [ERROR] replication.index.local: failed to persist checkpoint: error="failed to persist: node is not the leader"
Sep 15 10:04:52 vaults1 vault[3943]: 2022-09-15T10:04:52.286Z [ERROR] replication.index.periodic: failed to save checkpoint:
Sep 15 10:04:52 vaults1 vault[3943]: error=
Sep 15 10:04:52 vaults1 vault[3943]: | 2 errors occurred:
Sep 15 10:04:52 vaults1 vault[3943]: | * failed to persist checkpoint: failed to persist: leadership lost while committing log
Sep 15 10:04:52 vaults1 vault[3943]: | * failed to persist checkpoint: failed to persist: node is not the leader
Sep 15 10:04:52 vaults1 vault[3943]: |
Sep 15 10:04:52 vaults1 vault[3943]:

 

Solution : Disable automated upgrade in the autopilot

When disable_upgrade_migration flag is set to true, then when you add a new node to the cluster or when you upgrade your vault node, the node gets added back to the cluster as voter so you will not lose the quorum of the cluster during the process of upgrading all the standby nodes one by one. Upgrading your cluster this way, you would do the following:

  • Set the disable_upgrade_migration flag to true on the leader node using cli like this vault operator raft autopilot set-config -disable-upgrade-migration=true
$ vault operator raft autopilot set-config -disable-upgrade-migration=true
$ vault operator raft autopilot get-config
  #Key                                 Value
  #---                                 -----
  #Cleanup Dead Servers                false
  #Last Contact Threshold              10s
  #Dead Server Last Contact Threshold  24h0m0s
  #Server Stabilization Time           10s
  #Min Quorum                          0
  #Max Trailing Logs                   1000
  #Disable Upgrade Migration           true
  • Upgrade standby vault nodes one-by-one and make sure they join the cluster back as voter before moving on to the new node
  • Upgrade the active node last and properly shut down the node by issuing a SIGINT or SIGTERM. Replace the binary and start the node. Detailed instructions are found at Vault HA upgrades

Important

DO NOT attempt to issue a step-down operation at any time during the upgrade process.

 

References 

Vault HA upgrades

Raft Upgrade Automation

Autopilot

 

 

 

 

 

 

 

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB77","label":"Automation Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSLC9Y2","label":"IBM Vault Self-Managed"},"ARM Category":[{"code":"a8mgJ0000000E6EQAU","label":"Vault-\u003EOther"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"1.14.10;1.14.11;1.14.12;1.14.13;1.14.4;1.14.5;1.14.6;1.14.7;1.14.8;1.14.9;1.15.0;1.15.1;1.15.10;1.15.11;1.15.12;1.15.13;1.15.14;1.15.15;1.15.16;1.15.2;1.15.3;1.15.4;1.15.5;1.15.6;1.15.7;1.15.8;1.15.9;1.16.0;1.16.1;1.16.10;1.16.11;1.16.12;1.16.13;1.16.14;1.16.15;1.16.16;1.16.17;1.16.18;1.16.19;1.16.2;1.16.20;1.16.21;1.16.22;1.16.23;1.16.24;1.16.25;1.16.26;1.16.27;1.16.28;1.16.29;1.16.3;1.16.30;1.16.31;1.16.4;1.16.5;1.16.6;1.16.7;1.16.8;1.16.9;1.17.0;1.17.1;1.17.10;1.17.11;1.17.12;1.17.13;1.17.14;1.17.15;1.17.16;1.17.17;1.17.18;1.17.2;1.17.3;1.17.4;1.17.5;1.17.6;1.17.7;1.17.8;1.17.9;1.18.0;1.18.1;1.18.10;1.18.11;1.18.12;1.18.13;1.18.14;1.18.15;1.18.2;1.18.3;1.18.4;1.18.5;1.18.6;1.18.7;1.18.8;1.18.9;1.19.0;1.19.1;1.19.10;1.19.11;1.19.12;1.19.13;1.19.14;1.19.15;1.19.16;1.19.17;1.19.2;1.19.3;1.19.4;1.19.5;1.19.6;1.19.7;1.19.8;1.19.9;1.20.0;1.20.1;1.20.10;1.20.11;1.20.2;1.20.3;1.20.4;1.20.5;1.20.6;1.20.7;1.20.8;1.20.9;1.21.0;1.21.1;1.21.2;1.21.3;1.21.4;1.21.5;1.21.6;2.0.0;2.0.1;4.2.0-5.4.0"}]

Historical Number

7122445204755

Document Information

Modified date:
25 May 2026

UID

ibm17264470