Troubleshooting
Problem
Introduction
The Key-Value (KV) secrets engine stores and manages sensitive data such as passwords, API keys, and certificates within Vault’s physical storage. Vault supports two versions of the KV secrets engine:
KV Version 1 (KV v1): A simple key-value store without versioning capabilities.
KV Version 2 (KV v2):The key/value (
kv) secrets engine stores and versions arbitrary static secrets stored in Vault physical storage.- The
kvv2 plugin uses soft deletes to make data inaccessible while allowing data recovery. When an entry is permanently deleted, Vault purges the underlying version data and marks the key metadata as destroyed.
Many existing Vault deployments initially use KV v1 but may later require upgrading to KV v2 to take advantage of these additional features.
This document outlines the process of upgrading the KV secrets engine mount at the TEST path from version 1 to version 2. It describes the steps taken, the temporary unavailability of the mount during the upgrade, and provides an estimate of the upgrade duration based on observed data.
Prerequisite:
Update ACL rules
The kv v2 plugin uses different API path prefixes than kv v1. You must upgrade the relevant ACL policies before upgrading the plugin by changing v1 paths for read, write, or update policies to include the v2 path prefix, data/.
For example, the following kv v1 policy:
path "shared/dev/square-api/*" {
capabilities = ["create", "update", "read"]
}
becomes:
path "shared/data/dev/square-api/*" {
capabilities = ["create", "update", "read"]
}Steps Taken
-
Current state before upgrade:
- Mount path: TEST
- KV version: v1
- Number of secrets stored: ~10,002
Writing to TEST/secret10000 ...
Success! Data written to: TEST/secret10000
/tmp $ vault read sys/mounts/TEST/tune
Key Value
--- -----
default_lease_ttl 768h
description n/a
force_no_cache false
max_lease_ttl 768h
options map[version:1]
/tmp $ vault secrets listPath Type Accessor Description
---- ---- -------- -----------
TEST/ kv kv_79f9cd24 n/a
/tmp $ vault kv list TEST | wc -l
10002
-
Enable versioning (upgrade) on mount:
vault kv enable-versioning TEST/ This triggered Vault to convert the existing KV v1 backend into KV v2.
Vault logged:
Success! Tuned the secrets engine at: TEST/ -
Post upgrade verification:
Check tune options again:
/tmp $ vault read sys/mounts/TEST/tune
Key Value
--- -----
default_lease_ttl 768h
description n/a
force_no_cache false
max_lease_ttl 768h
options map[version:2]
What happens during upgrade?
- Vault performs key-by-key migration of existing secrets to the new versioned backend structure.
- This process makes the secrets engine unavailable temporarily until migration completes.
Error :"Upgrading from non-versioned to versioned data. This backend will be unavailable for a brief period and will resume service shortly." - Vault logs detail the progress of the migration, showing how many keys have been upgraded.
Logs:
The progress of the upgrade can be monitored in Vault's debug logs, which provide real-time insights into the number of keys being processed.
|
2025-09-12T20:48:51.061Z [INFO] core: mount tuning of options: path=TEST/ options=map[version:2] 2025-09-12T20:48:51.063Z [TRACE] core: adding local paths: paths=[] 2025-09-12T20:48:51.063Z [TRACE] core: adding write forwarded paths: paths=[] 2025-09-12T20:48:51.065Z [INFO] secrets.kv.kv_79f9cd24: collecting keys to upgrade 2025-09-12T20:48:51.066Z [INFO] secrets.kv.kv_79f9cd24: done collecting keys: num_keys=10001 2025-09-12T20:48:51.066Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=0/10001 2025-09-12T20:48:51.716Z [TRACE] activity: writing segment on timer expiration 2025-09-12T20:48:52.005Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=500/10001 2025-09-12T20:48:52.953Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=1000/10001 2025-09-12T20:48:53.914Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=1500/10001 2025-09-12T20:48:54.868Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=2000/10001 2025-09-12T20:48:56.126Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=2500/10001 2025-09-12T20:48:57.468Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=3000/10001 2025-09-12T20:48:58.807Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=3500/10001 2025-09-12T20:49:00.132Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=4000/10001 2025-09-12T20:49:01.455Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=4500/10001 2025-09-12T20:49:02.799Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=5000/10001 2025-09-12T20:49:04.189Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=5500/10001 2025-09-12T20:49:05.503Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=6000/10001 2025-09-12T20:49:06.881Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=6500/10001 2025-09-12T20:49:08.176Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=7000/10001 2025-09-12T20:49:09.474Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=7500/10001 2025-09-12T20:49:10.811Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=8000/10001 2025-09-12T20:49:12.220Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=8500/10001 2025-09-12T20:49:13.491Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=9000/10001 2025-09-12T20:49:14.778Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=9500/10001 2025-09-12T20:49:16.066Z [DEBUG] secrets.kv.kv_79f9cd24: upgrading keys: progress=10000/10001 2025-09-12T20:49:16.066Z [INFO] secrets.kv.kv_79f9cd24: upgrading keys finished |
Recommendations
Take a snapshot backup before performing the upgrade to ensure you can recover secrets in case of any unexpected issues.
Inform users in advance about the expected downtime, as the mount path will be inaccessible during the upgrade.
Plan the upgrade during a maintenance window to minimize impact on production workloads.
If the mount contains a large number of secrets or large-sized secrets, expect the upgrade to take more time.
Monitor Vault debug logs to track real-time progress of the upgrade, including key migration status.
Test the upgrade process in a non-production environment to validate behavior and estimate duration before applying it to production.
Additional Information
Document Location
Worldwide
Historical Number
44684220257555
Was this topic helpful?
Document Information
Modified date:
16 March 2026
UID
ibm17264864