Steps

Introduction

HashiCorp Vault provides detailed operational logs that are vital for debugging authentication flows and internal cluster health. While these are typically defined in a configuration file, administrators can use the Vault API to modify log levels directly in the server's memory. This allows for immediate, high-verbosity troubleshooting without requiring a service restart or a permanent configuration change.

Expected Outcome

After following this guide, you will be able to:

• Identify the current log level of the Vault core and its sub-systems as they exist in active memory.

• Temporarily shift the server or specific sub-loggers into trace or debug mode.

• Revert the server memory to standard levels once troubleshooting is complete.

Prerequisites

• Vault Token: A token with read and update capabilities on the sys/loggers path.

• Network Access: Connectivity to the Vault API port (8200).

• Tools: curl and jq installed on your machine.

• Environment Setup: Set your token as an environment variable to simplify the commands:

  export VAULT_TOKEN=<your_vault_token>

Use Case

• On-the-fly Debugging: You need to see verbose logging immediately, but do not want to restart the cluster or leave trace mode on permanently.

• Incident Response: Quickly quieting a "noisy" server that is filling up disk space with debug logs by pushing an info level change directly to memory.

Procedure

1. See the Server's Current Log-Level

To see all the individual log levels for every subsystem currently running in memory:

curl -sk --header "X-Vault-Token: $VAULT_TOKEN" \
    https://127.0.0.1:8200/v1/sys/loggers | jq -r '.data'

Understanding the data Output

The command above returns a map of sub-loggers. Key components include:

If you only want to see the base level (the core engine), use this targeted command:

curl -sk --header "X-Vault-Token: $VAULT_TOKEN" \
    https://127.0.0.1:8200/v1/sys/loggers | jq -r '.data.core'

2. Update the Log-Level in Memory

To increase verbosity across the entire server to trace (the most detailed level) or debug:

curl -sk --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"level": "trace"}' \
    https://127.0.0.1:8200/v1/sys/loggers

3. Update a Specific Sub-Logger

If you only need to debug a specific component without flooding the logs with data from other subsystems, you can target a specific endpoint.

Instruction: To target a sub-logger, append the logger name (from the keys found in the data output in Step 1) to the end of the v1/sys/loggers/ path.

• To target MFA: Use .../v1/sys/loggers/mfa

• To target Raft Storage: Use .../v1/sys/loggers/storage.raft

Example Command (Targeting MFA):

curl -sk --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"level": "trace"}' \
    https://127.0.0.1:8200/v1/sys/loggers/mfa

curl -sk --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"level": "info"}' \
    https://127.0.0.1:8200/v1/sys/loggers

curl -sk --header "X-Vault-Token: $VAULT_TOKEN" \
    --request DELETE \
    https://127.0.0.1:8200/v1/sys/loggers

sudo journalctl -u vault -f

Additional Information

[!IMPORTANT]

Memory vs. Persistence: These changes affect the active process memory only. If the Vault service restarts or the server reboots, Vault will revert to the log_level defined in your physical vault.hcl configuration file.

• Performance Impact: trace mode captures nearly every internal action, including hex dumps of network traffic. Only use this level for short periods of active troubleshooting.

   

Documentation & Resources

For more details on Vault logging and troubleshooting, refer to the official HashiCorp documentation:

• Vault API: System Loggers: Detailed documentation on the /sys/loggers endpoint.

• Vault Monitoring & troubleshooting: General strategies for debugging Vault issues.

• Vault Configuration Parameters: Reference for setting permanent log levels in the configuration file.