Troubleshooting Vault Secrets Operator (VSO)

Problem

Table of Contents

• Introduction
• Troubleshooting workflow phases
  • Pod State
    • CrashLoopBackOff
      • OOMKilled
  • Vault Connection
  • Vault Authentication
  • Secret Syncing
  • App Consumption
• Common error messages
  • Permission Denied
  • Failed to get Vault client
• Data Collection for further Troubleshooting
• Further Reading

Introduction

The Vault Secrets Operator is a Kubernetes Operator that syncs secrets between Vault and Kubernetes and allows Pods to consume Vault secrets directly as native Kubernetes Secrets.

A Kubernetes Operator is a software extension that uses custom resources to manage applications hosted on Kubernetes.

From VSO Tutorial

It's helpful to think of troubleshooting VSO into the phases of the VSO workflow:

• Pod state: is vault-secrets-operator-controller-manager pod running
• Vault Connection: Is the pod able to connect to Vault
• Vault authentication: Is the pod able to authenticate to Vault
• Secret Syncing: Is the pod able to sync the secret
• App Consumption: Can the app pod consume the native k8s secret

Troubleshooting workflow phases

Pod State

Is the pod running when running? Verify status with:

• kubectl get pods
• kubectl describe pod <vault-secrets-operator-controller-manager-uniq>

CrashLoopBackOff

VSO pod in this condition? Then describe the pod to look at the Last State reasons:

State: Waiting 
Reason: CrashLoopBackOff 
Last State: Terminated 
Reason: OOMKilled 
Exit Code: 137

OOMKilled

If your pods are running into OOM, consider increasing limits & requests values for the VSO container resources.

If your pod is running, consider the workflow

Vault Connection

• Is the vaultconnections CRD yaml configured correctly?
• Check the connection to VAULT_ADDR from pod using tools like curl, netcat, othertcp connectivity tools/methods
• Describe the VSO pod and review for clues / next steps

Vault Authentication

• Is the vaultauths CRD yaml configured correctly?
• Does the vault kubernetes authentication work?
  • test the authentication: vault write auth/kubernetes/login role=${your_role} jwt=...
• Auth failing? Check auth method configuration & role:
  • vault read auth/${kubernetes}/config
  • vault read auth/${kubernetes}/role/${role_name}

Secret Syncing

• Confirm sync status in pod logs
  • kubectl logs <vso-controller-pod>
• Review YAML files from the secret type(s) being used (ie - static, dynamic, pki, etc)

App Consumption

Check for underlying issues with app pods consuming the secrets

Common error messages

Permission Denied

• Make sure the auth configured for VSO has needed permissions for the secret being synced
  • Check policy permissions for your secret type. Some will only require read, some will require update, etc.
• Make sure the auth configured for VSO is working (See Vault Authentication section above) 

Failed to get Vault client

Ensure all CRD's are configured properly. This could stem from not configuring VaultAuth CRD, but having VaultConnection & secrets CRDs configured. 

Data Collection for further Troubleshooting

Need further help troubleshooting? When engaging with Support please share the following data in text output:

// Commands to run to gather relevant data to share with Support

// Kubernetes version

kubectl version

// Share environment running VSO

EKS, OpenShift, GKE, etc

// Share VaultConnection CRD information

kubectl get vaultconnection -A -o yaml

// Share VaultAuth CRD information

kubectl get vaultauth -A -o yaml

// Share confirmation Vault authentication works. Redact any sensitive info

vault write auth/kubernetes/login role=${your_role} jwt=...

// Describe the VSO pod and share output

kubectl describe pods <vault-secrets-operator-controller-manager-uniq-string>

// If Vault Authentication is failing, provide sanitized output of the following

vault read auth/${kubernetes}/config

vault read auth/${kubernetes}/role/${role_name}

// Share pod logs

kubectl logs <vso-controller-pod>

// Share CRD YAML configuration for secrets. Share output from CRD's used

kubectl get vaultdynamicsecret -A -o yaml
kubectl get vaultstaticsecret -A -o yaml
kubectl get vaultpkisecret -A -o yaml
kubectl get secrettransformation -A -o yaml

Further Reading

Vault Secrets Operator documentation

Running VSO in OpenShift

HashiCorp GitHub repository for VSO

VSO Tutorial