Deploying Vault Secrets Operator on OpenShift

This guide provides a comprehensive walkthrough for deploying and configuring the Vault Secrets Operator (VSO) in an OpenShift cluster. The VSO enables automatic synchronization of secrets from HashiCorp Vault to Kubernetes secrets, allowing applications to consume Vault secrets natively through Kubernetes.

The Vault Secrets Operator deployment consists of:

Vault Server: The source of truth for secrets.
VSO Controller: Manages the synchronization between Vault and Kubernetes.
VaultAuth: Custom resource defining authentication to Vault.
VaultStaticSecret: Custom resource defining which secrets to sync.
Kubernetes Secrets: Target secrets created and managed by VSO.

Prerequisites

Warning: Root tokens must not be used in production environments. This guide uses root token only for initial demonstration setup. Immediately proceed to Kubernetes authentication for production after initial setup. Configure Kubernetes authentication before deploying to production. Revoke the root token after setting up Kubernetes authentication.

Before beginning the installation, ensure the following requirements are met:

Vault Server: A running Vault server accessible within the cluster.
This guide assumes Vault is accessible at: http://vault.vault.svc.cluster.local:8200.

Vault can be running with in-memory storage or persistent storage.
Cluster Access: Verify you are logged into the OpenShift cluster:
```
oc whoami
```
Vault Root Token: Access to Vault with administrative privileges for initial configuration.
Helm: Helm 3.x installed and configured.

Procedure

Create a dedicated namespace for the Vault Secrets Operator:
```
oc create namespace vault-operator
```
Configure Vault server:
Configure Vault with the necessary secrets engine, authentication method, and policies. Execute these commands inside the Vault pod or through a port-forwarded connection.

Warning: CRITICAL SECURITY WARNING: Do not use root tokens in production environments. Root tokens should only be used for initial Vault setup. After initial configuration, root tokens should be revoked. Use proper authentication methods (Kubernetes, AppRole, LDAP, OIDC) for all operations. This guide uses root token only for demonstration of initial setup.
1. Access Vault pod:
```
oc exec -it <vault-0> -n <vault> -- /bin/sh
```
2. Set Vault address and authenticate:
  For Initial Setup Only (use proper auth methods afterward).
```
export VAULT_ADDR=http://127.0.0.1:8200
vault login <root_token>
```
  Note: After completing the initial setup in this section, proceed to the authentication methods for production section to configure proper authentication and revoke the root token.
3. Enable kv-v2 secrets engine:
```
vault secrets enable -path=kvv2 kv-v2
```
4. Create sample secret:
```
vault kv put kvv2/myapp/config username=<User_name> password=<password>
```
5. Enable approle authentication:
```
vault auth enable approle
```
6. Create policy for application access
  Create a policy that grants read access to the application secrets:
```
vault policy write myapp-policy - <<EOF
path "kvv2/data/myapp/config" {
  capabilities = ["read"]
}
EOF
```
7. Create approle:
```
vault write auth/approle/role/myapp-role \
  token_ttl=1h \
  token_max_ttl=4h \
  policies="myapp-policy"
```
8. Retrieve approle credentials:
  Get the Role ID:
```
vault read auth/approle/role/myapp-role/role-id
```
  Expected output:
```
Key        Value
---        -----
role_id    <your_role_id>
```
  Generate a Secret ID:
```
vault write -f auth/approle/role/myapp-role/secret-id
```
  Expected output:
```
Key                   Value
---                   -----
secret_id             <your_secret_id>
secret_id_accessor    <accessor_id>
```
  Important: Save both the role id and secret id as they will be needed in subsequent steps.
  
  Exit the Vault pod:
```
exit
```

Add HashiCorp helm repository and update the local cache:


helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

Create TLS certificate secret:
If your Vault server uses TLS (as configured in the values.yaml), you need to create a Kubernetes secret containing the CA certificate so VSO can verify Vault's TLS certificate.
1. Extract CA certificate from Vault.
  If Vault is running in the same cluster, you can extract the CA certificate from the Vault TLS secret:
```
oc get secret vault-tls -n vault -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
```
  Alternatively, if you have the CA certificate file from your certificate authority, use that file directly.
2. Create the CA certificate secret in the vault-operator namespace:
```
oc create secret generic vault-ca-cert \
  --from-file=ca.crt=ca.crt \
  -n vault-operator
```
3. Verify secret creation:
```
oc get secret vault-ca-cert -n vault-operator
```
  Note: If your Vault server uses skipTLSVerify: true (not recommended for production), you can skip this step and remove the caCertSecretRef line from the values.yaml file.
Install Vault secrets operator:
1. Create a file named vso-values.yaml with configuration for global settings, controller configuration, RBAC, service account, OpenShift-specific settings, and webhook configuration:
  - TLS Configuration: Uses HTTPS with CA certificate verification for secure Vault communication.
  - High Availability: 2 replicas with leader election, pod disruption budget, and pod anti-affinity.
  - Security Hardening: Runs as non-root user (UID 65532), read-only root filesystem, all capabilities dropped, privilege escalation prevented.
  - Resource Management: Defined requests and limits to ensure stable operation.
  - Priority Class: Uses system-cluster-critical to prevent eviction under resource pressure.
  - Client Cache: Enabled to reduce redundant Vault API calls.
  - RBAC: Automatically creates necessary cluster roles and bindings.
  - OpenShift Integration: Automatically creates SecurityContextConstraints (SCC).
  - Webhook Validation: Validates custom resources before creation/update.
  - Monitoring: Prometheus metrics enabled for observation.
2. Install VSO using helm:
```
helm install vault-secrets-operator hashicorp/vault-secrets-operator \
  -n vault-operator \
  -f vso-values.yaml
```
3. Check that the VSO pods are running to verify installation:
```
oc get pods -n vault-operator
```
  Expected output:
```
NAME                                                        READY   STATUS    RESTARTS   AGE
vault-secrets-operator-controller-manager-xxxxxxxxx-xxxxx   2/2     Running   0          30s
```
Create Kubernetes secret to store the AppRole secret id:
1. Create secret manifest:
  Create a file named token-secret.yaml and replace <yoursecretid> with the actual Secret ID obtained earlier.
2. Apply the secret:
```
oc apply -f token-secret.yaml
```
3. Verify secret creation:
```
oc get secret approle-secret -n vault-operator
```
Create VaultConnection resource:
The VaultConnection resource defines how VSO connects to your Vault server.
1. Create vaultconnection manifest:
  Create a file named vault-connection.yaml with configuration parameters for address, skipTLSVerify, and caCertSecretRef.
2. Apply the Vault connection:
```
oc apply -f vault-connection.yaml
```
3. Verify Vault connection:
```
oc get vaultconnection default -n vault-operator
oc describe vaultconnection default -n vault-operator
```
Define vaultauth custom resource:
The VaultAuth resource configures how VSO authenticates to Vault.
1. Create vaultauth manifest:
  Create a file named vault-auth.yaml and replace <your roleid> with the actual Role ID obtained earlier.
2. Apply the vaultauth resource:
```
oc apply -f vault-auth.yaml
```
3. Verify vaultauth status:
```
oc get vaultauth vault-auth -n vault-operator
```
  Check the status to ensure authentication is successful:
```
oc describe vaultauth vault-auth -n vault-operator
```
Define vaultstaticsecret resource to synchronize to Kubernetes:
1. Create a manifest file named vault-static-secret.yaml with configuration for vaultAuthRef, mount, type, path, refreshAfter, and destination:
2. Apply the vaultstaticsecret resource:
```
oc apply -f vault-static-secret.yaml
```
3. Verify vaultstaticsecret status:
```
oc get vaultstaticsecret vault-static-secret-v2 -n vault-operator
```
  Check the detailed status:
```
oc describe vaultstaticsecret vault-static-secret-v2 -n vault-operator
```

Verify synced Kubernetes secret:

The VSO should automatically create a Kubernetes secret with the data from Vault.

List the created secret:


oc get secret static-secret2 -n vault-operator

View secret contents:


oc get secret static-secret2 -n vault-operator -o yaml

Decode secret values:


oc get secret static-secret2 -n vault-operator -o jsonpath='{.data.username}' | base64 -d
oc get secret static-secret2 -n vault-operator -o jsonpath='{.data.password}' | base64 -d

Use secrets in applications:
Now that the secret is synchronized, you can use it in your applications by mounting secret as environment variables or as volume.

The Vault Secrets Operator is now deployed and configured. Secrets from Vault are automatically synchronized to Kubernetes secrets and can be consumed by applications.

Using secrets in applications

Example: mounting secret as environment variables.


apiVersion: v1
kind: Pod
metadata:
  name: myapp
  namespace: vault-operator
spec:
  containers:
  - name: myapp
    image: nginx:latest
    env:
    - name: USERNAME
      valueFrom:
        secretKeyRef:
          name: static-secret2
          key: username
    - name: PASSWORD
      valueFrom:
        secretKeyRef:
          name: static-secret2
          key: password

Example: mounting secret as volume.


apiVersion: v1
kind: Pod
metadata:
  name: myapp
  namespace: vault-operator
spec:
  containers:
  - name: myapp
    image: nginx:latest
    volumeMounts:
    - name: secrets
      mountPath: /etc/secrets
      readOnly: true
  volumes:
  - name: secrets
    secret:
      secretName: static-secret2

Best practices

Use Namespaces: Deploy VSO and applications in separate namespaces for better isolation.
Least Privilege: Create specific policies for each application with minimal required permissions.
Secret Rotation: Configure appropriate refreshAfter intervals based on your security requirements.
TLS Encryption: Always use TLS in production environments.
Monitor Sync Status: Regularly check VaultStaticSecret status to ensure secrets are syncing correctly.
Backup Credentials: Securely store AppRole credentials in a secrets management system.
Audit Logging: Enable Vault audit logging to track secret access.

Cleanup

To remove the VSO deployment:


# Delete secrets sync
oc delete vaultstaticsecret --all -n vault-operator

# Delete auth
oc delete vaultauth --all -n vault-operator

# Remove finalizer (if stuck)
oc patch vaultconnection default -n vault-operator \
  --type=merge \
  -p '{"metadata":{"finalizers":[]}}'

# Delete connection
oc delete vaultconnection --all -n vault-operator

# Delete app secret
oc delete secret approle-secret -n vault-operator

# Uninstall operator
helm uninstall vault-secrets-operator -n vault-operator

# Delete namespace
oc delete namespace vault-operator

Note: The patch command removes finalizers from the VaultConnection resource, which may prevent deletion if the resource is stuck in a terminating state. This is a common issue when cleaning up custom resources.