Secrets management best practices for developers
This guide outlines key strategies for securing APIs, credentials and infrastructure components. In turn, ensuring your regulatory compliance and protecting your organization’s assets. By implementing robust secrets management practices you and your organiztion can mitigate data breach risks.
It’s 2 AM, you receive a frantic Slack message from your CTO. A security researcher just posted that your company’s GitHub repo has been leaking AWS (Amazon Web Service) access keys for the past three months.
Even worse, those keys have full admin privileges and someone’s already spinning up 100 Elastic Compute Cloud (EC2) instances to mine cryptocurrency on your dime. Your database passwords are now completely exposed and your API keys for payment processing have been compromised. Now, the hardcoded SSH key in your config file is in the hands of a hacker who’s strolling through your Kubernetes cluster like it’s an open door.
This scenario might be hypothetical, but instances like this are happening right now to companies that thought, “We can fix secrets management later, we need to ship this immediately”. Unfortunately, “later” now also involves cleaning up an entirely avoidable mess.
Secrets are the skeleton keys to your entire digital empire. API keys, database credentials, IAM tokens and SSH keys are all the invisible threads holding your CI/CD pipelines, microservices and cloud infrastructure together. Lose control of them, and you don’t just risk a data breach, you put your entire business at risk. From anything like financial loss and reputational damage, to regulatory penalties under standards like SOC 2, HIPAA, PCI DSS and GDPR.
The good news is this disaster is 100% preventable. By implementing the right secrets management strategy, you can lock down access, automate security and hopefully sleep soundly knowing your keys are locked up tight. So, let’s focus on how we can prevent our next leak before it starts.
Secrets management refers to the processes and policies used to securely store, access and rotate sensitive credentials and other secrets. These secrets are used by applications, services and non-human identities (NHIs) (for example, CI/CD pipelines, microservices, Kubernetes pods). Unlike traditional user passwords, secrets are typically used by machines and services to authenticate and authorize access to resources without human intervention.
Types of secrets
• API keys (for example, for cloud services, payment gateways)
• Database credentials (for example, usernames/passwords for PostgreSQL, MySQL)
• SSH keys (for secure server access)
• IAM tokens (for cloud provider authentication)
• TLS/SSL certificates and private keys
• Environment variables (for example, DB_PASSWORD, AWS_ACCESS_KEY_ID)
• Service account credentials (for automated workflows)
Preventing data breaches
Hardcoded or improperly stored secrets are a prime target for attackers. According to research by GitGuardian, 28.6 million secrets—including API keys, database passwords and cloud credentials—were discovered in public GitHub commits. Once leaked, these secrets can be exploited to access sensitive data, escalate privileges or even take over entire systems. For example, a single exposed AWS access key can lead to unauthorized cloud infrastructure access, resulting in data theft or service disruption.
Mitigating secret sprawl
Secret sprawl occurs when secrets are scattered across code repositories, config files and general developer workspaces without any centralized control. This decentralization makes it difficult to track, rotate or revoke secrets. Overtime this decentralization leads to an increased risk of exposure. Centralized secrets management solutions, such as HashiCorp Vault, AWS Secrets Manager or Azure Key Vault, help regain control by storing secrets in a secure, centralized vault.
Compliance and auditing
Regulatory frameworks, like SOC 2, HIPAA, PCI DSS and ISO 2700, require organizations to implement strict access control and encryption for sensitive data. Secrets management tools provide audit logs that track who accessed which secret, when and for what purpose, ensuring compliance and availability for forensic investigations.
Enabling DevOps and CI/CD security
In DevOps and CI/CD pipelines, secrets are often required to deploy applications or integrate with third-party services. Without proper management, secrets might be embedded in GitHub Actions, Jenkinsfiles or Kubernetes manifests, creating vulnerabilities. Efficient secrets management will ensure that secrets are injected securely at runtime and automatically rotated to minimize exposure.
Supporting the principle of least privilege
The principle of least privilege dictates that users, services and applications should have only the minimum permissions necessary to perform their tasks. It is crucial that we make sure that not just anyone has access to everything, permissions should always be job-specific. Secrets management systems enforce this principle through role-based access control (RBAC), multifactor authentication (MFA) and fine-grained policies. These systems, when used correctly will reduce the blast radius of potential breaches.
Implementing effective secrets management requires a combination of technical tools, processes and the implementation of a security-minded culture within your organization. In short, you not only need the right tools for the job, but you need to make their use standard practice. Below are actionable best practices, along with guidance on how to implement them for use in real-world scenarios.
Never hardcode your secrets
Hardcoding secrets in source code, config files or environment variables in version control (for example, Git) is a leading cause of leaks. Even if removed later, secrets will remain in the Git history, so it’s best to avoid this practice from the start. Here, they can be discovered by attackers or secret scanning tools.
Instead, use environment variables for local development, but never commit .env files to version control. Tools like dotenv can load variables from .env files at run time, but these files should be excluded through .gitignore. Also, make sure to leverage secret management systems (for example, HashiCorp Vault, AWS Secrets Manager, Azure Key Vault) to store and retrieve secrets dynamically at run time.
Additionally, you should use pre-commit hooks to scan for secrets before commits. Tools like TruffleHog, GitLeaks or GitHub Secret Scanning can detect and block accidental commits of API keys, database passwords and other credentials.
Example (Never commit secrets):
Bash
Copy
# .gitignore
.env
.env.*
*.pem
*.key
config/secrets.yml
config/secrets.yaml
secrets.yml
secrets.yaml
Note: These files are loaded at run time (for example, from Vault/Kubernetes Secrets/CI environment variables), not stored in the repo.
Centralize secrets in a vault
Decentralized secrets, those secrets stored in code repositories, CI/CD tools or developer machines are difficult to manage, rotate and audit. If unchecked, this will inevitably lead to secret sprawl.
To ensure that it does not happen, deploy a centralized vault (for example, HashiCorp Vault, AWS Secrets Manager, Azure Key Vault) to store all secrets in a single, secure location. Also, make sure to enforce RBAC and least privilege so only authorized users and services can access specific secrets. Lastly, use dynamic secrets for short-lived credentials. These database passwords should expire after a few hours and will reduce the window of exposure.
Example (HashiCorp Vault):
Bash
Copy
# Retrieve a database password from Vault
vault kv get -field=password secret/database/prod
Automate secret rotation
Static secrets will increase the risk of compromise over time. These include long-lived database passwords or API keys and manual rotation of these secrets is error-prone and often neglected.
To counter this, you should use the automated rotation features provided by secrets management tools. For example, AWS Secrets Manager can rotate RDS database credentials automatically and HashiCorp Vault supports dynamic secrets for databases and cloud providers. Also, ensure that you implement short-lived secrets (for example, JWT tokens, OAuth tokens) that expire after a defined period.
Example (HashiCorp Vault):
get_vault_credentials.py
import os
import hvac
vault = hvac.Client(
url=os.environ["VAULT_ADDR"],
token=os.environ["VAULT_TOKEN"],
)
# Example role name from your Vault setup
role = "readonly"
# Dynamic credentials endpoint (Vault returns username/password with a lease/TTL)
# Exact path uses the database secrets engine and role name that you created.
creds = vault.read(f"database/creds/{role}")
username = creds["data"]["username"]
password = creds["data"]["password"]
# Use username/password to connect to DB...
print("Got fresh dynamic credentials:", username)
Secure secrets in CI/CD pipelines
CI/CD pipelines, like GitHub Actions, often require secrets to deploy applications or connect to services. As there is another major risk of exposure here, the last thing we should be doing is hardcoding these secrets in pipeline scripts or environment variables.
Use the following secret variables or secrets management integrations in your CI/CD tool to minimize your risk of exposure:
GitHub Actions: Store secrets in the repository’s Secrets tab and reference them as ${{ secrets.MY_SECRET }}.
GitLab CI: Use CI/CD variables (masked and protected).
Jenkins: Use the Credentials Plugin or integrate with HashiCorp Vault.
Example (GitHub Actions):
yaml
yaml
# .github/workflows/deploy.yml
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Deploy to AWS
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
run: ./deploy.sh
NOTE: Never log secrets in pipeline output. Most CI/CD tools automatically mask secrets in logs, but custom scripts should avoid printing them.
Enforce least privilege and access control
Another common problem is over-permissioned access to secrets. This leads to an increased risk of lateral movement and privilege escalation if there is a breach. We all know it’s impossible to prevent 100% of breaches, but that doesn’t mean we can’t minimize the damage associated with them.
To do this, we need to apply the principle of least privilege to all secrets. This means granting access only to the users, services or Kubernetes pods that absolutely need it. For example, in HashiCorp Vault, define policies that limit which secrets a Kubernetes pod can access. In AWS IAM, create roles with minimal permissions for CI/CD pipelines or microservices. Lastly, make sure to require multifactor authentication (MFA) for human access to secrets management systems.
Example (Vault Policy):
hcl
hcl
Copy
# Vault policy for a CI/CD pipeline
# Read-only access to a single KV v2 prefix
path "secret/data/ci/cd/*" {
capabilities = ["read"]
}
Audit and monitor secret access
Without audit trails, organizations cannot detect or investigate unauthorized access to secrets. Which can lead to something far worse than a data breach, one that goes undetected.
From the start, you should enable audit logging in your secrets management tool to track who accessed which secret and when, failed access attempts and the creation, rotation and deletion events associated with secrets. You should also ensure that you integrate logs with SIEM (Security Information and Event Management) tools for real-time monitoring and alerting.
Additionally, it’s best to regularly review audit logs for suspicious activity. These activities might be anything from unusual access patterns to off-hour requests. For example, Fidel from the New Orleans, LA office doesn’t usually sign on from Bangkok, Thailand at 0300 local time.
Example (Vault Audit Logs):
Bash
bash
Copy
# Enable file-based audit logging in Vault
vault audit enable file file_path=/var/log/vault-audit.log
Scan for exposed secrets
Even with the previous safeguards in place, secrets can accidentally leak into code repositories, logs or config files. None of us want to be known as the developer who leaks secrets, but it happens and it’s not the end of the world. The following methods will allow you to detect any exposed secrets:
• Code repositories: GitHub Secret Scanning and GitLab Secret Detection
• CI/CD pipelines: TruffleHog and GitLeaks
• Container images: Docker Scout and Trivy
You should take this one step further and automate scans as part of pre-commit hooks or pull request workflows. This automation will allow you to block leaks before they reach the main branch.
Example (GitHub Secret Scanning):
yaml
yaml
name: Secret Scan
on: [push, pull_request]
jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: TruffleHog scan
uses: trufflesecurity/trufflehog@main
with:
extra_args: "filesystem . --results=verified,unknown --fail"
Secure secrets in Kubernetes
Kubernetes environments often rely on Secrets (base64-encoded by default) or ConfigMaps. These are not encrypted at rest and can be retrieved by anyone with cluster access.
To prevent this, never store secrets in plain text in Kubernetes Secrets. Instead, use external secrets management (for example, HashiCorp Vault, AWS Secrets Manager) with Kubernetes External Secrets or Vault Agent Sidecar. Also, make sure to enable encryption at rest for Kubernetes Secrets by using a key management service (KMS). Ensure that you are restricting access to Kubernetes Secrets by using RBAC and network policies.
Example (Vault Agent Sidecar in Kubernetes):
yaml
yaml
apiVersion: v1
kind: Pod
metadata:
name: my-app
spec:
volumes:
- name: secrets
emptyDir: {}
- name: vault-agent-config
configMap:
name: vault-agent-config
containers:
- name: app
image: my-app
volumeMounts:
- name: secrets
mountPath: /etc/secrets
- name: vault-agent
image: hashicorp/vault:1.17
args:
- "agent"
- "-config=/etc/vault/config.hcl"
volumeMounts:
- name: vault-agent-config
mountPath: /etc/vault
- name: secrets
mountPath: /etc/secrets
Even with all the tools and techniques we’ve covered so far, human error (for example, committing hardcoded credentials, sharing secrets through Slack) remains a top cause of leaks.
To mitigate this issue, developers should be trained on the risks associated with hardcoding secrets and secret sprawl. They should also be continuously trained on how to use environment variables, secrets management tools and pre-commit hooks. Create secure practices for CI/CD pipelines, Kubernetes and multicloud environments.
You should also enforce code reviews to catch accidental commits of secrets because these accidents occur every so often. Finally, document your secrets management policies in a security.md file in your repository. Drafting up an easy-to-use guide or cheat sheet is also a great idea. This will help make sure your developers, especially new hires, are on the same page when implementing these procedures.
Secrets management is a critical discipline for modern development teams. The risks of hardcoded credentials, secret sprawl and data breaches are too great to ignore. Fortunately, you have this guide to refer to for implementing centralized vaults, automated rotation, least privilege access and audit trails.
By adopting the best practices we’ve outlined, your developers can:
• Eliminate hardcoded secrets from source code and configuration files.
• Centralize and secure secrets in a vault (for example, HashiCorp Vault, AWS Secrets Manager, Azure Key Vault).
• Automate rotation and enforce short-lived credentials.
• Monitor and audit access to detect and respond to threats.
• Integrate secrets management into CI/CD pipelines, Kubernetes and multicloud environments.
The switch from handling secrets in a haphazard manner to managing them in a structured, automated fashion not only reduces risk but also aligns with DevSecOps principles. In the future, your organization will have security baked into every stage of the development lifecycle. As you scale, so too should the way you manage secrets, ensuring that security isn’t sacrificed for progress.
Resources
Optimize your cloud with unified lifecycle automation—secure, scalable hybrid infrastructure designed for resilience and AI.
Safeguard your hybrid-cloud and AI environments with intelligent, automated protection across data, identity, and threats.
Protect and manage user access with automated identity controls and risk-based governance across hybrid-cloud environments.
Footnotes
“Secrets Management.” IBM Think
“Secrets Management Cheat Sheet.” OWASP Cheat Sheet Series
“A Survey of Secret Management in Cloud-Native Applications.” arXiv, 2022
“The State of Secrets Sprawl Report 2026.” GitGuardian, 2026
“TruffleHog.” GitHub
“Secret Scanning.” GitHub Docs
“Secrets.” Kubernetes Documentation