Best Practices for GitOps

Edit online

This section includes best practices recommendations when using GitOps for deployment.

Version control
  • Store all configuration files in Git: Maintain all configuration files in a version-controlled repository.
  • Use meaningful commit messages: Ensure commit messages clearly describe the changes made.
  • Tag releases: Use version tags to mark stable configurations and rollback easily, when required.
Environment separation
  • Use separate Git branches: Organize configurations by maintaining dedicated branches or directories for each environment such as development, staging, and production.
  • Example structure: 
    ivia-config/
     ├── base/
     │   ├── values.yaml
     │   └── ibmvia-autoconf.yaml
     ├── dev/
     │   ├── values.yaml
     │   └── ibmvia-autoconf.yaml
     ├── staging/
     │   ├── values.yaml
     │   └── ibmvia-autoconf.yaml
     └── production/
         ├── values.yaml
         └── ibmvia-autoconf.yaml
Secret management
  • Do not commit secrets to Git: Avoid storing secrets in Git version control.
  • Use external secret management: Store secrets in dedicated tools such as HashiCorp Vault or AWS Secrets Manager.
  • Use secure secret solutions: Consider using tools such as Sealed Secrets or the External Secrets Operator.
  • Rotate secrets regularly: Update and replace secrets periodically to reduce security risks.
Configuration versioning
  • Increment the version field: Always update the version field in ibmvia-autoconf.yaml when modifying secrets.
  • Version changes: Record all version updates and related changes clearly in commit messages.
  • Semantic versioning: Apply semantic versioning for major configuration changes to improve traceability and consistency.
Testing
  • Test changes in non-production environments: Validate configuration changes in development or staging environments before promoting them to production.
  • Preview Helm changes: Use helm template or helm show to preview changes.
  • Validate YAML syntax: Ensure all YAML files are correctly formatted and error-free before committing changes.
Monitoring
  • Monitor pod health: Monitor pod health and readiness status to ensure proper operation.
  • Configure alerts: Set up alerts to detect failed deployments.
  • Track configuration drift: Use ArgoCD to manage differences between the desired and actual cluster state.
  • Monitor application logs: Review logs regularly to identify and resolve configuration errors.
Backup and disaster recovery
  • Document recovery procedures: Maintain clear documentation for system recovery processes.
  • Test disaster recovery scenarios: Test and validate recovery procedures to ensure they function as expected.
  • Maintain version history: Retain multiple versions of configuration files in Git to support rollback and auditing.
Documentation
  • Document configuration parameters: Maintain clear documentation for all configuration parameters.
  • Maintain a changelog: Keep a detailed record of all configuration changes for traceability.
  • Document environment-specific settings: Clearly identify and document configurations that differ across environments.
  • Maintain operational runbooks: Provide runbooks for common operational tasks.