Problem

The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

Observing Consul service mesh metrics in Kubernetes is critical for monitoring the health and performance of your services. This guide walks you through configuring Helm overrides and pod annotations required to enable Prometheus metric collection for Consul sidecars and services.

Expected Outcome

Enable Prometheus monitoring for your Consul service mesh. This involves exposing metrics from both Envoy sidecars and your application services, making them accessible for scraping and analysis by Prometheus.

Prerequisites

• Consul 1.10 or later: The merged metrics feature is available from Consul 1.10 onwards. Ensure you have a compatible version installed.
• Consul Connect Injector: The Connect Injector is responsible for injecting the Envoy sidecar and configuring the necessary annotations for Prometheus. Verify that it's properly deployed and configured in your Kubernetes cluster.
• Consul-K8s: This is the official HashiCorp project for running Consul on Kubernetes. You'll need it for managing and integrating Consul with your Kubernetes environment.

• Kubernetes Environment:

  • Kubernetes Cluster: A running Kubernetes cluster is essential for deploying your applications and the Consul service mesh.
  • kubectl: The Kubernetes command-line tool, kubectl, is necessary for interacting with your cluster and managing resources.
  • Helm (Optional): If you're using Helm to manage Consul deployments, ensure you have Helm installed and configured.

   

• Prometheus Setup:

  • Prometheus Server: A running Prometheus server is required to scrape and store the metrics exposed by your Consul service mesh.
  • Prometheus Configuration: Configure Prometheus to scrape the appropriate endpoints (e.g., 0.0.0.0:20200) and any other relevant targets in your Consul environment.

Use Case

To gain comprehensive visibility into the health and performance of your services and infrastructure.

Table of Contents

• Introduction
• Configuring Connect-Inject Default Prometheus Annotations
  • Default Prometheus Annotations (Injected via Helm)
• Consul Service Mesh Sidecar Metrics Configuration
  • Global Configuration via Helm
  • Pod-Specific Configuration
• Application Merged Metrics Endpoints for Sidecars
  • Merging Envoy Sidecar and Connect Service Metrics
    • Helm Global Override
    • Pod Annotation
  • Application Service Metrics Configuration
  • Verifying the Merged Metrics Endpoints
• Verifying Metrics Collection
  • Validate Envoy Sidecar Listeners
• Conclusion
• References

Procedure

Configuring Connect-Inject Default Prometheus Annotations

By default, Consul Connect Injector adds Prometheus annotations to injected pods. When the applicable Helm Chart Values or Pod-Specific Annotations are configured (described further in the next section)

Default Prometheus Annotations (Injected via Helm)

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "20200"

These ensure Prometheus scrapers can collect metrics from Consul sidecars.

Consul Service Mesh Sidecar Metrics Configuration

Global Configuration via Helm

To enable global metric merging for all Consul dataplane proxies, modify your Helm chart as follows:

connectInject:
  metrics:
    defaultEnabled: true
    defaultPrometheusScrapePort: "20200"
    defaultPrometheusScrapePath: "/metrics"

This configuration ensures that every injected Consul sidecar proxy exposes Prometheus metrics on 0.0.0.0:20200.

Pod-Specific Configuration

For granular control at the pod level, use the following annotations:

annotations:
  consul.hashicorp.com/enable-metrics: "true"
  consul.hashicorp.com/prometheus-scrape-port: "20200"
  consul.hashicorp.com/prometheus-scrape-path: "/metrics"

These annotations ensure that both Envoy stats are accessible via Prometheus scrapers.

Application Merged Metrics Endpoints for Sidecars

Merging Envoy Sidecar and Application Service Metrics

Consul allows you to merge metrics from the Envoy sidecar and the Connect App's service into a single Prometheus endpoint. This can be enabled using:

Helm Global Override:

connectInject:
  metrics:
    defaultEnabled: true
    defaultEnableMerging: true

    # The port at which the consul-dataplane will listen on to return
    # merged metrics stats.
    defaultMergedMetricsPort: 20100

Pod Annotation:

The annotations below demonstrate per-pod annotation overrides for the above Helm chart value equivalent settings.

annotations:
  consul.hashicorp.com/enable-metrics: true
  consul.hashicorp.com/enable-metrics-merging: true
  consul.hashicorp.com/merged-metrics-port: "20100"
  consul.hashicorp.com/merged-metrics-path: "/stats/prometheus"

Application Service Metrics Configuration

To collect Prometheus metrics from your application service, you must define the application's metrics endpoint using pod-level annotations only:

annotations:
  consul.hashicorp.com/service-metrics-port: "<your_service_port>"
  consul.hashicorp.com/service-metrics-path: "<your_service_metrics_api_endpoint>"

These metrics will be aggregated and exposed at 0.0.0.0:20200, ensuring both Envoy and service metrics are available in a unified view.

Verifying the Merged Metrics Endpoints:

When enabled, the merged metrics are exposed at:

• Local (within pod): 127.0.0.1:20100/stats/prometheus
• External scraping endpoint: 0.0.0.0:20200

curl -s 127.0.0.1:20100/stats/prometheus | head -n 5
curl -s 0.0.0.0:20200/metrics | head -n 5

Both endpoints should return the same metrics data.

Verifying Metrics Collection

Validate Envoy Sidecar Listeners

curl -s 0.0.0.0:19000/listeners

Key listeners to verify:

• Merged Metrics Listener: envoy_prometheus_metrics_listener::0.0.0.0:20200
• Envoy Stats Listener: envoy_metrics_listener::127.0.0.1:20100

Conclusion

With these configurations, you can successfully integrate Consul-K8s for Prometheus-based monitoring. These settings ensure comprehensive visibility into both Envoy proxy metrics and application service metrics, facilitating proactive service health monitoring within a Consul service mesh environment.

Additional Information

• HashiCorp Consul Kubernetes Metrics Configuration
• Prometheus Scraping Configuration
• Consul Helm Chart Documentation