Datadog Consul and Envoy Metrics Monitoring | VM Edition

Problem

The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

This article is intended to be used as a guide outlining the major configuration changes necessary to configure Consul and Envoy to be monitored using Datadog agent.

Table of Contents

• Datadog Configuration Files
• Enable Consul Telemetry
• Enable Consul and Envoy Logging to File
• Expose Envoy Sidecar Proxy Metrics
• Create Datadog Account
• Create Datadog API Key
• Installing Datadog Agent on Host VM
• Configure Datadog Agent for Consul Metrics Capture
• Configure Datadog Agent for Envoy Metrics Capture
• Verify Consul to Datadog Agent Communication
• Verify Envoy to Datadog Agent Communication

Prerequisites

• Datadog Account and Login Info
• Linux OS Administrative User Access
• Running Consul Cluster using VMs
• Consul Connect Enabled
• Envoy Running as Sidecar Proxy for Connect Environment
• Shell Access to VMs hosting Consul

Datadog Configuration Files

Enable Consul Telemetry

Configure each Consul agent to expose and capture local agent metrics with Prometheus and DogstatsD:

telemetry = {
  prometheus_retention_time = "1h"
  dogstatsd_addr            = "127.0.0.1:8125"
}

If agent configuration entry for telemetry required updating, restart of Consul agent is required. Ensure, if restarting servers, to perform rolling-reboot of servers ensuring that the current Raft leader is restarted last.

sudo systemctl restart consul

Enable Consul and Envoy Logging to File

Consul Agent Configuration File

Ensure the following entries are set in any Consul agent configuration files that are desired for log exports to Datadog.

log_level          = "DEBUG"
syslog_facility    = "LOCAL1"
log_file           = "/var/log/consul/consul.log"

Envoy Extra Args 

Ensure any Envoy sidecar proxies are set to log at the desired level and to a known accessible log location. Below example uses the following extra arguments as an amendment to the consul connect envoy command:

-- --log-level trace --log-path /var/log/envoy/web-proxy.log --component-log-level upstream:trace,http:trace,router:trace,config:trace

# Example Sidecar Extra Config for Logging

consul connect envoy -sidecar-for web-1 -- --log-level trace --log-path /var/log/envoy/backend-proxy.log --component-log-level upstream:trace,http:trace,router:trace,config:trace

Expose Envoy Sidecar Proxy Metrics

# File: proxy-defaults.hcl
Kind        = "proxy-defaults"
Name        = "global"
Config      = {
  envoy_prometheus_bind_addr = "127.0.0.1:9102"
  envoy_dogstatsd_url        = "udp://127.0.0.1:8125"
}

# web.hcl - web service and sidecar proxy registration
service {
  name = "web"
  id   = "web-1"
  port = 8080

  connect {
    sidecar_service{
      proxy = {
        config = [
          {
              envoy_prometheus_bind_addr = "127.0.0.1:9102"
              envoy_dogstatsd_url        = "udp://127.0.0.1:8125"
          }
        ]
      }
    }
  }

If using global enablement, ensure to write configuration changes to Consul:

consul config write proxy-defaults.hcl

Re-run/restart Envoy's bootstrap from consul connect envoy command.

# systemd for web-sidecar-proxy
# ExecStart=consul connect envoy -sidecar-for web-1 -address

systemctl restart web-sidecar-proxy

Create Datadog Account

Create a Datadog account following the instructions found on Datadog's website.

Create Datadog API Key

To add a Datadog API key:

1. Login to Datadog using your account's username and password.
2. Navigate to Organization settings, then click the API keys tab.
3. Click the New Key button.
4. Enter a name for your key or token.
5. Click Create API key.
6. Take note of Key ID for later use when installing and configuring agent.

Official Datadog procedure found here.

Installing Datadog Agent on Host VM

There are two methods for installing and configuring the Datadog agent daemon on the Host Virtual Machine. Automatically using an s3 maintained script or manually. The steps below go through manually configuring Datadog to capture the concepts of configuration and generate the required Consul and Envoy specific files and directories.

• If not already done, copy Key ID from Step 6 of Create Datadog API Key above.
• Continue with configuration steps below.

Install and Configure Datadog for Consul and Envoy

Verify Consul to Datadog Agent Communication

Verify UDP DogstatsD Connection is Established with Consul

sudo netstat -nup | grep "127.0.0.1:8125.*ESTABLISHED"

udp 0 0 127.0.0.1:53874 127.0.0.1:8125 ESTABLISHED 23176/consul

sudo -u dd-agent -- datadog-agent check consul

sudo -u dd-agent -- datadog-agent check envoy

References:

• Datadog Consul Integration Guide
• Monitoring Consul with Datadog
• Datadog Envoy Configuration