How-to use OpenTelemetry with Agent metrics - Docker

Steps

Introduction

The HCP Terraform agent can export metrics to an OpenTelemetry Collector as documented in the agent telemetry documentation.

This article provides an example of using the OpenTelemetry fileexporter to write agent telemetry data to a local file.

Expected Outcome

After completing this guide, you will have a working Docker-based example of an HCP Terraform agent sending metrics to an OpenTelemetry collector.

Prerequisites

• An HCP Terraform or Terraform Enterprise instance.
• A configured Agent Pool within HCP Terraform or Terraform Enterprise.
• Docker installed on a server to run the agent and collector containers.

Procedure

1. Connect to the server where you will start the OpenTelemetry Collector container.

2. Create a directory named exported_data. The collector will write the output file to this directory.

   $ mkdir exported_data

3. Create a file named telemetry.yaml with the following content. This configuration instructs the collector to write telemetry data to /exported_data/telemetry_output.json inside the container.

   extensions:
     memory_ballast:
       size_mib: 128
   receivers:
     otlp/tfe_agent:
       protocols:
         grpc:
           endpoint: 0.0.0.0:4317
   processors:
     batch:
     memory_limiter:
       check_interval: 1s
       limit_mib: 1024
       spike_limit_percentage: 20
   exporters:
     file/no_rotation:
       path: /exported_data/telemetry_output.json
     logging:
       file:
         path: /dev/null
   service:
     extensions: [memory_ballast]
     pipelines:
       traces:
         receivers: [otlp/tfe_agent]
         exporters: [file]
       metrics/tfe_agent:
         receivers: [otlp/tfe_agent]
         processors: [memory_limiter, batch]
         exporters: [file/no_rotation]
     telemetry:
       logs:
         level: "debug"

4. Verify your directory structure.

   .
   ├── exported_data
   └── telemetry.yaml

5. Start the OpenTelemetry container using Docker. This command mounts your configuration and output directory into the container.

   $ docker run -d \
     --name collector \
     --volume $(pwd)/telemetry.yaml:/etc/otel/config.yaml \
     --volume $(pwd)/exported_data:/exported_data \
     -p 4317:4317 \
     --rm otel/opentelemetry-collector-contrib:0.73.0 \
     --config /etc/otel/config.yaml

6. Set the required environment variables for the HCP Terraform agent. The TFC_AGENT_OTLP_ADDRESS variable must point to the address where your OpenTelemetry container is running.

   $ export TFC_ADDRESS="https://<TFE_HOSTNAME>"
   $ export TFC_AGENT_TOKEN="<AGENT_TOKEN>"
   $ export TFC_AGENT_NAME="<AGENT_NAME>"
   $ export TFC_AGENT_OTLP_ADDRESS="<IP_ADDRESS_OR_HOSTNAME>:4317"

7. Start the HCP Terraform agent container.

   $ docker run \
     -e TFC_ADDRESS \
     -e TFC_AGENT_TOKEN \
     -e TFC_AGENT_NAME \
     -e TFC_AGENT_OTLP_ADDRESS \
     hashicorp/tfc-agent:1.14.1

8. Trigger a run in HCP Terraform or Terraform Enterprise that uses this agent. The agent sends metrics to the OpenTelemetry collector, which then writes them to the telemetry_output.json file in the exported_data directory.

9. Verify the output file contains telemetry data. The contents will be similar to the following example.

   {"key":"agent_name","value":{"stringValue":"agent-tele"}},{"key":"agent_pool_id","value":{"stringValue":"apool-RjP1Tazzy1HU1gGE"}}

Additional Information

• For a list of available metrics, refer to the HCP Terraform Agent metrics documentation.
• The OpenTelemetry Collector repository contains additional information about the collector.
• Details on the OpenTelemetry fileexporter are available in its official documentation.