Collecting metrics data with Prometheus is becoming more popular. With Instana, it is easy to capture Prometheus metrics and correlate them using our extensive knowledge graph. A typical example is custom business metrics.

The Instana prometheus sensor doesn't require a Prometheus server. The sensor captures metrics directly from the endpoints that are exposed by the monitored systems. For each Instana agent, specify which Prometheus endpoints you want to poll and which metrics should be collected from them using regular expressions.

Note that if your metrics are exposed by a Java library (like Micrometer, or the Prometheus Java client library, or jmx_exporter), there is no need to expose them via HTTP because our Java plug-in can collect metrics from these libraries directly:



Kubernetes configuration

The metric sources need to be defined in the agent configuration file <agent_install_dir>/etc/instana/configuration.yaml as a list of endpoints. The agent automatically recognizes when it is running in a Kubernetes environment, and gathers IPs and container ports from running pods. For Kubernetes environments, the url field should not contain a host and port information, only the metrics endpoint URL for example /prometheus/endpoint/metrics.

With a regular expression, you can define which metrics should be captured for a specific metric endpoint in the metricNameIncludeRegex. Prometheus sensor has basic authentication support that can be defined on the global level (all endpoints) or per an endpoint.

  # Global polling interval in seconds (optional) 
  poll_rate: 15                              # Default is 1 second
  # Global (all) endpoints username/password configuration (optional)
  username: ''
  password: ''
  - url: '/prometheus/endpoint/1'            # metrics endpoint, the IP and port are auto-discovered
    metricNameIncludeRegex: '.*'             # regular expression to filter metrics
    username: ''                             # endpoint specific username/password configuration
    password: ''
  - url: '/prometheus/endpoint/2'
    metricNameIncludeRegex: '.*'

Note: the sensor is not going to monitor endpoints specified as full URLs, for example,

Static configuration

The static configuration is meant to be used for non-Kubernetes environments. If you want to capture Prometheus metrics from local or remote endpoints, you can specify them using the customMetricSources configuration list.

  # Global polling interval in seconds (optional) 
  poll_rate: 15                              # Default is 1 second
  # Global (all) endpoints username/password configuration (optional)
  username: ''
  password: ''
  - url: 'http://localhost:8080/metrics'
    username: ''      # endpoint specific username/password configuration
    password: ''
    metricNameIncludeRegex: '^sample_app_request'
  - url: ''
    metricNameIncludeRegex: '^sample_app_request'

In case the metricNameIncludeRegex is not defined, the sensor collects all metric types up to the defined limit of 600 metrics per metric type.

Remote write

Starting with Agent bundle 1.1.587, the Instana Agent includes support for the remote_write endpoint, meaning it is able to ingest metrics and these are displayed as either a Prometheus Entity or part of the Process Custom Metrics.

To enable the remote_write endpoint, add the following to the agent's configuration.yaml:

    enabled: true

See the documentation for prometheus' remote_write on how to set up the sender (the component that sends the metrics). The remote_write endpoint is available on port 42699 at the /prometheus/v1/receive path. Therefore, the URL you need to configure in the prometheus configuration is going to be http://<agent_ip>:42699/prometheus/v1/receive.

Note: The optional Instana Agent Service provided on Kubernetes via the Instana Agent Helm Chart is very useful in combination with the remote_write API, as it ensures that the data is pushed to the Instana Agent running on the same Kubernetes node, ensuring the Instana Agent can correctly fill in the infrastructure correlation data.

Note: Currently, authentication is not supported for the remote_write endpoint, so the basic_auth and bearer_token (including bearer_token_file) options of prometheus should not be configured in the sender.

IMPORTANT: In order for Instana to be able to parse metrics correctly, the sender must send the metadata. This is the default in prometheus, so be sure not to turn it off.

Infrastructure correlation

For detailed information on infrastructure correlation on Linux hosts, review the documentation on the agent's HTTP API endpoint.

TLS encryption for remote write

TLS encryption can be enabled on the Agent, so that all data sends to the remote_write endpoint is TLS encrypted as well. See Enabling TLS Encryption for more information on how to set this up.

Metrics collection

To view the metrics, select Infrastructure in the sidebar of the Instana User interface, click a specific monitored host, and then you can see a host dashboard with all the collected metrics and monitored processes.

Prometheus metrics appear as "Prometheus Apps" associated with the host or with the process from where they are collected (in case of using the remote_write endpoint). You can query prometheus custom metrics using the "Dynamic Focus", "Event and Alerts", and the "Grafana plug-in" with entity.type:prometheus.

The sensor collects all core metric types, up to 600 metrics per type:

  • Counters
  • Gauges
  • Histograms
  • Summaries
  • Untyped


Instana provides a Prometheus Alertmanager webhook alert channel to send HTTP POST requests with the payload format of the Prometheus Alertmanager Webhook described at Prometheus Alertmanager Webhook Receiver configuration.

To configure a Prometheus Alertmanager Webhook alert channel, go to "Settings > Team Settings > Events & Alerts > Alert Channels > Add Alert Channel" and click "Prometheus Alertmanager Webhook".

Screenshot: Prometheus Alertmanager Webhook

Instana sends alerts through this alert channel as HTTP POST requests to the configured Prometheus Alertmanager Webhook Receiver, for example Alert Snitch or SNMP Notifier.

SNMP notifier

The SNMP Notifier project relays prometheus alerts as SNMP traps to any configured SNMP receiver.

To ensure proper interaction with Instana alerts, we provide a Docker image for SNMP Notifier with a custom alert mapping in our private Docker registry, see Run with Docker.

Alert channel configuration

Prometheus Alertmanager Webhook Receiver URL: http://{SNMP-Notifier-Host}:9464/alerts.

SNMP notifier configuration

To configure the SNMP Notifier service, specify the following as container environment variables:

SNMP_VERSION = SNMP version. "V2c" and "V3" are currently supported. (Default: "V2c")
SNMP_DESTINATION = SNMP trap server destination. (Default: "")
SNMP_RETRIES = SNMP number of retries. (Default: "1")
SNMP_TRAP_DEFAULT_OID = Trap OID to send if none is found in the alert labels. (Default: "")
SNMP_COMMUNITY = SNMP community (V2c only). (Default: "public")
SNMP_AUTHENTICATION_ENABLED = Enable SNMP authentication (V3 only). (Default: "false")
SNMP_AUTHENTICATION_PROTOCOL = Protocol for password encryption (V3 only). MD5 and SHA are currently supported. (Default: MD5)
SNMP_AUTHENTICATION_USERNAME = SNMP authentication username (V3 only).
SNMP_AUTHENTICATION_PASSWORD = SNMP authentication password (V3 only).
SNMP_PRIVATE_ENABLED = Enable SNMP encryption (V3 only). (Default: "false")
SNMP_PRIVATE_PROTOCOL = Protocol for SNMP data transmission (V3 only). DES and AES are currently supported (Default: "DES").
SNMP_PRIVATE_PASSWORD =  SNMP private password (V3 only).
SNMP_SECURITY_ENGINE_ID = SNMP security engine ID (V3 only).
SNMP_CONTEXT_ENGINE_ID = SNMP context engine ID (V3 only).
SNMP_CONTEXT_NAME = SNMP context name (V3 only).

Run with Docker

  1. Log in to Instana Docker registry.

    sudo docker login -u="_" -p="${AGENT_KEY}"
  2. Start Docker container.

    sudo docker run -p 9464:9464/tcp --env SNMP_DESTINATION= --env GROUP_ID=1001 --env USER_ID=1002

Run in Kubernetes

  1. Create secret for instana docker registry

    kubectl create secret docker-registry instana \
     --docker-server= \
     --docker-username=_ \
  2. Deployment and service descriptor

    apiVersion: apps/v1
    kind: Deployment
      name: instana-snmp-notifier-deployment
      labels: instana-snmp-notifier
      replicas: 1
          app: instana-snmp-notifier
            app: instana-snmp-notifier
            - env:
                - name: GROUP_ID
                  value: '1001'
                - name: USER_ID
                  value: '1002'
                - name: SNMP_DESTINATION
              name: instana-snmp-notifier
                - containerPort: 9464
                  name: web-endpoint
                  protocol: TCP
                  cpu: 500m
                  memory: 512Mi
                  cpu: 500m
                  memory: 256Mi
            - name: instana
    apiVersion: v1
    kind: Service
      name: instana-snmp-notifier-service
      type: ClusterIP
        - name: web-endpoint
          port: 9464
          targetPort: 9464
          protocol: TCP
        app: instana-snmp-notifier


Remote write high metric delay

Monitoring issue type: prometheus_remote_write_high_delay

Prometheus metrics that are being ingested from the remote_write endpoint, are being received with a high delay. This results in potentially delayed alerting and makes correlating metrics from other sources more difficult.

Look at tuning the Prometheus remote_write configuration. Specifically, adding the batch_send_deadline parameter as defined as follows, which limits the delay to 1 second:

  - url: ""
      batch_send_deadline: 1s

See the full configuration options in the Prometheus manual.