Prometheus

Kubernetes configuration

The metric sources need to be defined in the agent configuration file <agent_install_dir>/etc/instana/configuration.yaml as a list of endpoints. The agent automatically recognizes when it is running in a Kubernetes environment, and gathers IPs and container ports from running pods. For Kubernetes environments, the url field should not contain a host and port information, only the metrics endpoint URL for example /prometheus/endpoint/metrics.

With a regular expression, you can define which metrics should be captured for a specific metric endpoint in the metricNameIncludeRegex. Prometheus sensor has basic authentication support that can be defined on the global level (all endpoints) or per an endpoint.

com.instana.plugin.prometheus:
  # Global polling interval in seconds (optional) 
  poll_rate: 15                              # Default is 1 second
  # Global (all) endpoints username/password configuration (optional)
  username: ''
  password: ''
  customMetricSources:
  - url: '/prometheus/endpoint/1'            # metrics endpoint, the IP and port are auto-discovered
    metricNameIncludeRegex: '.*'             # regular expression to filter metrics
    username: ''                             # endpoint specific username/password configuration
    password: ''
  - url: '/prometheus/endpoint/2'
    metricNameIncludeRegex: '.*'

Note: the sensor is not going to monitor endpoints specified as full URLs, for example, https://prod-myapp.server.com/prometheus/metrics.

Static configuration

The static configuration is meant to be used for non-Kubernetes environments. If you want to capture Prometheus metrics from local or remote endpoints, you can specify them using the customMetricSources configuration list.

com.instana.plugin.prometheus:
  # Global polling interval in seconds (optional) 
  poll_rate: 15                              # Default is 1 second
  # Global (all) endpoints username/password configuration (optional)
  username: ''
  password: ''
  customMetricSources:
  - url: 'http://localhost:8080/metrics'
    username: ''      # endpoint specific username/password configuration
    password: ''
    metricNameIncludeRegex: '^sample_app_request'
  - url: 'http://223.58.1.10:9100/prometheus'
    metricNameIncludeRegex: '^sample_app_request'

In case the metricNameIncludeRegex is not defined, the sensor collects all metric types up to the defined limit of 600 metrics per metric type.

Infrastructure correlation

For detailed information on infrastructure correlation on Linux hosts, review the documentation on the agent's HTTP API endpoint.

TLS encryption for remote write

TLS encryption can be enabled on the Agent, so that all data sends to the remote_write endpoint is TLS encrypted as well. See Enabling TLS Encryption for more information on how to set this up.

Alert channel configuration

Prometheus Alertmanager Webhook Receiver URL: http://{SNMP-Notifier-Host}:9464/alerts.

SNMP notifier configuration

To configure the SNMP Notifier service, specify the following as container environment variables:

SNMP_VERSION = SNMP version. "V2c" and "V3" are currently supported. (Default: "V2c")
SNMP_DESTINATION = SNMP trap server destination. (Default: "127.0.0.1:162")
SNMP_RETRIES = SNMP number of retries. (Default: "1")
SNMP_TRAP_DEFAULT_OID = Trap OID to send if none is found in the alert labels. (Default: "1.3.6.1.4.1.98789.0.1")
SNMP_COMMUNITY = SNMP community (V2c only). (Default: "public")
SNMP_AUTHENTICATION_ENABLED = Enable SNMP authentication (V3 only). (Default: "false")
SNMP_AUTHENTICATION_PROTOCOL = Protocol for password encryption (V3 only). MD5 and SHA are currently supported. (Default: MD5)
SNMP_AUTHENTICATION_USERNAME = SNMP authentication username (V3 only).
SNMP_AUTHENTICATION_PASSWORD = SNMP authentication password (V3 only).
SNMP_PRIVATE_ENABLED = Enable SNMP encryption (V3 only). (Default: "false")
SNMP_PRIVATE_PROTOCOL = Protocol for SNMP data transmission (V3 only). DES and AES are currently supported (Default: "DES").
SNMP_PRIVATE_PASSWORD =  SNMP private password (V3 only).
SNMP_SECURITY_ENGINE_ID = SNMP security engine ID (V3 only).
SNMP_CONTEXT_ENGINE_ID = SNMP context engine ID (V3 only).
SNMP_CONTEXT_NAME = SNMP context name (V3 only).

Run with Docker

1. Log in to Instana Docker registry.

   sudo docker login https://containers.instana.io -u="_" -p="${AGENT_KEY}"
   

2. Start Docker container.

   sudo docker run -p 9464:9464/tcp --env SNMP_DESTINATION=127.0.0.1:162 --env GROUP_ID=1001 --env USER_ID=1002 containers.instana.io/instana/release/product/snmpnotifier:2.164.224-0
   

Run in Kubernetes

1. Create secret for instana docker registry

   kubectl create secret docker-registry instana \
    --docker-server=https://containers.instana.io \
    --docker-username=_ \
    --docker-password=${AGENT_KEY}
   

2. Deployment and service descriptor

   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: instana-snmp-notifier-deployment
     labels:
       instana.io/component: instana-snmp-notifier
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: instana-snmp-notifier
     template:
       metadata:
         labels:
           app: instana-snmp-notifier
       spec:
         containers:
           - env:
               - name: GROUP_ID
                 value: '1001'
               - name: USER_ID
                 value: '1002'
               - name: SNMP_DESTINATION
                 value: 127.0.0.1:162
             image: containers.instana.io/instana/release/product/snmpnotifier:2.164.224-0
             name: instana-snmp-notifier
             ports:
               - containerPort: 9464
                 name: web-endpoint
                 protocol: TCP
             resources:
               limits:
                 cpu: 500m
                 memory: 512Mi
               requests:
                 cpu: 500m
                 memory: 256Mi
         imagePullSecrets:
           - name: instana
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: instana-snmp-notifier-service
   spec:
     type: ClusterIP
     ports:
       - name: web-endpoint
         port: 9464
         targetPort: 9464
         protocol: TCP
     selector:
       app: instana-snmp-notifier