Host agent configuration

See the following host agent configurations:

Backend configurations

The <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg file contains the configurations that are used by the host agent to communicate with the Instana backend.

The values of the <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg file can be overridden by using the following environment variables:

Configuring multiple backends

In some cases, it might be desirable to have an agent report to multiple backends. For example, there are shared services that are used by separated environments. It is possible to manually configure agents to achieve this goal. The agent is counted in all backends separately as consuming a license, and the setup effectively multiplies the bandwidth consumption of the agent.

To have the host agent report to multiple backends, follow the steps:

  1. Rename the <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg file to <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend-1.cfg
  2. Create copies <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend-2.cfg, with the appropriate configurations for various backends that the agent reports to.

Each of files that are created in the previous step can be adjusted to describe a different host agent endpoint data and agent keys, and can even contain different proxy settings.

Notes:

Setting up an agent proxy

To communicate with the backend effectively, Instana uses the HTTP/2 protocol to transfer data. In many cases, direct communication from the host agent to the backend can be granted, which simplifies the agents' deployment. In some cases, one dedicated entry goes into or out of the network is required. For this reason, Instana can be used in combination with various proxies. In general, http, https, socks4s, and socks5 proxies are supported. The proxy needs to support CONNECT to pass through.

Note: A proxy that terminates the SSL connection and then tries to manage its own connection to the Instana backend is not supported. Secure HTTP/2 is used without ALPN.

For the host agent configuration, you need to modify two files. In the file <instana-agent-dir>/etc/mvn-settings.xml, the following information must be present, and not commented out in the <settings></settings> section:

<proxies>
  <proxy>
    <id>agent-proxy</id>
    <active>true</active>
    <protocol>http</protocol>
    <username>user-if-needed</username>
    <password>password-if-needed</password>
    <host>your-proxy-address-goes-here</host>
    <port>your-proxy-port-goes-here</port>
  </proxy>
</proxies>

It enables the Instana agent to download agent updates and extra features automatically. Furthermore, the file <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg needs to be reconfigured to use the proxy for the communication between the agent and the Instana backend. Make sure that the following lines are present and not commented out:

proxy.type=http
proxy.host=your-proxy-address-goes-here
proxy.port=your-proxy-port-goes-here
proxy.user=user-if-needed
proxy.password=password-if-needed
proxy.dns=true

Squid proxy configuration example

This document describes the configuration of a squid proxy (www.squid-cache.org) in combination with Instana, when there is no other proxies available.

There are several ways to install Squid on your system. Most Linux® distributions include Squid in their repositories, and the software can be installed by using the preferred package manager. In case no package is available, or you want to run Squid on Microsoft® Windows®, you can get Squid binaries from this location: http://wiki.squid-cache.org/SquidFaq/BinaryPackages. Once Squid is installed, an example configuration squid.conf is created. If you want to use the proxy for the Instana communication exclusively, you can back up the default configuration, and use this minimal squid.conf:

http_port 3128

acl instana_agent_net src 10.0.0.0/8

acl instana_backend dstdomain saas-eu-west-1.instana.io
#acl instana_backend dstdomain ec2-54-144-114-141.compute-1.amazonaws.com
#acl instana_backend dstdomain saas-us-east-1.instana.io
#acl instana_backend dstdomain saas-us-east-1.instana.io

acl instana_backend_port port 443

acl instana_repo dstdomain artifact-public.instana.io
acl instana_repo_port port 80
acl instana_repo_port_secure port 443

acl instana_backend_proto proto HTTP

acl instana_repo_proto proto HTTP
acl instana_repo_proto_secure proto HTTPS

http_access allow instana_agent_net instana_backend instana_backend_port
http_access allow instana_agent_net instana_repo instana_repo_port
http_access allow instana_agent_net instana_repo instana_repo_port_secure

http_access deny all

Setting up TLS Encryption for Agent Endpoint

Normally, network connections to the agent (HTTP on port 42699 and gRPC on port 4317) are not encrypted. TLS encryption can be enabled by adding certificates in the <agent_installation>/etc/certs/ folder. By default, the agent looks for the following files:

Other names are also allowed for the .crt and .key file if there is only one file of each in that folder.

After you add the certificates, restart the agent so that it will properly initialize the network connections.

Important: Currently, the agent does not yet allow _enforcing_ TLS encryption. TLS is only enabled on a connection when it's requested by the client.

Configuring host agent mode

Note: The One-Liner host agent installation method and the Instana Host Agent Docker image accept an extra mode, AWS mode. The AWS mode of the host agent is not used for monitoring hosts. It is simply the INFRASTRUCTURE mode plus some automatic configuration of AWS data collection, as described in the AWS Agent documentation.

The host agent mode is set by using the config file <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg:

mode = APM

Changing the <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg configuration file requires a restart of the host agent to take effect.

Preventing agent mode override from UI

As the agent mode can also be configured from the Instana UI, there is a flag in configuration file that might be used to disable this overriding. In this way, the agent mode can be configured only by using the configuration file or an environment variable locally to the installed agent.

If you don't want to allow setting the agent mode setting from UI, add the following to the <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg configuration file:

mode.web-override.allowed = false

Agent configuration file

Note: The format of this file is YAML, which is sensitive to white space. All indention levels allow only two blank spaces.

Most host agent configurations are applied by using one central configuration file in the location <instana-agent-dir>/etc/instana/configuration.yaml.

Multiple configuration files

To provide modularity in configurations, it is possible to create files <instana-agent-dir>/etc/instana/configuration-<suffix>.yaml. Replace <suffix> with an alphanumeric combination. All <instana-agent-dir>/etc/instana/configuration-<suffix>.yaml files are merged with <instana-agent-dir>/etc/instana/configuration.yaml in file system order. So configuration-cde.yaml comes after configuration-abc.yaml. Nested structures that are specified in different configuration files like tags are merged, and values are overwritten.

Integration with secret managers

The host agent can retrieve confidential data like credentials or other information that you do not want to put in clear text in the configuration.yaml file from secret managers.

Vault

Note: The Vault integration requires agent boostrap version 1.2.9 or higher.

Instana agent uses HashiCorp Vault to securely obtain values for sensitive settings in the agent configuration file. The following Vault integration configuration parameters need to be provided to the agent:

com.instana.configuration.integration.vault:
  connection_url: <vault_server_base_url> # The address (URL) of the Vault server instance(e.g. http://127.0.0.1:8200 or https://exapmle.com:8200)
  token: <vault_access_token> # Vault access token with assigned, at least, `read` access policy to relevant Vault paths
  path_to_pem_file: <path_to_X.509_CA_certificate> # X.509 CA certificate (UTF8 encoded) in unencrypted PEM format, used by the Agent when communicating with Vault over HTTPS
  secret_refresh_rate: 24 # This configuration option allows you to account for rotating credentials, refresh rate in hours, default 24
  kv_version: 2 # The Key/Value secrets engine version, default is 2

After the initial Vault configuring, you can specify the retrieval of secrets for various sensors as follows:

com.instana.plugin.mysql:
  user: 'instana'
  password:
    configuration_from:
      type: vault
      secret_key:
        path: <vault_path>
        key: <vault_secret_key>

For comparison, to configure the password in plain-text in the configuration.yaml file, see the following settings:

com.instana.plugin.mysql:
  user: 'instana'
  password: <my_password>

That is, you can swap out the string value for the YAML structure specifying the Vault coordinates, and the host agent automatically does the rest.

Also, you are not limited to using the Vault integration with a field that contains the password, but you can use it for every configuration that has a string as value in YAML.

For more information, see the following Vault concepts, relevant for the Instana agent integration:

Configurations from process environment and files

You might have cases in which you do not want to hardcode credentials or other configuration items in your configuration.yaml. Or you might have several instances of one technology (such as several MySQL nodes) on the same node, and each instance needs different configurations.

In these cases, the host agent can read configuration values from the process to be monitored (such as the MySQL database) looking up the values from specific environment variables (here only YAML literals are supported, like a string, a Boolean or a number) or as content of specific files (here also complex YAML structures are supported, like lists and maps).

There is an intentional symmetry between these capabilities of the host agent and the way you mount secrets in Kubernetes. However, these capabilities are available on all supported Operating Systems and all supported container runtimes.

Note: Numeric values in environment variables and files are converted to integer values, while the Boolean values true and false are converted to Booleans.

Configurations from process environment

For configuration values that are taken from the target process environment, the configuration.yaml looks as follows

com.instana.plugin.mysql:
  user: 'instana'
  password:
    configuration_from:
      type: env
      env_name: <environment variable name>
      default_value: <set this when no env var is found on this process>

If you do not specify a default value, and the environment variable is not present on the process to be monitored (such as the MySQL database), it is like you never entered a value in the configuration.yaml file.

Note: On Kubernetes, this function is designed to work with mounting ConfigMap entries as environment variables in your containers. In other words, you define configurations in ConfigMaps, mount them to the application containers as environment variables, and tell the host agent to look for those environment variables.

Note: Values from environment variables can only be YAML literals, like a string, a Boolean or a number. Complex YAML structures as lists and maps are not supported. The plug-in enabled/disabled setting cannot be changed by using configurations from process environment and files.

Configurations from process file system

For configuration values that are taken from a file, the configuration.yaml looks as follows:

com.instana.plugin.mysql:
  user: 'instana'
  password:
    configuration_from:
      type: file
      file_path: <absolute path to a file containing the configuration value>
      default_value: <set this when the file is not found>
      common_value: <set this in addition to the configuration value from file>

In this example, the agent reads /jmx-config-map.yaml from the process file system. type: file, and default_value accepts complex YAML structures. The common_value property is later merged with every process-specific configuration and is common to all monitored processes:

com.instana.plugin.java:
  jmx:
    configuration_from:
      type: file
      file_path: /jmx-config-map.yaml
      default_value:
        - object_name: 'java.lang:type=OperatingSystem'
          metrics:
          - attributes: 'CommittedVirtualMemorySize'
            type: 'absolute'
      common_value:
        - object_name: 'java.lang:type=OperatingSystem'
          metrics:
          - attributes: 'ProcessCpuLoad'
            type: 'delta'

In this example, /jmx-config-map.yaml gets mounted from a Kubernetes ConfigMap into the application container, providing the following content:

- object_name: 'java.lang:type=Compilation'
  metrics:
  - attributes: 'TotalCompilationTime'
    type: 'delta'
- object_name: 'java.lang:type=ClassLoading'
  metrics:
  - attributes: 'LoadedClassCount'
    type: 'absolute'

Resulting for this process in a final configuration like this. Note the common_value addition for ProcessCpuLoad:

com.instana.plugin.java:
  jmx:
  - object_name: 'java.lang:type=Compilation'
    metrics:
    - attributes: 'TotalCompilationTime'
      type: 'delta'
  - object_name: 'java.lang:type=ClassLoading'
    metrics:
    - attributes: 'LoadedClassCount'
      type: 'absolute'
  - object_name: 'java.lang:type=OperatingSystem'
    metrics:
    - attributes: 'ProcessCpuLoad'
      type: 'delta'

IMPORTANT: If the process to be monitored is running inside a container, the file absolute path is resolved against that container's file system.

Notes:

Monitor extra file systems

Note: These configurations are applied by using the agent configuration files.

By default, the agent monitors local file systems only. To add extra file systems, add the name of the file system (that is the first column in mtab or df).

For example, to monitor the file system server:/usr/local/pub /pub nfs rsize=8192,wsize=8192,timeo=14,intr, add the following line in the uncommented file systems list of the host section:

com.instana.plugin.host:
  filesystems:
    - 'server:/usr/local/pub'

Specify host tags

Note: These configurations are applied by using the agent configuration files.

To specify tags for an agent in the com.instana.plugin.host, add a section in the following format:

com.instana.plugin.host:
  tags:
    - 'production'
    - 'app1'

Alternatively, the INSTANA_TAGS=production,app1 environment variable can be used (tags from environment variable and config file are additive).

Two tags are added to the specific agent, which enables searching and filtering of tags in the UI:

Infrastructure tags

Extract the installed packages list

Note: These configurations are applied by using the agent configuration files.

The host agent can report to Instana which packages are installed on the underpinning Linux operating system.

Note: This configuration is disabled by default.

The following Linux distributions are currently supported:

When the collectInstalledSoftware property is set, installed packages on an operating system are extracted once a day.

com.instana.plugin.host:
  collectInstalledSoftware: false #  Valid values: true, false

Package list

Custom zones

Note: These configurations are applied by using the agent configuration files.

Instana uses Amazon Web Services, Google Compute Engine, or OpenStack Nova availability zone information to group hosts by default. To customize the default host grouping in Instana, a new group for the specific host can be defined in the section com.instana.plugin.generic.hardware by using the following format:

com.instana.plugin.generic.hardware:
  enabled: true
  availability-zone: 'Demozone'

It is also possible to provide this configuration by using the INSTANA_ZONE environment variable, which then overrides the zone that is specified in the configuration file.

The result is that hosts are grouped into zones on the infrastructure map:

Infrastructure zones

Monitor custom processes

Note: These configurations are applied by using the agent configuration files.

Instana automatically monitors process metrics of higher-level sensors like Java® or MySQL by default. If you want to monitor an OS process not automatically covered by Instana, you can configure it using both the process name or its arguments:

com.instana.plugin.process:
  processes:
    - 'sshd'
    - 'slapd'
  arguments:
    - '/opt/script.sh'

Secrets

Tracing data might contain sensitive data. Therefore, the Instana agent supports the specification of patterns for secrets, that is, data to be redacted agent-side from the tracing data. Data treated as secrets can't reach the Instana SaaS for processing and thus is not available for analysis in the UI or retrieval by using API.

Secrets are specified in the following way:

com.instana.secrets:
  # One of: 'equals-ignore-case', 'equals', 'contains-ignore-case', 'contains', 'regex'
  matcher: 'contains-ignore-case'
  list:
    - 'key'
    - 'password'
    - 'secret'

If a key matches an entry from the list, the value is redacted, and not sent to the Instana backend. If you do not specify any custom secrets configuration, Instana uses the afore-mentioned secrets configuration by default.

Note: Generally, secrets are filtered from static data. Filtering is highly optimized and changes to which secrets to scrub will not be applied while a process is running. To make a secret configuration effective, either restart the agent or the running monitored component, which make the new config in effect.

Instana supports secrets at infrastructure and platform level as follows:

Instana supports secrets in runtimes as follows:

Language Secrets in HTTP Query parameters Secrets in HTTP Matrix parameters
Go
Java®
NGINX
Node.js
.NET Core
.NET Framework
PHP
Python
Ruby

Notes:

Similarly, to the case of arbitrary path segments, literals from SQL queries are stripped, and you are recommended to use parametrized queries instead. The only exception lies with Instana's PHP support, which has an option to strip SQL literals (see sanitizeSql configuration option), at a potentially rather steep cost in terms of overhead of the Instana agent.

Capture custom HTTP headers

Note: These configurations are applied by using the agent configuration files.

By default, Instana collects HTTP headers when it traces HTTP calls, but this feature can be enabled on demand as follows:

com.instana.tracing:
  extra-http-headers:
    - 'x-request-id'
    - 'x-loadtest-id'
    - ...

The values are case-insensitive. The headers that are collected are shown in the call details (in the callee details section). You can also use the header names and their values to search for calls and traces (in the UI or by using the API). Finally, they can be used for service configuration.

Restrictions:

Tracer Request Headers on HTTP Entries Response Headers on HTTP Entries Request Headers on HTTP Exits Response Headers on HTTP Exits
Go
Java 1
.NET
Node.js
PHP
NGINX N.A. N.A.
Python
Ruby

Ignore processes

Note: These configurations are applied by using the agent configuration files.

Note for Windows only: The process names in the list are case-sensitive.

To ignore any monitoring of certain processes, uncomment the com.instana.ignore section and list all process names of processes that should not be monitored. Sometimes scripts or other jobs are started through the same command, but only the execution of a single command should be ignored. In this case, use arguments to specify any argument that should cause a process to be ignored.

com.instana.ignore:
  processes:
    - 'java'
    - 'httpd'
  arguments:
    - '-batch-file=/tmp/batch.def'

Deactivate agent capabilities triggered by UI

To tail the Instana Agent logs, do diagnostic actions on the agent and to enable the Trace Code View, users can trigger those capabilities through the Instana UI. In case you want to deactivate all of these capabilities, you can set the config by adding to: <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg

backchannel.enabled = false

Note: Commands are not sent directly to the Instana Agent, and they are received as response to sending new metrics.

Upload code source files to Instana

The Instana agent can retrieve on-demand source code of the processes that it monitors and associate it with the collected tracing data.

To disable the on-demand upload of source files into Instana, you can set the following configuration in the <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg file:

source.download.enabled = false

Note: The Instana Agent needs to be restarted to apply this setting.

Agent logging

By default, the Instana Agent logs to <instana-agent-dir>/data/log/agent.log, which will also be rotated if the file grows too large. If you are running the agent in a container, it logs to the console instead, which is managed by Docker and accessible by using docker logs.

The log level can be increased to debug by changing the level in the configuration file <instana-agent-dir>/etc/org.ops4j.pax.logging.cfg from:

log4j2.logger.instana.level=INFO

to:

log4j2.logger.instana.level=DEBUG

Note: Ensure that there is no trailing white space on the line.

Older agent installs need this line to be changed from:

log4j.logger.com.instana=INFO, out, osgi:*

to:

log4j.logger.com.instana=DEBUG, out, osgi:*

Log rotation

By default, the agent uses a log rotation of 10 times 5MB agent log files.

log4j2.appender.rolling.policy.type = SizeBasedTriggeringPolicy
log4j2.appender.rolling.policy.size = 5MB
log4j2.appender.rolling.strategy.type = DefaultRolloverStrategy
log4j2.appender.rolling.strategy.max = 10

Syslog

The Agent now uses Log4j2, which is a modern and flexible logging facility. For example, syslog can be configured like this:

log4j2.rootLogger.appenderRef.Syslog.ref = Syslog
log4j2.rootLogger.appenderRef.Syslog.level = ERROR
log4j2.appender.syslog.type=Syslog
log4j2.appender.syslog.name=Syslog
log4j2.appender.syslog.layout.type=PatternLayout
log4j2.appender.syslog.layout.pattern = ${log4j2.pattern}
log4j2.appender.syslog.facility=SYSLOG
log4j2.appender.syslog.host=localhost
log4j2.appender.syslog.port=514
log4j2.appender.syslog.protocol=UDP

Logs to STDOUT

In custom container images, you might want to log to STDOUT. You can do so by providing the following configuration in <instana-agent-install-dir>etc/org.ops4j.pax.logging.cfg:

log4j2.rootLogger.appenderRef.Console.ref = Console

log4j2.appender.console.type = Console
log4j2.appender.console.name = Console
log4j2.appender.console.layout.type = PatternLayout
log4j2.appender.console.layout.pattern = ${log4j2.pattern}

Log4j2

Because the logging is using the standard log4j2 logging, it can be configured in many more ways as described in the log4j2 documentation.

Log metrics or traces to file

The agent can temporarily log metrics or traces that are sent through that agent to a file on disk. It's often used to debug various issues that are related to metrics, traces, tracing, or spans.

To enable the feature, locate the file <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.File.cfg and update its contents with:

prefix=locallog

type=traces

As noted, the changes are hot-reloaded and can be activated immediately. It should only be done temporarily because leaving the feature enabled has the potential to fill up all available disk space given enough time and traffic.

Allow the logging to continue for a minute or two while generating traffic to the component in question when you trace issues. Then, revert the changes or comment all lines in that file. Again, changes that are written to disk are hot-reloaded by the Instana agent.

If you are working with a support ticket, attach the resulting log file to the support ticket. The log file is located in <instana-agent-dir>/data/log/locallog_*.log.

Limiting host agent CPU and memory

In certain scenarios, it is crucial to control resource consumption of processes closely. It can become especially handy in environments that use resource sharing and for systems with extremely low resources. While the Instana agent is designed to consume as few resources as possible it can be limited by doing the following instructions as another safeguard to respect clear resource limits.

All following examples limit Agent's CPU to computing resources equivalent to half a CPU (however, not all resources would necessarily come from the same CPU) and to a maximum of 512 MB memory.

Systemd

Create file called /etc/systemd/system/instana-agent.service.d/20-resource_limits.conf, and add the following content to it:

[Service]
CPUAccounting=true
CPUQuota=50%
MemoryAccounting=true
MemoryMax=512M

Run systemctl daemon-reload and restart the instana-agent service.

Docker

Run the instana-agent container with the following extra parameters:

Docker version 1.13 and newer:

--cpus=0.5 --memory=512m

Docker version 1.12 and older:

--cpu-period=100000 --cpu-quota=50000 --memory=512m

Kubernetes

Add the following configuration snippet to the agent's container configuration:

livenessProbe:
  httpGet: # Agent liveness is published on localhost:42699/status
    path: /status
    port: 42699
  initialDelaySeconds: 75
  periodSeconds: 5
resources:
  requests:
    memory: "256Mi"
    cpu: "0.5"
  limits:
    memory: "512Mi"
    cpu: "1.0"

The configuration also requests less memory and CPU for the instana-agent container.


  1. Supported technologies include Servlet, Spring, Tomcat, http4s.