Configuring host agents

Configuring multiple backends

In some cases, it might be desirable to have an agent report to multiple Instana backends. For example, seperated environments use the shared services. You can manually configure agents to achieve this goal. The network bandwidth that is used by the agent multiplies effectively when the agent sends the same data to multiple Instana backends.

To have the host agent report to multiple backends, follow the steps:

1. Rename the <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg file to <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend-1.cfg
2. Create copies <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend-2.cfg, with the appropriate configurations for various backends that the agent reports to.

Each of files that are created in the previous step can be adjusted to describe a different host agent endpoint data and agent keys, and can even contain different proxy settings.

Notes:

• Instead of 1, 2 and 3, any unique alphanumeric ID can be used.

• If a file <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg exists, all other backend files are silently ignored.

• The Instana host agent Docker images are specially configured so that adding extra backends can be easily done by mounting the backend files, such as com.instana.agent.main.sender.Backend-2.cfg.

  An example argument for a dockerized agent is as follows:

  --volume <path-to-additional-backend-config>:/opt/instana/agent/etc/instana/com.instana.agent.main.sender.Backend-2.cfg
  

Squid proxy configuration example

This document describes the configuration of a squid proxy (www.squid-cache.org) in combination with Instana, when there is no other proxies available.

There are several ways to install Squid on your system. Most Linux® distributions include Squid in their repositories, and the software can be installed by using the preferred package manager. In case no package is available, or you want to run Squid on Microsoft® Windows®, you can get Squid binaries from this location: http://wiki.squid-cache.org/SquidFaq/BinaryPackages. Once Squid is installed, an example configuration squid.conf is created. If you want to use the proxy for the Instana communication exclusively, you can back up the default configuration, and use this minimal squid.conf:

# The tcp port squid is listening on
http_port 3128

# Please specify subnet with instana agents
acl instana_agent_net src 10.0.0.0/8

# This is the ip of the instana backend
acl instana_backend dstdomain saas-eu-west-1.instana.io
#acl instana_backend dstdomain ec2-54-144-114-141.compute-1.amazonaws.com
#acl instana_backend dstdomain saas-us-east-1.instana.io
#acl instana_backend dstdomain saas-us-east-1.instana.io

# This is the port used by Instana
acl instana_backend_port port 443

# This is the repo to download updates and additional sensors
acl instana_repo dstdomain artifact-public.instana.io
acl instana_repo_port port 80
acl instana_repo_port_secure port 443

# Protocol used for instana backend
acl instana_backend_proto proto HTTP

# Protocol used for instana backend
acl instana_repo_proto proto HTTP
acl instana_repo_proto_secure proto HTTPS

http_access allow instana_agent_net instana_backend instana_backend_port
http_access allow instana_agent_net instana_repo instana_repo_port
http_access allow instana_agent_net instana_repo instana_repo_port_secure

# DO NOT REMOVE THIS RULE!
http_access deny all

Monitoring issues

Monitoring issue type: agent_tls_cert_expired

The certificate that is used to set up TLS encryption for Agent Endpoint expires. Make sure to replace it with the new certificate file.

Monitoring issue type: agent_tls_cert_about_to_expire

The certificate that is used to set up TLS encryption for Agent Endpoint is about to expire in few days. Make sure to replace it with the new certificate file.

Configuring updates of dynamic host agents

Dynamic host agents can update themselves, and thus reduce the management overhead. To configure the updates of dynamic host agents such as configuring the updating interval, see Configuring updates of dynamic host agents.

Preventing agent mode override from UI

As the agent mode can also be configured from the Instana UI, there is a flag in configuration file that might be used to disable this overriding. In this way, the agent mode can be configured only by using the configuration file or an environment variable locally to the installed agent.

If you don't want to allow setting the agent mode setting from UI, add the following to the <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg configuration file:

mode.web-override.allowed = false

Multiple configuration files

To provide modularity in configurations, it is possible to create files <instana-agent-dir>/etc/instana/configuration-<suffix>.yaml. Replace <suffix> with an alphanumeric combination. All <instana-agent-dir>/etc/instana/configuration-<suffix>.yaml files are merged with <instana-agent-dir>/etc/instana/configuration.yaml in file system order. So configuration-cde.yaml comes after configuration-abc.yaml. Nested structures that are specified in different configuration files like tags are merged, and values are overwritten.

Integration with secret managers

The host agent can retrieve confidential data like credentials or other information that you do not want to put in clear text in the configuration.yaml file from secret managers.

com.instana.configuration.integration.vault:
  connection_url: <vault_server_base_url> # The address (URL) of the Vault server instance(e.g. http://127.0.0.1:8200 or https://exapmle.com:8200)
  prefix: <optional_prefix> # Optional prefix path required if kv_version 2 is used and the /data/ must be injected further down
  token: <vault_access_token> # Vault access token with assigned, at least, `read` access policy to relevant Vault paths, optional if other auth providers are present
  github: # Optional auth method if Vault access token is not provided, has higher priority than approle if present
    github_token: <github_token> # Personal Access Token, must provide at least read:org scope, must be present if github is used as an auth provider
    auth_mount: github # Optional mount path for GitHub Auth, defaults to github
  approle: # Optional auth method if Vault access token or github auth is not provided
    role_id: <roleId> # AppRole RoleId, must be present if approle is used as an auth provider
    secret_id: <secretId> # AppRole SecretId, must be present if approle is used as an auth provider
    auth_mount: approle # Optional mount path for AppRole Auth, defaults to approle
  path_to_pem_file: <path_to_X.509_CA_certificate> # X.509 CA certificate (UTF8 encoded) in unencrypted PEM format, used by the Agent when communicating with Vault over HTTPS
  secret_refresh_rate: 24 # This configuration option allows you to account for rotating credentials, refresh rate in hours, default 24
  kv_version: 2 # The Key/Value secrets engine version, default is 2

com.instana.plugin.mysql:
  user: 'instana'
  password:
    configuration_from:
      type: vault
      secret_key:
        path: <vault_path>
        key: <vault_secret_key>

com.instana.plugin.mysql:
  user: 'instana'
  password: <my_password>

Configurations from process environment and files

You might have cases in which you do not want to hardcode credentials or other configuration items in your configuration.yaml. Or you might have several instances of one technology (such as several MySQL nodes) on the same node, and each instance needs different configurations.

In these cases, the host agent can read configuration values from the process to be monitored (such as the MySQL database) looking up the values from specific environment variables (here only YAML literals are supported, like a string, a Boolean or a number) or as content of specific files (here also complex YAML structures are supported, like lists and maps).

There is an intentional symmetry between these capabilities of the host agent and the way you mount secrets in Kubernetes. However, these capabilities are available on all supported Operating Systems and all supported container runtimes.

Note: Numeric values in environment variables and files are converted to integer values, while the Boolean values true and false are converted to Booleans.

com.instana.plugin.mysql:
  user: 'instana'
  password:
    configuration_from:
      type: env
      env_name: <environment variable name>
      default_value: <set this when no env var is found on this process>

com.instana.plugin.mysql:
  user: 'instana'
  password:
    configuration_from:
      type: file
      file_path: <absolute path to a file containing the configuration value>
      default_value: <set this when the file is not found>
      common_value: <set this in addition to the configuration value from file>

com.instana.plugin.java:
  jmx:
    configuration_from:
      type: file
      file_path: /jmx-config-map.yaml
      default_value:
        - object_name: 'java.lang:type=OperatingSystem'
          metrics:
          - attributes: 'CommittedVirtualMemorySize'
            type: 'absolute'
      common_value:
        - object_name: 'java.lang:type=OperatingSystem'
          metrics:
          - attributes: 'ProcessCpuLoad'
            type: 'delta'

- object_name: 'java.lang:type=Compilation'
  metrics:
  - attributes: 'TotalCompilationTime'
    type: 'delta'
- object_name: 'java.lang:type=ClassLoading'
  metrics:
  - attributes: 'LoadedClassCount'
    type: 'absolute'

com.instana.plugin.java:
  jmx:
  - object_name: 'java.lang:type=Compilation'
    metrics:
    - attributes: 'TotalCompilationTime'
      type: 'delta'
  - object_name: 'java.lang:type=ClassLoading'
    metrics:
    - attributes: 'LoadedClassCount'
      type: 'absolute'
  - object_name: 'java.lang:type=OperatingSystem'
    metrics:
    - attributes: 'ProcessCpuLoad'
      type: 'delta'

Monitor extra file systems

Note: These configurations are applied by using the agent configuration files.

By default, the agent monitors local file systems only. To add extra file systems, add the name of the file system (that is the first column in mtab or df).

For example, to monitor the file system server:/usr/local/pub /pub nfs rsize=8192,wsize=8192,timeo=14,intr, add the following line in the uncommented file systems list of the host section:

com.instana.plugin.host:
  filesystems:
    - 'server:/usr/local/pub'

Specify host tags

Note: These configurations are applied by using the agent configuration files.

To specify tags for an agent in the com.instana.plugin.host, add a section in the following format:

com.instana.plugin.host:
  tags:
    - 'production'
    - 'app1'

Alternatively, the INSTANA_TAGS=production,app1 environment variable can be used (tags from environment variable and config file are additive).

Two tags are added to the specific agent, which enables searching and filtering of tags in the UI:

Extract the installed packages list

Note: These configurations are applied by using the agent configuration files.

The host agent can report to Instana which packages are installed on the underpinning Linux operating system.

Note: This configuration is disabled by default.

The following Linux distributions are currently supported:

• Debian and derivatives that use dpkg as package manager
• Red Hat and derivates that use rpm and yum as package managers

When the collectInstalledSoftware property is set, installed packages on an operating system are extracted once a day.

com.instana.plugin.host:
  collectInstalledSoftware: false #  Valid values: true, false

Custom zones

Note: These configurations are applied by using the agent configuration files.

Instana uses Amazon Web Services, Google Compute Engine, or OpenStack Nova availability zone information to group hosts by default. To customize the default host grouping in Instana, a new group for the specific host can be defined in the section com.instana.plugin.generic.hardware by using the following format:

com.instana.plugin.generic.hardware:
  enabled: true
  availability-zone: 'Demozone'

It is also possible to provide this configuration by using the INSTANA_ZONE environment variable, which then overrides the zone that is specified in the configuration file.

The result is that hosts are grouped into zones on the infrastructure map:

Monitor custom processes

Note: These configurations are applied by using the agent configuration files.

Instana automatically monitors process metrics of higher-level sensors like Java® or MySQL by default. If you want to monitor an OS process not automatically covered by Instana, you can configure it using both the process name or its arguments:

com.instana.plugin.process:
  processes:
    - 'sshd'
    - 'slapd'
  arguments:
    - '/opt/script.sh'

Secrets

Tracing data might contain sensitive data. Therefore, the Instana agent supports the specification of patterns for secrets, that is, data to be redacted agent-side from the tracing data. Data treated as secrets can't reach the Instana SaaS for processing and thus is not available for analysis in the UI or retrieval by using API.

Secrets are specified in the following way:

com.instana.secrets:
  # One of: 'equals-ignore-case', 'equals', 'contains-ignore-case', 'contains', 'regex'
  matcher: 'contains-ignore-case'
  list:
    - 'key'
    - 'password'
    - 'secret'

If a key matches an entry from the list, the value is redacted, and not sent to the Instana backend. If you do not specify any custom secrets configuration, Instana uses the afore-mentioned secrets configuration by default.

Note: Generally, secrets are filtered from static data. Filtering is highly optimized and changes to which secrets to scrub will not be applied while a process is running. To make a secret configuration effective, either restart the agent or the running monitored component, which make the new config in effect.

Instana supports secrets at infrastructure and platform level as follows:

• Process environment variables
• Docker container information
• Kubernetes annotations and container environment variables (requires an extra configuration, see Kubernetes secrets for more details)

Instana supports secrets in runtimes as follows:

• Database connection strings
• HTTP query parameters (virtually for all supported runtimes, see the following support matrix), such as https://my.domain/accounts/status?account=<secret_1>&user=<secret_2>.
• HTTP matrix path parameters, such as https://my.domain/accounts/account=<secret_1>;user=<secret_1>/status (support available for some runtimes, see the following support matrix).

Notes:

• Instana does not support treating as secrets arbitrary segments in a URL path, for example, in case the URL is structured as https://my.domain/accounts/<secret>/status. The overhead of running, for example, regular expressions on URL paths to discard segments would be prohibitive.

• Although it is not really a matter that is related to secrets, the Instana does not capture the following information that is related to interaction with databases:

  1. SQL parameters from stored procedures
  2. Connection credentials

Similarly, to the case of arbitrary path segments, literals from SQL queries are stripped, and you are recommended to use parametrized queries instead. The only exception lies with Instana's PHP support, which has an option to strip SQL literals (see sanitizeSql configuration option), at a potentially rather steep cost in terms of overhead of the Instana agent.

Capture custom HTTP headers

Note: These configurations are applied by using the agent configuration files.

By default, Instana doesn't collect HTTP headers when it traces HTTP calls, but this feature can be enabled on demand as follows:

com.instana.tracing:
  extra-http-headers:
    - 'x-request-id'
    - 'x-loadtest-id'
    - ...

The values are case-insensitive. The headers that are collected are shown in the call details (in the callee details section). You can also use the header names and their values to search for calls and traces (in the UI or by using the API). Finally, they can be used for service configuration.

Restrictions:

• All tracers capture request headers on HTTP entries (HTTP calls that the instrumented process receives).
• Some, but not all tracers capture response headers on HTTP entries. See the following details.
• Some, but not all tracers capture request and response headers on HTTP exits (HTTP calls where the instrumented process is the client). See the following details.
• If the same header is present as a request header and a response header, one of the two values might not be captured by the tracer.

Kafka Trace Correlation Headers

You can configure the format for Kafka trace correlation headers that are used by Instana tracers with the setting com.instana.tracing.kafka.header-format. Valid values are binary, string, or both. Here is an example configuration:

com.instana.tracing:
  kafka:
    header-format: string # possible values: binary, both, string

You can also disable Kafka trace correlation entirely with the setting com.instana.tracing.kafka.trace-correlation: false (not recommended). See the following example:

com.instana.tracing:
  kafka:
    trace-correlation: false

For more information, see Kafka Header Migration.

Ignore processes

Note: These configurations are applied by using the agent configuration files.

Note for Windows only: The process names in the list are case-sensitive.

To ignore any monitoring of certain processes, uncomment the com.instana.ignore section and list all process names of processes that should not be monitored. Sometimes scripts or other jobs are started through the same command, but only the execution of a single command should be ignored. In this case, use arguments to specify any argument that should cause a process to be ignored.

com.instana.ignore:
  processes:
    - 'java'
    - 'httpd'
  arguments:
    - '-batch-file=/tmp/batch.def'

Deactivate agent capabilities triggered by UI

To tail the Instana Agent logs, do diagnostic actions on the agent and to enable the Trace Code View, users can trigger those capabilities through the Instana UI. In case you want to deactivate all of these capabilities, you can set the config by adding to: <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg

backchannel.enabled = false

Note: Commands are not sent directly to the Instana Agent, and they are received as response to sending new metrics.

Upload code source files to Instana

The Instana agent can retrieve on-demand source code of the processes that it monitors and associate it with the collected tracing data.

To disable the on-demand upload of source files into Instana, you can set the following configuration in the <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg file:

source.download.enabled = false

Note: The Instana Agent needs to be restarted to apply this setting.

Error report events (only AIX operating system)

Note: This configuration is applied by using the agent configuration files and is disabled by default.

To specify the time interval to poll error events from AIX system, set 'aixEventsPollRate' in seconds (minimum is 900 seconds).

com.instana.plugin.host:
  aixEventsPollRate: 900 # (Optional) Only for AIX systems. Setting a value to 900 seconds or larger will enable error report event collection and removing it will stop it.

To disable the feature, remove 'aixEventsPollRate' from the agent configuration files.

Log rotation

By default, the agent uses a log rotation of 10 times 5MB agent log files.

log4j2.appender.rolling.policy.type = SizeBasedTriggeringPolicy
log4j2.appender.rolling.policy.size = 5MB
log4j2.appender.rolling.strategy.type = DefaultRolloverStrategy
log4j2.appender.rolling.strategy.max = 10

Syslog

The Agent now uses Log4j2, which is a modern and flexible logging facility. For example, syslog can be configured like this:

log4j2.rootLogger.appenderRef.Syslog.ref = Syslog
log4j2.rootLogger.appenderRef.Syslog.level = ERROR
log4j2.appender.syslog.type=Syslog
log4j2.appender.syslog.name=Syslog
log4j2.appender.syslog.layout.type=PatternLayout
log4j2.appender.syslog.layout.pattern = ${log4j2.pattern}
log4j2.appender.syslog.facility=SYSLOG
log4j2.appender.syslog.host=localhost
log4j2.appender.syslog.port=514
log4j2.appender.syslog.protocol=UDP

Logs to STDOUT

In custom container images, you might want to log to STDOUT. You can do so by providing the following configuration in <instana-agent-install-dir>etc/org.ops4j.pax.logging.cfg:

log4j2.rootLogger.appenderRef.Console.ref = Console

log4j2.appender.console.type = Console
log4j2.appender.console.name = Console
log4j2.appender.console.layout.type = PatternLayout
log4j2.appender.console.layout.pattern = ${log4j2.pattern}

Log4j2

Because the logging is using the standard log4j2 logging, it can be configured in many more ways as described in the log4j2 documentation.

Log metrics or traces to file

The agent can temporarily log metrics or traces that are sent through that agent to a file on disk. It's often used to debug various issues that are related to metrics, traces, tracing, or spans.

To enable the feature, locate the file <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.File.cfg and update its contents with:

# Configuration of local logging. Changes will be hot-reloaded.
# Activate logging of outgoing payloads to local disk by setting a non-empty
# prefix. The log file will be written to data/log, and the file will have the
# defined prefix followed by a timestamp.
# Note: There is no automatic rotation of those files.
prefix=locallog

# The file can be filtered to either "metrics" or "traces".
# If empty or absent, there will be no filtering.
type=traces

As noted, the changes are hot-reloaded and can be activated immediately. It should only be done temporarily because leaving the feature enabled has the potential to fill up all available disk space given enough time and traffic.

Allow the logging to continue for a minute or two while generating traffic to the component in question when you trace issues. Then, revert the changes or comment all lines in that file. Again, changes that are written to disk are hot-reloaded by the Instana agent.

If you are working with a support ticket, attach the resulting log file to the support ticket. The log file is located in <instana-agent-dir>/data/log/locallog_*.log.

Systemd

Create file called /etc/systemd/system/instana-agent.service.d/20-resource_limits.conf, and add the following content to it:

[Service]
CPUAccounting=true
CPUQuota=50%
MemoryAccounting=true
MemoryMax=512M

Run systemctl daemon-reload and restart the instana-agent service.

Docker

Run the instana-agent container with the following extra parameters:

Docker version 1.13 and newer:

--cpus=0.5 --memory=512m

Docker version 1.12 and older:

--cpu-period=100000 --cpu-quota=50000 --memory=512m

Kubernetes

Add the following configuration snippet to the agent's container configuration:

livenessProbe:
  httpGet: # Agent liveness is published on localhost:42699/status
    path: /status
    port: 42699
  initialDelaySeconds: 75
  periodSeconds: 5
resources:
  requests:
    memory: "256Mi"
    cpu: "0.5"
  limits:
    memory: "512Mi"
    cpu: "1.0"

The configuration also requests less memory and CPU for the instana-agent container.