Configuring host agents by using the agent configuration file

Creating multiple configuration files

To provide modularity in configurations, you can create multiple <instana-agent-dir>/etc/instana/configuration-<suffix>.yaml files. Replace <suffix> with an alphanumeric combination.

All <instana-agent-dir>/etc/instana/configuration-<suffix>.yaml files are merged with <instana-agent-dir>/etc/instana/configuration.yaml file in the alphabetical order of the file system. So the configuration-cde.yaml file comes after the configuration-abc.yaml file.

Nested structures that are specified in different configuration files like tags are merged, and values are overwritten.

Integrating the host agent with secret managers

Instana doesn't support encrypt confidential data such as password directly in the agent configuration file. However, the host agent can retrieve confidential data like credentials or other information that you do not want to put in clear text in the configuration.yaml file from secret managers.

com.instana.configuration.integration.vault:
  connection_url: <secrets-manager-address> # The address (URL) of the IBM Cloud Secrets Manager server instance(e.g. https://f022446e-1024-4aa9-a00c-72bf15aa9e7b.us-south.secrets-manager.appdomain.cloud)
  ibm_secrets_manager: <iam_key> # IAM Key that can be used to create access tokens
  secret_refresh_rate: 24 # This configuration option allows you to account for rotating credentials, refresh rate in hours, default 24

com.instana.example:
  test:
    configuration_from:
      type: vault
      secret_key:
        path: <secret-id> # The id of the Secret within the IBM Secrets Manager, (e.g. cc32688d-89c0-6fa8-c0b4-6cc88c232e66)
        key: <kv-key-entry> # The Key inside the Secret Object of type KV (e.g. login)
      poll_rate: 300 # seconds

com.instana.configuration.integration.vault:
  connection_url: <vault_server_base_url> # The address (URL) of the Vault server instance(e.g. http://127.0.0.1:8200 or https://exapmle.com:8200)
  prefix: <optional_prefix> # Optional prefix path required if kv_version 2 is used and the /data/ must be injected further down
  token: <vault_access_token> # Vault access token with assigned, at least, `read` access policy to relevant Vault paths, optional if other auth providers are present
  github: # Optional auth method if Vault access token is not provided, has higher priority than approle if present
    github_token: <github_token> # Personal Access Token, must provide at least read:org scope, must be present if github is used as an auth provider
    auth_mount: github # Optional mount path for GitHub Auth, defaults to github
  approle: # Optional auth method if Vault access token or github auth is not provided
    role_id: <roleId> # AppRole RoleId, must be present if approle is used as an auth provider
    secret_id: <secretId> # AppRole SecretId, must be present if approle is used as an auth provider
    auth_mount: approle # Optional mount path for AppRole Auth, defaults to approle
  path_to_pem_file: <path_to_X.509_CA_certificate> # X.509 CA certificate (UTF8 encoded) in unencrypted PEM format, used by the Agent when communicating with Vault over HTTPS
  secret_refresh_rate: 24 # This configuration option allows you to account for rotating credentials, refresh rate in hours, default 24
  kv_version: 2 # The Key/Value secrets engine version, default is 2

com.instana.plugin.mysql:
  user: 'instana'
  password:
    configuration_from:
      type: vault
      secret_key:
        path: <vault_path>
        key: <vault_secret_key>

com.instana.plugin.mysql:
  user: 'instana'
  password: <my_password>

Obtaining configurations from process environment and files

You might have cases in which you do not want to hardcode the credentials or other configuration items in your configuration.yaml file. Or, you might have several instances of one technology, such as MySQL, on the same node, and each instance needs different configurations.

In these cases, the host agent can read configuration values from the process that are to be monitored, such as the MySQL database. The configuration values are read by looking up the values from specific environment variables (supports only YAML literals, like a string, a Boolean, or a number), or as content of specific files (supports complex YAML structures, like lists and maps).

An intentional symmetry exists between these capabilities of the host agent and the way that you mount secrets in Kubernetes. However, these capabilities are available on all supported Operating Systems and all supported container runtimes.

Numeric values in the environment variables and specific files are converted to integer values. The Boolean values true and false are converted to Booleans.

com.instana.plugin.mysql:
  user: 'instana'
  password:
    configuration_from:
      type: env
      env_name: <environment variable name>
      default_value: <set this when no env var is found on this process>

- Mount the ConfigMaps to the application containers as environment variables.
- Allow the configurations in the host agent to use those environment variables.

com.instana.plugin.mysql:
  user: 'instana'
  password:
    configuration_from:
      type: file
      file_path: <absolute path to a file containing the configuration value>
      default_value: <set this when the file is not found>
      common_value: <set this in addition to the configuration value from file>

com.instana.plugin.java:
  jmx:
    configuration_from:
      type: file
      file_path: /jmx-config-map.yaml
      default_value:
        - object_name: 'java.lang:type=OperatingSystem'
          metrics:
          - attributes: 'CommittedVirtualMemorySize'
            type: 'absolute'
      common_value:
        - object_name: 'java.lang:type=OperatingSystem'
          metrics:
          - attributes: 'ProcessCpuLoad'
            type: 'delta'

- object_name: 'java.lang:type=Compilation'
  metrics:
  - attributes: 'TotalCompilationTime'
    type: 'delta'
- object_name: 'java.lang:type=ClassLoading'
  metrics:
  - attributes: 'LoadedClassCount'
    type: 'absolute'

com.instana.plugin.java:
  jmx:
  - object_name: 'java.lang:type=Compilation'
    metrics:
    - attributes: 'TotalCompilationTime'
      type: 'delta'
  - object_name: 'java.lang:type=ClassLoading'
    metrics:
    - attributes: 'LoadedClassCount'
      type: 'absolute'
  - object_name: 'java.lang:type=OperatingSystem'
    metrics:
    - attributes: 'ProcessCpuLoad'
      type: 'delta'

Monitoring extra file systems

By default, the agent monitors local file systems only. To add extra file systems, add the name of the file system (that is the first column in mtab or df, the tools to get mounting information).

For example, to monitor the server:/usr/local/pub /pub nfs rsize=8192,wsize=8192,timeo=14,intr file system, add the following line in the uncommented file systems list of the host section:

com.instana.plugin.host:
  filesystems:
    - 'server:/usr/local/pub'

Specifying host tags

To specify tags for an agent in the com.instana.plugin.host configuration file, add a section in the following format:

com.instana.plugin.host:
  tags:
    - 'production'
    - 'app1'

Alternatively, you can use the INSTANA_TAGS=production,app1 environment variable to specify tags for an agent (tags from environment variable and config file are additive).

The production and app1 tags are added to the specific agent, which enables searching and filtering of tags in the UI:

Extracting the installed packages list

The host agent can report to Instana about the packages that are installed on the underpinning Linux operating system.

This configuration is disabled by default.

The following Linux distributions are supported:

• Debian and its derivatives that use dpkg as package manager.
• Red Hat OpenShift and its derivatives that use rpm and yum as package managers.

When the collectInstalledSoftware property is set, installed packages on an operating system are extracted once daily.

com.instana.plugin.host:
  collectInstalledSoftware: false #  Valid values: true, false

Setting custom zones

By default, Instana uses Amazon Web Services, Google Compute Engine, or OpenStack Nova availability zone information to group hosts. To customize the default host grouping in Instana, a new group for the specific host can be defined in the section com.instana.plugin.generic.hardware by using the following format:

com.instana.plugin.generic.hardware:
  enabled: true
  availability-zone: 'Demozone'

You can also provide this configuration by using the INSTANA_ZONE environment variable, which then overrides the zone that is specified in the configuration file. The result is that the hosts are grouped into zones on the infrastructure map:

Monitoring custom processes

By default, Instana automatically monitors the process metrics of higher-level sensors, like Java® or MySQL. If you want to monitor an OS process that is not automatically covered by Instana, then you can configure it by using both the process name or its arguments:

com.instana.plugin.process:
  processes:
    - 'sshd'
    - 'slapd'
  arguments:
    - '/opt/script.sh'

Configuring secrets

Tracing data might contain sensitive data. Therefore, the Instana agent supports the specification of patterns for secrets, that is, data to be redacted agent-side from the tracing data. Data treated as secrets can't reach the Instana SaaS for processing and thus is not available for analysis in the UI or retrieval by using API.

Secrets are specified in the following way:

com.instana.secrets:
  # One of: 'equals-ignore-case', 'equals', 'contains-ignore-case', 'contains', 'regex'
  matcher: 'contains-ignore-case'
  list:
    - 'key'
    - 'password'
    - 'secret'

If a key matches an entry from the list, the value is redacted, and not sent to the Instana backend. If you do not specify any custom secrets configuration, then Instana uses the aforementioned secrets configuration by default.

Generally, secrets are filtered from static data. While a process is running, filtering is highly optimized, and the changes that are made to configure secret scrubbing are not applied. To make a secret configuration effective, either restart the host agent or the running monitored component, which makes the new configuration in effect.

Instana supports secrets at the infrastructure and platform levels through the following configuration elements:

• Process environment variables
• Docker container information
• Kubernetes annotations and container environment variables (Supporting secrets through this option requires an extra configuration. For more information, see Kubernetes secrets).

Instana supports secrets in the following runtime elements:

• Database connection strings
• HTTP query parameters (virtually for all supported runtimes, see the following support matrix), such as https://my.domain/accounts/status?account=<secret_1>&user=<secret_2>.
• HTTP matrix path parameters, such as https://my.domain/accounts/account=<secret_1>;user=<secret_1>/status.

Support is available for some runtimes, see the following support matrix:

Notes:

• Instana does not support treating the arbitrary segments in a URL path as secrets. For example, if the URL is structured as https://my.domain/accounts/<secret>/status. The overhead of running, for example, the regular expressions on URL paths to discard segments is prohibited.

• The Instana does not capture the following information that is related to the interaction with the databases:

  1. SQL parameters from stored procedures
  2. Connection credentials

Similarly, for arbitrary path segments, the literals from SQL queries are stripped, and you can use parametrized queries instead. The only exception lies with Instana PHP support, which can strip SQL literals (see sanitizeSql configuration option), at a potentially rather steep cost in terms of overhead of the Instana agent.

Capturing custom HTTP headers

By default, Instana doesn't collect HTTP headers when it traces HTTP calls. But, this feature can be enabled on demand as follows:

com.instana.tracing:
  extra-http-headers:
    - 'x-request-id'
    - 'x-loadtest-id'
    - ...

The values are case-insensitive. The headers that are collected are shown in the call details (in the callee details section). You can also use the header names and their values to search for calls and traces (in the UI or by using the API). Finally, they can be used for service configuration.

Restrictions:

• All tracers capture request headers on HTTP entries (HTTP calls that the instrumented process receives).
• Some, but not all tracers capture response headers on HTTP entries. See the following details.
• Some, but not all tracers capture request and response headers on HTTP exits (HTTP calls where the instrumented process is the client). See the following details.
• If the same header is present as a request header and a response header, one of the two values might not be captured by the tracer.

Configuring Kafka trace correlation headers

You can configure the format for Kafka trace correlation headers that are used by Instana tracers with the setting com.instana.tracing.kafka.header-format. Valid values are binary, string, or both. See the following example:

com.instana.tracing:
  kafka:
    header-format: string # possible values: binary, both, string

Do not disable the Kafka trace correlation entirely. But, if you need to disable the Kafka trace correlation entirely, then set com.instana.tracing.kafka.trace-correlation: false. See the following example:

com.instana.tracing:
  kafka:
    trace-correlation: false

For more information, see Kafka Header Migration.

Ignoring processes

Attention: For Windows, the process names in the list are case-sensitive.

Processes and arguments are not related or codependent.

You can define either processes or arguments. For example, if you define a process called nginx and an argument called /opt/server/server.js, both the NGINX process and the nodeJS process (that calls the file /opt/server/server.js) are ignored.

To ignore monitoring of certain processes, uncomment the com.instana.ignore section and list all the process names of the processes that do not require monitoring. Sometimes, scripts or other jobs are started through the same command, but only the execution of a single command must be ignored. In this case, use arguments to specify any argument that causes a process to be ignored.

When you specify arguments, make sure to retrieve arguments and environmental variables that are defined in Process > Arguments in the Instana UI.

Instana takes in arguments based on the white spaces between arguments. For example, if you run a command with /usr/bin/java -p 4654 -Djava.locale.providers=CLDR,JRE,SPI, the white space between -p and 4654 causes Instana to identify them as two different arguments.

com.instana.ignore:
  processes:
    - 'java'
    - 'httpd'
  arguments:
    - '-batch-file=/tmp/batch.def'

Deactivating agent capabilities triggered by Instana UI

To get the recent data from the Instana agent logs, do diagnostic actions on the agent.

To enable the Trace Code View, you can trigger the agent capabilities through the Instana UI. If you want to deactivate all the agent capabilities, then you can set the configuration by adding the following command to the <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg configuration file:

backchannel.enabled = false

Commands are not sent directly to the Instana agent, and they are received as response to sending new metrics.

Uploading code source files to Instana

The Instana agent can retrieve on-demand source code of the processes that it monitors and associate it with the collected tracing data.

To disable the on-demand upload of the source files into Instana, complete the following steps:

1. Set the following configuration in the <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg configuration file:

source.download.enabled = false

2. Restart the Instana agent to apply this setting.

Configuring error report events (only AIX operating system)

To specify the time interval to poll the error events from AIX system, set the aixEventsPollRate event in seconds (minimum is 900 seconds).

com.instana.plugin.host:
  aixEventsPollRate: 900 # (Optional) Only for AIX systems. Setting a value to 900 seconds or larger will enable error report event collection and removing it will stop it.

To disable the feature, remove aixEventsPollRate event from the agent configuration files.