Configuring host agents by using the agent configuration file

Most host agent configurations are applied by using the agent configuration file (<instana-agent-dir>/etc/instana/configuration.yaml). You can configure a host agent by editing the agent configuration file.

The format of the agent configuration file is YAML, which is sensitive to white space. All indention levels allow only two blank spaces.

Creating multiple configuration files

To provide modularity in configurations, you can create multiple <instana-agent-dir>/etc/instana/configuration-<suffix>.yaml files. Replace <suffix> with an alphanumeric combination.

All <instana-agent-dir>/etc/instana/configuration-<suffix>.yaml files are merged with <instana-agent-dir>/etc/instana/configuration.yaml file in the alphabetical order of the file system. So the configuration-cde.yaml file comes after the configuration-abc.yaml file.

Nested structures that are specified in different configuration files like tags are merged, and values are overwritten.

Integrating the host agent with secret managers

Instana doesn't support encrypt confidential data such as password directly in the agent configuration file. However, the host agent can retrieve confidential data like credentials or other information that you do not want to put in clear text in the configuration.yaml file from secret managers.

IBM Cloud Secrets Manager

The IBM Cloud Secrets Manager is based on open-source HashiCorp Vault and provides the same API and the same configuration as HashiCorp Vault. For more information about configuring the integration, see the HashiCorp Vault section.

Starting with version 1.0.11 of the Vault component, the IBM Cloud Secrets Manager SDK with IAM Keys are supported to be used in Vault.

com.instana.configuration.integration.vault:
  connection_url: <secrets-manager-address> # The address (URL) of the IBM Cloud Secrets Manager server instance(e.g. https://f022446e-1024-4aa9-a00c-72bf15aa9e7b.us-south.secrets-manager.appdomain.cloud)
  ibm_secrets_manager: <iam_key> # IAM Key that can be used to create access tokens
  secret_refresh_rate: 24 # This configuration option allows you to account for rotating credentials, refresh rate in hours, default 24

Endpoint descriptions are available in IBM Cloud Docs - Secrets Manager or within the IBM Secrets Manager dashboard. To create an IAM key, see IBM Cloud - IAM Keys.

To use the secrets inside IBM Cloud Secret Manager in Instana, see the configuration as follows:

com.instana.example:
  test:
    configuration_from:
      type: vault
      secret_key:
        path: <secret-id> # The id of the Secret within the IBM Secrets Manager, (e.g. cc32688d-89c0-6fa8-c0b4-6cc88c232e66)
        key: <kv-key-entry> # The Key inside the Secret Object of type KV (e.g. login)
      poll_rate: 300 # seconds

HashiCorp Vault

The Vault integration requires the agent bootstrap version 1.2.9 or later.

Instana agent uses HashiCorp Vault to securely obtain values for sensitive settings in the agent configuration file.

You need to provide the Vault integration configuration parameters to the host agent by adding the following lines to the agent configuration file:

com.instana.configuration.integration.vault:
  connection_url: <vault_server_base_url> # The address (URL) of the Vault server instance(e.g. http://127.0.0.1:8200 or https://exapmle.com:8200)
  prefix: <optional_prefix> # Optional prefix path required if kv_version 2 is used and the /data/ must be injected further down
  token: <vault_access_token> # Vault access token with assigned, at least, `read` access policy to relevant Vault paths, optional if other auth providers are present
  github: # Optional auth method if Vault access token is not provided, has higher priority than approle if present
    github_token: <github_token> # Personal Access Token, must provide at least read:org scope, must be present if github is used as an auth provider
    auth_mount: github # Optional mount path for GitHub Auth, defaults to github
  approle: # Optional auth method if Vault access token or github auth is not provided
    role_id: <roleId> # AppRole RoleId, must be present if approle is used as an auth provider
    secret_id: <secretId> # AppRole SecretId, must be present if approle is used as an auth provider
    auth_mount: approle # Optional mount path for AppRole Auth, defaults to approle
  path_to_pem_file: <path_to_X.509_CA_certificate> # X.509 CA certificate (UTF8 encoded) in unencrypted PEM format, used by the Agent when communicating with Vault over HTTPS
  secret_refresh_rate: 24 # This configuration option allows you to account for rotating credentials, refresh rate in hours, default 24
  kv_version: 2 # The Key/Value secrets engine version, default is 2

After the initial Vault configuring, you can specify the retrieval of secrets for various sensors as follows:

com.instana.plugin.mysql:
  user: 'instana'
  password:
    configuration_from:
      type: vault
      secret_key:
        path: <vault_path>
        key: <vault_secret_key>

For comparison, configure the password in plain-text in the agent configuration file as follows:

com.instana.plugin.mysql:
  user: 'instana'
  password: <my_password>

That is, you can swap out the string value for the YAML structure that specifies the Vault coordinates, and the host agent automatically does the rest.

Also, you are not limited to using the Vault integration with a field that contains the password, but you can use it for every configuration that has a string as value in the <instana-agent-dir>/etc/instana/configuration.yaml file.

For more information, see the following Vault concepts that are relevant for the Instana agent integration:

Sample Vault configuration for MySQL

See the following sample configuration in the <instana-agent-dir>/etc/instana/configuration.yaml file to integrate HashiCorp Vault with MySQL database:

Before you start configuring the Vault for MySQL, ensure that Vault and MySQL are installed and configured in your system.

com.instana.configuration.integration.vault:
  connection_url: http://127.0.0.1:8200
  prefix: secret
  token:  secret-vault-token

The prefix attribute is optional.

com.instana.plugin.mysql:
  user: 'myuser'
  password:
    configuration_from:
      type: vault
      secret_key:
        path: secret/my-secret
        key: password

The /data/ path is not mentioned in the MySQL plug-in. If you use kv_version 2, omit any /data/ within the path.

Run the following command to see the secrets:

vault kv get secret/my-secret

The secrets are displayed as follows:

==== Secret Path ====
secret/data/my-secret

======= Metadata =======
Key                Value
---                -----
created_time       2024-07-08T06:53:10.830770915Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    *****
username    myuser

Kubernetes secrets

To get sensitive information from Kubernetes secrets into the agent's sensor configuration, combine the following two basic concepts:

By combining these two concepts, your workloads can carry sensitive information in pod-specific or process-specific environment variables or files. In addition, the agent can get these environment variables or files to read the sensitive information and apply the information to the workload-specific configuration.

Obtaining configurations from process environment and files

You might have cases in which you do not want to hardcode the credentials or other configuration items in your configuration.yaml file. Or, you might have several instances of one technology, such as MySQL, on the same node, and each instance needs different configurations.

In these cases, the host agent can read configuration values from the process that are to be monitored, such as the MySQL database. The configuration values are read by looking up the values from specific environment variables (supports only YAML literals, like a string, a Boolean, or a number), or as content of specific files (supports complex YAML structures, like lists and maps).

An intentional symmetry exists between these capabilities of the host agent and the way that you mount secrets in Kubernetes. However, these capabilities are available on all supported Operating Systems and all supported container runtimes.

Numeric values in the environment variables and specific files are converted to integer values. The Boolean values true and false are converted to Booleans.

Configurations from process environment

For configuration values that are taken from the target process environment, the configuration.yaml file looks as follows:

com.instana.plugin.mysql:
  user: 'instana'
  password:
    configuration_from:
      type: env
      env_name: <environment variable name>
      default_value: <set this when no env var is found on this process>

If you do not specify a default value and the environment variable is not present on the process to be monitored, such as the MySQL database, then the configuration.yaml file gives the impression that no value was entered.

Notes:

On Kubernetes, this function is designed to work with mounting ConfigMap entries as environment variables in your containers. That is, you need to complete the following steps:

- Mount the ConfigMaps to the application containers as environment variables.
- Allow the configurations in the host agent to use those environment variables.

The values from environment variables can be only YAML literals, like a string, a Boolean, or a number. Complex YAML structures, such as lists and maps, are not supported. You cannot change the plug-in enabled/disabled setting by using configurations from the process environment and specific files.

Configurations from process file system

For configuration values that are taken from a file, the configuration.yaml file looks as follows:

com.instana.plugin.mysql:
  user: 'instana'
  password:
    configuration_from:
      type: file
      file_path: <absolute path to a file containing the configuration value>
      default_value: <set this when the file is not found>
      common_value: <set this in addition to the configuration value from file>

In the following example, the agent reads the /jmx-config-map.yaml file from the process file system with the property type: file. The default_value property accepts complex YAML structures. The common_value property is later merged with every process-specific configuration and is common to all monitored processes:

com.instana.plugin.java:
  jmx:
    configuration_from:
      type: file
      file_path: /jmx-config-map.yaml
      default_value:
        - object_name: 'java.lang:type=OperatingSystem'
          metrics:
          - attributes: 'CommittedVirtualMemorySize'
            type: 'absolute'
      common_value:
        - object_name: 'java.lang:type=OperatingSystem'
          metrics:
          - attributes: 'ProcessCpuLoad'
            type: 'delta'

In the following example, the /jmx-config-map.yaml file gets mounted from a Kubernetes ConfigMap into the application container that provides the following content:

- object_name: 'java.lang:type=Compilation'
  metrics:
  - attributes: 'TotalCompilationTime'
    type: 'delta'
- object_name: 'java.lang:type=ClassLoading'
  metrics:
  - attributes: 'LoadedClassCount'
    type: 'absolute'

This process results in the following final configuration. Note the common_value addition for ProcessCpuLoad:

com.instana.plugin.java:
  jmx:
  - object_name: 'java.lang:type=Compilation'
    metrics:
    - attributes: 'TotalCompilationTime'
      type: 'delta'
  - object_name: 'java.lang:type=ClassLoading'
    metrics:
    - attributes: 'LoadedClassCount'
      type: 'absolute'
  - object_name: 'java.lang:type=OperatingSystem'
    metrics:
    - attributes: 'ProcessCpuLoad'
      type: 'delta'

IMPORTANT: If the process that is to be monitored is running inside a container, then the absolute path of the file is resolved against that container's file system.

Notes:

  • Referencing files on the host's operating system when you try to resolve a configuration value for a containerized process is not supported.

  • On Kubernetes, the function is designed to work with mounting ConfigMap entries as volumes in your containers. That is, you can define the configurations in ConfigMaps as follows:

  • Mount the ConfigMaps to pods as files.

  • Mount the ConfigMap volume into the application containers with a volume mount.

  • Allow the configurations in the host agent to read particular configurations from the file path that you used to mount the ConfigMap volume inside the application container.

  • YAML structures are supported for process-specific file-based configurations. The plug-in enabled/disabled cannot be changed by using the configurations from the process environment and specific files.

Monitoring extra file systems

By default, the agent monitors local file systems only. To add extra file systems, add the name of the file system (that is the first column in mtab or df, the tools to get mounting information).

For example, to monitor the server:/usr/local/pub /pub nfs rsize=8192,wsize=8192,timeo=14,intr file system, add the following line in the uncommented file systems list of the host section:

com.instana.plugin.host:
  filesystems:
    - 'server:/usr/local/pub'

Specifying host tags

To specify tags for an agent in the com.instana.plugin.host configuration file, add a section in the following format:

com.instana.plugin.host:
  tags:
    - 'production'
    - 'app1'

Alternatively, you can use the INSTANA_TAGS=production,app1 environment variable to specify tags for an agent (tags from environment variable and config file are additive).

The production and app1 tags are added to the specific agent, which enables searching and filtering of tags in the UI:

Infrastructure tags

Extracting the installed packages list

The host agent can report to Instana about the packages that are installed on the underpinning Linux operating system.

This configuration is disabled by default.

The following Linux distributions are supported:

  • Debian and its derivatives that use dpkg as package manager.
  • Red Hat OpenShift and its derivatives that use rpm and yum as package managers.

When the collectInstalledSoftware property is set, installed packages on an operating system are extracted once daily.

com.instana.plugin.host:
  collectInstalledSoftware: false #  Valid values: true, false

Package list

Setting custom zones

By default, Instana uses Amazon Web Services, Google Compute Engine, or OpenStack Nova availability zone information to group hosts. To customize the default host grouping in Instana, a new group for the specific host can be defined in the section com.instana.plugin.generic.hardware by using the following format:

com.instana.plugin.generic.hardware:
  enabled: true
  availability-zone: 'Demozone'

You can also provide this configuration by using the INSTANA_ZONE environment variable, which then overrides the zone that is specified in the configuration file. The result is that the hosts are grouped into zones on the infrastructure map:

Infrastructure zones

Monitoring custom processes

By default, Instana automatically monitors the process metrics of higher-level sensors, like Java® or MySQL. If you want to monitor an OS process that is not automatically covered by Instana, then you can configure it by using both the process name or its arguments:

com.instana.plugin.process:
  processes:
    - 'sshd'
    - 'slapd'
  arguments:
    - '/opt/script.sh'

Configuring secrets

Tracing data might contain sensitive data. Therefore, the Instana agent supports the specification of patterns for secrets, that is, data to be redacted agent-side from the tracing data. Data treated as secrets can't reach the Instana SaaS for processing and thus is not available for analysis in the UI or retrieval by using API.

Secrets are specified in the following way:

com.instana.secrets:
  # One of: 'equals-ignore-case', 'equals', 'contains-ignore-case', 'contains', 'regex'
  matcher: 'contains-ignore-case'
  list:
    - 'key'
    - 'password'
    - 'secret'

If a key matches an entry from the list, the value is redacted, and not sent to the Instana backend. If you do not specify any custom secrets configuration, then Instana uses the aforementioned secrets configuration by default.

Generally, secrets are filtered from static data. While a process is running, filtering is highly optimized, and the changes that are made to configure secret scrubbing are not applied. To make a secret configuration effective, either restart the host agent or the running monitored component, which makes the new configuration in effect.

Instana supports secrets at the infrastructure and platform levels through the following configuration elements:

  • Process environment variables
  • Docker container information
  • Kubernetes annotations and container environment variables (Supporting secrets through this option requires an extra configuration. For more information, see Kubernetes secrets).

Instana supports secrets in the following runtime elements:

  • Database connection strings
  • HTTP query parameters (virtually for all supported runtimes, see the following support matrix), such as https://my.domain/accounts/status?account=<secret_1>&user=<secret_2>.
  • HTTP matrix path parameters, such as https://my.domain/accounts/account=<secret_1>;user=<secret_1>/status.

Support is available for some runtimes, see the following support matrix:

Language Secrets in HTTP Query parameters Secrets in HTTP Matrix parameters
Go
Java®
NGINX
Node.js
.NET Core
.NET Framework
PHP
Python
Ruby
HTTPd

Notes:

  • Instana does not support treating the arbitrary segments in a URL path as secrets. For example, if the URL is structured as https://my.domain/accounts/<secret>/status. The overhead of running, for example, the regular expressions on URL paths to discard segments is prohibited.

  • The Instana does not capture the following information that is related to the interaction with the databases:

    1. SQL parameters from stored procedures
    2. Connection credentials

Similarly, for arbitrary path segments, the literals from SQL queries are stripped, and you can use parametrized queries instead. The only exception lies with Instana PHP support, which can strip SQL literals (see sanitizeSql configuration option), at a potentially rather steep cost in terms of overhead of the Instana agent.

Capturing custom HTTP headers

By default, Instana doesn't collect HTTP headers when it traces HTTP calls. But, this feature can be enabled on demand as follows:

com.instana.tracing:
  extra-http-headers:
    - 'x-request-id'
    - 'x-loadtest-id'
    - ...

The values are case-insensitive. The headers that are collected are shown in the call details (in the callee details section). You can also use the header names and their values to search for calls and traces (in the UI or by using the API). Finally, they can be used for service configuration.

Restrictions:

  • All tracers capture request headers on HTTP entries (HTTP calls that the instrumented process receives).
  • Some, but not all tracers capture response headers on HTTP entries. See the following details.
  • Some, but not all tracers capture request and response headers on HTTP exits (HTTP calls where the instrumented process is the client). See the following details.
  • If the same header is present as a request header and a response header, one of the two values might not be captured by the tracer.
Tracer Request Headers on HTTP Entries Response Headers on HTTP Entries Request Headers on HTTP Exits Response Headers on HTTP Exits
Go
Java [1]
.NET
Node.js
PHP
NGINX N.A. N.A.
Python
Ruby
HTTPd N.A. N.A.

Configuring Kafka trace correlation headers

You can configure the format for Kafka trace correlation headers that are used by Instana tracers with the setting com.instana.tracing.kafka.header-format. Valid values are binary, string, or both. See the following example:

com.instana.tracing:
  kafka:
    header-format: string # possible values: binary, both, string

Do not disable the Kafka trace correlation entirely. But, if you need to disable the Kafka trace correlation entirely, then set com.instana.tracing.kafka.trace-correlation: false. See the following example:

com.instana.tracing:
  kafka:
    trace-correlation: false

For more information, see Kafka Header Migration.

Ignoring processes

Attention: For Windows, the process names in the list are case-sensitive.

Processes and arguments are not related or codependent.

You can define either processes or arguments. For example, if you define a process called nginx and an argument called /opt/server/server.js, both the NGINX process and the nodeJS process (that calls the file /opt/server/server.js) are ignored.

To ignore monitoring of certain processes, uncomment the com.instana.ignore section and list all the process names of the processes that do not require monitoring. Sometimes, scripts or other jobs are started through the same command, but only the execution of a single command must be ignored. In this case, use arguments to specify any argument that causes a process to be ignored.

When you specify arguments, make sure to retrieve arguments and environmental variables that are defined in Process > Arguments in the Instana UI.

Instana takes in arguments based on the white spaces between arguments. For example, if you run a command with /usr/bin/java -p 4654 -Djava.locale.providers=CLDR,JRE,SPI, the white space between -p and 4654 causes Instana to identify them as two different arguments.

com.instana.ignore:
  processes:
    - 'java'
    - 'httpd'
  arguments:
    - '-batch-file=/tmp/batch.def'

Deactivating agent capabilities triggered by Instana UI

To get the recent data from the Instana agent logs, do diagnostic actions on the agent.

To enable the Trace Code View, you can trigger the agent capabilities through the Instana UI. If you want to deactivate all the agent capabilities, then you can set the configuration by adding the following command to the <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg configuration file:

backchannel.enabled = false

Commands are not sent directly to the Instana agent, and they are received as response to sending new metrics.

Uploading code source files to Instana

The Instana agent can retrieve on-demand source code of the processes that it monitors and associate it with the collected tracing data.

To disable the on-demand upload of the source files into Instana, complete the following steps:

  1. Set the following configuration in the <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg configuration file:
source.download.enabled = false
  1. Restart the Instana agent to apply this setting.

Configuring error report events (only AIX operating system)

To specify the time interval to poll the error events from AIX system, set the aixEventsPollRate event in seconds (minimum is 900 seconds).

com.instana.plugin.host:
  aixEventsPollRate: 900 # (Optional) Only for AIX systems. Setting a value to 900 seconds or larger will enable error report event collection and removing it will stop it.

To disable the feature, remove aixEventsPollRate event from the agent configuration files.


  1. Supported technologies include Servlet, Spring, Tomcat, and http4s. ↩︎