Configuring host agents using agent configuration file

Creating multiple configuration files

To provide modularity in configurations, you can create multiple <instana-agent-dir>/etc/instana/configuration-<suffix>.yaml files. Replace <suffix> with an alphanumeric combination.

All <instana-agent-dir>/etc/instana/configuration-<suffix>.yaml files are merged with <instana-agent-dir>/etc/instana/configuration.yaml file in the alphabetical order of the file system. For example, the configuration-cde.yaml file comes after the configuration-abc.yaml file. Nested structures that are specified in different configuration files such as tags are merged, and values are overwritten.

See the following sections to learn how to use the agent configuration file to configure the agent:

Integrating the host agent with secret managers

Instana doesn't support encrypting confidential data such as password directly in the agent configuration file.

Therefore, when you add sensitive information (such as credentials or other information) in the agent configuration file, the contents might look like the following example:

com.instana.example:
  user: 'instana'
  password: <password>

However, you might not want to put the sensitive information in plain text in the agent configuration file. In such circumstances, you can use secret managers such as IBM Cloud Secrets Manager or HashiCorp Vault to store the secrets and retrieve them into the agent configuration file as required.

com.instana.configuration.integration.vault:
  connection_url: <vault_server_base_url> # The address (URL) of the Vault server instance(e.g. http://127.0.0.1:8200 or https://exapmle.com:8200)
  prefix: <optional_prefix> # Optional prefix path required if kv_version 2 is used and the /data/ must be injected further down
  token: <vault_access_token> # Vault access token with assigned, at least, `read` access policy to relevant Vault paths, optional if other auth providers are present
  github: # Optional auth method if Vault access token is not provided, has higher priority than approle if present
    github_token: <github_token> # Personal Access Token, must provide at least read:org scope, must be present if github is used as an auth provider
    auth_mount: github # Optional mount path for GitHub Auth, defaults to github
  approle: # Optional auth method if Vault access token or github auth is not provided
    role_id: <roleId> # AppRole RoleId, must be present if approle is used as an auth provider
    secret_id: <secretId> # AppRole SecretId, must be present if approle is used as an auth provider
    auth_mount: approle # Optional mount path for AppRole Auth, defaults to approle
  path_to_pem_file: <path_to_X.509_CA_certificate> # X.509 CA certificate (UTF8 encoded) in unencrypted PEM format, used by the Agent when communicating with Vault over HTTPS
  secret_refresh_rate: 24 # This configuration option allows you to account for rotating credentials, refresh rate in hours, default 24
  kv_version: 2 # The Key/Value secrets engine version, default is 2

com.instana.example:
  user: 'instana'
  password:
    configuration_from:
      type: vault
      secret_key:
        path: <vault_path>
        key: <vault_secret_key>

com.instana.configuration.integration.vault:
  connection_url: <secrets-manager-address> # The address (URL) of the IBM Cloud Secrets Manager server instance(e.g. https://f022446e-1024-4aa9-a00c-72bf15aa9e7b.us-south.secrets-manager.appdomain.cloud)
  ibm_secrets_manager: <iam_key> # IAM Key that can be used to create access tokens
  secret_refresh_rate: 24 # This configuration option allows you to account for rotating credentials, refresh rate in hours, default 24

com.instana.example:
  user: 'instana'
  password:
    configuration_from:
      type: vault
      secret_key:
        path: <secret-id> # The id of the Secret within the IBM Secrets Manager, (e.g. cc32688d-89c0-6fa8-c0b4-6cc88c232e66)
        key: <kv-key-entry> # The Key inside the Secret Object of type KV (e.g. login)
      poll_rate: 300 # seconds

com.instana.configuration.integration.vault:
  connection_url: http://127.0.0.1:8200
  prefix: secret
  token:  secret-vault-token

com.instana.plugin.mysql:
  user: 'myuser'
  password:
    configuration_from:
      type: vault
      secret_key:
        path: secret/my-secret
        key: password

vault kv get secret/my-secret

==== Secret Path ====
secret/data/my-secret

======= Metadata =======
Key                Value
---                -----
created_time       2024-07-08T06:53:10.830770915Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    *****
username    myuser

Obtaining configurations from process environment and files

You can configure the agent to read configuration values from the process that are to be monitored. This approach is useful in the following situations:

• When you do not want to hardcode any information in your configuration.yaml.
• When you have several instances of one technology, such as MySQL, on the same node, and each instance needs different configurations.

The agent reads the configuration values by looking up the values from specific environment variables (supports only YAML literals, such as a string, a Boolean, or a number), or as content of specific files (supports complex YAML structures, like lists and maps).

An intentional symmetry exists between these capabilities of the host agent and the way that you mount secrets in Kubernetes. However, these capabilities are available on all supported Operating Systems and all supported container runtimes.

Numeric values in the environment variables and specific files are converted to integer values. The Boolean values true and false are converted to Booleans.

com.instana.plugin.mysql:
  user: 'instana'
  password:
    configuration_from:
      type: env
      env_name: <environment variable name>
      default_value: <set this when no env var is found on this process>

com.instana.plugin.mysql:
  user: 'instana'
  password:
    configuration_from:
      type: file
      file_path: <absolute path to a file containing the configuration value>
      default_value: <set this when the file is not found>
      common_value: <set this in addition to the configuration value from file>

com.instana.plugin.java:
  jmx:
    configuration_from:
      type: file
      file_path: /jmx-config-map.yaml
      default_value:
        - object_name: 'java.lang:type=OperatingSystem'
          metrics:
          - attributes: 'CommittedVirtualMemorySize'
            type: 'absolute'
      common_value:
        - object_name: 'java.lang:type=OperatingSystem'
          metrics:
          - attributes: 'ProcessCpuLoad'
            type: 'delta'

- object_name: 'java.lang:type=Compilation'
  metrics:
  - attributes: 'TotalCompilationTime'
    type: 'delta'
- object_name: 'java.lang:type=ClassLoading'
  metrics:
  - attributes: 'LoadedClassCount'
    type: 'absolute'

com.instana.plugin.java:
  jmx:
  - object_name: 'java.lang:type=Compilation'
    metrics:
    - attributes: 'TotalCompilationTime'
      type: 'delta'
  - object_name: 'java.lang:type=ClassLoading'
    metrics:
    - attributes: 'LoadedClassCount'
      type: 'absolute'
  - object_name: 'java.lang:type=OperatingSystem'
    metrics:
    - attributes: 'ProcessCpuLoad'
      type: 'delta'

Monitoring extra file systems

By default, the agent monitors local file systems only. To add extra file systems, add the name of the file system (the first column in mtab or df).

For example, to monitor the server:/usr/local/pub /pub nfs rsize=8192,wsize=8192,timeo=14,intr file system, add the following line in the uncommented file systems list of the host section in the agent configuration file:

com.instana.plugin.host:
  filesystems:
    - 'server:/usr/local/pub'

Specifying host tags

To specify tags for an agent in the com.instana.plugin.host configuration file, add a section in the following format:

com.instana.plugin.host:
  tags:
    - 'production'
    - 'app1'

Alternatively, you can use the INSTANA_TAGS=production,app1 environment variable to specify tags for an agent (tags from environment variable and config file are additive).

The production and app1 tags are added to the specific agent, which enables searching and filtering of tags in the UI:

Extracting the installed packages list

The host agent can report to Instana about the packages that are installed on the underpinning Linux operating system.

This configuration is disabled by default.

The following Linux distributions are supported:

• Debian and its derivatives that use dpkg as package manager.
• Red Hat OpenShift and its derivatives that use rpm and yum as package managers.

To enable this feature, set the collectInstalledSoftware property to true.

com.instana.plugin.host:
  collectInstalledSoftware: false #  Valid values: true, false

After you enable the feature, the installed packages on an operating system are extracted once daily.

Setting custom zones

By default, Instana uses Amazon Web Services, Google Compute Engine, or OpenStack Nova availability zone information to group hosts. To customize the default host grouping in Instana, you can define a new group for the specific host in the section com.instana.plugin.generic.hardware by using the following format:

com.instana.plugin.generic.hardware:
  enabled: true
  availability-zone: 'Demozone'

You can also provide this configuration by using the INSTANA_ZONE environment variable, which then overrides the zone that is specified in the configuration file. After the configuration, the hosts are grouped into zones on the infrastructure map.

Monitoring custom processes

By default, Instana automatically monitors the process metrics of higher-level sensors, such as Java® or MySQL. To monitor an OS process that is not automatically covered by Instana, configure it by using both the process name and its arguments:

com.instana.plugin.process:
  processes:
    - 'sshd'
    - 'slapd'
  arguments:
    - '/opt/script.sh'

Configuring secrets

Tracing data might contain sensitive data. Therefore, the Instana agent supports the specification of patterns for secrets (data to be redacted agent-side from the tracing data). Data that is treated as secrets can't reach the Instana SaaS for processing, and therefore they are not available for analysis in the UI or retrieval by using API.

Specify the secrets as follows:

com.instana.secrets:
  # One of: 'equals-ignore-case', 'equals', 'contains-ignore-case', 'contains', 'regex'
  matcher: 'contains-ignore-case'
  list:
    - 'key'
    - 'password'
    - 'secret'

If you do not specify any custom secrets configuration, then Instana uses this secrets configuration by default. If a key matches an entry from the list, the value is redacted, and not sent to the Instana backend.

Generally, secrets are filtered from static data. When a process is running, filtering is highly optimized, and the changes that are made to configure secret scrubbing are not applied. To make a secret configuration effective immediately, either restart the host agent or the running monitored component.

Instana supports secrets at the infrastructure and platform levels through the following configuration elements:

• Process environment variables
• Docker container information
• Kubernetes annotations and container environment variables (Supporting secrets through this option requires an extra configuration. For more information, see Kubernetes secrets).

Instana supports secrets in the following runtime elements:

• Database connection strings
• HTTP query parameters (virtually for all supported runtimes, see the following support matrix), such as https://my.domain/accounts/status?account=<secret_1>&user=<secret_2>.
• HTTP matrix path parameters, such as https://my.domain/accounts/account=<secret_1>;user=<secret_1>/status.

The following runtimes are supported:

Capturing custom HTTP headers

By default, Instana doesn't collect HTTP headers when it traces HTTP calls.

If required, you can enable this feature by applying the following configuration in the agent configuration file:

com.instana.tracing:
  extra-http-headers:
    - 'x-request-id'
    - 'x-loadtest-id'
    - ...

The values are case-insensitive. The headers that are collected are shown in the call details (in the Call details section). You can also use the header names and their values to search for calls and traces (in the UI or by using the API) and for service configuration.

This feature currently has the following restrictions:

• All tracers capture request headers on HTTP entries (HTTP calls that the instrumented process receives).
• Some tracers might capture response headers on HTTP entries. See Table 2.
• Some tracers might capture request and response headers on HTTP exits (HTTP calls where the instrumented process is the client). See Table 2.
• If the same header is present as a request header and a response header, one of the two values might not be captured by the tracer.

Configuring Kafka trace correlation headers

You can configure the format for Kafka trace correlation headers that are used by Instana tracers by using the setting com.instana.tracing.kafka.header-format. Valid values are binary, string, or both. See the following example:

com.instana.tracing:
  kafka:
    header-format: string # possible values: binary, both, string

You must not disable the Kafka trace correlation entirely. However, if you need to disable the Kafka trace correlation entirely, then set com.instana.tracing.kafka.trace-correlation: false. See the following example:

com.instana.tracing:
  kafka:
    trace-correlation: false

For more information, see Kafka Header Migration.

Ignoring endpoints

You can exclude specific endpoints from tracing. For example, if you are using the redis package and want to avoid the tracing of commands, such as GET, TYPE, or others, you can use the ignore-endpoints configuration option.

For more information about the specific packages and tracers that support this feature, see Support information.

Filtering rules

The rules for filtering traces are as follows:

• When a trace is ignored, all subsequent downstream traces are also ignored.
• Use * to ignore all endpoints or methods.
• Endpoint values (such as Kafka topic names) remain consistent across services.
• Method names might vary depending on the programming language and technology. To determine the correct method and endpoint for your service, refer to the Instana UI.

The following Instana UI screenshot provides a visual reference for identifying the correct method and endpoint for configuration:

com.instana.tracing:
  ignore-endpoints:

    # Filtering by Method Name  
    redis:  
      - 'get'   
      - 'type'  
    dynamodb:
      - 'query'
    kafka:
      - 'send'

    # Filtering by Method Name and Endpoint for Kafka  
    kafka: 
      - methods: ["consume"]
        endpoints: ["topic1", "topic2"]  # Exclude consume calls for topic1 and topic2

      - methods: ["consume", "send"]
        endpoints: ["topic3"]  # Exclude both consume and send calls for topic3

      - methods: ["*"]  
        endpoints: ["topic4"]  # Exclude all methods for topic4

      - methods: ["consume"]
        endpoints: ["*"]  # Exclude consume method for all topics

Ignoring processes

You can exclude specific processes from tracing.

To ignore monitoring of certain processes, uncomment the com.instana.ignore section in the agent configuration file and list all the process names of the processes that do not require monitoring.

You can define either processes or arguments. For example, if you define a process called nginx and an argument called /opt/server/server.js, both the NGINX process and the nodeJS process (that calls the file /opt/server/server.js) are ignored. See the following example:

com.instana.ignore:
  processes:
    - 'java'
    - 'httpd'
  arguments:
    - '-batch-file=/tmp/batch.def'

For Windows, the process names in the list are case-sensitive.

Processes and arguments are not related or codependent.

Sometimes, scripts or other jobs are started through the same command, but only the execution of a single command must be ignored. In this scenario, use arguments to specify any argument that causes a process to be ignored.

When you specify arguments, make sure to retrieve arguments and environmental variables that are defined in Process > Arguments in the Instana UI.

Instana takes in arguments based on the blank spaces between arguments. For example, if you run a command with /usr/bin/java -p 4654 -Djava.locale.providers=CLDR,JRE,SPI, the space between -p and 4654 causes Instana to identify them as two different arguments.

Deactivating agent capabilities that are enabled in Instana UI

If you want to deactivate all the agent capabilities that are enabled in the Instana UI, add the following line to the <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg configuration file:

backchannel.enabled = false

Uploading code source files to Instana

The Instana agent can retrieve on-demand source code of the processes that it monitors and associate it with the collected tracing data.

To disable the on-demand upload of the source files into Instana, complete the following steps:

1. Set the following configuration in the <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg configuration file:

source.download.enabled = false

2. Restart the Instana agent to apply this setting.

Configuring error report events (only AIX operating system)

To specify the time interval to poll the error events from AIX system, set the aixEventsPollRate event in seconds (minimum is 900 seconds).

com.instana.plugin.host:
  aixEventsPollRate: 900 # (Optional) Only for AIX systems. Setting a value to 900 seconds or larger will enable error report event collection and removing it will stop it.

To disable the feature, remove aixEventsPollRate event from the agent configuration files.