Configuring host agents by using the agent configuration file
Most host agent configurations are applied by using the agent configuration file (
<instana-agent-dir>/etc/instana/configuration.yaml). You can configure a host agent by editing the agent configuration file.
Note: The format of the agent configuration file is YAML, which is sensitive to white space. All indention levels allow only two blank spaces.
- Creating multiple configuration files
- Integrating the host agent with secret managers
- Obtaining configurations from process environment and files
- Monitoring extra file systems
- Specifying host tags
- Extracting the installed packages list
- Setting custom zones
- Monitoring custom processes
- Configuring secrets
- Capturing custom HTTP headers
- Capturing Kafka trace correlation headers
- Ignoring processes
- Deactivating agent capabilities triggered by Instana UI
- Uploading code source files to Instana
- Configuring error report events (only AIX operating system)
Creating multiple configuration files
To provide modularity in configurations, you can create multiple
<instana-agent-dir>/etc/instana/configuration-<suffix>.yaml files. Replace
<suffix> with an alphanumeric combination.
<instana-agent-dir>/etc/instana/configuration-<suffix>.yaml files are merged with
<instana-agent-dir>/etc/instana/configuration.yaml file in the alphabetical order of the file system. So the
configuration-cde.yaml file comes after the
Nested structures that are specified in different configuration files like tags are merged, and values are overwritten.
Integrating the host agent with secret managers
The host agent can retrieve confidential data like credentials or other information that you do not want to put in clear text in the
configuration.yaml file from secret managers.
IBM Cloud Secrets Manager
The IBM Cloud Secrets Manager is based on open-source HashiCorp Vault and provides the same API and the same configuration as HashiCorp Vault. For more information about configuring the integration, see the HashiCorp Vault section.
Starting with version 1.0.11 of the Vault component, the IBM Cloud Secrets Manager SDK with IAM Keys are supported to be used in Vault.
com.instana.configuration.integration.vault: connection_url: <secrets-manager-address> # The address (URL) of the IBM Cloud Secrets Manager server instance(e.g. https://f022446e-1024-4aa9-a00c-72bf15aa9e7b.us-south.secrets-manager.appdomain.cloud) ibm_secrets_manager: <iam_key> # IAM Key that can be used to create access tokens secret_refresh_rate: 24 # This configuration option allows you to account for rotating credentials, refresh rate in hours, default 24
Endpoint descriptions are available in IBM Cloud Docs - Secrets Manager or within the IBM Secrets Manager dashboard. To create an IAM key, see IBM Cloud - IAM Keys.
To use the secrets inside IBM Cloud Secret Manager in Instana, see the configuration as follows:
com.instana.example: test: configuration_from: type: vault secret_key: path: <secret-id> # The id of the Secret within the IBM Secrets Manager, (e.g. cc32688d-89c0-6fa8-c0b4-6cc88c232e66) key: <kv-key-entry> # The Key inside the Secret Object of type KV (e.g. login) poll_rate: 300 # seconds
Note: The Vault integration requires the agent bootstrap version
1.2.9 or later.
Instana agent uses HashiCorp Vault to securely obtain values for sensitive settings in the agent configuration file.
You need to provide the Vault integration configuration parameters to the host agent by adding the following lines to the agent configuration file:
com.instana.configuration.integration.vault: connection_url: <vault_server_base_url> # The address (URL) of the Vault server instance(e.g. http://127.0.0.1:8200 or https://exapmle.com:8200) prefix: <optional_prefix> # Optional prefix path required if kv_version 2 is used and the /data/ must be injected further down token: <vault_access_token> # Vault access token with assigned, at least, `read` access policy to relevant Vault paths, optional if other auth providers are present github: # Optional auth method if Vault access token is not provided, has higher priority than approle if present github_token: <github_token> # Personal Access Token, must provide at least read:org scope, must be present if github is used as an auth provider auth_mount: github # Optional mount path for GitHub Auth, defaults to github approle: # Optional auth method if Vault access token or github auth is not provided role_id: <roleId> # AppRole RoleId, must be present if approle is used as an auth provider secret_id: <secretId> # AppRole SecretId, must be present if approle is used as an auth provider auth_mount: approle # Optional mount path for AppRole Auth, defaults to approle path_to_pem_file: <path_to_X.509_CA_certificate> # X.509 CA certificate (UTF8 encoded) in unencrypted PEM format, used by the Agent when communicating with Vault over HTTPS secret_refresh_rate: 24 # This configuration option allows you to account for rotating credentials, refresh rate in hours, default 24 kv_version: 2 # The Key/Value secrets engine version, default is 2
After the initial Vault configuring, you can specify the retrieval of secrets for various sensors as follows:
com.instana.plugin.mysql: user: 'instana' password: configuration_from: type: vault secret_key: path: <vault_path> key: <vault_secret_key>
For comparison, configure the password in plain-text in the agent configuration file as follows:
com.instana.plugin.mysql: user: 'instana' password: <my_password>
That is, you can swap out the
string value for the YAML structure that specifies the Vault coordinates, and the host agent automatically does the rest.
Also, you are not limited to using the Vault integration with a field that contains the password, but you can use it for every configuration that has a
string as value in the
For more information, see the following Vault concepts that are relevant for the Instana agent integration:
To get sensitive information from Kubernetes secrets into the agent's sensor configuration, combine the following two basic concepts:
- Configure your workloads to mount Kubernetes secrets into environment variables or files within the workload's pod.
- Adopt the agent's
configuration.yamlfile to read configuration values from process-specific environment variables or files.
By combining these two concepts, your workloads can carry sensitive information in pod-specific or process-specific environment variables or files. In addition, the agent can get these environment variables or files to read the sensitive information and apply the information to the workload-specific configuration.
Obtaining configurations from process environment and files
You might have cases in which you do not want to hardcode the credentials or other configuration items in your
configuration.yaml file. Or, you might have several instances of one technology, such as MySQL, on the same node, and
each instance needs different configurations.
In these cases, the host agent can read configuration values from the process that are to be monitored, such as the MySQL database. The configuration values are read by looking up the values from specific environment variables (supports only YAML literals, like a string, a Boolean, or a number), or as content of specific files (supports complex YAML structures, like lists and maps).
An intentional symmetry exists between these capabilities of the host agent and the way that you mount secrets in Kubernetes. However, these capabilities are available on all supported Operating Systems and all supported container runtimes.
Note: Numeric values in the environment variables and specific files are converted to integer values. The Boolean values
false are converted to Booleans.
Configurations from process environment
For configuration values that are taken from the target process environment, the
configuration.yaml file looks as follows:
com.instana.plugin.mysql: user: 'instana' password: configuration_from: type: env env_name: <environment variable name> default_value: <set this when no env var is found on this process>
If you do not specify a default value and the environment variable is not present on the process to be monitored, such as the MySQL database, then the
configuration.yaml file gives the impression that no value was entered.
On Kubernetes, this function is designed to work with mounting ConfigMap entries as environment variables in your containers. That is, you need to complete the following steps:
- Mount the ConfigMaps to the application containers as environment variables. - Allow the configurations in the host agent to use those environment variables.
Note: The values from environment variables can be only YAML literals, like a string, a Boolean, or a number. Complex YAML structures, such as lists and maps, are not supported. You cannot change the plug-in
enabled/disabled setting by using configurations from the process environment and specific files.
Configurations from process file system
For configuration values that are taken from a file, the
configuration.yaml file looks as follows:
com.instana.plugin.mysql: user: 'instana' password: configuration_from: type: file file_path: <absolute path to a file containing the configuration value> default_value: <set this when the file is not found> common_value: <set this in addition to the configuration value from file>
In the following example, the agent reads the
/jmx-config-map.yaml file from the process file system with the property
type: file. The
default_value property accepts complex YAML structures. The
common_value property is later merged with every process-specific configuration and is common to all monitored processes:
com.instana.plugin.java: jmx: configuration_from: type: file file_path: /jmx-config-map.yaml default_value: - object_name: 'java.lang:type=OperatingSystem' metrics: - attributes: 'CommittedVirtualMemorySize' type: 'absolute' common_value: - object_name: 'java.lang:type=OperatingSystem' metrics: - attributes: 'ProcessCpuLoad' type: 'delta'
In the following example, the
/jmx-config-map.yaml file gets mounted from a Kubernetes ConfigMap into the application container that provides the following content:
- object_name: 'java.lang:type=Compilation' metrics: - attributes: 'TotalCompilationTime' type: 'delta' - object_name: 'java.lang:type=ClassLoading' metrics: - attributes: 'LoadedClassCount' type: 'absolute'
This process results in the following final configuration. Note the
common_value addition for
com.instana.plugin.java: jmx: - object_name: 'java.lang:type=Compilation' metrics: - attributes: 'TotalCompilationTime' type: 'delta' - object_name: 'java.lang:type=ClassLoading' metrics: - attributes: 'LoadedClassCount' type: 'absolute' - object_name: 'java.lang:type=OperatingSystem' metrics: - attributes: 'ProcessCpuLoad' type: 'delta'
IMPORTANT: If the process that is to be monitored is running inside a container, then the absolute path of the file is resolved against that container's file system.
Referencing files on the host's operating system when you try to resolve a configuration value for a containerized process is not supported.
On Kubernetes, the function is designed to work with mounting ConfigMap entries as volumes in your containers. That is, you can define the configurations in ConfigMaps as follows:
Mount the ConfigMaps to pods as files.
Mount the ConfigMap volume into the application containers with a volume mount.
Allow the configurations in the host agent to read particular configurations from the file path that you used to mount the ConfigMap volume inside the application container.
YAML structures are supported for process-specific file-based configurations. The plug-in
enabled/disabledcannot be changed by using the configurations from the process environment and specific files.
Monitoring extra file systems
By default, the agent monitors local file systems only. To add extra file systems, add the name of the file system (that is the first column in mtab or df, the tools to get mounting information).
For example, to monitor the
server:/usr/local/pub /pub nfs rsize=8192,wsize=8192,timeo=14,intr file system, add the following line in the uncommented file systems list of the host section:
com.instana.plugin.host: filesystems: - 'server:/usr/local/pub'
Extracting the installed packages list
The host agent can report to Instana about the packages that are installed on the underpinning Linux operating system.
Note: This configuration is disabled by default.
The following Linux distributions are supported:
- Debian and its derivatives that use
dpkgas package manager.
- Red Hat OpenShift and its derivatives that use
yumas package managers.
collectInstalledSoftware property is set, installed packages on an operating system are extracted once daily.
com.instana.plugin.host: collectInstalledSoftware: false # Valid values: true, false
Setting custom zones
By default, Instana uses Amazon Web Services, Google Compute Engine, or OpenStack Nova availability zone information to group hosts. To customize the default host grouping in Instana, a new group for the specific host can be defined in the section
com.instana.plugin.generic.hardware by using the following format:
com.instana.plugin.generic.hardware: enabled: true availability-zone: 'Demozone'
You can also provide this configuration by using the
INSTANA_ZONE environment variable, which then overrides the zone that is specified in the configuration file. The result is that the hosts are grouped into zones on the infrastructure
Monitoring custom processes
By default, Instana automatically monitors the process metrics of higher-level sensors, like Java® or MySQL. If you want to monitor an OS process that is not automatically covered by Instana, then you can configure it by using both the process name or its arguments:
com.instana.plugin.process: processes: - 'sshd' - 'slapd' arguments: - '/opt/script.sh'
Tracing data might contain sensitive data. Therefore, the Instana agent supports the specification of patterns for secrets, that is, data to be redacted agent-side from the tracing data. Data treated as secrets can't reach the Instana SaaS for processing and thus is not available for analysis in the UI or retrieval by using API.
Secrets are specified in the following way:
com.instana.secrets: # One of: 'equals-ignore-case', 'equals', 'contains-ignore-case', 'contains', 'regex' matcher: 'contains-ignore-case' list: - 'key' - 'password' - 'secret'
If a key matches an entry from the list, the value is redacted, and not sent to the Instana backend. If you do not specify any custom secrets configuration, then Instana uses the aforementioned secrets configuration by default.
Note: Generally, secrets are filtered from static data. While a process is running, filtering is highly optimized, and the changes that are made to configure secret scrubbing are not applied. To make a secret configuration effective, either restart the host agent or the running monitored component, which makes the new configuration in effect.
Instana supports secrets at the infrastructure and platform levels through the following configuration elements:
- Process environment variables
- Docker container information
- Kubernetes annotations and container environment variables (Supporting secrets through this option requires an extra configuration. For more information, see Kubernetes secrets).
Instana supports secrets in the following runtime elements:
- Database connection strings
- HTTP query parameters (virtually for all supported runtimes, see the following support matrix), such as
- HTTP matrix path parameters, such as
Support is available for some runtimes, see the following support matrix:
|Language||Secrets in HTTP Query parameters||Secrets in HTTP Matrix parameters|
Instana does not support treating the arbitrary segments in a URL path as secrets. For example, if the URL is structured as
https://my.domain/accounts/<secret>/status. The overhead of running, for example, the regular expressions on URL paths to discard segments is prohibited.
The Instana does not capture the following information that is related to the interaction with the databases:
- SQL parameters from stored procedures
- Connection credentials
Similarly, for arbitrary path segments, the literals from SQL queries are stripped, and you can use parametrized queries instead. The only exception lies with Instana PHP support, which can strip SQL literals (see
sanitizeSql configuration option),
at a potentially rather steep cost in terms of overhead of the Instana agent.
Capturing custom HTTP headers
By default, Instana doesn't collect HTTP headers when it traces HTTP calls. But, this feature can be enabled on demand as follows:
com.instana.tracing: extra-http-headers: - 'x-request-id' - 'x-loadtest-id' - ...
The values are case-insensitive. The headers that are collected are shown in the call details (in the callee details section). You can also use the header names and their values to search for calls and traces (in the UI or by using the API). Finally, they can be used for service configuration.
- All tracers capture request headers on HTTP entries (HTTP calls that the instrumented process receives).
- Some, but not all tracers capture response headers on HTTP entries. See the following details.
- Some, but not all tracers capture request and response headers on HTTP exits (HTTP calls where the instrumented process is the client). See the following details.
- If the same header is present as a request header and a response header, one of the two values might not be captured by the tracer.
|Tracer||Request Headers on HTTP Entries||Response Headers on HTTP Entries||Request Headers on HTTP Exits||Response Headers on HTTP Exits|
Capturing Kafka trace correlation headers
You can configure the format for Kafka trace correlation headers that are used by Instana tracers with the setting
com.instana.tracing.kafka.header-format. Valid values are
See the following example:
com.instana.tracing: kafka: header-format: string # possible values: binary, both, string
Do not disable the Kafka trace correlation entirely. But, if you need to disable the Kafka trace correlation entirely, then set
com.instana.tracing.kafka.trace-correlation: false. See the following example:
com.instana.tracing: kafka: trace-correlation: false
For more information, see Kafka Header Migration.
Attention: For Windows, the process names in the list are case-sensitive.
To ignore monitoring of certain processes, uncomment the
com.instana.ignore section and list all the process names of the processes that do not require monitoring. Sometimes, scripts or other jobs are started through the same command,
but only the execution of a single command must be ignored. In this case, use arguments to specify any argument that causes a process to be ignored.
com.instana.ignore: processes: - 'java' - 'httpd' arguments: - '-batch-file=/tmp/batch.def'
Deactivating agent capabilities triggered by Instana UI
To get the recent data from the Instana agent logs, do diagnostic actions on the agent.
To enable the Trace Code View, you can trigger the agent capabilities through the Instana UI. If you want to deactivate all the agent capabilities, then you can set the configuration by adding the following command to the
<instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg configuration file:
backchannel.enabled = false
Note: Commands are not sent directly to the Instana agent, and they are received as response to sending new metrics.
Uploading code source files to Instana
The Instana agent can retrieve on-demand source code of the processes that it monitors and associate it with the collected tracing data.
To disable the on-demand upload of the source files into Instana, complete the following steps:
- Set the following configuration in the
source.download.enabled = false
- Restart the Instana agent to apply this setting.
Configuring error report events (only AIX operating system)
To specify the time interval to poll the error events from AIX system, set the
aixEventsPollRate event in seconds (minimum is 900 seconds).
com.instana.plugin.host: aixEventsPollRate: 900 # (Optional) Only for AIX systems. Setting a value to 900 seconds or larger will enable error report event collection and removing it will stop it.
To disable the feature, remove
aixEventsPollRate event from the agent configuration files.
Supported technologies include