Configuring host agent
After you install the host agent, you can configure the host agent as you need.
- Configuring Instana backend
- Setting up an agent proxy
- Setting up TLS encryption for agent endpoint
- Configuring host agent mode
- Configuring host agents by using the agent configuration file
- Agent logging
- Limiting host agent CPU and memory
- Converting a static agent to a dynamic agent
Configuring Instana backend
The Instana host agent connects to the Instana backend by using the HTTP/2 protocol with TLSv1.3
encryption. The connection is always established in a secure and encrypted way. For more information, see Setting up TLS encryption for agent endpoint.
The <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg
file contains the configurations that are used by the host agent to communicate with the Instana backend.
The values of the <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg
file can be overridden by using the following environment variables:
INSTANA_AGENT_ENDPOINT
- The host name of the Instana backend/service that your host agent connects to.INSTANA_AGENT_ENDPOINT_PORT
- The TCP port that your host agent connects to. The default is 443.INSTANA_AGENT_KEY
- The agent key used to create a relationship between the host agent and the Instana backend/service.
Values for the Instana agent endpoint and port are reported in the agent log; for example:
2024-04-16T05:09:09.627+0000 | INFO | ... | Backend | 55 - com.instana.agent - 1.1.718 | Connected using HTTP/2 to ingress-pink-saas.instana.rocks:443 ...
The endpoint, port, and agent key values are shown on Agent deployment screens in the Instana UI; for example, on a Linux - Automatic Installation (One-liner) screen, the deployment code includes ... -a aGeNTKEY0vaLuO0Eu1ABc ... -e ingress-green-saas.instana.io:443
.
Configuring multiple backends
In some cases, you might need an agent to report to multiple backends. For example, if shared services are used by separated environments, you can manually configure an agent to report to multiple backends in these separated environments.
The agent is counted in all backends separately because the number of licenses that it uses, and the setup effectively multiplies the bandwidth consumption of the agent.
To configure the host agent to report to multiple backends, follow the steps:
- Rename the
<instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg
configuration file to<instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend-1.cfg
- Create copies of the
<instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend-2.cfg
configuration file with the appropriate configurations for various backends that the agent reports to.
Each of the files that are created in the previous step can be adjusted to describe a different host agent endpoint and agent keys. These files can even contain different proxy settings.
Notes:
-
You can use any numeric or alphanumeric ID in the configuration file. For example,
<instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend-<alphanumeric>.cfg
-
If the
<instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg
configuration file exists, then all the other backend files are ignored. -
The Instana host agent Docker images are specially configured, so that adding extra backends can be easily done by mounting the backend files, such as
com.instana.agent.main.sender.Backend-2.cfg
.An example argument for a dockerized agent is as follows:
--volume <path-to-additional-backend-config>:/opt/instana/agent/etc/instana/com.instana.agent.main.sender.Backend-2.cfg
Setting up an agent proxy
To communicate with the backend effectively, Instana uses the HTTP/2 protocol to transfer the data.
In many cases, direct communication from the host agent to the backend can be granted to simplify the deployment of the agent.
In some cases, one dedicated entry is required that goes in or out of the network. For this reason, use Instana with various proxies. In general, HTTP, HTTPS, SOCKS4, and SOCKS5 proxies are supported. The proxy needs to support the CONNECT method to pass through. TLS termination at the proxy level is supported as long as HTTP/2 with ALPN is supported by the proxy.
For the host agent configuration, modify the following files:
<instana-agent-dir>/etc/mvn-settings.xml
<instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg
In the <instana-agent-dir>/etc/mvn-settings.xml
file, the following information must be present and not commented out in the <settings></settings>
section:
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
http://maven.apache.org/xsd/settings-1.0.0.xsd">
<localRepository>data/repo</localRepository>
<servers>
<server>
<id>features</id>
<username>_</username>
<password>_</password>
</server>
<server>
<id>shared</id>
<username>_</username>
<password>_</password>
</server>
<!-- for mirrors requiring auth
<server>
<id>features-mirror</id>
<username>myuser</username>
<password>mypassword</password>
</server>
<server>
<id>shared-mirror</id>
<username>myuser</username>
<password>mypassword</password>
</server>
-->
</servers>
<!-- If you want to access the Instana repository using an HTTP/Socks proxy,
uncomment the proxies section and insert proxy details. -->
<!-- https://maven.apache.org/guides/mini/guide-proxies.html -->
<!--
<proxies>
<proxy>
<id>agent-proxy</id>
<active>true</active>
<protocol>http</protocol>
<username></username>
<password></password>
<host></host>
<port></port>
</proxy>
</proxies>
-->
<!-- If you want to use a company internal mirror, make sure that mirror is
configured to use the Instana repository, uncomment the mirrors section
and insert maven mirror details.
If the mirror also requires auth it can be added above. -->
<!-- https://maven.apache.org/guides/mini/guide-mirror-settings.html -->
<!-- Repos that need to be mirrored -->
<!-- feature: https://artifact-public.instana.io/artifactory/features-public -->
<!-- shared: https://artifact-public.instana.io/artifactory/shared -->
<!--
<mirrors>
<mirror>
<id>features-mirror</id>
<url>http://repo.mycompany.com/artifactory/instana-public</url>
<mirrorOf>features</mirrorOf>
</mirror>
<mirror>
<id>shared-mirror</id>
<url>http://repo.mycompany.com/artifactory/instana-shared</url>
<mirrorOf>shared</mirrorOf>
</mirror>
</mirrors>
-->
</settings>
The configuration for the proxies uses the Instana agent in an environment with a proxy to communicate with the Maven endpoint configured in the mvn-settings.xml
file.
Furthermore, you need to reconfigure the <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg
configuration file to use the proxy to communicate between the agent and the Instana backend.
Make sure that the following lines are present and not commented out:
proxy.type=http
proxy.host=your-proxy-address-goes-here
proxy.port=your-proxy-port-goes-here
proxy.user=user-if-needed
proxy.password=password-if-needed
proxy.dns=true
Squid proxy configuration example
You can configure a squid proxy (www.squid-cache.org) in combination with Instana, when no other proxies are available.
Several ways are available to install Squid on your system. Most Linux® distributions include Squid in their repositories, and the software can be installed by using the preferred package manager.
If no package is available, or you want to run Squid on Microsoft® Windows®, then you can get Squid binary files from the (Squid Web Cache documentation).
After Squid is installed, an example configuration squid.conf is created, and it has a default configuration. If you want to use the proxy for the Instana communication exclusively, then you can back up the default configuration, and use the following squid.conf configuration:
# The tcp port squid is listening on
http_port 3128
# Please specify subnet with instana agents
acl instana_agent_net src 10.0.0.0/8
# This is the ip of the instana backend
acl instana_backend dstdomain saas-eu-west-1.instana.io
#acl instana_backend dstdomain ec2-54-144-114-141.compute-1.amazonaws.com
#acl instana_backend dstdomain saas-us-east-1.instana.io
#acl instana_backend dstdomain saas-us-east-1.instana.io
# This is the port used by Instana
acl instana_backend_port port 443
# This is the repo to download updates and additional sensors
acl instana_repo dstdomain artifact-public.instana.io
acl instana_repo_port port 80
acl instana_repo_port_secure port 443
# Protocol used for instana backend
acl instana_backend_proto proto HTTP
# Protocol used for instana backend
acl instana_repo_proto proto HTTP
acl instana_repo_proto_secure proto HTTPS
http_access allow instana_agent_net instana_backend instana_backend_port
http_access allow instana_agent_net instana_repo instana_repo_port
http_access allow instana_agent_net instana_repo instana_repo_port_secure
# DO NOT REMOVE THIS RULE!
http_access deny all
Setting up TLS encryption for agent endpoint
By default, HTTP
network connections to the agent on port 42699 and gRPC connections on port 4317 are not encrypted.
You can configure the agent to accept TLS-encrypted requests.
The following TLS versions are enabled: TLSv1
, TLSv1.1
, TLSv1.2
, and TLSv1.3
. The available TLS versions also apply when the agent itself issues secure requests, such as when the agent connects
to external metrics resources.
You can enable TLS encryption on the agent endpoints on port 42699 and 4317 by adding certificates in the <agent_installation>/etc/certs/
directory. By default, the agent looks for the following files:
<agent_installation>/etc/certs/tls.crt
<agent_installation>/etc/certs/tls.key
Other names, such as <agent_installation>/etc/certs/<your_certificate_name>.crt
or <agent_installation>/etc/certs/<your_key_name>.key
, are also allowed for the .crt
and .key
files if the <agent_installation>/etc/certs/
directory has only one file of each.
After you add the certificates, restart the agent to initialize the network connections.
Important: The host agent does not allow _enforcing_TLS
encryption. TLS is enabled on a connection only on the client's request.
Monitoring issues
You might encounter the following monitoring issues while setting up TLS encryption for an agent endpoint. These issues are displayed within the agent dashboard in the Instana UI. You need to resolve them before you proceed.
Monitoring issue type: agent_tls_cert_expired
The certificate that is used to set up the TLS encryption for agent endpoint expires. Make sure that you replace the expired certificate with a new certificate file.
Monitoring issue type: agent_tls_cert_about_to_expire
The certificate that is used to set up the TLS encryption for agent endpoint is about to expire in few days. Make sure that you to replace the certificate with a new certificate file.
Configuring host agent mode
The One-Liner host agent installation method and the Instana Host Agent Docker image accept an extra mode that is the AWS mode.
The AWS mode of the host agent is not used for monitoring hosts. The INFRASTRUCTURE
mode plus some automatic configuration of AWS data collection, as described in the AWS Agent documentation
is used for monitoring hosts.
You can set the host agent mode by configuring the <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg
configuration file:
mode = APM
# APM, INFRASTRUCTURE or OFF
After you change the <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg
configuration file, you must restart the host agent for the changes to take effect.
Configuring updates of dynamic host agents
Dynamic host agents can update themselves, and thus reduce the management overhead. To configure the updates of dynamic host agents such as configuring the updating interval, see Configuring updates of dynamic host agents.
Preventing agent mode override from UI
Because the agent mode can also be configured from the Instana UI, the configuration file has a flag that might be used to disable this overriding. In this way, the agent mode can be configured by using only the configuration file or an environment variable that is local to the installed agent.
If you don't want to set the agent mode from the UI, then add the following line to the <instana-agent-dir>/etc/instana/com.instana.agent.main.config.Agent.cfg
configuration file:
mode.web-override.allowed = false
Configuring host agents by using the agent configuration file
Most host agent configurations are applied by using the agent configuration file (<instana-agent-dir>/etc/instana/configuration.yaml
).
By using the agent configuration file, you can achieve the following goals:
- Create multiple configuration files
- Integrate the host agent with secret managers
- Obtain configurations from process environment and files
- Monitor extra file systems
- Specifying host tags
- Extract the installed packages list
- Set custom zones
- Monitor custom processes
- Configure secrets
- Capture custom HTTP headers
- Configure Kafka trace correlation headers
- Ignore processes
- Deactivate agent capabilities triggered by Instana UI
- Upload code source files to Instana
- Configure error report events (only AIX operating system)
For more information, see the Configuring host agents by using the agent configuration file topic.
Agent logging
By default, the Instana agent logs to the <instana-agent-dir>/data/log/agent.log
log file, which is rotated if the file grows too large. If you are running the agent in a container, the Instana agent logs to the console instead,
which is managed by the Docker and is accessible by using the Docker logs.
The log level can be increased to debug by changing the level in the <instana-agent-dir>/etc/org.ops4j.pax.logging.cfg
configuration file from log4j2.logger.instana.level=INFO
to log4j2.logger.instana.level=DEBUG
.
Important: Ensure that the line has no trailing white space.
The older agent installs need this line to be changed from log4j.logger.com.instana=INFO, out, osgi:*
to log4j.logger.com.instana=DEBUG, out, osgi:*
Log rotation
By default, the agent uses a log rotation of 10 times 5 MB agent log files. That is, at every 5 MB, the file is rotated and 10 files are kept, and the file 11 is deleted.
log4j2.appender.rolling.policy.type = SizeBasedTriggeringPolicy
log4j2.appender.rolling.policy.size = 5MB
log4j2.appender.rolling.strategy.type = DefaultRolloverStrategy
log4j2.appender.rolling.strategy.max = 10
Syslog
The agent uses Log4j2, which is a modern and flexible logging facility. See the following example to configure syslog:
log4j2.rootLogger.appenderRef.Syslog.ref = Syslog
log4j2.rootLogger.appenderRef.Syslog.level = ERROR
log4j2.appender.syslog.type=Syslog
log4j2.appender.syslog.name=Syslog
log4j2.appender.syslog.layout.type=PatternLayout
log4j2.appender.syslog.layout.pattern = ${log4j2.pattern}
log4j2.appender.syslog.facility=SYSLOG
log4j2.appender.syslog.host=localhost
log4j2.appender.syslog.port=514
log4j2.appender.syslog.protocol=UDP
Logs to STDOUT
In custom container images, you might want to log to STDOUT
. You can do so by providing the following configuration in <instana-agent-install-dir>etc/org.ops4j.pax.logging.cfg
configuration file:
log4j2.rootLogger.appenderRef.Console.ref = Console
log4j2.appender.console.type = Console
log4j2.appender.console.name = Console
log4j2.appender.console.layout.type = PatternLayout
log4j2.appender.console.layout.pattern = ${log4j2.pattern}
Log4j2
Because the logging uses the standard log4j2 logging, it can be configured in many more ways as described in the log4j2 documentation.
Logging metrics or traces to file
The agent can temporarily log metrics or traces that are sent through this agent to a file on the disk. The feature to log metrics or traces to a file is often used to debug various issues that are related to metrics, traces, tracing, or spans.
To enable the feature, locate the <instana-agent-dir>/etc/instana/com.instana.agent.main.sender.File.cfg
configuration file, and update its contents with the following commands:
# Configuration of local logging. Changes will be hot-reloaded.
# Activate logging of outgoing payloads to local disk by setting a non-empty
# prefix. The log file will be written to data/log, and the file will have the
# defined prefix followed by a timestamp.
# Note: There is no automatic rotation of those files.
prefix=locallog
# The file can be filtered to either "metrics" or "traces".
# If empty or absent, there will be no filtering.
type=traces
As noted, the changes are hot-reloaded, and can be activated immediately. The feature must be enabled temporarily only because leaving the feature enabled has the potential to occupy all the available disk space, if given enough time and traffic.
Allow the logging to continue for a minute or two while the traffic is generated to the component when you trace the issues. Then, revert the changes or comment all lines in that file. Again, the changes that are written to disk are hot-reloaded by the Instana agent.
If you are working with a support ticket, attach the resulting log file to the support ticket. The log file is located in the <instana-agent-dir>/data/log/locallog_*.log
log file.
Limiting host agent CPU and memory
In certain scenarios, controlling resource consumption of processes closely is crucial. This controlling can become especially helpful in environments that use resource sharing, and for the systems with low resources. While the Instana agent is designed to use few resources as possible, the resources can be further limited by following these instructions as another safeguard to respect clear resource limits.
All the following examples limit the agent's CPU to computing resources equivalent to half a CPU and to a maximum of 512 MB memory. However, not all resources would necessarily come from the same CPU.
Systemd
- Create a configuration file named
/etc/systemd/system/instana-agent.service.d/20-resource_limits.conf
, and add the following content to it:
[Service]
CPUAccounting=true
CPUQuota=50%
MemoryAccounting=true
MemoryMax=512M
- Run
systemctl daemon-reload
. - Restart the
instana-agent
service.
Docker
Run the `instana-agent` container with the following extra parameters:
For Docker 1.13 and later:
--cpus=0.5 --memory=512m
For Docker 1.12 and earlier:
--cpu-period=100000 --cpu-quota=50000 --memory=512m
Kubernetes
Add the following configuration snippet to the host agent's container configuration:
livenessProbe:
httpGet: # Agent liveness is published on localhost:42699/status
path: /status
port: 42699
initialDelaySeconds: 75
periodSeconds: 5
resources:
requests:
memory: "256Mi"
cpu: "0.5"
limits:
memory: "512Mi"
cpu: "1.0"
The configuration requests less memory and CPU for the instana-agent
container.
Converting a static agent to a dynamic agent
To change a static agent to a dynamic agent, update the following configuration files:
-
Update the
<agent-dir>/etc/org.ops4j.pax.url.mvn.cfg
file as shown:org.ops4j.pax.url.mvn.repositories=https://artifact-public.instana.io/artifactory/features-public@id=features@snapshots@snapshotsUpdate=always,https://artifact-public.instana.io/artifactory/shared@id=shared@snapshots@snapshotsUpdate=always
-
Update the
<agent-dir>/etc/instana/com.instana.agent.main.config.UpdateManager.cfg
file as shown:mode=AUTO
-
In
<agent-dir>/etc/instana/com.instana.agent.bootstrap.AgentBootstrap.cfg
file, ignore or remove theversion
orpin
based on your requirements as shown:#version=<hash>
-
Delete all the references to the version of the BOMs present in the system directory by executing the following command:
find <agent-dir>/system/ -type d -name '1.0.0-SNAPSHOT' -exec rm -rv {} \;
Attention: If you install a dynamic agent first, you cannot change it to a static agent.