Managing host agents

All host agents that report to Instana are displayed on the Agents View. This page gives a quick overview of all agents, their important properties, and general health. Furthermore, each host agent has its own Agent Management View, which displays real-time information about important metrics and allows you to configure the agent in various ways.

Agent management dashboard

To view an Agent Management Dashboard, on any host dashboard click Open Agent Management.

Agent management

Furthermore, all Agent Management Dashboards can be reached through the Agents View.

The Agent Management Dashboard displays the agent configuration and current runtime metrics. In the management area, configurations can be changed, and operations such as a restart can be run. Further down runtime metrics, and the current agent log are shown.

Agent dash

Changing agent modes

The host agent mode is relevant for host-based licensing, and it is used to decide whether the agent counts as an Infrastructure or an APM agent. The host agent mode can be toggled by using Change Agent Mode control.

In the Change Agent Mode dialog, the following options are provided:

  • APM, which maps to the APM host agent mode
  • Infrastructure, which maps to the INFRASTRUCTURE host agent mode
  • Disabled, which maps to the OFF host agent mode

An agent that is Disabled does not report any data and as such is not included in the Managed Virtual Servers (MVS) count. 

Changing log level

The log level defines the level of detail that the agent logs contain.

In the Change Agent Log Level dialog, the following options are provided:

  • INFO, normal level of details
  • DEBUG, extended details for troubleshooting
  • TRACE, most details for further problem analysis

Updating agents

This button manually triggers the agent auto update as described at Updates of Dynamic Host Agents.

During the update, the host agent cannot provide any data so you might see a short gap.

Resetting agents

Resetting the agent is similar to restarting the agent process, but the current process remains active. It means that OS level watchdogs and service scripts will not see the process identifier (pid) changing.

During the reset, the host agent cannot provide any data, so you might see a short gap.

Rebooting agents

Rebooting the agent stops the current agent process and create a new one. The process of the used start script remains unchanged, but operating system level watchdogs and service scripts might see a changed process identifier (pid) depending on their watch target.

During the restart, the host agent cannot provide any data, which results in a short gap.

Sensor information

To view the list of sensors and host agent components that are used by the agent, click Sensors Info.

Configuration management

Git-based configuration management can be enabled and configured in the configuration management section.

Details of this feature, and the used nomenclature are described on the Git-based configuration management page.

To support Configuration Management at least agent-bootstrap version 1.2.11 is required.

To have access to this feature, users need to have a role that with the Configuration of agents permission. For more information about users, roles, and permissions, see managing users docs.

Agent Health View permission

Initializing configuration management

If no configuration management is active, it can be initialized with the Initialize button in the Configuration Management section.

Initializing configuration management

The initialization dialog provides the settings:

  • Remote Name, name the Git remote should be set to
  • Remote Branch, name of the branch to use for updates
  • Remote URI, URI of the Git repository

Initialize & Restart configures the agent according to these settings and restart it to fetch the newest configuration.

Updating configuration management

If configuration management is already active, its details are displayed in the Configuration Management section. It includes information about the configured remote, like branch name and URI, and the currently used commit, like hash and commit message.

To update the configuration, the Configuration Management dialog can be opened by using the Update button in the section.

Initializing configuration management

Similar to the initialization dialog the following settings can be changed:

  • Remote Name, name the Git remote should be set to
  • Remote Branch, name of the branch to use for updates
  • Remote URI, URI of the Git repository

Update & Restart configures the agent according to these settings and restart it to fetch the newest configuration.

Self monitoring

Not only is the host agent monitoring other software that is running on the host, but it also does constant lightweight self monitoring. Gathered metrics can be used to observe host agent performance and resource consumption:

Agent metrics

Agents view

The Agents view, which is located in the main navigation bar under More > Agents, provides an overview of all host agents that report to the Instana tenant unit.

When you use the search bar on the page, the view can be narrowed down to a subset of agents by using Dynamic Focus Queries.

Agent Health View

Bulk agent management

To have access to this feature, users need to have a role with the Configuration of agents permission. For more information about users, roles, and permissions, see managing users docs.

Agent Health View permission

The Agents view allows for quick access to the following features:

  • Update All Agents: updates all reporting agents to the latest version.
  • Reset All Agents: resets all reporting agents.
  • Instana Agent Installation: agent installation instructions for all supported platforms.

Host agent health status

The host agent health monitoring table contains the following information:

  • Agent entity link, which links to the agent management dashboard
  • Agent version
  • Agent boot version
  • Origin of installation
  • Update Mode
  • Mode (Disabled, Infrastructure, or APM)
  • Log level
  • JVM name and version
  • Reporting status

Reported agents

As with most features in Instana, the host agent health monitoring view strongly depends on the current time configuration.

There are two important things to understand:

  1. The table presents all agents, which reported within the current time window.
  2. The status (reporting or not reporting) is based on the selected time.

The following conceptual image shows how the reporting indicator behaves.

Agent Health View concept

In this example image, the time window was set so that three agents are listed. The end of the selected time window was set somewhere around 6:20am. The agent table would now indicate Agent 1 as not reporting because it stops reporting at 6:00am, so before 6:20am. The other two agents report at that time.

Debugging

Creating a host agent heap dump

To create a heap dump of the host agent, run the following command:

TS=`date +%s` && /opt/instana/agent/jvm/bin/jmap -dump:file=/tmp/agent-dump-$TS.hprof `cat /opt/instana/agent/agent.pid` && gzip /tmp/agent-dump-$TS.hprof

Troubleshooting

There might be cases where your setup does not work at first. If this troubleshooting section does not answer the questions you have, contact the IBM Instana support team with information about your experience, so that we can help you and update our documentation accordingly.

Instana Agent contains vulnerable Log4j library

Monitoring issue type: agent_log4j_vulnerability

Due to a vulnerability discovered in Apache Log4j (CVE-2021-44228 and CVE-2021-45046) we have provided an update to our agent component. It is based on the Pax Logging library as used by our agent packages.

We recommend to update your Instana Agent installation packages to (re-)install the latest Agent version as described here: Installing host agents. Latest packages contain upgraded Pax Logging libraries that mitigate these CVE's. For more information about how Instana mitigates the risk for the Apache Log4j CVE-2021-44228 and CVE-2021-45046, see incident page.

Instana Agent is missing the /tmp directory

Monitoring issue type: agent_tmp_directory_missing

The Instana host agent creates temporary JAR files in the $TMP/.instana directory. These JAR files are required for the JVM monitoring. This warning indicates that either the necessary JAR files are missing or the Instana tmp directory is unavailable, which can lead to Java attachment issues. The Instana Agent periodically monitors the state of the Instana tmp directory and its relevant JAR files to prevent Java attachment failure. To avoid the issue , make sure that the tmp location has read and write permissions and avoid unnecessary cleanup of the /tmp location.

Insufficient disk space for the /tmp directory

Monitoring issue type: insufficient-disk-space-for-tmp-directory

At run time, the Instana agent container creates temporary JAR files in the $TMP/.instana directory. These JAR files are required for the JVM monitoring. This warning indicates that insufficient space in the tmp location to accommodate temporary files, which can lead to a Java attachment failure. To address this issue, increase the allocated space for tmp in the container deployment.

Missing TLSv1.3 support for Instana agent

Monitoring issue type: agent_jvm_tls_1_3_missing

The Instana agent requires TLSv1.3 support on the installed host. If TLSv1.3 is not supported, monitoring capabilities will be degraded.

You are recommended to update your Instana agent TLS version to TLSv1.3. The following TLS versions are enabled: TLSv1, TLSv1.1, TLSv1.2, TLSv1.3. For more information, see Setting up TLS encryption for agent endpoint.

Solaris JVM file permission issue

Monitoring issue type: solaris_jspawnhelper_executable_issue

This monitoring issue indicates that the jspawnhelper JVM file on the Solaris host does not have the executable permission.

To fix the Solaris JVM file permission issue, provide the executable permission to the jspawnhelper JVM file in the JVM path using the following command:

chmod +x <jvm-path>/jspawnhelper

Where JVM path for earlier version of Java 9 is <instana-agent-dir>/jvm/jre/lib/sparcv9/jspawnhelper and for Java 9 or later is <instana-agent-dir>/jvm/jre/lib/jspawnhelper.