Managing host agents

Edit online

You can view and manage all the agents that report to Instana by using the Agents page.

To open the Agents page, from the navigation menu of the Instana UI, select Data sources > Instana Agents.

The Instana Agents tab displays the following information:

  • Total Agents: Displays the total number of agents that are configured in the Instana tenant unit (including reporting and not reporting).
  • Reporting Agents: Provides a chart of all agents that report to the Instana tenant unit within the selected period. Agents that do not report during the selected period are excluded. You can select a bar to zoom in for details, highlight the details across all charts, and export the data in JSON and CSV format.
  • Agent details: Provides a quick overview of all agents, including their key properties and status.
Note: Use the search bar on the page to narrow down the details to a subset of agents. For more information, see Dynamic Focus Queries.

In addition, you can perform the following actions directly from the Agents page:

  • Update all agents: Updates all reporting agents to the latest version.
  • Reset all agents: Resets all reporting agents.
  • Install agents: Opens the Agent deployment page. You can select the required agent from the grid to open its installation page.
  • Install agent-based integrations: Installs the IBM Monitoring Integration Service by using one-liner installation. For more information, see Agent-based integrations.
Note: To see these options, your user account must have a role with the Configuration of agents permission. For more information about users, roles, and permissions, see Managing user access.

Managing agents by using agent dashboard

Edit online

To manage an individual agent, you must open its dashboard. You can view real-time information about important metrics and configure the agent as you require by using the agent dashboard.

To open the dashboard of an agent, select the agent from the Agent details table.

Alternatively, click Infrastructure in the navigation menu of the Instana UI. Select the required host and click Open dashboard. In the host dashboard, scroll to the end and click Open Agent Management.

You can perform the following tasks by using the agent dashboard:

Changing agent modes

Edit online

You must specify the host agent mode for host-based licensing. Host agent mode is used to identify if the agent is an Infrastructure or an APM agent.

To change the agent mode, click Change Agent Mode. In the Change Agent Mode dialog, select the required agent mode from the drop-down list. The following modes are available in the list:

  • APM: Indicates the APM host agent mode.
  • Infrastructure: Indicates the INFRASTRUCTURE host agent mode.
  • Disabled: Indicates the OFF host agent mode.
Note: If you set the mode as Disabled, the agent does not report any data and is not included in the Managed Virtual Servers (MVS) count.

Changing log level

Edit online

You can set the log level to specify the level of detail that is included in the agent logs.

To change the log level, click Change Log Level. In the Change Log Level dialog, select the required level from the drop-down list. The following levels are available in the list:

  • INFO: Provides normal level of details.
  • DEBUG: Provides extended details for troubleshooting.

Resetting agents

Edit online

You can reset the agent and sensors by clicking Reset Agent and Reset sensors.

Resetting the agent is similar to the agent restart process, but the current processes remain active. This means that the OS-level watchdogs and service scripts do not see the process identifier (pid) changing.

Note: During the reset, the host agent cannot provide any data.

Rebooting agents

Edit online

You can reboot the agent to stop the current agent process and create a new one.

The process of the used start script is unchanged, but operating system level watchdogs and service scripts might see a changed process identifier (pid) depending on their watch target.

Note: During the restart, the host agent cannot provide any data.

Viewing agent and sensor information

Edit online

To view the list of sensors and host agent components that are used by the agent, click Sensors Info.

To view the agent configuration, click Agent configuration.

To download agent logs, click Agent logs.

Enabling Git-based configuration management

Edit online

You can enable and configure Git-based configuration management from the Configuration Management section in the agent dashboard.

Configuration Management requires the following prerequisites:

  • You have installed agent-bootstrap version 1.2.11 or later on your machine.
  • Your user account has a role with the Configuration of agents permission. For more information about users, roles, and permissions, see Managing user access.

Initializing configuration management

Edit online

If you meet the prerequisites, you can initialize the configuration management.

If no configuration management is active, initialize Git-based configuration management by clicking Initialize in the Configuration Management section.

Then, enter the following information in the Configuration Management dialog:

  • Remote Name: The name for the Git remote.
  • Remote Branch: The name for the branch to use for updates.
  • Remote URI: The URI of the Git repository.

Click Initialize & Restart to configure the agent according to the settings and restart to fetch the latest configuration.

Updating configuration management

Edit online

If configuration management is already active, you can see the details in the Configuration Management section. The details include information about the configured remote, such as branch name and URI, and information about the currently used commit, such as hash and commit message.

To update the configuration, click Update and enter the required details in the Configuration Management dialog.

Click Update & Restart to configure the agent according to the settings and restart to fetch the latest configuration.

Monitoring the agent

Edit online

The Instana agent monitors itself and gathers metrics regarding performance and resource consumption. You can view these metrics, such as Monitoring Issues, CPU Load, Garbage Collection, Network, and so on, in the agent dashboard. The metrics also include information about sensors and agent log output.

Debugging the agent

Edit online
Identify the Java runtime that your agent is using. The agent runs on either a HotSpot-based Java runtime or an OpenJ9-based Java runtime. To identify your agent's underlying Java runtime, go to the agent management dashboard in the Instana UI and observe the "Java runtime" property in the Configuration window:
  • HotSpot-based Java runtimes identify as OpenJDK
  • OpenJ9-based Java runtimes identify as OpenJ9.

Create a heap dump for the host agent on Linux with a HotSpot-based Java runtime

Edit online

To create a heap dump for the host agent on a Linux system, run the following command in a Linux shell:

TS=`date +%s` && /opt/instana/agent/jvm/bin/jmap -dump:file=/tmp/agent-dump-$TS.hprof `cat /opt/instana/agent/agent.pid` && gzip /tmp/agent-dump-$TS.hprof

The command creates the heap dump and saves the compressed file in /tmp/agent-dump-<timestamp>.hprof.zip.

Create a heap dump for the host agent on Windows with a HotSpot-based Java runtime

Edit online

To create a heap dump for the host agent on a Window system, run the following command in a PowerShell:

$TS = [int][double]::Parse((Get-Date -UFormat %s)); C:\"Program Files"\Instana\instana-agent\jvm\bin\jmap -dump:file="$env:TEMP\agent-dump-$TS.hprof" (Get-Content "C:\Program Files\Instana\instana-agent\agent.pid"); Compress-Archive -Path "$env:TEMP\agent-dump-$TS.hprof" -DestinationPath "$env:TEMP\agent-dump-$TS.hprof.zip" -Force; Remove-Item "$env:TEMP\agent-dump-$TS.hprof"

The command creates the heap dump and saves the compressed file in $TEMP\agent-dump-<timestamp>.hprof.zip.

Create a heap dump for the host agent on Linux with an OpenJ9-based Java runtime

Edit online

To create a heap dump for the host agent on a Linux system, complete the following steps:

An ad-hoc heap dump can be created only when the agent runs with the -Dcom.ibm.tools.attach.enable=yes setting. To verify this setting, open the agent management dashboard in the Instana UI, navigate to the Instana agent process entity on the left side of the process dashboard, and locate com.ibm.tools.attach.enable in the process arguments list.

The following steps assume that the Instana agent is installed in the default location at /opt/instana/agent and the agent is running as a Systemd service. You need to adopt the paths if you installed Instana agent into a different location.

Note: If your agent is already running with the -Dcom.ibm.tools.attach.enable=yes setting, you can skip steps 1 through 4.
  1. Log on to the Linux host as root user or change your shell session to the root user.
  2. Open the /opt/instana/agent/bin/setenv file for editing.
  3. Locate the line that starts with EXTRA_JAVA_OPTS= and change the property -Dcom.ibm.tools.attach.enable=no to -Dcom.ibm.tools.attach.enable=yes.
  4. Restart the agent with systemctl restart instana-agent.
  5. Let the agent run until the situation to pull the heap dump occurs.
  6. To create a heap dump for the host agent on a Linux system, run the following command: 
    TS=`date +%s` && /opt/instana/agent/jvm/bin/jcmd `cat /opt/instana/agent/agent.pid` Heap.dump /tmp/agent-dump-$TS.phd  && gzip /tmp/agent-dump-$TS.phd

    The command creates the heap dump and saves the compressed file in /tmp/agent-dump-<timestamp>.phd.zip.

Create a heap dump for the host agent on Windows with an OpenJ9-based Java runtime

Edit online

To create a heap dump of the host agent on a Windows system, complete the following steps:

An ad-hoc heap dump can be created only when the agent runs with the -Dcom.ibm.tools.attach.enable=yes setting. To verify this setting, open the agent management dashboard in the Instana UI, navigate to the Instana agent process entity on the left side of the process dashboard, and locate com.ibm.tools.attach.enable in the process arguments list.

The following steps assume that the Instana agent is installed in the default location at C:\Program Files\Instana\instana-agent and the agent is running as a Systemd service. You need to adopt the paths if you installed Instana agent into a different location.

Note: If your agent is already running with the -Dcom.ibm.tools.attach.enable=yes setting, you can skip steps 1 through 4.
  1. Log on to the Windows host as Administrator.
  2. Open C:\Program Files\Instana\instana-agent\bin\setenv.bat file for editing.
  3. Locate the line that starts with SET EXTRA_JAVA_OPTS= and change the property -Dcom.ibm.tools.attach.enable=no to -Dcom.ibm.tools.attach.enable=yes.
  4. In the "Services" Microsoft Management Console, locate the "Instana Agent Service" and restart the service.
  5. Let the agent run until the situation to pull the heap dump occurs.
  6. Create the file C:\Users\Administrator\agent-dump.ps1 with the following content: 
    $TS = [int][double]::Parse((Get-Date -UFormat %s))
$AGENT_PID = (Get-Content "C:\Program Files\Instana\instana-agent\agent.pid")
C:\"Program Files"\Instana\instana-agent\jvm\bin\jcmd $AGENT_PID Dump.heap C:\Windows\Temp\agent-dump-$TS.phd
Compress-Archive -Path C:\Windows\Temp\agent-dump-$TS.phd -DestinationPath C:\Windows\Temp\agent-dump-$TS.phd.zip -Force
Remove-Item C:\Windows\Temp\agent-dump-$TS.phd
  7. Open a PowerShell that runs with Administrator privileges.
  8. Run the following commands to create and run a ScheduledTask that executes the heap dump as the SYSTEM account: 
    $SchedScript = New-ScheduledTaskAction -Execute "PowerShell.exe" -Argument '-ExecutionPolicy Bypass -file C:\Users\Administrator\agent-dump.ps1'
Register-ScheduledTask -User SYSTEM -taskname !_AgentDump -Action $SchedScript | Out-Null
Start-ScheduledTask -TaskName "!_AgentDump" 
Unregister-ScheduledTask -TaskName "!_AgentDump" -Confirm:$false

    The command creates the heap dump and saves the compressed file in C:\Windows\Temp\agent-dump-<timestamp>.phd.zip.

Troubleshooting the agent

Edit online

See the troubleshooting methods to the common issues that are noticed during agent installation and operation.

Analyzing agent performance with self-profiler

Edit online

You can analyze the performance metrics of the Instana host agent by using the agent self-profiler feature.

The self-profiler is only supported on Linux systems with x86_64-bit architecture. {: note} To self-profile the host agent, complete the following steps:

  1. In the Support section of the agent dashboard, click Profile Agent. The profiler runs for 10 minutes. You can check the agent logs to see that the profiler is started. See the following example: 
    2025-03-19T14:21:48.077+00:00 | INFO  | instana-executor-thread-2-1      | ndRequestHandler | com.instana.agent-self-profiling - 1.0.0 | Starting Agent profiler for 10 minutes
2025-03-19T14:21:52.998+00:00 | INFO  | instana-scheduler-thread-3-4     | Profiler         | com.instana.agent-self-profiling - 1.0.0 | Profiler installed

  2. After you start the profiler, wait for 2 minutes, and then refresh the dashboard. A new button, Analyze Profiles appears in the agent dashboard.

  3. Click the Analyze Profiles button to open the Analyze profiles page.

  4. On the Analyze profiles page, click View all. You can share the Flame graph along with any additional information that is requested by IBM Support for agent performance analysis.

Instana Agent is missing the /tmp directory

Edit online

Monitoring issue type: agent_tmp_directory_missing

The Instana host agent creates temporary JAR files in the $TMP/.instana directory. These JAR files are required for the JVM monitoring. This warning indicates that either the necessary JAR files are missing or the Instana tmp directory is unavailable, which can lead to Java attachment issues. The Instana agent periodically monitors the state of the Instana tmp directory and its relevant JAR files to prevent Java attachment failure.

To avoid the issue, make sure that the tmp location has read and write permissions, and avoid unnecessary cleanup of the /tmp location.

Insufficient disk space for the /tmp directory

Edit online

Monitoring issue type: insufficient-disk-space-for-tmp-directory

At run time, the Instana agent container creates temporary JAR files in the $TMP/.instana directory. These JAR files are required for the JVM monitoring. This warning indicates insufficient space in the tmp location to accommodate temporary files, which can lead to a Java attachment failure. To address this issue, increase the allocated space for tmp in the container deployment.

Memory leak in Pax URL 2.6.16

Edit online

Monitoring issue type: pax_url_mvn_leak

The OPS4J Pax URL version 2.6.16 has a memory leak that is reset every 24 hours as part of the agent update. Depending on the number of sensors that are stopped and started during this period, the agent might encounter an OutOfMemoryError error. To address this issue, a fix is added in the agent assemblies that are released after 18 June 2025. If you notice this monitoring issue, update your Instana agent assembly to a version that is released on or after 18 June 2025.

For more information about this issue, see the issue and the corresponding pull request in the pax-url GitHub repository.

Missing TLSv1.3 support for Instana agent

Edit online

Monitoring issue type: agent_jvm_tls_1_3_missing

The Instana agent requires TLSv1.3 support on the installed host. If TLSv1.3 is not supported, monitoring capabilities are degraded.

To resolve this issue, update your Instana agent TLS version to TLSv1.3. The following TLS versions are enabled: TLSv1, TLSv1.1, TLSv1.2, TLSv1.3. For more information, see Setting up TLS encryption for agent endpoint.

Solaris JVM file permission issue

Edit online

Monitoring issue type: solaris_jspawnhelper_executable_issue

This monitoring issue indicates that the jspawnhelper JVM file on the Solaris host does not have the executable permission.

To fix the Solaris JVM file permission issue, provide the executable permission to the jspawnhelper JVM file in the JVM path by using the following command:

chmod +x <jvm-path>/jspawnhelper

For Java 9 or later, specify the JVM path as *instanaAgentDir*/jvm/jre/lib/jspawnhelper. For earlier versions of Java, specify the JVM path as *instanaAgentDir*/jvm/jre/lib/sparcv9/jspawnhelper.

If this troubleshooting section does not resolve your issue, contact IBM Instana support with the information about your experience.