Agent bundle releases

Updated Bundles

This agent release updates the following bundles:

• grpc core 1.0.15: This bundle updates protobuf-java to version 4.29.3, protobuf-java-util to version 4.29.3, and proto-common-google-common-protos to version 2.50.1.
• agent process handling 1.0.7: This bundle adds the logic to identify the location of ctr on a Tanzu Cluster.

Additionally, the following bundles are updated to maintain compatibility with the updated Protobuf version:

• agent cri client 1.0.1
• agent grpc channel cache 1.0.1
• agent grpc channel cache api 1.0.1
• agent open telemetry base 2.2.2
• agent open telemetry service 1.1.4
• agent open telemetry grpc 1.1.4
• agent open telemetry http 1.1.4
• agent otel sensorsdk api 1.0.3
• otel sensorsdk utils 1.0.1
• otel sensorsdk database 1.0.11
• otel sensorsdk host 1.0.5
• otel sensorsdk llmonitor 1.0.6
• otel sensorsdk dcgm 1.0.1
• otel sensorsdk jvm 1.0.2
• otel sensorsdk k8s 1.0.1
• agent prometheus 1.0.12

Fixes

This release fixes the following issues:

• Previously, the location of ctr on Tanzu Clusters was not identified correctly, which caused issues with container operations. This issue is now fixed by adding the logic to accurately detect the ctr location in such environments.

Features

This agent release adds the following features:

• The UnifiedProcess abstraction is added to offer extended functionalities to manage normal processes and CRI processes.
• CriClient and its associated configurations are added for seamless interaction with CRI API gRPC endpoints.
• The agent cgroup bundle is added to enhance container metric management.
• The agent util bundle now provides improved resource and thread management.

Fixes

This release fixes the following issues:

• Previously, entities in the backend were marked as offline due to missing data. This issue is now fixed by keeping entities alive with empty data.
• Previously, the agent's handling of empty files caused unexpected behavior. This issue is now fixed by returning a zero-length byte array when empty files are encountered.
• Previously, the asynchronous process for adding public IP gateway interests caused delays. This issue is now fixed by changing the operation to be synchronous.
• Previously, in rootless Podman environments, temporary files that were created with the correct set of permissions were overwritten because of the umask settings. This issue is now fixed by explicitly setting the file permissions after the temporary file has been created.

Features

This agent release adds the following features:

• The Instana agent has stopped the support for Java 8. You must use Java 11. When you start newly installed agents with Java 8, the agent now shows a monitoring event. This monitoring event is relevant only if you install or use your own JVM. Containerized or package-based agents already use Java 11.

Fixes

This release fixes the following issues:

• Previously, in rootless Podman environments, copying files to the target container might fail as the agent did not define a permission set for the temporary file. This issue is now fixed, and the temporary files are created with rwxr-xr-x.
• Previously, the agent failed to set any socket information for a containerD instance when multiple sockets are present. This issue is now fixed, and the agent sets the correct socket information for a containerD instance.
• Previously, the agent had a race condition on JVM connections and kept a connection open even though it was internally already marked as closed. The wrong connection was removed only when the agent was restarted. This issue is now fixed, and the agent correctly closes the broken connection.

Fixes

This release fixes the following issues:

• Previously, in rootless Podman environments, the df command failed a check that is used to verify the available size within a container before a JVM attachment. This issue is fixed, and the df command is now run by the user as required.
• Previously, during the startup, the agent failed to execute some commands that are necessary to identify if the agent is running on a host that supports various container environments. The failure occurred because the default timeout to run the command was one second, which was not sufficient. This issue is fixed, and you can now control the timeout by using a new environment variable INSTANA_COMMAND_RUNNER_TIMEOUT_SECONDS=5, which has a default duration of 5 seconds.
• Previously, the agent did not include the --user argument while executing a nsenter command when the environment variable NSENTER_MODE_ENABLED=true was set. This issue is fixed, and the agent now uses the --user argument when required.

Fixes

This release fixes the following issues:

• Previously, the uniqueId from the agent on Linux was not deterministic when the main interface identified by Sigar was a vlan interface. This was because these interfaces are not ordered by the host. This issue is fixed, and the vlan interfaces are now sorted first if names that start with vlan are identified.
• Previously, the uniqueId from the agent on Windows used the KM-TEST Loopback Adapter interface. Because this is a loop-back interface, the MAC address was not unique. This issue is fixed, and the agent now ignores the KM-TEST Loopback Adapter interface when other interfaces are available.
• Previously, the agent did not log when the agent key for a backend was wrong. This issue is fixed, and the agent now logs a warning every 10 minutes when it receives a 403 error report from the backend, which indicates that the agent key is wrong.

Features

This agent release adds the following features:

• The agent now has a Selective Monitoring mode. This mode adds the ability to monitor only the processes that hold specific environment variables.

  The agent checks if the process is started with the environment variable INSTANA_SELECTIVE_MONITORING.

  • If INSTANA_SELECTIVE_MONITORING is set to DEFAULT or has an empty value, the agent operates in normal mode. All processes with INSTANA_MONITORING=false and INSTANA_IGNORE=<any value> are ignored.

  • If INSTANA_SELECTIVE_MONITORING is set to OPT_IN, the agent operates in Selective Monitoring mode. The agent monitors only the processes that have INSTANA_MONITORING=true, INSTANA_MONITORING=apm, or INSTANA_MONITORING=infra. All processes with INSTANA_MONITORING=false and INSTANA_IGNORE=<any value> are ignored.

  Currently, there is no differentiation between apm or infra modes.

Fixes

This release fixes the following issues:

• Previously, the agent always executed the id command in the context of the agent, which could fail if the agent was running in a container. This issue is fixed, and the id command is now executed on the host.
• Previously, the agent failed to start because of Snakeyaml imports that required a minimum version of 2. This version was not present in the agents with agent bootstrap version earlier than 1.2.9. This issue is fixed, and the agent util now imports Snakeyaml versions 1 to 3.

Fixes

This release fixes the following issue:

• Previously, the agent only checked if the /etc/passwd file contained a user ID to identify a user. This issue is fixed, and the agent now correctly identifies a user by using the id command instead.
• Previously, the agent had a fallback setting to use the sudo command on AIX for the JVM attachment. However, the fallback was not always possible as the sudo command is not available on AIX by default. This issue is fixed, and the agent now only uses the su command on AIX for the JVM attachment.

Features

This agent release adds the following features:

• The agent now has a dedicated bundle to handle all activities that are related to processes. These activities also include the abstraction around containerized processes.
• The agent now collects a validation of the confiugration.yaml. You can retrieve this data by clicking on the Collect Support Information button on the agent dashboard.

Fixes

This release fixes the following issues:

• Previously, the agent did not check if the /etc/passwd file had more characters to evaluate when a user ID was not found in the file. This issue is now fixed, and the agent correctly identifies when the checking of the file is finished.
• Previously, the agent did not evaluate all the available namespaces of the host system when executing the nsetner -a command. This issue is fixed, and the agent now checks all available namespaces and adds them explicitly to the nsenter command.

Features

This agent release adds the following features:

• The agent now collects a thread dump of all agent threads. You can retrieve this data by clicking on the Collect Support Information button on the agent dashboard.

Fixes

This release fixes the following issue:

• Previously, the agent did not use the correct PATH when it was run in a container on RKE2. This issue is now fixed, and the agent correctly identifies when it is run on RKE2. The agent also adds /var/lib/rancher/rke2/bin to the PATH if it is missing.
• Previously, the agent used a thread from the scheduler during rebooting or resetting. When the agent installed other features at the same time, the agent could get stuck on the reboot or reset. This issue is now fixed, and the agent uses a thread from the main group and finishes the reboot or reset.
• Previously, the agent used the su command to change the user on IBM i when the agent was run as the root user QSECOFR. This attachment failed because IBM i does not support su. This issue is now fixed, and the agent uses the custom ibmi-sudo executable to change the user.
• Previously, the agent used the sudo command to change the user on z/OS as a fallback. This attachment failed because z/OS does not support sudo. This issue is now fixed, and the agent only uses the su command to change the user.
• Previously, the agent did not provide details about the threads that are stuck when the scheduler and executor stopped the threads for the agent to stop, restart, or update. This issue is now fixed, and the agent writes a thread dump into the logs when the threads are stopped.
• Previously, the agent attached to JVMs that are part of a buildah step. This issue is now fixed, and the agent ignores JVMs that have a parent process with the name buildah-chroot-exec.

Fixes

This release fixes the following issue:

• Previously, the agent returned a NumberFormatException error in the PodmanUtil service and failed to start most of the internal services on AIX systems. This error occurred because the UIDs on AIX exceeded the maximum integer range that was specified in the Agent 1.2.6 release. This issue is now fixed, and the agent parses long UIDs and starts up correctly.

Features

This agent release adds the following features:

• The agent now supports rootless Podman to identify if a process is started by the root or a user. To use this feature, make sure that Podman sockets are available for both root and user.
• The agent now provides an abstraction layer to access files for rootless Podman. The abstraction is required because the existing Java Nio-based file access has the same user as the agent and therefore cannot interact with the user file system.
• You can now configure the agent to send less data to the backend from the sensors that collect data at lower granular intervals. For example, process sensors and container sensors. Decreasing the amount of data that is sent to the backend reduces the counted traffic and the computation on the backend.
• The agent now generates lesser number of metrics on the backend to debug traces and logs. This reduces the load on the backend.

Fixes

This release fixes the following issues:

• Previously, the agent marked a process as not running after an update of the agent feature. This was because the configuration service that is used to determine whether a process must be ignored is not ready at startup time. This is now fixed and the agent awaits the configuration service before deciding whether a process is ignored.

Fixes

This release fixes the following issues:

• Previously, the agent failed to run a docker exec command with a user switch. This was because the agent added the -u flag after containerId, which is invalid. This issue was noticed in the Agent 1.2.3 release. This issue is now fixed.
• Previously, the agent created a jvm_instrumentation_shared_absent monitoring issue when a JVM was absent or when the connection to the JVM was already closed. This issue is now fixed and the agent creates the jvm_instrumentation_shared_absent monitoring issue when the JVM is still active.

Fixes

This release fixes the following issues:

• Previously, the agent became unresponsive if multiple restart requests were raised at the same time. This issue is now fixed and the agent ignores incoming restart requests in the first minute.
• Previously, the agent did not check whether the instrumentation-shared files exist. If the files were absent, the java-tracer did not start and a NoSuchFileException was added to the logs. This issue is now fixed and the agent checks if the instrumentation-shared file is present. The agent also creates a monitoring issue with the name jvm_instrumentation_shared_absent if the instrumentation-shared is absent.
• Previously, the agent generated multiple logs when an update occurred while the agent was forwarding logs to the backend. This issue is now fixed and logs are not created when the agent is updated.

Features

This agent release adds the following features:

• The agent now collects the configurations that are shared with a sensor at startup time or as part of a configuration update. You can retrieve this data by clicking on the Collect Support Information button on the agent dashboard.
• The agent now supports JDK-based attachments along with Byte Buddy-based attachments. You can specify to use JDK-based attachments by setting the environment variable INSTANA_JVM_BYTEBUDDY_ATTACHMENT_ENABLED as false.

Fixes

This release fixes the following issue:

• Previously, the agent created a new string when comparing the inventory as part of the discovery check. This issue is now fixed and new strings are not created anymore.

Fixes

This release fixes the following issue:

• Previously, the vault configuration integration failed to start and became unusable when the agent bundle was updated after the vault was started. This issue is now fixed, and the vault configuration integration works after an update.

Bundle split

This release splits the agent bundle into multiple bundles. The splitting of bundles reduces the duration of agent downtimes during an update and ensures that the components that are not updated are run without interruption. This agent release contains the following bundles:

• agent 1.1.729: The main agent bundle.
• agent backend API 1.0.0: Services required for other agent bundles for backend connection.
• agent backend 1.0.0: The implementation of the backend connection.
• agent backend util 1.0.0: Utility bundle for backend-related components.
• agent internal API 1.0.8: Services required for other bundles that are provided by the agent bundle.
• agent Jaeger tracer 1.0.0: The Jaeger Tracer component.
• agent memory API 1.0.0: Services required for other bundles for memory access.
• agent memory 1.0.0: Implementation of the memory access.
• agent runtime API 1.0.0: Services required for other bundles of the agent that allow access to the runtime-related data.
• agent runtime 1.0.0: Implementation of the runtime-related data.
• agent self profiling 1.0.0: Self-profiling capability of the agent.
• agent socket server API 1.0.0: Services required for other bundles of the agent for the main agent socket server 42699. Mainly holds counter related information.
• agent socket server 1.0.0: The implementation of the main agent socket server that runs on port 42699.
• agent task orchestrator API 1.0.0: Services required for other bundles for task orchestration.
• agent task orchestrator 1.0.0: The implementation for the task orchestration. Holds the Scheduler and Executor services.
• agent util 1.0.0: Helper bundle with static helpers that are not part of any API.
• agent Zipkin tracer 1.0.0: The Zipkin Tracer component.

Features

This agent release adds the following features:

• Java Instrumentation now supports Java 23.
• The agent now supports a special environment variable INSTANA_BIND_ALL_INTERFACES. If this environment variable is set to true, the agent binds to all interfaces and announces the interfaces when the agent monitors Java workloads. This feature is required when the container uses Macvlan.
• The agent now tracks aborted requests on the 42699 endpoint for internal monitoring.

Fixes

This release fixes the following issues:

• Previously, an internal service of the agent would be absent if a temporary directory on the host was corrupted. Now, the agent catches this exception and the internal service remains intact.
• Previously, the agent would time out on the java -version check after 5 seconds. This timeout is now increased to 10 seconds.

Features

This agent release adds the following feature:

• The agent now runs on AS400/IBM i OS.

Fixes

This release fixes the following issues:

• Previously, the agent showed the agent-bundle version on the / path. This issue is now fixed, and the agent now displays the agent version.
• Previously, the agent ignored connections from JVMs if they were marked as unstable. These markings did not timeout. The timeout for the markings is now fixed, and the potential unstable connections are retried again after 5 minutes of the timeout.

Features

This agent release adds the following feature:

• An internal API is added to the host agent that enables the sensors to request for the process environment configuration that is already in discovery.

Fixes

This release fixes the following issue:

• Previously, J9-based JVMs were excluded from the new attachment that was added with Agent 1.1.721 because these JVMs required the stat command, which was not available on AIX. This issue is now fixed and the JVMs are included in the new attachment because the stat command is replaced with the istat command.

Fixes

This release fixes the following issues:

• Previously, the Host Agent added a dedicated Thread Pool for discovery requests. An edge case was identified when the Agent was receiving too many discovery requests on startup. In that case, the Host Agent might not handle any other Request and failed to respond to health checks. This overload might lead to restarts on Kubernetes. This issue is now fixed and the Host Agent is no longer restarted.
• On Solaris, the Host Agent checks if the <java-home>/jvm/jre/lib/sparcv9/jspawnhelper is executable. Now, the Host Agent notifies the user with a Monitoring event.
• Previously, the Host Agent showed the command that is used on the java -version process in an error case. As this command can display secrets, the used command is no longer shown.

Changes

• Previously, Opentelemetry monitoring was deactivated by default. The new default is activated. For more information, see Configuring OpenTelemetry data ingestion.

Fixes

This release fixes the following issue:

• Previously, the Host Agent verified incoming data on the connection and closed it if no traffic was observed within 600 seconds. An edge case was identified when the Agent loses connection to the Instana backend every 30 seconds. Closing the connection led to extra load within the JVM. This issue is now fixed and the Host Agent now checks the incoming data at 200, 400, and 600 seconds. If no data reaches the agent for 600 seconds, the connection is marked as unstable and closed.

Fixes

This release fixes the following issues:

• Previously, the Host Agent ignored a process after reading the content in /proc/<pid>/<something> had a timeout. The default timeout was 5 seconds, which got triggered mistakenly in Host Agents running on overloaded hosts. In such cases, the Agent mistakenly ignored the process and did not retry it. This issue is now fixed, and the Host Agent handles potentially broken processes with a staged approach. The Host Agent now supports multiple stages for a process. Successful reading of process data resets the counter. The list of the ignored processes is reset every 10 minutes.
• Previously, in some cases within the Host Agent, the flow expected a connection to be removed to run a full reattachment. This reattachment is only possible if the connection is removed and is not added back within a timeout of 20 seconds. This issue is now fixed, and the connection between the Host Agent and JVM is now removed, if the connection is marked as closed.
• Previously, a regression with J9 based JVMs occured that required the stat command, which was not available on AIX. This issue is now fixed and now the JVMs that run on AIX are excluded from the new attachment that is added with Agent 1.1.721. The new attachment is enabled again when stat command is replaced with istat command.

Fixes

This release fixes the following issues:

• An environment variable CONTAINERD_SUPPRESS_DEPRECATION_WARNINGS with the value true is added to the ctr exec operations. This environment variable is to exclude deprecation warnings that might interfere with parsers in the Agent.
• The evaluation of a process and the PID that reaches the 42699 port of the Agent is delayed. Computation is completed when the field is requested in code. When you make computation optional, the Agent executes other important tasks.

Fixes

This release fixes the following issues:

• Process with Arguments on Windows were ignored if they contained characters outside of the UTF-8 area.
• The Javaagent Loader inside a JVM leaks on Exit Hooks for file references of the Agent when the attachment works, but the communication to the Agent failed afterward
• When the agent attaches to a JVM, the Agent places files in <TMPDIR>/.instana with read and execute permissions and excludes the write permission for everyone.
• The Agent now parses the output of docker info when Docker is available to save the location of the Docker Root Dir. This information is used to correctly identify Docker containerIds.

Fixes

This release fixes the following issues:

• Better logging when java -version fails. The log now contains the command and the output that failed to be parsed.
• JVM attachments now always use the attach-tools-<version>.jar, which is placed in <TMPDIR>/.instana to allow attachments against JREs and 32-bit JVMs on Windows.
• The Agent monitors the Metaspace and issues a restart if the available space is under 10 MB for over 30 minutes. This restart reduces the risk of an OutOfMemoryError:Metaspace.

Fixes

This release fixes the following issue:

• JVM monitoring: Ignore SAP NetWeaver classes to mitigate class loading issues.

Fixes

This release fixes the following issues:

• JVM monitoring: Support for Virtual Threads.
• Reduction in Memory by deduplicating Strings on internal tasks.
• Vault is available to other Sensors by an internal Service.