Known issues and limitations
Review the known issues and limitations for IBM Instana observability.
Known issues
Incompatibility of CrowdStrike Falcon sidecar with the Instana agent
The Instana agent container is incompatible with the CrowdStrike Falcon sidecar. You can use Falcon only as a DaemonSet.
Memory issue with Instana agent on Windows Server 2016
On Windows Server 2016 hosts, a library used by the Instana agent gradually consumes memory without freeing it correctly, which might cause increased memory usage on the affected Windows systems. To keep memory consumption under control, the Instana agents are restarted automatically. The automatic restart feature is to prevent the memory issue from impacting automatic agent updates. All the Instana agents that are deployed on Windows Server 2016 restart every 7 days between 05:00 and 05:30 AM, by default. This feature is available since Instana agent bundle 1.1.702. The agent bundle version can be checked in the Instana agent startup logs.
To change the default restart behavior, set the following environment variables before you initiate the Instana agent.
| Environment Variable Name | Default | Note |
|---|---|---|
INSTANA_AUTO_RESTART_WINDOWS_SERVER_2016_ENABLED |
true |
The flag to enable or disable automatic restart. |
INSTANA_AUTO_RESTART_WINDOWS_SERVER_2016_RESTART_TIME |
5:15 |
The time at which the Instana agent restarts. |
INSTANA_AUTO_RESTART_WINDOWS_SERVER_2016_JITTER_IN_MIN |
30 |
The jitter in minutes to set the time range in which the agent restarts. The jitter is to prevent the Instana agents from restarting simultaneously. The restart time is randomly selected within the specified range. To modify the randomization of restart time, set a value greater than 1. The time range for the restart time is calculated by using the formula INSTANA_AUTO_RESTART_WINDOWS_SERVER_2016_RESTART_TIME - (INSTANA_AUTO_RESTART_WINDOWS_SERVER_2016_JITTER_IN_MIN/2) and INSTANA_AUTO_RESTART_WINDOWS_SERVER_2016_RESTART_TIME + (INSTANA_AUTO_RESTART_WINDOWS_SERVER_2016_JITTER_IN_MIN/2). For example, if the restart time is set to 5.15 AM with a jitter of 30 minutes, the restart occurs between 05:00 and 05:30 AM. |
INSTANA_AUTO_RESTART_WINDOWS_SERVER_2016_RESTART_MIN_INTERVAL_IN_DAYS |
7 |
The interval before the next restart takes place. Set a value greater than zero. |
Agent ID conflicts due to identical MAC addresses
When multiple systems share the same MAC address, they generate identical agent IDs, which causes host identity conflicts. This results in agents overwriting each other’s data, and process from one host might be incorrectly attributed to another. To resolve this issue, you can configure the agent to include the FQDN in the agent ID by setting the environment variable INSTANA_APPEND_FQDN_TO_AGENT_ID=true. This configuration ensures that each agent ID is unique regardless of identical MAC addresses.
CRI-API connection failure on older Kubernetes and containerd setups
In Kubernetes environments running older or customized versions of RKE2 and Containerd, the monitoring agent might fail to establish communication with the container runtime through the Container Runtime Interface (CRI) gRPC API.
Example versions:
- Container Runtime Version: containerd://1.4.12-k3s1
- Kubelet Version: v1.21.8+rke2r1
This failure manifests as errors similar to:
io.grpc.StatusRuntimeException: UNIMPLEMENTED: unknown service runtime.v1.RuntimeService
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268) ~[?:?]
This issue occurs because older versions of Kubernetes and Containerd might not fully support the CRI gRPC API expected by the agent, causing connection failures during interaction with the container runtime.
As a workaround, set the environment variable INSTANA_USE_CONTAINERD_CRI_CLIENT=false on the agent to disable interaction with the container runtime through the CRI client. The agent then uses ctr to collect the metrics.
Known limitations
Processor group awareness on Windows
On Windows Servers between 2008 and 2019 that run more than 64 cores, the Instana agent can monitor only the cores within the processor group that is assigned to it. This limitation occurs because the current version of the agent is not process group-aware, which means it cannot recognize the cores that are not part of the processor group.
Analyzing traces and calls
When calls are grouped by log.level or log.message, the special tag group Tag not present is not displayed in the same way as other tags.
For more information about this feature, see Analyzing traces and calls.
Filtering with Dynamic Focus
For more information about this feature, see Filtering with Dynamic Focus
Syntax constraints
- White space is not allowed between the
!and the query to be negated. Use!<query>orNOT <query>instead. - When multiple of the same
entity.application.name,entity.service.name,entity.endpoint.nameDynamic Focus keywords are used together, they can only be joined by theORoperator. E.g. the queriesentity.application.name:foo AND entity.application.name:bar,entity.application.name:foo entity.application.name:barare not supported. The queryentity.application.name:foo OR entity.application.name:baris supported. - When multiple of different
entity.application.name,entity.service.name,entity.endpoint.nameDynamic Focus keywords are used together, they can only be joined by theANDoperator. E.g. the queryentity.application.name:foo OR entity.service.name:baris not supported, whereas the queryentity.application.name:foo AND entity.service.name:baris supported. - Queries containing a negation of application, service or endpoint name are not supported. Consequently, neither
NOT entity.application.name:foo,!entity.service.name:barnorNOT entity.endpoint.name:foobarare valid. However, filtering out events of an application, service or endpoint is still possible by usingNOT event.entity.label:fooinstead. - Dynamic Focus queries
entity.application.name:<term>,entity.service.name:<term>andentity.endpoint.name:<term>are evaluated using the stringcontainslogic, e.g. Dynamic Focus Queryentity.application.name:shopwould also matchentity.application.name:shop-frontend. - Dynamic Focus query keywords
entity.application.name,entity.service.nameandentity.endpoint.nameare only supported in boolean queries (see boolean logic constraints above), term queries, phrase queries and prefix queries.
Query limitations
- Dynamic Focus query on
event.textcontaining hyphens might return unexpected results. For e.g.event.text:"foo-bar-random"would also return events containing justfooorbarorrandomdue to tokenization of the Lucene analyzer. - Entities as follows a Docker container, such as Host or Availability Zone, cannot be selected when scoping on most Kubernetes keywords. As an example, the query
entity.kubernetes.deployment.name:my-K8s-deployment AND entity.selfType:hostdoes not return any host. In contrast, the queryentity.kubernetes.deployment.name:my-K8s-deployment AND entity.selfType:dockerreturns all Docker containers related to the deployment. However, there are a few Kubernetes keywords that can be used to select Host entites, nameleyentity.kubernetes.cluster.*andentity.kubernetes.node.*. For instance, the queryentity.kubernetes.cluster.label:my-K8s-cluster AND entity.selfType:hostreturns all related hosts of the cluster.
Infrastructure metrics
- Entities are limited to show at most 3000 metrics in the Instana UI for performance reasons. Dashboards with more metrics than that limit will have missing metrics.
- Negative metric values are not displayed.
Monitoring IBM App Connect Enterprise (ACE)
- If HTTPS is enabled for REST API interface, you need to specify keystore and keystorePassword parameters in
<agent_install_dir>/etc/instana/configuration.yaml. For keystore type, only JKS or P12 is supported. - The ACE user exit tracing method supports only single-node setups. High Availability (HA) setups and clustered environments are not supported. For HA deployments or clustered environments, use native OpenTelemetry tracing support instead of the user exit method.
For more information about this feature, see Monitoring IBM App Connect Enterprise (ACE)
Monitoring IBM API Connect application
- DataPower Tracing supports only the API gateway and doesn't support other services such as the multiprotocol gateway and web service proxy.
For more information about this feature, see Monitoring IBM API Connect application
Jaeger tracing
-
The tracing data collected from Jaeger will not be correlated with the tracing data collected via AutoTrace, resulting in separate traces even if the systems traced by Jaeger and Instana AutoTrace, respectively, were directly interacting with one another.
-
Since the Jaeger tracing data does not convey a notion of which process is sending them to the host agent, Instana correlates those traces to the host underpinning the host agent. This prevents the association with the process, and therefore also container and platform hierarchy (for example, Kubernetes pod, namespace, and cluster).
-
Jaeger has no notion of user monitoring (although this may eventually change with the adoption of W3C TraceContext.md); as such, beacons collected with Instana Website Monitoring will not be correlated with backend traces collected from Jaeger.
-
The host agent supports the collection of Jaeger traces only over HTTP. UDP, which is the protocol used if you configure the
JAEGER_AGENT_HOSTandJAEGER_AGENT_PORTenvironment variables, is not supported.
For more information about this feature, see Jaeger
OpenTelemetry tracing
- OpenTelemetry
linksare not supported. - Mixing Instana default context propagator with context propagators other than
W3C Trace Contextis not supported.W3C Trace Contextis the default context propagator of OpenTelemetry.
For more information about this feature, see OpenTelemetry Tracing
IBM MQ Tracing
A known issue on the Windows platform prevents the use of the destination queue monitor levels (IBMMQ_DEST_MONITOR_LEVEL_*) and the global monitor level (MONITOR_LEVEL) together.
On the AIX platform, the OTLP gRPC protocol (activated when you set OTLP_EXPORTER_GRPC_ENDPOINT) is unavailable, so HTTP is used by default.
Zipkin tracing
-
The tracing data collected from Zipkin will not be correlated with the tracing data collected via AutoTrace, resulting in separate traces even if the systems traced by Zipkin and Instana AutoTrace, respectively, were directly interacting with one another.
-
Zipkin has no notion of user monitoring (although this may eventually change with the adoption of W3C TraceContext.md); as such, beacons collected with Instana Website Monitoring will not be correlated with backend traces collected from Zipkin.
-
The host agent supports the collection of Zipkin traces only over HTTP, which corresponds to the
COLLECTOR_HTTP_ENABLEDsetting of Zipkin.
For more information about this feature, see Zipkin
Monitoring Containerd
The CPU Total Normalized and Memory Working Set Usage % metrics are available only on systems that are using Control Group v1 (cgroupv1).
Monitoring Amazon MSK
The Amazon MSK sensor supports monitoring only for the provisioned MSK cluster type. The serverless cluster type is not supported.
ALPN support
The Instana agent and backend support the Application-Layer Protocol Negotiation (ALPN) connection when the following requirements are met:
- Netty 4.1.49.Final or later
- Java Development Kit (JDK) 1.8.0_251 or later
Addressing high CPU usage
High CPU usage is observed in hotspot-based agents with agent container 1.263.3 and earlier. To resolve this issue, reinstall the agent. The reserved code cache size is now increased to support high CPU usage. For the agents that run on a host environment, remove the agent and then reinstall it.
If you are using Kubernetes (k8s) or OpenShift Container Platform (OCP), complete the following steps:
-
Uninstall agent:
kubectl delete agents.instana.io instana-agent -n instana-agent --wait; helm uninstall instana-agent -n instana-agent; kubectl delete crd agents.instana.io; kubectl delete namespace instana-agent -
Install agent:
To install the Instana agent, use Helm Chart on Kubernetes or OpenShift Container Platform.
Synthetic test attributes
Jest as a scriptType is not supported for Browser Simple, API Script, and API Simple tests on UI and API. Jest as a scriptType can be used only with Browser Script tests.
Some Synthetic test attributes are not supported in the Instana UI. To set these attributes, use the Open API or the synctl tool. The unsupported attributes in the Instana UI are listed in the following table:
| Test attribute | Browser Script | Browser Simple | API Simple | API Script |
|---|---|---|---|---|
| browser | ☒ | ☒ | N/A | N/A |
| expectExists | N/A | N/A | ☒ | N/A |
| expectNotEmpty | N/A | N/A | ☒ | N/A |
| ScriptType | ☒ | N/A | N/A | ☒ |
| HEAD (Operation) | N/A | N/A | ☒ | N/A |
| DELETE (Operation) | N/A | N/A | ☒ | N/A |
| OPTIONS (Operation) | N/A | N/A | ☒ | N/A |
| PATCH (Operation) | N/A | N/A | ☒ | N/A |
| POST (Operation) | N/A | N/A | ☒ | N/A |
| PUT (Operation) | N/A | N/A | ☒ | N/A |
The symbol ☒ indicates that the attribute is not supported in the Instana UI.
N/A indicates that the attribute is not applicable for the test type.
Business processes
A known limitation in certain versions of Business Automation Workflow (BAW) that run on the IBM J9 JVM can affect business process capabilities. This issue can result in some or all processes becoming invisible or inaccessible within the system.
Business perspectives
The business perspectives tab is disabled if Instana does not detect any connected agents. To view your business perspectives, make sure that you have at least one agent connected.
Canvas rendering
Certain canvases may not render correctly in the Instana UI when using Safari. This includes:
- The Business processes flow map
- The Infrastructure graph
- The Event probable root cause topology
- The OTel Collector configuration editor graph
Use other browsers, such as Chrome or Firefox, for better rendering.