Configuring Java sensor

After you install the Instana agent, the Java sensor and tracer is automatically set up. They provide metrics and distributed tracing without any additional configuration. However, you can customize the configuration as outlined in the following sections.

For more information about monitoring JVM, see Monitoring Java virtual machine (JVM).

Customizing the application name

The application name identifies the Java application in Instana. The application name can be configured in the following ways: automatic and manual.

Automatic naming

Typically, Instana picks the application name automatically from the following sources:

  • Application .jar file: Within the .jar file, the following standard attributes are available in META-INF/Manifest.MF:

    Implementation-Title=Demo App
    Implementation-Version=1.0.0
    

    By using build tools, you can set both attributes when you build the application. For more information, see the Oracle documentation.

  • Command line: If the name is unavailable, the command line is used to find the name in the same way Java tools display JVMs.

  • System properties: If no name is obtained from the .jar file, Instana inspects various system properties, for example -Dcatalina.home for Tomcat containers.

Manual naming

To customize the application name, use one of the following options:

  • Configure the name during JVM initialization: To explicitly set the application name, specify the -Dcom.instana.agent.jvm.name="My Custom Name" attribute when you start the JVM.

  • Use existing system property: To configure the agent to use an existing custom system property, specify a list of properties in the agent configuration.yaml file. The first existing property names the JVM:

com.instana.plugin.java:
  name-properties:
    - 'acme.app'
    - 'DEPLOYMENT'

Configuring custom JMX metrics

To configure which JMX metrics to monitor, edit the agent configuration.yaml file as shown in the following example:

com.instana.plugin.java:
  jmx:
    # JMX is NOT hot-reloaded and needs to be set before a JVM is discovered.
    # Supported attribute types are Number and Boolean
    # delta calculation is only supported for Number
    - object_name: 'java.lang:type=Compilation'
      metrics:
        - attributes: 'TotalCompilationTime'
          type: 'delta' # delta will report the change to the previous second
    - object_name: 'java.lang:type=ClassLoading'
      metrics:
        - attributes: 'LoadedClassCount'
          type: 'absolute' # absolute will report the value as-is

The preceding example specifies the following types of metrics:

  • delta: Used for values that increase over time, such as counters, where the relative change is considered interesting.
  • absolute: Used to show the actual value each time the metric is accessed.

The specified attribute must either be a Number (or one of its subclasses, such as Integer or Double) or a String representing a number.

Configuring JMX metrics with wildcard patterns

The object_name might contain wildcard patterns as specified in the Java ObjectName API documentation.

The following example shows the configuration with wildcard patterns:

com.instana.plugin.java:
  jmx:
    - object_name: 'kafka.*:type=*-metrics,client-id=*,*'
      metrics:
        - attributes: 'incoming-byte-rate'
          type: 'absolute'
        - attributes: 'incoming-byte-total'
          type: 'absolute'
        - attributes: 'outgoing-byte-rate'
          type: 'absolute'
        - attributes: 'outgoing-byte-total'
          type: 'absolute'

Viewing custom JMX metrics

On the JVM dashboard, the JMX metrics are displayed under the Custom JMX metrics.

Configuring Java Trace instrumentation

After you install the Instana host agent, the default (non-SDK) Java Trace instrumentation is enabled by default, but the OpenTracing instrumentation and Java Trace SDK are not enabled.

Disabling Java Trace instrumentation

By default, Java trace instrumentation is enabled for all the Java processes. You can disable it for all processes or for a single process.

Disabling for all Java processes

To disable the default Java Trace instrumentation for all processes, configure the agent configuration.yaml file as follows:

com.instana.plugin.javatrace:
   instrumentation:
     enabled: false

Restart the Java application to make sure that the instrumentation is disabled.

To explicitly enable Java Trace instrumentation, set enabled: false to true.

Disabling for a single Java process

Java trace instrumentation is enabled for all the Java processes by default.

To disable trace instrumentation for a Java process, use one of the following options:

  • Add the -Dinstana.java.tracer.enabled=false system property as a Java virtual machine (JVM) argument.
  • Add the **INSTANA_JAVA_TRACER_ENABLED=false **system environment variable.

The system property is prioritized over the system environment variable if both are provided.

Enabling OpenTracing instrumentation

To enable the OpenTracing instrumentation of Java Trace, configure the agent configuration.yaml file as follows:

com.instana.plugin.javatrace:
   instrumentation:
     opentracing: true

To disable the OpenTracing instrumentation, set opentracing: false to false.

Enabling Java Trace SDK

To enable the Java Trace SDK, configure the agent configuration.yaml file by providing a list of packages that need to be scanned for SDK annotation as follows. By default, no packages are scanned for SDK annotations.

com.instana.plugin.javatrace:
   instrumentation:
     sdk:
       packages:
         - 'com.instana.backend'
         - 'com.instana.frontend'

To disable the Java Trace SDK, comment the entire packages section.

Capturing intermediate spans

Instana can capture intermediate spans for Java applications in Kafka Streams and WebMethods Integration Server. However, this feature is disabled by default.

Enabling intermediate spans results in a substantial increase in the number of spans, multiplying the volume of trace data and potentially impacting the application performance.

To enable this feature, see the following sections:

Enabling intermediate spans for Kafka Streams

In Kafka Streams, every stream operator is captured as an intermediate span. This feature impacts the performance of the application if you have large data inflows. Do not use this feature in production environments.

To enable capturing of intermediate spans for Kafka Streams, add the following lines to your agent configuration.yaml file:

com.instana.plugin.javatrace:
   instrumentation:
    plugins:
     Kafka011StreamIntermediate: true

Enabling internal service calls for WebMethods Integration Server

You can enable the feature of capturing every service call in WebMethods Integration Server as an intermediate span. But if the server handles heavy workloads, this feature can impact the performance of your application.

To enable capturing internal service calls for WebMethods Integration Server, add the following lines to your agent configuration YAML file:

com.instana.plugin.javatrace:
   instrumentation:
     plugins:
       WebmethodsIntermediate: true

Disabling stack trace capture

Java Tracer captures stack traces for exit and intermediate spans. However, stack traces for entry spans are disabled by default. If an application has a deeper stack or a request results in several spans, capturing stack traces might impact application performance. If you are facing performance issues, you can disable stack trace to optimize performance.

To disable capturing of stack trace, start the JVM with the following property:

-Dinstana.tracing.stack.exit=-1 -Dinstana.tracing.stack.log=-1

The following example displays how to disable capturing of stack trace:

$ java -Dinstana.tracing.stack.exit=-1 -Dinstana.tracing.stack.log=-1 -jar application.jar

Configuring trace correlation

Instana trace correlation is disabled by default for technologies, such as IBM Db2, WebLogic T3, IBM MQ, and Kafka, which use Java Tracer for tracing. To enable trace correlation for these technologies, see the following sections:

Configuring IBM Db2 trace correlation

To configure IBM Db2 trace correlation, use one of the following configuration options:

  • Agent configuration: Configure the agent configuration.yaml file as follows:

      com.instana.tracing:
        db2:
          trace-correlation: true
          db2CorrelationMode: dont-override
          db2CorrelationDelimiter: '|'
          db2CorrelationField: ClientHostname
    
  • Environmental variable configuration: Set the following environmental variables:

     INSTANA_DB2_TRACE_CORRELATION=true
     INSTANA_DB2_TRACE_CORRELATION_MODE=dont-override
     INSTANA_DB2_TRACE_CORRELATION_DELIMITER='|'
     INSTANA_DB2_TRACE_CORRELATION_FIELD=ClientHostname
    
  • System property configuration: Set the following system properties in the instrumented JVM:

     -Djdbc.db2.correlation.enable=true
     -Djdbc.db2.correlation.mod=dont-override
     -Djdbc.db2.correlation.delimiter=|
     -Djdbc.db2.correlation.field=ClientHostname
    

By default, correlation (propagating TraceId and SpanId) is disabled. Propagation happens by using client information properties of Db2. The correlation field sets the field name as ClientHostname by default. The default delimiter is |. If the correlation mode is set to override (default), the correlation data overwrites any existing value. However, if it is not set to override, the correlation data is appended to the existing value.

Configuring WebLogic T3 trace correlation

Enterprise Java Bean (EJB) calls support WebLogic T3 correlation.

To configure WebLogics T3 trace correlation, use one of the following configuration options:

  • Agent configuration: Configure the agent configuration.yaml file as follows:

      # General Tracing Settings
      com.instana.tracing:
        weblogic:
          trace-correlation: true
    
  • Environmental variable configuration: Set the following environment variable:

    INSTANA_WEBLOGIC_EJB_TRACE_CORRELATION=true
    
  • System property configuration: Set the following system property in the instrumented JVM:

    -Dcom.instana.weblogic.ejb.trace.correlation=true
    

    This configuration works only if the T3 communication happens between WebLogic servers. However, if the WebLogic client is used outside the WebLogic server, correlation is not possible.

Enabling IBM MQ trace correlation

In some IBM MQ consumer clients, the presence of extra header data in IBM MQ messages can cause message processing errors and message rejection. Therefore, Instana Trace header correlation is not enabled by default for IBM MQ Java applications.

You must test IBM MQ trace correlation in a nonproduction environment before you enable it in a production environment. If you encounter errors on the IBM MQ client applications after you enable trace correlation, troubleshoot as follows:

  • If you can alter the application, update the application to ignore the additional IBM MQ header data that is added by Instana. For assistance to make IBM MQ client changes, contact IBM MQ support.
  • If you cannot alter the application, do not enable the IBM MQ trace correlation support.

To enable trace correlation for IBM MQ, use one of the following configuration options:

  • Agent configuration: Edit the agent configuration.yaml file as shown in the following example:

    # General Tracing Settings
    com.instana.tracing:
      ibm-mq:
        trace-correlation: true
        ibmmq-optin-queues-regexp: '^QueueName.*'
    

    Set the ibmmq-optin-queues-regexp regular expression value to filter the queue names for which you want to enable the trace correlation. The regular expression value is optional and if it is missing then the trace correlation applies to all queues.

    This configuration is picked up in any of the following scenarios:

    • When the JVMs starts
    • When the Java Tracer is updated
    • After the Instana agent is shut down for approximately 15 minutes

    It is not hot-reloaded at run time. A quick agent restart might not pick up this configuration change because this feature is optimized. A configuration change needs a full reinstrumentation.

  • System property configuration: Alternately, specify a system property on the application JVMs that contain IBM MQ clients as shown in the following example:

    -Dinstana.ibm.mq.trace.correlation.enable=true -Dcom.instana.ibm.mq.trace.optin.queues="^QueueName.*"
    

The IBM MQ sensor also provides a trace capability by using user exits (server-side tracing for IBM MQ). For more information about the IBM MQ sensor, see Monitoring IBM MQ.

Configuring Kafka trace correlation headers

To enable the header format for Kafka trace correlation headers that are used by the Java Tracer, use one of the following configuration options:

  • Agent configuration: Configure the agent configuration.yaml file. For more information, see the Instana host agent.

  • Environment variable configuration: Set the environment variable as shown in the following example:

    INSTANA_KAFKA_HEADER_FORMAT= both # possible values: binary, both, string
    

    Valid values are binary, string, or both.

  • System property configuration: Set the following property when you start the JVM:

    -Dcom.instana.kafka.trace.header.format=binary # possible values: binary, both, string
    

To disable Kafka Trace correlation entirely, use one of the following configuration options:

  • Agent configuration: See Instana host agent.
  • Environment variable configuration: Set the environment variable INSTANA_KAFKA_TRACE_CORRELATION=false, but the use of this option is not suggested.
  • System property configuration: Provide the option -Dcom.instana.kafka.trace.correlation=false when you start the JVM.

For more information, see Kafka Header Migration.

Configuring span batching

Span batching is a feature that merges multiple spans and sends them to the agent in batches, rather than individually. When span batching is enabled, spans with the same name, type, and duration (less than 10 ms) are aggregated. Typically, high-frequency and short spans, such as database calls, are batched together.

The following image shows the implementation of span batching. The dashboard highlights 5 sub calls, of which 3 are batched and listed in the Calls section.

Span batching
Figure 1. Span batching

Span batching is enabled by default. To explicitly enable span batching, set the INSTANA_SPANBATCHING_ENABLED parameter to true.

Span batching is enabled automatically for the following applications: Aerospike, Amazon s3, Cassandra, Data Grid, DynamoDB, Elasticsearch, FTP, Google Cloud Bigtable, Google Cloud storage, HBase, Hazelcast, IMS, JavaMail, Kinesis, LDAP, Memcached, MongoDB, Neo4j, and Redis.

For Data Grid and JDBC, specific span merge strategies are followed. For more information, see Span batching strategies.

Disabling span batching

If you disable span batching, the number of individual spans sent increases, resulting in a higher data volume.

To disable span batching, use any one of the following configuration options:

  • Agent configuration: Configure the agent configuration file as follows:

      com.instana.plugin.javatrace:
        instrumentation:
          spanBatchingEnabled: false
    
  • Environmental variable configuration: Set the environmental variable as follows:

    INSTANA_SPANBATCHING_ENABLED=false
    
  • System property configuration: Set the system property in the instrumented JVM as follows:

    -DspanBatchingEnabled=false
    

Span batching strategies for Data Grid and JDBC spans

Applications, such as Data Grid and JDBC, implement the following custom batching strategies for span batching:

  • Data Grid: Span are merged only if both datagrid.operation and datagrid.cacheName have the same value in sequential spans. Span batching is enabled only if this condition is met.

  • JDBC: The following types of span batching strategies are supported:

    • Default batching (enabled by default)

      Spans are merged regardless of the value of JDBC span attributes.

    • Custom batching (optional)

      Spans are merged only if the value of JDBC.statement is identical in sequential spans.

Enabling custom batching for JDBC spans

The size of the ingested data increases if you enable custom batching for JDBC spans.

To enable custom batching for JDBC spans, use any of the following configuration options:

  • Agent configuration: Configure the agent configuration file as follows:

      com.instana.plugin.javatrace:
        instrumentation:
          customJDBCSpanBatchingEnabled: true
    
  • Environmental variable configuration: Set the environmental variables as follows:

    INSTANA_CUSTOM_JDBC_SPANBATCHING_ENABLED=true
    
  • System property configuration: Set the system property in the instrumented JVM as follows:

    -DcustomJDBCSpanBatchingEnabled=true
    

Configuring logging levels

You can enable and disable the logging levels in Instana. Instana supports the following logging levels:

  • OFF
  • ERROR
  • WARN

When a logging level is set, all higher priority logging levels are included in tracing. For example, if ERROR is configured, the logging levels ERROR and FATAL are traced.

Enabling customer-defined logging level

To configure the logging level, use one of the following configuration options:

  • System property configuration: Set instana_log_level_capture to ERROR or WARN.
  • Environment variable configuration: Set INSTANA_LOG_LEVEL_CAPTURE to ERROR or WARN.

If you provide an invalid logging level (for example, INFO or TRACE), the following warning is logged in the agent logs and the default logging level WARN is used.

The value 'INFO' for 'INSTANA_LOG_LEVEL_CAPTURE' is not valid. Valid values are 'WARN', 'ERROR' and 'OFF'.

Disabling logging plug-ins

To disable logging for all logging plug-ins, you can use one of the following configuration options:

  • Agent configuration: Configure Java tracing to disable all logging plug-ins in the agent configuration.yaml file:

        com.instana.plugin.javatrace:
          instrumentation:
            plugins:
              Logging: false
    
  • System property configuration: Set instana_log_level_capture="OFF".

  • Environment variable configuration: Set INSTANA_LOG_LEVEL_CAPTURE="OFF".

Configuring Java Tracer for JVMs running incompatible agents

The Instana agent does not automatically monitor JVMs that also run any of the following agents:

  • AppDynamics
  • DataDog
  • Dynatrace
  • Glow Root
  • Kanela, the Kamon Instrumentation Agent
  • ManageEngine APM Insight Java Agent
  • New Relic
  • Oracle Java Flight Recorder
  • OpenTelemetry Instrumentation for Java
  • Riverbed AppInternals
  • Spring Loaded
  • Wily Introscope, also known as CA Application Performance Management and Broadcom DX Application Performance Management

To monitor JVMs that run any of these agents, configure the agent configuration.yaml file as follows:

com.instana.plugin.javatrace:
  trace-jvms-with-problematic-agents: true

Ignoring classes

To disable instrumentation of a particular class or classes in a package, you can use one of the following configuration options:

  • Agent configuration: Configure Java Trace sensor to ignore instrumentation of a class or classes in a package in the agent configuration.yaml file as follows:

        com.instana.plugin.javatrace:
          instrumentation:
             ignoredClass: "^ClassName or PackageName"
    

    For example, use the following configuration to disable instrumentation of all classes in the org.apache.derby package:

           com.instana.plugin.javatrace:
             instrumentation:
                ignoredClass: org.apache.derby.
    

    Restart the agent or application after editing the agent configuration.yaml file.

  • System property configuration: Set ignoredClass="^ClassName or PackageName" as shown in the following example:

    -DignoredClass=org.postgresql
    
  • Environment variable configuration: Set IGNORED_CLASS_NAME="^ClassName or PackageName".