Java agent tracing

The IBM® Z OMEGAMON® AI for JVM Java agent provides its own log facility, and writes message and trace data to stderr.

Overview

Note: A command prefix agent is provided that allows users on the enhanced 3270UI to request JVM dumps and to dynamically enable OMEGAMON AI for JVM agent tracing.

A trace utility option allows secured users on the enhanced 3270UI to gather JVM diagnostics, including Java core dump, heap dump, and SVC dump. Additionally, and to avoid recycling the monitored JVM job, the existing Java agent debug tracing is dynamically enabled.

During normal operation, the Java™ agent, which runs in a monitored Java virtual machine (JVM), writes only a few informational messages to the standard error file (stderr). By default, it also writes warning and error messages, when produced, to stderr.

If a problem is suspected in the Java agent or if more information is required to debug a scenario, IBM Software Support might ask that you enable more in-depth tracing against specific components in the Java agent. You designate trace specifications, as directed by IBM Software Support, in the KJJ_DEBUG Java system property.

The KJJ_DEBUG Java system property has the following format.
KJJ_DEBUG=unit_name:level
where
unit_name
Indicates the full module name or the prefix of modules that you want to trace. All Java agent modules begin with KJJ.
level
Indicates the level of tracing for the specified unit.
  • ERROR returns error messages only.
  • WARN returns warning and error messages.
  • INFO returns informational, warning, and error messages. INFO is the typical setting for the KJJ unit so that all Java agent messages are written to the log, which is the default behavior.
  • STATE records INFO-level data, plus the condition of flags and variables in the process and the results (for example, return code) of a method call or results of a process.
  • FLOW records STATE-level traces, plus method entry and exit points.
  • DETAIL records FLOW-level traces, plus detailed tracing information.
You can specify a maximum of 8 comma-separated unit_name:level pairs. In addition to setting one or more specific traces, also specify KJJ:INFO as the last pair in the list to ensure that all informational, warning, and error messages are logged during your debugging session. If KJJ unit debug level is not specified, KJJ:INFO is automatically added to the end of the list, and it is counted as one of the limited 8 comma-separated unit_name:level pairs.
The way that you set KJJ_DEBUG, such as in an options file or on the java command line, depends on the JVM environment. The following example illustrates using the -D option of the java command to set KJJ_DEBUG as a Java system property. In the example, flow-level tracing is set for module KJJ_GCListener, and informational, warning, and error messages are logged for all Java agent components.
-DKJJ_DEBUG=KJJ_GCListener:FLOW,KJJ:INFO
Tip: The stderr log can grow very large if the debug unit specification is too broad. Be mindful with the levels of tracing that you specify. Make sure that you restore the overall level to KJJ:INFO after you complete problem diagnosis by removing the KJJ_DEBUG Java system property and then, recycle the JVM.
Tip: If any error is found in KJJ_DEBUG specification, default KJJ:INFO debugging level will be used.
Notes on security:

  • Use of the OMEGAMON AI for JVM diagnostic utilities, including the dynamic trace command, are protected by Take Action profiles defined to the System Authorization Facility (for example RACF). Refer to the OMEGAMON shared documentation at https://www.ibm.com/docs/en/om-shared?topic=esoe3ui-define-take-action-profiles-control-access-data-actions.

  • The profile used to protect OMEGAMON AI for JVM Take Action commands is KJJ.msn.TAKEACTION, where msn is the managed system name that the OMEGAMON AI for JVM TEMA runs on. In most circumstances, a generic profile name KJJ.**.TAKEACTION may be used.

Enabling Java agent tracing on a running JVM using the trace utility option in the enhanced 3270UI

  1. From the Health Summary workspace of the enhanced 3270UI, open the Options menu and select option U (JVM Utilities). The Utility Selection panel is displayed.
  2. From the Utility Selection panel, select T (OMEGAMON Java Agent Trace). The OMEGAMON Java Agent Trace pop-up is displayed.
  3. Specify trace options. The trace options are specified the same way as the environment variable KJJ_DEBUG is defined, using unit_name:level pairs. Similarly, up to eight different trace options can be specified, separated by commas.
    unit_name
    Indicates the full module name or the prefix of modules that you want to trace. All Java agent modules begin with KJJ. There are different trace units for the Java agent, the JVMTI agent and the z/OS Connect interceptor. These will normally be provided to you by IBM Software Support.
    level
    Indicates the level of tracing for the specified unit.
    • ERROR returns error messages only.
    • WARN returns warning and error messages.
    • INFO returns informational, warning, and error messages. INFO is the typical setting for the KJJ unit so that all Java agent messages are written to the log, which is the default behavior.
    • STATE records INFO-level data, plus the condition of flags and variables in the process and the results (for example, return code) of a method call or results of a process.
    • FLOW records STATE-level traces, plus method entry and exit points.
    • DETAIL records FLOW-level traces, plus detailed tracing information.
  4. Press Enter when you are finished specifying trace options. When the trace command is submitted, it is immediately activated. You do not need to stop and restart the JVM.
    Note:

    Once you have reproduced the problem to be diagnosed, the trace must be disabled by returning it to the default level using trace option:

    KJJ:INFO