Resolving The Problem

The QIBM_XJ9_TRACE environment variable can be set at the *SYS or *JOB level to gather detailed trace information about the IBM Technology for Java JDK when experiencing issues, especially startup problems.

Valid trace option values include:

* ALL : All common trace points (not ALL!! Options in red not included)
* HOME : Home library and directory for J9
* JNI : Ile side JNI functions
* PASEJNI : Pase side JNI functions
* MIPTR : MIPtr class operations
* LOADLIB : load ILE library
* RSLVNAT : resolve ILE native method
* REGNATIV : RegisterNative and UnregisterNative
* SIGCONV : native method signiture convert
* ILECALL : call return to/from Ile native method
* FINDPASE : Find Pase EntryPoint
* HOOK : Trace JVM hook points
* COMPAT : Classic JVM Compatability Options
* RSLVPASENAT : resolve PASE native methods
* CALLPASECNT : counts the number of calls to pase via Qxj9PaseAccess
* JOB : Job Name
* PASECRT : Pase Creation Status
* ILEENV : Ile envvars before call to create PASE
* PASEENV : Pase envvars during JNI_CreateJavaVM
* JVMCRTOPT : JNI_CreateJavaVM options
* INCRTOPT : All options (and their sources)
* CRTOPTMOD : All modified options
* ILEREF : All Ile Object Ref conversions for 64 bit
* LOCREF : Ile Local Object Ref conversions for 64 bit
* GLOBREF : Ile Global Object Ref conversions for 64 bit
* WEAKREF : Ile Weak Global Object Ref converts for 64 bit
* DUMPLOCR : Dump Ile Local Object Ref tables for 64 bit
* DUMPGLOBR : Dump Ile Global and Weak Global Object Ref tables for 64 bit
* FLDID : Ile FieldId for 64 bit
* MTHID : Ile MethodId for 64 bit
* DUMPFLDID : Dump Ile FieldId tables for 64 bit
* DUMPMTHID : Dump Ile MethodId tables for 64 bit
* CLLDR : Dump Ile Class Loader
* DUMPCLLDR : Dump Ile Class Loader tables
* THREADHEAP : Thread Heap
* DIAGMSG : Diag messages sent from PASE
* JCLNTV : JCL natives implemented in ILE (in many cases, only failures are traced)
* JCLSPAWN : spawn()'s from the JCL (Runtime.exec)

The QIBM_XJ9_TRACE_FILE environment variable specifies the output destination file path for trace information turned on by the QIBM_XJ9_TRACE environment variable. IBM recommends the QIBM_XJ9_TRACE_FILE environment variable be set in addition to QIBM_XJ9_TRACE in order to customize where the trace data is logged.

Valid values for the QIBM_XJ9_TRACE_FILE variable are:

* STDOUT: Default behavior. Output goes to standard out.
* STDERR: Output goes to standard error.
* <filename>: Output is directed to the specified file in IFS.

Setting the QIBM_XJ9_TRACE environment variable system-wide turns on tracing capabilities for all JVMs on the system. IBM recommends the QIBM_XJ9_TRACE environment variable be set at the job scope if possible by changing the LEVEL parameter to *JOB on the ADDENVVAR command. The ADDENVVAR command would then have to be executed at the beginning of the program/job before the JVM is created. In some cases, it may be easier to set the QIBM_XJ9_TRACE at the *SYS level (system-wide) if you are unable to easily call the ADDENVVAR command in the job before the JVM is created.

Following are two commands that are typically used to enable the trace at the *JOB level:

ADDENVVAR ENVVAR(QIBM_XJ9_TRACE) VALUE(ALL) LEVEL(*JOB)
ADDENVVAR ENVVAR(QIBM_XJ9_TRACE_FILE) VALUE('/tmp/J9Trace-%PID.txt') LEVEL(*JOB)

Following are two commands that are typically used to enable the trace system-wide at the *SYS level:

ADDENVVAR ENVVAR(QIBM_XJ9_TRACE) VALUE(ALL) LEVEL(*SYS)
ADDENVVAR ENVVAR(QIBM_XJ9_TRACE_FILE) VALUE('/tmp/J9Trace-%PID.txt') LEVEL(*SYS)

Notes:
1. The `%PID` keyword, will cause each process to replace `PID` in the file name with its own process ID (PID). Use of `%PID` is highly recommended to ensure accurate data and isolate the trace data per JVM process.
2. Using (*SYS) on the ADDENVVAR command will cause all J9 JVMs that are started to pick them up. After the J9 JVM has started and the issue has been recreated, you can use the RMVENVVAR command to remove the QIBM_XJ9_TRACE and/or QIBM_XJ9_TRACE FILE environment variables.