Troubleshooting data for Java Virtual Machine (JVM) server problems

CICS® MustGather for Java™ Virtual Machine (JVM) server problems

The PERFORM JVMSERVER(jvmserver) JVM GATHER DIAGNOSTICS command (the GATHER SPI) can help to automate the collection of much of the following JVM data. If you choose to use it, some steps can be entirely omitted, and some can be partially omitted. When available, this option is noted at the beginning of the relevant step.

Gather the following diagnostic information before contacting the CICS support team to troubleshoot your JVM server problems.

Required data:
  1. The CICS job log that is sent in tersed binary. This contains various CICS messages that are relevant to the JVM server and any entries in the SYSOUT and SYSnnnn DD cards.

  2. Skip this step if you choose to use the GATHER SPI.

    The JVM profile contents. Ensure that you identify the JVM profile correctly. The JVMSERVER resource defines the file name and the location of the JVM profile. You can use the EXEC CICS INQUIRE JVMSERVER or the CEMT INQUIRE JVMSERVER command to inquire on the JVMSERVER resource - the JVM profile is returned by the PROFILE attribute.

    To manually find the file name of the JVM profile:

    Look at the JVMPROFILE attribute in the JVMSERVER definition - the file name has an extension of .jvmprofile. For more information, see JVMSERVER attributes.

    To locate the directory that contains the JVM profile:
    • For a JVM server that is defined using the CSD or BAS, the directory is the one specified by the JVMPROFILEDIR SIT parameter for the region. For more information, see JVMPROFILEDIR.
    • For a JVM server that is defined in a CICS bundle, this is the CICS bundle subdirectory where the JVM profile is stored.
  3. Skip this step if you choose to use the GATHER SPI.

    The STDERR, STDOUT, JVMTRACE and JVMLOG files. You can use the EXEC CICS INQUIRE JVMSERVER STDERR/STDOUT/TRACE/LOG or the CEMT INQUIRE JVMSERVER STDERR/STDOUT/TRACE/LOG command to inquire on the JVMSERVER resource - the required paths are returned by the attributes of the same name.

    To locate these files manually, refer to the JVM server options in the JVM profile to determine their location. For more information, see JVM server options. By default, they are located in the WORK_DIR/<applid>/<jvmserver> directory on zFS, where WORK_DIR is specified in your JVM profile. The STDERR file contains the Java stack trace when a Java exception occurs. The STDOUT file contains user application written information. TheJVMTRACE file contains any CICS trace information that is produced by the JVM server and the JVMLOG file contains important messages written by the JVMSERVER.

  4. JVM trace that is written to the CICS internal trace and is included in the MVS™ system dump when tracing is active. Ensure that the internal trace table size is big enough to contain sufficient data for diagnosis; for example, you can use a table size of 20480K. Turn on level 1-2 tracing for the AP and EI components, and level to 1-5 tracing for the SJ component. Levels 1-2 write the internal Java domain trace, while levels 3,4 and 5 are for information, debug and deep debug levels respectively and are written to the zFS JVMTRACE file. For more information, see Using trace for CICS problem determination on z/OS®.
    Note: Besides the trace that is written to the internal trace table, trace entries for each JVM server are written to a unique file on zFS. For more information, see Activating and managing tracing for JVM servers.
  5. Collect any Java and LE dumps that have been generated as a result of failures in the JVM server. Additionally you may pre-emptively generate Java dumps if you suspect there are JVM problems. These can be generated either by sending a UNIX QUIT signal to the JVM, having first specified the desired dump types using the -Xdump option in the JVM profile, or by using the PERFORM JVMSERVER DUMP command.
  6. If you have hang or loop problems with CICS and Java, take an MVS system dump of the CICS region and OMVS ASID. For more information, see Using dumps for CICS problem determination on z/OS. Use the following MVS command to capture the dump:
    DUMP COMM=(dumpname)
R yy,JOBNAME=(cicsjob),CONT
R yy,DSPNAME=('OMVS'.*),ASID=(omvs),CONT
R yy,SDATA=(PSA,CSA,LPA,LSQA,RGN,SQA,SUM,SWA,TRT,ALLNUC,GRSQ)

    Where:

    dumpname
    Specifies the name you want to give the dump.
    yy
    Specifies the reply identifier.
    cicsjob
    Specifies the job name of your CICS region.
    omvs
    Specifies the ASID of OMVS. To see your OMVS ASID in the BPXO042I message, enter the MVS command D OMVS. For more information, see BPX messages in z/OS MVS System Messages.

  7. Skip this step if you choose to use the GATHER SPI.

    If an abend occurs because of a failure in the JVM or in the Language Environment® (LE) enclave that the JVM runs, an MVS system dump of CICS taken at the point of the abend is of limited use. In this case, the MVS system dump is unlikely to contain the JVM storage. This is because the LE enclave might have terminated by the time CICS notices the failure and takes the dump. However, it is useful to provide any of the accompanying JAVADUMP, SYSDUMP (TDUMP), and CEEDUMP dumps with the MVS system dump.

    You can control the behavior of JAVADUMP, SYSDUMP, and CEEDUMP dumps by specifying the -Xdump option in your JVM profile. For more information, see -Xdump. For example, to collect information on a java.lang.OutOfMemoryError exception, set the -Xdump option as follows:
    -Xdump:java+heap+system:events=systhrow,filter=java/lang/OutOfMemoryError
    1. JAVADUMP is a formatted dump that is taken by the JVM. The dump summarizes the internal state of the JVM when it crashes unexpectedly. The output file is located in the directory named by the WORK_DIR option in the JVM profile.
    2. A SYSDUMP is produced by the JVM using IEATDUMP and is a dump of the system areas and the program’s address spaces. The following example shows messages that are written to STDERR when the JVM requests a SYSDUMP.
      JVMDG217: Dump Handler is Processing Signal 4 - Please Wait.
JVMHP002: JVM requesting System Transaction Dump
JVMHP012: System Transaction Dump written to PLOWMAN.JVM.TDUMP.COMKZCES.D050419.T231519
JVMDG303: JVM Requesting Java core file
JVMDG304: Java core file written to
/u/plowman/cics630/workdir/JAVADUMP.20050419.231606.83886118.txt
JVMDG215: Dump Handler has Processed Exception Signal 4.

      You can use MVS Interactive Problem Control System (IPCS) to format the dump. For more information, see Using IPCS to format and analyze CICS dumps: Overview.

      To override the default dump name, use the environment variable JAVA_DUMP_TDUMP_PATTERN in the JVM profile. For more information, see APAR II13292 and diagnostics information in IBM SDK, Java Technology Edition product documentation.

    3. CEEDUMP is a formatted application level dump that is taken by the LE service CEE3DMP after any SYSDUMP processing. The following example shows the LE traceback information in CEEDUMP. For more information, see Using trace for CICS problem determination on z/OS.
      TRACEBACK:
DSA ADDR PROGRAM UNIT PU ADDR PU OFFSET ENTRY E ADDR E OFFSET STATEMENT LOAD MOD SERVICE STATUS
2E63E178 /u/sovbld/hm131s/hm131s-20020723/src/hpi/pfm/threads_utils.c
2EB098A0 +000006AA ThreadUtils_CoreDump
2EB098A0 +000006AA 1662 *PATHNAM h020723 CALL
2E63E068 /u/sovbld/hm131s/hm131s-20020723/src/hpi/pfm/interrupt_md.c
2EAEC108 +000004B0 userSignalHandler
2EAEC108 +000004B0 376 *PATHNAM h020723 CALL
2E63DFB8 /u/sovbld/hm131s/hm131s-20020723/src/hpi/pfm/interrupt_md.c
2EAEC670 +000000B6 intrDispatch 2EAEC670 +000000B6 642 *PATHNAM h020723 CALL
2E63DF00 2DED9938 +0000001A @@GETFN 2DED9890 +000000C2 CEEEV003 CALL
2E63DE80 CEEVASTK 2DCCC420 +00000126 CEEVASTK 2DCCC420 +00000126 CEEPLPKA CALL
2E32BCC0 2DFF2B80 +00000736 __zerros 2DFF2B80 +00000736 CEEEV003 DRIVER9 CALL
2E329128 CEEHDSP 2DC16848 +00000C0C CEEHDSP 2DC16848 +00000C0C CEEPLPKA CALL
2E3286A8 /u/sovbld/hm131s/hm131s-20020723/src/zip/sov/ZipEntry.c
31F6EB38 +00000252 Java_java_util_zip_ZipEntry_initFields
31F6EB38 +00000252 99 *PATHNAM h020723
EXCEPTION
2E3285C8 java/util/zip/ZipEntry.java
32CEE95C +000000E4 java/util/zip/ZipEntry.initFields(J)V
32CEE95C +000000E4 0 CALL
  8. The Java version that you run - you can collect this from numerous places including a Java core dump, the messages.log output or the CICS output.

    See CICS support for application programming languages for a list of CICS releases and supported Java versions.

    Go to Service for Java on z/OS for a list of service summaries for different Java version. The summaries contain tables that list the SDK service levels and their corresponding PTFs. Install the latest service level applicable to your CICS TS to take advantage of improved service, performance, and stability.

Optional data:
  • The output from specifying the PRINT_JVM_OPTIONS=YES and PRINT_PROFILE=YES options in the JVM profile. When these options are specified, all the options that are passed to the JVM at startup, including the contents of the class paths, are printed to SYSPRINT.