Collect troubleshooting data for pooled JVM (Java) problems in CICS TS for z/OS

Resolving The Problem

For the most up-to-date documentation, refer to Collecting CICS troubleshooting data (MustGather) for IBM Support in the CICS TS documentation. You will find the General information you should gather for every type of problem followed by the Component-specific information which includes Troubleshooting data for pooled JVM (Java) problems.

The following is the original MustGather information:

• Collecting general information for all problems
• Collecting data for pooled JVM problems
• Troubleshooting hints and tips
• Exchanging data with IBM Support


• Collect troubleshooting data for CICS products - Read first

Collecting general information for all problems

If you have already contacted IBM Support, you can continue on to the component-specific MustGather information. Otherwise, go to Collect troubleshooting data for CICS products - Read first.

Back to top

Collecting data for pooled JVM problems

It is highly recommended that you run the "Hello World" and "Hello CICS World" sample programs to verify that Java has been installed and set up correctly in your CICS region. Afterwards, gather the following documentation for your Java problem and call IBM support:

Required (MustGather) data:

• CICS message log containing DFHSJxxxx, DFHIIxxxx, and DFHEJxxxx messages. Alternatively, cut and paste the messages into your problem record.
  
  
• The JVM profile and the properties file (if applicable). Properties files are optional with CICS TS V3.2 and higher.
  
  
• If using the Shared Class Cache, profile DFHJVMCD. Shared Class Cache can be used with CICS TS V4.1 and higher, or when running Java V1.5 with CICS TS V3.2.
  
  
• The output from specifying the PRINT_JVM_OPTIONS=YES option in the JVM profile. When this option is specified, all the options passed to the JVM at startup, including the contents of the class paths, are printed to SYSPRINT. The output is written every time a JVM is started, so you should add the option to the appropriate JVM profile, wait for a JVM to be started with the profile (or enter the PERFORM JVMPOOL command to manually start a JVM with the profile), and then immediately remove the option from the profile.
  
  
• If running CICS TS V3.1 or below, Event log containing all of the property settings and details for unresettable events. CICS creates the event log when a JVM initializes in the directory defined by the WORK_DIR option in the JVM profile. Ensure that option ibm.jvm.events.output is set in the system properties file referenced by its JVM profile. For more details, see section 8.3.3 Logging events in the JVM of the IBM Redbooks publication Java Application Development for CICS.
  
  If the problem is re-creatable and you have unresettable events, set the following properties to request the maximum information for unresettable events:
  ibm.jvm.events.output=event.log
  ibm.jvm.unresettable.events.level=max
  ibm.jvm.resettrace.events
  ibm.jvm.crossheap.events
  
  Note that this does not apply to versions of CICS TS after V3.1 because the resettable mode of the JVM is no longer supported.
  
  
• STDERR file containing the Java stack trace when there is a Java exception. Output from Java applications is written to an HFS file that is named by the STDERR option in the JVM profile for the JVM. The output file is located in the directory named by the WORK_DIR option in the JVM profile.
  
  
• Hang or loop problems with CICS and Java require an MVS system dump of the CICS region and OMVS ASID. Use the following MVS command to capture the dump:
  DUMP COMM=(dumpname)
   R yy,JOBNAME=( cicsjob),CONT
   R yy,DSPNAME=('OMVS'.*),ASID=(omvs),CONT
   R yy,SDATA=(PSA,CSA,LPA,LSQA,RGN,SQA,SUM,SWA,TRT,
               ALLNUC,GRSQ)
  where dumpname is the name you want to give the dump, yy is the reply id, cicsjob is the job name of your CICS region, and omvs is the ASID of OMVS. Enter MVS command D OMVS to see your OMVS ASID in message BPXO042I.
  
  
• An MVS system dump of CICS taken when an abend occurs that is the result of a failure in the JVM or the Language Environment (LE) enclave that it runs is almost always of limited use. This is because it probably will not contain the JVM storage and the LE enclave may have already ended. However in this case, JAVADUMP and SYSDUMP (TDUMP) and CEEDUMP dumps can be very useful.
  
  
• You can control the behavior of JAVADUMP, SYSDUMP, and CEEDUMP dumps by using the JAVA_DUMP_OPTS environment variables in your JVM profile. By default CICS TS sets JAVA_DUMP_OPTS to
  ONANYSIGNAL(JAVADUMP,CEEDUMP,SYSDUMP),ONINTERRUPT(NONE).
  
  You must apply APAR PK13558 for CICS TS V3.1 before you can specify JAVA_DUMP_OPTS to override the default. You do not have to apply an APAR for CICS TS V3.2 or higher. DUMP=YES or DUMP=NO in the JVM profile has no affect in any release of CICS.
  
  
• JAVADUMP is a formatted dump taken by the JVM to summarize the internal state of the JVM when it crashes unexpectedly. The output file is located in the directory named by the WORK_DIR option in the JVM profile.
  
  
• SYSDUMP (TDUMP) is a standard SVC dump taken by the JVM when the JVM crashes. See SYSDUMP under section 8.3.1 Javadumps in the Redbooks publication Java Applications Development for CICS for an example of the messages written to stderr when the JVM requests a sysdump and instructions on formatting the dump.
  
  To override the default dump name, use the environment variable JAVA_DUMP_TDUMP_PATTERN in the JVM profile for SDK V1.4, V5, and V6. See II13292 and the Java Diagnostics Guide for common questions and more details.
  
  
• CEEDUMP is a formatted application level dump taken by the LE service CEE3DMP after any SYSDUMP processing. See CEEDUMP under section 8.3.1 Javadumps in the Redbooks publication Java Applications Development for CICS for an example of the LE traceback information that can be found in CEEDUMP.
  
  
• Java version you are running. Determine this by doing one of the following:
• See message DFHSJ0901I written at JVM startup when you specify either of the following in the JVM profile:
  -Dcom.ibm.cics.ShowJavaVersion=true
  DISPLAY_JAVA_VERSION=YES
  
  
• From OMVS, enter command java -version.
  
  
• Take the Java version from the JAVADUMP if one is available.

See the table below for a list of CICS releases and supported Java releases. Make sure that you are using a supported release of Java. For more details, see Java Support under High-level language Support.

CICS TS release	Supported Java (SDK) release
4.2	6.0.1 (64-bit)
4.1	6
3.2	1.4.2 and 5 **
3.1	1.4.2
2.3	1.4.2

** APAR PK59577 provides support for Java V5.

Select the release of Java that you are running from the following table to be taken to a service summary. The summaries contain tables that list the SDK service levels and their corresponding PTFs. Install the latest service level to take advantage of improved service, performance, and stability.

Service levels and corresponding PTFs

IBM 64-bit SDK for z/OS, Java Technology Edition, V6.0.1

IBM 31-bit SDK for z/OS, Java Technology Edition, V6

IBM 31-bit SDK for z/OS, Java 2 Technology Edition, V5

IBM SDK for z/OS, Java 2 Technology Edition, V1.4

Optional (MustGather) data:

• If the original cause of the failure happens a lot earlier than the time of the exception itself, a SYSDUMP (TDUMP) taken at the time you receive an earlier Java message can be helpful. Specify the following options in the JVM profile to capture a dump for a specific Java message:
  
  IBM_XE_COE_NAME=AnyValidJava/message
  JAVA_DUMP_OPTS=ONEXCEPTION(SYSDUMP),ONERROR(SYSDUMP)
  JAVA_DUMP_TDUMP_PATTERN=\
  "DUMP.JVM.TDUMP.&APPLID;.&JVM_NUM;.LATEST"
  
  Replace AnyValidJava/message with any valid Java message such as java/lang/OutOfMemoryError.
  
  To continue a parameter on a second line, you end the first line with the backslash and a blank (\   ) continuation characters. This is seen above for JAVA_DUMP_TDUMP_PATTERN=\ .
  
  

• JVM trace that is written to CICS internal trace and included in the MVS system dump when tracing is active. The trace should be at least 10240K and when possible level 1 tracing should be on for all CICS components. See Defining and activating traces for pooled JVMs in the Problem Determination Guide for more details. Note: JVM trace can also be written to CICS auxiliary trace or GTF Trace.

Back to top

Troubleshooting hints and tips

Diagnostic tips:

• Review the logs and dumps generated at the point of failure.
• Search the CICS support site for known problems using symptoms like the message number and error codes.
• If you find a fixing PTF, see Ordering CICS products and maintenance for the options that are available to order CICS maintenance.
• Gather the documentation and work with the CICS support team to resolve your problem.
  
  

Learn more about pooled JVMs:

• Java programming and support roadmap (CICS TS 4.2)
• Java programming and support roadmap (CICS TS 4.1)
• Java programming and support roadmap (CICS TS 3.2)
• Java programming and support roadmap (CICS TS 3.1)
• Redbooks: Java Applications Development for CICS (CICS TS 3.2)
• IBM developer kits: Diagnosis documentation (Java Diagnostics Guide)

Exchanging data with IBM Support

1. See Exchanging information with IBM Technical Support for FTP and email instructions using the IBM Enhanced Customer Data Repository (ECuRep).
   
2. Go to the Service Request page to open or update a problem. If you have a SoftwareXcel enterprise edition for zSeries contract, you can also ask a non-defect installation and usage questions.
   
   Note: Always update your Service Request (problem record) to indicate that data has been sent.

If you need to speak to an IBM technical support representative call your country representative. If you need to speak to an IBM technical support representative in the US, call 1-800-IBM-SERV.

Back to top