MustGather: Java Virtual Machine General Issues: How to Collect Data for the IBM Technology for Java VM

Resolving The Problem

1. Collect the following information when the Java Virtual Machine process (job) is experiencing the problem.

Issue the following commands to create the following spooled files that should be sent to IBM.

These items can be collected BEFORE the issue occurs:
WRKPTFGRP (use F6 to print or send a screen shot showing the installed status of PTFs)
DSPPTF OUTPUT(*PRINT)
WRKSBS OUTPUT(*PRINT)
WRKSHRPOOL OUTPUT(*PRINT)

These items need to be collected when the issue occurs:
WRKSYSSTS OUTPUT(*PRINT)
WRKACTJOB OUTPUT(*PRINT)
DSPJOBLOG JOB(JOBNUMBER/JOBUSER/JOBNAME) OUTPUT(*PRINT)


2. Collect JVM Summary information.

For IBM i 6.1 and later...
Execute the PRTJVMJOB command by doing the following:

• Issue the WRKJVMJOB command.
• Type 13 on the target job, and press Enter to print (a spooled file is generated).

NOTE: This should be done in tandem with collecting the Java dumps.

If the WRKJVMJOB command is not available, you can generate a JVM summary dump using the JVMINFO SST Advanced Analysis Command.

Use the JVMINFO Advanced Analysis Macro. Specify the -all <task> option.
For additional information on the JVMINFO macro, refer to IBM technote, IBM Technology for Java (J9) Serviceability - JVMINFO and How to Gather System, Java, Heap, and GcCycle Dumps in IBM i 5.4 and Later. Under V5R4 and later section, you will specify -all <task> for Step 14.


3. Gather three or four Java Dumps for the JVM when it is experiencing the issue.

For IBM i 6.1 and later...
Execute the GENJVMDMP JOB(x/x/x)TYPE(*JAVA) command by doing the following:

• Issue the WRKJVMJOB command.
• Type 12 on the target job, and press Enter to initiate the Java dump. A javacore.xxx.txt file will be produced. The location of this dump will be listed in the JVM job log.

NOTE: This should be done in tandem with collecting the JVM Summary dumps. Gather a Java Dump 3-4 times separated by 1 minute intervals so we can see changes in thread activity. The Java dump will create a javacore.xxx.xxxx.xxxx.xxxx.txt file in the directory defined for the "user.dir" JVM property. For any JVM, you can find the exact location of the core file by viewing the JVM job log (DSPJOBLOG).

If the WRKJVMJOB command is not available, you can also generate a Java dump using the Qshell kill command.

a. Find the ProcessID (PID) from the joblog of the JVM.
WRKJOB JOB(NUMBER/USER/NAME), Option 10 to Display job log
Example message text: "Java Virtual machine is IBM Technology for Java. PID(5706)
5706 is the PID of the JVM.

b. On the operating system command line, type the following:
QSH
Press the Enter key.

c. Type in the following command:
kill -quit <PID>
where <PID> is the ProcessID of the JVM recorded in Step 1.
Press the Enter key.

d. The dump file location can be found by checking the joblog of the JVM.
Example messages returned: "JVMDUMP007I JVM Requesting Java Dump using
'.../javacore.xxx.xxx.xxx.xxx.txt'"
"JVMDUMP010I Java Dump written to '.../javacore.xxx.xxx.xxx.xxx.txt'"

e. Wait one minute.
kill -quit <PID>
where <PID> is the ProcessID of the JVM recorded in Step 1.

f. Repeat step e 2 to 3 more times.


4. Gather a GcCycle Dump for the JVM when it is experiencing the issue.

For IBM i 6.1 and later...
Dump the GcCycles data by doing the following:

• Issue the WRKJVMJOB command.
• Type option 9 on the target job, and press ENTER. Then, press F7 to print the GC Tables to a spool file.

If the WRKJVMJOB command is not available, you can generate a GcCycle dump using the JVMINFO SST Advanced Analysis Command.

Use the JVMINFO Advanced Analysis Macro. Specify the -gcCycles <task> option.
For additional information on the JVMINFO macro, refer to IBM technote, IBM Technology for Java (J9) Serviceability - JVMINFO and How to Gather System, Java, Heap, and GcCycle Dumps in IBM i 5.4 and Later. Under V5R4 and later section, you will specify -all <task> for Step 14.

5. Check for License Internal Code logs that occurred during the occurrence of the issue.

a. Issue the STRSST command.
b. Enter SST UserID/PWD.
c. Option 1. Start a service tool (press Enter).
d. Option 5. Licensed Internal Code log (press Enter).
e. Option 1. Select entries from the Licensed Internal Code (LIC) log (press Enter two times).
f. Specify a Enty type: Major code of 4700 and minor code of 0000. Then, press Enter.
g. Look at the date and time the entry was produced. If a log exists at the time of the issue, then enter a 1 next to it and press Enter to print the log.

6. How to collect and send diagnostic information to IBM Support

i. Collect the following spooled files.

QSYSPRT (WRKPTFGRP)
QPDSPSTS (WRKSYSSTS)
QPDSPSBS (WRKSBS)
QSYSPRT (WRKSHRPOOL)
QPDSPJOBLOG (DSPJOBLOG)
QPJVAWRKVM (JVM Summary (V6R1) & GcCycles) (IBM Technology for Java Only)
QPCSMPRT (JVM Summary (V5R4))
QPCSMPRT (License Internal Code Logs)

Directions for collecting spooled files that you own

To retrieve the spooled files that you own, do the following:

a.	Open the System i Navigator.
b.	Expand your system where the files are located. Log in with your user ID (preferably QSECOFR or equivalent).
c.	Expand Basic Operations, and then left click on Printer Output to display the output files.
e.	Locate the following spooled files: Refer to the list above.
f.	Finally, right click the files from System i Navigator, select Export..., select the destination folder, and click Save.

Directions for collecting spooled files under a different user than yourself

To retrieve the spooled files that another user owns, do the following:

a.	Open the System i Navigator.
b.	Expand your system where the files are located. Log in with your user ID (preferably QSECOFR or equivalent).
c.	Expand Basic Operations, and then right click on Printer Output, and select Customize this View.
d.	Under the General tab, click Browse, from the list of users, and select the user who owns the spooled file.
e.	Locate the following spooled file(s): Refer to the list above.
f.	Finally, right click the files from System i Navigator, select Export..., select the destination folder, and click Save.

ii. Collect the javacore.xxx.xxx.txt file from the IFS of the System i.

Directions for collecting files located in the IFS

To retrieve the file(s), do the following:

a.	Open the System i Navigator.
b.	Expand your system where the files are located. Log in with your user ID (preferably QSECOFR or equivalent).
c.	Expand File Systems, and drill down to the directory that contains the file(s) requested by this document.
d.	Locate the following file(s): The java dump file can be found by checking the joblog of the JVM. WRKJOB JOB(NUMBER/USER/NAME), Option 10 to Display job log Java Dump: Example messages returned: "JVMDUMP007I JVM Requesting Java Dump using '.../javacore.xxx.xxx.xxx.xxx.txt'" "JVMDUMP010I Java Dump written to '.../javacore.xxx.xxx.xxx.xxx.txt'"
e.	Finally, drag and drop the files from System i Navigator to your desktop.

iii. Upload the data to IBM Support.

Transferring files to IBM