MUSTGATHER: Java Application hangs or is taking 100% of the CPU

Answer

1. Set up Java to turn on and collect the verbose gc (garbage collection) output:

If you are using RAD 7:

Backup your current eclipse.ini file.

Now edit your eclipse.ini file, change the line that reads
C:\Program Files\IBM\SDP70\jdk\jre\bin\javaw.exe
to
C:\Program Files\IBM\SDP70\jdk\jre\bin\java.exe

and add in the lines
-debug
-consolelog
just before the line that starts with "-vmargs"

Your eclipse.ini will now look like this:

-vm
C:\Program Files\IBM\SDP70\jdk\jre\bin\java.exe
-debug
-consolelog
-vmargs
-Xquickstart
-Xms40m
-Xmx768m
-Xgcpolicy:gencon
-Xscmx96m
-Xshareclasses:singleJVM,keep
-Xnolinenumbers
-XX:MaxPermSize=512M

Note: (black = unchanged, green = changed, blue = new lines)

You may have to change the "C:\Program Files\IBM\SDP70\jdk\jre\bin\java.exe" to point to the proper JRE for your installation.

The use of java.exe (instead of javaw.exe), along with "-debug" and "-consolelog" will allow us to get debug information about the java executeable.

Then add the following lines to your new eclipse.ini file:

-verbose:gc
-Xverbosegclog:[DIR_PATH][FILE_NAME]

where:
[DIR_PATH] is the directory where the debug/logging info will be written
[FILE_NAME] is the name of the file to write the debug/logging info to
eg. java -Xverbosegclog:C:\temp\verbose_output.xml

Note: there are no trailing white spaces after any of the switches, and the text for all switches start in the first column.

The eclipse.ini file will be in the "Installation Directory" directory of your RAD 7.0 installation.
By default, this is "C:\Program Files\IBM\SDP70"

If you don't know where your "Installation Directory" is, start the Installation Manager and click on "Update Packages", select the "IBM Software Development Platform" entry under "Installation Localtions", and you will get a dialog that looks like this:

Then, launch eclipse.exe : eclipse.exe >debug.dta 2>&1
All needed output will go to "debug.dta"


If you are using RAD 6:

Start a command line session in your RAD 6 installation directory which is, by default
C:\Program Files\IBM\Rational\SDP\6.0

change to the Eclipse directory within it.

Start RAD6 using the following command line:

jre\bin\java -debug -verbose:gc -Djava.compiler= -Dosgi.adaptor=org.eclipse.core.runtime.adaptor.EclipseAdaptor -cp startup.jar org.eclipse.core.launcher.Main >debug.txt 2>&1

The meaning of
-verbose:gc
is the same as previously mentioned.


2. OPTIONAL - Once you have your Java application running, enable CPU, process size, and page size monitoring using Perfmon. Ideally, do this step after the application starts, but before you actually start to do anything with it.

The PerfMon tool is packaged as part of the Windows operating system and can be used to monitor a wide variety of counters, including those related to the process size and virtual address space usage. PerfMon PerfMon can usually be started by typing perfmon on the command line. Alternatively, PerfMon can be activated going to Start > Control Panel > Administrative Tools > Performance Activating PerfMon Logging Expand Performance logs and alerts Right-click on Counter Logs and select New Log Settings. Enter a name for the log and select the OK button. This causes any output to immediately start logging to the log file specified above. Monitoring the CPU Usage Click on the + button on the toolbar. Click the Thread performance object Select the following counters from the list: % Processor Time ID Process ID Thread 1. Click on the instance of the Java process to monitor. Ensure that you select all of the Java threads. 2. Click Add and then Close Monitoring the Process Size Click the + button on the toolbar. Click t the Process performance object Select Virtual Bytes from the counters list Select the instance for the Java process to monitor Click Add and then Close Monitoring the Paging Usage Click the + button on the toolbar. Click on the Memory performance object Select the Pages/sec counter Click on Add and then Close Note: You will have to do this after closing RAD, using the TaskManager. Stopping PerfMon Logging In order to stop logging and obtain the log file, the following needs to be done: Right-click Log Name in the counter window Click Stop The log file is referenced in the Log File Name column.

3. Follow whatever steps you normally need to in order to reproduce the hang or CPU busy condition, then go to the console the java.exe executable was launched from and hit CTRL-BREAK 3 to 10 times; each hit should be about 10 seconds apart. This will cause a javacore*.txt to be produced in the directory containing the java executable.


4. Gather up any javacore*.txt and core*.DMP files

Take the files from the above 3 steps and upload them to the Rational Support FTP site

• The verbose gc log & debug.dta file.
• The Perfmon logs on CPU useage, Process Size, and Paging Size -- if any exist.
• Any Javacore*txt and core*.dmp files -- they'll be in your "shared resources" folder.

.