HealthCenter Thread Dumps To Save Your Sanity!

Body

 

 

 

HELP!  I AM UNSUCCESSFUL IN CAPTURING THREAD DUMPS DURING A REQUIRED TIME-FRAME?

You have lived the story before:  There is an intermittent delay in your IBM JVM application serving environment.  You contact support and they ask for the Performance MustGather collected DURING the time of the delay.  While the actual act of collecting the MustGather data is simple, since the delay is intermittent, you just can't seem to capture the MustGather data DURING the time of the performance issue.  No MustGather = No way to understand the problem.  This can be a frustrating situation for both you and the support person trying to assist.

 

SOLUTION: 

Add -Xhealthcenter:level=headless to the generic JVM arguments. Save, synchronize and restart the JVM.

Java Health Center - IBM Application Server Performance Cookbook

 

IDEAL OUTCOME:

Gather the generated *.hcd file from the WebSphere Application Server profile directory, and view results in the Java Health Center client.

The healthCenter*.hcd file will be produced on JVM stop.

 

*********************************************************

IF YOU DO NOT WISH TO WAIT FOR JVM STOP:

Prior to JVM stop, the JHC data is retained in a tmp directory which can be harvested in it's raw form:

*********************************************************

LOCATE:

The WebSphere Application Server profile directory for a folder named tmp_<mmddyy>_<hhmmss> "threads" file.

The threads file contains JHC raw thread dump data which you will want to convert to a human readable format:

 

 

ZIP: 

The threads file to a new threads.zip file.

I suggest to name the threads.zip file with the PID and timestamp if possible: PID12345_100817_002136_threads.zip

LOAD:

Upload the threads.zip file to WebSphere Commerce Health Center Report

This tool was created by the Commerce Team but it works just as well for WebSphere Application Server JHC data.

CLICK:

Threads -> Thread Dumps

This step will parse the threads*.zip data, convert it into separate human-readable thread dump files, and zip the individual files into a single hcthreads.zip file.

 

 

DOWNLOAD:

hcthreads.zip:

 

The hcthreads.zip will contain thread dumps collected at 30 second intervals.

I suggest renaming this hcthreads.zip file to the same PID and timestamp which the *.threads.zip was named for.

Note:

The number of files that Health Center creates depends on the length of time that you have been monitoring the application, and the number of backing files that you chose to keep

 

REVIEW:

Each dump will be named with the timestamp of when it was collected by the JVM.

Find the thread dumps that match the time of slow performance.

Threads that do not change their state over time or threads that are "BLOCKED" are suspect for contention.

 

FURTHER ASSISTANCE:

If you are unable to determine the cause of the performance delay by reviewing the thread dumps over time, engage WebSphere Application Server Support for further assistance.

 

RELATED:

developerWorks : Technical Topics : Java technology : IBM Developer kits : Tools : IBM Monitoring and Diagnostic Tools - Health Center

IBM MustGather: Read first for WebSphere Application Server

Java Health Center - IBM Application Server Performance Cookbook

 

**Using the documented MustGathers for collecting performance data is the recommended method to collect the most complete set of data to use when working with IBM Support.  The suggestion above is an alternative method for capturing thread data when the MustGather method has been unsuccessful or if a long running set of thread dump data is required.

IBM MustGather: Performance, hang, or high CPU issues with WebSphere Application Server on AIX

IBM MustGather: Performance, hang, or high CPU issues with WebSphere Application Server on Linux

IBM MustGather: Performance, hang, or high CPU issues on Windows

 

***The overhead for running JHC is very low at ~1% with the default settings.