HealthCenter Thread Dumps To Save Your Sanity!
MicheleCalcavecchia 270000HCF1 Visits (5749)
HELP! I AM UNSUCCESSFUL IN CAPTURING THREAD DUMPS DURING A REQUIRED TIME-FRAME?
You have lived the story before: There is an intermittent delay in your IBM JVM application serving environment. You contact support and they ask for the Performance MustGather collected DURING the time of the delay. While the actual act of collecting the MustGather data is simple, since the delay is intermittent, you just can't seem to capture the MustGather data DURING the time of the performance issue. No MustGather = No way to understand the problem. This can be a frustrating situation for both you and the support person trying to assist.
Gather the generated *.hcd file from the WebSphere Application Server profile directory, and view results in the Java
The healthCenter*.hcd file will be produced on JVM stop.
IF YOU DO NOT WISH TO WAIT FOR JVM STOP:
Prior to JVM stop, the JHC data is retained in a tmp directory which can be harvested in it's raw form:
The WebSphere Application Server profile directory for a folder named tmp_
The threads file contains JHC raw thread dump data which you will want to convert to a human readable format:
The threads file to a new threads.zip file.
I suggest to name the threads.zip file with the PID and timestamp if possible: PID1
Upload the threads.zip file to WebS
This tool was created by the Commerce Team but it works just as well for WebSphere Application Server JHC data.
Threads -> Thread Dumps
This step will parse the threads*.zip data, convert it into separate human-readable thread dump files, and zip the individual files into a single hcthreads.zip file.
The hcthreads.zip will contain thread dumps collected at 30 second intervals.
I suggest renaming this hcthreads.zip file to the same PID and timestamp which the *.threads.zip was named for.
The number of files that Health Center creates depends on the length of time that you have been monitoring the application, and the number of backing files that you chose to keep
Each dump will be named with the timestamp of when it was collected by the JVM.
Find the thread dumps that match the time of slow performance.
Threads that do not change their state over time or threads that are "BLOCKED" are suspect for contention.
If you are unable to determine the cause of the performance delay by reviewing the thread dumps over time, engage WebS
**Using the documented MustGathers for collecting performance data is the recommended method to collect the most complete set of data to use when working with IBM Support. The suggestion above is an alternative method for capturing thread data when the MustGather method has been unsuccessful or if a long running set of thread dump data is required.
***The overhead for running JHC is very low at ~1% with the default settings.