Topic
1 reply Latest Post - ‏2012-06-22T04:27:01Z by DGDT_ashwini_kumar
yansqrt
yansqrt
1 Post
ACCEPTED ANSWER

Pinned topic Collecting javacore and heapdump caused java application hang

‏2012-01-18T07:51:03Z |
Hi,

I was taking a snapshot using "kill -3 $pid", but hit warnings for dump and error for core:

 JVMDUMP006I Processing dump event "user", detail "" - please wait.
 JVMDUMP022W The requested heap compaction has not been performed because exclusive access was not requested or could not be obtained.
 JVMDUMP023W The requested heap preparation has not been performed because exclusive access was not requested or could not be obtained.
 JVMDUMP027W The requested heapdump has not been produced because another component is holding the VM exclusive lock.
 JVMDUMP032I JVM requested Java dump using '/logs/javacore.20120112.000455.2818088.0024.txt' in response to an event
 JVMDUMP012E Error in Java dump: /logs/javacore.20120112.000455.2818088.0024.txt
 JVMDUMP013I Processed dump event "user", detail "".

After that, the application seemed working strangely, so I typed the second "kill -3", the result was:

 JVMDUMP006I Processing dump event "user", detail "" - please wait.
 JVMDUMP022W The requested heap compaction has not been performed because exclusive access was not requested or could not be obtained.
 JVMDUMP023W The requested heap preparation has not been performed because exclusive access was not requested or could not be obtained.
 JVMDUMP027W The requested heapdump has not been produced because another component is holding the VM exclusive lock.
 JVMDUMP032I JVM requested Java dump using '/logs/javacore.20120112.101036.2818088.0026.txt' in response to an event
 JVMDUMP010I Java dump written to /logs/javacore.20120112.101036.2818088.0026.txt
 JVMDUMP013I Processed dump event "user", detail "".

The javacore was successfully collected, which told me most of the threads have been blocked. In JVM System Monitor Dump, they
were waiting to get the "VM class table lock" which was locked by thread named osthread:
...
2LKREGMON     VM class loader blocks lock (0x30010988): <unowned>
2LKREGMON     VM class table lock (0x300109E4): Flat locked by "osthread" (0x30011724), entry count 1
3LKWAITERQ      Waiting to enter:
3LKWAITER         "JIT Compilation Thread" (0x305BC000)
3LKWAITER         "Thread-3" (0x31C14300)
3LKWAITER         "NIMessageRouterDiscoveryThread - 1721" (0x31DB2E00)
3LKWAITER         "NIMessageRouterInboundThread" (0x31DB3600)
3LKWAITER         "Thread-4" (0x31F4AE00)
3LKWAITER         "Thread-12" (0x323A2000)
3LKWAITER         "NICommandProcessor 16" (0x32BBFA00)
3LKWAITER         "Thread-11" (0x32C97900)
...
3LKWAITER         "Thread-4031" (0x33266A00)
2LKREGMON     VM string table lock (0x30010A40): <unowned>
2LKREGMON     VM segment lock (0x30010A9C): <unowned>
2LKREGMON     VM JNI frame lock (0x30010AF8): <unowned>
...

After the java application was restarted, the problem no longer persisted and I can't recreate it again.

My question is did anyone use to hit similar issue? And what is that osthread? I had a search but in most
other cases 'osthread' was holding the 'VM thread list lock', and one case 'VM class table lock' was held by
JIT Compilation Thread, which was fixed in java 6.0 (IZ67874 / PM10711).

I was using IBM JDK 5.0 on AIX.
Thanks in advance ~
Updated on 2012-06-22T04:27:01Z at 2012-06-22T04:27:01Z by DGDT_ashwini_kumar
  • DGDT_ashwini_kumar
    DGDT_ashwini_kumar
    1 Post
    ACCEPTED ANSWER

    Re: Collecting javacore and heapdump caused java application hang

    ‏2012-06-22T04:27:01Z  in response to yansqrt
    I have hit the same problem. I am wondering if you were able to get any answers to this problem. I am unable to fetch the heap and thread dump that is essential to solving this issue.
    Thanks
    Ashwin