Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
4 replies Latest Post - ‏2012-11-15T07:55:29Z by SystemAdmin
SystemAdmin
SystemAdmin
2736 Posts
ACCEPTED ANSWER

Pinned topic Not all native thread stacks captured in AIX 7.1 + Java6 SR9 FP2

‏2012-11-14T18:44:14Z |
Hi folks
Please let me know if there's something im missing here.

I have a javacore and coredump from a 'user' signal (-Xdump:system+java:events=user)
Javacore shows me 53 threads with native stacks (albeit, just the IP and library+offset , not the function name) but native stack nonetheless.

However the corresponding core file , which was generated first given its sequence number core.time.PID.001.dmp, has just ONE thread's native stack.
This is all i could get:

IOT/Abort trap in pthread_kill at 0x900000000804830 ($t1)
0x900000000804830 (pthread_kill+0xb0) e8410028 ld r2,0x28(r1)

(dbx) thread

thread state-k wchan state-u k-tid mode held scope function
>$t1 run running 56295537 k no sys pthread_kill
(dbx) where
pthread_kill(??, ??) at 0x900000000804830
_p_raise(??) at 0x9000000008040e4
raise.raise(??) at 0x900000000038be8
abort() at 0x900000000053be4
j9dump_create() at 0x900000000cfdbb0
doSystemDump() at 0x900000000dbf064
protectedDumpFunction() at 0x900000000dc35e0
j9sig_protect() at 0x900000000ce6f90
runDumpFunction() at 0x900000000dc3584
runDumpAgent() at 0x900000000dc3370
triggerDumpAgents() at 0x900000000ddfb24
sigQuitHandler() at 0x900000001f498c0
j9sig_protect() at 0x900000000ce6f90
sigQuitWrapper() at 0x900000001f49724
asynchSignalReporter() at 0x900000000ce7f2c
thread_wrapper() at 0x900000000cc2c6c

It looks like the core has just the native stack of the thread responsible for the SIGQUIT.
This was an issue with Linux where the triggerring the core generation from within the IBM JVM caused just the singal handling thread's context to be captured.. (apologies if i have gotten this wrong)
Has this emerged on AIX now?

Any help is appreciated!

Advance thanks!
Updated on 2012-11-15T07:55:29Z at 2012-11-15T07:55:29Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    2736 Posts
    ACCEPTED ANSWER

    Re: Not all native thread stacks captured in AIX 7.1 + Java6 SR9 FP2

    ‏2012-11-14T18:55:30Z  in response to SystemAdmin
    Some more info..

    The core file is ~ 1.2GB

    Java version:
    =============
    JRE 1.6.0 IBM J9 2.4 AIX ppc64-64 build jvmap6460sr9-20110624_85526
    VM build 20110624_085526
    JIT enabled, AOT enabled - r9_20101028_17488ifx17
    GC - 20101027_AA

    OS Info:
    ========
    OS Level : AIX 7.1
    Processors -
    Architecture : ppc64
    How Many : 4
    Options of Importance:
    =======================
    -Xdump:system+java:events=user
    -Xms512m
    -Xmx1024m
    Limits:
    =======

    User Limits (in bytes except for NOFILE and NPROC)

    type soft limit hard limit
    RLIMIT_AS unlimited unlimited
    RLIMIT_CORE unlimited unlimited
    RLIMIT_CPU unlimited unlimited
    RLIMIT_DATA unlimited unlimited
    RLIMIT_FSIZE unlimited unlimited
    RLIMIT_NOFILE unlimited unlimited
    RLIMIT_RSS unlimited unlimited
    RLIMIT_STACK 4294967296 4294967296
    • SystemAdmin
      SystemAdmin
      2736 Posts
      ACCEPTED ANSWER

      Re: Not all native thread stacks captured in AIX 7.1 + Java6 SR9 FP2

      ‏2012-11-15T05:20:43Z  in response to SystemAdmin
      Hi

      I tried it on our environment using the same SDK/JVM & OS level. The generated core image contains the complete JVM process context.

      1CIJAVAVERSION JRE 1.6.0 IBM J9 2.4 AIX ppc64-64 build jvmap6460sr9-20110624_855
      26
      1CIVMVERSION VM build 20110624_085526
      1CIJITVERSION JIT enabled, AOT enabled - r9_20101028_17488ifx17
      1CIGCVERSION GC - 20101027_AA

      (dbx) where
      coredump(??) at 0x9000000002e1a68
      j9dump_create() at 0x90000001670cae4
      doSystemDump() at 0x90000001677e064
      protectedDumpFunction() at 0x9000000167825e0
      j9sig_protect() at 0x9000000166f5f90
      runDumpFunction() at 0x900000016782584
      runDumpAgent() at 0x900000016782370
      triggerDumpAgents() at 0x90000001679eb24
      sigQuitHandler() at 0x90000001700e8c0
      j9sig_protect() at 0x9000000166f5f90
      sigQuitWrapper() at 0x90000001700e724
      asynchSignalReporter() at 0x9000000166f6f2c
      thread_wrapper() at 0x9000000166e1c6c

      (dbx) thread
      thread state-k wchan state-u k-tid mode held scope function
      $t1 run 0xf1000f0a10028340 blocked 42139713 k no sys _event_sleep
      $t2 run 0xf1000f0a10011d40 blocked 18677951 k no sys _event_sleep
      >$t3 run running 25690121 k no sys coredump

      I can see difference wrt to the call stack of the User dump handler thread. Can you try with simple program & confirm it again.
    • SystemAdmin
      SystemAdmin
      2736 Posts
      ACCEPTED ANSWER

      Re: Not all native thread stacks captured in AIX 7.1 + Java6 SR9 FP2

      ‏2012-11-15T05:51:12Z  in response to SystemAdmin
      Can you also check the system property "Enable full CORE dump". If it is set to false, JVM is expected to generate core image with limited threading information.

      Given steps to enable "full CORE dump" in the following link,

      http://publib.boulder.ibm.com/infocenter/javasdk/v6r0/index.jsp?topic=%2Fcom.ibm.java.doc.diagnostics.60%2Fdiag%2Fproblem_determination%2Faix_setup_full_core.html
      • SystemAdmin
        SystemAdmin
        2736 Posts
        ACCEPTED ANSWER

        Re: Not all native thread stacks captured in AIX 7.1 + Java6 SR9 FP2

        ‏2012-11-15T07:55:29Z  in response to SystemAdmin
        Param,
        The limits in my previous post show that full core is enabled.
        I will try with a standalone as you have suggested.