Topic
  • 4 replies
  • Latest Post - ‏2012-11-15T07:55:29Z by SystemAdmin
SystemAdmin
SystemAdmin
2736 Posts

Pinned topic Not all native thread stacks captured in AIX 7.1 + Java6 SR9 FP2

‏2012-11-14T18:44:14Z |
Hi folks
Please let me know if there's something im missing here.

I have a javacore and coredump from a 'user' signal (-Xdump:system+java:events=user)
Javacore shows me 53 threads with native stacks (albeit, just the IP and library+offset , not the function name) but native stack nonetheless.

However the corresponding core file , which was generated first given its sequence number core.time.PID.001.dmp, has just ONE thread's native stack.
This is all i could get:

IOT/Abort trap in pthread_kill at 0x900000000804830 ($t1)
0x900000000804830 (pthread_kill+0xb0) e8410028 ld r2,0x28(r1)

(dbx) thread

thread state-k wchan state-u k-tid mode held scope function
>$t1 run running 56295537 k no sys pthread_kill
(dbx) where
pthread_kill(??, ??) at 0x900000000804830
_p_raise(??) at 0x9000000008040e4
raise.raise(??) at 0x900000000038be8
abort() at 0x900000000053be4
j9dump_create() at 0x900000000cfdbb0
doSystemDump() at 0x900000000dbf064
protectedDumpFunction() at 0x900000000dc35e0
j9sig_protect() at 0x900000000ce6f90
runDumpFunction() at 0x900000000dc3584
runDumpAgent() at 0x900000000dc3370
triggerDumpAgents() at 0x900000000ddfb24
sigQuitHandler() at 0x900000001f498c0
j9sig_protect() at 0x900000000ce6f90
sigQuitWrapper() at 0x900000001f49724
asynchSignalReporter() at 0x900000000ce7f2c
thread_wrapper() at 0x900000000cc2c6c

It looks like the core has just the native stack of the thread responsible for the SIGQUIT.
This was an issue with Linux where the triggerring the core generation from within the IBM JVM caused just the singal handling thread's context to be captured.. (apologies if i have gotten this wrong)
Has this emerged on AIX now?

Any help is appreciated!

Advance thanks!
Updated on 2012-11-15T07:55:29Z at 2012-11-15T07:55:29Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    2736 Posts

    Re: Not all native thread stacks captured in AIX 7.1 + Java6 SR9 FP2

    ‏2012-11-14T18:55:30Z  
    Some more info..

    The core file is ~ 1.2GB

    Java version:
    =============
    JRE 1.6.0 IBM J9 2.4 AIX ppc64-64 build jvmap6460sr9-20110624_85526
    VM build 20110624_085526
    JIT enabled, AOT enabled - r9_20101028_17488ifx17
    GC - 20101027_AA

    OS Info:
    ========
    OS Level : AIX 7.1
    Processors -
    Architecture : ppc64
    How Many : 4
    Options of Importance:
    =======================
    -Xdump:system+java:events=user
    -Xms512m
    -Xmx1024m
    Limits:
    =======

    User Limits (in bytes except for NOFILE and NPROC)

    type soft limit hard limit
    RLIMIT_AS unlimited unlimited
    RLIMIT_CORE unlimited unlimited
    RLIMIT_CPU unlimited unlimited
    RLIMIT_DATA unlimited unlimited
    RLIMIT_FSIZE unlimited unlimited
    RLIMIT_NOFILE unlimited unlimited
    RLIMIT_RSS unlimited unlimited
    RLIMIT_STACK 4294967296 4294967296
  • SystemAdmin
    SystemAdmin
    2736 Posts

    Re: Not all native thread stacks captured in AIX 7.1 + Java6 SR9 FP2

    ‏2012-11-15T05:20:43Z  
    Some more info..

    The core file is ~ 1.2GB

    Java version:
    =============
    JRE 1.6.0 IBM J9 2.4 AIX ppc64-64 build jvmap6460sr9-20110624_85526
    VM build 20110624_085526
    JIT enabled, AOT enabled - r9_20101028_17488ifx17
    GC - 20101027_AA

    OS Info:
    ========
    OS Level : AIX 7.1
    Processors -
    Architecture : ppc64
    How Many : 4
    Options of Importance:
    =======================
    -Xdump:system+java:events=user
    -Xms512m
    -Xmx1024m
    Limits:
    =======

    User Limits (in bytes except for NOFILE and NPROC)

    type soft limit hard limit
    RLIMIT_AS unlimited unlimited
    RLIMIT_CORE unlimited unlimited
    RLIMIT_CPU unlimited unlimited
    RLIMIT_DATA unlimited unlimited
    RLIMIT_FSIZE unlimited unlimited
    RLIMIT_NOFILE unlimited unlimited
    RLIMIT_RSS unlimited unlimited
    RLIMIT_STACK 4294967296 4294967296
    Hi

    I tried it on our environment using the same SDK/JVM & OS level. The generated core image contains the complete JVM process context.

    1CIJAVAVERSION JRE 1.6.0 IBM J9 2.4 AIX ppc64-64 build jvmap6460sr9-20110624_855
    26
    1CIVMVERSION VM build 20110624_085526
    1CIJITVERSION JIT enabled, AOT enabled - r9_20101028_17488ifx17
    1CIGCVERSION GC - 20101027_AA

    (dbx) where
    coredump(??) at 0x9000000002e1a68
    j9dump_create() at 0x90000001670cae4
    doSystemDump() at 0x90000001677e064
    protectedDumpFunction() at 0x9000000167825e0
    j9sig_protect() at 0x9000000166f5f90
    runDumpFunction() at 0x900000016782584
    runDumpAgent() at 0x900000016782370
    triggerDumpAgents() at 0x90000001679eb24
    sigQuitHandler() at 0x90000001700e8c0
    j9sig_protect() at 0x9000000166f5f90
    sigQuitWrapper() at 0x90000001700e724
    asynchSignalReporter() at 0x9000000166f6f2c
    thread_wrapper() at 0x9000000166e1c6c

    (dbx) thread
    thread state-k wchan state-u k-tid mode held scope function
    $t1 run 0xf1000f0a10028340 blocked 42139713 k no sys _event_sleep
    $t2 run 0xf1000f0a10011d40 blocked 18677951 k no sys _event_sleep
    >$t3 run running 25690121 k no sys coredump

    I can see difference wrt to the call stack of the User dump handler thread. Can you try with simple program & confirm it again.
  • SystemAdmin
    SystemAdmin
    2736 Posts

    Re: Not all native thread stacks captured in AIX 7.1 + Java6 SR9 FP2

    ‏2012-11-15T05:51:12Z  
    Some more info..

    The core file is ~ 1.2GB

    Java version:
    =============
    JRE 1.6.0 IBM J9 2.4 AIX ppc64-64 build jvmap6460sr9-20110624_85526
    VM build 20110624_085526
    JIT enabled, AOT enabled - r9_20101028_17488ifx17
    GC - 20101027_AA

    OS Info:
    ========
    OS Level : AIX 7.1
    Processors -
    Architecture : ppc64
    How Many : 4
    Options of Importance:
    =======================
    -Xdump:system+java:events=user
    -Xms512m
    -Xmx1024m
    Limits:
    =======

    User Limits (in bytes except for NOFILE and NPROC)

    type soft limit hard limit
    RLIMIT_AS unlimited unlimited
    RLIMIT_CORE unlimited unlimited
    RLIMIT_CPU unlimited unlimited
    RLIMIT_DATA unlimited unlimited
    RLIMIT_FSIZE unlimited unlimited
    RLIMIT_NOFILE unlimited unlimited
    RLIMIT_RSS unlimited unlimited
    RLIMIT_STACK 4294967296 4294967296
    Can you also check the system property "Enable full CORE dump". If it is set to false, JVM is expected to generate core image with limited threading information.

    Given steps to enable "full CORE dump" in the following link,

    http://publib.boulder.ibm.com/infocenter/javasdk/v6r0/index.jsp?topic=%2Fcom.ibm.java.doc.diagnostics.60%2Fdiag%2Fproblem_determination%2Faix_setup_full_core.html
  • SystemAdmin
    SystemAdmin
    2736 Posts

    Re: Not all native thread stacks captured in AIX 7.1 + Java6 SR9 FP2

    ‏2012-11-15T07:55:29Z  
    Can you also check the system property "Enable full CORE dump". If it is set to false, JVM is expected to generate core image with limited threading information.

    Given steps to enable "full CORE dump" in the following link,

    http://publib.boulder.ibm.com/infocenter/javasdk/v6r0/index.jsp?topic=%2Fcom.ibm.java.doc.diagnostics.60%2Fdiag%2Fproblem_determination%2Faix_setup_full_core.html
    Param,
    The limits in my previous post show that full core is enabled.
    I will try with a standalone as you have suggested.