1 reply Latest Post - ‏2012-10-20T18:59:07Z by ChrisBailey
2736 Posts

Pinned topic Segmentation fault: gpf java core dump being produced

‏2012-10-15T17:28:52Z |
I am running a Java application on AIX 6.1 and at least once a day the application crashes producing a core dump. Running dbx on it I get the following:


for help. [using memory image in core.20121011.182320.991266.0001.dmp] reading symbolic information ...   Signal 0 in coredump at 0xd03a3efc ($t44) 0xd03a3efc (coredump+0x3c) 80410014        lwz   r2,0x14(r1) (dbx) where coredump(??) at 0xd03a3efc j9dump_create() at 0x3012a450 doSystemDump() at 0x302750c0 protectedDumpFunction() at 0x3027939c j9sig_protect() at 0x30114ba8 runDumpFunction() at 0x30279340 runDumpAgent() at 0x3027912c triggerDumpAgents() at 0x30294a90 generateDiagnosticFiles() at 0x3008ebd4 j9sig_protect() at 0x30114ba8 structuredSignalHandler() at 0x3008e728 masterSynchSignalHandler() at 0x30116084 jitCheckCast(), line 53 in 
"nathelp.s" javaProtectedThreadProc() at 0x3008b90c j9sig_protect() at 0x30114ba8 javaThreadProc() at 0x3008b82c thread_wrapper() at 0x30101cbc (dbx) listi 0xd03a3efc (coredump+0x3c) 80410014        lwz   r2,0x14(r1) 0xd03a3f00 (coredump+0x40) 81810058        lwz   r12,0x58(r1) 0xd03a3f04 (coredump+0x44) 38210050       addi   r1,0x50(r1) 0xd03a3f08 (coredump+0x48) 7d8803a6       mtlr   r12 0xd03a3f0c (coredump+0x4c) 83e1fffc        lwz   r31,-4(r1) 0xd03a3f10 (coredump+0x50) 4e800020        blr 0xd03a3f14 (coredump+0x54) 83c20714        lwz   r30,0x714(r2) 0xd03a3f18 (coredump+0x58) 80020004        lwz   r0,0x4(r2) 0xd03a3f1c (coredump+0x5c) 2c000000       cmpi   cr0,0x0,r0,0x0 0xd03a3f20 (coredump+0x60) 41820080        beq   0xd03a3fa0 (coredump+0xe0)

And the java core dump looks like:

NULL           ------------------------------------------------------------------------ 0SECTION       TITLE subcomponent dump routine NULL           =============================== 1TISIGINFO     Dump Event 
"gpf" (00002000) received 1TIDATETIME    Date:                 2012/10/11 at 18:23:36 1TIFILENAME    Javacore filename:    /tmp/VM-Temp-Dump-aLicib/javacore.20121011.182320.991266.0002.txt 1TIREQFLAGS    Request Flags: 0x81 (exclusive+preempt) 1TIPREPSTATE   Prep State: 0x0 1TIPREPINFO    Exclusive VM access not taken: data may not be consistent across javacore sections NULL           ------------------------------------------------------------------------ 0SECTION       GPINFO subcomponent dump routine NULL           ================================ 2XHOSLEVEL     OS Level         : AIX 6.1 2XHCPUS        Processors - 3XHCPUARCH       Architecture   : ppc 3XHNUMCPUS       How Many       : 2 3XHNUMASUP       NUMA is either not supported or has been disabled by user NULL 1XHEXCPCODE    J9Generic_Signal_Number: 00000004 1XHEXCPCODE    Signal_Number: 0000000B 1XHEXCPCODE    Error_Value: 00000000 1XHEXCPCODE    Signal_Code: 00000032 1XHEXCPCODE    Handler1: 30156208 1XHEXCPCODE    Handler2: 30150F1C NULL NULL 1XHREGISTERS   Registers: 2XHREGISTER      R0: 31964BD0 2XHREGISTER      R1: 3499E490 2XHREGISTER      R2: 30A778CC 2XHREGISTER      R3: 32294BA0 2XHREGISTER      R4: 3007B960 2XHREGISTER      R5: 00000038 2XHREGISTER      R6: 29F05079 2XHREGISTER      R7: 00000038 2XHREGISTER      R8: 00000001 2XHREGISTER      R9: 713A0B1C 2XHREGISTER      R10: 00000010 2XHREGISTER      R11: 001D0001 2XHREGISTER      R12: 316B4BCC 2XHREGISTER      R13: 34742900 2XHREGISTER      R14: 3474FCF0 2XHREGISTER      R15: 0007647D 2XHREGISTER      R16: 300EDBAC 2XHREGISTER      R17: 00000000 ... 2XHREGISTER      FPR30: 40CFF00000000000 2XHREGISTER      FPR31: 4080000000000000 2XHREGISTER      IAR: 30910F40 2XHREGISTER      LR: 31964D68 2XHREGISTER      MSR: 0000D032 2XHREGISTER      CTR: 311FE494 2XHREGISTER      CR: 42800084 2XHREGISTER      FPSCR: 82000000 2XHREGISTER      XER: 00000004 2XHREGISTER      TID: 00000000 2XHREGISTER      MQ: 347421F8 NULL 1XHFLAGS       VM flags:00000000 ...

Note that no specific module is stated in the java core dump.

The java version info is:

NULL           ------------------------------------------------------------------------ 0SECTION       ENVINFO subcomponent dump routine NULL           ================================= 1CIJAVAVERSION JRE 1.6.0 IBM J9 2.4 AIX ppc-32 build jvmap3260sr9-20110203_74623 1CIVMVERSION   VM build 20110203_074623 1CIJITVERSION  JIT enabled, AOT enabled - r9_20101028_17488ifx3 1CIGCVERSION   GC - 20101027_AA 1CIRUNNINGAS   Running as a standalone JVM

Within the java core dump the current thread is given as:

NULL           ------------------------------------------------------------------------ 0SECTION       THREADS subcomponent dump routine NULL           ================================= NULL 1XMCURTHDINFO  Current thread NULL           ---------------------- 3XMTHREADINFO      
"Quartz-4" J9VMThread:0x34742900, j9thread_t:0x3452D724, java/lang/Thread:0x70ABD790, state:R, prio=5 3XMTHREADINFO1            (

native thread ID:0x1A2095, 

native priority:0x5, 

native policy:UNKNOWN) 1INTERNAL                    Unable to walk in-flight data on call stack 3XMTHREADINFO3           Native callstack: 4XENATIVESTACK               (0x3008B910) 4XENATIVESTACK               (0x30114BAC) 4XENATIVESTACK               (0x3008B830) 4XENATIVESTACK               (0x30101CC0) 4XENATIVESTACK               (0xD04A1C50) NULL NULL

I am confused as to why the crash is occurring. I have tried only using -Xnoaot, and the problem still occurs. It seems that turning off the jit compiler causes the issue to go away.

I have already tried reducing the jit optimization levels, decreasing level by level, but even the noOpts level still causes a crash. Completely turning off the jit compiler is not a viable solution.

Any help would be appreciated.

Updated on 2012-10-20T18:59:07Z at 2012-10-20T18:59:07Z by ChrisBailey
  • ChrisBailey
    190 Posts

    Re: Segmentation fault: gpf java core dump being produced

    ‏2012-10-20T18:59:07Z  in response to SystemAdmin
    Hi Derek:

    To do much analysis on this is looks like we'd need access to the core file itself in order to work out where the failure is. I know its not a great suggestion, but do you have any scope to upgrade? There's been a number (80 or 90) JIT fixes published since the level you have, any of which could solve the problem.