Topic
  • 12 replies
  • Latest Post - ‏2013-09-05T13:18:53Z by ArtyomSemenikhin
SystemAdmin
SystemAdmin
2736 Posts

Pinned topic Unhandled exception occured on USS on z/OS

‏2013-03-18T07:37:50Z |
Hi~
I met a big problem about JAVA when I had been installing a WebSphere Application Server for z/OS. Before this question is written, I found similar threads on this site. But I couldn't understand the messages completely because of lack of my knowledge or the difference of platform.

The purpose of these works is the migration of WAS 7.1 for z to WAS 8.5 for z.
At first, I installed a WAS v8.5 for z using IBM Installation Manager for z and did some works on IBM WCT and then uploaded the jobs created by WCT to z/OS 1.13 installed newly. And Some jobs are finished successfully.
But, when the step CRPROF of the job BBOWDPRO was executed, many dumping messages are created in SYSLOG. Followins:

JVMDUMP039I Processing dump event "gpf", detail "" at 2013/03/18 703
12:07:23 - please wait.

JVMDUMP032I JVM requested System dump using 'JHNA.JVM.JHNA2.D130318.T1
704
20723.X&DS' in response to an event

IGD101I SMS ALLOCATED TO DDNAME (SYS00012) 705
I thought that it is created by problem of java itself.
So I had executed "java -version" in java 1.6(64bit) ./bin directory in OMVS shell so followng messages are displayed :

:> java -version
Unhandled exception
Type=Segmentation error vmState=0x000529ff
J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000035
Handler1=000000480871DCF8 Handler2=0000004808626890
gpr0=000000488C2122C4 gpr1=0002008900060011 gpr2=0000000000000000 gpr3=000000488C3A3478
gpr4=0000004862BFB020 gpr5=00000048190E26B0 gpr6=0000000012978C28 gpr7=0000000012D4910E
gpr8=000000488C3A3478 gpr9=0000000000000001 gpr10=000000488C3A3D28 gpr11=0000000000000000
gpr12=0000000000000000 gpr13=00000048190B0200 gpr14=00000048086E3E78 gpr15=00000048190B11F8
fpc=0008000000000000 psw0=0785240180000000 psw1=0000000012978C32
fpr0=C1E0000000000000 fpr1=0000000000000000 fpr2=0000000000000000 fpr3=0000000000000000
fpr4=406F000000000000 fpr5=0000000000000000 fpr6=0000000000000000 fpr7=0000000000000000
fpr8=0000000000000000 fpr9=0000000000000000 fpr10=0000000000000000 fpr11=0000000000000000
fpr12=0000000000000000 fpr13=0000000000000000 fpr14=0000000000000000 fpr15=0000000000000000

Method_being_compiled=java/lang/J9VMInternals.initialize(Ljava/lang/Class;)V
Target=2_60_20120322_106210 (z/OS 01.13.00)
CPU=s390x (8 logical CPUs) (0x280000000 RAM)

Stack Backtrace
protectedIntrospectBacktraceSymbols+0x845ea5e0 (0x0000000000000000)
j9sig_protect+0x845d1d90 (0x0000000000000000)
j9introspect_backtrace_symbols+0x845ea6c0 (0x0000000000000000)
generateDiagnosticFiles+0x8439f688 (0x0000000000000000)
j9sig_protect+0x845d1d90 (0x0000000000000000)
structuredSignalHandler+0x8439e2b0 (0x0000000000000000)
masterSynchSignalHandler+0x845d3698 (0x0000000000000000)
(0x0000000000000000)
__zerros+0xeddd52f0 (0x0000000000000000)
CEEHDSP+0xee205ce0 (0x0000000000000000)
CEEOSIGJ+0xedfb6240 (0x0000000000000000)
CELQHROD+0xee1f2488 (0x0000000000000000)
CEEOSIGG+0xedfbcc30 (0x0000000000000000)
CELQHROD+0xee1f2488 (0x0000000000000000)
TR_Block::getLastRealTreeTop()+0xed6873d8 (0x0000000000000000)
TR_RegisterCandidates::candidatesOverlap(TR_Block*,TR_RegisterCandidate*,TR_RegisterCandida...+0xed2b7a70 (0x0000000000000000)
TR_RegisterCandidates::computeAvailableRegisters(TR_RegisterCandidate*,int,int,TR_Block**,T...+0xed2ac2a8 (0x0000000000000000)
TR_RegisterCandidates::assign(int&,int&)+0xed2b5510 (0x0000000000000000)
TR_GlobalRegisterAllocator::perform()+0xed2a61e0 (0x0000000000000000)
TR_OptimizerImpl::performOptimization(TR_OptimizerImpl::Optimization*,int,int,int)+0xed639ae0 (0x0000000000000000)
TR_OptimizerImpl::performOptimization(TR_OptimizerImpl::Optimization*,int,int,int)+0xed639ae0 (0x0000000000000000)
TR_OptimizerImpl::optimize()+0xed63a250 (0x0000000000000000)
TR_Compilation::performOptimizations()+0xed77dfa0 (0x0000000000000000)
TR_Compilation::compile()+0xed77f400 (0x0000000000000000)
TR_CompilationInfoPerThreadBase::compile(J9VMThread*,TR_Compilation*,TR_Options*,TR_Resolve...+0xeda6b4e8 (0x0000000000000000)
TR_CompilationInfoPerThreadBase::wrappedCompile(J9PortLibrary*,CompileParameters*)+0xeda71318 (0x0000000000000000)
j9sig_protect+0x845d1d90 (0x0000000000000000)
TR_CompilationInfoPerThreadBase::compile(J9VMThread*,TR_MethodToBeCompiled*,bool)+0xeda763b0 (0x0000000000000000)
protectedCompilationThreadProc(J9PortLibrary*,TR_CompilationInfoPerThread*)+0xeda77868 (0x0000000000000000)
j9sig_protect+0x845d1d90 (0x0000000000000000)
compilationThreadProc(void*)+0xeda7abf0 (0x0000000000000000)
thread_wrapper+0xedb51ad8 (0x0000000000000000)

JVMDUMP039I Processing dump event "gpf", detail "" at 2013/03/18 12:17:07 - please wait.
JVMDUMP032I JVM requested System dump using 'JHNA.JVM.JHNA5.D130318.T121707.X&DS' in response to an event
IEATDUMP in progress with options SDATA=(LPA,GRSQ,LSQA,NUC,PSA,RGN,SQA,SUM,SWA,TRT)
IEATDUMP success for DSN='JHNA.JVM.JHNA5.D130318.T121707.X&DS'
JVMDUMP040I System dump written to dataset(s) using name template JHNA.JVM.JHNA5.D130318.T121707.X&DS
JVMDUMP032I JVM requested Java dump using '//javacore.20130318.121707.83951691.0002.txt' in response to an event
Unhandled exception
Type=Segmentation error vmState=0x000529ff
J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000035
Handler1=000000480871DCF8 Handler2=0000004808626890
gpr0=00000000000A0000 gpr1=000000488FAC9FBC gpr2=0000004890CE73CC gpr3=0000000000000000
...........................
(omited)
...........................
j9sig_protect+0x845d1d90 (0x0000000000000000)
TR_CompilationInfoPerThreadBase::compile(J9VMThread*,TR_MethodToBeCompiled*,bool)+0xeda763b0 (0x0000000000000000)
protectedCompilationThreadProc(J9PortLibrary*,TR_CompilationInfoPerThread*)+0xeda77868 (0x0000000000000000)
j9sig_protect+0x845d1d90 (0x0000000000000000)
compilationThreadProc(void*)+0xeda7abf0 (0x0000000000000000)
thread_wrapper+0xedb51ad8 (0x0000000000000000)

JVMDUMP039I Processing dump event "gpf", detail "" at 2013/03/18 12:17:14 - please wait.
JVMDUMP032I JVM requested System dump using 'JHNA.JVM.JHNA5.D130318.T121714.X&DS' in response to an event
IEATDUMP in progress with options SDATA=(LPA,GRSQ,LSQA,NUC,PSA,RGN,SQA,SUM,SWA,TRT)
IEATDUMP success for DSN='JHNA.JVM.JHNA5.D130318.T121714.X&DS'
JVMDUMP040I System dump written to dataset(s) using name template JHNA.JVM.JHNA5.D130318.T121714.X&DS
JVMDUMP032I JVM requested Java dump using '//javacore.20130318.121714.83951691.0004.txt' in response to an event
java version "1.6.0"
Java(TM) SE Runtime Environment (build pmz6460_26sr2ifix-20120419_02(SR2+IV19661))
IBM J9 VM (build 2.6, JRE 1.6.0 z/OS s390x-64 20120322_106210 (JIT enabled, AOT enabled)
J9VM - R26_Java626_SR2_20120322_1722_B106210
JIT - r11_20120322_22976
GC - R26_Java626_SR2_20120322_1722_B106210
J9CL - 20120322_106210)
JCL - 20120316_01
;;;
I am wondering why these messages are created and why java isn't run correctly and what I can do.

I found that it runs well with java 32bit and not with 64bit.

I wanna get your helps.
Updated on 2013-03-25T01:15:36Z at 2013-03-25T01:15:36Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    2736 Posts

    Re: Unhandled exception occured on USS on z/OS

    ‏2013-03-18T08:27:26Z  
    To verify whether the problem is related to JIT compiler or not. Could you run the program with -Xint.
    For example: java -Xint -version
  • SystemAdmin
    SystemAdmin
    2736 Posts

    Re: Unhandled exception occured on USS on z/OS

    ‏2013-03-18T08:38:01Z  
    To verify whether the problem is related to JIT compiler or not. Could you run the program with -Xint.
    For example: java -Xint -version
    I did it .

    :> java -Xint -version
    java version "1.6.0"
    Java(TM) SE Runtime Environment (build pmz6460_26sr2ifix-20120419_02(SR2+IV19661))
    IBM J9 VM (build 2.6, JRE 1.6.0 z/OS s390x-64 20120322_106210 (JIT disabled, AOT disabled)
    J9VM - R26_Java626_SR2_20120322_1722_B106210
    GC - R26_Java626_SR2_20120322_1722_B106210
    J9CL - 20120322_106210)
    JCL - 20120316_01

    It seems done well.
  • SystemAdmin
    SystemAdmin
    2736 Posts

    Re: Unhandled exception occured on USS on z/OS

    ‏2013-03-18T09:32:33Z  
    I did it .

    :> java -Xint -version
    java version "1.6.0"
    Java(TM) SE Runtime Environment (build pmz6460_26sr2ifix-20120419_02(SR2+IV19661))
    IBM J9 VM (build 2.6, JRE 1.6.0 z/OS s390x-64 20120322_106210 (JIT disabled, AOT disabled)
    J9VM - R26_Java626_SR2_20120322_1722_B106210
    GC - R26_Java626_SR2_20120322_1722_B106210
    J9CL - 20120322_106210)
    JCL - 20120316_01

    It seems done well.
    It seems to be the problem is something related to JIT compiler. This is the reason the JVM was able to initialize when we disable JIT compiler.

    To proceed further & verification,

    a) Reinstall the JDK (Since JIT compiler failing at the very initial stage of VM initialization which is unexpected & I doubt the installation)
    b) Install the latest JDK SR(If above step not successful then verify whether the problem is addressed in latest JDK level)
    c) Report the problem to IBM JDK service organisation through your support contract.
  • SystemAdmin
    SystemAdmin
    2736 Posts

    Re: Unhandled exception occured on USS on z/OS

    ‏2013-03-19T01:05:57Z  
    It seems to be the problem is something related to JIT compiler. This is the reason the JVM was able to initialize when we disable JIT compiler.

    To proceed further & verification,

    a) Reinstall the JDK (Since JIT compiler failing at the very initial stage of VM initialization which is unexpected & I doubt the installation)
    b) Install the latest JDK SR(If above step not successful then verify whether the problem is addressed in latest JDK level)
    c) Report the problem to IBM JDK service organisation through your support contract.
    Thank you for your answer.

    This JDK was installed automatically when WebSphere Application Server V8.5 for z/OS. It was included in WAS installation package.
    /usr/lpp/zWebSphere/V8R5/java64/...
    When I reinstall a new JDK to this system, will there be no problem?

    And one more thing I wonder:
    I installed a new JDK 1.7 after I installed the WAS for Z. And the 64bit module wasn't run correctly and 32bit module was run well also. Likewise, JDK 1.5 installed aleady on z/OS was run correctly with only 32 bit module.
    Regardless of JDK version, it seems that 64bit module isn't run correctly.

    I guess I will have to find solutions for this problem in z/OS USS system.
    Anyway, Thank you for your concern.
  • brataj
    brataj
    41 Posts

    Re: Unhandled exception occured on USS on z/OS

    ‏2013-03-22T13:05:10Z  
    Thank you for your answer.

    This JDK was installed automatically when WebSphere Application Server V8.5 for z/OS. It was included in WAS installation package.
    /usr/lpp/zWebSphere/V8R5/java64/...
    When I reinstall a new JDK to this system, will there be no problem?

    And one more thing I wonder:
    I installed a new JDK 1.7 after I installed the WAS for Z. And the 64bit module wasn't run correctly and 32bit module was run well also. Likewise, JDK 1.5 installed aleady on z/OS was run correctly with only 32 bit module.
    Regardless of JDK version, it seems that 64bit module isn't run correctly.

    I guess I will have to find solutions for this problem in z/OS USS system.
    Anyway, Thank you for your concern.
    If all your 64-bit Java's are failing, do check that you have a sufficient MEMLIMIT, which controls the amount of memory you can allocate above the bar (64-bit memory). As noted by ParamS, the failure is early in the JVM startup, so possibly it ran out of 64-bit memory at a sensitive point in the processing.

    See the SDK Guide via http://www-03.ibm.com/systems/z/os/zos/tools/java/products/j6pcont64.html
  • SystemAdmin
    SystemAdmin
    2736 Posts

    Re: Unhandled exception occured on USS on z/OS

    ‏2013-03-25T01:15:36Z  
    • brataj
    • ‏2013-03-22T13:05:10Z
    If all your 64-bit Java's are failing, do check that you have a sufficient MEMLIMIT, which controls the amount of memory you can allocate above the bar (64-bit memory). As noted by ParamS, the failure is early in the JVM startup, so possibly it ran out of 64-bit memory at a sensitive point in the processing.

    See the SDK Guide via http://www-03.ibm.com/systems/z/os/zos/tools/java/products/j6pcont64.html
    Thank you for your answer.

    I tried to run correctly with your answer, but no changes.

    If anyone has few time, could you connect to my system?
    My system is open to internet.
    If anyone having time notice to me, I'll send to him user id and password temporarily.
  • ArtyomSemenikhin
    ArtyomSemenikhin
    4 Posts

    Re: Unhandled exception occured on USS on z/OS

    ‏2013-09-04T11:28:53Z  
    • brataj
    • ‏2013-03-22T13:05:10Z
    If all your 64-bit Java's are failing, do check that you have a sufficient MEMLIMIT, which controls the amount of memory you can allocate above the bar (64-bit memory). As noted by ParamS, the failure is early in the JVM startup, so possibly it ran out of 64-bit memory at a sensitive point in the processing.

    See the SDK Guide via http://www-03.ibm.com/systems/z/os/zos/tools/java/products/j6pcont64.html

    hi,

    do you mean some JCL jobparm MEMLIMIT? or the one that is set by ulimit command?

  • brataj
    brataj
    41 Posts

    Re: Unhandled exception occured on USS on z/OS

    ‏2013-09-04T12:08:55Z  

    hi,

    do you mean some JCL jobparm MEMLIMIT? or the one that is set by ulimit command?

    By MEMLIMIT I meant whatever means of providing 64-bit memory is applicable to the environment in which you're running Java. This may be coding a MEMLIMIT on the jobcard, having a MEMLIMIT on your TSO logon proc etc.

  • ArtyomSemenikhin
    ArtyomSemenikhin
    4 Posts

    Re: Unhandled exception occured on USS on z/OS

    ‏2013-09-04T14:04:20Z  
    • brataj
    • ‏2013-09-04T12:08:55Z

    By MEMLIMIT I meant whatever means of providing 64-bit memory is applicable to the environment in which you're running Java. This may be coding a MEMLIMIT on the jobcard, having a MEMLIMIT on your TSO logon proc etc.

    I tried to re-install JDK 1.6.0.1 to the last available version however got the same error that is very similar that is described by topic starter..

    Any suggestions?

    javacore in an attachment.

  • brataj
    brataj
    41 Posts

    Re: Unhandled exception occured on USS on z/OS

    ‏2013-09-04T14:51:45Z  

    I tried to re-install JDK 1.6.0.1 to the last available version however got the same error that is very similar that is described by topic starter..

    Any suggestions?

    javacore in an attachment.

    There's nothing really obviously incorrect. I'd suggest trying again with a 4G MEMLIMIT and removing all mention of Java from the LIBPATH (where it is not required if using the java command).

    Also do mount the Java filesystem readonly; there have been a number of customers who inadvertantly overwrote parts.

  • ArtyomSemenikhin
    ArtyomSemenikhin
    4 Posts

    Re: Unhandled exception occured on USS on z/OS

    ‏2013-09-04T15:39:12Z  
    • brataj
    • ‏2013-09-04T14:51:45Z

    There's nothing really obviously incorrect. I'd suggest trying again with a 4G MEMLIMIT and removing all mention of Java from the LIBPATH (where it is not required if using the java command).

    Also do mount the Java filesystem readonly; there have been a number of customers who inadvertantly overwrote parts.

    hi, btw.. I'm using ADCD113S
    IBMUSER:/u/ibmuser: >ulimit -a
    core file         8192b
    cpu time          7203
    data size         unlimited
    file size         unlimited
    stack size        unlimited
    file descriptors  400
    address space     1606632k
    memory above bar  4096m


    IBMUSER:/Z113S/usr/lpp/java/J6.0.N_64/J6.0.1_64/bin: >echo $LIBPATH
    /lib:/usr/lib:.

    Same error :( probably I need some specific OMVS parameters enabled?

  • ArtyomSemenikhin
    ArtyomSemenikhin
    4 Posts

    Re: Unhandled exception occured on USS on z/OS

    ‏2013-09-05T13:18:53Z  

    hi, have you got your problem resolved? I have the same one..