IBM Support

Why does my JVMSERVER remain in a "BEING DISABLED" state?

Question & Answer


Question

When I try to disable a JVMSERVER it becomes hung and remains in a "BEING DISABLED" state. This also occurs when I shut down the CICS region. The CICS region ultimately has to be cancelled.

Answer

Looking at the Kernel Summary you see the CEMT is running and it is running on the TP001 the TCB.

 ===KE: Kernel Domain KE_TASK Summary
 
 KE_NUM KE_TASK  STATUS       TCA_ADDR TRAN_# TRANSID KE_KTCB  TCB    
 00F3   1A895100 ***Running** 0004B080 02723  CEMT    19170F68 008ACE88 
 002B   1C08C100 KTCB TP001   00000000                19170F68 008ACE88
                                                                   

Kernel stack shows that the CEMT task is TERMINATE_JVM_ENCLAVE:

 KE_NUM @STACK   LEN  TYPE ADDRESS  LINK REG OFFSET ERR NAME                  
                                                                              
 00F3   1A896040 01E0 Bot  18E03C00 98E04230 000630     DFHKETA               
 00F3   1A896220 03E0 Dom  18E203B8 98E205FC 000244     DFHDSKE               
 00F3   1A896600 1130 Dom  18E54438 98E55952 00151A     DFHXMTA               
 00F3   1A897730 0AB0 Dom  19526E08 99528120 001318     DFHPGPG               
                      Int   +000374 99526FA4 00019C     INITIAL_LINK          
 00F3   1A8981E0 0E90 Dom  1989CD00 998A933A 00C63A     DFHAPLI1              
                      Int   +002AA2 9989DA3E 000D3E     CICS_INTERFACE        
 00F3   1A899070 0FB8 Lifo 198A9100 998AAC92 001B92     DFHEPC                
 00F3   1A89A028 0DA0 Dom  19519300 9951AA74 001774     DFHPGLE               
                      Int   +0005CC 995194CE 0001CE     LINK_EXEC             
 00F3   1A89ADC8 0E90 Dom  1989CD00 80082CDE 000000     DFHAPLI1              
                      Int   +002AA2 9989DA3E 000D3E     CICS_INTERFACE        
 00F3   1A89BC58 0D20 Lifo 19F17E00 99F193BA 0015BA     DFHEIQSY              
 00F3   1A644040 1D70 Dom  197384B8 99741086 008BCE     DFHSJJS               
                      Int   +00531E 99738744 00028C     SET_JVMSERVER         
                      Int   +007B8A 9973E7A0 0062E8     DISABLE_JVMSERVER
 00F3   1A85F040 0BA0 Dom  18E94198 98E957B0 001618     DFHDSIT               
                      Int   +00055E 98E94334 00019C     PROCESS               
                      Int   +00148A 98E9475E 0005C6     DELETE_TCB            
 00F3   1A85FBE0 07A0 Dom  19728AF0 997295CE 000ADE     DFHSJIN               
                      Int   +000A78 99728EFA 00040A     NOTIFY_DELETE_TCB
 00F3   1A860380 05F0 Dom  19727DC0 99728126 000366     DFHSJDS               
                      Int   +000224 99727F4E 00018E     NOTIFY_DELETE_TCB
 00F3   1A6DE040 07A0 Dom  19728AF0 99728CFA 00020A     DFHSJIN               
                      Int   +001EC2 99728ED8 0003E8     TERMINATE_JVM_ENCLAVE

Looking at the traceback for the IPT TCB 008ACB88, it shows the DestroyJavaVM was called to unload the JVM but ended up waiting. This function will wait until the current thread is the only non-daemon user thread before it destroys the JVM.

 Traceback:                                                             
   DSA      Entry                    E  Offset                          
                                                                        
   1        CEEOPCW                  +00002014                          
   2        pthread_cond_wait        +00000152                          
   3        monitor_wait_original    +00000B42                          
   4        j9thread_monitor_wait    +00000070                          
   5        protectedDestroyJavaVM   +000000F2                          
   6        j9sig_protect            +0000076E                          
   7        DestroyJavaVM            +000003B4                          
   8        DestroyJavaVM            +0000003C                        
   9        terminate_jvm            +00000110                        
   10       ipt_control_loop         +00000606
   11       CELQPCAS                 +000016F6 
                    

Using the jdmpview tool to list out the Java threads from the dump I can see that thread id 1976af00 is marked as a non-daemon thread.

 thread id: 0x 1976af00                                               
 TCB=0x8ad8b0                                                         
 associated Java thread:                                              
  name:          Thread-5                                             
  Thread object: com/ibm/CORBA/iiop/KeepAlive @ 0x48a6ee3ea8          
  Daemon:        false                                                
  JVMTI state:   ALIVE WAITING WAITING_INDEFINITELY IN_OBJECT_WAIT
                 waiting to be notified on:                       
                 com/ibm/CORBA/iiop/KeepAlive@0x48a6ee3ea8 

This corba keepalive thread is indeed what is preventing the JVM from terminating normally.

From the dump submitted, jdmpview info proc reveals the java level is -Djava.runtime.version=pmz6470sr5-20130619_01 (SR5)

and also that IBM Heath Center is enabled

-Xhealthcenter:port=1972

The corba thread noted has been seen before when IBM Health Center is enabled. APAR IV53219 addresses a shutdown error, and the fix was shipped within PTF UI14491 (java7 sr6 fp1) .

The PTF can be applied, or delete (or comment out) the -Xhealthcenter option to disable IBM Health Center to resolve the problem.

[{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SSGMGV","label":"CICS Transaction Server"},"Platform":[{"code":"PF035","label":"z\/OS"}],"Component":"Java","Version":"","Line of Business":{"code":"LOB17","label":"Mainframe TPS"}}]

Product Synonym

CICS/TS CICSTS CICS TS CICS Transaction Server

Document Information

Modified date:
26 August 2015

UID

dwa1190901