PM35764: WLM INFINITE LOOP IN HASMAP

Fixes are available

APAR status

Closed as program error.

Error description

Customer reports an ABENDSEC3 RSN04130004 in the Adjunct region.
Dump shows a TCB ran a request in the Adjunct region for 60
minutes. The call stack for this TCB (native and java) is:

Entry      Offset     Function
-----      ------     --------
1c5aaed0   000020c4   CEEOPCT
1c88fed8   000000c0   pthread_cond_timedwait
1e983bb0   0000044e   monitor_wait_original
1e987700   00000084   j9thread_monitor_wait_abortable
1e961560   00000010   callMonitorWaitAbortable
1e8e9c08   00000418   Z_OBJECTMONITORENTERBLOCKING
1e94adf0   00000014   objectMonitorEnterBlocking
1e8db838   fffdcac4   RUNCALLINMETHOD
1e918058   0000003e   gpProtectedRunCallInMethod
1e917e30   0000001c   signalProtectAndRunGlue
1e9f1218   0000071a   j9sig_protect
1e91ae50   000000a4   gpCheckCallin
1e9135a8   0000005e   callVirtualObjectMethodA
1cbbc5f8   00001230   ORBEJSBridge::invoke(void*)
1cd40988   00000ea8   threadDispatch(BOSS_Object_Key*,
                      Internal_CORBA_Request&,ORB_Request*)
1cd55060   00001ba2   SR_ExecutionThread::RemoveAndProcessWork(
                      ThreadCleanUp*,TCB*)
1cd57ff8   000001b8   SR_ExecutionRoutine
1c5a6df0   00000f50   CELQPCMM

Java:

com/ibm/ws/sib/utils/ThreadLocalStack.push
    ThreadLocalStack.java:87
com/ibm/ws/sib/utils/ras/SibTr.push
    SibTr.java:448
com/ibm/ws/sib/admin/impl/HAManagerMessagingEngineImpl.membershi
    pChanged
    HAManagerMessagingEngineImpl.java:669
com/ibm/ws/hamanager/proxy/DispatchHAGroupCallbackImpl.membershi
    pChanged
    DispatchHAGroupCallbackImpl.java:166
com/ibm/ws/hamanager/proxy/_DispatchHAGroupCallbackImplBase._inv
    oke
    _DispatchHAGroupCallbackImplBase.java:140
com/ibm/ws390/orb/CommonBridge.invoke
    CommonBridge.java:1190
com/ibm/ws390/orb/ORBEJSBridge.invoke
    ORBEJSBridge.java:270
sun/reflect/GeneratedMethodAccessor33.invoke
sun/reflect/DelegatingMethodAccessorImpl.invoke
    DelegatingMethodAccessorImpl.java:37
java/lang/reflect/Method.invoke
    Method.java:600
com/ibm/ws390/orb/parameters/ORBEJSBridgeInvoke.ORBEJSBridgeInvo
    keParmSetter
    ORBEJSBridgeInvoke.java:186

The extracted javacore shows TCB 8B64F8 is waiting for a monitor
owned by the SIB Activation Thread on TCB 8b5a48 (again, in the
adjunct)

2LKMONINUSE      sys_mon_t:0x0000000A352CC700 infl_mon_t:
                    0x0000000A352CC770:
3LKMONOBJECT
com/ibm/ws/sib/admin/impl/JsMessagingEngineImpl
                      @0x000000081CBB92E0/0x000000081CBB92F8:
                      Flat locked by "SIB Activation Thread"
                      (0x000000080943ED00), entry count 1
3LKWAITERQ            Waiting to enter:
3LKWAITER                "WebSphere t=008b64f8"
(0x0000000809428600)

TCB 8B5A48 has gone outbound with a request:

TCB 8b5a48 caa c13dfecf0 pthread_id 1d4579000000007a
           SIB Activation Thread J9VMThread 80943ed00 (daemon)

Dsa         Entry      Offset     Function
---         -----      ------     --------
c13bfd1e0   1cf065d0   00000000
bboosout(BBOOSOUT_Functions,ORB_Request_SharedMemberData*,long
long,int)
c13bfd4e0   1cd7b480   00000594
ORB_Request::comm_outbound_request()
c13bfd720   1cdce160   00000c0e
Java_com_ibm_ws390_orb_ClientDelegate_invokeRequestCFW
c13bfe420   1e8dbdf0   e1724210   RUNJAVATHREAD
c13bfe860   1e953ae8   000000e0   javaProtectedThreadProc
c13bfe960   1e9f1218   0000071a   j9sig_protect
c13bff160   1e9565e8   00000072   javaThreadProc
c13bff260   1e9855e0   0000051c   thread_wrapper
c13bff360   1c5a6df0   00000f50   CELQPCMM

Java stack:
-----------
com/ibm/ws390/orb/ClientDelegate.invokeRequestCFW
    ClientDelegate.java
com/ibm/ws390/orb/ClientDelegate.commonInvoke
    ClientDelegate.java:1018
com/ibm/ws390/orb/ClientDelegate.invoke
    ClientDelegate.java:865
org/omg/CORBA/portable/ObjectImpl._invoke
    ObjectImpl.java:484
com/ibm/ws/cluster/zos/_ControllerClusterManagerImpl_Stub.publis
    h
    _ControllerClusterManagerImpl_Stub.java:544
com/ibm/ws/cluster/zos/SRDescriptionManager.publish
    SRDescriptionManager.java:287
com/ibm/ws/cluster/zos/SRDescriptionManager.publish
    SRDescriptionManager.java:364
com/ibm/ws/cluster/zos/ZDescriptionManager.publish
    ZDescriptionManager.java:196
com/ibm/websphere/cluster/topography/DescriptionA.updateMemento
    DescriptionA.java:160
com/ibm/ws/cluster/topography/ClusterDescriptionImpl.addMember
    ClusterDescriptionImpl.java:387
com/ibm/ws/cluster/service/ClusterManagementImpl.joinCluster
    ClusterManagementImpl.java:283
com/ibm/ws/cluster/zos/ZosClusterManagementImpl.joinCluster
    ZosClusterManagementImpl.java:108
com/ibm/ws/cluster/service/ClusterMemberServiceImpl.joinCluster
    ClusterMemberServiceImpl.java:503
com/ibm/ws/sib/trm/wlm/server/Join.join
    Join.java:211
com/ibm/ws/sib/trm/wlm/server/Join.toShortToLongName
    Join.java:136
com/ibm/ws/sib/trm/wlm/server/Manager.start
    Manager.java:144
com/ibm/ws/sib/trm/TrmMeMainImpl.start
    TrmMeMainImpl.java:266
com/ibm/ws/sib/admin/impl/JsMessagingEngineImpl.start
    JsMessagingEngineImpl.java:609
com/ibm/ws/sib/admin/impl/HAManagerMessagingEngineImpl.activate
    HAManagerMessagingEngineImpl.java:995
com/ibm/ws/sib/admin/impl/JsActivationThread.run
    JsActivationThread.java:92

THIS IS THE LOOPING TCB in the Controller region, processing the
request on
TCB=8b5a48 TCB 8b6e00 caa a4f7fecf0 pthread_id 1d952000000006d
WebSphere t=008b6e00
J9VMThread 8098df000 (daemon)

Dsa         Entry      Offset     Function
---         -----      ------     --------
b2fafa4e0   1fe29838   fd722688   RUNCALLINMETHOD
b2fafa920   1fe66058   0000003e   gpProtectedRunCallInMethod
b2fafaa20   1fe65e30   0000001c   signalProtectAndRunGlue
b2fafab20   1ff3f218   0000071a   j9sig_protect
b2fafb320   1fe68e50   000000a4   gpCheckCallin
b2fafb460   1fe615a8   0000005e   callVirtualObjectMethodA
b2fafb560   1cbc15f8   00001230   ORBEJSBridge::invoke(void*)
b2fafc560   1cd45988   00000ea8 threadDispatch(BOSS_Object_Key*,
            Internal_CORBA_Request&, ORB_Request*)
b2fafd1a0   1cd47ba8   00000a56   ACR_ExecutionThread::
            ProcessInboundRequest(acrwObj*,
            ThreadCleanUp*,BOSS_Object_Key&,  Interna...
b2fafe3a0   1cd577e8   0000086e   ACR_ExecutionThread::
            RemoveAndProcessWork(ThreadCleanUp*)
b2faff220   1cd599c8   0000022e   ACR_ExecutionRoutine
b2faff360   1c5a9df0   00000f50   CELQPCMM

Java stack:
-----------
java/util/HashMap.findNonNullKeyEntry
    HashMap.java:526
java/util/HashMap.putImpl
    HashMap.java:622
java/util/HashMap.put
    HashMap.java:605
java/util/HashSet.add
    HashSet.java:95
com/ibm/ws/cluster/zos/CRDescriptionManagerImpl.publish
    CRDescriptionManagerImpl.java:206
com/ibm/ws/cluster/zos/ControllerClusterManagerImpl.publish
    ControllerClusterManagerImpl.java:582
com/ibm/ws/cluster/zos/_ControllerClusterManagerImpl_Tie.publish
    _ControllerClusterManagerImpl_Tie.java:255
com/ibm/ws/cluster/zos/_ControllerClusterManagerImpl_Tie._invoke
    _ControllerClusterManagerImpl_Tie.java:84
sun/reflect/GeneratedMethodAccessor15.invoke
sun/reflect/DelegatingMethodAccessorImpl.invoke
    DelegatingMethodAccessorImpl.java:37
java/lang/reflect/Method.invoke
    Method.java:600
com/ibm/ws390/orb/RIRProtectedProxyHandler.invoke
    RIRProtectedProxyHandler.java:186
$Proxy0._invoke
com/ibm/ws390/orb/CommonBridge.invoke
    CommonBridge.java:1190
com/ibm/ws390/orb/ORBEJSBridge.invoke
    ORBEJSBridge.java:270
sun/reflect/GeneratedMethodAccessor11.invoke
sun/reflect/DelegatingMethodAccessorImpl.invoke
    DelegatingMethodAccessorImpl.java:37
java/lang/reflect/Method.invoke
    Method.java:600
com/ibm/ws390/orb/parameters/ORBEJSBridgeInvoke.ORBEJSBridgeInvo
    keParmSetter
    ORBEJSBridgeInvoke.java:186

Evidence suggest the loop on TCB=8b5a48 ran for 1 hour.

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:  All users of IBM WebSphere Application      *
*                  Server z/OS on versions 7.0 & 8.0           *
****************************************************************
* PROBLEM DESCRIPTION: Infinite loop in HashMap                *
****************************************************************
                                                               *
****************************************************************
Unprotected concurrent access to a hashmap in the WLM class
com/ibm/ws/cluster/zos/CRDescriptionManagerImpl can cause an
infinite loop in the underlying HashMap inplementation.  A
customer hitting this problem would see symptoms as described
above, any stack should have the following piece present:
com/ibm/ws/cluster/zos/CRDescriptionManagerImpl.publish
CRDescriptionManagerImpl.java:206

Problem conclusion

The WLM code in question had synchronization added around the
usage of the hashmap to prevent the infinite loop from
occurring.

APAR PM35764 is currently targeted for inclusion in
Service Level (Fix Pack) 7.0.0.19, 8.0.0.1 of WebSphere
Application Server

Please refer to the Recommended Updates page for delivery
information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

Temporary fix

Comments

APAR Information

APAR number
PM35764
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
700
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2011-03-28
Closed date
2011-07-08
Last modified date
2011-10-04

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500

Applicable component levels

R700 PSY UK71280
UP11/09/10 P F109

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
27 October 2021

Tips

PM35764: WLM INFINITE LOOP IN HASMAP

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R700 PSY UK71280

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

Document Information

Share your feedback

Need support?