APAR status
Closed as program error.
Error description
A WebSphere MQ messaging provider activation specification in WebSphere Application Server for z/OS may hang while being paused due to a logical deadlock between two threads. In a Javacore you will see the following lock information: 3LKMONOBJECT com/ibm/mq/connector/inbound/ ServerSessionImpl$IsInUseLock@0x 000000482E732E40/0x000000482E732E58: 3LKNOTIFYQ Waiting to be notified: 3LKWAITNOTIFY "WebSphere t=00ab8408" (0x0000004808CDFC00) 2LKMONINUSE sys_mon_t:0x0000004812E59AC0 infl_mon_t: 0x0000004812E59B30: This is followed by thread information where two threads are shown to be in the following state: WebSphere t=00ab8408" J9VMThread:0x0000004808CDFC00, Java callstack: at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:167 (Compiled Code)) at com/ibm/mq/connector/inbound/ ServerSessionImpl.close(ServerSessionImpl.java:299) at com/ibm/mq/connector/inbound/ServerSessionPoolImpl. closeInternal(ServerSessionPoolImpl.java:648) at com/ibm/mq/connector/inbound/ServerSessionPoolImpl. close(ServerSessionPoolImpl.java:557) at com/ibm/mq/connector/inbound/MessageEndpointDeployment. stop(MessageEndpointDeployment.java:473) at com/ibm/mq/connector/ResourceAdapterImpl. endpointDeactivation(ResourceAdapterImpl.java:523) and "WebSphere:ORB.thread.pool t=00ac0528" Java callstack: at com/ibm/ws390/orb/ClientDelegate. invokeRequestCFW(Native Method) at com/ibm/ws390/orb/ClientDelegate. commonInvoke(ClientDelegate.java:1042 (Compiled Code)) at com/ibm/ws390/orb/ClientDelegate. invoke(ClientDelegate.java:889) at org/omg/CORBA/portable/ObjectImpl. _invoke(ObjectImpl.java:484) at com/ibm/ws390/management/connector/corba/ _CorbaConnectorStub.invoke (_CorbaConnectorStub.java:634) at com/ibm/ws390/management/connector/corba/ CorbaConnectorClient.invoke (CorbaConnectorClient.java:903) at com/ibm/ws390/management/InterProcessAdminService. invoke(InterProcessAdminService.java:371) at com/ibm/ws/management/AdminServiceImpl$1. run(AdminServiceImpl.java:1432) at com/ibm/ws/security/util/AccessController. doPrivileged(AccessController.java:118 (Compiled Code)) at com/ibm/ws/management/AdminServiceImpl. invoke(AdminServiceImpl.java:1224) at com/ibm/ws/wmqra/utils/EndpointPauser$1.run (EndpointPauser.java:99) at com/ibm/ws/sib/security/auth/AuthUtilsImpl$5. run(AuthUtilsImpl.java:1454) at com/ibm/ws/security/auth/ContextManagerImpl. runAs(ContextManagerImpl.java:5365) at com/ibm/ws/security/auth/ContextManagerImpl. runAsSystem(ContextManagerImpl.java:5453) at com/ibm/ws/sib/security/auth/AuthUtilsImpl. runAsSystem(AuthUtilsImpl.java:1449) at com/ibm/ws/wmqra/utils/EndpointPauser. pauseJ2CMessageEndpoint(EndpointPauser.java:96) at com/ibm/ws/wmqra/helper/WASRuntimeHelper. deliveryFailed(WASRuntimeHelper.java:326) at com/ibm/mq/connector/inbound/WorkImpl. callDeliveryFailed(WorkImpl.java:511) The Job Log might also report the threads as hung, as per WebSphere Application Server hung thread detection processing. For example: BBOO0221W: WSVR0605W: Thread "WebSphere WLM Dispatch Thread t=00ac0528" (00000011) has been active for 706600 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
Local fix
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of: - The WebSphere Application Server for z/OS V7 WebSphere MQ messaging provider. - The WebSphere Application Server for z/OS V8 WebSphere MQ messaging provider. - The WebSphere Application Server for z/OS V8.5 WebSphere MQ messaging provider. who are using an activation specification to drive a Message-Driven Bean (MDB) application which consumes messages from WebSphere MQ. Platforms affected: z/OS **************************************************************** PROBLEM DESCRIPTION: The default configuration of an activation specification defined against the WebSphere MQ messaging provider within WebSphere Application Server is to stop the messaging endpoint on the first failure to deliver a message. This behaviour is configurable using the properties: - Stop endpoint if message delivery fails and - Number of sequential delivery failures before suspending endpoint Using the default configuration, if a Message-Driven Bean (MDB) application rolls a message back because it could not be processed, WebSphere Application Server will attempt to stop the activation specification. When this is requested, the activation specification waits for all current activity (included queued up work) to complete, either successfully or due to an error, before stopping. It was observed that when the activation specification on WebSphere Application Server for z/OS attempted to pause, it ended up hanging because of two Java threads were in a logical deadlock processing the stop request. The scenario was as follows. A thread that was running a Server Session and had driven the MDB application with a message requested the endpoint to stop. For WebSphere Application Server for z/OS, this thread was not necessarily the thread that performed the actual stop of the endpoint and the subsequent closure of the Server Session Pool associated with the activation specification. In this case, the thread associated with the Server Session waited for the deactivation of the endpoint to complete, while the Server Session was still marked as "in use". However, the thread that was processing the endpoint deactivation waited for all Server Sessions from the activation specification's Server Session Pool to complete their processing and to be returned to the pool. This resulted in the thread stopping the endpoint waiting for a Server Session to be returned to the pool, which was busy waiting for the processing of stopping the endpoint to complete.
Problem conclusion
The WebSphere MQ resource adapter code has been changed to ensure when WebSphere Application Server for z/OS requests a stop of the messaging endpoint (activation specification), the thread processing the stop request does not wait for the Server Session that requested the stop to complete before pausing. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v7.0 7.0.1.14 v7.1 7.1.0.7 v7.5 7.5.0.6 v8.0 8.0.0.3 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT06660
Reported component name
WMQ WINDOWS V7
Reported component ID
5724H7220
Reported release
710
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2015-01-26
Closed date
2015-03-30
Last modified date
2015-03-30
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
PI35752
Fix information
Fixed component name
WMQ WINDOWS V7
Fixed component ID
5724H7220
Applicable component levels
R710 PSY
UP
[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSFKSJ","label":"WebSphere MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.1"}]
Document Information
Modified date:
09 March 2021