IV26267: WEBSPHERE APPLICATION SERVER V7 REPORTS HUNG THREADS WHEN PAUSING WEBSPHERE MQ ACTIVATION SPECIFICATIONS

WebSphere MQ V7.5 Fix Pack 7.5.0.1
WebSphere MQ V7.0 Fix Pack 7.0.1.10
Fix Pack 7.1.0.3 for WebSphere MQ V7.1

APAR status

• Closed as program error.

Error description

• A Message Driven Bean(MDB) running on WebSphere Application
  Server(WAS) v7.0.x fails to process messages that are received
  from WebSphere MQ v7.0.x.This results in a hang situation.
  
  The following errors are written to WAS SystemOut.log:
  CWWMQ0007W: The message endpoint XXX has been paused by the
  system.  Message delivery failed to the endpoint more than 2
  times.The last attempted delivery failed with the following
  error: javax.jms.TransactionRolledBackException:
  at com.ibm.mq.connector.inbound.WorkImpl.call
    DeliveryFailed(WorkImpl.java:511)
  at com.ibm.mq.connector.inbound.WorkImpl.xa
    StateChanged(WorkImpl.java:460)
  at com.ibm.mq.connector.xa.XAObservable.
    update(XAObservable.java:120)
  at com.ibm.mq.connector.xa.XARWrapper.
    rollback(XARWrapper.java:444)
  at com.ibm.tx.jta.JTAXAResourceImpl.
    rollback(JTAXAResourceImpl.java:363)
  at com.ibm.tx.jta.RegisteredResources.
    deliverOutcome(RegisteredResources.java:1647)
  
  J2CA0140W: The Message Endpoint pause operation failed for
  ActivationSpec XXX and MDB Application XYZ because the
  endpoint is currently deactivated or stopped, .or an
  unexpected exception occurred deactivating the endpoint.
  
  J2CA0530I: The Message Endpoint for ActivationSpec XXX and
  MDB Application XYZ is partially activated but has been
  disabled from handling incoming messages.
  
  J2CA0520W: The Message Endpoint resume operation failed for
  ActivationSpec XXX and MDB Application XYZ because the
  endpoint is currently activated or stopped, or an unexpected
  exception occurred activating the endpoint.
  
  javacores show hung threads with the following stack:
  Blocking Thread:Thread:1:
  at java/lang/Object.wait(Native Method)
  at java/lang/Object.wait(Object.java:167(Compiled Code))
  at com/ibm/mq/connector/inbound/ServerSessionImpl.close
    (ServerSessionImpl.java:298)
  at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.
    closeInternal(ServerSessionPoolImpl.java:648)
  at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.
    close(ServerSessionPoolImpl.java:557)
  at com/ibm/mq/connector/inbound/MessageEndpointDeployment.
    stop(MessageEndpointDeployment.java:455)
  at com/ibm/mq/connector/ResourceAdapterImpl.endpoint
    Deactivation(ResourceAdapterImpl.java:523)
  at com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivate
    UnderRAClassLoaderContext(ActivationSpecWrapperImpl.
    java:508)
  at com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivate
    EndPoint(ActivationSpecWrapperImpl.java:411)
  at com/ibm/ejs/j2c/mbeans/MessageEndpointMBeanImpl.pause
    (MessageEndpointMBeanImpl.java:162)
  
  Blocked thread:Thread:2
  at com/ibm/mq/connector/ResourceAdapterImpl.endpoint
    Deactivation(ResourceAdapterImpl.java:516)
  at com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivate
    UnderRAClassLoaderContext(ActivationSpecWrapperImpl.
    java:508)
  at com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivate
    EndPoint(ActivationSpecWrapperImpl.java:411)
  at com/ibm/ejs/j2c/mbeans/MessageEndpointMBeanImpl.pause
    (MessageEndpointMBeanImpl.java:162)
  

Local fix

Problem summary

• ****************************************************************
  USERS AFFECTED:
  This issue affects users of the WebSphere MQ Messaging Provider
  supplied with WebSphere Application Server, who have configured
  one or more Activation Specifications for the Messaging
  Provider, and are attempting to pause the activation
  specification(s) either manually using the application server
  administration console, or automatically as a result of a
  delivery failure.
  
  Platforms affected:
  All Distributed (iSeries, all Unix and Windows) +Java +Java zOS
  ****************************************************************
  PROBLEM SUMMARY:
  When delivery of a message to a message driven bean (MDB)
  fails, or the message driven bean reports an exception during
  message processing, the WebSphere MQ Messaging Provider
  notifies the application server of the failure event, in order
  to allow the application server to make the appropriate action.
  At this point, if the application server configuration permits,
  the application server may respond with the decision to pause
  the activation specification.
  
  When pausing an activation specification, the thread performing
  the pause must wait for all other threads associated with the
  Activation Specification to complete their current work before
  the pause operation can be completed.
  
  In the case where multiple threads are associated with the
  Activation Specification concurrently receive notification of a
  failed delivery, all threads receiving this notification will
  also receive the pause decision from the application server.
  This results in multiple threads simultaneously attempting to
  pause the Activation Specification.
  
  This potentially lead to a deadlock whereby multiple threads
  are waiting for each other to complete so that they can
  complete complete the pause operation.
  
  The same behaviour was also be observed if a user attempted to
  pause the Activation Specification from the WebSphere
  Application Server administration console at the same time as
  the application server was notifying an Activation
  Specification thread of a failed delivery.
  

Problem conclusion

• The WebSphere MQ Messaging Provider code has been modified so
  that only one thread may concurrently attempt to pause a
  particular Activation Specification. This allows all other
  threads associated with the Activation Specification to complete
  as normal before the pause operation takes place, avoiding the
  deadlock scenario.
  
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
                     v7.0
  Platform           Fix Pack 7.0.1.10
  --------           --------------------
  Windows            U200352
  AIX                U853055
  HP-UX (PA-RISC)    U853082
  HP-UX (Itanium)    U853087
  Solaris (SPARC)    U853083
  Solaris (x86-64)   U853089
  iSeries            7.0.1.10
  Linux (x86)        U853084
  Linux (x86-64)     U853088
  Linux (zSeries)    U853085
  Linux (Power)      U853086
  zOS                7.0.1.10
  
                     v7.1
  Platform           Fix Pack 7.1.0.3
  --------           --------------------
  Windows            7.1.0.3
  AIX                7.1.0.3
  HP-UX (Itanium)    7.1.0.3
  Solaris (SPARC)    7.1.0.3
  Solaris (x86-64)   7.1.0.3
  iSeries            7.1.0.3
  Linux (x86)        7.1.0.3
  Linux (x86-64)     7.1.0.3
  Linux (zSeries)    7.1.0.3
  Linux (Power)      7.1.0.3
  zOS                7.1.0.3
  
  Platform           v7.5
  --------           --------------------
  Multiplatforms     7.5.0.1
  
  The latest available maintenance can be obtained from
  'WebSphere MQ Recommended Fixes'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
  
  If the maintenance level is not yet available information on
  its planned availability can be found in 'WebSphere MQ
  Planned Maintenance Release Dates'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  PI15088

Fix information

• Fixed component name

  WMQ LIN X86-64

• Fixed component ID

  5724H7230

Applicable component levels

• R700 PSY

     UP