Fixes are available
APAR status
Closed as program error.
Error description
There was an infinite loop of MQExceptions reporting reason code 2016 (MQRC_GET_INHIBITED) over a period of time which lead to the OutOfMemoryError when running in WebSphere Application Server 7 using the WebSphere MQ Messaging provider. Heap dumps and java cores show a large number of Monitors: Example: 705 monitors of com/ibm/mq/jmqi/remote/internal/RemoteDispatchThread out of a total of 770 monitors. There are also unusually high number of dispatch threads. Threads are named "DispatchThread : 1" with its stack trace as follows 3XMTHREADINFO "DispatchThread: 1" J9VMThread:0x48799800,j9thread_t:0x5506FF04, java/lang/Thread:0x84DF2EB0, state:CW, prio=5 3XMTHREADINFO1 (native thread ID:0x1EB0117, native priority:0x5, native policy:UNKNOWN) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at java/lang/Object.wait(Native Method) 4XESTACKTRACE at java/lang/Object.wait(Object.java:196(Compiled Code)) 4XESTACKTRACE at com/ibm/mq/jmqi/remote/internal/RemoteDispatchThread.waitOnSleep ingEvent(RemoteDispatchThread.java:116(Compiled Code)) 4XESTACKTRACE at com/ibm/mq/jmqi/remote/internal/RemoteDispatchThread.run(RemoteD ispatchThread.java:664(Compiled Code)) 4XESTACKTRACE at com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.runTas k(WorkQueueItem.java:209(Compiled Code)) 4XESTACKTRACE at com/ibm/msg/client/commonservices/workqueue/SimpleWorkQueueItem. runItem(SimpleWorkQueueItem.java:78(Compiled Code)) 4XESTACKTRACE at com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.run(Wo rkQueueItem.java:224(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/wmqcsi/workqueue/WorkQueueManagerImpl$WorkQueueRunnab le.run(WorkQueueManagerImpl.java:648(Compiled Code)) 4XESTACKTRACE at java/lang/Thread.run(Thread.java:736(Compiled Code))
Local fix
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of: - The WebSphere MQ V7 Resource Adapter. - The WebSphere Application Server V7 WebSphere MQ messaging provider. - The WebSphere Application Server V8 WebSphere MQ messaging provider. - The WebSphere Application Server V8.5 WebSphere MQ messaging provider. who are using an Message-Driven Bean (MDB) configured with a WebSphere MQ Activation Specification using WebSphere MQ messaging provider normal mode. Platforms affected: All Distributed (iSeries, all Unix and Windows) +Java +Java zOS **************************************************************** PROBLEM SUMMARY: The 'Get messages' property on a queue defined on the WebSphere MQ (WMQ) queue manager was changed to 'Inhibited'. This resulted in the WMQ Resource Adapter that was listening to that queue to fail with the MQRC_GET_INHIBITED error and the corresponding WMQ connection to be closed. The WMQ Resource Adapter reconnected and encountered the same exception again as the queue was still 'Inhibited'. This pattern then reoccurred over and over again until the WMQ queue property was changed to allow applications to get messages. The result of the constant reconnect and disconnect was an increase in the number of "DispatchThread: 1" threads within the Application Server JVM, causing a potential OutOfMemoryError. The number of client connections to the WMQ queue manager was also increased.
Problem conclusion
The WebSphere MQ classes for JMS have been update so that the WMQ Resource Adapter handles the MQRC_GET_INHIBITED return code correctly by keeping the WMQ connection open and waiting for the WMQ queue to be modified to allow applications to get messages. When the queue is inhibited the following message will be output to the Application Server log: JMSWMQ1119: The queue '[queue name]' is currently inhibited for get requests as the WebSphere MQ call returned with reason '2016' ('MQRC_GET_INHIBITED'). Once the queue has been re-enabled the following message will be see in the Application Server log: JMSWMQ1118: The queue '[queue name]' that was previously inhibited for get has been re-enabled for get as the WebSphere MQ call returned with reason '2494' ('MQRC_GET_ENABLED') --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: v7.0 Platform Fix Pack 7.0.1.11 -------- -------------------- Windows 7.0.1.11 AIX 7.0.1.11 HP-UX (PA-RISC) 7.0.1.11 HP-UX (Itanium) 7.0.1.11 Solaris (SPARC) 7.0.1.11 Solaris (x86-64) 7.0.1.11 iSeries 7.0.1.11 Linux (x86) 7.0.1.11 Linux (x86-64) 7.0.1.11 Linux (zSeries) 7.0.1.11 Linux (Power) 7.0.1.11 zOS 7.0.1.11 v7.1 Platform Fix Pack 7.1.0.3 -------- -------------------- Windows 7.1.0.3 AIX 7.1.0.3 HP-UX (Itanium) 7.1.0.3 Solaris (SPARC) 7.1.0.3 Solaris (x86-64) 7.1.0.3 iSeries 7.1.0.3 Linux (x86) 7.1.0.3 Linux (x86-64) 7.1.0.3 Linux (zSeries) 7.1.0.3 Linux (Power) 7.1.0.3 zOS 7.1.0.3 Platform v7.5 -------- -------------------- Multiplatforms 7.5.0.1 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IV36010
Reported component name
WMQ AIX V7
Reported component ID
5724H7221
Reported release
701
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2013-01-28
Closed date
2013-01-31
Last modified date
2013-02-28
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WMQ AIX V7
Fixed component ID
5724H7221
Applicable component levels
R701 PSY
UP
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSDEZSF","label":"IBM WebSphere MQ Managed File Transfer for z\/OS"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0.1","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
31 March 2023