IBM Support

IV36010: JMS APPLICATION CONTINUOUSLY RECEIVES 2016 (MQRC_GET_INHIBITED) ERROR WHICH LEADS TO A JAVA OUTOFMEMORYERROR

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • There was an infinite loop of MQExceptions reporting reason
    code 2016 (MQRC_GET_INHIBITED) over a period of time which lead
    to the OutOfMemoryError when running in WebSphere Application
    Server 7 using the WebSphere MQ Messaging provider.
    
    Heap dumps and java cores show a large number of Monitors:
    Example: 705 monitors of
    com/ibm/mq/jmqi/remote/internal/RemoteDispatchThread out of a
    total of 770 monitors.
    
    There are also unusually high number of dispatch threads.
    Threads are named "DispatchThread : 1" with its stack trace as
    follows
    
    3XMTHREADINFO      "DispatchThread: 1"
    J9VMThread:0x48799800,j9thread_t:0x5506FF04,
    java/lang/Thread:0x84DF2EB0, state:CW, prio=5
    3XMTHREADINFO1      (native thread ID:0x1EB0117, native
    priority:0x5, native policy:UNKNOWN)
    3XMTHREADINFO3      Java callstack:
    4XESTACKTRACE  at java/lang/Object.wait(Native Method)
    4XESTACKTRACE  at
    java/lang/Object.wait(Object.java:196(Compiled Code))
    4XESTACKTRACE  at
    com/ibm/mq/jmqi/remote/internal/RemoteDispatchThread.waitOnSleep
    ingEvent(RemoteDispatchThread.java:116(Compiled Code))
    4XESTACKTRACE  at
    com/ibm/mq/jmqi/remote/internal/RemoteDispatchThread.run(RemoteD
    ispatchThread.java:664(Compiled Code))
    4XESTACKTRACE  at
    com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.runTas
    k(WorkQueueItem.java:209(Compiled Code))
    4XESTACKTRACE  at
    com/ibm/msg/client/commonservices/workqueue/SimpleWorkQueueItem.
    runItem(SimpleWorkQueueItem.java:78(Compiled Code))
    4XESTACKTRACE  at
    com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.run(Wo
    rkQueueItem.java:224(Compiled Code))
    4XESTACKTRACE  at
    com/ibm/ws/wmqcsi/workqueue/WorkQueueManagerImpl$WorkQueueRunnab
    le.run(WorkQueueManagerImpl.java:648(Compiled Code))
    4XESTACKTRACE  at java/lang/Thread.run(Thread.java:736(Compiled
    Code))
    

Local fix

Problem summary

  • ****************************************************************
    USERS AFFECTED:
    This issue affects users of:
    
    - The WebSphere MQ V7 Resource Adapter.
    - The WebSphere Application Server V7 WebSphere MQ messaging
    provider.
    - The WebSphere Application Server V8 WebSphere MQ messaging
    provider.
    - The WebSphere Application Server V8.5 WebSphere MQ messaging
    provider.
    
    who are using an Message-Driven Bean (MDB) configured with a
    WebSphere MQ Activation Specification using WebSphere MQ
    messaging provider normal mode.
    
    Platforms affected:
    All Distributed (iSeries, all Unix and Windows) +Java +Java zOS
    ****************************************************************
    PROBLEM SUMMARY:
    The 'Get messages' property on a queue defined on the WebSphere
    MQ (WMQ) queue manager was changed to 'Inhibited'. This
    resulted in the WMQ Resource Adapter that was listening to that
    queue to fail with the MQRC_GET_INHIBITED error and the
    corresponding WMQ connection to be closed. The WMQ Resource
    Adapter reconnected and encountered the same exception again as
    the queue was still 'Inhibited'. This pattern then reoccurred
    over and over again until the WMQ queue property was changed to
    allow applications to get messages.
    
    The result of the constant reconnect and disconnect was an
    increase in the number of "DispatchThread: 1" threads within the
    Application Server JVM, causing a potential OutOfMemoryError.
    The number of client connections to the WMQ queue manager was
    also increased.
    

Problem conclusion

  • The WebSphere MQ classes for JMS have been update so that the
    WMQ Resource Adapter handles the MQRC_GET_INHIBITED return code
    correctly by keeping the WMQ connection open and waiting for the
    WMQ queue to be modified to allow applications to get messages.
    
    When the queue is inhibited the following message will be output
    to the Application Server log:
    
    JMSWMQ1119: The queue '[queue name]' is currently inhibited for
    get requests as the WebSphere MQ call returned with reason
    '2016' ('MQRC_GET_INHIBITED').
    
    Once the queue has been re-enabled the following message will be
    see in the Application Server log:
    
    JMSWMQ1118: The queue '[queue name]' that was previously
    inhibited for get has been re-enabled for get as the WebSphere
    MQ call returned with reason '2494' ('MQRC_GET_ENABLED')
    
    ---------------------------------------------------------------
    The fix is targeted for delivery in the following PTFs:
    
                       v7.0
    Platform           Fix Pack 7.0.1.11
    --------           --------------------
    Windows            7.0.1.11
    AIX                7.0.1.11
    HP-UX (PA-RISC)    7.0.1.11
    HP-UX (Itanium)    7.0.1.11
    Solaris (SPARC)    7.0.1.11
    Solaris (x86-64)   7.0.1.11
    iSeries            7.0.1.11
    Linux (x86)        7.0.1.11
    Linux (x86-64)     7.0.1.11
    Linux (zSeries)    7.0.1.11
    Linux (Power)      7.0.1.11
    zOS                7.0.1.11
    
                       v7.1
    Platform           Fix Pack 7.1.0.3
    --------           --------------------
    Windows            7.1.0.3
    AIX                7.1.0.3
    HP-UX (Itanium)    7.1.0.3
    Solaris (SPARC)    7.1.0.3
    Solaris (x86-64)   7.1.0.3
    iSeries            7.1.0.3
    Linux (x86)        7.1.0.3
    Linux (x86-64)     7.1.0.3
    Linux (zSeries)    7.1.0.3
    Linux (Power)      7.1.0.3
    zOS                7.1.0.3
    
    Platform           v7.5
    --------           --------------------
    Multiplatforms     7.5.0.1
    
    The latest available maintenance can be obtained from
    'WebSphere MQ Recommended Fixes'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
    
    If the maintenance level is not yet available information on
    its planned availability can be found in 'WebSphere MQ
    Planned Maintenance Release Dates'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
    ---------------------------------------------------------------
    

Temporary fix

Comments

APAR Information

  • APAR number

    IV36010

  • Reported component name

    WMQ AIX V7

  • Reported component ID

    5724H7221

  • Reported release

    701

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2013-01-28

  • Closed date

    2013-01-31

  • Last modified date

    2013-02-28

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WMQ AIX V7

  • Fixed component ID

    5724H7221

Applicable component levels

  • R701 PSY

       UP

[{"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSCPQ63","label":"APAR \/ Maintenance"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0.1","Edition":"","Line of Business":{"code":"","label":""}}]

Document Information

Modified date:
28 February 2013