APAR status
Closed as program error.
Error description
After running for a long period of time, an IBM MQ AMQP channel stops unexpectedly and an FDC containing the following information is generated: ************************************************************* ... | Source Class :- com.ibm.mq.MQXRService.AMQPServerSessionV10 | Source Method :- processProtonUpdates | Probe Id :- XR10072014 ... | Thread :- ServerWorker1 ... :--------------------------------------------------------------- -----------------------------------: : Exception cause: : :--------------------------------------------------------------- -----------------------------------: java.lang.OutOfMemoryError: Java heap space at java.lang.StringCoding.encode(StringCoding.java:...) at java.lang.String.getBytes(String.java:786) at java.io.UnixFileSystem.getBooleanAttributes0(Native Method) at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java: ...) at java.io.File.exists(File.java:...) at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:. ..) at sun.misc.URLClassPath$FileLoader.findResource(URLClassPath.java: ...) at sun.misc.URLClassPath.findResource(URLClassPath.java:...) at java.net.URLClassLoader$1.run(URLClassLoader.java:...) at java.net.URLClassLoader$1.run(URLClassLoader.java:...) at java.security.AccessController.doPrivileged(AccessController.jav a:...) at java.net.URLClassLoader.findResource(URLClassLoader.java:...) at java.lang.ClassLoader.getResource(ClassLoader.java:...) at java.lang.Class.getResource(Class.java:...) at com.ibm.mq.AMQP.AMQPCompInfo$1.run(AMQPCompInfo.java:...) at java.security.AccessController.doPrivileged(AccessController.jav a:...) at com.ibm.mq.AMQP.AMQPCompInfo.<init>(AMQPCompInfo.java:...) at com.ibm.mq.MQXRService.AMQPServerSessionV10.receiveMessage(AMQPS erverSessionV10.java:...) at com.ibm.mq.MQXRService.AMQPServerSessionV10.processProtonUpdates (AMQPServerSessionV10.java:...) at com.ibm.mq.MQXRService.AMQPServerSessionV10.handleReceive(AMQPSe rverSessionV10.java:...) at com.ibm.mq.MQXRService.AMQPServerWireContext.receive(AMQPServerW ireContext.java:...) at com.ibm.mq.communications.NonBlockingConnection.receive(NonBlock ingConnection.java:...) at com.ibm.mq.communications.NonBlockingWorker.run(NonBlockingWorke r.java:...) at java.lang.Thread.run(Thread.java:...) :--------------------------------------------------------------- -----------------------------------: : Object :- java.lang.OutOfMemoryError: Java heap space : : Class@Hashcode :- java.lang.OutOfMemoryError@87a8347b : :--------------------------------------------------------------- -----------------------------------: :--------------------------------------------------------------- -----------------------------------: : FFDC Origin: : :--------------------------------------------------------------- -----------------------------------: com.ibm.mq.util.logging.FFDCFileHandler.publish(FFDCFileHandler. java:...) com.ibm.mq.util.logging.FFDC.print(FFDC.java:...) com.ibm.mq.util.logging.FFDC.processException(FFDC.java:...) com.ibm.mq.MQXRService.AMQPServerSessionV10.processProtonUpdates (AMQPServerSessionV10.java:...) com.ibm.mq.MQXRService.AMQPServerSessionV10.handleReceive(AMQPSe rverSessionV10.java:...) com.ibm.mq.MQXRService.AMQPServerWireContext.receive(AMQPServerW ireContext.java:...) com.ibm.mq.communications.NonBlockingConnection.receive(NonBlock ingConnection.java:...) com.ibm.mq.communications.NonBlockingWorker.run(NonBlockingWorke r.java:...) java.lang.Thread.run(Thread.java:...) ************************************************************* At the time that the channel stops, the IBM MQ AMQP service writes the following messages to its log file (amqp0.log): AMQXR0002E: On channel <channel_name>, a throwable java.lang.OutOfMemoryError: Java heap space resulted when the MQXR service received a message from an MQTT client ID:<client_identifier>. AMQCO1006I: Channel '<channel_name>' has stopped A Java heapdump collected at the time that the OutOfMemoryError occurs shows that most most of the memory is associated with a single instance of an: "org.apache.qpid.proton.engine.impl.ConnectionImpl" object, which is maintaining references to a large number of: "org.apache.qpid.proton.engine.impl.SessionImpl" objects.
Local fix
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of: - The IBM MQ 9.1 AMQP service. - The IBM MQ 9.2 AMQP service. Platforms affected: AIX, Linux on Power, Linux on x86-64, Linux on zSeries, Solaris x86-64, z/OS, Solaris SPARC **************************************************************** PROBLEM DESCRIPTION: The MQ AMQP service uses Apache Qpid Proton to handle the communication with AMQP clients. When the service starts up, each internal worker thread creates its own "org.apache.qpid.proton.engine.impl.ConnectionImpl" object. Whenever an AMQP client connects, it is associated with a worker thread. The thread then uses its "org.apache.qpid.proton.engine.impl.ConnectionImpl" object to create a "org.apache.qpid.proton.engine.impl.SessionImpl" object to hold the state information for that client. The "org.apache.qpid.proton.engine.impl.SessionImpl" object is then added to a list maintained by its parent "org.apache.qpid.proton.engine.impl.ConnectionImpl" object. Here is a simple diagram that shows the relationship between these objects: org.apache.qpid.proton.engine.impl.ConnectionImpl |-org.apache.qpid.proton.engine.impl.SessionImpl |-org.apache.qpid.proton.engine.impl.SessionImpl |-org.apache.qpid.proton.engine.impl.SessionImpl ..... Now, when the AMQP client disconnected from the MQ AMQP service, the service called the method: org.apache.qpid.proton.engine.impl.SessionImpl.close() to indicate that the client was no longer connected. However, it did not call the method: org.apache.qpid.proton.engine.impl.SessionImpl.free() As a result, the internal resources for that AMQP client were not cleaned up and the "org.apache.qpid.proton.engine.impl.SessionImpl" object remained in the list maintained by its parent "org.apache.qpid.proton.engine.impl.ConnectionImpl" object. Over time, as lots of AMQP clients connected and disconnected, the list of "org.apache.qpid.proton.engine.impl.SessionImpl" objects for a given "org.apache.qpid.proton.engine.impl.ConnectionImpl" object grew until eventually the MQ AMQP service wrote the following messages to it's log file (amqp0.log): AMQXR0002E: On channel <channel_name>, a throwable java.lang.OutOfMemoryError: Java heap space resulted when the MQXR service received a message from an MQTT client ID:<client_identifier>. AMQCO1006I: Channel '<channel_name>' has stopped and generated an FDC containing the information shown below: ************************************************************* ... | Source Class :- com.ibm.mq.MQXRService.AMQPServerSessionV10 | Source Method :- processProtonUpdates | Probe Id :- XR10072014 ... | Thread :- ServerWorker1 ... :--------------------------------------------------------------- -----------------------------------: : Exception cause: : :--------------------------------------------------------------- -----------------------------------: java.lang.OutOfMemoryError: Java heap space ... 	at com.ibm.mq.MQXRService.AMQPServerSessionV10.receiveMessage(AMQPS erverSessionV10.java:...) 	at com.ibm.mq.MQXRService.AMQPServerSessionV10.processProtonUpdates (AMQPServerSessionV10.java:...) 	at com.ibm.mq.MQXRService.AMQPServerSessionV10.handleReceive(AMQPSe rverSessionV10.java:...) 	at com.ibm.mq.MQXRService.AMQPServerWireContext.receive(AMQPServerW ireContext.java:...) 	at com.ibm.mq.communications.NonBlockingConnection.receive(NonBlock ingConnection.java:...) 	at com.ibm.mq.communications.NonBlockingWorker.run(NonBlockingWorke r.java:...) 	at java.lang.Thread.run(Thread.java:...) :--------------------------------------------------------------- -----------------------------------: : Object :- java.lang.OutOfMemoryError: Java heap space : : Class@Hashcode :- java.lang.OutOfMemoryError@87a8347b : :--------------------------------------------------------------- -----------------------------------: :--------------------------------------------------------------- -----------------------------------: : FFDC Origin: : :--------------------------------------------------------------- -----------------------------------: com.ibm.mq.util.logging.FFDCFileHandler.publish(FFDCFileHandler. java:...) com.ibm.mq.util.logging.FFDC.print(FFDC.java:...) com.ibm.mq.util.logging.FFDC.processException(FFDC.java:...) com.ibm.mq.MQXRService.AMQPServerSessionV10.processProtonUpdates (AMQPServerSessionV10.java:...) com.ibm.mq.MQXRService.AMQPServerSessionV10.handleReceive(AMQPSe rverSessionV10.java:...) com.ibm.mq.MQXRService.AMQPServerWireContext.receive(AMQPServerW ireContext.java:...) com.ibm.mq.communications.NonBlockingConnection.receive(NonBlock ingConnection.java:...) com.ibm.mq.communications.NonBlockingWorker.run(NonBlockingWorke r.java:...) java.lang.Thread.run(Thread.java:...) *************************************************************
Problem conclusion
The MQ AMQP service has been updated to call the methods: org.apache.qpid.proton.engine.impl.SessionImpl.close() org.apache.qpid.proton.engine.impl.SessionImpl.free() when an AMQP client disconnects from it. This ensures that: - All of the resources for that AMQP client are cleaned up. - And the "org.apache.qpid.proton.engine.impl.SessionImpl" object is removed from the list maintained by its parent "org.apache.qpid.proton.engine.impl.ConnectionImpl" object. which prevents the OutOfMemoryError occurring. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v9.1 LTS 9.1.0.10 v9.2 LTS 9.2.0.4 v9.x CD 9.3.1 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT37726
Reported component name
IBM MQ BASE MP
Reported component ID
5724H7271
Reported release
910
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2021-07-22
Closed date
2021-10-06
Last modified date
2022-09-30
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
IBM MQ BASE MP
Fixed component ID
5724H7271
Applicable component levels
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"910","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
30 September 2022