APAR status
Closed as program error.
Error description
After the MQ Telemetry service reports the following error to its log file: AMQXR0002E: On channel <channel_name>, a throwable com.ibm.mq.MQXRService.LowMemoryException: AMQXR1016W: com.ibm.mq.MQXRService.MQXRService$<object_reference>(Object) discovered that most of the available memory has been used. resulted when the MQXR service received a message from client <client_identifier>. it starts generating FDCs where the "Exception cause:" section shows that an MQ API call failed with reason code 2219 (MQRC_CALL_IN_PROGRESS). Some example of the "Exception cause" section of the FDCs that are generated in this situation are shown below: ************************************************************ Example 1: :--------------------------------------------------------------- -----------------------------------: : Exception cause: : :--------------------------------------------------------------- -----------------------------------: com.ibm.mq.MQXRService.MQInvocationException: AMQXR0004E: MQSeries verb=MQOPEN(String) returned cc=2(int) MQCC_FAILED rc=2219(int) MQRC_CALL_IN_PROGRESS. at com.ibm.mq.MQXRService.MQConnection.getMQException(MQConnection. java:...) at com.ibm.mq.MQXRService.MQConnection.open(MQConnection.java:...) at com.ibm.mq.MQXRService.LifecycleMessage.sendMessage(LifecycleMes sage.java:...) at com.ibm.mq.MQXRService.MQTTServerSession.sendWillMessages(MQTTSe rverSession.java:...) at com.ibm.mq.MQXRService.MQTTServerSessionV3.willClose(MQTTServerS essionV3.java:...) at com.ibm.mq.MQXRService.MQTTServerContext.willClose(MQTTServerCon text.java:...) at com.ibm.mq.communications.NonBlockingConnection.close(NonBlockin gConnection.java:...) at com.ibm.mq.MQXRService.MQTTServerSessionV3.requestDisconnect(MQT TServerSessionV3.java:...) at com.ibm.mq.MQXRService.MQXRService$1.handleNotification(MQXRServ ice.java:...) at javax.management.NotificationBroadcasterSupport.handleNotificati on(NotificationBroadcasterSupport.java:...) at javax.management.NotificationBroadcasterSupport$SendNotifJob.run (NotificationBroadcasterSupport.java:...) at javax.management.NotificationBroadcasterSupport$1.execute(Notifi cationBroadcasterSupport.java:...) at javax.management.NotificationBroadcasterSupport.sendNotification (NotificationBroadcasterSupport.java:...) at com.ibm.java.lang.management.internal.LazyDelegatingNotifier.sen dNotification(LazyDelegatingNotifier.java:...) at com.ibm.lang.management.internal.MemoryNotificationThread.dispat chMemoryNotificationHelper(MemoryNotificationThread.java:...) at com.ibm.lang.management.internal.MemoryNotificationThread.proces sNotificationLoop(Native Method) at com.ibm.lang.management.internal.MemoryNotificationThread.run(Me moryNotificationThread.java:...) ************************************************************ Example 2: :--------------------------------------------------------------- -----------------------------------: : Exception cause: : :--------------------------------------------------------------- -----------------------------------: com.ibm.mq.MQXRService.MQInvocationException: AMQXR0004E: MQSeries verb=MQCMIT(String) returned cc=2(int) MQCC_FAILED rc=2219(int) MQRC_CALL_IN_PROGRESS. at com.ibm.mq.MQXRService.MQConnection.commit(MQConnection.java:... ) at com.ibm.mq.MQXRService.WorkerContext.synchronousWorkCompleted(Wo rkerContext.java:...) at com.ibm.mq.communications.NonBlockingWorker.synchronousWorkCompl eted(NonBlockingWorker.java:...) at com.ibm.mq.communications.NonBlockingWorker.run(NonBlockingWorke r.java:...) at java.lang.Thread.run(Thread.java:...) ************************************************************ Example 3: :--------------------------------------------------------------- -----------------------------------: : Exception cause: : :--------------------------------------------------------------- -----------------------------------: com.ibm.mq.MQXRService.MQInvocationException: AMQXR0004E: MQSeries verb=MQSUB(String) returned cc=2(int) MQCC_FAILED rc=2219(int) MQRC_CALL_IN_PROGRESS. at com.ibm.mq.MQXRService.MQConnection.getMQException(MQConnection. java:...) at com.ibm.mq.MQXRService.MQConnection.inquireNamedSubscribers(MQCo nnection.java:...) at com.ibm.mq.MQXRService.MQTTServerSession.clearDurableSubs(MQTTSe rverSession.java:...) at com.ibm.mq.MQXRService.MQTTServerSession.purge(MQTTServerSession .java:...) at com.ibm.mq.MQXRService.MQTTServerSession.prepareAsynchronousWork (MQTTServerSession.java:...) at com.ibm.mq.MQXRService.MQTTServerContext.prepareAsynchronousWork (MQTTServerContext.java:...) at com.ibm.mq.communications.NonBlockingWorker.synchronousWorkCompl eted(NonBlockingWorker.java:...) at com.ibm.mq.communications.NonBlockingWorker.run(NonBlockingWorke r.java:...) at java.lang.Thread.run(Thread.java:...) ************************************************************ Example 4: :--------------------------------------------------------------- -----------------------------------: : Exception cause: : :--------------------------------------------------------------- -----------------------------------: com.ibm.mq.MQXRService.MQInvocationException: AMQXR0004E: MQSeries verb=MQOPEN(String) returned cc=2(int) MQCC_FAILED rc=2219(int) MQRC_CALL_IN_PROGRESS. at com.ibm.mq.MQXRService.MQConnection.getMQException(MQConnection. java:...) at com.ibm.mq.MQXRService.MQConnection.open(MQConnection.java:...) at com.ibm.mq.MQXRService.MQTTServerSessionV311.receivePublish(MQTT ServerSessionV311.java:...) at com.ibm.mq.MQXRService.MQTTServerSessionV311.handleReceive(MQTTS erverSessionV311.java:...) at com.ibm.mq.MQXRService.MQTTServerContext.handleReceive(MQTTServe rContext.java:...) at com.ibm.mq.MQXRService.MQTTWireContext.receive(MQTTWireContext.j ava:...) at com.ibm.mq.communications.NonBlockingConnection.receive(NonBlock ingConnection.java:...) at com.ibm.mq.communications.NonBlockingWorker.run(NonBlockingWorke r.java:...) at java.lang.Thread.run(Thread.java:...) ************************************************************
Local fix
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of: - The MQ 9.1 LTS Telemetry service. - The MQ 9.2 LTS Telemetry service. who have MQTT client applications that connect to the service and register a last will and testament (LWT) message that will be published to a topic if the MQTT client is disconnected unexpectedly. Platforms affected: AIX, Linux on x86-64, Windows, Linux on S390 **************************************************************** PROBLEM DESCRIPTION: The MQ Telemetry service contains a number of internal worker threads, each of which has its own connection to the queue manager that the service is connected to. Whenever an MQTT client application connects to the MQ Telemetry service, it is associated with one of these internal threads. If the MQTT client performs an operation that requires the MQ Telemetry service to communicate with the queue manager, then the worker thread will use its connection to the queue manager to issue an MQ API call, and process the response. MQTT clients can specify a last will and testament (LWT) message when they connect to the MQ Telemetry service. If the MQTT client does this, and then disconnects unexpectedly, then the MQ Telemetry service will publish the client's LWT message to the specified topic. This is done using the queue manager connection associated with the internal worker thread that the MQTT client was using. Now, the MQ Telemetry service also contains some functionality to monitor the amount of memory that it is using. When its memory usage gets to within 80% of its maximum value, the MQ Telemetry service will write the message: AMQXR0002E: On channel <channel_name>, a throwable com.ibm.mq.MQXRService.LowMemoryException: AMQXR1016W: com.ibm.mq.MQXRService.MQXRService$<object_reference>(Object) discovered that most of the available memory has been used. resulted when the MQXR service received a message from client <client_identifier>. to its log file, randomly select 10% of the MQTT clients that are currently connected to it and then disconnect them to free up memory. This processing is performed on a dedicated internal thread, called the "MemoryMXBean notification dispatcher thread". When the issue reported in this APAR occurred, the MQ Telemetry service had detected that it was running low on memory, and was disconnecting some MQTT clients that had registered LWT messages. As part of this processing, the service tried to send the LWT messages for those MQTT clients - in order to do so, it: - Looked up the internal worker thread that was being used to - Obtained a reference to the connection to the queue manager that was associated with that internal worker thread. - And then used that connection to issue the following MQ API calls: - MQOPEN - MQPUT - MQCLOSE to publish the LWT message. However, there was no locking around the connection. This meant that it was possible for the connection to be used by both the internal worker thread, and the "MemoryMXBean notification dispatcher thread" at the same time. If this happened, then: - The first thread would make an MQ API call, which would complete successfully. - The second thread would try to make an MQ API call, and the call would fail with reason code 2219 (MQRC_CALL_IN_PROGRESS). The second thread would then generate an FDC, where the "Exception cause:" section would show that an MQ API call failed with this reason code.
Problem conclusion
To resolve this issue, the MQ Telemetry service has been updated to include some locking around its connections to the queue manager. This ensures that only one thread can use a connection to make an MQ API call at a time, which prevents the FDCs reported in the APAR from occurring. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v9.1 LTS 9.1.0.9 v9.2 LTS 9.2.0.4 v9.x CD 9.2.4 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT36442
Reported component name
IBM MQ BASE MP
Reported component ID
5724H7271
Reported release
910
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2021-04-02
Closed date
2021-06-23
Last modified date
2021-06-23
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
IBM MQ BASE MP
Fixed component ID
5724H7271
Applicable component levels
[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU029","label":"Software"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"910"}]
Document Information
Modified date:
24 June 2021