IT37881: SYSTEM.MQTT.TRANSMIT.QUEUE fills up with messages for MQTT clients that are no longer connected

APAR status

• Closed as program error.

Error description

• An MQTT client with client identifier "Client1" connects to the
  MQ Telemetry service requesting a clean session (by by setting
  MqttConnectOptions.cleanSession to true) to take out a
  non-durable subscription on a topic. After processing a number
  of messages that were published on that topic, the MQTT client
  disconnects.
  
  Some time later, the MQ Telemetry service reports the following
  error in its log file (mqxr_0.log):
  
  AMQXR0002E: On channel <channel name>, a throwable
  com.ibm.mq.MQXRService.LowMemoryException: AMQXR1016W:
  com.ibm.mq.MQXRService.MQXRService$1@<object reference>(Object)
  discovered that most of the available memory has been used.
  resulted when the MQXR service received a message from client
  <client identifier>.
  
  
  At the time that the AMQXR0002E error is written to the log
  file, the MQ Telemtry service system queue
  SYSTEM.MQTT.TRANSMIT.QUEUE contains a large number of
  unprocessed messages for the MQTT client that had disconnected,
  as shown in the extract from the amqsbcg output for the queue
  below:
  
  00000000 : 58514820 01000000 53595354 454d2e4d : XQH
  ....SYSTEM.M
  00000010 : 5154542e 5055424c 49434154 494f4e2e :
  QTT.PUBLICATION.
  00000020 : 51554555 45202020 20202020 20202020 : QUEUE
  00000030 : 20202020 20202020 436c6965 6e743120 :         Client1
  00000040 : 20202020 20202020 20202020 20202020 :
  00000050 : 20202020 20202020 20202020 20202020 :
  .....
  

Local fix

Problem summary

• ****************************************************************
  USERS AFFECTED:
  This issue affects users of:
  
  - The MQ 9.1 Telemetry service.
  - The MQ 9.2 Telemetry service.
  
  who have MQTT clients that connect to the MQ Telemetry service
  requesting a clean session (by setting
  MqttConnectOptions.cleanSession to true).
  
  
  Platforms affected:
  AIX, Windows, Linux on x86-64, Linux on zSeries
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  When an MQTT client connects to the MQ Telemetry service
  requesting a clean session (by setting
  MqttConnectOptions.cleanSession to true) and subscribes to a
  topic, the MQ Telemetry service will create a non-durable
  subscription for that MQTT client on its associated queue
  manager. When a message is published on that topic, a copy of it
  is put onto the SYSTEM.MQTT.TRANSMIT.QUEUE. A callback function
  within the MQ Telemetry service is then invoked, which picks the
  message up and sends it to the MQTT client. Depending on the
  quality of service (QOS) of the publications, the copy of the
  message is either removed from the SYSTEM.MQTT.TRANSMIT.QUEUE
  after it has been sent (for QOS 0) or left on the queue until
  the MQTT client has acknowledged that it has received it (for
  QOS 1 or 2).
  
  Now, when an MQTT client that had subscribed to a topic
  requesting a clean session became disconnected from the MQ
  Telemetry service, the service would:
  
  - Unsubscribe to the topic, to prevent any new messages for the
  MQTT client being put to the SYSTEM.MQTT.TRANSMIT.QUEUE.
  - And remove any unacknowledged messages on the
  SYSTEM.MQTT.TRANSMIT.QUEUE that had already been sent to the
  MQTT client via the callback function.
  
  However, any messages for that MQTT client which had not yet
  been processed by the MQ Telemetry service's callback function
  were left on the SYSTEM.MQTT.TRANSMIT.QUEUE.
  
  When the callback function eventually processed these messages,
  it detected that they were for a MQTT client which was no longer
  connected. It then left the messages on the queue, and stored
  some details about the messages internally, which used up memory
  within the service which would never be released. If the
  SYSTEM.MQTT.TRANSMIT.QUEUE contained a large number of messages
  for MQTT clients which had connected requesting a clean session
  and were no longer connected, then a significant amount of
  memory would be allocated to messages that would never be sent,
  which could eventually result in the MQ Telemetry service
  reporting the following message to it's log file (mqxr_0.log):
  
  AMQXR0002E: On channel <channel name>, a throwable
  com.ibm.mq.MQXRService.LowMemoryException: AMQXR1016W:
  com.ibm.mq.MQXRService.MQXRService$1@<object reference>(Object)
  discovered that most of the available memory has been used.
  resulted when the MQXR service received a message from client
  <client identifier>.
  

Problem conclusion

• To resolve this issue, the MQ Telemetry service has been updated
  so that if an MQTT client connects to the service requesting a
  clean session and then disconnects, then any unprocessed
  messages for that MQTT client will be removed from the
  SYSTEM.MQTT.TRANSMIT.QUEUE. As well as preventing messages from
  building up on the queue, this change also helps reduce the
  memory usage of the MQ Telemetry service as it will no longer
  store information about messages on the
  SYSTEM.MQTT.TRANSMIT.QUEUE which will never be sent to MQTT
  clients.
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v9.1 LTS   9.1.0.10
  v9.2 LTS   9.2.0.5
  v9.x CD    9.2.5
  
  The latest available maintenance can be obtained from
  'WebSphere MQ Recommended Fixes'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
  
  If the maintenance level is not yet available information on
  its planned availability can be found in 'WebSphere MQ
  Planned Maintenance Release Dates'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  IBM MQ BASE MP

• Fixed component ID

  5724H7271

Applicable component levels