MQ Monitoring agent is writing to the MQ Dead Letter Queue

Body

Product: WebSphere MQ monitoring agent (ITCAM)

This article also applies to MQ monitoring agent from APM V8.1.x product.

Problem

You have MQ monitoring agent running, and all looks good.  Then you get a call from your MQ Admin team, they are concerned about many messages from ITCAM MQ monitoring agent ending up in the Dead Letter Queue (DLQ).

How is agent writing to the DLQ? How to resolve this problem? This is the topic of following discussion.

Cause of the problem

A dead-letter queue (DLQ), sometimes referred to as an undelivered-message queue, is a holding queue for messages that cannot be delivered to their destination queues, for example because the queue does not exist, or because it is full.

The monitoring agent creates a dynamic queue named KMQ.IRA.AGENT.QUEUE.*, and uses this as reply-to queue for requests for data from the Queue Manager.  The number of messages received in this reply-queue varies depending on the agent configuration, queue manager configuration, number of MQ objects being monitored, situations defined etc. When the number of messages exceeds the maximum depth (MAXDEPTH) setting for this queue, the un-delivered messages will end up in the DLQ of the queue manager.

Resolving the problem

The dynamic queue KMQ.IRA.AGENT.QUEUE.* is created by agent by using SYSTEM.DEFAULT.MODEL.QUEUE as a template, hence it inherits MAXDEPTH setting from the model queue. The solution is to increase the MAXDEPTH of agent reply-queue so that it can hold the number of response messages to hold monitoring data it receives.

The default value for MAXDEPTH in QMGR configuration is 5000. You can check this setting using runmqsc tool as follows:

runmqsc qmgr_name

DIS QUEUE(SYSTEM.DEFAULT.MODEL.QUEUE) MAXDEPTH

QUEUE(SYSTEM.DEFAULT.MODEL.QUEUE)       TYPE(QMODEL)
MAXDEPTH(5000)

DIS QL(KMQ.IRA.AGENT.QUEUE*) MAXDEPTH

 QUEUE(KMQ.IRA.AGENT.QUEUE.5C352495030E8822)
 TYPE(QLOCAL)                            MAXDEPTH(5000)

There are two ways to increase the MAXDEPTH of the agent reply-queue as described below.

Option 1: To resolve the problem of messages in DLQ, increase the MAXDEPTH of the model queue as follows, and then restart the monitoring agent:

runmqsc qmgr_name

    ALTER QMODEL(SYSTEM.DEFAULT.MODEL.QUEUE) MAXDEPTH(10000)

    AMQ8008I: IBM MQ queue changed.

    DIS QL(KMQ.IRA.AGENT.QUEUE*) MAXDEPTH

    QUEUE(KMQ.IRA.AGENT.QUEUE.5C352495030E8822)
    TYPE(QLOCAL)                            MAXDEPTH(10000)

This should resolve the problem of MQ monitoring agent messages ending up in DLQ.

You may need to adjust the model queue MAXDEPTH setting further. You can try setting to 50000 and reduce from there to get to an optimum value for your queue manager activity and agent configuration.

Option 2: The MQ Admin team may not agree to increasing the MAXDEPTH setting for the SYSTEM.DEFAULT.MODEL.QUEUE system queue. In this case, you can create a local queue named 'KMQ.IRA.AGENT.QUEUE' for the agent and adjust the MAXDEPTH setting for this local queue.  This solution works because MQ agent first looks for the KMQ.IRA.AGENT.QUEUE, and uses it if it exists, and if it is not found, then it creates a dynamic KMQ.IRA.AGENT.QUEUE*.

Use following commands to create KMQ.IRA.AGENT.QUEUE  local queue, and then restart the monitoring agent.

runmqsc qmgr_name

    DEFINE QLOCAL(KMQ.IRA.AGENT.QUEUE)
    AMQ8006I: IBM MQ queue created.
    ALTER QLOCAL(KMQ.IRA.AGENT.QUEUE) MAXDEPTH(10000)
           AMQ8008I: IBM MQ queue changed.

The agent will now use the KMQ.IRA.AGENT.QUEUE local queue and not create any dynamic queues KMQ.IRA.AGENT.QUEUE.*.

As mentioned in Option 1, you may have to adjust the MAXDEPTH of the queue if the data collection is generating more than 10000 messages and DLQ is still getting the overflow from agent queue.

References

IBM MQ: Dead-letter queues

MQ Agent: Configuration best practices

(The best practices link applies to MQ agent on both the distributed and zOS platforms.)