IBM Support

Collect IBM MQ MustGather data to solve dead letter queue problems on Linux, UNIX, Windows and IBM i

Troubleshooting


Problem

An IBM MQ queue manager is placing messages on its dead-letter queue (DLQ), and you need to collect the MustGather data to find a solution.

Environment

These instructions apply only to IBM MQ V9.2, V9.1, V9.0 and V8.0, and IBM WebSphere MQ V7.5, V7.1, V7.0 AIX, HP-UX, Linux, Solaris and Windows, and to IBM MQ V9.2, V9.1, V9.0 and V8.0, IBM WebSphere MQ V7.1, V7.0 and V6.0 on IBM i. Refer to the IBM MQ Read First page for instructions on other operating systems:

Resolving The Problem

Please answer these questions about the problem and then follow the steps below

  • What dead-letter queue problem did you observe on the system?
  • What time did the dead-letter queue problem start and when did it stop?
  • Where are the dead-letter messages coming from, and what is their intended route?

Step 1: Generate Data

If messages are going to the dead letter queue right now, or if you can reproduce the problem which cases messages to go there, generate data to provide more information about the problem:

Generate a trace of the queue manager while messages are going to the dead-letter queue:
Browse the messages on the dead-letter queue using a sample program like the amqsbcg. For example:
amqsbcg QMA.DEAD.LETTER.QUEUE QMA > QMA.DLQ.browse.txt

Step 2: Collect Data


Place the browse output file, i.e. QMA.DLQ.browse.txt, from Step 1 directly in the high-level error log directory, i.e. var/mqm/errors on Linux and MQ_INSTALLATION_PATH\errors on Windows. Both the RAS automation tool (runmqras) and the manual collection steps below collect files found there.
 

Collecting Data Automatically

If you are running IBM MQ V9.2, V9.1, V9.0 or V8.0, IBM WebSphere MQ V7.5, V7.1.0.1 or later, or V7.0.1.8 or later collect data automatically with the runmqras command on both sides of the channel. Be sure to collect the runmqras defs, cluster and trace sections, and to specify your case number:

runmqras -section defs,cluster,trace -qmlist QMA -caseno TS001234567

Collecting IBM MQ Data Manually

  1. If your system has more than one IBM MQ installation, use the setmqenv command to choose the one with the problem before proceeding:

    Linux and UNIX

    sh> . /path/to/mqm/bin/setmqenv -n InstallationX

    Windows

    C:\> "C:\Program Files\IBM\MQ\bin\setmqenv" -n InstallationX
  2. Record the IBM MQ version and maintenance level on both sides of the channel. Alternatively, collect the IBM MQ data manually on both sides of the channel.

  3. Record the Operating System version and maintenance level on both sides of the channel.

  4. Record the IP addresses and hostnames of the systems on both sides of the channel.

  5. Save the IBM  MQ configuration information, for example, registry keys and .ini files.

  6. If your system has more than one IBM MQ installation, record your IBM MQ installation details:

    Linux and UNIX

    sh> dspmqinst > /tmp/dspmqinst.txt

    Windows

    C:\>dspmqinst > %TEMP%/dspmqinst.txt

  7. Record the status of your queue managers:

    Linux and UNIX

    sh> dspmq -a > /tmp/dspmq.txt

    Windows

    C:\> dspmq -a > %TEMP%/dspmq.txt

    IBM i command line

    ===> WRKMQM

    IBM i Qshell

    ===> /QSYS.LIB/QMQM.LIB/DSPMQ.PGM -a > /tmp/dspmq.txt

  8. Record the IBM MQ processes active on your system:

    Linux and UNIX

    sh> ps -ef | grep mq > /tmp/ps.txt

    Windows

    C:\> TASKLIST /V > %TEMP%/tasklist.txt

    IBM i command line

    ===> WRKACTJOB SBS(QMQM)

    IBM i Qshell

    ===> ps -ef | grep mq > /tmp/ps.txt

  9. On IBM WebSphere MQ V7.1 and later installations, use dmpmqcfg command to record the queue manager configuration:

    Linux and UNIX

    sh> dmpmqcfg -m QMA >/tmp/QMA.config.txt

    Windows

    C:\> dmpmqcfg -mQMA>%TEMP%\QMA.config.txt

    IBM i Qshell

    ===> /QSYS.LIB/QMQM.LIB/DMPMQCFG.PGM -mQMA > /tmp/QMA.config.txt

  10. Otherwise, on IBM WebSphere MQ V7.0 and earlier installations, use runmqsc to record the queue manager configuration. If any command returns an error, carry on with the others:

    DISPLAY QMGR ALL
    DISPLAY Q(*) ALL
    DISPLAY SUB(*) ALL
    DISPLAY TOPIC(*) ALL
    DISPLAY CHANNEL(*) ALL
    DISPLAY SERVICE(*) ALL
    DISPLAY PROCESS(*) ALL
    DISPLAY LISTENER(*) ALL
    DISPLAY NAMELIST(*) ALL

  11. On all IBM MQ server installations, use runmqsc to record status information from the queue manager . If any command returns an error, carry on with the others:

    DISPLAY PUBSUB ALL
    DISPLAY QMSTATUS ALL
    DISPLAY CHSTATUS(*) ALL
    DISPLAY LSSTATUS(*) ALL
    DISPLAY SVSTATUS(*) ALL
    DISPLAY SBSTATUS(*) ALL
    DISPLAY CONN(*) TYPE(*) ALL
    DISPLAY QSTATUS(*) TYPE(QUEUE) ALL
    DISPLAY QSTATUS(*) TYPE(HANDLE) ALL
    DISPLAY TPSTATUS('#') TYPE(PUB) ALL
    DISPLAY TPSTATUS('#') TYPE(SUB) ALL
    DISPLAY TPSTATUS('#') TYPE(TOPIC) ALL

  12. If your channel is having difficulty connecting, use your Operating System tools to list network connections on both sides immediately before and after the connection attempt:

    Displaying network connections on Linux and UNIX

    sh> netstat -an

    Displaying network connections on Windows

    C:\>NETSTAT -AN

    Displaying IPv4 and IPv6 network connections at the IBM i command line

    ===> NETSTAT OPTION(*CNN) ===> NETSTAT OPTION(*CNN6)

  13. Manually package your files for IBM , including files containing the output from the commands listed in Step 1 and Step 2.

Step 3: Send Data to IBM

A good description of the problem and the data is the most important information you can provide to IBM. Please do not send data without providing a description!

Back to top

[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSFKSJ","label":"WebSphere MQ"},"ARM Category":[{"code":"a8m0z00000008QrAAI","label":"Components and Features->Dead Letter Queue"}],"ARM Case Number":"","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF012","label":"IBM i"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"All Version(s)"}]

Product Synonym

IBMMQ WebSphere MQ WMQ

Document Information

Modified date:
02 April 2021

UID

swg21290897