Collect IBM MQ MustGather data to solve hang and high CPU problems on Linux, UNIX, Windows and IBM i

Resolving The Problem

Please answer these questions about the problem and then follow the steps below:

• What performance problem or hang did you observe on the system?
• What time did the problem start and when did it stop?
• Which processes were involved in the performance problem or hang?
• Were there any recent changes to the system or to your applications before the problem?

Step 1: Generate Data

It is essential to gather information from the system when the performance problem or hang is happening in order to identify the cause:

1. Generate stack dumps and other debugging data from the queue managers and applications showing the problem:
   
   

   Generating debugging data from managed file transfer processes on Linux, UNIX, Windows and IBM i

   Generate three javacores from the hanging managed file transfer process, delaying approximately one minute between each one. Replace AGENTNAME and LOGGERNAME with the name of the managed file transfer agent or logger that is hanging, and COORDQMNAME with the name of the coordination queue manager:

   
   
   

   1. If you are running MQ V7.5 or later, you can generate debugging data from managed file transfer agents and loggers very easily:
      
      

      V7.5 and later: Generating three agent javacores on Linux, UNIX and Windows

      
      fteSetAgentTraceLevel -jc AGENTNAME
      ...
      fteSetAgentTraceLevel -jc AGENTNAME
      ...
      fteSetAgentTraceLevel -jc AGENTNAME
      
      
      

      V7.5 and later: Generating three logger javacores on Linux, UNIX and Windows

      
      fteSetLoggerTraceLevel -jc LOGGERNAME
      ...
      fteSetLoggerTraceLevel -jc LOGGERNAME
      ...
      fteSetLoggerTraceLevel -jc LOGGERNAME
      
      
      The java cores generated by this method are stored in the file transfer data directory, based on the coordination queue manager name and the agent name. For example:
      
      

      V7.5 Javacore location on Linux and UNIX

      
      /var/mqm/mqft/logs/COORDQMNAME/loggers/LOGGERNAME
      /var/mqm/mqft/logs/COORDQMNAME/agents/AGENTNAME
      
      

      V7.5 Javacore location on Windows (may vary)

      
      C:\Program Files (x86)\IBM\WebSphere MQ\mqft\logs\COORDQMNAME\agents\AGENTNAME
      C:\Program Files (x86)\IBM\WebSphere MQ\mqft\logs\COORDQMNAME\loggers\LOGGERNAME
      
      
   
   
   

   If you are running WebSphere MQ File Transfer Edition V7.0, you must configure the agent in advance in order to generate javacores. This method of triggering javacores is available in later versions, but is now redundant.

   
   
   
   2. Stop the agent and edit its agent.properties file to include the javaCoreTriggerFile property, which should name a file of your choosing. On Windows systems, you must use double blackslashes as path separators. For example:
      
      

      Setting the javaCoreTriggerFile property in agent.properties on Linux, UNIX and IBM i

      
      javaCoreTriggerFile=/tmp/javatrigger.txt
      
      

      Setting the javaCoreTriggerFile property in agent.properties on Windows

      
      javaCoreTriggerFile=C:\\TEMP\\JAVATRIGGER.TXT
      
      
   
   
   
   3. Restart the agent to pick up the new property. Whenever the trigger file is altered, the agent will generate a javacore. For example:
      
      

      Generating three agent javacores on Linux and UNIX

      
      sh> touch /tmp/javatrigger.txt
          ...
      sh> touch /tmp/javatrigger.txt
          ...
      sh> touch /tmp/javatrigger.txt
      
      

      Generating three agent javacores on Windows

      
      C:\> ECHO "NOW" > C:\TEMP\JAVATRIGGER.TXT
           ...
      C:\> ECHO "NOW" > C:\TEMP\JAVATRIGGER.TXT
           ...
      C:\> ECHO "NOW" > C:\TEMP\JAVATRIGGER.TXT
      
      

      Generating three agent javacores in the IBM i Qshell

      
      ===> touch /tmp/javatrigger.txt
           ...
      ===> touch /tmp/javatrigger.txt
           ...
      ===> touch /tmp/javatrigger.txt
      
      
   
   
   
   4. Delete the trigger file when done. You may remove the javaCoreTriggerFile property the next time you restart the agent, or leave it in place to gather more javacores in the future.
      
      

      Deleting the trigger file on Linux and UNIX

      
      sh> rm /tmp/javatrigger.txt
      
      

      Deleting the trigger file on Windows

      
      C:\> DEL C:\TEMP\JAVATRIGGER.TXT
      
      

      Deleting the trigger file in the IBM i Qshell

      
      ===> rm /tmp/javatrigger.txt
      
      
   
   
   
   5. The javacores generated by this procedure will be written to the same agent directories shown in the previous step for V7.5 and to the following directories for V7.0:
      
      

      V7.0 Javacore location on Linux and UNIX

      
      /var/mqm/config/logs/COORDQMNAME/agents/AGENTNAME
      
      

      V7.0 Javacore location on Windows (may vary)

      
      C:\Documents and Settings\All Users\Application Data\IBM\WMQFTE\config\logs\COORDQMNAME\agents\AGENTNAME
      
      

      V7.0 Javacore location on IBM i

      
      /QIBM/UserData/WMQFTE/V7/config/COORDQMNAME/agents/AGENTNAME
      
      
   
   
   

   For all other managed file transfer commands, follow these instructions to generate three javacores from the process. The javacores or thead dumps in this case are typically written to the working directory of the command:

   
   
   
   6. On AIX, HP-UX, Linux and Solaris, list the Java™ virtual machines using ps and find the one running the hanging managed file transfer command. Then send SIGQUIT to that process identifier (PID) to generate a javacore or thread dump. Note that the "kill -QUIT" command does not terminate Java virtual machines on Linux and UNIX but instead makes them create a javacore or thread dump. For example:
      
      

      Generating javacores from a managed file transfer command on AIX, HP-UX, Linux and Solaris

      
      sh> ps -ef | egrep 'PID|StartAgent'
        UID   PID  PPID   C STIME   TTY           TIME CMD
       7001 37789     1   0 Sun03PM ??         3:07.35 java ... com.ibm.wmqfte.api.StartAgent AGENT1
       7001 69177 64373   0  2:35PM ttys003    0:00.00 egrep PID|StartAgent
      sh> kill -QUIT 37789
          ...
      sh> kill -QUIT 37789
          ...
      sh> kill -QUIT 37789
      
      
   
   
   
   7. On Windows, start the managed file transfer command from the Windows command prompt. Be sure to add the -F option to the fteStartAgent, fteStartLogger, and fteStartDatabaseLogger commands so that will run in the foreground and not in the background or as a Windows service. Then type the Ctrl+Break keyboard sequence to generate a javacore from the process. For example:
      
      

      Generating javacores from a managed file transfer process on Windows

      
      C:\> fteStartLogger -F LOGGER1
      ...
      Ctrl+Break
      ...
      Ctrl+Break
      ...
      Ctrl+Break
      
      
   
   
   
   8. On IBM i, list the Java Virtual Machine jobs in the system using WRKJVMJOB option 7 to find the one running the hanging managed file transfer command. Then press F3 to exit and use the job Number, User and Job name to generate a Java thread dump from the job. For example:
      
      

      Generating javacores from a managed file transfer process using the IBM i Command Line

      
      ===> WRKJVMJOB
      
      Opt  Job Name    User        Number  Function          Status
           QJVACMDSRV  QMQM        136365  PGM-StartAgent     THDW
           QYPSJSVR    QYPSJSVR    136415  PGM-jvmStartPa     SIGW
      
      Use option 7 to find the right job and F3 to return to the Command Line
      
      ===> GENJVMDMP JOB(136365/QMQM/QJVACMDSRV) TYPE(*JAVA)
      
      
   
   

   Generating debugging data from processes on Linux and UNIX

   
   
   1. Download the IBM stackit and sigdump scripts. On HP-UX and Linux systems you must install the GNU debugger (GDB, called WDB on HP-UX), even if temporarily, for stackit to work:
      
      
      • Download stackit
      • Download sigdump
      • Download GDB for Linux
      • Download WDB for HP-UX
   
   
   
   2. Run the stackit script three times against the affected MQ queue managers and applications, with a delay of a minute or less between each run:
      
      

      Running stackit against queue managers QMA and QMB and against and the program myapp

      
      sh> stackit -m QMA -m QMB -n myapp -f /var/mqm/errors/stackit-1.txt
      sh> sleep 30
      sh> stackit -m QMA -m QMB -n myapp -f /var/mqm/errors/stackit-2.txt
      sh> sleep 30
      sh> stackit -m QMA -m QMB -n myapp -f /var/mqm/errors/stackit-3.txt
      
   
   
   
   3. Run the sigdump script once against the affected MQ queue managers. The sigdump script will cause each queue manager to generate diagnostic FFST files:
      
      

      Runnning sigdump against queue managers QMA and QMB

      
      sh> sigdump -m QMA -m QMB
      
   
   

   Generating debugging data from processes on Windows

   
   
   1. Download the following debugging utilities from Microsoft if you do not have them on your system:
      
      
      • Download Microsoft PsList
      • Download Microsoft Handle
      • Download Microsoft ADPlus
      • Download Microsoft Process Monitor
   
   
   
   2. Display the list of processes:
      
      

      Displaying processes on Windows

      
      C:\> tasklist -v
      
   
   
   
   3. Display additional information about each process:
      
      

      Displaying process statistics on Windows

      
      C:\> pslist -x
      
   
   
   
   4. Display information about MQ processes and any affected applications by passing the first few characters of each process name to the handle program, for example:
      
      

      Displaying process handles on Windows

      
      C:\> handle -a -p amq
      C:\> handle -a -p runmq
      C:\> handle -a -p myapp
      
   
   
   
   5. Gather data from hangs (or even crashes) of MQ processes and any affected applications, for example:
      
      

      Gathering hang and crash information on Windows

      
      C:\> adplus -hang -pn amqzxma0.exe
      C:\> adplus -hang -pn amqzlaa0.exe
      C:\> adplus -crash -pn runmqchi.exe
      
   
   
   
   6. Use the Microsoft Process Monitor tool to provide real-time stack data, loaded modules, environment information, files accessed, libr/aries used, registry keys accessed, and more information. Please note that this tool can be very CPU intensive, even with filtering options set. Refer to the section "Scripting Process Monitor" in the included procmon.chm help file for information on using it in a script or batch file.
   
   

   Generating debugging data from processes on IBM i

   
   
   1. Download and run the IBM MQSTACK tool as described here. MQSTACK will show the status of all threads for all queue manager processes, however it does not show information about non-MQ processes:
      
      
      • Download and run MQSTACK
   
   
   
   2. For processes which are not part of the queue manager, such as application programs, run the SERVICEDOCS utility as described below. SERVICEDOCS will show the stack for the main thread of every process on the system:
      
      
      • Generate SERVICEDOCS information
   
   
2. Generate an MQ trace while the problem is happening. Please stop the trace after a short period of time (e.g. a minute or less) to avoid worsening the system performance:
   
   
   • Linux and UNIX
   • Windows

   • IBM i
   • Java and JMS



3. If the hang or high CPU usage is happening inside WebSphere Application Server, please complete the WebSphere Application Server MustGather instructions for your platform:
   
   
   • AIX
   • HP-UX
   • Linux

   • Solaris
   • Windows
   • IBM i



4. On Linux and UNIX systems, save the output from the mqconfig command. If you are running IBM MQ V9.0 or V8.0, or WebSphere MQ V7.5.0.2 or later, or V7.1.0.3 or later, the mqconfig command is already available as part of the MQ server installation.

Step 2: Collect Data

1. Place the debug files from Step 1, and on Linux and UNIX the mqconfig output, directly in the top-level MQ errors directory. Both the runmqras automation tool and the manual collection steps below collect files found there.



2. Collect data automatically with the runmqras command if you are running IBM MQ V9.0 or V8.0, or WebSphere MQ V7.5, V7.1.0.1 or later, or V7.0.1.8 or later. Be sure to collect the runmqras defs, cluster and trace sections, and to specify your PMR number:



Collecting runmqras output from queue manager QMA


runmqras -section defs,cluster,trace -qmlist QMA -pmrno 12345,67R,890




3. Alternatively, collect the MQ data manually.
   
   

   Collecting MQ data manually

   
   
   1. If your system has more than one MQ installation, use the setmqenv command to choose the one with the problem before proceeding:
      
      

      Linux and UNIX

      
      sh> . /path/to/mqm/bin/setmqenv -n InstallationX
      
      

      Windows

      
      C:\> "C:\Program Files\IBM\MQ\bin\setmqenv" -n InstallationX
      
   
   
   
   2. Record the MQ version and maintenance level.
   
   
   
   3. Record the operating system version and maintenance level.
   
   
   
   4. Save the MQ configuration information, for example registry keys and ini files.
   
   
   
   5. If your system has more than one MQ installation, record your MQ installation details:
      
      

      Linux and UNIX

      
      sh> dspmqinst > /tmp/dspmqinst.txt
      
      

      Windows

      
      C:\> dspmqinst > %TEMP%/dspmqinst.txt
      
   
   
   
   6. Record the MQ processes active on your system:
      
      

      Linux and UNIX

      
      sh> ps -ef | grep mq > /tmp/ps.txt
      
      

      Windows

      
      C:\> TASKLIST /V > %TEMP%/tasklist.txt
      
      

      IBM i Command Line

      
      ===> WRKACTJOB SBS(QMQM)
      
      

      IBM i Qshell

      
      ===> ps -ef | grep mq > /tmp/ps.txt
      
   
   
   
   7. On MQ V7.1 and later installations, use dmpmqcfg to record the queue manager configuration:
      
      

      Linux and UNIX

      
      sh> dmpmqcfg -m QMA > /tmp/QMA.config.txt
      
      

      Windows

      
      C:\> dmpmqcfg -m QMA > %TEMP%\QMA.config.txt
      
      

      IBM i Qshell

      
      ===> /QSYS.LIB/QMQM.LIB/DMPMQCFG.PGM -m QMA > /tmp/QMA.config.txt
      
   
   
   
   8. Otherwise, on MQ V7.0 and earlier installations, use runmqsc to record the queue manager configuration. If any command gives an error, carry on with the others:
      
      DISPLAY QMGR ALL
      DISPLAY Q(*) ALL
      DISPLAY SUB(*) ALL
      DISPLAY TOPIC(*) ALL
      DISPLAY CHANNEL(*) ALL
      DISPLAY SERVICE(*) ALL
      DISPLAY PROCESS(*) ALL
      DISPLAY LISTENER(*) ALL
      DISPLAY NAMELIST(*) ALL
      
   
   
   
   9. On all MQ installations, use runmqsc to record status information from the queue manager. If any command gives an error, carry on with the others:
      
      DISPLAY PUBSUB ALL
      DISPLAY QMSTATUS ALL
      DISPLAY CHSTATUS(*) ALL
      DISPLAY LSSTATUS(*) ALL
      DISPLAY SVSTATUS(*) ALL
      DISPLAY SBSTATUS(*) ALL
      DISPLAY CONN(*) TYPE(*) ALL
      DISPLAY QSTATUS(*) TYPE(QUEUE) ALL
      DISPLAY QSTATUS(*) TYPE(HANDLE) ALL
      DISPLAY TPSTATUS('#') TYPE(PUB) ALL
      DISPLAY TPSTATUS('#') TYPE(SUB) ALL
      DISPLAY TPSTATUS('#') TYPE(TOPIC) ALL
      
   
   
   
   10. On MQ installations, use runmqsc to record information about cluster objects known to the queue manager. If any command gives an error, carry on with the others:
       
       DISPLAY CLUSQMGR(*) ALL
       DISPLAY QCLUSTER(*) ALL
       DISPLAY TCLUSTER(*) ALL
       
   
   
   
   11. Manually package your files for IBM, including files containing the output from the commands listed in Step 1 and 2.
   
   

Step 3: Send Data to IBM

1. Send your data to the /toibm/websphere directory on the IBM ECuRep repository by standard or secure HTTP or FTP, or by using the IBM Secure Diagnostic Data Upload Utility (SDDUU) Java application. Be sure to include the full PMR number in the file name so that it will be associated with your problem, e.g. 12345,69R,890-hangdata.zip. Small files may be emailed to websphere_support@ecurep.ibm.com if you put the full PMR number in the subject of your note.



2. While the data is transferring, send an email or use the IBM Service Request tool to update your PMR with your description of the problem and of the data you are sending.




A good description of the problem and the data is the most important information you can provide to IBM. Please do not send data without providing a description!



3. Contact your country representative if you need to speak to an IBM technical support representative, or in the US call 1-800-IBM-SERV. Refer to the IBM Software Support Handbook for more information on working with IBM.




Back to top