WMB/IIB MustGather: Problems with Hangs, High CPU and Performance

1. Message flow accounting and statistics can be started and viewed in IBM Integration Explorer to monitor the message flows processing messages when such a problem is experienced. Also check the following example in the Knowledge Center.

   By reviewing the data, you can isolate the problem down to a specific message flow node where the problem may be occurring. For example: average elapsed time (in milliseconds), average CPU time and the number of times a node is invoked can help you identify a particular node that may be a bottleneck.



2. Check for known issues:
   IIB diagnostic tools guide
   How to use the flow analysis to find further performance problems
   IIB and WMB FAQ for memory
   Dynamic guide to help troubleshooting Message Broker
   Troubleshooting performance problems
   Message splitter pattern
   IBM Integration Bus support site
   

1. mqsidc
   See: mqsidc usage instructions
   
   • This command line tool is only available for WMB 7.0.0.5+, WMB 8.0.0.1+, IIB 9.0.0.0+, and IIB 10.0.0.0+.
   • When running the tool, select the option for 'Broker Collector', when prompted.



2. Message flow accounting and statistics
   See: Procedure for collecting accounting and statistics data as XML messages



3. Stack dump

   In hang situations, collect 4-5 stack dumps of the threads running in the DataFlowEngine with a gap of about 10 seconds each.

   
   Use these commands, where <PID> is the process ID of the DataFlowEngine:
   
   • procstack <PID> (On AIX)
   • pstack <PID> (On Linux, Solaris)




4. Project Interchange files for the problematic flow/set/ESQL/model
   
   • You can export your Message Flow and Message Set project(s) into an archive file for easy transmission to IBM Support.
     See: Exporting files from the Workbench



5. Traces If the problem occurs during a runtime test or with the Toolkit test facility, WMB execution group traces can be used to gain a better understanding of the problem.
   • A service level trace is intended to be provided to IBM Support to assist in the diagnosis of your issue. Please run the service level trace independently and not while the above other items are being collected



6. General Broker information
   If the mqsidc tool was not run, then capture this information manually
   • Record the IBM Integration Bus or WebSphere Message Broker version, release, and maintenance level.
     This can be captured using the command 'mqsiservice -v'..
   • Record the operating system version, release, and maintenance level.
   • Record the version, release, and maintenance level of any related products and components for the problematic application.
   • Collect the local error log. On UNIX and Linux systems, the local error log is the syslog. The location of your syslog is configured in the syslog daemon.
     See: Configuring the syslog daemon
   • Collect the Standard Output/Error logs.
     WMB writes information to both STDOUT and STDERR. These files are located under the Message Broker workpath.
     See: Standard System Logs



7. Additional information
   • Output of command: mqsilist -r -d2

ATTENTION: A good problem description is one of the most important tools IBM needs to analyze your data!
When sending data to IBM, be sure to update your PMR or send a note with the following information:

• Tell us what errors you saw, where you saw them, and what time they happened
• Let us know if you made any changes to WebSphere Message Broker or the system before the problem
• Share any other observations which you think will help us to better understand the problem

• Use IBM Service Request to open or view a problem record with IBM.
• Send your data to IBM for further analysis.