How to Troubleshoot when your WebSphere MQ queue manager is creating FDCs
ValerieLampkin 27000182R2 Visits (15860)
Often, when a user is having problems with WebSphere MQ one of the first questions asked by the IBM MQ support team is "Are there any FDC files?"
The terms FFST and FDC are sometimes used synonymously. These are files written by MQ to record errors and dump information about what occurred at the time MQ encountered the error.
FFST stands for First Failure Support Technology, and is technology within WebSphere MQ designed to create detailed reports with information about the current state of a queue manager together with historical data. These errors are written to files named as AMQxxxxx.y.FDC where xxxxx is the process ID (PID) that encountered the error. Hence, the *FDC files include the FFST reports. All threads in a process will append their FFSTs to the same file.
A useful tool is "ffstsummary". You simply type that command when you are in the /var/mqm/errors directory and it produces a summary of the FFST reports, sorted into time order.
Usually the first FDC that occurred at the time/date of the current problem is the most relevant. Other subsequent FDC files may be due to the resulting symptoms but not the actual root cause. This is why it is important to analyze the FDCs times to determine the origin of the problem.
I often redirect the output to a file for ease of review. Example:
$ ffstsummary > FFST.output
You can then view the FFST.output file and should see something like this:
!AMQ348308.0.FDC 2012/09/07 07:25:19.153937-4 amqzmur0 348308 1 XC034255 xcsWaitEventSem
Using the information from the ffstsummary output, try to correlate the times to entries in the Queue Manager logs. Together this information is a good starting point for trying to troubleshoot a WebSphere MQ problem.
For example, the columns in the output above show:
Once you have the Probe ID information from the FDC file and possibly a corresponding error code from the queue manager log, you can search online for possible matches to APAR, technotes, etc. to expedite your problem resolution.