Tracing on AIX and Linux

On AIX® and Linux®, you can use the strmqtrc and endmqtrc commands to start and end tracing, and dspmqtrc to display a trace file. On AIX, you can use AIX system trace in addition to using the strmqtrc and endmqtrc commands.

Before you begin

To format or view a trace file, you must be either the creator of the trace file, or a member of the mqm group.

Before you enable tracing in IBM® MQ, ensure that there is plenty of free disk space in the trace directory, otherwise, if this directory gets filled with trace files, it could impact the running of the queue managers due lack of available disk space.

When taking long traces of an IBM MQ queue manager or a C-based IBM MQ client application, you can mitigate any potential disk space issues by using the environment variable MQTRACEPATH to point to a directory that has more disk space as shown in the following example:

export MQTRACEPATH=/other/mytrace

Notes:

Before setting MQTRACEPATH, make sure that the queue manager is not already running, otherwise setting this environment variable will NOT change the location when traces are generated.
You must ensure that the directory has the same file permissions and ownership as /var/mqm/trace.

For more information, see the IBM Support technote Approaches to mitigate the disk space issue when taking long traces for IBM MQ

About this task

Traces can capture the root cause of many IBM MQ problems. It is important to limit how long tracing is active in order to avoid affecting performance or generating excessively large trace files and filling up your disk.

On AIX and Linux, you use the following commands to control the trace facility:

strmqtrc

On AIX and Linux systems, you enable or modify tracing using the strmqtrc control command. The command has optional parameters that specify the level of tracing you want. For example:

For selective component tracing, use the -t and -x parameters to control the amount of trace detail to record. Specify the points that you want to trace by using the -t parameter or the points you do not want to trace by using the -x parameter. You can run multiple traces with different parameters at the same time, if necessary.
For selective process tracing, use the -p parameter to restrict trace generation to specified named processes.
If you need to prevent IBM MQ from including any message data in its traces, use the -d 0 parameter.

For more information, see strmqtrc (start trace).

endmqtrc

To stop tracing, you use the endmqtrc control command. The endmqtrc command ends only the trace that is described by its parameters. Using endmqtrc with no parameters ends early tracing of all processes. For more information, see endmqtrc (end trace).

dspmqtrc filename

Trace output is unformatted; use the dspmqtrc control command to format trace output before viewing. For more information, see dspmqtrc (display formatted trace).

[Linux] On Linux x86-64 systems, you can alternatively use IBM MQ Explorer to start and stop tracing. However, you can trace only everything using the function provided, equivalent to using the commands strmqtrc -e and endmqtrc -e.

The trace facility uses a number of files, which are:

One file for each entity being traced, in which trace information is recorded.
One additional file on each machine, to provide a reference for the shared memory used to start and end tracing.
One file to identify the semaphore used when updating the shared memory.

Files associated with trace are created in a fixed location in the file tree, which is /var/mqm/trace. All client tracing takes place to files in this directory.

Note: Make sure that the /var/mqm/trace directory is on a local file system that is separate from the file systems containing your queue managers. Otherwise, you risk an outage if IBM MQ trace fills the file system that your queue managers are using. The trace directory must have sufficient free space since a partial trace might not contain the information that IBM Support needs to solve the problem.

Trace files are named AMQppppp.qq.TRC where the variables are:

ppppp: The ID of the process reporting the error.
qq: A sequence number, starting at 0. If the full file name exists, this value is incremented by one until a unique trace file name is found. A trace file name can exist if a process is reused.

Note:

The process identifier can contain fewer, or more, digits than shown in this example.
There is one trace file for each process running as part of the entity being traced.

SSL trace files have the names AMQ.SSL.TRC and AMQ.SSL.TRC.1. You cannot format SSL trace files; send them unchanged to IBM Support.

Note: You can accommodate the production of large trace files by mounting a temporary file system over the directory that contains your trace files. Alternatively, rename the trace directory and create the symbolic link /var/mqm/trace to a different directory.

Procedure

If you have multiple installations of IBM MQ on a single AIX or Linux system, make sure that your environment is set for the installation that you want to trace.
For example, if you are tracing a queue manager or application associated with Installation1, then you must use the setmqenv command to switch to Installation1 before running any of the commands to start, stop, or format traces, otherwise, your trace will capture no useful information.
Check the /etc/opt/mqm/mqinst.ini file to determine the location of the IBM MQ installations on your system. Use the setmqenv command to set your environment for the installation you want to trace.
For example:
Setting your environment for the installation with queue manager QMA
```
/opt/mq93/bin/setmqenv -m QMA
```
Setting your environment for installation Installation2
```
/opt/mq93/bin/setmqenv -n Installation2
```
Start tracing with the strmqtrc command.
By default all trace points are enabled and a default-detail trace is generated. The parameters passed to strmqtrc control which processes generate trace files and what kinds of data they trace. For example:
Starting a default detail trace of queue manager QMA
```
strmqtrc -m QMA
```
Starting a trace of queue manager QMA with only output data associated with using Transport Layer Security (TLS) channel security
```
strmqtrc -m QMA -t ssl
```
Starting a default detail trace of queue manager QMB with no message data included
```
strmqtrc -m QMB -d 0
```
Starting an API trace of processes named amqsput with all message data included
```
strmqtrc -t api -p amqsput -d all
```
Starting a high detail trace of everything with files that wrap at 100MB
```
strmqtrc -e -t all -t detail -l 100
```
For selective component tracing on AIX, use the environment variable MQS_TRACE_OPTIONS to activate the high detail and parameter tracing functions individually. Because MQS_TRACE_OPTIONS enables tracing to be active without high detail and parameter tracing functions, you can use it to reduce the effect on performance and trace size when you are trying to reproduce a problem with tracing enabled. Typically, MQS_TRACE_OPTIONS must be set in the process that starts the queue manager, and before the queue manager is started, or it is not recognized. Set MQS_TRACE_OPTIONS before tracing starts. If it is set after tracing starts it is not recognized.
Note: Only set the environment variable MQS_TRACE_OPTIONS if you have been instructed to do so by IBM Support.
Reproduce the problem while tracing is active, making every attempt to keep the trace as short as possible.
To check trace status use the -s parameter of the strmqtrc command:
```
strmqtrc -s
```
Stop tracing with the endmqtrc command.
As soon as the problem has occurred, stop tracing immediately. If you wait to stop the trace, the data that IBM Support needs might be lost or overwritten. You can stop all traces at once, or you can stop each trace individually. For example:
Stopping all traces in an installation
```
endmqtrc -a
```
Stopping a trace of queue manager QMA
```
endmqtrc -m QMA
```
Stopping a trace of processes named amqsput
```
endmqtrc -p amqsput
```
Format the traces with the dspmqtrc command.
On AIX and Linux, the trace files that IBM MQ generates are binary, and must be formatted before they can be read. The binary trace files are named like AMQ*.TRC, and if you started a wrapping trace, some files might also be named like AMQ*.TRS:
```
cd /var/mqm/trace
dspmqtrc AMQ*.TR?
```
Clean up the trace directory.
Clean up the /var/mqm/trace directory to reclaim space and also to make sure that you are ready to generate new traces in the future, if necessary. Save a copy of the binary trace files for as long as you are working with IBM Support, but do not send both binary and formatted traces to IBM. Delete the binary trace files (except AMQ.SSL.TRC and AMQ.SSL.TRC.1, if present) before sending your trace files to IBM.
```
cd /var/mqm/trace
rm -f AMQ*.?.TR?
```
Collect the traces and any other troubleshooting information relating to your problem.
If you are using the runmqras command to collect IBM MQ troubleshooting information and you need to include trace data, you must include trace in the section names that you specify for the -section parameter. If the traces are very large, you might need to use the -workdirectory option to package the files in a directory with plenty of free space. For more information, see Collecting troubleshooting information automatically with runmqras and Sending troubleshooting information to IBM.
Send the traces and any other troubleshooting information that you have collected for the same problem to IBM.
For more information, see Sending troubleshooting information to IBM.