- Turning on Application Activity Trace
- Configuring Application Activity Trace
- Choosing the Application Activity Trace level
- Performance impact of Application Activity Trace
- Formatting Application Activity Trace output
- Troubleshooting application behaviour using Application Activity Trace
- Downloadable resources
- Related topics
Increasing the visibility of messages using WebSphere MQ Application Activity Trace
This article uses scenarios to demonstrate potential uses of the IBM® WebSphere® MQ Application Activity Trace, including problem determination and maintaining an audit trail of WebSphere MQ messages. Two tools will be used in the scenarios to format the Application Activity Trace records for analysis:
- The command-line tool amqsact, which is shipped with WebSphere MQ as a sample.
- The events and statistics viewer included in SupportPac MS0P: WebSphere MQ Explorer – Extended Management Plug-ins. Download SupportPac MS0P.
Turning on Application Activity Trace
Application Activity Trace activation is controlled by a queue manager attribute called ACTVTRC. To turn tracing on, use the runmqsc command
ALTER QMGR ACTVTRC(ON). To turn tracing off, use the runmqsc command
ALTER QMGR ACTVTRC(OFF).
You don't need to restart your queue manager to start or stop Application Activity Trace.
You can also turn Application Activity Trace on and off in WebSphere MQ Explorer, in the online monitoring section of the queue managers properties, as shown in Figure 1:
Figure 1. Enabling Application Activity Trace using WebSphere MQ Explorer
You can also turn on Application Activity Trace in the MQCONNX call. For more information, see MQCONNX options in the WebSphere MQ information centre.
Configuring Application Activity Trace
The configuration file mqat.ini enables you to control the frequency and level of detail in Application Activity Trace. The mqat.ini file is located in the queue manager data directory:
- On Linux® and UNIX®: /var/mqm/qmgrs/<qm_name>
- On Microsoft® Windows®: C:\Program Files\IBM\WebSphere MQ\qmgrs\<qm_name>
Where <qm_name> is the name of your queue manager.
Changes to the ini file can be made dynamically without restarting the queue manager. You just need to turn Application Activity Trace off and back on for changes to be picked up.
Choosing the Application Activity Trace level
You have a number of options for configuring the level of detail in Application Activity Trace output. Configurable options in the mqat.ini file include:
- The ability to restrict the applications that the trace runs against. For example, you can specify the application you want to trace by specifying the ApplName in the ApplicationTrace stanza. Since the WebSphere MQ V7.5 Java client now contains the real application name instead of the generic WebSphere MQ client for Java, you now also have more flexibility in tracing Java applications.
- The option to include message data in the Application Activity Trace message.
- The ability to configure the time interval between trace messages, and to batch multiple trace records into a single message. This capability enables you to modify the number of messages generated by Application Activity Trace in cases where performance is a priority. The default is to generate an Application Activity Trace record for every Message Queue Interface (MQI) call made to the queue manager. Each Application Activity Trace message contains 100 trace records.
- The ability to configure the level of detail included in the Application Activity Trace message, as described in the next section.
Depending on the granularity you need, you can set the trace level to LOW, MEDIUM or HIGH via the TraceLevel parameter in the Application Activity Trace configuration file mqat.ini. For example, the following extract from mqat.ini shows the TraceLevel set to MEDIUM (which is the default setting):
AllActivityTrace: # Global settings stanza ActivityInterval=1 # Time interval between trace messages # Values: 0-99999999 (0=off) # Default: 0 ActivityCount=100 # Number of operations between trace msgs # Values: 0-99999999 (0=off) # Default: 0 TraceLevel=MEDIUM # Amount of data traced for each operation # Values: LOW | MEDIUM | HIGH # Default: MEDIUM TraceMessageData=0 # Amount of message data traced # Values: 0-104857600 # Default: 0 StopOnGetTraceMsg=ON # Stop trace on get of activity trace message # Values: ON | OFF # Default: ON
As an example of the output generated with different trace levels, Figure 2 below shows the output parameters for the MQPUT call. This data was taken from Application Activity Trace of a WebSphere MQ client application (the amqsputc sample) connecting to a queue manager and putting a message on a local queue:
Figure 2. MQPut parameters recorded with different trace levels
The appropriate trace level depends on your reason for turning on Application Activity Trace. If you are troubleshooting a codepage conversion issue and need to check the CCSID of the message, then you need to at least set the level of trace detail to MEDIUM. On the other hand, if you are only interested in finding out the number of successful MQPUTs to a queue, then a LOW level of detail may be sufficient. To determine the appropriate level of trace detail for your requirements, turn on Application Activity Trace for a short period of time and try out different levels of trace detail to determine which level provides the amount of detail that you need.
Performance impact of Application Activity Trace
The impact of Application Activity Trace on performance depends on the level of detail you specify, the number of applications being traced, and the frequency with which activity messages are generated. Choose the lowest level of detail and frequency that gives you the information you need, especially if performance is critical. Ensure that the monitoring application reading the activity trace messages can keep up with the trace generation, so that queuing of activity trace messages does not create a bottleneck. For more information on the performance impact of activity tracing and on ways to reduce that impact, see Performance impact of Application Activity Trace in the WebSphere MQ information centre.
Formatting Application Activity Trace output
You can write your own application to read and format Application Activity Trace messages. For more information, see Application Activity Trace message reference in the WebSphere MQ information centre.
The scenarios in this article use the amqsact sample program and the Application Activity Trace viewer in WebSphere MQ Explorer to format trace messages.
amqsact sample program
amqsact formats Application Activity Trace messages for you and is provided with WebSphere MQ. The source code is also provided, so you can easily see how it works and can extend and modify the behaviour to meet your requirements. The compiled program is located in the samples directory:
- On Linux and UNIX: MQ_INSTALLATION_PATH?samp?bin
- On Windows: MQ_INSTALLATION_PATH\tools\c\Samples\Bin;
amqsact [-m QMgrName] [-q QName] # Override default queue name [-t TopicString] # Subscribe to event topic [-b] # Only browse records [-v] # Verbose output [-d <depth>] # Number of records to display [-w <timeout>] # Time to wait (in seconds) [-s <startTime>] # Start time of record to process [-e <endTime>] # End time of record to process
Here is an example of the output for an MQCONN API call when formatted using the command:
amqsact –m TESTQM -v :
MonitoringType: MQI Activity Trace Correl_id: 00000000: 414D 5143 5445 5354 514D 2020 2020 2020 'AMQCTESTQM ' 00000010: B5F6 4251 2000 E601 '.öBQ .æ. ' QueueManager: 'TESTQM' Host Name: 'ADMINIB-1VTJ6N1' IntervalStartDate: '2013-03-15' IntervalStartTime: '12:08:10' IntervalEndDate: '2013-03-15' IntervalEndTime: '12:08:10' CommandLevel: 710 SeqNumber: 0 ApplicationName: 'WebSphere MQ_1\bin\amqsput.exe' Application Type: MQAT_WINDOWS_NT ApplicationPid: 14076 UserId: 'Emma_Bushby' API Caller Type: MQXACT_EXTERNAL API Environment: MQXE_OTHER Application Function: '' Appl Function Type: MQFUN_TYPE_UNKNOWN Trace Detail Level: 2 Trace Data Length: 0 Pointer size: 4 Platform: MQPL_WINDOWS_NT MQI Operation: 0 Operation Id: MQXF_CONN ApplicationTid: 1 OperationDate: '2013-03-15' OperationTime: '12:08:10' ConnectionId: 00000000: 414D 5143 5445 5354 514D 2020 2020 2020 'AMQCTESTQM ' 00000010: FFFFFFB5FFFFFFF6 4251 2000 FFFFFFE601 '.öBQ .æ. ' QueueManager: 'TESTQM' Completion Code: MQCC_OK Reason Code: 0 Connect Options: 256
WebSphere MQ Explorer Application Activity Trace viewer
SupportPac MSOP: WebSphere MQ Explorer – Extended Management Plug-ins provides an Application Activity Trace viewer that formats Application Activity Trace messages and also sorts the trace messages by application, which enables you to easily view the flow of messages by application.
After installing the Extended Management Plug-in for MQ Explorer, you can format Application Activity Trace messages by right-clicking on SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE in the content pane and clicking on Format Activity Records. The window shown below opens, enabling you can specify a number of options for formatting Application Activity Trace messages:
Figure 3. Options for displaying Application Activity Trace
The Merge Activity option specifies that an attempt is made to associate chains of report messages as part of the same application. The scenarios in this article all use this option. The formatted messages are displayed in the events and statistics viewer, as shown below:
Figure 4. Events and Statistics Viewer
Troubleshooting application behaviour using Application Activity Trace
Application Activity Trace can help you troubleshoot issues with WebSphere MQ application behaviour because it gives you a detailed view of the MQI calls that an application makes. Three scenarios in this section show you how to use Application Activity Trace to troubleshoot issues with application behaviour. The scenarios in this section are based on a simple WebSphere MQ environment at Company X, as shown in Figure 5 below. A number of WebSphere MQ client applications connect into the queue manager CLUSQM_1, which is a member of the WebSphere MQ cluster TESTSCLUSTER, and this cluster contains two other queue managers, CLUSQM_2 and CLUSQM_3.
Figure 5. WebSphere MQ environment at Company X
Scenario 1. Messages are not being work-load balanced between cluster queues
In Scenario 1, the WebSphere MQ administrator at Company X notices that messages destined for one of their cluster queues are not being evenly distributed between all the instances of the queue across the cluster. The flow of messages in this scenario originates from a number of client applications external to the WebSphere MQ Cluster. Messages are sent from the applications to the queue CLUS_Q, which is hosted on two queue managers in the cluster, CLUSQM_2 and CLUSQM_3, via the queue manager CLUSQM_1.
Development standards at Company X specify that the MQOPEN option MQOO_BIND_NOT_FIXED should be used when putting to cluster queues to enable workload balancing of messages across all the instances of a cluster queue. The administrator does not have access to the application code to verify whether the MQOPEN options are correct or whether one of the applications is incorrectly using the MQOO_BIND_ON_OPEN option, causing the uneven distribution of messages to CLUS_Q.
In order to verify whether MQ_BIND_NOT_FIXED is being used by all of the connecting applications, the administrator turns on Application Activity Trace on CLUSQM_1, and uses the Events and Statistics Viewer in MQ Explorer format the messages. Figure 6 is an extract from the Events and Statistics Viewer that clearly shows that one of the applications is using the MQOO_BIND_ON_OPEN option, enabling the administrator to quickly identify the application causing the unexpected behaviour.
Figure 6. An application using BIND_ON_OPEN
Scenario 2. Queue manager hitting maxchannels limit
The administrator at Company X has configured the queue manager CLUSQM_1 with a maxchannels value of 500, which means that no more than 500 channels can be connected to a queue manager at a time. This setting has been in place for a number of years without issues, but in recent weeks the administrator has received alerts saying that the maxchannels value has been reached, and the administrator has temporarily increased the maxchannels value for the queue manager in order to accommodate all of the connections.
In order to determine whether there is a legitimate increase in the number of connections, or whether applications are keeping connections open with the queue manager by failing to disconnect from the queue manager when they have finished processing, the administrator turns on application activity tracing on CLUSQM_1. As the Activity Trace Viewer merges together records for each application, it clearly shows which applications are disconnecting from the queue manager and which ones are keeping connections open.
Figure 7 below from the Events and Statistics viewer shows an application that issues a disconnect for every connection it makes to the queue manager, whilst Figure 8 shows an application that hasn't issued a disconnect. Running the Application Activity Trace over a longer period of time will enable the administrator to confirm whether a disconnect is ever issued.
Figure 7. An application disconnecting from a queue manager after closing a queue
Figure 8. An application failing to disconnect from the queue manager
Scenario 3. Using Application Activity Trace to keep an audit trail of messages
Another use for Application Activity Trace is to keep an audit trail of critical messages. Since an audit trail requires Application Activity Trace to be turned on permanently, set the trace level to LOW if possible in order to minimize the performance impact. You can reduce the number of Application Activity Trace messages sent across the network by tuning the ActivityInterval and ActivityCount parameters in the mqat.ini file. Each MQI operation produces an activity record, and these parameters let you configure how many of these records are bundled into an activity trace message based on time or record count thresholds.
In Scenario 3, the administrator for Company X is using Application Activity Trace to maintain an audit trail of the transfer of messages across the three queue managers in the WebSphere MQ cluster TESTCLUSTER. In order to centralize the collection of Application Activity Trace messages for all of the queue managers in the WebSphere MQ cluster, you can reroute all messages from the SYSTEM.ADMIN.ACTIVITY.TRACE.QUEUE to a publish/subscribe topic by redefining the SYSTEM.ADMIN.ACTIVITY.TRACE.QUEUE as a clustered topic object. Then all Application Activity Trace records will be rerouted to the newly defined topic, and you can then configure your formatting application to subscribe to the topic, so they can be monitored centrally. Here are the steps to reroute the Activity Trace records:
- On CLUSQM_1, define a clustered topic to send the Application Activity Trace messages, as shown in this example runmqsc command:
DEFINE TOPIC(TRACETOPIC)+ TOPICSTR('/Application/Activity/Trace')+ CLUSTER(TESTCLUSTER)
- Delete the SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE queue and redefine it as an alias queue resolving to the new topic defined in Step 1, as shown in this example runmqsc:
DELETE QLOCAL(SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE) DEFINE QALIAS(SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE)+ TARGTYPE(TOPIC) TARGET(TRACETOPIC)
- On CLUSQM_2 and CLUSQM_3, delete the SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE queue and redefine it as an alias queue resolving to the new topic defined in Step 1, as shown in this example runmqsc:
DELETE QLOCAL(SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE DEFINE QALIAS(SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE)+ TARGTYPE(TOPIC) TARGET(TRACETOPIC)
Another option is to partition the topic tree by including the queue manager name in the topic string, and then subscribe to events using a wildcard. For example, you could publish to the following topic strings on CLUSQM_1, CLUSQM_2, and CLUSQM_3 respectively:
/Application/Activity/Trace/CLUSQM_1 /Application/Activity/Trace/CLUSQM_2 /Application/Activity/Trace/CLUSQM_3
The subscribing application could then subscribe to /Application/Activity/Trace/* to get Application Activity Trace messages for all of the queue managers, or it could subscribe to /Application/Activity/Trace/CLUSQM_1 to just get messages for CLUSQM_1. Since you have all of the Application Activity Trace messages publishing to a central location, it is easier to track messages across an environment. As an example, the application team sends a message with the following message id to CLUS_Q via CLUSQM_1:
00000000: 414D 5120 434C 5553 514D 5F31 2020 2020 'AMQ CLUSQM_1 ' 00000010: FFFFFFE5FFFFFFD7 FFFFFF9951 2006 1303 'å.™Q ... '
They have asked for confirmation of which queue manager the message has been sent to, and whether it has been successfully put onto the queue CLUS_Q. Using amqsact to format the Application Activity Trace output, the administrator can verify this information by searching for the message id within the output. To collect and format the messages for the topic /Application/Activity/Trace, you ca use the following command:
amqsact –m CLUSQM_1 –t /Application/Activity/Trace –w 30 > output_file.txt
Firstly, the administrator can confirm that the message is successfully put to CLUSQM_1 with the following MQPUT record:
MQI Operation: 1 Operation Id: MQXF_PUT ApplicationTid: 28 OperationDate: '2013-05-21' OperationTime: '17:05:14' High Res Time: 1369152314903178 Completion Code: MQCC_OK Reason Code: 0 Hobj: 13676096 Put Options: 8260 Msg length: 1 Known_dest_count: 0 Unknown_dest_count: 1 Invalid_dest_count: 0 Object_type: MQOT_Q Object_name: 'CLUS_Q' Object_Q_mgr_name: '' Resolved_Q_Name: 'CLUS_Q' Resolved_Q_mgr: 'CLUSQM_3' Resolved_local_Q_name: 'CLUS_Q' Resolved_local_Q_mgr: 'CLUSQM_1' Resolved_type: MQOT_Q Report Options: 0 Msg_type: MQMT_DATAGRAM Expiry: -1 Format_name: 'MQSTR' Priority: -1 Persistence: 2 Msg_id: 00000000: 414D 5120 434C 5553 514D 5F31 2020 2020 'AMQ CLUSQM_1 ' 00000010: FFFFFFE5FFFFFFD7 FFFFFF9951 2006 1303 'å.™Q ... ' Correl_id: 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 '................' 00000010: 0000 0000 0000 0000 '........ ' Reply_to_Q : '' Reply_to_Q_Mgr: '' Coded_char_set_id: 437 Encoding: 546 Put_date: '20130521' Put_time: '16051491'
You can see that the resolved queue manager is CLUSQM_3, so the administrator checks whether the message has arrived on CLUS_Q on CLUSQM_3. The following trace record confirms that it has:
MQI Operation: 1 Operation Id: MQXF_PUT ApplicationTid: 6 OperationDate: '2013-05-21' OperationTime: '17:05:14' High Res Time: 1369152314985842 Completion Code: MQCC_OK Reason Code: 0 Hobj: 14459304 Put Options: 272388 Msg length: 1 Recs_present: 0 Known_dest_count: 1 Unknown_dest_count: 0 Invalid_dest_count: 0 Object_type: MQOT_Q Object_name: 'CLUS_Q' Object_Q_mgr_name: 'CLUSQM_3' Resolved_Q_Name: 'CLUS_Q' Resolved_Q_mgr: 'CLUSQM_3' Resolved_local_Q_name: 'CLUS_Q' Resolved_local_Q_mgr: 'CLUSQM_3' Resolved_type: MQOT_Q Report Options: 0 Msg_type: MQMT_DATAGRAM Expiry: -1 Format_name: 'MQSTR' Priority: 0 Persistence: 0 Msg_id: 00000000: 414D 5120 434C 5553 514D 5F31 2020 2020 'AMQ CLUSQM_1 ' 00000010: FFFFFFE5FFFFFFD7 FFFFFF9951 2006 1303 'å.™Q ... ' Correl_id: 00000000: 0000 0000 0000 0000 0000 0000 0000 0000 '................' 00000010: 0000 0000 0000 0000 '........ ' Reply_to_Q : ' ' Reply_to_Q_Mgr: 'CLUSQM_1' Coded_char_set_id: 437 Encoding: 546 Put_date: '20130521' Put_time: '16051491'
This time, the resolved queue manager and the resolved local queue manager are the same, which means that this is the intended final destination for the message. If a message is put as part of a unit of work, then the message will not be available to a retrieving application until the unit of work has been committed. Application Activity Trace will also write a message for the commit or backout with operation IDs of MQXF_CMIT or MQXF_BACK respectively. To confirm whether a message put under syncpoint arrives at the final destination, use the Application Activity Trace output to confirm whether the message has been committed or backed out.
The Application Activity Trace feature gives you a detailed view of the MQI calls made on a queue manager. This article has demonstrated some of its uses and configurable options, as well as some examples of how to format and interpret Application Activity Trace output.
The author thanks IBM colleagues Jonathan Rumsey (Senior Software Engineer, WebSphere MQ Development) and Mark E. Taylor (Software Developer, WebSphere MQ Technical Strategy) for their assistance and guidance in putting together this article.
- Application Activity Trace
- SupportPac MS0P: WebSphere MQ Explorer – Extended Management Plug-ins
- IBM MQ documentation in IBM Knowledge Center
- IBM MQ Developer Center
- IBM MQ documentation library
- Download a free trial version of IBM MQ
- developerWorks on Twitter