Increasing the visibility of messages using WebSphere MQ Application Activity Trace

This article describes the WebSphere MQ Application Activity Trace, a new feature in WebSphere MQ V7.1 on all distributed platforms. Until now it has been difficult for WebSphere MQ administrators to see what the applications connecting to queue managers are actually doing. Application Activity Trace provides a detailed view of application behaviour and application interaction with a WebSphere MQ queue manager and its resources.

Share:

Emma Bushby, Accelerated Value Program Specialist, WebSphere Message Broker and WebSphere MQ, IBM

Photo of Emma BushbyEmma Bushby is an Accelerated Value Program Specialist for WebSphere Message Broker and WebSphere MQ at the IBM Software Lab in Warwick, UK. She has been a WebSphere MQ product specialist for 12 years, including seven years in WebSphere MQ Level 2 Support. In her current role in the Accelerated Value Program, she works with clients in the Distribution, Financial, and Retail industries to help them maximize their return from using WebSphere MQ. You can contact Emma at emma_bushby@uk.ibm.com.



12 June 2013

Also available in Chinese

Introduction

This article uses scenarios to demonstrate potential uses of the IBM® WebSphere® MQ Application Activity Trace, including problem determination and maintaining an audit trail of WebSphere MQ messages. Two tools will be used in the scenarios to format the Application Activity Trace records for analysis:

  • The command-line tool amqsact, which is shipped with WebSphere MQ as a sample.
  • The events and statistics viewer included in SupportPac MS0P: WebSphere MQ Explorer – Extended Management Plug-ins. Download SupportPac MS0P.

Turning on Application Activity Trace

Application Activity Trace activation is controlled by a queue manager attribute called ACTVTRC. To turn tracing on, use the runmqsc command ALTER QMGR ACTVTRC(ON). To turn tracing off, use the runmqsc command ALTER QMGR ACTVTRC(OFF). You don't need to restart your queue manager to start or stop Application Activity Trace.

You can also turn Application Activity Trace on and off in WebSphere MQ Explorer, in the online monitoring section of the queue managers properties, as shown in Figure 1:

Figure 1. Enabling Application Activity Trace using WebSphere MQ Explorer
Enabling Application Activity Trace using WebSphere MQ Explorer

You can also turn on Application Activity Trace in the MQCONNX call. For more information, see MQCONNX options in the WebSphere MQ information centre.

Configuring Application Activity Trace

The configuration file mqat.ini enables you to control the frequency and level of detail in Application Activity Trace. The mqat.ini file is located in the queue manager data directory:

  • On Linux® and UNIX®: /var/mqm/qmgrs/<qm_name>
  • On Microsoft® Windows®: C:\Program Files\IBM\WebSphere MQ\qmgrs\<qm_name>

Where <qm_name> is the name of your queue manager.

Changes to the ini file can be made dynamically without restarting the queue manager. You just need to turn Application Activity Trace off and back on for changes to be picked up.

Choosing the Application Activity Trace level

You have a number of options for configuring the level of detail in Application Activity Trace output. Configurable options in the mqat.ini file include:

  • The ability to restrict the applications that the trace runs against. For example, you can specify the application you want to trace by specifying the ApplName in the ApplicationTrace stanza. Since the WebSphere MQ V7.5 Java client now contains the real application name instead of the generic WebSphere MQ client for Java, you now also have more flexibility in tracing Java applications.
  • The option to include message data in the Application Activity Trace message.
  • The ability to configure the time interval between trace messages, and to batch multiple trace records into a single message. This capability enables you to modify the number of messages generated by Application Activity Trace in cases where performance is a priority. The default is to generate an Application Activity Trace record for every Message Queue Interface (MQI) call made to the queue manager. Each Application Activity Trace message contains 100 trace records.
  • The ability to configure the level of detail included in the Application Activity Trace message, as described in the next section.

Depending on the granularity you need, you can set the trace level to LOW, MEDIUM or HIGH via the TraceLevel parameter in the Application Activity Trace configuration file mqat.ini. For example, the following extract from mqat.ini shows the TraceLevel set to MEDIUM (which is the default setting):

AllActivityTrace:           # Global settings stanza
  ActivityInterval=1        # Time interval between trace messages
                            #   Values: 0-99999999 (0=off)
                            #   Default: 0
  ActivityCount=100         # Number of operations between trace msgs
                            #   Values: 0-99999999 (0=off)
                            #   Default: 0
  TraceLevel=MEDIUM      # Amount of data traced for each operation
                            #   Values: LOW | MEDIUM | HIGH
                            #   Default: MEDIUM
  TraceMessageData=0        # Amount of message data traced
                            #   Values: 0-104857600
                            #   Default: 0
  StopOnGetTraceMsg=ON      # Stop trace on get of activity trace message
                            #   Values: ON | OFF
                            #   Default: ON

As an example of the output generated with different trace levels, Figure 2 below shows the output parameters for the MQPUT call. This data was taken from Application Activity Trace of a WebSphere MQ client application (the amqsputc sample) connecting to a queue manager and putting a message on a local queue:

Figure 2. MQPut parameters recorded with different trace levels
MQPut parameters recorded with different trace levels

The appropriate trace level depends on your reason for turning on Application Activity Trace. If you are troubleshooting a codepage conversion issue and need to check the CCSID of the message, then you need to at least set the level of trace detail to MEDIUM. On the other hand, if you are only interested in finding out the number of successful MQPUTs to a queue, then a LOW level of detail may be sufficient. To determine the appropriate level of trace detail for your requirements, turn on Application Activity Trace for a short period of time and try out different levels of trace detail to determine which level provides the amount of detail that you need.

Performance impact of Application Activity Trace

The impact of Application Activity Trace on performance depends on the level of detail you specify, the number of applications being traced, and the frequency with which activity messages are generated. Choose the lowest level of detail and frequency that gives you the information you need, especially if performance is critical. Ensure that the monitoring application reading the activity trace messages can keep up with the trace generation, so that queuing of activity trace messages does not create a bottleneck. For more information on the performance impact of activity tracing and on ways to reduce that impact, see Performance impact of Application Activity Trace in the WebSphere MQ information centre.

Formatting Application Activity Trace output

You can write your own application to read and format Application Activity Trace messages. For more information, see Application Activity Trace message reference in the WebSphere MQ information centre.

The scenarios in this article use the amqsact sample program and the Application Activity Trace viewer in WebSphere MQ Explorer to format trace messages.

amqsact sample program

amqsact formats Application Activity Trace messages for you and is provided with WebSphere MQ. The source code is also provided, so you can easily see how it works and can extend and modify the behaviour to meet your requirements. The compiled program is located in the samples directory:

  • On Linux and UNIX: MQ_INSTALLATION_PATH?samp?bin
  • On Windows: MQ_INSTALLATION_PATH\tools\c\Samples\Bin;

amqsact options

amqsact [-m QMgrName]
               [-q QName]        # Override default queue name
               [-t TopicString]  # Subscribe to event topic
               [-b]              # Only browse records
               [-v]          # Verbose output
               [-d <depth>]      # Number of records to display
               [-w <timeout>]    # Time to wait (in seconds)
               [-s <startTime>]  # Start time of record to process
               [-e <endTime>]    # End time of record to process

Example output

Here is an example of the output for an MQCONN API call when formatted using the command: amqsact –m TESTQM -v :

MonitoringType: MQI Activity Trace
Correl_id:
00000000:  414D 5143 5445 5354 514D 2020 2020 2020  'AMQCTESTQM      '
00000010:  B5F6 4251 2000 E601                      '.öBQ .æ.        '
QueueManager: 'TESTQM'
Host Name: 'ADMINIB-1VTJ6N1'
IntervalStartDate: '2013-03-15'
IntervalStartTime: '12:08:10'
IntervalEndDate: '2013-03-15'
IntervalEndTime: '12:08:10'
CommandLevel: 710
SeqNumber: 0
ApplicationName: 'WebSphere MQ_1\bin\amqsput.exe'
Application Type: MQAT_WINDOWS_NT
ApplicationPid: 14076
UserId: 'Emma_Bushby'
API Caller Type: MQXACT_EXTERNAL
API Environment: MQXE_OTHER
Application Function: ''
Appl Function Type: MQFUN_TYPE_UNKNOWN
Trace Detail Level: 2
Trace Data Length: 0
Pointer size: 4
Platform: MQPL_WINDOWS_NT
MQI Operation: 0
   Operation Id: MQXF_CONN
   ApplicationTid: 1
   OperationDate: '2013-03-15'
   OperationTime: '12:08:10'
   ConnectionId:
   00000000:  414D 5143 5445 5354 514D 2020 2020 2020  'AMQCTESTQM      '
   00000010:  FFFFFFB5FFFFFFF6 4251 2000 FFFFFFE601    '.öBQ .æ.        '
   QueueManager: 'TESTQM'
   Completion Code: MQCC_OK
   Reason Code: 0
   Connect Options: 256

WebSphere MQ Explorer Application Activity Trace viewer

SupportPac MSOP: WebSphere MQ Explorer – Extended Management Plug-ins provides an Application Activity Trace viewer that formats Application Activity Trace messages and also sorts the trace messages by application, which enables you to easily view the flow of messages by application.

After installing the Extended Management Plug-in for MQ Explorer, you can format Application Activity Trace messages by right-clicking on SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE in the content pane and clicking on Format Activity Records. The window shown below opens, enabling you can specify a number of options for formatting Application Activity Trace messages:

Figure 3. Options for displaying Application Activity Trace
Options for displaying Application Activity Trace

The Merge Activity option specifies that an attempt is made to associate chains of report messages as part of the same application. The scenarios in this article all use this option. The formatted messages are displayed in the events and statistics viewer, as shown below:

Figure 4. Events and Statistics Viewer
Events and Statistics Viewer

Troubleshooting application behaviour using Application Activity Trace

Application Activity Trace can help you troubleshoot issues with WebSphere MQ application behaviour because it gives you a detailed view of the MQI calls that an application makes. Three scenarios in this section show you how to use Application Activity Trace to troubleshoot issues with application behaviour. The scenarios in this section are based on a simple WebSphere MQ environment at Company X, as shown in Figure 5 below. A number of WebSphere MQ client applications connect into the queue manager CLUSQM_1, which is a member of the WebSphere MQ cluster TESTSCLUSTER, and this cluster contains two other queue managers, CLUSQM_2 and CLUSQM_3.

Figure 5. WebSphere MQ environment at Company X
WebSphere MQ environment at Company X

Scenario 1. Messages are not being work-load balanced between cluster queues

In Scenario 1, the WebSphere MQ administrator at Company X notices that messages destined for one of their cluster queues are not being evenly distributed between all the instances of the queue across the cluster. The flow of messages in this scenario originates from a number of client applications external to the WebSphere MQ Cluster. Messages are sent from the applications to the queue CLUS_Q, which is hosted on two queue managers in the cluster, CLUSQM_2 and CLUSQM_3, via the queue manager CLUSQM_1.

Development standards at Company X specify that the MQOPEN option MQOO_BIND_NOT_FIXED should be used when putting to cluster queues to enable workload balancing of messages across all the instances of a cluster queue. The administrator does not have access to the application code to verify whether the MQOPEN options are correct or whether one of the applications is incorrectly using the MQOO_BIND_ON_OPEN option, causing the uneven distribution of messages to CLUS_Q.

In order to verify whether MQ_BIND_NOT_FIXED is being used by all of the connecting applications, the administrator turns on Application Activity Trace on CLUSQM_1, and uses the Events and Statistics Viewer in MQ Explorer format the messages. Figure 6 is an extract from the Events and Statistics Viewer that clearly shows that one of the applications is using the MQOO_BIND_ON_OPEN option, enabling the administrator to quickly identify the application causing the unexpected behaviour.

Figure 6. An application using BIND_ON_OPEN
An application using BIND_ON_OPEN

Scenario 2. Queue manager hitting maxchannels limit

The administrator at Company X has configured the queue manager CLUSQM_1 with a maxchannels value of 500, which means that no more than 500 channels can be connected to a queue manager at a time. This setting has been in place for a number of years without issues, but in recent weeks the administrator has received alerts saying that the maxchannels value has been reached, and the administrator has temporarily increased the maxchannels value for the queue manager in order to accommodate all of the connections.

In order to determine whether there is a legitimate increase in the number of connections, or whether applications are keeping connections open with the queue manager by failing to disconnect from the queue manager when they have finished processing, the administrator turns on application activity tracing on CLUSQM_1. As the Activity Trace Viewer merges together records for each application, it clearly shows which applications are disconnecting from the queue manager and which ones are keeping connections open.

Figure 7 below from the Events and Statistics viewer shows an application that issues a disconnect for every connection it makes to the queue manager, whilst Figure 8 shows an application that hasn't issued a disconnect. Running the Application Activity Trace over a longer period of time will enable the administrator to confirm whether a disconnect is ever issued.

Figure 7. An application disconnecting from a queue manager after closing a queue
An application disconnecting from a queue manager after closing a queue
Figure 8. An application failing to disconnect from the queue manager
An application failing to disconnect from the queue manager

Scenario 3. Using Application Activity Trace to keep an audit trail of messages

Another use for Application Activity Trace is to keep an audit trail of critical messages. Since an audit trail requires Application Activity Trace to be turned on permanently, set the trace level to LOW if possible in order to minimize the performance impact. You can reduce the number of Application Activity Trace messages sent across the network by tuning the ActivityInterval and ActivityCount parameters in the mqat.ini file. Each MQI operation produces an activity record, and these parameters let you configure how many of these records are bundled into an activity trace message based on time or record count thresholds.

In Scenario 3, the administrator for Company X is using Application Activity Trace to maintain an audit trail of the transfer of messages across the three queue managers in the WebSphere MQ cluster TESTCLUSTER. In order to centralize the collection of Application Activity Trace messages for all of the queue managers in the WebSphere MQ cluster, you can reroute all messages from the SYSTEM.ADMIN.ACTIVITY.TRACE.QUEUE to a publish/subscribe topic by redefining the SYSTEM.ADMIN.ACTIVITY.TRACE.QUEUE as a clustered topic object. Then all Application Activity Trace records will be rerouted to the newly defined topic, and you can then configure your formatting application to subscribe to the topic, so they can be monitored centrally. Here are the steps to reroute the Activity Trace records:

  1. On CLUSQM_1, define a clustered topic to send the Application Activity Trace messages, as shown in this example runmqsc command:
    DEFINE TOPIC(TRACETOPIC)+ TOPICSTR('/Application/Activity/Trace')+
    CLUSTER(TESTCLUSTER)
  2. Delete the SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE queue and redefine it as an alias queue resolving to the new topic defined in Step 1, as shown in this example runmqsc:
    DELETE QLOCAL(SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE) 
    DEFINE QALIAS(SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE)+ TARGTYPE(TOPIC) TARGET(TRACETOPIC)
  3. On CLUSQM_2 and CLUSQM_3, delete the SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE queue and redefine it as an alias queue resolving to the new topic defined in Step 1, as shown in this example runmqsc:
    DELETE QLOCAL(SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE
    DEFINE QALIAS(SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE)+ TARGTYPE(TOPIC) TARGET(TRACETOPIC)

Another option is to partition the topic tree by including the queue manager name in the topic string, and then subscribe to events using a wildcard. For example, you could publish to the following topic strings on CLUSQM_1, CLUSQM_2, and CLUSQM_3 respectively:

/Application/Activity/Trace/CLUSQM_1
/Application/Activity/Trace/CLUSQM_2
/Application/Activity/Trace/CLUSQM_3

The subscribing application could then subscribe to /Application/Activity/Trace/* to get Application Activity Trace messages for all of the queue managers, or it could subscribe to /Application/Activity/Trace/CLUSQM_1 to just get messages for CLUSQM_1. Since you have all of the Application Activity Trace messages publishing to a central location, it is easier to track messages across an environment. As an example, the application team sends a message with the following message id to CLUS_Q via CLUSQM_1:

00000000:  414D 5120 434C 5553 514D 5F31 2020 2020  'AMQ CLUSQM_1    '
00000010:  FFFFFFE5FFFFFFD7 FFFFFF9951 2006 1303    'å.™Q ...        '

They have asked for confirmation of which queue manager the message has been sent to, and whether it has been successfully put onto the queue CLUS_Q. Using amqsact to format the Application Activity Trace output, the administrator can verify this information by searching for the message id within the output. To collect and format the messages for the topic /Application/Activity/Trace, you ca use the following command:

amqsact –m CLUSQM_1 –t /Application/Activity/Trace –w 30 > output_file.txt

Firstly, the administrator can confirm that the message is successfully put to CLUSQM_1 with the following MQPUT record:

MQI Operation: 1
   Operation Id: MQXF_PUT
   ApplicationTid: 28
   OperationDate: '2013-05-21'
   OperationTime: '17:05:14'
   High Res Time: 1369152314903178
   Completion Code: MQCC_OK
   Reason Code: 0
   Hobj: 13676096
   Put Options: 8260
   Msg length: 1
   Known_dest_count: 0
   Unknown_dest_count: 1
   Invalid_dest_count: 0
   Object_type: MQOT_Q
   Object_name: 'CLUS_Q'
   Object_Q_mgr_name: ''
   Resolved_Q_Name: 'CLUS_Q'
   Resolved_Q_mgr: 'CLUSQM_3'
   Resolved_local_Q_name: 'CLUS_Q'
   Resolved_local_Q_mgr: 'CLUSQM_1'
   Resolved_type: MQOT_Q
   Report Options: 0
   Msg_type: MQMT_DATAGRAM
   Expiry: -1
   Format_name: 'MQSTR'
   Priority: -1
   Persistence: 2
   Msg_id:
   00000000:  414D 5120 434C 5553 514D 5F31 2020 2020  'AMQ CLUSQM_1    '
   00000010:  FFFFFFE5FFFFFFD7 FFFFFF9951 2006 1303    'å.™Q ...        '
   Correl_id:
   00000000:  0000 0000 0000 0000 0000 0000 0000 0000  '................'
   00000010:  0000 0000 0000 0000                      '........        '
   Reply_to_Q : ''
   Reply_to_Q_Mgr: ''
   Coded_char_set_id: 437
   Encoding: 546
   Put_date: '20130521'
   Put_time: '16051491'

You can see that the resolved queue manager is CLUSQM_3, so the administrator checks whether the message has arrived on CLUS_Q on CLUSQM_3. The following trace record confirms that it has:

MQI Operation: 1
   Operation Id: MQXF_PUT
   ApplicationTid: 6
   OperationDate: '2013-05-21'
   OperationTime: '17:05:14'
   High Res Time: 1369152314985842
   Completion Code: MQCC_OK
   Reason Code: 0
   Hobj: 14459304
   Put Options: 272388
   Msg length: 1
   Recs_present: 0
   Known_dest_count: 1
   Unknown_dest_count: 0
   Invalid_dest_count: 0
   Object_type: MQOT_Q
   Object_name: 'CLUS_Q'
   Object_Q_mgr_name: 'CLUSQM_3'
   Resolved_Q_Name: 'CLUS_Q'
   Resolved_Q_mgr: 'CLUSQM_3'
   Resolved_local_Q_name: 'CLUS_Q'
   Resolved_local_Q_mgr: 'CLUSQM_3'
   Resolved_type: MQOT_Q
   Report Options: 0
   Msg_type: MQMT_DATAGRAM
   Expiry: -1
   Format_name: 'MQSTR'
   Priority: 0
   Persistence: 0
   Msg_id:
   00000000:  414D 5120 434C 5553 514D 5F31 2020 2020  'AMQ CLUSQM_1    '
   00000010:  FFFFFFE5FFFFFFD7 FFFFFF9951 2006 1303    'å.™Q ...        '
   Correl_id:
   00000000:  0000 0000 0000 0000 0000 0000 0000 0000  '................'
   00000010:  0000 0000 0000 0000                      '........        '
   Reply_to_Q : '                                                '
   Reply_to_Q_Mgr: 'CLUSQM_1'
   Coded_char_set_id: 437
   Encoding: 546
   Put_date: '20130521'
   Put_time: '16051491'

This time, the resolved queue manager and the resolved local queue manager are the same, which means that this is the intended final destination for the message. If a message is put as part of a unit of work, then the message will not be available to a retrieving application until the unit of work has been committed. Application Activity Trace will also write a message for the commit or backout with operation IDs of MQXF_CMIT or MQXF_BACK respectively. To confirm whether a message put under syncpoint arrives at the final destination, use the Application Activity Trace output to confirm whether the message has been committed or backed out.

Conclusion

The Application Activity Trace feature gives you a detailed view of the MQI calls made on a queue manager. This article has demonstrated some of its uses and configurable options, as well as some examples of how to format and interpret Application Activity Trace output.

Acknowledgements

The author would like to thank IBM colleagues Jonathan Rumsey (Senior Software Engineer, WebSphere MQ Development) and Mark E. Taylor (Software Developer, WebSphere MQ Technical Strategy) for their assistance and guidance in putting together this article.

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=932934
ArticleTitle=Increasing the visibility of messages using WebSphere MQ Application Activity Trace
publish-date=06122013