Transaction traceability of enterprise files, data, and applications: Part 3: Tracking operational transaction data in files, data, and applications using IBM Tivoli Composite Application Manager for Transactions

This article series describes a scenario involving JK Financials, a fictitious bank that needs to gain end-to-end visibility over its integrated enterprise IT infrastructure. Part 3 shows you how to use IBM Tivoli Composite Application Manager for Transactions to track end-to-end movement of files, transactions, applications, and components using IBM Sterling Connect:Direct. It will show you how to use WebSphere Message Broker to emit transaction data to IBM Tivoli Composite Application Manager for Transactions, and quickly identify operational and transactional problems involving a money transfer application.

Share:

Devaprasad Nadgir (ndevapra@in.ibm.com), Senior Software Engineer, Emerging Technologies team, IBM

Photo of Devaprasad NadgirDevaprasad Nadgir is an IBM Certified Senior IT Architect and works with the WebSphere Service Registry and Repository and SOA Governance team at the IBM India Software Lab in Bangalore. You can contact Devaprasad at ndevapra@in.ibm.com.



Abhinav Priyadarshi (pabhinav@in.ibm.com), Technical Project Lead, WebSphere Message Broker Development team, IBM

Photo of Abhinav PriyadarshiAbhinav Priyadarshi is an Advisory IT Architect and Technical Lead for the IBM Integration Bus Development Team at the IBM India Software Lab in Bangalore. He has 12 years of experience with IBM. He develops new features and nodes for IBM Integration Bus, and provides consultancy on IBM Integration Bus Proofs of Concept, EAI design, and implementation for customers in India and the Asia-Pacific region. You can contact Abhinav at pabhinav@in.ibm.com.



12 September 2012

Also available in Russian

Introduction

This article series describes a scenario involving JK Financials, a fictitious bank that needs to gain end-to-end visibility over its integrated enterprise IT infrastructure. This visibility will help provide transaction traceability and ways to address the IT infrastructure problems that arise during regular operations.

Part 1 described the challenges faced by large enterprises today in integrating file-based systems with enterprise applications spread across multiple networks and supporting different protocols. Part 1 showed you how to integrate IBM® Sterling Connect:Direct (hereafter called Sterling C:D) with IBM WebSphere® MQ in order to generate MQ messages that can be used by an orchestration engine such as WebSphere Message Broker or any application capable of consuming an MQ event.

Part 2 showed you how to build a WebSphere Message Broker orchestration that correlates file arrival events in a WebSphere Message Broker flow, generates a unique ID to track file transfer details using monitoring tools, and computes a routing endpoint and protocol based on the file contents in the Sterling C:D file.

Part 3 will show you how to use IBM Tivoli® Composite Application Manager for Transactions (hereafter called ITCAM for Transactions) to track end-to-end movement of files, transactions, applications, and components using IBM Sterling C:D. It will also show you how to use WebSphere Message Broker to emit transaction data to ITCAM for Transactions, and quickly identify operational and transactional problems involving a money transfer application.

Installing and configuring WebSphere Message Broker and ITCAM for Transactions

Here are the detailed instructions from the information center: Installing Transaction Tracking components of ITCAM for Transactions V7.3. Here is an overview:

  1. Download ITCAM for Transactions V7.3 Transaction Tracking for the appropriate platform.
  2. After installing ITCAM for Transactions V7.3, make sure that Transaction Collector and Transaction Reporter V7.3 are installed.
  3. Verify the installation:
    Figure 1. Verify installation of ITCAM for Transactions
    Verify installation of ITCAM for Transactions
  4. Search for Transaction Reporter and Transaction Collector in the list of installed Tivoli agents, as shown below:
    Figure 2. Search results for Transaction Collector and Transaction Reporter
    Search results for Transaction Collector and Transaction Reporter

Configuring ITCAM for Transactions with WebSphere Message Broker

This section shows you how to install ITCAM for Transactions to monitor transaction details from WebSphere Message Broker.

  1. In a command window, change directory to C:\IBM\ITM\TMAITM6\k3\bin.
  2. Run configDC.bat –enable. Make sure the command is successful:
    Figure 3. Configuration command execution and results
    Configuration command execution and results
  3. Using the mqsichangebroker command, set the active user exits name as KK3UserExit:
    C:\IBM\ITM\TMAITM6\k3\bin>mqsichangebroker <BrokerName> -e KK3UserExit

    Here the user exits are provided by ITCAM, which captures the required transaction data from WebSphere Message Broker.

  4. Verify that the user exits are successfully set in WebSphere Message Broker using the command mqsireportbroker <BrokerName>, as shown below:
    Figure 4. Command execution results for a user exit configuration
    Command execution results for a user exit configuration
  5. Restart WebSphere Message Broker.

Implementing transaction tracking using WebSphere Message Broker and ITCAM for Transactions

Part 1 of this article series explained the JK Financials scenario shown below in Figure 5. This scenario involves transaction tracking using ITCAM for Transactions, and managed file transfer within its enterprise network using IBM Sterling C:D. In this scenario, a branch of JK Financials at a different location from the data center sends money transfer instructions in a file using secure transfer protocol:

Figure 5. Architectural overview using WebSphere Message Broker
Architectural overview using WebSphere Message Broker

The message flow that implements the banking scenario is shown below. For more details about the flow implementation, see Part 2 of this article series.

Figure 6. Correlating events of file arrival using WebSphere Message Broker flow
Correlating events of file arrival using WebSphere Message Broker flow
  1. Deploy the message flow shown in Figure 5 on the ITCAM configured WebSphere Message Broker execution group.
  2. Run the message flow as explained in Part 2 of this article series.
  3. Open Tivoli Enterprise portal and then open the Transaction Topology workspace.
  4. Launch Tivoli Enterprise Portal and then open the Transaction Reporter workspace: Select Transaction Reporter => Transactions => Transaction Topology:
    Figure 7. Transaction Viewer in Tivoli Enterprise Portal
    Transaction Viewer in Tivoli Enterprise Portal

The Transaction Aggregate Topology captures the transaction processing path through various application components, and it shows you how different applications in the system interact to process the transaction. Figure 8 shows the Transaction Aggregate Topology view:

Figure 8. Transaction Aggregate Topology view in Transaction Reporter
Transaction Aggregate Topology view in Transaction Reporter

In the Transaction topology workspace, you can see how transactions flow through various application end points connected to a WebSphere Message Broker orchestration. Here the event message comes through SCCQueue, goes to a message flow called db2admin, and then goes to either OUTPUTQ or ERRORQ.

In this topology, you can see individual transaction instances captured by ITCAM, check transaction performance, and view details about individual transactions and context views, as shown below in Figure 9. As you can see, there are some slow transactions, which you can act on in order to meet service-level agreements (SLAs).

Figure 9. Transaction instances view in ITCAM for Transactions
Transaction instances view in ITCAM for Transactions

Similarly, you can view the Application Topology and Component Topology workspaces to view the applications and components involved in a transaction in a topology diagram. If any application is down, then ITCAM will detect it and show an alert.

You can use the Situation Editor and set some criteria to show an alert in ITCAM using the Universal Message Console. For example, you could specify that if the currentDepth of a queue on the Message Broker queue ERRORQ is greater than 0, then show an alert. In the Transaction Topology view, you can select ERRORQ as an application endpoint where a failure message arrives, and send a Short Message Service (SMS) text. An SMS gateway needs to be installed, and then you run a command to send an SMS text when a message arrives in the ERRORQ. You can configure alerts like these, as shown below.

The Situation Event Console below shows various system alerts classified as "Warning" and "Critical," which helps the administrator of ITCAM for Transactions react appropriately.

Figure 10. Situation Event Console for alerts
Situation Event Console for alerts

The Universal Message Console shows customized situation events. Here a situation event is created to show an alert when any transaction message is routed to ERRORQ. The current depth property of ERRORQ is used to capture such event. If the current depth of ERRORQ exceeds 0, the Situation Handler sends an alert to the Universal Message Console:

Figure 11. Viewing failed transactions in Universal Message Console
Viewing failed transactions in Universal Message Console

Viewing transactions using ITCAM for Transactions

Part 1 of this article series explained the architecture of the JK Financials multi-protocol branch integration using WebSphere Message Broker. Figure 12 shows the overall architecture:

Figure 12. Component overview using WebSphere Message Broker
Component overview using WebSphere Message Broker

Part 2 of this article series explained the message flow for the scenario, as shown below:

Figure 13. Simplified WebSphere Message Broker orchestration
Simplified WebSphere Message Broker orchestration

This flow receives the file transferred by different bank branches to the enterprise data center at CDNode A1. The input messages file details are provided in Part 1. It processes all transactions available in each file. The CDInput node sends each transaction in the file to the next compute node, called BaseOnRequestTypeRouteIndividualTransaction. This compute node routes the messages to either the NEFT queue or DD Print endpoints, based on the RequestType field value in the input message.

The message flow is deployed and transaction data files are sent to the enterprise data center using the Sterling C:D Server available at different bank branches. The Sterling C:D secondary node at the enterprise data center receives the data files from all branches. The CDInput node fetches the data file received at the enterprise data center after it is reprocessed through the local Sterling C:D node at the enterprise data center.

Running the message flow

After deploying the message flow in the execution group, open the Sterling C:D Requester Tool and process MsgFile2.txt from the branchesdatacollection folder messages into the CDWindows\download\ folder. Send all files from different branches using the Send/Receive File Utility of the Sterling C:D node:

Figure 14. Testing end-to-end flow using Sterling C:D Console
Testing end-to-end flow using Sterling C:D Console

Use the amqsget utility to receive any messages routed to the NEFF queue as shown in Figure 15 below. Two DDn.txt messages will be stored in the file system.

Figure 15. Running amqsget utility to send sample data file
Running amqsget utility to send sample data file

The two other transactions on DD print will be stored in the file system using the CDOutput node. For the two transaction requests of type DD, the message is routed to the CDOutput node, where it generates DD16.txt and DD17.txt files in the output file system folder, as shown below:

Figure 16. Testing arrival of a file in Sterling C:D configured node folder
Testing arrival of a file in Sterling C:D configured node folder

After processing the messages through the Sterling C:D Server and the WebSphere Message Broker flow, open the Tivoli Enterprise Portal, using userid sysadmin and password sysadmin.

Go to Transaction Reporter. The Sterling C:D Node connected to the broker system is shown. Here it can detect a remote C:D node also and show the C:D node details. Figure 17 shows that the broker system ISLRPBEIXV378 is connected to C:D node islrpbeixv378. The C:D server node is a local node. It will also show the remote C:D server node connection if any file transfer happens through a remote C:D server node to the broker system ISLRPBEIXV378.

Figure 17. Server Component Topology view
Server Component Topology view

Viewing the application in ITCAM

Three applications have been detected:

  • DEMOQM -- WebSphere MQ application
  • itcamEG -- WebSphere Message Broker application execution group process
  • ISLRPBEIXV378 -- Sterling C:D node server application

In Figure 18, the Applications panel on the left shows four applications: DEMOQM, ISLRPBEIXV378, islrpbeixv378, and itcamEG. It shows the availability of these applications in the Lowest Availability panel on the right. The three green bars show that three applications are running well, while the yellow bar indicates that DEMOQM has a performance issue and message processing through it is slow:

Figure 18. Applications detected by ITCAM for Transactions
Applications detected by ITCAM for Transactions
  1. Open the Application Aggregate Topology workspace to view how these composite applications are connected.
  2. In the Tivoli Enterprise Portal, select Transaction Reporter => Applications => Workspace => Application Topology.
  3. The Application Topology view is shown below. The Application Aggregate Topology shows you how various applications are wired together in the overall solution:
    Figure 19. Application Aggregate Topology view
    Application Aggregate Topology view
  4. In the Application Topology workspace view, you see that a Sterling C:D Server ISLRPBEIXV378 is connected to the WebSphere Message Broker execution group itcamEG, which is connected to the WebSphere MQ application DEMOQM. DEMOQM is lightly coupled with amqsget.exe, which consumes the NEFT messages from the queue. The execution group itcamEG application is connected to the Sterling CD server islrbpeix378 to create an output file for any DD type request.

Viewing the Components in ITCAM

Select the components in the Transaction Reporter, as shown below in the Tivoli Enterprise Portal.

In the Components view, you see that it has detected four components: two Sterling C:D, one WebSphere Message Broker, and one WebSphere MQ.

In Figure 20, the Components panel on the left shows the four components detected by the Transaction Collector and Transaction Reporter. The Lowest Availability panel on the right shows the status of these components:

Figure 20. Components view in Transaction Reporter
Components view in Transaction Reporter

To see how these components are aggregated, perform the following steps:

  1. Select Transaction Reporter => Components => Workspace => Component Topology.
  2. The Component Aggregate Topology shows you how various application components are wired together and running in WebSphere Message Broker, as shown in Figure 21. The Sterling C:D Server is the starting point that sends the file to WebSphere Message Broker, which picks up the transactions in the transferred file and routes them to either WebSphere MQ or C:D components. The WebSphere MQ component puts the message into a queue, while the C:D component creates an output file. Finally, the WebSphere MQ client component consumes the messages in the message queue. The dotted line shows that the WebSphere MQ client is loosely coupled, while the other components are tightly coupled:.
    Figure 21. Component Aggregate Topology view
    Component Aggregate Topology view

Viewing the transaction in ITCAM

The Transaction view lets you view transaction performance at five minute intervals, and you can adjust this polling interval.

  1. Open the Tivoli Enterprise Portal and select Transaction Reporter => Transaction.
  2. There are four transaction endpoints as shown below in Figure 22 in different colored boxes. PROC3 is the process that initiates the transaction and starts the file transfer. FileTransfer_CDNodes is the flow that consumes the file transfer started from PROC3 and routes messages inside the file to NEFTQ and two messages to DDPrint, since the Sterling C:D server created two different processes to create a DDPrint output file. The Transactions panel on the left contains columns for different performance evaluation parameters. Total Time shows the time elapsed in each of these application components:
    Figure 22. Transaction view in Transaction Reporter
    Transaction view in Transaction Reporter
  3. To view details about transaction performance in these application components, open the Transaction Aggregate Topology view: Select Transaction Reporter => Workspace => Transaction Topology.
  4. In this scenario, the Sterling C:D server process PROC3 sends a file having multiple transaction records to the message flow FileTransfer_CDNodes, which routes the transaction records to either NEFTQ (a WebSphere MQ queue) or DDPrint process (a Sterling C:D process to create the output file). The Transaction Aggregate Topology view shows how these individual applications are wired as part of the overall transaction processing system. The topology is detected automatically by the Transaction Collector and Transaction Reporter of ITCAM for Transactions.
  5. In Figure 23 below, the ITCAM Transaction Reporter detects the file transfer initiated by the Sterling C:D server process PROC3. It reaches the WebSphere Message Broker flow FileTransfer_CDNodes, and then the message is routed either to the CDOutput node or to the MQOutput node in the message flow, based on the message contents. Either the CDOutput node's Sterling C:D Server process DDPrint writes the output file to the file system, or else the MQOutput node puts the message to NEFTQ. In Figure 23, the FileTransfer_CDNodes message flow contains two routing paths: DDPrint and NEFTQ . The amqsget application shown as db2admin consumes the messages that arrive in the NEFTQ output queue:
    Figure 23. Transaction Aggregate Topology view
    Transaction Aggregate Topology view

Viewing the transaction instance

An individual transaction instance can help you identify details about a specific transaction, so that you can query on what happened to an individual transaction using a localtransactionId or sub-transaction, in case a transaction contains multiple sub-transactions. These kinds of queries can help you maintain security and SLAs. For example, if a message is routed to an error queue or dead letter queue instead of to the desired destination, you can diagnose and correct the problem and resend the request.

In this scenario, a file contains four transactions, which get processed and routed individually to various destinations by the WebSphere Message Broker flow. In the individual Transaction Instance view, you can see that the transaction file transferred by the bank contains four transaction records, shown below in pink. Two of them (shown in green) are routed to DDPrint, and two (shown in blue) are routed to NEFTQ. The view also shows that the NEFTQ transaction is slow, and therefore you should take some corrective action to maintain the SLA.

Figure 24. Transaction Instance Topology view
Transaction InstanceTopology view

The CDInput node of the message flow FileTransfer_CDNodes reads each of the four messages individually, and the message flow Compute node routes two of them to NEFTQ and two of them to DDPrint to a file to print the demand draft. The slow transaction on the WebSphere MQ NEFT queue is shown as an alert in yellow, indicating that WebSphere MQ has a performance issue. If the file contains more transaction records, then the destination of each transaction record is shown in the Transaction Instance Topology view.

Transaction Contexts view

The Transaction Contexts view is part of the Transaction Instance Topology view, and it shows details about an individual input or output transaction instance in a table.

In the Contexts view shown in Figures 25 and 26, you see that the transaction instance contains a file named MsgFile2.txt transferred by the Sterling C:D Server PROC3 (in pink) and it comes to the WebSphere Message Broker flow as a transaction. MsgFile2.txt contains four sub-transaction messages identified by LocalTransactionID: MsgFile2.txt_20120311_124421_197168. Due to a limitation in ITCAM, the individual sub-transaction ID is not getting overridden or populated, and therefore four sub-transactions are shown with the same ID. However, the LocalTransactionID shows where all four transactions have gone -- two of them to NEFTQ with respective WebSphere MQ MessageId's (shown in blue), and two to the DDPrint application (shown in green), which is a Sterling C:D application and creates output files named DD72.txt and DD73.txt.

Figure 25. Contexts table view of individual transaction instance
Contexts table view of Individual transaction instance
Figure 26. Contexts table view of Individual transaction instance
Contexts table view of Individual transaction instance

To achieve the transaction tracking in WebSphere Message Broker, ITCAM for Transactions provides a user exit that instruments its code into the WebSphere Message Broker runtime, and fetches the necessary transaction information, and displays it in various Transaction Reporter workspace views. The Transaction Collector uses the WebSphere Message Broker user exit to collect various transaction messages processed through the deployed message flow, and Transaction Reporter then shows the activities in its various workspaces.

Conclusion

This article showed you how to use ITCAM for Transactions to trace a transaction that is processed through IBM Sterling Connect:Direct, WebSphere Message Broker, and WebSphere MQ. ITCAM for Transactions can help you meet SLAs by showing alerts for slow transactions and enabling you to act before an SLA is violated.

This article series showed you how to integrate IBM Sterling Connect:Direct and WebSphere Message Broker to provide end-to-end visibility of transactions involving files and application-specific data across different protocols and networks.

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=834988
ArticleTitle=Transaction traceability of enterprise files, data, and applications: Part 3: Tracking operational transaction data in files, data, and applications using IBM Tivoli Composite Application Manager for Transactions
publish-date=09122012