Transaction traceability of enterprise files, data, and applications: Part 2: Orchestrating WebSphere Message Broker flow to process IBM Sterling Connect:Direct file transfers

This article series describes a scenario involving JK Financials, a fictitious bank that needs to gain end-to-end visibility over its integrated enterprise IT infrastructure. Part 2 shows you how to build a WebSphere Message Broker orchestration that correlates file arrival events in a WebSphere Message Broker flow, generates a unique ID to track file transfer details using monitoring tools, and computes a routing endpoint and protocol based on the file contents in the Sterling Connect:Direct file.

Devaprasad Nadgir (ndevapra@in.ibm.com), Senior Software Engineer, Emerging Technologies team, IBM

Photo of Devaprasad NadgirDevaprasad Nadgir is an IBM Certified Senior IT Architect and works with the WebSphere Service Registry and Repository and SOA Governance team at the IBM India Software Lab in Bangalore. You can contact Devaprasad at ndevapra@in.ibm.com.



Mallanagouda Patil (pmallana@in.ibm.com), Software Developer, WebSphere Message Broker Development team, IBM

Photo of Mallanagouda PatilMallanagouda Patil is a Software Developer on the WebSphere Message Broker Development team at the IBM India Software Lab in Bangalore. He has 10 years of experience in messaging and host integration technology, and he currently works on WebSphere Message Broker web services development, and interoperability testing with .NET, WebSphere Application Server, and other application servers. You can contact Mallanagouda at pmallana@in.ibm.com.



Amar Shah (samar@in.ibm.com), Software Developer, WebSphere Message Broker Level-3 Support team, IBM

Photo of Amar ShahAmar Shah is a Software Developer on the WebSphere Message Broker Level-3 Support team with IBM India. He also does serviceability improvements for WebSphere Message Broker and he is a co-author of the IBM Redbook "Managing WebSphere Message Broker Resources in a Production Environment." You can contact Amar at samar@in.ibm.com.



Shashi R Thambrahalli (shashikanth@in.ibm.com), Advisory Software Engineer, WebSphere MQ Client Development team, IBM

Photo of Shashi R. ThambrahalliShashikanth (Shashi) R. Thambrahalli is an Advisory Software Engineer on the WebSphere MQ Client Development team at the IBM India Software Lab in Bangalore. He has nine years of experience with IBM, focusing on WebSphere MQ clients, C/C++, and Microsoft .NET, XMS .NET, and WCF. He develops new features for MQ clients and provides consulting services on WebSphere MQ, WebSphere MQ FTE, and WebSphere MQ Advanced Message Security for customers in India and the Asia-Pacific region. You can contact Shashi at shashikanth@in.ibm.com.



11 July 2012

Also available in Chinese Russian

Introduction

This article series describes a scenario involving JK Financials, a fictitious bank that needs to gain end-to-end visibility over its integrated enterprise IT infrastructure. This visibility will help provide transaction traceability and ways to address the IT infrastructure problems that arise during regular operations.

Part 1 described the challenges faced by large enterprises today in integrating file-based systems with enterprise applications spread across multiple networks and supporting different protocols. Part 1 showed you how to integrate IBM® Sterling Connect:Direct (hereafter called Sterling C:D) with IBM WebSphere MQ in order to generate MQ messages that can be used by an orchestration engine such as WebSphere Message Broker or any application capable of consuming an MQ event.

Part 2 will show you how to build a WebSphere Message Broker orchestration that correlates file arrival events in a WebSphere Message Broker flow, generates a unique ID to track file transfer details using monitoring tools, and computes a routing endpoint and protocol based on the file contents in the Sterling C:D file.

To illustrate new features in WebSphere Message Broker V7 that support Sterling C:D nodes, this article shows sample implementations of the flow in both WebSphere Message Broker V6.1 and V7.

Money transfer orchestration using WebSphere Message Broker V6.1

WebSphere Message Broker V6.1 has no first-class support for processing files transferred by Sterling C:D, and does not provide a primitive to read events and contents from a configured C:D node. This article shows you how to transform a file that is sent by Sterling C:D node into records using WebSphere Message Broker File nodes, and then route the records to appropriate destinations based on record contents using other WebSphere Message Broker nodes. This article also shows you how to define an end-to-end transaction correlation using events emitted by Sterling Control Center. A WebSphere Message Broker message flow demonstrating this scenario is shown below.

Figure 1. Message flow to correlate file arrival event
Message flow to correlate file arrival event

Correlating file arrival event in the WebSphere Message Broker flow

This section describes the various activities performed by the message flow nodes.

SCC Event Input – MQInput node -- Receives events published by Sterling Control Center during file transfer.

Event filtering -- Sterling Control Center publishes events that describe the C:D node receiving a file. From the list of events, it looks for the event indicating successful completion of a file transfer to the configured directory. One such event from Sterling Control Center is shown below:

<scc:event xmlns:scc="http:///www.IBM   .com/SCC/SCCEvents.xsd">
    <scc:actionsFlag>1321512264468</scc:actionsFlag> 
    <scc:destFile>C:\Server\DOWNLOAD\testcd9.txt</scc:destFile> 
    <scc:direction>outBound</scc:direction> 
    <scc:eventId>86606373241817173</scc:eventId> 
    <scc:eventType>2</scc:eventType> 
    <scc:fileSize>19</scc:fileSize> 
    <scc:fromNode>P</scc:fromNode> 
    <scc:processId>93</scc:processId> 
    <scc:processName>SENDRECV</scc:processName> 
    <scc:recordId>CTRC</scc:recordId> 
    <scc:remoteNode>APACHE</scc:remoteNode> 
    <scc:returnCode>0</scc:returnCode> 
    <scc:shortText>Copy operation successful.</scc:shortText> 
    <scc:sourceFile>C:\CDWindows_file\upload\testcd9.txt</scc:sourceFile> 
    <scc:STPT>2011/11/17 05:33:13.000</scc:STPT> 
    <scc:STRT>2011/11/17 05:33:12.000</scc:STRT> 
    <scc:submitterId>cdadmin</scc:submitterId> 
</scc:event>

The event contains metadata for the transferred file, including the file name and status of the file transfer, as shown below:

Figure 2. Event details of a Sterling Control Center event used for correlation
Event details of a Sterling Control Center event used for correlation

Receive File -- A File Input node configured to poll the directory where the Sterling C:D node will place the transferred file, and to transform file into records. The message structure and the LocalEnvironment tree are populated automatically, as shown below:

Figure 3. Receive File node
Receive File node

Collector Node -- Correlates messages received from different sources by matching the name of the file from event and file node details. In this example, three records from a file and one event record from WebSphere MQ are configured to form the collection:

Figure 4. Collector node
Collector node

Generating a unique ID to track file transfer details using monitoring tools

Since messages and records were originally part of the file that is now being propagated as individual messages to disparate sources, there must be way to track them. Therefore you need to generate a unique identifier before propagating the message, which can be done by concatenating the file name and MQ message identifier. You can extract the file name from the file message and extract the MQ message identifier from the Sterling Control Center event, as shown below:

Figure 5. MQ message identifier
MQ message identifier
SET OutputRoot.XMLNSC.Customer.TransactionID = InputRoot.Collection.FileIn.XMLNSC.
Customer.FileName || '_'|| CAST(InputRoot.Collection.MQIn.MQMD.MsgId AS CHARACTER);

You compute a routing endpoint and routing protocol based on the contents of the Sterling C: D file. Records from the file are routed to different destinations depending on their contents, as shown below:

Figure 6. Computing the routing endpoint
Computing the routing endpoint

Money transfer orchestration using WebSphere Message Broker V7

WebSphere Message Broker V7.0.0.3 or later supports the processing of files transferred by Sterling C:D. This article shows you how to transform a file sent by a Sterling C:D server into records using the WebSphere Message Broker CDInput node and then routing the record to the appropriate destination based on the record contents using other nodes of WebSphere Message Broker. You do not need to handle Sterling Control Center events in this case, because the CDInput node receives both the contents of the file and the metadata provided by Sterling C:D on the transfer. You use the WebSphere Message Broker CDInput node to receive messages that have been transferred to a given C:D server, and you use WebSphere Message Broker CDOutput node to serialize the message tree to a file and then transfer it to a secondary C:D server using a primary C:D server.

The following example shows you how to use the C:D nodes when working with Sterling C:D in conjunction with WebSphere Message Broker.

Figure 7. Message flow to correlate file arrival event
Message flow to correlate file arrival event

Correlating the file arrival event in the WebSphere Message Broker flow

A CDInput node is configured to receive the file from Sterling C:D server and to transform the file into records. The message structure and LocalEnvironment tree are populated automatically. Figure 8 below shows data representation of messages after CDInput transforms the file into records and messages:

Figure 8. Data representation after CDInput transforms the file
Data representation after CDInput transforms the file

Generating a unique ID to track file transfer details using monitoring tools

Since messages and records were originally part of the file that is now being propagated as individual messages to disparate sources, there must be way to track them. Since C:D transfers do not provide a unique identifier, you need to generate one before propagating the message, which you can do by concatenating the filename with the timestamp of the file arrival. The CDInput node stores the filename and timestamp in the LocalEnvironment tree. The unique identifier is stored in the Environment structure, where it can be used by Tivoli Composite Application Manager to track the record. Figure 9 below shows the message data representation along with the unique identifier:

Figure 9. Message data representation and unique identifier
Message data representation and unique identifier

Computing a routing endpoint and routing protocol based on contents of the Sterling C:D file

Records from the file are routed to different destinations depending record contents, as shown below:

Figure 10. Computing routing endpoint
Computing routing endpoint

Conclusion

Part 2 of this article series has described the architectural elements involved in providing an integrated enterprise solution involving files, data, messages, events, and applications. The example has shown how to add the capabilities of reading a file and converting a message flow to different protocols supported by different networks and applications, giving JK Financials end-to-end visibility of customer records that arrive in a batch file and are processed as individual records for consumption by various back-end applications. This visibility provides transaction traceability and ways to address IT infrastructure problems that arise during operations. The example uses Sterling C:D as the managed file transfer product, but it would work equally well with any managed file transfer product supported by WebSphere Message Broker, such as WebSphere MQ File Transfer Edition (FTE).

Part 3 will bring together all pieces of the solution by introducing IBM Tivoli Composite Application Manager for Transactions to provide end-to-end transaction visibility for files and data transferred via the techniques described in Parts 1 and 2.

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=825144
ArticleTitle=Transaction traceability of enterprise files, data, and applications: Part 2: Orchestrating WebSphere Message Broker flow to process IBM Sterling Connect:Direct file transfers
publish-date=07112012