Transaction traceability of enterprise files, data, and applications
Part 2: Orchestrating WebSphere Message Broker flow to process IBM Sterling Connect:Direct file transfers
This content is part # of # in the series: Transaction traceability of enterprise files, data, and applications
This content is part of the series:Transaction traceability of enterprise files, data, and applications
Stay tuned for additional content in this series.
This article series describes a scenario involving JK Financials, a fictitious bank that needs to gain end-to-end visibility over its integrated enterprise IT infrastructure. This visibility will help provide transaction traceability and ways to address the IT infrastructure problems that arise during regular operations.
Part 1 described the challenges faced by large enterprises today in integrating file-based systems with enterprise applications spread across multiple networks and supporting different protocols. Part 1 showed you how to integrate IBM® Sterling Connect:Direct (hereafter called Sterling C:D) with IBM WebSphere MQ in order to generate MQ messages that can be used by an orchestration engine such as WebSphere Message Broker or any application capable of consuming an MQ event.
Part 2 will show you how to build a WebSphere Message Broker orchestration that correlates file arrival events in a WebSphere Message Broker flow, generates a unique ID to track file transfer details using monitoring tools, and computes a routing endpoint and protocol based on the file contents in the Sterling C:D file.
To illustrate new features in WebSphere Message Broker V7 that support Sterling C:D nodes, this article shows sample implementations of the flow in both WebSphere Message Broker V6.1 and V7.
Money transfer orchestration using WebSphere Message Broker V6.1
WebSphere Message Broker V6.1 has no first-class support for processing files transferred by Sterling C:D, and does not provide a primitive to read events and contents from a configured C:D node. This article shows you how to transform a file that is sent by Sterling C:D node into records using WebSphere Message Broker File nodes, and then route the records to appropriate destinations based on record contents using other WebSphere Message Broker nodes. This article also shows you how to define an end-to-end transaction correlation using events emitted by Sterling Control Center. A WebSphere Message Broker message flow demonstrating this scenario is shown below.
Figure 1. Message flow to correlate file arrival event
Correlating file arrival event in the WebSphere Message Broker flow
This section describes the various activities performed by the message flow nodes.
SCC Event Input – MQInput node -- Receives events published by Sterling Control Center during file transfer.
Event filtering -- Sterling Control Center publishes events that describe the C:D node receiving a file. From the list of events, it looks for the event indicating successful completion of a file transfer to the configured directory. One such event from Sterling Control Center is shown below:
<scc:event xmlns:scc="http:///www.IBM .com/SCC/SCCEvents.xsd"> <scc:actionsFlag>1321512264468</scc:actionsFlag> <scc:destFile>C:\Server\DOWNLOAD\testcd9.txt</scc:destFile> <scc:direction>outBound</scc:direction> <scc:eventId>86606373241817173</scc:eventId> <scc:eventType>2</scc:eventType> <scc:fileSize>19</scc:fileSize> <scc:fromNode>P</scc:fromNode> <scc:processId>93</scc:processId> <scc:processName>SENDRECV</scc:processName> <scc:recordId>CTRC</scc:recordId> <scc:remoteNode>APACHE</scc:remoteNode> <scc:returnCode>0</scc:returnCode> <scc:shortText>Copy operation successful.</scc:shortText> <scc:sourceFile>C:\CDWindows_file\upload\testcd9.txt</scc:sourceFile> <scc:STPT>2011/11/17 05:33:13.000</scc:STPT> <scc:STRT>2011/11/17 05:33:12.000</scc:STRT> <scc:submitterId>cdadmin</scc:submitterId> </scc:event>
The event contains metadata for the transferred file, including the file name and status of the file transfer, as shown below:
Figure 2. Event details of a Sterling Control Center event used for correlation
Receive File -- A File Input node configured to poll the directory where the Sterling C:D node will place the transferred file, and to transform file into records. The message structure and the LocalEnvironment tree are populated automatically, as shown below:
Figure 3. Receive File node
Collector Node -- Correlates messages received from different sources by matching the name of the file from event and file node details. In this example, three records from a file and one event record from WebSphere MQ are configured to form the collection:
Figure 4. Collector node
Generating a unique ID to track file transfer details using monitoring tools
Since messages and records were originally part of the file that is now being propagated as individual messages to disparate sources, there must be way to track them. Therefore you need to generate a unique identifier before propagating the message, which can be done by concatenating the file name and MQ message identifier. You can extract the file name from the file message and extract the MQ message identifier from the Sterling Control Center event, as shown below:
Figure 5. MQ message identifier
SET OutputRoot.XMLNSC.Customer.TransactionID = InputRoot.Collection.FileIn.XMLNSC. Customer.FileName || '_'|| CAST(InputRoot.Collection.MQIn.MQMD.MsgId AS CHARACTER);
You compute a routing endpoint and routing protocol based on the contents of the Sterling C: D file. Records from the file are routed to different destinations depending on their contents, as shown below:
Figure 6. Computing the routing endpoint
Money transfer orchestration using WebSphere Message Broker V7
WebSphere Message Broker V220.127.116.11 or later supports the processing of files transferred by Sterling C:D. This article shows you how to transform a file sent by a Sterling C:D server into records using the WebSphere Message Broker CDInput node and then routing the record to the appropriate destination based on the record contents using other nodes of WebSphere Message Broker. You do not need to handle Sterling Control Center events in this case, because the CDInput node receives both the contents of the file and the metadata provided by Sterling C:D on the transfer. You use the WebSphere Message Broker CDInput node to receive messages that have been transferred to a given C:D server, and you use WebSphere Message Broker CDOutput node to serialize the message tree to a file and then transfer it to a secondary C:D server using a primary C:D server.
The following example shows you how to use the C:D nodes when working with Sterling C:D in conjunction with WebSphere Message Broker.
Figure 7. Message flow to correlate file arrival event
Correlating the file arrival event in the WebSphere Message Broker flow
A CDInput node is configured to receive the file from Sterling C:D server and to transform the file into records. The message structure and LocalEnvironment tree are populated automatically. Figure 8 below shows data representation of messages after CDInput transforms the file into records and messages:
Figure 8. Data representation after CDInput transforms the file
Generating a unique ID to track file transfer details using monitoring tools
Since messages and records were originally part of the file that is now being propagated as individual messages to disparate sources, there must be way to track them. Since C:D transfers do not provide a unique identifier, you need to generate one before propagating the message, which you can do by concatenating the filename with the timestamp of the file arrival. The CDInput node stores the filename and timestamp in the LocalEnvironment tree. The unique identifier is stored in the Environment structure, where it can be used by Tivoli Composite Application Manager to track the record. Figure 9 below shows the message data representation along with the unique identifier:
Figure 9. Message data representation and unique identifier
Computing a routing endpoint and routing protocol based on contents of the Sterling C:D file
Records from the file are routed to different destinations depending record contents, as shown below:
Figure 10. Computing routing endpoint
Part 2 of this article series has described the architectural elements involved in providing an integrated enterprise solution involving files, data, messages, events, and applications. The example has shown how to add the capabilities of reading a file and converting a message flow to different protocols supported by different networks and applications, giving JK Financials end-to-end visibility of customer records that arrive in a batch file and are processed as individual records for consumption by various back-end applications. This visibility provides transaction traceability and ways to address IT infrastructure problems that arise during operations. The example uses Sterling C:D as the managed file transfer product, but it would work equally well with any managed file transfer product supported by WebSphere Message Broker, such as WebSphere MQ File Transfer Edition (FTE).
Part 3 will bring together all pieces of the solution by introducing IBM Tivoli Composite Application Manager for Transactions to provide end-to-end transaction visibility for files and data transferred via the techniques described in Parts 1 and 2.
- Sterling, Tivoli, and WebSphere
Message Broker resources
- Documentation for IBM Sterling Connect:Direct
- Installing ITCAM for Transactions using the ITCAM for Transactions installer
- Youtube tutorial: Integrating Microsoft .NET code in a
WebSphere Message Broker V8 message flow
This five-minute youtube tutorial shows you how simple it is to use WebSphere Message Broker V8 to build a message flow that includes Microsoft .NET code. Microsoft Visual Studio is used to build .NET code in C#, which is then integrated into a message flow using Message Broker and an HTTP RESTful interface.
- WebSphere Message Broker V8 Development Training
In this IBM Training course, you will learn about the components of the WebSphere Message Broker development and runtime environments. The course also explores message flow problem determination, and shows you how to construct message flows that use ESQL, Java, and PHP to transform messages.
- WebSphere Message Broker V8 information center
A single Web portal to all WebSphere Message Broker V8 documentation, with conceptual, task, and reference information on installing, configuring, and using your WebSphere Message Broker environment.
- WebSphere Message Broker developer resources
Technical resources to help you use WebSphere Message Broker for connectivity, universal data transformation, and enterprise-level integration of disparate services, applications, and platforms to power your SOA.
- WebSphere Message Broker product page
Product descriptions, product news, training information, support information, and more.
- Download free trial version of WebSphere Message
WebSphere Message Broker is an ESB built for universal connectivity and transformation in heterogeneous IT environments. It distributes information and data generated by business events in real time to people, applications, and devices throughout your extended enterprise and beyond.
- WebSphere Message Broker documentation
WebSphere Message Broker specifications and manuals.
- WebSphere Message Broker forum
Get answers to your technical questions and share your expertise with other WebSphere Message Broker users.
- WebSphere Message Broker support page
A searchable database of support problems and their solutions, plus downloads, fixes, and problem tracking.
- WebSphere resources
- developerWorks WebSphere developer
Technical information and resources for developers who use WebSphere products. developerWorks WebSphere provides product downloads, how-to information, support resources, and a free technical library of more than 2000 technical articles, tutorials, best practices, IBM Redbooks, and online product manuals.
- developerWorks WebSphere application integration developer
How-to articles, downloads, tutorials, education, product info, and other resources to help you build WebSphere application integration and business integration solutions.
- developerWorks WebSphere business process management
WebSphere BPM how-to articles, downloads, tutorials, education, product info, and other resources to help you model, assemble, deploy, and manage business processes.
- Most popular WebSphere trial downloads
No-charge trial downloads for key WebSphere products.
- WebSphere forums
Product-specific forums where you can get answers to your technical questions and share your expertise with other WebSphere users.
- WebSphere on-demand demos
Download and watch these self-running demos, and learn how WebSphere products and technologies can help your company respond to the rapidly changing and increasingly complex business environment.
- developerWorks WebSphere weekly newsletter
The developerWorks newsletter gives you the latest articles and information only on those topics that interest you. In addition to WebSphere, you can select from Java, Linux, Open source, Rational, SOA, Web services, and other topics. Subscribe now and design your custom mailing.
- WebSphere-related books from IBM Press
Convenient online ordering through Barnes & Noble.
- WebSphere-related events
Conferences, trade shows, Webcasts, and other events around the world of interest to WebSphere developers.
- developerWorks WebSphere developer resources
- developerWorks resources
downloads for IBM software products
No-charge trial downloads for selected IBM® DB2®, Lotus®, Rational®, Tivoli®, and WebSphere® products.
Join a conversation with developerWorks users and authors, and IBM editors and developers.
cloud computing resources
Access the IBM or Amazon EC2 cloud, test an IBM cloud computing product in a sandbox, see demos of cloud computing products and services, read cloud articles, and access other cloud resources.
- developerWorks tech briefings
Free technical sessions by IBM experts to accelerate your learning curve and help you succeed in your most challenging software projects. Sessions range from one-hour virtual briefings to half-day and full-day live sessions in cities worldwide.
- developerWorks podcasts
Listen to interesting and offbeat interviews and discussions with software innovators.
- developerWorks on
Check out recent Twitter messages and URLs.
- IBM Education Assistant
A collection of multimedia educational modules that will help you better understand IBM software products and use them more effectively to meet your business requirements.
- Trial downloads for IBM software products