Enterprises both large and small need to process various types of files and connect them to disparate and distributed applications spread across enterprise data centers. As the underlying infrastructure gets more dispersed and complex, it becomes a challenge to gain end-to-end visibility of transactions that involve files, data transformations, connectivity logic, protocol conversions, and final data posted to different applications.
Scenario and prerequisites
This article series describes a scenario involving JK Financials, a fictitious bank that needs to gain end-to-end visibility over its integrated enterprise IT infrastructure. This visibility will help provide transaction traceability and ways to address the IT infrastructure problems that arise during regular operations.
Part 1 will describe the high-level architecture of how JK Financials processes instruction subsets received from its bank branches as files. It will show how these files are processed using the managed file transfer capabilities of IBM® Sterling Connect:Direct (hereafter called Sterling C:D). Reliably received files are processed by a WebSphere® Message Broker orchestration to correlate the arrival of a file and make routing decisions and protocol conversions for multiple bank applications. In order to effectively trace all the paths that an instruction in the form of a file takes, various components of the architecture are integrated with IBM Tivoli® Composite Application Manager for Transactions (hereafter called ITCAM), which correlates events using the transaction ID to give different views of transactions that are completed, aborted, in error, or suspended.
This scenario assumes that the following IBM products are installed and configured to work individually in the JK Financials network.
- Sterling C:D Nodes V4.5.01
- Sterling Control Centre V5.3
- WebSphere MQ V7
- WebSphere Message Broker V6.1 and V7
- ITCAM V7.3
JK Financials uses Sterling C:D to provide managed file transfer (MFT) capabilities within its enterprise network. A branch bank sends money transfer instructions in a file using a secure transfer protocol, as shown below:
Figure 1. Scenario overview and architecture
The files that arrive in the enterprise data centre are saved in a shared storage repository in the file system. The arrival of a file at Sterling C:D Node A1 generates an MQ event to trigger a WebSphere Message Broker orchestration to further process the file. This trigger involves integrating the Sterling Control Centre with WebSphere MQ as explained in the following section. The WebSphere Message Broker orchestration acts like an enterprise service bus and is responsible for the following actions.
- Correlate events from Sterling Control Centre to initiate the orchestration.
- Make sure that the file has arrived successfully in the shared storage.
- Examine the incoming file contents and create a unique ID based on the file name and timestamp of the file arrival.
- Compute the routing path based on the file contents.
- Based on the endpoints selected for routing, do protocol conversion.
- Create new messages or files and post them to applications.
This article describes both WebSphere Message Broker V6.1 and V7:
- For V6.1, use events from IBM Sterling Control Center (hereafter called SCC) to correlate the file arrival and initiate the orchestration.
- For V7, use out-of-the box functionality to listen for the C:D file transfer and arrival, and then automatically initiating file processing.
The applications that consume the data in this scenario are listed below:
- A money transfer SAP workflow that is triggered by an MQ message arriving at the application endpoint.
- A legacy core banking system that takes raw bytes over TCP/IP to process money transfer instructions.
- A logging and compliance system that uses a file for logging compliance for all financial transactions in the organization.
The scenario combines a robust file processing and batch-oriented managed file transfer system like Sterling C:D with a WebSphere Message Broker connectivity system that can process extreme transaction volumes and handle multiple types of networks and protocols.
File structure used in this scenario
A data file can contain one or more transaction requests (messages) from various JK Financial bank branches. The request is sent to the main branch data center. Each transaction message is in XML format and a file can contain multiple XML messages delimited by a new line character. The MsgFile.txt sample file below contains four transactions, each one with a unique TransactionID to identify and track the transaction. It also contains a RequestType element to provide the request type, which can be a DD request or an NEFT money transfer request. Based on the request type, these messages are routed to different business flows:
<Customer><TransactionID>2342194</TransactionID><AccountNo>234234</AccountNo> <Name>Abhi</Name><RequestType>DD</RequestType><BeneficiaryName>John</BeneficiaryName> <Amount>10000</Amount><Branch>payable at</Branch><Date>31/10/2011</Date></Customer> <Customer><TransactionID>2342195</TransactionID><AccountNo>121332</AccountNo> <Name>Vijay</Name><RequestType>NEFT</RequestType><ToAccountNo>122334</ToAccountNo> <Name>James</Name><IFSCCode>1234</IFSCCode><Branch>Bangalore</Branch> <Amount>20000</Amount><Date>31/10/2011</Date></Customer> <Customer><TransactionID>2342196</TransactionID><AccountNo>234234</AccountNo> <Name>David</Name><RequestType>DD</RequestType><BeneficiaryName>Martin</BeneficiaryName> <Amount>10000</Amount><Branch>payable at</Branch><Date>31/10/2011</Date></Customer> <Customer><TransactionID>2342197</TransactionID><AccountNo>121332</AccountNo> <Name>Mike</Name><RequestType>NEFT</RequestType><ToAccountNo>122334</ToAccountNo> <Name>James</Name><IFSCCode>1234</IFSCCode><Branch>Bangalore</Branch> <Amount>20000</Amount><Date>31/10/2011</Date></Customer>
This scenario uses XML as a the data format, but any structured format of text or binary data can be used.
Creating a unique TransactionID for end-to-end traceability
The TransactionID is a concatenation of the transaction data file name and timestamp. For example, if the file name is Msgfile.txt and the time stamp when the file arrived at the Message Broker flow is 20120205_133117_820340, then the ID for the transactions would be Msgfile.txt_20120205_133117_820340. ITCAM for Transactions will show four transactions and the destination of each, enabling you to trace details for each one.
Configuring Sterling C:D nodes and testing for file arrival
This section shows you how to configure Sterling Control Center (SCC) to monitor a Sterling C:D node, and how to publish events from SCC to WebSphere MQ. Figure 2 shows the topology:
Figure 2. Topology for Sterling C:D nodes, SCC, and WebSphere MQ
Configuring System A
This article uses Microsoft® Windows® on both System A and System B.
- Install Sterling C:D Windows on System A as the primary node.
- Install Sterling C:D Windows on System B as the secondary node.
- C:D has two functions -- C:D Server (the one that can accept inbound
files or requests) and C:D Requester (the one that can initiate a
request for transfer of outbound files) During installation, ensure
that no applications are using Ports 1363 and 1364, as these are the
default ports used by C:D Windows. Using the C:D Requester tool from
the Program Files menu, insert a local node using the navigation as
Figure 3. Adding the C:D node
Configuring the Functional Authorities
- On the C:D node, double-click on Functional
Authorities. A list of local domain users will be
displayed. To create a C:D user, click New User. For
a new or existing user, click on Properties to
configure it using the dialog shown below:
Figure 4. Configuring the Functional Authorities
- Retain the defaults on the Main tab. On the Directories tab, configure the file upload and download directories.
Netmap configuration is an important step in C:D configuration. It is where you define the secondary C:D nodes in the case of C:D to C:D transfer, or configure the C:D server adapter in the case of C:D to Sterling File Gateway transfer. This scenario involves C:D to C:D transfer, so you need to configure Netmap for the secondary C:D node:
- Double click on Netmap from the C:D node to open a dialog as shown
Figure 5. Configuring Netmap
- Enter the name of the secondary C:D node.
- Go to the TCP/IP tab and provide the IP/Hostname and Port details for the secondary C:D node.
- On the Communication Paths tab, select TCPCommPath into the list of selected paths. For the other configuration settings, accept the defaults and click OK.
- Do similar configuration steps for the C:D secondary node. The last step of configuring Netmap is optional on the secondary node, and needs to be done only if you are going to use the secondary node to initiate transfers to the primary node.
You have completed the basic C:D configuration and are now ready to initiate a file transfer from the primary node (System A) to the secondary node (System B).
Sending and receive files using C:D
- On the local node configured on the primary C:D node, double click on Send/Receive Files. In the dialog that opens as shown below, select the secondary Node (SNode) as configured in Netmap.
- Specify the path to the file that needs to be sent to SNode and the
destination file name. You do not need to specify the destination path
because the file will be delivered to the download directory
configured for the SNode user.
Figure 6. Sending and receiving of files using C:D Requester
- After the transfer job is completed, you can see the results in the
Exec Status window, as shown below. Right-click on the relevant record
and select View Details to see the Process Execution
Figure 7. Process statistics in C:D requester window
Using IBM Sterling Control Center to monitor Sterling C:D nodes
- Install SCC on System B and configure it to monitor the secondary C:D node.
- Get the installable for the SCC for Windows, unzip it, and double click CCInstall.exe to start the installation. SCC installation requires a database -- this scenario uses DB2 for Windows V9.7. SCC creates two database instances -- a staging database and the actual database instance.
- The SCC installation wizard prompts you to enter the information to install SCC. Specify db2jcc.jar.
- If the installation fails during any of the configuration steps, you can retry it by running the configCC.bat file from the ControlCenter/bin directory on the installed path. The Installation steps described here are not a substitution for the Control Center Installation Guide -- they are just the minimum information to get a setup working to demonstrate its abilities.
- Start the SCC Console from the Desktop icon or from the Program Files menu. It prompts you for the information below. HostName indicates that the SCC Console can be installed and launched from a machine that need not be the machine where the SCC engine is installed. In this scenario, the SCC Engine and the Console are installed on the same machine.
- The default HTTP port number is 58080, which you can change depending on the port availability on your machine.
- When you start the SCC Console for the first time, the default
userid/password is admin/admin. Change the password after you log in:
click Change Password in the SCC Console Menu.
Figure 8. Logging in to the SCC console
- After logging in, the next step is to add the server that you want SCC
to monitor. SCC monitors servers that are either Connect:Direct,
Connect:Enterprise, Sterling B2B Integrator, or FTP. On the Servers
panel, right click on Control Center and click on
Figure 9. Configuring the C:D server in SCC console 1
- A dialog opens as shown below:
Figure 10. Configuring the C:D server in SCC console 2
- In the Connection tab, enter the IP and port number of the C:D node that you intend to add to SCC for monitoring, and specify the userid/password of the C:D user.
- To verify that the information you have provided is correct, ensure
that the C:D server is running and click Test
Figure 11. Configuring the C:D server in SCC console 3
- This article does not cover the other tabs -- just accept the defaults and finish adding the server to SCC.
- Since you have added a C:D server, it will be listed in the SCC
console under Connect:Direct, as shown below. The icon next to the
server node indicates that the server is available for monitoring:
Figure 12. SCC console showing added C:D server
- If the server is unavailable for SCC to monitor, you will see a red icon next to the icon to indicate a connection failure, which needs to be corrected and the server to be monitored made available to SCC.
- To view the statistics or the processes that have been carried out by
the C:D node, use the SCC console. Right-click on the C:D server node
and click Process History:
Figure 13. Viewing process history from SCC console
- A report is displayed showing a list of processes that the secondary
C:D node carried out:.
Figure 14. Process History view of C:D node as seen in SCC console
- To display the details on each of step carried out during the file
transfer, click Select Statistics:
Figure 15. SCC Select Statistics displaying all statistics
Integrating IBM Sterling Control Centre with WebSphere MQ
In order to have a unified view of C:D file transfers coupled with other WebSphere applications, you need a way to publish events so that they can be consumed by other WebSphere products. You can meet this goal by configuring SCC V5.3 to publish events to a topic in WebSphere MQ, and configuring WebSphere MQ as shown below:
- Install WebSphere MQ V7.
- Create a queue manager.
- Create a topic with a topic string such as scc.
- Create a server connection channel
- Configure the Listener to listen on a given port.
- Create a local queue called something like SCCQueue.
- Create a subscription queue for the Topic . The events published to this Topic will be put to this queue.
Any application that needs to consume these SCC events can be configured to read from the SCCQueue.
- In order to enable SCC to emit events, configure SCC by running the configCC.bat file from the installed location.
- During the execution of this batch file, you will be prompted to
change various installation parameters of SCC, but you only need to
change the JMS configuration for SCC. Figure 16 shows the parameters
you will be prompted for and the values that you need to specify:
Figure 16. Enabling SCC to emit JMS events
- After the configuration changes are completed, restart SCC.
- If your JMS configuration changes to SCC are done correctly, and if
WebSphere MQ is configured as explained above, you will see that when
a file is received by the C:D secondary node monitored by SCC, then
related events are published in the configured topic in WebSphere MQ,
as shown below in Figure 17:
Figure 17. List of SCC events in SCCQueue
Part 1 described the overall integration solution with managed file transfer, brokering, and end-to-end monitoring for the JK Financial bank scenario. The article showed you how integrating Sterling C:D with WebSphere Message Broker provides cross-platform and cross-protocol connectivity in the enterprise, giving you end-to-end visibility and transaction traceability. It also showed you how to set up the managed file transfer solution with Sterling C:D. Part 2 will show you how to use WebSphere Message Broker to develop, orchestrate, and execute a message flow with the files received through the C:D integration with Message Broker.
- Sterling, Tivoli, and WebSphere
Message Broker resources
- Documentation for IBM Sterling Connect:Direct
- Installing ITCAM for Transactions using the ITCAM for Transactions installer
- Youtube tutorial: Integrating Microsoft .NET code in a
WebSphere Message Broker V8 message flow
This five-minute youtube tutorial shows you how simple it is to use WebSphere Message Broker V8 to build a message flow that includes Microsoft .NET code. Microsoft Visual Studio is used to build .NET code in C#, which is then integrated into a message flow using Message Broker and an HTTP RESTful interface.
- WebSphere Message Broker V8 Development Training
In this IBM Training course, you will learn about the components of the WebSphere Message Broker development and runtime environments. The course also explores message flow problem determination, and shows you how to construct message flows that use ESQL, Java, and PHP to transform messages.
- WebSphere Message Broker V8 information center
A single Web portal to all WebSphere Message Broker V8 documentation, with conceptual, task, and reference information on installing, configuring, and using your WebSphere Message Broker environment.
- WebSphere Message Broker developer resources
Technical resources to help you use WebSphere Message Broker for connectivity, universal data transformation, and enterprise-level integration of disparate services, applications, and platforms to power your SOA.
- WebSphere Message Broker product page
Product descriptions, product news, training information, support information, and more.
- Download free trial version of WebSphere Message
WebSphere Message Broker is an ESB built for universal connectivity and transformation in heterogeneous IT environments. It distributes information and data generated by business events in real time to people, applications, and devices throughout your extended enterprise and beyond.
- WebSphere Message Broker documentation
WebSphere Message Broker specifications and manuals.
- WebSphere Message Broker forum
Get answers to your technical questions and share your expertise with other WebSphere Message Broker users.
- WebSphere Message Broker support page
A searchable database of support problems and their solutions, plus downloads, fixes, and problem tracking.
- WebSphere resources
- developerWorks WebSphere developer
Technical information and resources for developers who use WebSphere products. developerWorks WebSphere provides product downloads, how-to information, support resources, and a free technical library of more than 2000 technical articles, tutorials, best practices, IBM Redbooks, and online product manuals.
- developerWorks WebSphere application integration developer
How-to articles, downloads, tutorials, education, product info, and other resources to help you build WebSphere application integration and business integration solutions.
- developerWorks WebSphere business process management
WebSphere BPM how-to articles, downloads, tutorials, education, product info, and other resources to help you model, assemble, deploy, and manage business processes.
- Most popular WebSphere trial downloads
No-charge trial downloads for key WebSphere products.
- WebSphere forums
Product-specific forums where you can get answers to your technical questions and share your expertise with other WebSphere users.
- WebSphere on-demand demos
Download and watch these self-running demos, and learn how WebSphere products and technologies can help your company respond to the rapidly changing and increasingly complex business environment.
- developerWorks WebSphere weekly newsletter
The developerWorks newsletter gives you the latest articles and information only on those topics that interest you. In addition to WebSphere, you can select from Java, Linux, Open source, Rational, SOA, Web services, and other topics. Subscribe now and design your custom mailing.
- WebSphere-related books from IBM Press
Convenient online ordering through Barnes & Noble.
- WebSphere-related events
Conferences, trade shows, Webcasts, and other events around the world of interest to WebSphere developers.
- developerWorks WebSphere developer resources
- developerWorks resources
downloads for IBM software products
No-charge trial downloads for selected IBM® DB2®, Lotus®, Rational®, Tivoli®, and WebSphere® products.
Join a conversation with developerWorks users and authors, and IBM editors and developers.
cloud computing resources
Access the IBM or Amazon EC2 cloud, test an IBM cloud computing product in a sandbox, see demos of cloud computing products and services, read cloud articles, and access other cloud resources.
- developerWorks tech briefings
Free technical sessions by IBM experts to accelerate your learning curve and help you succeed in your most challenging software projects. Sessions range from one-hour virtual briefings to half-day and full-day live sessions in cities worldwide.
- developerWorks podcasts
Listen to interesting and offbeat interviews and discussions with software innovators.
- developerWorks on
Check out recent Twitter messages and URLs.
- IBM Education Assistant
A collection of multimedia educational modules that will help you better understand IBM software products and use them more effectively to meet your business requirements.
- Trial downloads for IBM software products