Transaction traceability of enterprise files, data, and applications: Part 1: Scenario overview: Integrating IBM Sterling Connect:Direct with WebSphere MQ

This article series describes a scenario involving JK Financials, a fictitious bank that needs to gain end-to-end visibility over its integrated enterprise IT infrastructure. Part 1 shows you how to use the managed file transfer capabilities of IBM Sterling Connect:Direct with a WebSphere Message Broker orchestration to make routing decisions and protocol conversions for multiple bank applications.

Devaprasad Nadgir (ndevapra@in.ibm.com), Senior Software Engineer, Emerging Technologies team, IBM

Photo of Devaprasad NadgirDevaprasad (Dev) Nadgir is a Senior Software Engineer on the Emerging Technologies team at the IBM India Software Lab in Bangalore. He is an IBM Certified and The Open Group Architecture Framework (TOGAF) Master Certified Architect with experience in development, architecture, and consulting. You can contact Dev at ndevapra@in.ibm.com.



Abhinav Priyadarshi (pabhinav@in.ibm.com), Technical Project Lead, WebSphere Message Broker Development team, IBM

Photo of Abhinav PriyadarshiAbhinav Priyadarshi is an Advisory IT Architect and Technical Lead for the IBM Integration Bus Development Team at the IBM India Software Lab in Bangalore. He has 12 years of experience with IBM. He develops new features and nodes for IBM Integration Bus, and provides consultancy on IBM Integration Bus Proofs of Concept, EAI design, and implementation for customers in India and the Asia-Pacific region. You can contact Abhinav at pabhinav@in.ibm.com.



Ms. Sreelatha Sankaranarayanan (sreelathas@in.ibm.com), Software Engineer, WebSphere Adapters Development team, IBM

Photo of Sreelatha SankaranarayananSreelatha Sankaranarayana is a Software Engineer on the WebSphere Adapters Development team at the IBM India Software Lab in Bangalore, and is currently Technical Lead for WebSphere Adapter for Oracle e-Business Suite. Previously, she was part of the WebSphere Partner Gateway and WebSphere Data Interchange Support and Development teams. You can contact Sreelatha at <a href="mailto:sreelathas@in.ibm.com">sreelathas@in.ibm.com</a>.



27 June 2012

Also available in Chinese Russian

Introduction

Enterprises both large and small need to process various types of files and connect them to disparate and distributed applications spread across enterprise data centers. As the underlying infrastructure gets more dispersed and complex, it becomes a challenge to gain end-to-end visibility of transactions that involve files, data transformations, connectivity logic, protocol conversions, and final data posted to different applications.

Scenario and prerequisites

This article series describes a scenario involving JK Financials, a fictitious bank that needs to gain end-to-end visibility over its integrated enterprise IT infrastructure. This visibility will help provide transaction traceability and ways to address the IT infrastructure problems that arise during regular operations.

Part 1 will describe the high-level architecture of how JK Financials processes instruction subsets received from its bank branches as files. It will show how these files are processed using the managed file transfer capabilities of IBM® Sterling Connect:Direct (hereafter called Sterling C:D). Reliably received files are processed by a WebSphere® Message Broker orchestration to correlate the arrival of a file and make routing decisions and protocol conversions for multiple bank applications. In order to effectively trace all the paths that an instruction in the form of a file takes, various components of the architecture are integrated with IBM Tivoli® Composite Application Manager for Transactions (hereafter called ITCAM), which correlates events using the transaction ID to give different views of transactions that are completed, aborted, in error, or suspended.

This scenario assumes that the following IBM products are installed and configured to work individually in the JK Financials network.

  • Sterling C:D Nodes V4.5.01
  • Sterling Control Centre V5.3
  • WebSphere MQ V7
  • WebSphere Message Broker V6.1 and V7
  • ITCAM V7.3

JK Financials uses Sterling C:D to provide managed file transfer (MFT) capabilities within its enterprise network. A branch bank sends money transfer instructions in a file using a secure transfer protocol, as shown below:

Figure 1. Scenario overview and architecture
Scenario overview and architecture

The files that arrive in the enterprise data centre are saved in a shared storage repository in the file system. The arrival of a file at Sterling C:D Node A1 generates an MQ event to trigger a WebSphere Message Broker orchestration to further process the file. This trigger involves integrating the Sterling Control Centre with WebSphere MQ as explained in the following section. The WebSphere Message Broker orchestration acts like an enterprise service bus and is responsible for the following actions.

  1. Correlate events from Sterling Control Centre to initiate the orchestration.
  2. Make sure that the file has arrived successfully in the shared storage.
  3. Examine the incoming file contents and create a unique ID based on the file name and timestamp of the file arrival.
  4. Compute the routing path based on the file contents.
  5. Based on the endpoints selected for routing, do protocol conversion.
  6. Create new messages or files and post them to applications.

This article describes both WebSphere Message Broker V6.1 and V7:

  • For V6.1, use events from IBM Sterling Control Center (hereafter called SCC) to correlate the file arrival and initiate the orchestration.
  • For V7, use out-of-the box functionality to listen for the C:D file transfer and arrival, and then automatically initiating file processing.

The applications that consume the data in this scenario are listed below:

  • A money transfer SAP workflow that is triggered by an MQ message arriving at the application endpoint.
  • A legacy core banking system that takes raw bytes over TCP/IP to process money transfer instructions.
  • A logging and compliance system that uses a file for logging compliance for all financial transactions in the organization.

The scenario combines a robust file processing and batch-oriented managed file transfer system like Sterling C:D with a WebSphere Message Broker connectivity system that can process extreme transaction volumes and handle multiple types of networks and protocols.

File structure used in this scenario

A data file can contain one or more transaction requests (messages) from various JK Financial bank branches. The request is sent to the main branch data center. Each transaction message is in XML format and a file can contain multiple XML messages delimited by a new line character. The MsgFile.txt sample file below contains four transactions, each one with a unique TransactionID to identify and track the transaction. It also contains a RequestType element to provide the request type, which can be a DD request or an NEFT money transfer request. Based on the request type, these messages are routed to different business flows:

<Customer><TransactionID>2342194</TransactionID><AccountNo>234234</AccountNo>
<Name>Abhi</Name><RequestType>DD</RequestType><BeneficiaryName>John</BeneficiaryName>
<Amount>10000</Amount><Branch>payable at</Branch><Date>31/10/2011</Date></Customer>

<Customer><TransactionID>2342195</TransactionID><AccountNo>121332</AccountNo>
<Name>Vijay</Name><RequestType>NEFT</RequestType><ToAccountNo>122334</ToAccountNo>
<Name>James</Name><IFSCCode>1234</IFSCCode><Branch>Bangalore</Branch>
<Amount>20000</Amount><Date>31/10/2011</Date></Customer>

<Customer><TransactionID>2342196</TransactionID><AccountNo>234234</AccountNo>
<Name>David</Name><RequestType>DD</RequestType><BeneficiaryName>Martin</BeneficiaryName>
<Amount>10000</Amount><Branch>payable at</Branch><Date>31/10/2011</Date></Customer>

<Customer><TransactionID>2342197</TransactionID><AccountNo>121332</AccountNo>
<Name>Mike</Name><RequestType>NEFT</RequestType><ToAccountNo>122334</ToAccountNo>
<Name>James</Name><IFSCCode>1234</IFSCCode><Branch>Bangalore</Branch>
<Amount>20000</Amount><Date>31/10/2011</Date></Customer>

This scenario uses XML as a the data format, but any structured format of text or binary data can be used.

Creating a unique TransactionID for end-to-end traceability

The TransactionID is a concatenation of the transaction data file name and timestamp. For example, if the file name is Msgfile.txt and the time stamp when the file arrived at the Message Broker flow is 20120205_133117_820340, then the ID for the transactions would be Msgfile.txt_20120205_133117_820340. ITCAM for Transactions will show four transactions and the destination of each, enabling you to trace details for each one.

Configuring Sterling C:D nodes and testing for file arrival

This section shows you how to configure Sterling Control Center (SCC) to monitor a Sterling C:D node, and how to publish events from SCC to WebSphere MQ. Figure 2 shows the topology:

Figure 2. Topology for Sterling C:D nodes, SCC, and WebSphere MQ
Topology for Sterling C:D nodes, SCC, and WebSphere MQ

Configuring System A

This article uses Microsoft® Windows® on both System A and System B.

  1. Install Sterling C:D Windows on System A as the primary node.
  2. Install Sterling C:D Windows on System B as the secondary node.
  3. C:D has two functions -- C:D Server (the one that can accept inbound files or requests) and C:D Requester (the one that can initiate a request for transfer of outbound files) During installation, ensure that no applications are using Ports 1363 and 1364, as these are the default ports used by C:D Windows. Using the C:D Requester tool from the Program Files menu, insert a local node using the navigation as shown below:.
    Figure 3. Adding the C:D node
    Adding the C:D node

Configuring the Functional Authorities

  1. On the C:D node, double-click on Functional Authorities. A list of local domain users will be displayed. To create a C:D user, click New User. For a new or existing user, click on Properties to configure it using the dialog shown below:
    Figure 4. Configuring the Functional Authorities
    Configuring the Functional Authorities
  2. Retain the defaults on the Main tab. On the Directories tab, configure the file upload and download directories.

Configuring Netmap

Netmap configuration is an important step in C:D configuration. It is where you define the secondary C:D nodes in the case of C:D to C:D transfer, or configure the C:D server adapter in the case of C:D to Sterling File Gateway transfer. This scenario involves C:D to C:D transfer, so you need to configure Netmap for the secondary C:D node:

  1. Double click on Netmap from the C:D node to open a dialog as shown below:
    Figure 5. Configuring Netmap
    Configuring Netmap
  2. Enter the name of the secondary C:D node.
  3. Go to the TCP/IP tab and provide the IP/Hostname and Port details for the secondary C:D node.
  4. On the Communication Paths tab, select TCPCommPath into the list of selected paths. For the other configuration settings, accept the defaults and click OK.
  5. Do similar configuration steps for the C:D secondary node. The last step of configuring Netmap is optional on the secondary node, and needs to be done only if you are going to use the secondary node to initiate transfers to the primary node.

You have completed the basic C:D configuration and are now ready to initiate a file transfer from the primary node (System A) to the secondary node (System B).

Sending and receive files using C:D

  1. On the local node configured on the primary C:D node, double click on Send/Receive Files. In the dialog that opens as shown below, select the secondary Node (SNode) as configured in Netmap.
  2. Specify the path to the file that needs to be sent to SNode and the destination file name. You do not need to specify the destination path because the file will be delivered to the download directory configured for the SNode user.
    Figure 6. Sending and receiving of files using C:D Requester
    Sending and receiving of files using C:D Requester
  3. After the transfer job is completed, you can see the results in the Exec Status window, as shown below. Right-click on the relevant record and select View Details to see the Process Execution Status:
    Figure 7. Process statistics in C:D requester window
    Process statistics in C:D requester window

Using IBM Sterling Control Center to monitor Sterling C:D nodes

  1. Install SCC on System B and configure it to monitor the secondary C:D node.
  2. Get the installable for the SCC for Windows, unzip it, and double click CCInstall.exe to start the installation. SCC installation requires a database -- this scenario uses DB2 for Windows V9.7. SCC creates two database instances -- a staging database and the actual database instance.
  3. The SCC installation wizard prompts you to enter the information to install SCC. Specify db2jcc.jar.
  4. If the installation fails during any of the configuration steps, you can retry it by running the configCC.bat file from the ControlCenter/bin directory on the installed path. The Installation steps described here are not a substitution for the Control Center Installation Guide -- they are just the minimum information to get a setup working to demonstrate its abilities.
  5. Start the SCC Console from the Desktop icon or from the Program Files menu. It prompts you for the information below. HostName indicates that the SCC Console can be installed and launched from a machine that need not be the machine where the SCC engine is installed. In this scenario, the SCC Engine and the Console are installed on the same machine.
  6. The default HTTP port number is 58080, which you can change depending on the port availability on your machine.
  7. When you start the SCC Console for the first time, the default userid/password is admin/admin. Change the password after you log in: click Change Password in the SCC Console Menu.
    Figure 8. Logging in to the SCC console
    Logging in to the SCC console
  8. After logging in, the next step is to add the server that you want SCC to monitor. SCC monitors servers that are either Connect:Direct, Connect:Enterprise, Sterling B2B Integrator, or FTP. On the Servers panel, right click on Control Center and click on Add Server:
    Figure 9. Configuring the C:D server in SCC console 1
    Configuring the C:D server in SCC console 1
  9. A dialog opens as shown below:
    Figure 10. Configuring the C:D server in SCC console 2
    Configuring the C:D server in SCC console 2
  10. In the Connection tab, enter the IP and port number of the C:D node that you intend to add to SCC for monitoring, and specify the userid/password of the C:D user.
  11. To verify that the information you have provided is correct, ensure that the C:D server is running and click Test Connection:
    Figure 11. Configuring the C:D server in SCC console 3
    Configuring the C:D server in SCC console 3
  12. This article does not cover the other tabs -- just accept the defaults and finish adding the server to SCC.
  13. Since you have added a C:D server, it will be listed in the SCC console under Connect:Direct, as shown below. The icon next to the server node indicates that the server is available for monitoring:
    Figure 12. SCC console showing added C:D server
    SCC console showing added C:D server
  14. If the server is unavailable for SCC to monitor, you will see a red icon next to the icon to indicate a connection failure, which needs to be corrected and the server to be monitored made available to SCC.
  15. To view the statistics or the processes that have been carried out by the C:D node, use the SCC console. Right-click on the C:D server node and click Process History:
    Figure 13. Viewing process history from SCC console
    Viewing process history from SCC console
  16. A report is displayed showing a list of processes that the secondary C:D node carried out:.
    Figure 14. Process History view of C:D node as seen in SCC console
    Process History view of C:D node as seen in SCC console
  17. To display the details on each of step carried out during the file transfer, click Select Statistics:
    Figure 15. SCC Select Statistics displaying all statistics
    SCC Select Statistics displaying all statistics

Integrating IBM Sterling Control Centre with WebSphere MQ

In order to have a unified view of C:D file transfers coupled with other WebSphere applications, you need a way to publish events so that they can be consumed by other WebSphere products. You can meet this goal by configuring SCC V5.3 to publish events to a topic in WebSphere MQ, and configuring WebSphere MQ as shown below:

  1. Install WebSphere MQ V7.
  2. Create a queue manager.
  3. Create a topic with a topic string such as scc.
  4. Create a server connection channel
  5. Configure the Listener to listen on a given port.
  6. Create a local queue called something like SCCQueue.
  7. Create a subscription queue for the Topic . The events published to this Topic will be put to this queue.

Any application that needs to consume these SCC events can be configured to read from the SCCQueue.

  1. In order to enable SCC to emit events, configure SCC by running the configCC.bat file from the installed location.
  2. During the execution of this batch file, you will be prompted to change various installation parameters of SCC, but you only need to change the JMS configuration for SCC. Figure 16 shows the parameters you will be prompted for and the values that you need to specify:
    Figure 16. Enabling SCC to emit JMS events
    Enabling SCC to emit JMS events
  3. After the configuration changes are completed, restart SCC.
  4. If your JMS configuration changes to SCC are done correctly, and if WebSphere MQ is configured as explained above, you will see that when a file is received by the C:D secondary node monitored by SCC, then related events are published in the configured topic in WebSphere MQ, as shown below in Figure 17:
    Figure 17. List of SCC events in SCCQueue
    List of SCC events in SCCQueue

Conclusion

Part 1 described the overall integration solution with managed file transfer, brokering, and end-to-end monitoring for the JK Financial bank scenario. The article showed you how integrating Sterling C:D with WebSphere Message Broker provides cross-platform and cross-protocol connectivity in the enterprise, giving you end-to-end visibility and transaction traceability. It also showed you how to set up the managed file transfer solution with Sterling C:D. Part 2 will show you how to use WebSphere Message Broker to develop, orchestrate, and execute a message flow with the files received through the C:D integration with Message Broker.

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=823141
ArticleTitle=Transaction traceability of enterprise files, data, and applications: Part 1: Scenario overview: Integrating IBM Sterling Connect:Direct with WebSphere MQ
publish-date=06272012