Before you can extend the Managed File Transfer Sample you must have completed one of the following tasks:
After you have imported or built the sample, you can extend it:
If problems arise when you run the sample, see Preparing the environment for WebSphere MQ File Transfer Edition nodes or, see Resolving problems when running samples in the IBM Integration Bus documentation.
The sample handles one branch. To extend the sample to work with multiple branches on the FTEInputBranch2HQMFlow message flow, change the FTEInput node filters. Filtering allows the FTEInput node to choose which messages to pick up and process. For example:
To filter by directory location, set the Directory filter field to point to the directory to which the branch outlets agents transfer the messages.
You must also simulate an agent for every additional branch. In the sample scenario, the .mbtest client is used to simulate sending a stock replenishment request message from the branch outlet. To indicate which branch the message was sent from, the application origin data header field is populated with the branch number. The header is then used to set the file name by setting the wildcard match in the local environment.
To enable the sample to operate with multiple branches on the FTEOutputHQ2BranchMFlow message flow, change the following FTEOutput node properties:
In the local environment set the Destination directory or Agent name, or both, to route the messages to the right branch outlet, see FTE metadata.
The sample can now process files from multiple branches with local and remote agents.
If the sample retail scenario is extended to operate with multiple branches, typically the flow instances are set to operate with additional instances for better throughput.
With additional instances, the stock application receives and processes messages from different branches, and mixes up the ordering of the received data. The message flow that uses the FTEOutput node might open and process files for many branches at any given time, and start to build up the files concurrently.
With multi-flow processing, the order of the data output by the stock application is not guaranteed to be in the order in which it was received. It is critical that the End of Data record is received last for a particular branch, so that the file is closed only when complete. The order of the product data is in XML format, and can be written in any order.
The following diagram shows two approaches to this problem. In the top flow, a Resequence node is placed before the FTEOutput node. The Resequence node controls the order in which incoming messages of a branch are propagated through the node, based on the sequence number that is inserted into each record. To maintain the integrity of the product data of a branch, a message group tag (such as the branch name) is inserted to ensure that the Resequence node assembles only the products for its destination branch. However, the Resequence node requires that messages belonging to the same branch (sequence group) are handled only by a single thread. This scenario is contrary to the original requirement for multi-instance operation. The correct sequencing is achieved at the expense of multi-instance flow operation.
If the message flow is redesigned as shown in the bottom flow, this conflict is overcome. Here, the Resequence node is placed after the FTEOutput node.
The flow can now operate in multi-threaded mode, writing the product data as it arrives at the input terminal. Output from the FTEOutput node is captured by a Resequence node, which always waits for the last record to be received before propagating records. The End of Data record is sent directly to the Resequence node, at which point the node propagates the record only if the node has received the full set of records for a branch. A Filter node discards the data records, and sends only the End of Data record to the FTEOutput node. The FTEOutput node can now close the file and call the agent to transfer the file.
This design enables the FTEOutput node to operate in flows with additional instances, and maintain the integrity of the data destined for a particular file.
The basic sample uses the temporary integration node FTE agent.
For details about how to use the WMQFTE remote agents with
the integration node agent and how to use the IBM Integration Explorer Tooling
to create a transfer and check the progress
of the transfer, see
Welcome to WebSphere MQ File Transfer
Edition.
The sample includes commented code snippets and descriptions in the Branch2HQDriver ESQL code. For example:
-- When you use the FTEInput node, it stores information that you can access in the LocalEnvironment.FTE and -- LocalEnvironment.FTE.Transfer message trees. The LocalEnvironment.FTE message tree stores information relating -- to the current record and is populated by the integration node. The LocalEnvironment.FTE.Transfer message tree contains -- information received from WebSphere® MQ File Transfer Edition regarding the file --LocalEnvironment.FTE.TimeStamp (CHARACTER) --LocalEnvironment.FTE.Offset (INTEGER) --LocalEnvironment.FTE.Record (INTEGER) --LocalEnvironment.FTE.Delimiter (CHARACTER) --LocalEnvironment.FTE.Transfer.Directory (CHARACTER)
For information about FTE metadata, see Managed file transfers using WebSphere MQ File Transfer Edition in the IBM Integration Bus documentation.
The basic sample demonstrates the request/reply model. The sample uses the branch number from the XML message and MQMD to ensure request/reply correlation. This section focuses on the use of the transfer identifier and job name for request/reply correlation.
When an FTE agent transfers a message, the agent populates the local environment with information about the transfer. This information can be overwritten, see FTE metadata, or the information can be used as it is to correlate the messages and ensure all the messages are sent back to the correct FTE agent. You can see this information in the properties of the transfer from the IBM Integration Explorer Tooling view, for example:
# STARTING
<?xml version="1.0" encoding="UTF-8"?>
<transaction xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
ID="414d51204d4237514d47522020202020bb73ea4b20046302"
agentRole="sourceAgent" version="3.00" xsi:noNamespaceSchemaLocation="TransferLog.xsd">
<action time="2010-05-12T11:34:12.421Z">started</action>
<sourceAgent QMgr="IB9QMGR" agent="IB9NODE.FTEBRANCH123">
<systemInfo architecture="x86" name="Windows XP" version="5.1 build 2600 Service Pack 3"/>
</sourceAgent>
<destinationAgent QMgr="IB9QMGR" agent="IB9NODE.FTEHQ"/>
<originator>
<hostName>IBM-27BDFF76CFB</hostName>
<userID>SYSTEM</userID>
<mqmdUserID>SYSTEM</mqmdUserID>
</originator>
<transferSet bytesSent="0" startTime="2010-05-12T11:34:12.421Z" total="1">
<metaDataSet>
<metaData key="com.ibm.wmqfte.TransferId">414d51204d4237514d47522020202020bb73ea4b20046302</metaData>
<metaData key="com.ibm.wmqfte.MqmdUser">SYSTEM</metaData>
<metaData key="com.ibm.wmqfte.DestinationAgent">IB9NODE.FTEHQ</metaData>
<metaData key="com.ibm.wmqfte.OriginatingHost">IBM-27BDFF76CFB</metaData>
<metaData key="com.ibm.wmqfte.OriginatingUser">SYSTEM</metaData>
<metaData key="com.ibm.wmqfte.JobName">Branch123</metaData>
<metaData key="com.ibm.wmqfte.SourceAgent">IB9NODE.FTEBRANCH123</metaData>
</metaDataSet>
</transferSet>
<job>
<name>Branch123</name>
</job>
</transaction>