Using WebSphere DataPower and WebSphere MQ File Transfer Edition to manage file transfers


This article describes how to integrate WebSphere DataPower and WebSphere MQ File Transfer Edition (FTE). At the time of writing, most of the available documentation is out-of-date and mainly focused on the use of managed file transfer in the context of B2B. DataPower firmware V4.x introduced a new WebSphere MQ FTE specific protocol handler that allows for a better integration between the two products. Firmware V5.x introduced extended memory support, thereby removing some of the limitations for large files.

WebSphere MQ FTE and DataPower are complementary products as the former is intended to centrally manage file transfers within the secure enterprise domain, while the latter acts as a secure gateway and extends connectivity to different security domains. Figure 1 shows the typical DataPower network deployment, where DataPower is located in a DMZ and acts as a gateway between the WebSphere MQ network and external partners.

Figure 1. DataPower Network Deployment
DataPower Network Deployment picture
DataPower Network Deployment picture

B2B Gateway Service vs. Multi-Protocol Gateway in the context of Managed File Transfer

Before firmware 4.0.0, the integration between DataPower and WebSphere MQ FTE was mainly based on shared file systems. That integration was sufficient in many contexts, but it did not take advantage of any WebSphere MQ premium features, such as transactionality and assured delivery. In addition, the B2B Transaction Viewer was not fully integrated with WebSphere MQ FTE.

Currently, FTP and SFTP are commonly used to transport B2B documents. Therefore, integrating WebSphere MQ FTE and DataPower through a B2B Gateway is the most common use case. For more information, see this document, WebSphere DataPower Release 4.0.1 B2B MQFTE.

The B2B Gateway Service (B2BGW) is available only on the “XB*” model and supports popular B2B messaging protocols, such as EDIINT AS1, AS2, and AS3 as well as ebMS v2.0. It provides the ability to properly manage file transfers as part of a B2B transaction, log the operation to persistent storage, and to integrate the FTE metadata with the B2B Transaction Viewer.

The B2B Gateway also provides the ability to organize file routing through the “Business Partner” objects. This construct is an abstraction that allows business partner information to be stored in a trading partner profile and is used to enforce trading partner agreements between two parties as well as providing a logical link between partner identity and one or more destination URLs.

The main points are that, in this configuration, streaming is not possible and files are always buffered in memory. Even when a large file is segmented in many WebSphere MQ messages, the B2BGW will rebuild the whole file in memory before processing it.

Given the characteristics of a B2BGW, there are a few critical aspects that need to be considered in the design phase:

  • Protocols
  • Integration with B2BViewer
  • File size
  • Auditing
  • Transactions rate
  • Transactionality and error handling
  • Capacity planning and B2B metadata and payload retention periods

Protocols are critical because if external partners use B2B specific protocols, such as AS/1/2/3 protocols, then there is no alternative to the B2B gateway service. This is also true if the integration with the B2BViewer is a key requirement for the solution, or if the customer wants to use the profile management or viewing capabilities unique to the B2BGW Service.

The next key element to be considered is the file size. For very large files, the Multi-Protocol Gateway (MPGW) is usually a better choice because it supports streaming and it does not save message payloads by default. However, MPGW has no ability to provide a view that shows the state of the transaction, nor does it have the ability to use profile management.

This last point leads to the next consideration: auditing. As already mentioned, B2BGWs are designed for critical B2B transactions. They provide, by default, a rich set of functionality, such as transaction auditing, and provide meta-information for the B2B Transaction Viewer.

MPGWs are simpler objects with no built-in auditing capability. Any processing, including logging and auditing, requires explicit configuration and eventually some (usually minimal) XSLT coding.

Besides the ability to stream, MPGWs simplicity and flexibility bring another benefit by supporting high transactions rate. MPGWs usually perform better than B2BGWs because they do not persist any metadata or payloads

The last important aspect to be considered is the need for transactionality and error handling, which can now rely on the traditional DataPower-MQ integration features, such as unit-of-work and back-out settings. The specific details around transactionality and error handling change depend on the direction of the transfer (if WebSphere MQ is the source or the destination of the file) and will be discussed in detail later in this article.

Table 1 summarizes the main elements to consider when choosing a B2B Gateway or a Multi-Protocol Gateway and integrating DataPower and WebSphere MQ FTE.

Table 1. Choosing B2BGW vs. MPGW
B2BGW (buffering mode only)MPGW (Buffering Mode)MPGW (Streaming Mode)
Support for B2B protocols, such as EDIINT AS1, AS2, and AS3 as well as ebMS v2.0.Platforms that support WebSphere MQ FTE protocol, but the B2BGW is not available (XI* and XG*).High network latency
Want to use the B2B Transaction ViewerHigh transaction rate“Extended memory support” is not available and the file size is greater than 100 MB.
Built in auditingOr “extended memory support” is available and the file size is greater than 3 GB.
BB2GW can be slower than MPGW for highly transactional workload (many small messages)No B2B Transaction ViewerNo B2B Transaction Viewer
Streaming is not supported.AS1/2/3 is not supported.AS1/2/3 and ebMS are not supported.
Available only on XB*Auditing needs to be implemented in the processing policyComplex processing policies can break streaming, limited auditing capability
HA and DR are more complex for a B2BGW than a MPGW

Transactionality and message persistence

WebSphere MQ FTE uses the WebSphere MQ infrastructure to move files across the enterprise. For performance reasons, it has been designed to reliably move files using non-persistent messages for the file transfer data. Persistent messaging is used for the initial WebSphere MQ FTE agent to agent handshake and also transfer restart messages between agents. When moving a file to a queue using the “fteCreateTransfer” command, by default, the messages are persistent. At that point, the transfer is completed from a WebSphere MQ FTE perspective. If the messages were not persistent and the Queue Manager failed after the FTE transfer is complete and before the DataPower transfer starts, the file would disappear from the system without any explicit transfer error.

The use of persistent messages has an important implication in terms of Queue Manager tuning as described in this article, Configuring and tuning WebSphere MQ for performance on Windows and UNIX, which specifically mentions that persistent messages require appropriate tuning for the Queue Manager logs. This is critical because in our experience, insufficient log space on the Queue Manager is the most common cause of errors.

Another reason to put some effort in tuning the Queue Manager logs is that in a B2B context, messages are typically consumed within a transaction. This means that the final commit happens only when DataPower has completed the file transfer to the target system, thus keeping transactions potentially pending for a long time, depending on the total file size and network latency.

Finally, note that it is important to choose the right persistence setting for each queue because, by default, DataPower sets the message persistence according to the queue setting. For finer control, DataPower default behavior can be overwritten by explicitly setting the persistence flag in the MQ Message Descriptor header (MQMD) in the processing policy. Default queue persistence is not critical for WebSphere MQ FTE because it is designed to optimize the use of persistent messages.

Configuration examples

This article will focus on a number of typical use cases:

This article will refer to the following basic configuration:

  • A DataPower XB62 with a B2B Gateway and a Multiprotocol Gateway
  • A VMware image with the WMQFTE installation acting as the internal partner
  • An additional VMware image with an FTP Server and client acting as the external partner

For the sake of this configuration, we will use plain protocols. In a real production environment, all communication has been protected by encryption.

This DataPower configuration is extremely simple and some of its details will be highlighted while discussing the use cases.

WebSphere MQ FTE configuration

In any WebSphere MQ FTE configuration, you must have:

  • One WebSphere MQ Queue Manager that is designated as the WebSphere MQ FTE Coordination Queue Manager. The Coordination Queue manager uses WebSphere MQ queues to process file transfers and WebSphere MQ publish/subscribe to store information relating to the WebSphere MQ FTE Agents associated with this configuration. The WebSphere MQ FTE Coordination Queue Manager must be at WebSphere MQ version 7 or higher for the required publish/subscribe functionality.
  • One or more WebSphere MQ Queue Managers that acts as WebSphere MQ FTE Command Queue managers. The Command Queue manager is used to pass command requests to WebSphere MQ FTE.
  • One or more WebSphere MQ Queue Managers that acts as WebSphere MQ FTE Agent Queue Managers. Agent Queue Managers provide the WebSphere MQ queues required to support the transfer of files between agents using WebSphere MQ messages.
  • One or more WebSphere MQ FTE Agents that performs file transfers. Each agent connects to one Agent Queue Manager. An Agent Queue Manager can support many agents.

Our WebSphere MQ FTE configuration used WebSphere MQ V7.0.1.8 and WebSphere MQ FTE V7.0.4.1. All WebSphere MQ and WebSphere MQ FTE components were running on Windows®. With the announcement of WebSphere MQ V7.5, WebSphere MQ FTE becomes an optional component of WebSphere MQ 7.5 and will be known as "WebSphere Managed File Transfer".

We used a simple WebSphere MQ FTE configuration on Windows to test the DataPower to WebSphere MQ FTE use cases described above. This configuration consisted of a single WebSphere MQ Queue Manager named FTE_QM. That fulfilled the roles of Coordination, Command, and Agent Queue Manager described above. FTE_QM was created with the default circular log allocation of three primary and two secondary log files. We found that we had to amend this when we started transferring large files. In a production configuration, it would be more typical to have agents connected to a number of different agent queue managers.

On FTE_QM, we defined two WebSphere MQ queues as "FTE_TO_B2BGW" and "FTE_TO_MPGW". These queues were used as the destination queue for the File-to-Queue transfers initiated between AGT1_FTE_FROM_FILE and AGT3_DP_FTE_QM. The messages placed on these queues were then read by DataPower for transfer onto the final destination. Each queue had a backout queue: "B2B_BACKOUT" for FTE_TO_B2B and "MPGW_BACKOUT" for FTE_TO_MPGW.

An additional queue named "DP_TO_FTE" was created for transfers between DataPower to FTE. The default persistence flag for this queue was set to “persistent”.

We then created three WMQ_FTE agents on the same Windows machine as follows:

  • AGT1_FTE_FROM_FILE: This is the source agent for transfers from WebSphere MQ FTE to DataPower. This agent was used to read the source to be transferred from the Windows file system.
  • AGT2_FTE_TO_QUEUE_FOR_DP: This is the destination agent for transfers from WebSphere MQ FTE to DataPower. Files transferred from AGT1_FTE_FROM_FILE were sent to this agent as a File-to-Queue transfer, resulting in the file arriving as a group of WebSphere MQ messages on the destination queue. This queue was monitored by the DataPower box and when all the messages in the group had arrived, DataPower then processed the transfer onto the final destination defined in DataPower. In our case, the final destination for WebSphere MQ FTE to DataPower was an FTP server running on separate Windows machine from WebSphere MQ and WebSphere MQ FTE. To enable file-to-queue transfers, the property enableQueueInputOutput was set to "true" in the AGT3_DP_FTE's file.
  • AGT3_DP_TO_FTE: This is the source agent for transfer from DataPower to WebSphere MQ FTE. The file is read by DataPower and then sent as one of the WebSphere MQ messages to a WebSphere MQ queue. This queue is monitored by a WebSphere MQ FTE Monitor. It then starts a WebSphere MQ FTE transfer when the messages arrive onto the queue. This agent is the source agent in this triggered transfer and AGT1_FTE_FROM_FILE is the target.

The three WebSphere MQ FTE agents are connected to the FTE_QM agent Queue Manager using cross memory – or bindings – connections, rather than a WebSphere MQ client connection. All three agents were defined to run as a Windows service. Figure 2 shows the WebSphere MQ FTE configuration, highlighting the agents, the queue manager, and the queues.

Figure 2. WebSphere MQ FTE configuration
WebSphere MQ FTE configuration
WebSphere MQ FTE configuration

The WebSphere MQ FTE to MPGW interaction is similar. The difference is that the WebSphere MQ queue used for the outbound transfers is "FTE_TO_MPGW".

Use case 1: Outbound file transfer using a B2BGW

The B2BGW routing is now well defined and described in the WebSphere DataPower Release 4.0.1 B2B MQFTE document.

The first step is to create an internal and an external partner. For the external partner, add an FTP destination. For the internal partner, add a WebSphere MQ FTE destination.

It is a good practice to manage partial file transfers by enabling transactionality on the WebSphere MQ FTE destination URL and by renaming the file after the transfer for the FTP destination.

Similarly, create a WebSphere MQ FTE front side handler to pick up files from the internal partner and a FTP server FSH to accept files from the external partner.

The last step is to create a B2BGW by adding the two front side handlers and the two business partners.

The WebSphere MQ FTE file transfer will include the following headers:

  • DPMQFTESenderID: Business ID of the internal partner
  • DPMQFTEReceiverID: Business ID of the external partner
  • DPMQFTEContentType: Content type of the message payload

An example is shown in Listing 1.

Listing 1. fteCreateTransfer that includes the DPMMQFTE headers
fteCreateTransfer -sa AGT1_FTE_FROM_FILE -da AGT2_FTE_TO_QUEUE_FOR_DP -dm FTE_QM -dq FTE_TO_B2BGWGW@FTE_QM -dqp true -qmp true -md DPMQFTESenderId=InternalPartner, DPMQFTEReceiverId=ExternalPartner,DPMQFTEContentType=application/binary -qs 1M -t binary c:\temp\file03.bin

In the fteCreateTransfer command shown in Listing 1, –sa and –da are the source (AGT1_FTE_FROM_FILE) and destination (AGT2_FTE_TO_QUEUE_FOR_DP) WebSphere MQ FTE agents, respectively. -dm denotes the destination Queue Manager (FTE_QM), and -dq the destination queue name (FTE_TO_B2BGW), which also determines that this is a File to Queue transfer. "-dqp true" determines that the messages put to the destination queue will be persistent. "-qmp true" implies that first message written to the destination queue will have message properties set.

The values following –md are name-value pairs that are passed into WebSphere MQ FTE. Finally, "-qs 1m" is used to tell WebSphere MQ FTE to split the transfer into multiple messages that form a WebSphere MQ message group. This group has a maximum message size of 1MB. We found that if we did not use the –qs setting for the large files, the WebSphere MQ FTE transfer failed with reason code 2030, which is MQRC_MSG_TOO_BIG_FOR_Q. To prevent this error, we adjusted the maximum message size settings on both the FTE_QM Queue Manager and on the FTE_TO_B2BGW queue. This is the MAXMSGL setting that defaults to 4Mb. Advice on altering the WebSphere MQ MAXMSGL settings and the WebSphere MQ FTE maxInputOutputMessageLength agent property is found in Guidance for setting WebSphere MQ and WebSphere MQ File Transfer Edition properties associated with message size.

Unit-Of-Work (UOW) is enabled on the DataPower Queue Manager object (see Figure 3). In case of errors, the transfer is re-tried until the backout count in reached, and then the file is moved to the backout queue. Given that the file is at first assembled in memory and then pushed to the back-out queue, it is not surprising that messages in the backout queue can potentially have a different size from the messages in the request queue.

Figure 3. DataPower Unit of Work configuration
How to enable DataPower Unit of Work configuration
How to enable DataPower Unit of Work configuration

Finally, it is worth noticing that the GMO settings in the WebSphere MQ FTE protocol handler are overwritten by the settings in the DataPower Queue Manager Object. When the UOW is enabled, the messages are consumed under syncpoint, irrespective of their persistence and vice-versa.

WebSphere MQ FTE can optionally log transfer details into a database. DataPower B2B Transaction Viewer can retrieve and visualize that information as part of the file transfer by configuring the DataSource object on the advanced panel of the B2B Gateway.

Since only one logger DB can be referenced by a B2BGW, both inbound and outbound flows going through the same DataPower service are recorded in the same logger.

Use case 2: Outbound file transfer using a MPGW (streaming)

This is probably the simplest configuration. On the DataPower side, there is a Multi-Protocol Gateway with a dynamic backend and a WebSphere MQ FTE front side protocol handler.

Since the goal of this configuration is to stream large files, the processing policy is usually extremely simple, as shown in Figure 4.

Figure 4. DataPower rule definition
MPGW DataPower rule definition
MPGW DataPower rule definition

The minimal processing required is to set a destination. The WebSphere MQ FTE agent can pass the metadata to DataPower using RFH headers that DataPower can use to set the destination.

The example shown in Listing 2 uses a custom header called “DPMQFTEDestination”. MQFTE transfers a file to a queue using the command shown in Listing 2.

Listing 2. fteCreateTransfer that uses the DPMQFTEDestination header
fteCreateTransfer -sa AGT1_FTE_FROM_FILE -da AGT2_FTE_TO_QUEUE_FOR_DP -dm FTE_QM -dq FTE_TO_MPGW@FTE_QM -dqp true -qmp true -md DPMQFTEDestination=ftp://user:password@xx.xx.xx.xx/foo.Part?Rename=Foo.Complete -qs 1M -t binary c:\temp\file01.bin

In order to avoid undetected failures, messages in the WebSphere MQ queue, that is the WebSphere MQ FTE transfer destination, are persistent.

When streaming the segmented WebSphere MQ FTE messages, a good practice is to keep the message size relatively small (less than 1MB).

In this example, the DPMQFTEDestination header is set to an FTP server, which is accessed using a user ID of “user” and a password of “password”. The FTP server is located at IP address xx.xx.xx.xx. The file is transferred in parts with a name prefix of “foo.part”. When the transfer is complete, the file is "Foo.Complete".

The route action in the DataPower stylesheet uses this header to set the actual destination, as shown in Listing 3.

Listing 3. DataPower XML Style sheet for the MPGW routing
<?xml version="1.0" encoding="UTF-8"?> <xsl:stylesheet xmlns:xsl="" version="1.0" xmlns:dp="" extension-element-prefixes="dp" exclude-result-prefixes="dp"> <xsl:template match="/"> <!-- Get the MQRFH2 headers --> <xsl:variable name="MQRFH2" select="dp:request-header('MQRFH2')" /> <!-- Parse the MQRFH2 headers to XML format --> <xsl:variable name="parsedMQRFH2" select="dp:parse($MQRFH2)" /> <!-- log MQRFH2 --> <xsl:message dp:priority="debug"> <xsl:copy-of select="$parsedMQRFH2" /> </xsl:message> <!-- extract destination URL --> <xsl:variable name="finalDestination" select="$parsedMQRFH2//DPMQFTEDestination"></xsl:variable> <!-- set destination --> <dp:set-variable name="'var://service/routing-url'" value="$finalDestination" /> </xsl:template> </xsl:stylesheet>

As for the MPGW settings, the request type is NON-XML, the response is set as “Pass-Thru”, the flow control is enabled, and both the request and response are streamed as shown in Figure 5.

Figure 5. DataPower configuration for streaming
DataPower configuration for streaming
DataPower configuration for streaming

As already pointed out, DataPower firmware V5.x introduced the extended memory support that greatly reduces the need for streaming. However, for very large files (greater than 1GB), streaming might still be the best option. You need to consider that in buffering mode, the two transfers from FTE to DataPower and from DataPower to the FTP Server are strictly sequential. However, in streaming mode, the two file transfers happen largely in parallel, diminishing the overall latency.

Finally, this service is not integrated with the B2BViewer.

Use case 3: Inbound file transfer using a B2BGW

The DataPower configuration is the same described in Use case 1.

This time the routing, such as the identification of the partnerIDs, follows the usual B2BGW schema. For the generic binary files, the b2b-routing stylesheet needs to be customized. In our example, we will identify the external partner based on the source IP and the internal partner based on the destination directory.

The internal partner destination is a WebSphere MQ FTE URL. Note that DataPower includes into the message the final destination within the WebSphere MQ FTE network. A typical WebSphere MQ FTE destination looks similar to Listing 4.

Listing 4. mqfte back-end URL when using a DataPower MQ Q-Manager object
dpmqfte://FTE-QM/?RequestQueue=DP_TO_FTE&DestAgent= AGT3_DP_TO_FTE&DestQM=FTE_QM&DestFile=dummyPath&Transactional=true

The required URL parameters are:

  • DestAgent: The destination agent in WebSphere MQ FTE network that receives a message.
  • DestQM: The destination queue manager to which the source agent sends a message.
  • DestFile: The name of the file where the destination agent stores a received message.

For this use case, we created a third WebSphere MQ FTE agent named "AGT3_DP_TO_FTE", again running as a Windows service and with "enableQueueInputOutput=true" in the file to enable Queue-to-File transfers. WebSphere MQ FTE needs to monitor the destination WebSphere MQ Queue that DataPower writes to and trigger a WebSphere MQ FTE transfer to process the file. To do this, we created a WebSphere MQ FTE monitor using the fteCreateMonitor command shown in Listing 5.

Listing 5. fteCreateMonitor command for Use case 3
 FteCreateMonitor.cmd -ma AGT3_DP_TO_FTE -mq DP_TO_FTE -mn XB62_Monitor -mt D:\Products\WMQFTE\Datapower\Stefano\XB62_Monitor.xml -pi 10 -pu seconds -tr completeGroups

In this command, –ma denotes the name of the monitoring agent, which is set to "AGT3_DP_FTE_QM". The agent we created for this use case is used as the source agent for the triggered transfer. The –mq value of DP_TO_FTE is the name of the WebSphere MQ Queue to be monitored, which is the target of the transfer from DataPower.

–mn is the name of the monitor to be created, which is "XB62_Monitor" in this case. –pu is set to seconds, then the –pi value of 10 sets a polling interval of seconds. The –tr value of completeGroups specifies that the WebSphere MQ FTE transfer command defined in the –mt value is only initiated when a complete group of WebSphere MQ messages has arrived on the monitored queue. The WebSphereMQ FTE transfer command, submitted when the message group is complete, is defined in the monitor task definition file specified via the –mt parameter. This is an XML file that, in our scenario, contained the following shown in Listing 6.

Listing 6. FTE monitor task definition XML
<?xml version="1.0" encoding="UTF-8"?> <request version="5.00" xmlns:xsi="" xsi:noNamespaceSchemaLocation="FileTransfer.xsd"> <managedTransfer> <originator> <hostName></hostName> <userID>fteuser</userID> </originator> <sourceAgent QMgr="FTE_QM" agent="AGT3_DP_FTE_QM"/> <destinationAgent QMgr="${DPMQFTEDestinationQM}" agent="${DPMQFTEDestinationAgent}"/> <transferSet> <item checksumMethod="MD5" mode="binary"> <source disposition="delete" recursive="false" type="queue"> <queue>DP_TO_FTE</queue> </source> <destination exist="overwrite" type="file"> <file>d:\temp\FromXB62\${DPMQFTEDestinationFile}</file> </destination> </item> </transferSet> </managedTransfer> </request>

This file was created using the –gt option on the fteCreateTransfer command. Using the –gt option to create this file is the easiest way of making sure that the format is correct. The generated task definition file can then be modified as required, notably for variable substitution.

In the above task definition file shown above, sourceAgent defines the source WebSphere MQ FTE agent. It is associated with the Queue Manager for the transfer that is to be initiated when a group of messages arrives from DataPower. In this case, these are AGT3_DP_TO_FTE and FTE_QM, respectively. The destinationAgent defines the name of the target agent and it is the Queue Manager for the transfer. In the example above, the Queue Manager is set to “${DPMQFTEDestinationQM}” and the agent to “${DPMQFTEDestinationAgent}”. These values are substituted at runtime by the value of the variables DPMQFTEDestinationQM and DPMQFTEDestinationAgent set by DataPower.

Finally, the name of the destination file is defined in the “file” XML tag. This time the path is hard coded to “d:\temp\FromXB62\”, but the file name itself is set by the variable DPMQFTEDestinationFile.

It is important to note that DataPower, by default, puts messages using the default queue persistence. Therefore, that needs to be set carefully.

Persistent messages require careful tuning of the Queue Manager logs, but can survive a Queue Manager crash. Non-persistent messages, on the other side, perform better, but a file can potentially disappear without any clear error.

Use case 4: Inbound file transfer using a MPGW (streaming)

The WebSphere MQ FTE configuration in this scenario is exactly the same as the previous one.

The main difference is that DataPower routing is not based anymore on Business Partner IDs, but on some logic in the processing policy.

This configuration supports streaming for incoming files and is a good fit for very large file transfers.


This article described a clear criteria on choosing between B2BGWs and MPGWs when integrating DataPower and WebSphere MQ FTE. It also provided concise examples of working configurations.


The authors would like to thank Richard Kinard and Adrian Preston for their contributions and reviews of this article.

Downloadable resources

Related topics


Sign in or register to add and subscribe to comments.

ArticleTitle=Using WebSphere DataPower and WebSphere MQ File Transfer Edition to manage file transfers