WebSphere MQ File Transfer Edition walkthrough

WebSphere MQ File Transfer Edition is a reliable managed file transfer solution for moving files, regardless of their size, between IT systems without the need for programming. It uses the enterprise-scale WebSphere MQ messaging system as its transport mechanism. This article provides a simple walkthrough of installation, configuration, basic file transfers, monitoring, and scripting with Apache Ant.

Martin Phillips (mphillip@uk.ibm.com), Software Engineer, IBM

Martin Phillips photoMartin Phillips is currently a test team leader working on IBM WebSphere MQ. He has worked for IBM since 1999, mainly as a tester for IBM WebSphere Application Server, and particularly with the service integration bus. Recently he was a developer on IBM WebSphere Business Monitor Version 6.1.



10 March 2010

Introduction

IBM® WebSphere® MQ File Transfer Edition (FTE) is a reliable managed file transfer solution for moving files, regardless of their size, between IT systems without the need for programming. This article is a simple walkthrough that shows you how to install, configure, perform file transfers, and do some more advanced tasks with WebSphere MQ FTE.

Prerequisites

  • WebSphere MQ File Transfer Edition V7.0.2 -- Server
  • WebSphere MQ File Transfer Edition V7.0.2 -- Documentation and Tools
  • WebSphere MQ V7.0.1 or later, including MQ Explorer
  • Working knowledge of WebSphere MQ and WebSphere MQ administration. (This article does not explain some WebSphere MQ concepts and configuration steps, because it focuses on WebSphere MQ FTE.)
  • A queue manager that you have defined and started. This walkthrough uses a queue manager called QM1.

Downloads

Concepts and terminology

(role)
The coordination, agent, and command queue managers (see below) are roles rather than specific queue managers. A single queue manager can fulfill the role of all three, or you can have a queue manager dedicated to each role. The choice depends on your existing MQ infrastructure and the expected load on the network. This walkthrough uses a single queue manager to fulfill all three roles, primarily to make things simpler and avoid the additional MQ configuration required to enable connectivity between multiple queue managers (MQ must know how to send messages from one queue manager to the other one, which can be done using sender/receiver channel pairs or a cluster).
Agent
An FTE agent is a JVM process that runs on a machine and performs file transfers to and from other agents. In order to transfer files to or from a machine using FTE, you must have an agent running on that machine. Every agent connects to a WebSphere MQ queue manager and uses MQ to communicate with other agents. The queue manager that an agent connects to fulfills the agent queue manager role, and will have a number of queues defined on it for the agent. An agent will connect only to its agent queue manager, which it does using bindings or client connection modes. For more information, see WebSphere MQ FTE V7 Information Center.
Agent queue manager (role)
A queue manager that hosts an agent's queues. Each agent needs a number of queues defined to be able to function. A queue manager can be the agent queue manager for any number of agents. When transferring files between two agents, their respective agent queue managers must have connectivity.
Coordination queue manager (role)
A queue manager in the MQ network that acts as a central location for collecting and broadcasting audit and file transfer information. It also gathers information about all the agents in the network at a single point where it can be queried. All agent queue managers in the MQ network must have connectivity to the coordination queue manager to route appropriate information there.
Command queue manager (role)
Submitting commands to an agent (such as requesting a transfer) involves sending an MQ message to the agent's command queue on the agent queue manager. The command queue manager is the queue manager used by the command-line commands to connect to in order to submit these command messages to agents. WebSphere MQ routes the messages from the command queue manager to to the appropriate agent queue manager. The command queue manager must have connectivity to all agent queue managers.

Installing WebSphere MQ FTE

This walkthrough assumes that you are working on a Windows® system. For a Unix® system, substitute appropriate paths and commands and ensure that you have read/write access to all relevant directories. This walkthrough uses a single queue manager for the coordination, command, and agent queue managers.

Before you start

  • Make sure that you have a queue manager defined and started on your system. This walkthrough uses a queue manager called QM1
  • If you are running on Unix, there is no need to run the install as root. Instead, run the install as the user you will use to run your agents, but you must ensure that this user has access to write to the product install directory (product binaries) and data directory (configuration data) that you will be using. The default product install directory is /opt/IBM/WMQFTE, and the default data directory is /var/IBM/WMQFTE/config, but you may change those to any directories for which you have write access.
  1. Start the Server install by running the install executable from the unzipped download or CD: Windows_x86\Disk1\InstData\VM\install.
  2. On the Flash panel, click Select.
  3. On the Introduction panel, click Next.
  4. On the License panel, accept the license and click Next.
  5. On the Product installation panel, either accept the defaults or else enter the directory where you wish to install the product, and then click Next. This walkthrough assumes that you have used c:\WMQFTE as the product installation directory:
    Figure 1
  6. On the Product data panel, either accept the defaults or else enter the directory where you wish to install the product data directory, and then click Next. This walkthrough assumes that you have used c:\WMQFTE\config as the product data directory:
    Figure 2
  7. On the Coordination queue manager name panel, enter the name of your MQ V7 queue manager (QM1). Select Bindings for the connection transport and then click Next. If you select client, you will be prompted for additional MQ client connection information, such as host, port, and channel name on the next panel. Client connection will also require you to configure a listener and start it on your queue manager:
    Figure 3
  8. On the Agent details panel, enter a name and description for your agent. This walkthrough will use the agent name AGENT1. For the Agent queue manager name, enter the name of your MQ V7 queue manager (QM1). Select Bindings for the connection transport and click Next: xxx
    Figure 4
  9. On the Command queue manager panel , enter the name of your MQ V7 queue manager (QM1 ). Select Bindings for the connection transport
    Figure 5
  10. On the Pre-installation summary panel, click Install and wait for the installation to continue.
  11. The Run on coordination queue manager panel will soon open. Read the instructions on the panel. The install has created an MQSC script that will configure the coordination queue manager for File Transfer Edition.
  12. From a command line, run the following command to configure the coordination queue manager: runmqsc QM1 < C:\WMQFTE\config\QM1\QM1.mqsc. Click Next.:
    Figure 6
  13. The Run on agent queue manager panel will soon open. Read the instructions on the panel. The install has created an MQSC script that will configure the agent queue manager for your File Transfer Edition agent. From a command-line, run the following command to configure the agent queue manager and create the queues that the agent requires: runmqsc QM1 < C:\WMQFTE\config\QM1\agents\AGENT1\AGENT1_create.mqsc. Click Next.:
    Figure 7
  14. On the Install complete panel, click Done..

Installing WebSphere MQ FTE remote tools and documentation

  • Make sure that MQ Explorer is not running before starting this install.
  • Make sure that you have a queue manager defined and started on your system. This walkthrough uses a queue manager called QM1.
  • If you are running on Unix, you do not need to run the install as root. Run the install as the user you intend to use to run your agents, but you must ensure that this user has access to write to the product install directory (product binaries) and data directory (configuration data) that you will be using. The default product install directory is /opt/IBM/WMQFTE and the default data directory is /var/IBM/WMQFTE/config, however you may change these to any directories that you have access to write to.
  1. Start the install by running the install executable from the unzipped download or CD: Windows_x86\Disk1\InstData\VM\install.
  2. On the Flash panel click Select to select your installation language.
  3. On the Introduction panel click Next.
  4. On the License panel, accept the license and click Next.
  5. On the Set of features panel, click Complete installation and then click Next:
    Figure 8
  6. On the Product installation directory panel, enter the same directories you used for the Server installation (for example c:\WMQFTE) for the product installation directory. Click Next.
  7. On the Product data directory panel, enter the same directories you used for the Server installation (for example c:\WMQFTE\config) for the data directory. Click Next
  8. A pop-up will open stating that the configuration directory already exists and is in use. Select Use existing to configure the Tools install to use the same configuration data as the Server install:
    Figure 9
  9. On the Pre-installation summary panel, click Install and wait for the installation to continue.
  10. On the Install complete panel, click Done.

Creating, starting, and stopping an agent on the command line

OK, you've installed the various parts of the product, and as part of the server install, you have created an FTE agent AGENT1. Normally you will have agents on several machines and will transfer files between them, but for this simple walkthrough you will create a second agent on the same machine and perform transfers between them. To do this use the command line:

  1. On the command line, change directory to the bin directory under the product install directory. For example: C:\WMQFTE\bin.
  2. Run the fteCreateAgent command. To display the full help for this command, run it with the -? option: fteCreateAgent -agentName AGENT2 -agentQMgr QM1 -?.
  3. You will see from the output of this command that another MQSC script has been created. Direct the commands from the MQSC script to the queue manager to configure the agent queue manager and create the queues that the agent requires: runmqsc QM1 < C:\WMQFTE\config\QM1\agents\AGENT2\AGENT2_create.mqsc.
  4. Now start the agents: fteStartAgent AGENT1, fteStartAgent AGENT2. You should see similar output to this:
    C:\WMQFTE\bin>fteStartAgent.cmd agent1
    5655-U80, 5724-R10 Copyright IBM Corp.  2008, 2009.  ALL RIGHTS RESERVED
    BFGCL0030I: The request to start agent 'AGENT1' on this machine has been submitted.
    BFGCL0031I: Agent log files located at: C:\WMQFTE\config\sioux\agents\AGENT1
  5. To stop an agent, use the fteStopAgent command: fteStopAgent AGENT1.

Using MQ Explorer to create transfers

Now you get to do the fun bit and actually move some files around.

  1. Make sure both of your agents are started.
  2. Start up MQ Explorer (for example by using the strmqcfg command).
  3. You should see a new Managed File Transfer section in MQ Explorer:
    Figure 10
  4. Under Managed File Transfer, right-click on QM1 and select New Transfer to start the New Transfer wizard:
    Figure 11
  5. Use the dropdown box to select AGENT1 as the source agent in the From section:
    Figure 12
  6. Type in the path to a file that exists on your machine that you want to transfer:
    • You can specify an individual file name. If you do not specify the fully qualified file path, the agent will use the home directory of the user who started the directory.
    • You can use * as a wildcard to transfer multiple files, such as *.jpg. If you want to recurse all subfolders searching for wildcards that match the one you specified, check Include subdirectories.
    • You can specify a directory name if you want to transfer the directory and all files in it. If you want to recurse all subfolders in the directory, check Include subdirectories.
  7. Use the dropdown box to select AGENT2 as the destination agent in the To section.
  8. Type in a destination directory to transfer the file to.
  9. If you specified a specific source filename, you can specify the name of the file as it should appear on the destination system. If you specified a source directory or wildcard, the files will keep their original names on the destination system:
    Figure 13
  10. To define a transfer for multiple file specifications (such as *.txt and *.jpg, thisfile.doc and thatfile.tar, or *.zip and index.html), click Add to group and enter more source and destination file specifications:
    Figure 14
  11. Click Finish now and the transfer should be carried out.
  12. You can monitor transfer progress in the Current transfer progress tab at the bottom of MQ Explorer:
    Figure 15
  13. In the left navigation panel, click Transfer log to see the results of the transfer:
    Figure 16
  14. The above transfer was only partially successful, because some of the files specified did not exist. Explanations of why transfers fail are listed here.

Using the command line to create transfers

To create a transfer using the command line, use the fteCreateTransfer command:

C:\WMQFTE\bin>fteCreateTransfer.cmd -sa AGENT1 -sm QM1 -da AGENT2 -dm QM1 -dd
"c:\test\target" "c:\test\source\20M_bin.zip" 5655-U80, 5724-R10
Copyright IBM Corp. 2008, 2009. ALL RIGHTS RESERVED BFGCL0035I: Transfer request issued.
The request ID is: 414d512073696f757820202020202020f962754b20002f03 BFGCL0182I:
The request is waiting to be processed by the agent.

Parameters

-sa
Name of source agent
-sm
Name of source agent's queue manager. You did not need to enter this information on the GUI, because the GUI can extract it from the agent information published on the coordination queue manager. The command line does not look this information up, in order to reduce the dependency on the availability of the coordination queue manager.
-da
Name of destination agent
-dm
Name of destination agent's queue manager (see above)
-dd
destination directory
Last parameter
Path of files to transfer

The command line returns the request ID, a unique ID to identify this transfer request. The command says that the request is waiting to be processed by the agent. The fteCreateTransfer command submits the transfer request, but does not wait for the transfer to finish. If you want the command line to wait until the transfer is completed, use the -w option:

C:\WMQFTE\bin>fteCreateTransfer.cmd -sa AGENT1 -sm QM1 -da AGENT2 -dm QM1 -w -dd
"c:\test\target" "c:\test\source\20M_bin.zip" 5655-U80, 5724-R10 
Copyright IBM Corp.  2008, 2009.  ALL RIGHTS RESERVED
BFGCL0035I: Transfer request issued.  The request ID is: 
    414d512073696f757820202020202020f962754b20003303
BFGCL0142I: The file transfer request has been submitted. 
    The command is waiting for notification of the transfer's completion.
BFGCL0139I: The requested file transfer has successfully completed.

There are many options when requesting transfers on the command line. Use the -h parameter to display help for the command: fteCreateTransfer -h.

Using MQ Explorer to create monitors

FTE Agents can be configured to watch directories for new files appearing, and then transfer then when they do. There are a couple of common monitor scenarios. One scenario is simply to watch for any new files and then move them when they appear. A second common scenario is to wait for a special trigger file to appear and then transfer a specific group of files.

Scenario 1. Move new files when they appear

  1. Under Managed File Transfer => QM1, right-click on Monitors and select New Monitor to start up the New Monitor wizard:
    Figure 17
    1. Enter a descriptive name for the Monitor you are going to create, such as MonitorXMLFiles.
    2. Enter the directory to monitor, such as C:\test\input. Make sure you have created this directory.
    3. Enter a file pattern to watch for, such as *.xml.
    4. Click on the Advanced tab:
      Figure 18
  2. The Advanced tab is where you configure how often you want the monitor to check the directory. Set the poll interval to 10 seconds so that you get a quick response for this test and click Next:
    Figure 19
  3. This panel lets you specify what to transfer. For this scenario you want to transfer every new xml file that appears in the c:\test\input directory. The monitor will create a transfer for every file it detects that matches the *.xml trigger condition, so when the transfer happens you want to transfer just the file that triggered the monitor. You can use variable substitution to generically specify the file name that triggers the transfer. For details on variable substitution, click F1 to see the online help.
    1. Select AGENT1 for the source agent.
    2. Enter ${FilePath} to tell the agent to use the the file that triggered the transfer.
    3. Select AGENT2 for the destination agent
    4. Enter a destination directory, such as C:\test\target.
    5. Enter ${FileName}-${LastModifiedTime} for the destination file name, which will append the file's last modified time to the source file name to create the destination file name. This technique is useful for making multiple file backups.
    6. Check Remove source files after completion to delete the source files after they have been transferred.
  4. Click Finish:
    Figure 20
  5. Under Managed File Transfer => QM1, click Monitors to display the list of monitors in your FTE network. Your new monitor should be listed:
    Figure 21
  6. Now, put some XML files in the monitored directory and they will transfer. Look at the Transfer Log and Current Transfer Progress to see the results. Check your target directory to make sure that the files transferred.

Scenario 2: Single trigger file initiates the transfer of other files

  1. Under Managed File Transfer => QM1, right-click on Monitors and select New Monitor to start the New Monitor wizard.
    1. Enter a descriptive name for the Monitor you are going to create, such as MonitorGOFile.
    2. Enter the directory to monitor, such as C:\test\input. Make sure you have created this directory.
    3. Enter a file pattern to watch for, such as go.file.
    4. Click on the Advanced tab and set your polling interval.
  2. Click Next.
  3. For this scenario, you want to transfer every zip file in the c:\test\input directory when go.file appears. Before the go.file is present, none of the zip files will be transferred.
    1. Select AGENT1 for the source agent.
    2. Enter C:\test\input\*.zip to tell the agent to transfer all zip files.
    3. Select AGENT2 for the destination agent.
    4. Enter a destination directory. If you want each batch of files to go somewhere different, use variable substitution in the directory name, such as C:\test\target\${LastModifiedTime}.
    5. Since the source file specification contains a wildcard, you may be transferring multiple files with this transfer and a target file name is not required.
    6. Check Remove source files after completion to delete the source files after they have been transferred.
  4. Click Finish:
    Figure 22
  5. Look at the monitors panel to see your new monitor.
  6. Create some zip files in the input directory. They will not be transferred until you create go.file.
  7. Create go.file. The zip files will be moved as a single multi-file transfer:
    Figure 23
  8. In this example, go.file is not moved or deleted as part of the transfer. It remains in the source directory, but it will not trigger any more transfers of zip files until it has been updated, since the monitor remembers that it has already triggered on a go.file of a particular last modified time. If go.file is deleted and a new one is created, or if the last modified time of go.file is updated, the monitor will trigger again, transferring *.zip. On some operating systems, copying or moving a file does not update the last modified time, so if you copy the same go.file into the monitored directory on top of an old one, it may not trigger a transfer. Therefore you should always edit the file or else delete and recreate it to ensure that the last modified time of the file is updated.

Using FTE Apache Ant scripting

WebSphere MQ FTE provides tasks that let you script transfers using the Apache Ant scripting language. We shall walk through one of the samples supplied with WebSphere MQ FTE, but for more information on the FTE Ant tasks, see the information center topic Using Apache Ant with FTE.

  1. Make a copy of of the timeout sample Ant script to modify and use. You can find the script at <WMQFTE_Install_Location>/samples/fteant/timeout/timeout.xml.
  2. This script kicks off a transfer, waits a set amount of time for the transfer to complete, and if the transfer has not completed, the script cancels the transfer. To force the transfer to time out, transfer between AGENT1 and AGENT2 with AGENT2 stopped, so that the transfer will not be able to complete. Stop AGENT2 with the command fteStopAgent AGENT2.
  3. Open your copy of the timeout.xml script file in an appropriate editor. In the init target, edit the variables defined there to match your system. For example:
    Figure 24
  4. Run the Ant script using the fteAnt command:
    C:\WMQFTE\bin>fteAnt.cmd -f C:\WMQFTE\samples\fteant\timeout\timeout.xml
    5655-U80, 5724-R10 Copyright IBM Corp.  2008, 2009.  ALL RIGHTS RESERVED
    BFGCL0211I:
    BFGCL0211I: init:
    BFGCL0211I:
    BFGCL0211I: step1:
    BFGCL0211I:
    BFGCL0211I: step2:
  5. In MQ Explorer, go to the Transfer Log section. You should see the cancelled transfer from AGENT1 to AGENT2:
    Figure 25

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=473068
ArticleTitle=WebSphere MQ File Transfer Edition walkthrough
publish-date=03102010