Customizing file transfers using Ant scripts with WebSphere MQ File Transfer Edition

This article shows you how to use Ant scripts and Java classes to customize WebSphere MQ FTE to automatically transfer new and modified files from a source folder to a destination folder, and ensure that files in source and destination folders are always synchronized as efficiently as possible.

Shashi R Thambrahalli (shashikanth@in.ibm.com), Advisory Software Engineer, WebSphere MQ Client Development team, IBM

Photo of Shashi R. ThambrahalliShashikanth (Shashi) R. Thambrahalli is an Advisory Software Engineer on the WebSphere MQ Client Development team at the IBM India Software Lab in Bangalore. He has nine years of experience with IBM, focusing on WebSphere MQ clients, C/C++, and Microsoft .NET, XMS .NET, and WCF. He develops new features for MQ clients and provides consulting services on WebSphere MQ, WebSphere MQ FTE, and WebSphere MQ Advanced Message Security for customers in India and the Asia-Pacific region. You can contact Shashi at shashikanth@in.ibm.com.



Bala Subramanian Vetrivel (bvetrive@in.ibm.com), Technical Specialist, IBM

Photo of Bala VetrivelBala Subramanian Vetrivel is a Technical Specialist on the Worldwide WebSphere Business Partner team. He has seven years of experience in the IT Industry and his areas of specialization include BPM, C/C++, IBM Blueworks, IBM Component Business Model, IBM PureSystems, IBM Rational products, IBM Workload Deployer, Java, SOA, WebSphere Application Server, WebSphere Business Modeler and Monitor, WebSphere Business Services Fabric, WebSphere Industry Content Packs, WebSphere Integration Developer, WebSphere MQ, WebSphere Process Server, and WebSphere TX. In his current role, he works with IBM business partners and customers on architecture design, proofs of concepts, requests for proposals, and technology enablement. He also has contributed to developerWorks articles, the Biztech project, and IBM Redbooks. You can contact Bala at bvetrive@in.ibm.com.



20 June 2012

Also available in Russian Portuguese

Introduction

IBM® WebSphere® MQ File Transfer Edition (FTE) provides an enterprise-class platform for managed file transfer operations. For most of the file transfer requirements, WebSphere MQ FTE provides a straight forward way to configure file transfers. But in some situations, WebSphere MQ FTE needs to be extended to develop a customized file transfer solution. This article explains a customer scenario with the following requirements:

At the source:

  • Transfer only those files created or modified after a given time, and copy those files recursively.

At the destination:

  • Delete files that are not present in the same folders at the source, and do this deletion recursively.

At the end of this operation, the destination should be in sync with the source for a given time with minimal effort and without copying entire source folders. Only those files modified after a given time have to be considered for synchronization.

Like WebSphere MQ, WebSphere MQ FTE can invoke user exits before a file transfer begins and after it is completed. You can have exits at both source and destination agents. At the source agent before the file transfer begins, you can use an exit to rename the file on the destination system. After the file transfer ends, you can use the exit at the destination agent to send an event notification to those who need to know the status of file transfer. For more information on WebSphere MQ FTE exits, see WebSphere MQ FTE source and destination user exit routines in the WebSphere MQ FTE information center.

WebSphere MQ FTE also provides a rich set of Ant tasks that you can use to customize your file transfer solutions. You can use the fteAnt command to run an Ant script in an environment with WebSphere MQ FTE.

Scenario and solution overview

A meteorological department has weather monitoring centers spread across England -- for example, in Bristol, Dover, Manchester, an Southampton. These monitoring centers measure weather parameters such as temperature, humidity, and air pressure. Measurements go on throughout the day, and the data is stored in files on servers located at each monitoring center. The department has a central data center at London where data from each monitoring center is stored in a folder named for each center, such as Data_Bristol. Monitoring centers send the files containing weather data to the data center at predefined intervals throughout the day, and the files at the monitoring centers and the data center must always be in sync. This article shows you how to set up an environment to maintain this synchronization using a combination of WebSphere MQ FTE Ant scripts, XML, and Java classes at both source and destination.

At the source:

  • Transfer only those files in source folders that are created or modified after a given time.
  • Transfer files from subfolders also.

At the destination:

  • Delete all files that are not present in the same folder in the source folders and subfolders.
  • Create a log of all deleted files.

For more information on WebSphere MQ FTE Ant Scripts see Using Apache Ant with WebSphere MQ FTE in the WebSphere MQ FTE information center.


Prerequisites

Define the following WebSphere MQ FTE environment:

  1. On a server, define and install the following components. A Microsoft® Windows® server is used as the admin server in the "data center."
  2. Install WebSphere MQ V7.0.3 or later, including WebSphere MQ Explorer.
  3. Define a WebSphere MQ queue manager called LONDON.DATA.CENTER.QMGR, which will be used as the coordination, command, and agent queue manager.
  4. Define a listener on an unused port and start this listener. It will be used by the remote agents.
  5. Install WebSphere MQ FTE V7.0.3 Server Edition.
  6. As part of the server installation, define an agent called LONDON.DATA.CENTER.AGNT. Since it is a local agent to the queue manager, it can be defined as a bindings connection agent.
  7. Install the WebSphere MQ FTE V7.0.3 WebSphere MQ Explorer plug-in from the Remote Tools and Documentation installation.
  8. On the servers that are acting as monitoring centers (use Bristol and Dover for this example), install WebSphere MQ FTE Client Edition.
  9. Define a client connection agent on each server -- BRISTOL.AGENT and DOVER.AGENT -- so that they client connect to the LONDON.DATA.CENTER.QMGR queue manager using the port that the listener defined earlier is listening on. Use the LONDON.DATA.CENTER.QMGR queue manager as their agent, command, and coordination queue manager.

If you are new to WebSphere MQ FTE and need information on setting it up, see the developerWorks article WebSphere MQ FTE walkthrough. You are now ready to follow Steps 1 through 6 below.

1. Develop exit for source agent

The main task of the exit is to remove all the files whose created or modified time stamp is before the specified time stamp. You can specify the time stamp in a number of ways -- this article uses an XML file like the one below:

<property  name="syncTimeStamp" value="06/21/2012 23:55:00" />

You can also use the user-defined Transfer metadata to pass a timestamp and modify the exit to use this timestamp to exclude files from the transfer list. The exit implements the com.ibm.wmqfte.exitroutine.api.SourceTransferStartExit interface and overrides onSourceTransferStart. WebSphere MQ FTE will invoke this method before a file transfer starts.

Compile the source to create a class. Here is the sample listing of the method:

public class RemoveUnmodifiedFilesFromList implements SourceTransferStartExit {
    public TransferExitResult onSourceTransferStart (String sourceAgentName, 
        String destinationAgentName, 
        Map <String, String> envMetaData,
        Map <String, String> transferMetaData,
        List <SourceFileExitFileSpecification>fileSpec)
    {
        // Read timestamp from XML file
        final String parmTS = transferMetaData.get("syncTimeStamp");
        if(syncTS > createdDate)
        {
            fileSpecs.remove(i); // No So remove it
        }
    }
}

2. Configure source agent

  1. Create an XML file as shown below. Change the property value as needed.
    <property  name="syncTimeStamp" value="06/21/2012 23:55:00" />
  2. Specify the exit class in the source agent's agent.properties file:
    sourceTransferStartExitClasses=com.ibm.wmqfte.example.RemoveUnmodifiedFilesFromList
  3. Copy the exit class to the agent coordination queue manager's exits folder. For example:
    <fte install folder> config/LONDON.DATA.CENTER.QMGR/agents/BRISTOL.AGENT/exits

3. WebSphere MQ FTE Ant script at source agent

  1. The WebSphere MQ FTE Ant file calls a custom Ant task to list all files at the source. After file transfer, this list will be used at the destination agent to delete files not present in the source agent.
    <listFiles dir="${srcDir}" />
    <echo message="recurive.Files***" />
    <echo message="${recurive.Files}" />
  2. Call the external Ant script file to delete unnecessary files at the destination:
    <fte:filespec srcfilespec="${srcDir}" conversion="binary" 
         dstdir="${dstDir}" overwrite="true" />
    <fte:postdst command="DeleteDest.xml" type="antscript" successrc="0">
    <fte:property name="recurive.Files" value="${recurive.Files}"/>
    <fte:property name="destDir" value="${dstDir}"/>
    </fte:postdst>
  3. In the external Ant script, call a custom Ant task that deletes unnecessary files by comparing the source file listing with the destination folder:
    <SyncDir srcList="${recurive.Files}" destDir="${destDir}"/>

4. Configure destination agent

  1. Specify the location of the external Ant script in the destination agent's
    agent.properties

    file:
    commandPath=C\:\\IBM\\WMQFTE\\jre\\bin;C\:\\IBM\\WMQFTE\\bin\\;
    C\:\\WINDOWS\\system32\\;

5. Creating the custom Ant task

  1. Extend the class org.apache.tools.ant.Task.
  2. Override the execute method and add your functionality:
    public @Override  void execute() throws BuildException
  3. Package this custom task as a JAR file and place it in C:\IBM\WMQFTE\ant\lib.

Step 6. Calling the custom Ant task

  1. Add the taskdef resource:
    <taskdef resource="net/sf/antcontrib/antlib.xml"/>
  2. Define the custom Ant task by providing a class name:
    <taskdef 
    name="listFiles" classname="com.ibm.mq.fte.file.sync.ant.util.ListFilesTask" 
    classpath="{ant.home}/lib" 
    />
    <taskdef 
    name="listDir" classname="com.ibm.mq.fte.file.sync.ant.util.ListDirectoriesTask“ 
         classpath="{ant.home}/lib" 
    />
  3. Place the following JAR files inside ant/lib: Ant-contrib-1.0b3-bin and AntExtras.

Download files

The following files are contained in the download file AntScript.zip:

agent.properties
Specifies location of exit class of source agent.
mq-ant-utils.jar
Classes related to the custom Ant command. Place in the ant library of the source and target agents.
SyncDirWithLoggingSingleXferSyncTS.xml
FTE Ant script to be executed at the source agent.
DeleteDest.xml
FTE script invoked by the source agent's FTE Ant script. Specify the location of this external Ant script in the agent.properties file of the destination agent.

Download

DescriptionNameSize
Code sampleAntScript.zip15 KB

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=821760
ArticleTitle=Customizing file transfers using Ant scripts with WebSphere MQ File Transfer Edition
publish-date=06202012