Introduction

During software product development life cycle, product engineering teams (particularly Development, Test and Support) often need access to most recent product builds available. At various phases during product development, a single change in the source code causes a new build to be published. Downloading product builds is kind of daily activity. In product development, which contains geographically distributed teams; there is always need to maintain local build repository, which contains few recent builds published.

Whenever new build is published, it should be downloaded at local build repository. This saves significant time of each team member from downloading the build files directly from central build machine. Once new build is generated, it is published at a central location from where each team member can access the build files. Central build machine maintains build repository where each build is identified by its unique build name. At central build machine, a directory with the build name is created under which all corresponding build files, binaries are published. There are various methods followed about how to assign an unique name to each build published, which varies product to product.
Typical examples of assigning unique name to a build :

1. Assigning numerical value as build name, which will be incremented by one for next build, example: 5510, 5511, 5512.
2. Build naming by Date, example: Build published on 21st March 2004, will be named as 040321.
3. Custom naming, example. ibmdidev_050401a, ibmdidev_050401b, ibmdidev_050402a.

There is also a chance that the files published under that build may not contain file names as fixed strings. For example: A build published as name "5510" might consist of AIX setup file named as setupAIX_5510.bin.
If we analyze the above problem, here are few essential things required with this:

1. Ability to detect a new build is published in a generic way.
2. Download specified build files and create a local build copy.

Back to top

How ITDI can help in this?

With ITDI, we can consider this as synchronization problem of local and remote build repositories.

Most often, this problem is addressed by writing automated scripts using scripting languages supported with underlying Operating Systems. These scripts are scheduled to be executed daily, weekly depending upon the frequency of new builds getting published. But this may not answer following questions:

1. Is it generic enough to work with other products (which may have different mechanism of assigning unique name to each build)?
2. Is it generic enough to execute from any other platform other than it was developed?
3. Uploading of selective build files at specified locations once the build is downloaded?
4. Ability to customize this process depending upon the needs specified in an easier way?

The solution developed using ITDI can answer the above questions and provide customizable functionality with this:

1. Ability to detect arrival of new builds at central build repository for any type of Build Naming method.
2. Schedule build download process by specifying standard "crontab" style format.
3. Downloading of selective build files from the newer build.
4. Uploading of build files after download is complete at multiple FTP locations.
5. Retry functionality in case of connection or data transfer failures.
6. E-mail notification to specified mailing list of users with detailed activities logged.

Here is the list of main ITDI components those are used while developing this solution:

• FTP Client Connector: To connect to central build repository where builds are published and retrieves the build list using FTP.
• Delta Feature: FTP Client connector in combination with Delta feature gives the ability to detect a new build is arrived at central build machine.
• Timer event Handler: Schedules the build download process on specified time interval.
• FTP Client class: The instance of com.ibm.di.protocols.FTPClient class, which actually perform uploading and downloading of files using FTP.
• Script Component: Script component allows writing JavaScript code that is required at several places with this solution.
• External Properties file: The property file where all FTP connection details, file details are provided by user which can be modified as per the need.

Back to top

Solution Architecture

Detecting arrival of a new or user specified startup build is followed by downloading build files in local repository. Once the download is complete, uploading user specified build files is performed on each upload machine. These all activities are performed sequentially at each scheduled time interval.

Timer Event Handler (timerEh) is scheduled for a specified time (example: daily at 10 AM) to trigger the build download process. When specified time is arrived, the Assembly line "DetectBuild" is called and event handler waits for next timer event to happen. The assembly line "DetectBuild" consists of FTP Client connector in iterator mode with Delta feature enabled.
FTP Client Connector connects to machine where builds are published. It is assumed that the directory with the new build name is created under which build files are published. If in the external properties, user has specified any startup build name (BUILD_startFromBuild) then work entry containing only that entry will be processed further through out the execution in order to download the build files from that directory. Else entry with operation as "add" is considered as arrival of the new build. If there are more than one add operation occurs then the entry with first add operation will be processed further and rest other entries are ignored. Once the build is detected as new build to be downloaded, the script component "CheckBuild" ensures whether it is already downloaded or not. If not then build download begins. Downloading of build files is driven by the parameter "BUILD_fileNames" which consists of list of file names user want to download. Simple Regular expressions (e.g. *.exe) can be specified with this.

The download module prepares list of files to be downloaded and downloads these files under the same name in which build files are published. This FTP directory name is nothing but the build name, which is unique. Once build is downloaded, the build files are uploaded at respective machines specified in external properties file. After entire upload is completed, mail notification is sent to the list of e-mail ids specified along with all the details of new build download and upload activities.

This process gets repeated at next time interval when Time Event Handler is triggered.

Back to top

ITDI Components

1. Timer EventHandler

Specify Crontab style schedule with it. The function ontimer() is invoked when timer occurs. This event handler clears previous execution logs and calls the Assembly Line "DetectBuild".

2. FTP Client Connector with Delta feature

FTP connection details are specified from the external properties file so no need to specify over here. LineReader parser is associated with FTP Client connector.

The option "Enable Delta" of FTP Client Connector is selected in order to enable the Delta feature. "Unique Attribute Name" is an entry attribute having unique value. As FTP Client Connector iterates build names which are unique and present in the attribute name "line" of conn object.

3. Script component "CheckBuild"

FTP Client connector passes unchanged or newly added entries those are iterated from the FTP server. These entries are nothing but the build names. This script validates whether user has specified any particular build name for download (which is called as startup build) or a new entry with operation "add" is passed by the FTP Client connector.

If a newer build is arrived and it is not present locally then it will be downloaded. This script component also performs some minor functions like validating parameter values provided in external properties file and creation of mail template, which will be used later to send mail. This script component calls download functions to download the specified build files.

After the download is complete, upload is performed depending upon the number of FTP Servers specified by user. After data transfer is complete, mail is sent to the list of mail-ids specified.

4. Upload and Download Functions

Downloading of build files is performed once the build is identified for download. Here is the demo script to download files by creating instance of com.ibm.di.protocols.FTPClient class.

The function shown is a demo code, which shows how to download files, but similar code is added in the script module "downloadFiles".

				
function demoDownloadfile(remoteFilePath, localFileName)
{
 var ftpObj=new Packages.com.ibm.di.protocols.FTPClient();
 try{
	task.logmsg("++++++++Connecting to FTPServer, attempt: " + iAttempts);
	var ftpServer=system.getExternalProperty("BUILD_ftpServer");
	var ftpPort=system.getExternalProperty("BUILD_ftpServerPort");
	var ftpUser=system.getExternalProperty("BUILD_ftpUser");
	var ftpPassword=system.getExternalProperty("BUILD_ftpUserPassword");
	ftpObj.connect(ftpServer,ftpPort);
	task.logmsg("++++++++Connected to FTP Server ++++++++");
	ftpObj.login(ftpUser,ftpPassword);
	task.logmsg("++++++++Logged in to FTP Server ++++++++");
	task.logmsg("++++++++Downloading file ++++++++");
	ftpObj.getFile(remoteFilePath, localFileName,true);
	task.logmsg("++++++++File Downloaded Successfully ++++++++");
 }  

 catch(ex){
		task.logmsg("####### FTP Error occurred: " +ex); 
 }	

}

Once the build files are downloaded, it can be uploaded at any location specified. For uploading a file using instance of com.ibm.di.protocols.FTPClient class, use method "putFile" whereas for downloading "getFile" method is used, rest of other methods are same.

				
ftpObj.putFile(localFilePath, remoteFileName,true);

5. SendMail

This module takes necessary SMTP input parameter values specified in external properties file and list of e-mail ids to send the mail. Once download of files is done and are uploaded at specified locations, an e-mail with all this activity is sent along with the activity log (Build.log) in details. If there was any problem with the connection or anything related to data transfer, this will be logged in Build.log file, which is sent as an attachment.

The function listed below shows how to send an e-mail from ITDI, but similar code is added in the script module "sendMail".

				
var mailSubject = "Test mail Subject"
var mailBody = " These are mail contents"
system.setJavaProperty("mail.smtp.host", smtpHostIP);
system.setJavaProperty("mail.smtp.port", smtpPort);
system.sendMail( mailFrom, mailTo, mailSubject,mailBody,"Attachment.txt");

Back to top

Using the solution

1. Setting the Configuration Parameters

Open the extern.props file and set the appropriate parameter values. Here are the details about each parameter:

2. Prerequisites

1. Install ITDI 6.0
2. Copy the GenericBuildDownloader.xml and extern.props file in the ITDI Solution directory.
3. Parameters from extern.props file are set to the appropriate values. For more details, refer to section "Setting the configuration Parameters" mentioned above.

3. Running the solution

Method 1

1. From command prompt, change directory to ITDI install directory.
2. Execute command: ibmdisrv -c GenericBuildDownloader.xml -t timerEh

Method 2

1. Launch the Config Editor (ibmditk) from ITDI's install directory.
2. Open the config file GenericBuildDownloader.xml with it(File=>Open).
3. Select the Event handler "timerEh" and select Run.

Execution Log

Below is a snap shot of the execution log that show a newer build is detected and further data transfer is done.

Back to top

Conclusion

As described with this article, ITDI provides a generic way of detecting arrival of new build as well as download and upload of files all using FTP. This solution can be used with any software product for daily build downloads and maintaining local build repository in-sync with the central build repository.

Back to top

Download

Information about download methods

Resources

• Learn more about IBM Tivoli Directory Integrator
  
  

About the author

Rate this article

Comments

Back to top

Table of contents

• Introduction
• How ITDI can help in this?
• Solution Architecture
• ITDI Components
• Using the solution
• Conclusion
• Download
• Resources
• About the author
• Comments

Next steps from IBM

• Try: Tivoli Directory Integrator
• Briefing: Introduction to Tivoli service management environment
• Buy: Tivoli Directory Integrator

Dig deeper into Tivoli on developerWorks

Discover knowledge paths

Skill-building guides for Linux, open source, cloud, Java, business analytics, and more.

Special offers