Contents


Simplify file transfer from the enterprise to the cloud with IBM MQ Advanced

Migrate multiple file transfer applications to a single cloud solution using the managed file transfer capability

Comments

The accumulation of multiple file transfer solutions—legacy or proprietary—in an organization's environment adds to maintenance costs and recurring expenses. As organizations undergo transformation to the cloud, replacing these file transfer solutions becomes complex, particularly when an organization chooses a single file transfer solution provider.

If you are considering moving your multiple file transfer applications to a cloud platform, you must adopt the predefined standard software stack that is provided by the cloud vendor. Regardless of your situation, this migration from multiple file transfer applications to a single solution can pose several challenges. For example, you might have a different solution approach for file transfers that are used currently, or you might have an application that must use both the existing file transfer method and the newly adopted method. You can avoid such challenges if your approach and design are well thought out from the start of your project.

This article highlights a single solution to the most commonly encountered migration scenarios by using the managed file transfer capability of IBM® MQ Advanced. MQ Advanced is a secure and convenient method to manage file transfers on the cloud from business applications in an enterprise. This article explains how you can migrate to MQ Advanced from commonly used file transfer protocols in various scenarios. Because the focus is on a single solution, this article can serve as a handy reference guide to both developers and architects.

Try IBM MQStart building in Bluemix

Preliminary factors to consider

Before you move to any single-vendor file transfer solution, you must be aware of several factors, such as the following examples, that can affect the migration from a multivendor file transfer solution:

  • The existing file transfer mechanism that is used, such as File Transfer Protocol (FTP), Secure FTP (SFTP), XCOM, and Axway File Broker (XFB)
  • The platform on which the application runs, such as Microsoft® Windows® or Linux®
  • The gateway that is used for external communication
  • The integration between the application and the existing file transfer mechanism (tightly or loosely coupled)
  • The scheduling of the file transfer and dependency on other jobs or workflows

Depending on the factors in your environment, the migration approach will vary. This article addresses some of these factors and how they affect the solution.

Architectural decisions and assumptions

The file transfer migration scenarios in this article follow these assumptions and architectural decisions:

  • The target file transfer mechanism is using MQ Advanced.
  • The applications have a local MQ Advanced agent installed on the node and communicate by using a shared MQ Advanced concentrator (client/server model).
  • Applications that use their own MQ queue manager for file transfer are converted to this client/server model. They no longer have a dedicated queue manager for file transfer.
  • You use configuration scripts to specify the parameters to be used for the file transfer. Each interface has its own MQ Advanced File Transfer ID and a related script with a source, target, and concentrator. The source folder/file and target folder/file are predefined (to check access), but you can define parameters to overwrite them.
  • Each application has its own unique MQ Advanced agent name. An application that already uses MQ Advanced gets a new agent name on the target. If the application uses MQ Advanced for file transfers, the MQ Advanced Transfer ID remains unchanged.
  • After a successful transfer, post-processing scripts are used on the source node, for example, to rename or delete a source file. These scripts can also be used on the destination node, for example, to move, rename, or change the permissions of the target file.
  • The application itself can trigger the file transfer, or you can use any scheduler software, such as Windows Scheduler and UNIX® Cron jobs.
  • An initiator can push and pull a file. You need a separate script to upload and download a file (source and target switched).
  • You can start multiple file transfer scripts simultaneously.

The following figure shows an overview of a loosely coupled file transfer between applications. The file transfer is a push (upload) from A to B (initiated by application A (App A)). Alternatively, application A can also initiate a pull (download) from application B (App B) to application A. And, application B can also initiate a file transfer as a push (upload) from application B to application A and a pull (download) from application A to application B. Each file transfer connection has its own file transfer ID. You can use an initialization (INI) file that contains the connection details for the file transfer.

Loosely coupled file transfers between                     applications
Loosely coupled file transfers between applications

The source and target agent names are predefined in the INI file, but you can specify the files and folders when you start the transfer. To start the file transfer, you enter the following command:
start_transfer <transferID> <sourcePath> [<sourceFile>] [<destFile>] [<destPath>]

Required application changes

When you plan to migrate any existing file transfer mechanism to MQ Advanced, you must make the following changes to the source and target applications that currently use a different file transfer method:

  • Use file transfer scripts to start uploading or downloading the files.
  • Decide whether the transfer is "scheduled" or "application started." Either the scheduling tool or the application must start the file transfer script.
  • Resolve any MQ Advanced file transfer script return codes.
  • Call transfers by using the "wait for completion" or "fire-and-forget" option.
  • Remove tight coupling between the application and the file transfer mechanism.
  • Remove any non-supported functions in MQ Advanced or direct file transfer API calls.
  • Design or implement alternatives for functions that MQ Advanced does not support.
  • Call or schedule the file transfer scripts with the appropriate parameters of Source files and Target files.
  • Use wildcards to transfer multiple files at one time.
  • Address the impact of the migration on mainframe job schedules or workflows.
  • Optional: Create PostScripts.

Common migration scenarios

The following sections highlight three common file transfer migration scenarios where the target is IBM MQ Advanced. The scenarios depict two hypothetical business applications (referred to as App A and App B in the figures) that use different file transfer mechanisms that are not ready for migration to the MQ Advanced together. For these scenarios, the common assumption is that the MQ Advanced concentrator is available and set-up on a Linux environment.

Scenario 1: Migrating from FTP, SFTP, SSH, or XFB on Linux or Windows

The most common scenario entails applications that use any of the popular file transfer methods, such as FTP, SFTP, Secure Shell (SSH), or XFB. The source and target operating system can be either Windows or Linux.

In this scenario, the existing application set (application A and application B) uses FTP as the current file transfer method. Application A must move to the new environment with MQ Advanced. Application B will continue to run on the current platform with FTP. The existing file transfer through FTP between these two applications is encrypted.

As shown in the existing environment in the following figure, applications A and B use FTP to transfer files to each other. A scheduler is used to trigger the file transfer between the two applications. The file to be transferred is on a local disk. Both the source and target environments are either Windows or Linux.

Existing state of scenario 1
Existing state of scenario 1

To migrate from FTP, SFTP, SSH, or XFB on Linux or Windows, you make the following changes:

  1. Move application A to the new environment.
  2. Install the MQ Advanced agent on the servers that host applications A and B.
  3. Create transfer scripts for application A for all transfers.
  4. Create transfer scripts for application B for transfers to application A.
  5. Make any necessary code changes that are required for applications A and B to remove tight coupling.
  6. Make necessary code changes that are required for applications A and B to use the MQ Advanced concentrator for initiating the file transfer.

Applications A and B use the MQ Advanced concentrator in the new environment to initiate the transfers. Application B can still use FTP for transfers to other applications (other than application A). Otherwise, you can remove the FTP agent from this server.

After these changes are made, you arrive at the state that is shown in the following figure.

Migrated state of scenario 1
Migrated state of scenario 1

Scenario 2: Migrating from XCOM on Linux or Windows to a mainframe

In this scenario, applications A and B use XCOM, a proprietary file transfer method, as their current file transfer solution. Application A runs on a Linux or Windows environment, and application B runs on a mainframe. Application A must move to the new environment. Application B runs on the mainframe and is not migrated to the new environment. The file transfers are usually scheduled because some of them are part of a complicated or integrated job schedule or workflow on the mainframe. The following figure illustrates the current state of this scenario.

Existing state of scenario 2
Existing state of scenario 2

To migrate from XCOM on Linux or Windows to a mainframe, you must make the following changes:

  1. Move application A to the new environment.
  2. Install the MQ Advanced agent on the servers that host applications A and B.
  3. Create new transfer scripts for all transfers from application A to application B.
  4. Create new transfer scripts for all transfers from application B to application A.
  5. Make any necessary code changes that are required for applications A and B to remove tight coupling.
  6. Update the job schedules and workflow for applications A and B when applicable.
  7. Make necessary code changes that are required for applications A and B to use the MQ Advanced concentrator for initiating the file transfer.

Application B on the mainframe uses the MQ Advanced agent to initiate transfers to application A.

Application B can still use XCOM for transfers to other applications, if needed. Otherwise, you can remove the XCOM agent from the server.

After these changes are made, you arrive at the state that is shown in the following figure.

Migrated state of scenario 2
Migrated state of scenario 2

Scenario 3: Migrating from Connect:Direct on Linux to Oracle Solaris

This scenario involves a Connect:Direct (C:D) gateway as the file transfer mechanism on Linux, with Oracle Solaris as the target operating system. Application A communicates with application B through the C:D gateway. Application A is migrated to use MQ Advanced on the new environment. Application B is an application behind the C:D gateway.

Application A is using SFTP or SSH to communicate with the C:D gateway. The C:D gateway does not have an MQ Advanced agent. The following figure illustrates the current state of this scenario.

Existing state of scenario 3
Existing state of scenario 3

To migrate from Connect:Direct on Linux to the Oracle Solaris operating system, you make the following changes:

  1. Move application A to the cloud.
  2. Install the MQ Advanced agent for application A.
  3. Install the MQ Advanced agent for application B.
  4. Create transfer scripts for all transfers from application A to application B.
  5. Create transfer scripts for all transfers from application B to application A.
  6. Make any necessary code changes that are required for applications A and B to remove tight coupling.
  7. Configure the C:D gateway to support MQ Advanced.
  8. Update the transfer schedules of applications A and B when applicable.

Therefore, applications A and B use an MQ Advanced Concentrator in the new environment to initiate transfers. Application B can still use the C:D gateway for transfers to other applications. Otherwise, remove the C:D agent.

After these changes are made, you arrive at the state that is shown in the following figure.

Migrated state of scenario 3
Migrated state of scenario 3

Conclusion

Moving from multiple file transfer solutions to a single file transfer solution is among the primary technical challenges that enterprises face in any modernization or cloud transformation project. This article showed how you can adopt a single file transfer solution, by using IBM MQ Advanced as an example, from multiple file transfer solutions. It outlined preliminary factors, architectural decisions, and assumptions that you must consider before you undertake this type of migration project. Then, it demonstrated three common scenarios for migrating to an IBM MQ Advanced solution from commonly used file transfer protocols.

Acknowledgements

The authors thank Somen Mandal for reviewing this article.


Downloadable resources


Related topics


Comments

Sign in or register to add and subscribe to comments.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Middleware, Cloud computing
ArticleID=1042377
ArticleTitle=Simplify file transfer from the enterprise to the cloud with IBM MQ Advanced
publish-date=04302017