MFT resource monitoring concepts

An overview of the key concepts of the Managed File Transfer resource monitoring feature.

Monitors

You create a resource monitor by using the fteCreateMonitor command, which creates and starts a new resource monitor from the command line. The resource monitor is associated with a Managed File Transfer agent, and is only active when that agent is started. When the monitoring agent stops, so does the monitor. If the agent is already started when the monitor is created, the monitor starts immediately. The monitoring agent must also be the source agent of the task that is initiated by the monitor.

Monitor names must be unique within their agent. The monitor name must be a minimum of one character in length and must not contain asterisk (*), percent (%) or question mark (?) characters. The case of supplied monitor names is ignored and the monitor name is converted to uppercase. If you try to create a monitor with a name that is already present, the request is ignored and the attempt is logged to the monitor log topic.

Note: You cannot create a resource monitor with a task definition that contains scheduled transfers.

There is no restriction on the number of monitors that can be created on an agent, and all run with the same priority. Consider the implications of overlapping monitored resources, conflicting trigger conditions and how frequently the resources are polled.

Overlapping resource monitors can cause:

Possible contention on the source location/items.
Possible duplicate transfer requests for same source items.
Unexpected errors or failures for transfers due to conflicts of source items.

If multiple monitors scan the same location and can trigger on the same items, you can potentially end up with the problem of two different monitors submitting managed transfer requests for the same item.

Resource monitors look at the contents of resources after every poll interval period. The contents of the resource are compared with the trigger conditions and if those conditions are satisfied, the task associated with the monitor is called.

The task is started asynchronously. If there is a condition match, and the task is started, the monitor continues to poll for further changes to the resource contents. So for example, if a match occurred because a file called reports.go arrived in a monitored directory, the task would be started once. At the next poll interval, even if the file still exists, the task is not started again. However, if the file is deleted and then placed in the directory again, or the file is updated (such that the last modified date attribute is changed), the next trigger condition check causes the task to be called again.

Resources

Resource monitors in Managed File Transfer can poll the contents of the following two types of resource:

Directories or nested directory structures

A common scenario is to monitor a directory for the presence of a trigger file. An external application might be processing multiple files and placing them in a known source directory. When the application has completed its processing, it indicates that the files are ready to be transferred, or otherwise acted upon, by placing a trigger file into a monitored location. The trigger file can be detected by a Managed File Transfer resource monitor and the transfer of those files from the source directory to another Managed File Transfer Agent is initiated.

By default, the specified directory is monitored. To also examine sub-directories set the recursion level in the fteCreateTransfer command.

Two examples of monitoring a directory are as follows:

Monitor for a trigger file (for example trigger.file) and then transfer a wildcard (for example, *.zip).
Monitor for *.zip and then transfer ${FilePath} (for example, the file that triggered the transfer). For more information about variable substitution, see Customizing MFT resource monitor tasks with variable substitution.

Note: Do not create a monitor that monitors for *.zip, and then transfers *.zip. The monitor tries to start a transfer of *.zip for every .zip file on your system. That is, the monitor generates * number of transfers for *.zip.

For an example of creating a resource monitor to monitor a directory, see Monitoring a directory and using variable substitution.

IBM® MQ queues: An example of monitoring a queue is that an external application might be generating messages and placing them on a known queue with the same group ID. When the application has completed putting messages on the queue, it indicates that the group is complete. The complete group of messages can be detected by a Managed File Transfer resource monitor and the transfer of the group of messages from the source queue to a file is initiated. For an example of creating a resource monitor to monitor a queue, see Example: Configuring an MFT resource.
Note: You can specify only one monitor per queue. If you specify more than one monitor to poll an IBM MQ queue, unpredictable behavior occurs.

Monitoring data sets is not supported.

Trigger conditions

The condition is met when the resource contains a value that matches some other string or pattern. Conditions can be one of the following:

Match on file name (pattern)
No match on file name (pattern)
File size
Match if file size remains the same for a number of polls

File name matching can be expressed as:

Exact string match
Simple wildcard match as described in Using wildcard characters with MFT
Regular expression match

File names can also be excluded from file name matching by using a wildcard or Java regular expression that identifies file names that are never matched.

When a matching file is detected, its last modified time stamp is retained. If subsequent polls detect that the file has been changed, the trigger condition is satisfied again, and the task is started. If the condition is to detect when a file does not exist, if no file in the monitored directory matches the file name pattern, the task is started. If a file is then added to the directory that does match the file name pattern, the task is only started if the file is then deleted.

Tasks

Managed File Transfer supports the following two types of task that you can configure to be started by resource monitors:

File transfer
Command

File transfer tasks are defined in the same way as any other file transfer. A useful way to generate the task XML required by a monitor is to run the fteCreateTransfer command with the -gt parameter. This command generates a task definition as an XML document, including the transfer specification. You then pass the name of the task XML document as the value for the -mt parameter on the fteCreateMonitor command. When the fteCreateMonitor is run, it reads the task XML document. After the fteCreateMonitor is run, any changes that are made to the task XML file are not used by the monitor.

Command tasks can run Ant scripts, call executable programs, or run JCL jobs. For more information, see Configuring monitor tasks to invoke commands and scripts.

When using a file transfer task, you can select how many trigger conditions are batched into a task. The default is for one trigger condition to start one task. You can run the fteCreateMonitor command with the -bs option to select the number of trigger conditions that are batched together into one task.

Backing up and restoring resource monitors

You can back up the resource monitors that you have already defined so that you can re-use them in the future. There are various options that you can use as follows:

Use the fteCreateMonitor command with the -ox parameter to export a resource monitor configuration to an XML file, and with the -ix parameter to restore a resource monitor by importing the resource monitor configuration from an XML file.
Use the fteListMonitors command with the -ox to export the definition for a single resource monitor to an XML file.
From IBM MQ 9.0.5, use the fteListMonitors command with the -od to export multiple resource monitor definitions to a specified directory. Each resource monitor definition is saved to separate XML file. You can also use the -od option to export a single resource monitor definition to a specified directory.

For more information, see Backing up and restoring MFT resource monitors.

Resource monitor logging

From IBM MQ 9.0.3, Managed File Transfer includes resource monitor logging. For more information, see Logging MFT resource monitors.