Configuring an MFT resource monitor to avoid overloading an agent

You can configure the property and parameter values of a Managed File Transfer resource monitor to reduce the load on an agent. Reducing the load on the agent improves the performance of that agent. There are several settings you can use, and you may need to use trial and error to find the best settings for your system configuration.

About this task

When a resource monitor polls a directory or a queue, the agent completes the following stages:
  • Find all the files that match a trigger pattern (for example, all the *.txt files in the directory). Or find all complete groups of messages on the queue.
  • Determine which files are new or changed, or determine which groups are new on the queue.
  • Initiate transfers for the files or groups that match the criteria in the two previous stages.
  • Add to the list of files and groups already transferred so they are not transferred again until they change.
For a directory monitor, the more files in the source directory and the broader the triggering pattern, the bigger the list of files the agent has to parse and compare against the list of files already transferred.

For a queue monitor, the more groups on the queue the bigger the list of groups the agent has to compare against the list of groups already transferred.

Procedure

  • Use agent property monitorMaxResourcesInPoll to set the maximum number of files or groups the agent includes on each poll.

    Using this parameter limits the number of transfers in a polling interval. It also means that the agent has less parsing to do before initiating a transfer for that number of files or groups. The next time the directory monitor or queue monitor polls, the agent includes the next set of files or groups.

  • When creating a directory monitor, ensure that the transfer definition you configure has a source disposition of delete.

    Setting this disposition means that when the file transfer completes it is removed from the monitored directory and the agent no longer keeps it on its internal list.

  • When creating a directory monitor, use the -rl parameter in the fteCreateMonitor command to limit the number of levels of the directory the agent has to recurse through.

    Using this parameter means that lower-level directories are not scanned unnecessarily.

What to do next

The process of resource monitor polling consumes agent resources. Increasing the polling interval of a monitor reduces the load placed on the agent. However, the setting of the polling interval must be balanced against generating too many transfers per polling interval. Consider the following when you set the polling interval for a resource monitor:
  • How quickly you need a transfer to be initiated after a file is placed in a directory, or a group on a queue.
  • The rate which files are placed into a directory, or groups onto a queue.
  • The maximum transfer rate of the agent. The agent must be able to handle all the transfers that a monitor generates.

The polling interval is specified when the resource monitor is created with the fteCreateMonitor command by specifying the -pi (polling interval) and -pu (polling interval units) parameters. You may need to experiment to determine the best settings for your configuration.

An option to improve the stability of highly loaded agents that run resource monitors, is to reduce the agent property value of maxSourceTransfers. With this option the agent splits its processing time between the resource monitor and transferring files. The higher the value of agent property maxSourceTransfers, the more processing time is consumed by transferring files and less is available for the resource monitor. If you reduce the value of agent property maxSourceTransfers, the agent does fewer transfers in parallel, but it should have enough processing time to poll its resource monitors. If you lower the value of this agent property you should consider increasing the value of agent property maxQueuedTransfers because the number of queued transfers may increase.

If after optimizing your monitor you find that some transfers enter recovery, consider increasing an agent timeout value. The heavy load placed on the agent, may mean that the transfers timeout when negotiating the start of the transfer with the destination agent. This timeout causes the transfer to go into recovery and delays the completion of the transfer. The agent property maxTransferNegotiationTime specifies the time the source agent waits for a response from the destination agent. If this time is exceeded the transfer goes into recovery. The default value of this property is 30000 milliseconds (30 seconds). Increasing the value of the property, for example to 300000 Milliseconds (5 minutes), may allow the transfers to continue without timing out and avoid going into recovery.