Using resource monitors effectively
Matthew Whitehead 120000DAG9 Visits (3447)
Plenty of people have mentioned in emails or in person that it's easy to grind an agent to a halt if you define a resource monitor that matches large numbers of files (typically 50k+) in a directory.
A couple of years ago we published a tech note to try and help people get round some of the problems. You can read the current tech note here
The tech note describes one way of trying to reduce performance degradation (increasing the maxT
When a resource monitor polls a directory, it has to go through the following steps:
It's possible to tune each of the above steps so that the performance and stability of the agent can be optimised. For each of the above steps I've described some of things you can do to improve agent behaviour.
Step 1 - Finding all the files that match the trigger pattern
This might sound fairly obvious but the more files you put in your source directory and the broader your triggering pattern, the bigger the list of files the agent will have to parse and compare against the list of files we already transferred. To reduce the time the agent takes on each poll of the resource monitor, consider the following options:
Step 2 - Determining which files are new or have changed
As mentioned above, you should make sure you are using a source disposition of delete. If you use source disposition of 'leave' then FTE has to permanently keep a record of every file. Any future polls of the resource monitor will have to compare every file it finds with the internal list which FTE maintains. If that list contains all the 100,000 files in your monitored directory, it will take the agent a long time to run down that list and make sure the files we've found in the directory haven't already been transferred.
Step 3 - Initiating a transfer for matched files
In early versions of FTE, every file that a resource monitor matched would cause a new transfer to be submitted. This meant that if you had 1000 .txt files in the monitored directory, 1000 new transfers would be submitted to the agent. In version 7.0.3 a new option was introduced called batch size (the -bs flag on fteCreateMonitor). By setting the batch size of a resource monitor (say, to 100) the resource monitor would create a new transfer for every 100 files it matched. So with 1000 .txt files you'd only get 10 new transfers being started. Since every transfer incurs some overhead relating to starting the transfer, publishing started/finished event messages etc. it can be more efficient to submit fewer transfers of a larger number of files, rather than lots of transfers all of just one file.
Step 4 - Updating the list of transferred files
As mentioned already, the bigger the list of files FTE needs to maintain, the longer it takes to parse that list on every poll of the resource monitor. In order to minimise the size of the list FTE maintains, consider the recommendations above such as using source disposition of delete, or using a source transfer exit to move files from the monitored directory to an archive directory once they've been tran