• 1 reply
  • Latest Post - ‏2010-08-03T10:01:21Z by SystemAdmin
64 Posts

Pinned topic Directory Monitoring

‏2010-07-28T01:11:51Z |
I'm setting up some directory monitoring. The monitored directory is always the same, but the files I want to move are all different and depending on the filename (or pattern match including an alias) then the files need to go to different destinations.

Is the only way around this to set up 1 monitor for each filename/wildcard match -> destination ??

I could end up with about 30 monitors on an agent. I guess I could set up dedicated agents on the nodes that require monitoring. But I would really like to understand how many system resources 30 agents would use? Also - if we have 20 monitored nodes - then thats potentially an awful lot of monitors to manage and maintain.

The other way I was considering was the match on everything and then read in the ${FileName} and do a lookup on the destination in a properties file from within an ant script.
Updated on 2010-08-03T10:01:21Z at 2010-08-03T10:01:21Z by SystemAdmin
  • SystemAdmin
    64 Posts

    Re: Directory Monitoring

    There are a variety of options, depending on how flexible your setup is.

    Multiple monitors is certainly a way of doing this. 30 Monitors running in 1 agent should not be a problem. Do ensure that the poll interval is not too low (e.g. don't set the poll interval down to 1 second or you will have 30 monitors trying to scan a directory (or recursively scan a directory if that's what you are doing) every second). Depending on the number/frequency of files that will end up being transferred you may want to look at how much memory is being used up by the agent running these monitors and may want to increase the maximum JVM heap size that the agent is allowed to use if it is getting close to the default maximum. A downside to multiple monitors like this is managing them. If you had to make a change to how the monitors behaved you would have to delete them all and recreate them with whatever changes were required as unfortunately FTE doesn't support editing monitors yet. If you do go down this route you may want to look at scripting the creation of the monitors so that they are easy to recreate should the need arise.
    Multiple agents is probably not a good idea. It would be much more effort to manage 1 monitor in each of 30 agents that it is to manage 30 monitors in one agent. A single agent should be capable of running 30 monitors.
    An Ant script that gets run from a single monitor is another good possibility. It will probably take longer to set up initially as you develop it and make it work, and you need the skills to develop the ant script, but once you have it it would be easier to maintain, and any changes you have to make in the future would be easier to deploy as you only have to change a single ant script/monitor instead of 30.
    If your system is flexible enough that you can change what directories files get generated into you may want to take a look at some of the advanced substitution variable options that are available on Monitors. If you can get the files generated into different directories, named for their destination agent/queue manager then you could write 1 single monitor without an ant task that could do the routing you desire.

    For example:


    A single Monitor recursively monitoring /source could use the substitution variable ${FilePath{token=2}} for the destination agent name, and ${FilePath{token=3}} for the destination queue manager name in the transfer task.

    With a bit of experimentation, and depending on your naming convention, you might even be able to use parts of the filename itself rather than directory names.


    ${FilePath{token=2}{separator=_}} = destination agent
    ${FilePath{token=3}{separator=_}} = destination QM

    This would allow you to add new destinations without having to create new monitors specifically looking for the new destination names, though both of these would require that the application generating the files be aware of agent and queue manager names which is not great.

    If the filename/path cannot contain agent names and there is going to be a required 'lookup' step to map from filename to destination agent/qm, then an ant script is possibly the best bet as it would have the least number of things that you need to configure.

    Hope this has been helpful, please do ask if you have any further questions.