Topic
  • 5 replies
  • Latest Post - ‏2013-02-06T19:00:30Z by Kevin_Foster
Kevin_Foster
Kevin_Foster
98 Posts

Pinned topic Timing question for DirectoryScan operator

‏2013-02-04T22:46:24Z |
I've got a question about the timing of scans by the DirectoryScan operator.

Suppose I have a file open that I'm still writing when the DirectoryScan operator does a periodic scan. I'm assuming that the file name won't be placed on the output stream as the file is still being written?

Now suppose that I close the file. Will the file name be discovered by DirectoryScan and placed on the output stream the next time that the folder is scanned?

The scenario is that we're copying files to a folder that's watched by a DirectoryScan() operator, but periodically we're getting orphaned files. Renaming a file "cures" the problem for that particular file, so doesn't seem to be a problem with the file itself...

-Kevin
  • hnasgaard
    hnasgaard
    200 Posts

    Re: Timing question for DirectoryScan operator

    ‏2013-02-04T23:00:02Z  
    The intended behavior is as you describe. DS should pick up the modified file once ctime is updated. Can you tell me if you are using NFS or GPFS? I've had a report of a similar issue using GPFS. Does touch fix the problem? I'm not sure how a file could get lost.
  • Kevin_Foster
    Kevin_Foster
    98 Posts

    Re: Timing question for DirectoryScan operator

    ‏2013-02-05T07:21:07Z  
    Customer is using SAN storage attached with optic cable, connected on the Streams machine as "ext3" file system.

    Yes, "touch" also works i.e. seems to make the file visible to the DirectoryScan.

    Sorry, forgot to mention that this is Streams 2.0.0.4.

    -Kevin
  • hnasgaard
    hnasgaard
    200 Posts

    Re: Timing question for DirectoryScan operator

    ‏2013-02-05T12:56:48Z  
    Customer is using SAN storage attached with optic cable, connected on the Streams machine as "ext3" file system.

    Yes, "touch" also works i.e. seems to make the file visible to the DirectoryScan.

    Sorry, forgot to mention that this is Streams 2.0.0.4.

    -Kevin
    There seems to be a fair bit of debug info generated by the DirectoryScan operator. Would it be possible to run the application with tracing on for that operator and collect the application logs. Is the problem easily reproduced or is it a rare and unpredictable event?
  • bmwilli
    bmwilli
    41 Posts

    Re: Timing question for DirectoryScan operator

    ‏2013-02-06T17:55:58Z  
    Kevin,

    We saw the same behavior your experiencing. We had re-factor our file processing before streams to use a file move into the directory being watched rather than a copy.

    Would be nice if the DirectoryScan did not send the filename multiple times.

    Brian
  • Kevin_Foster
    Kevin_Foster
    98 Posts

    Re: Timing question for DirectoryScan operator

    ‏2013-02-06T19:00:30Z  
    This certainly could be file system related as it's not typical attached storage...

    But since this is Streams 2.0.0.4 and we have no easy way to reproduce this where we can turn on tracing, we're decided to take the easy route of first writing the data to disk and then moving the file (on that same disk) to make the file visible to DirectoryScan.

    Thanks for the replies everyone.

    -Kevin