Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
5 replies Latest Post - ‏2013-02-06T19:00:30Z by Kevin_Foster
Kevin_Foster
Kevin_Foster
98 Posts
ACCEPTED ANSWER

Pinned topic Timing question for DirectoryScan operator

‏2013-02-04T22:46:24Z |
I've got a question about the timing of scans by the DirectoryScan operator.

Suppose I have a file open that I'm still writing when the DirectoryScan operator does a periodic scan. I'm assuming that the file name won't be placed on the output stream as the file is still being written?

Now suppose that I close the file. Will the file name be discovered by DirectoryScan and placed on the output stream the next time that the folder is scanned?

The scenario is that we're copying files to a folder that's watched by a DirectoryScan() operator, but periodically we're getting orphaned files. Renaming a file "cures" the problem for that particular file, so doesn't seem to be a problem with the file itself...

-Kevin
  • hnasgaard
    hnasgaard
    200 Posts
    ACCEPTED ANSWER

    Re: Timing question for DirectoryScan operator

    ‏2013-02-04T23:00:02Z  in response to Kevin_Foster
    The intended behavior is as you describe. DS should pick up the modified file once ctime is updated. Can you tell me if you are using NFS or GPFS? I've had a report of a similar issue using GPFS. Does touch fix the problem? I'm not sure how a file could get lost.
  • Kevin_Foster
    Kevin_Foster
    98 Posts
    ACCEPTED ANSWER

    Re: Timing question for DirectoryScan operator

    ‏2013-02-05T07:21:07Z  in response to Kevin_Foster
    Customer is using SAN storage attached with optic cable, connected on the Streams machine as "ext3" file system.

    Yes, "touch" also works i.e. seems to make the file visible to the DirectoryScan.

    Sorry, forgot to mention that this is Streams 2.0.0.4.

    -Kevin
    • hnasgaard
      hnasgaard
      200 Posts
      ACCEPTED ANSWER

      Re: Timing question for DirectoryScan operator

      ‏2013-02-05T12:56:48Z  in response to Kevin_Foster
      There seems to be a fair bit of debug info generated by the DirectoryScan operator. Would it be possible to run the application with tracing on for that operator and collect the application logs. Is the problem easily reproduced or is it a rare and unpredictable event?
  • bmwilli
    bmwilli
    41 Posts
    ACCEPTED ANSWER

    Re: Timing question for DirectoryScan operator

    ‏2013-02-06T17:55:58Z  in response to Kevin_Foster
    Kevin,

    We saw the same behavior your experiencing. We had re-factor our file processing before streams to use a file move into the directory being watched rather than a copy.

    Would be nice if the DirectoryScan did not send the filename multiple times.

    Brian
  • Kevin_Foster
    Kevin_Foster
    98 Posts
    ACCEPTED ANSWER

    Re: Timing question for DirectoryScan operator

    ‏2013-02-06T19:00:30Z  in response to Kevin_Foster
    This certainly could be file system related as it's not typical attached storage...

    But since this is Streams 2.0.0.4 and we have no easy way to reproduce this where we can turn on tracing, we're decided to take the easy route of first writing the data to disk and then moving the file (on that same disk) to make the file visible to DirectoryScan.

    Thanks for the replies everyone.

    -Kevin