Topic
  • 1 reply
  • Latest Post - ‏2015-07-12T13:01:22Z by kobil
kobil
kobil
2 Posts

Pinned topic Deleted part files during transfer

‏2015-07-07T08:08:38Z | fte mqft wmqfte

Hi all,

 

I have a scenario where I got tow agents transferring files, the files can be at the range of 1K up to 1G,

the files are being transfer to a destination directory on the destination agent host, on that directory we got a legacy system which just pull out any new file which arrived  into that directory.

The problem is that this legacy system also taking the .part files, created during the transfer, and then we get this error and the transfer failed:

BFGIO0083E: A file open for write failed due to a Java IOException with message text D:\dir\a.txt.part (The process cannot access the file because it is being used by another process.)

 

The destination agent is running on Windows Server 2003 R2, and we used the "enableMandatoryLocking=true" on it's properties file.

 

We want to know if anyone have any recommendation on how to workaround this situation,

out current thoughts are:

1. use the "doNotUseTempOutputFile=true" property, but we worry about the large files, and  we are not sure if the destination system won't take the files before the transfer complete.

2. using Interim folder on the destination host, that the files where first arrived to that folder and then another transfer move them to the real destination folder. by that we are hopping to have less part files, and minimize the time when the destination system can take the uncompleted files.

3. separate the agent, so we will have agent for large files and agent for small files, and the preventing the small files agent to create part file (by the doNotUseTempOutputFile property), because the frequent of small files it's much greater than small files, so it minimize the failures, but won't prevent them at all.

 

I will glad to hear any other thoughts based on your experience, 

Thanks,

Kobi.

 

Updated on 2015-07-07T09:47:36Z at 2015-07-07T09:47:36Z by kobil
  • kobil
    kobil
    2 Posts

    Re: Deleted part files during transfer

    ‏2015-07-12T13:01:22Z  

    An update:

    Since we didn't found any good enough solution at this time, we decide to use a program-call at the post-destination phase, which call a batch file that doing move command.

    The full 'solution' is:

    1. move the file, using monitor, to a temporary directory.

    2. At the post-destination step, call batch file (need to use the commandPath property), with the ${FileName} as parameter.

    3. in the batch file having the command: move /Y C:\temp_dir\%1 C:\dest_dir  (%1 will have our file name).

     

    Since the move command is fast enough, no files are stolen before fully writing to the destination directory. 

    In case, for any reason, the move command failed, the transfer will failed with the reason from the move command output.

    This is a temporary solution, and we will be glad to hear any other, more elegant, solutions.

     

    Thanks.