Topic
  • 4 replies
  • Latest Post - ‏2005-12-27T14:58:07Z by SystemAdmin
SystemAdmin
SystemAdmin
332 Posts

Pinned topic Hash file generates single file for each record in data fold

‏2005-06-05T04:18:21Z |
Hi all,

I have DataStage 7.5 in Windows 2003 server .. the problem is:

I have a jobs that writes data to hashed file, hashed file format is the defaults of Type 30 (Dynamic). One job deletes then create the file, then some other jobs append records to the same file. One day I found that the data folder for this file instead of containing 2 files (data and metadata), it contains a single file for each record with file name equal to record key. so if the file should hold 5 million records , its data folder contains 5 million files !!!

I tried to delete all files manually (this required more than 4 hours) then recompile the job and run it again for 100 records only, again it generated 100 files !!! It only work correct again if file path changes which is not always possible.

I need a solution to correct this up normal job behavior without changing the file path.

Also I need a fast way to delete 5 million files in a single folder.

Thanks.
Updated on 2005-12-27T14:58:07Z at 2005-12-27T14:58:07Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    332 Posts

    Re: Hash file generates single file for each record in data

    ‏2005-06-05T05:03:28Z  
    You don't have a hashed file. A dynamic hashed file is a folder containing exactly three files, called DATA.30, OVER.30 and .Type30.

    If someone has deleted any of these, or added any other file to the folder, then it can not be opened as a hashed file.

    What you need to do is to delete every file except the three named above, or delete the folder itself and its file dictionary (a file called D_hashedfile) and, if the hashed file was created in Account (rather than in Directory), its VOC entry, then re-create the hashed file.
  • SystemAdmin
    SystemAdmin
    332 Posts

    Re: Hash file generates single file for each record in data

    ‏2005-06-05T10:15:46Z  
    There's really no need to post the exact same thing in multiple forums.
  • SystemAdmin
    SystemAdmin
    332 Posts

    Re: Hash file generates single file for each record in data

    ‏2005-12-27T14:07:10Z  
    just to throw a little info in, I had the same problem, and I think it came from having one process reading the hash file while another was trying to re-create it. i'm guessing a read-lock kept the delete from working and then the folder/files became out of synch. deleting the folder and D_ file would resolve it on the next run....
  • SystemAdmin
    SystemAdmin
    332 Posts

    Re: Hash file generates single file for each record in data

    ‏2005-12-27T14:58:07Z  
    This can also happen if you have the "Create File" and "Clear File Before Writing" options selected at the same time. When you run the job, it will delete part of the .30 files, but leave some intact because they are locked.

    Brett