Topic
  • 3 replies
  • Latest Post - ‏2013-03-04T14:45:12Z by hnasgaard
SystemAdmin
SystemAdmin
1245 Posts

Pinned topic Processing zip files.

‏2013-03-04T09:59:35Z |
I'm having a problem reading a compressed directory that contains zip files I need to process. Apparently, FileSource() operator can't still read the files inside the zip file. Please see my filesource operator I'm sure I'm missing something important.

stream<rstring line> FileGet = FileSource()
{
param
format : line;
file : "File.dat.zip";
compression : zlib;
}
  • hnasgaard
    hnasgaard
    200 Posts

    Re: Processing zip files.

    ‏2013-03-04T12:59:49Z  
    The FileSource can read a file that has compressed data, but not a zip file that is a compressed set of files. You'd have to unzip them first before feeding them to the FileSource. You could write something like DirectoryScan, that calls unzip on your file, then sends the file names to FileSource one at a time.
  • SystemAdmin
    SystemAdmin
    1245 Posts

    Re: Processing zip files.

    ‏2013-03-04T14:21:43Z  
    • hnasgaard
    • ‏2013-03-04T12:59:49Z
    The FileSource can read a file that has compressed data, but not a zip file that is a compressed set of files. You'd have to unzip them first before feeding them to the FileSource. You could write something like DirectoryScan, that calls unzip on your file, then sends the file names to FileSource one at a time.
    Basically I can unzip files using DirectoryScan() operator? But as I can see here, there are no option to unzip files using DirectoryScan().
    http://publib.boulder.ibm.com/infocenter/streams/v2r0/index.jsp

    Or I should really decompress the files before I can process them?
  • hnasgaard
    hnasgaard
    200 Posts

    Re: Processing zip files.

    ‏2013-03-04T14:45:12Z  
    Basically I can unzip files using DirectoryScan() operator? But as I can see here, there are no option to unzip files using DirectoryScan().
    http://publib.boulder.ibm.com/infocenter/streams/v2r0/index.jsp

    Or I should really decompress the files before I can process them?
    DirectoryScan does not have an option to unzip a zip file of files. It can scan the directory after the files are unzipped. You might want to unzip them one directory and move them into the directory to be scanned. I'm not sure how it would work if DirectoryScan was scanning while files were being unzipped.