Topic
3 replies Latest Post - ‏2013-07-16T14:04:35Z by AaronHoffer
AaronHoffer
AaronHoffer
9 Posts
ACCEPTED ANSWER

Pinned topic Import CSV records as lists

‏2013-07-09T01:12:52Z |

I want to import data from a CSV file. The CSV is a series of images, stored as 0-255 gray values. Each record is 1 image. The first column is a label (also an integer). I hoped to represent each record as a tuple:

composite Main
{
    type Img = tuple<uint32 label, list<uint32> pixels>;
  
    graph
        stream<listSchema2d> Images = FileSource()
        {
        param
            file: "train.csv";
            format: csv;
        }

    //FileSink here

}

However, this code causes a runtime error. The first two errors are:

ERROR #splapptrc,J[0],P[0],Images,spl_operator M[Images.cpp:processOneFile:155]  - CDISR5028E: The read of the tuple<list<uint32> pixels> type failed for the following tuple number: 1.

ERROR #splapptrc,J[0],P[0],Images,spl_operator M[Images.cpp:processOneFile:172]  - CDISR5155E: An error occurred during the processing of tuple 1 in the train.csv file. The error is: The read of the tuple<list<uint32> pixels> type failed for the following tuple number: 1.

I am new to streams. I cannot figure out how to drill into these errors to discover what is going wrong.

Can anyone point me in the right direction?

  • Jason T
    Jason T
    31 Posts
    ACCEPTED ANSWER

    Re: Import CSV records as lists

    ‏2013-07-10T21:48:19Z  in response to AaronHoffer

    Hello,

    Can you please attach a small subset of your data?

    Thank You,

    Jason

  • BruceGlassford
    BruceGlassford
    71 Posts
    ACCEPTED ANSWER

    Re: Import CSV records as lists

    ‏2013-07-16T09:23:08Z  in response to AaronHoffer

    When reading in collections (tuples, lists, maps, sets) from a csv file, the input file must be formatted the same way it would be by a FileSink - for lists, this means the list would be formatted with enclosing { and } around the set of elements.  You would probably be better off reading it in in line mode and parsing the tuples from the lines using tokenise.

    Hope this helps.

     

    ... Bruce

    • AaronHoffer
      AaronHoffer
      9 Posts
      ACCEPTED ANSWER

      Re: Import CSV records as lists

      ‏2013-07-16T14:04:35Z  in response to BruceGlassford

      Thanks Bruce. That did the trick.

      Also, I took your advice and read in the data as lines and then parsed out the fields.

      Regards, Aaron