Variable File Node

You can use Variable File nodes to read data from free-field text files (files whose records contain a constant number of fields but a varied number of characters), also known as delimited text files. This type of node is also useful for files with fixed-length header text and certain types of annotations. Records are read one at a time and passed through the stream until the entire file is read.

Note regarding reading in geospatial data

If the node contains geospatial data, and the node was created as an export from a flat file, you must follow some extra steps to set up the geospatial metadata. For more information, see Importing geospatial data into the Variable File Node.

Notes for reading in delimited text data

  • Records must be delimited by a newline character at the end of each line. The newline character must not be used for any other purpose (such as within any field name or value). Leading and trailing spaces should ideally be stripped off to save space, although this is not critical. Optionally these spaces can be stripped out by the node.
  • Fields must be delimited by a comma or other character that ideally is used only as a delimiter, meaning it does not appear within field names or values. If this is not possible, then all text fields can be wrapped in double quotation marks, if none of the field names or text values contains a double quotation mark. If field names or values do contain double quotation marks, then text fields can be wrapped in single quotation marks as an alternative, again if single quotation marks are not used elsewhere within values. If neither single or double quotation marks can be used, then text values need to be amended to remove or replace either the delimiter character, or single or double quotation marks.
  • Each row, including the header row, should contain the same number of fields.
  • The first line should contain the field names. If not, clear Read field names from file to give each field a generic name such as Field1, Field2, and so on.
  • The second line must contain the first record of data. There must no blank lines or comments.
  • Numeric values must not include the thousands separator or grouping symbol—without the comma in 3,000.00, for example. The decimal indicator (period or full-stop in the US or the UK) must be used only where appropriate.
  • Date and time values should be in one of the formats that are recognized in the Stream Options dialog box, such as DD/MM/YYYY or HH:MM:SS. All dates and time fields in the file should ideally follow the same format, and any field that contains a date must use the same format for all values within that field.