Infer File Schema

The schema or metadata for delimited and CSV files (such as file datatypes) can now be detected automatically. The inference is supported for both runtime and metadata import.

Configure File Connector stage to infer schema

  1. Configure the File connector in source context to as target to write data to a file.
  2. Enable and select the Runtime column propagation property
  3. Select the Infer schema option under Delimited format properties and then select Record definition.

Metadata import

  • During metadata import if the schema is not specified in .osh file or in the header, the inference gets invoked and the metadata is automatically detected.

Limitations

  • During runtime, only default formats are supported for Date, Time and Timestamp column types.
  • Inference works better with when the file has 10 or more rows.
  • The : field delimiter is currently not supported for metadata import.