Data types

The following sections describe the various data types and their valid values:

• Value formats
  
  • Parsing values from JSON
    
  • Serializing to JSON and CSV
    
• Input and output values in Code operators
  
  • Data values at input
    
  • Expected Code operator output values
    
• Numbers
  
• Dates
  
  • Traits
    
  • Parsing from source messages
    
  • Formatting for Target operators (schema-less and schema-based)
    
• Binary data

Value formats

The section is about parsing and formatting of data, depending on its type.

Parsing values from JSON

The following table gives examples of valid values for the supported data types. Messages with invalid values are skipped and reported in the Notifications pane of the Metrics page. This data type handling applies to all Source operators that output data in JSON format, for example, HTTP, Kafka, MQTT, and Watson IoT.

For valid output values in Code operators, see section Expected Code operator output values.

Serializing values to JSON and CSV

Formatted values are in quotation marks (“) only for data types Text and Date. The following table shows examples:

^(*) For more information about inserting Date values into schema-less or schema-based Target operators, see Formatting Date output for Target operators.

Input and output values in Code operators

Data values at Code operator input

The following table describes the data type mapping between flow columns and Python dictionary values at Code operator inputs. It applies to the Code operator in Sources and Targets, and to the Python Model operator in Processing and Analytics.

Expected Code operator output values

The section applies to the Code operator in Sources, and to the Code operator and the Python Model operator in Processing and Analytics.

The following table gives examples of valid output values, by data type. Outputs with invalid values are skipped and reported in the Notifications pane of the Metrics page.

(*) False is a Boolean literal, but "False" is a string literal. Any non-empty string is interpreted as True, whether it’s “True”, “False”, or “yes sir”.

When your output dictionary has no entry for an output column, or when its value is None, then the streaming runtime acts in the following way:

• If a same-name input column with the same column data type exists, its value is used (the value is passed through).

• Otherwise, the default value for the output column’s data type is used.

Numbers

• When Number column values are parsed from JSON messages of Source operators such as Event Streams or Watson IoT, the values are always mapped to float values in input dictionaries.

• When Number column values are coming in from other operators, the Python type can be mapped to float or int values, depending on the operator logic. For example, the count function in the Aggregation operator produces values of type int.

• In the Code operator, you can output either int or float values for output columns of type Number.

Dates

The following traits apply to Date values:

• Date values express a point in time, and as such, include date and time information similar to a UNIX timestamp.

• Split-second precision of Date values is up to microseconds (up to 6 digits after the decimal point).

• Date values do not carry time zone information.

Parsing dates from Source operator messages

• When parsing data that comes from Source operators such as IBM Event Streams or IBM Watson IoT, a streams flow assumes that all Date values are given in ISO 8601 format. The streams flow normalizes the date values internally to Coordinated Universal Time (UTC).

• Separator characters other than -, T, and : are also accepted. For example, the date might use a space as a separator:
  2018-03-26 15:3:47.123456

• This parsing is applicable to all Source operators except Code (in Sources).

The following table gives examples of supported formats:

(*) When no time zone is indicated, the time is assumed to be given as UTC.

Tip

When processing dates from multiple time zones, their textual representation in source messages ideally have the respective time zone offset. If not, then you can use a Code operator to enhance the Date values with time zone information that is not explicit in the source. You can also use a Code operator for parsing formats other than ISO 8601.

Formatting Date output for Target operators

• Outputting Date values to schema-less Target operators, such as Kafka or Redis

  The values are formatted according to ISO 8601:

  • 2018-03-26T11:38:47Z (value has no split seconds)

  • 2018-03-26T11:38:47.123400Z (value contains split seconds)

Binary data

• For ingesting Binary data from source operators that output a single data attribute, select the parsing option “None”. This will stream the data as raw data, without the built-in parsing. The following table gives examples for such operators:

  Source operator	Data attribute name
  Kafka	event_message
  HTTP	http_body

• For writing Binary data in target operators that usually need a format setting for serializing the incoming data, set the format option to “None” and make sure this operator’s output schema has exactly one attribute. This will pass the data as raw data, without the built-in serialization.

• In operators that receive or output structured data as typed attributes, the attributes can be of type Binary. Examples of such operators are Code and Streams.