Flight data requests

Data request syntax and properties

To find out which interaction properties are supported by which connector, see Data and AI Common Core API. You can also call the v2/datasources_types endpoint to Data and AI Common Core API list defined types of data source.

A data request has the following structure. Actual instances of data requests typically do not include all of the listed properties and might use other interaction properties that are specific to a certain data source.

Example of a request to /v2/datasource_types:

curl --request GET --url 'https://<host:port>/v2/datasource_types?connection_properties=true&interaction_properties=true' --header 'Accept: application/json' --header 'Authorization: Bearer ${TOKEN}'

Example of a request to /v2/connections:

curl --request GET --url 'https://<host:port>/v2/connections?entity.name=<your_connection_name>' --header 'Accept: application/json' --header 'Authorization: Bearer ${TOKEN}'

Data request object syntax

The following data request shows the syntax in Python notation. The description is also valid for R.

data_request = {

    # The Flight Service expects assets to be identified by their GUID.
    # "asset_id" is the ID of connected data asset or a connection asset.
    "asset_id": "ASSET_GUID",
    "project_id|space_id|catalog_id": "GUID",

    # itc_utils.flight_service adds support for using asset names
    # use either one of ...
    "data_name" : "name of a plain data asset",
    "connection_name" : "name of a connection asset",
    "connected_data_name" : "name of a 'connected data' asset",
    # by default the assets will be looked up in the current project
    # itc_utils will set "asset_id" from the provided name.

    # "asset_id" and "connection_properties" are mutually exclusive
    "connection_properties": { # optional
        # not needed in most cases
    },

    "interaction_properties": {
        # commonly used properties, all optional
        "file_name" : "file in a connected data source",
        "schema_name" : "schema of a table in a connected database",
        "table_name" : "name of a table in a connected database",
        "row_limit" : NROWS,
        "infer_schema" : false|true,
        "invalid_data_handling" : "fail|row|column",
        ...
    },
    "fields": [...] # optional
    "num_partitions": N, # optional
    "batch_size": NROWS, # optional
    "commit_frequency" : NROWS # optional

}

Supported properties, values, and descriptions

fields property

The fields property enumerates the fields, also referred to as columns, to be selected from a data source. When reading from a CSV or delimited file, this list of fields defines the names and types of the fields in the file. The layout of the values in the file must exactly match the list of fields. Each field must specify both the name and the data type.

Supported values for the type attribute in fields are (names are not case-sensitive): ARRAY, BIGINT, BINARY, BIT, BLOB, BOOLEAN, CHAR, CLOB, DATALINK, DATE, DECIMAL, DISTINCT, DOUBLE, FLOAT, INTEGER, JAVA_OBJECT, LONGNVARCHAR, LONGVARBINARY, LONGVARCHAR, NCHAR, NCLOB, NULL, NUMERIC, NVARCHAR, OTHER, REAL, REF, REF_CURSOR, ROWID, SMALLINT, SQLXML, STRUCT, TIME, TIME_WITH_TIMEZONE, TIMESTAMP, TIMESTAMP_WITH_TIMEZONE, TINYINT, VARBINARY, VARCHAR, VECTOR
The values for the type attribute in fields are the names that are used by the Flight service. When data is read, these types are mapped to the data types used in open source Arrow Flight. For more information, see Data Types and In-Memory Data Model. For example, TINYINT gets mapped to int8() and BIGINT to int64().

However, with some data sources, for example when reading from a Parquet file or from a table in an external SQL database, the list of fields can specify a subset of the columns to be read from the source.

Flight service won't read other columns. By default, all columns of the table would be retrieved. Each field must specify both the name and the data type. The type can override the type of the column in the external table. For example, a column that is stored as VARCHAR(10) can be mapped to bigint in Flight. The fields property is not used when writing to a database; instead, Flight service uses the schema that is provided in do_put(...). The following example shows how to use the fields property in Python notation. However, the description is also valid for R.

```json {: .codeblock}
    "fields": [{"name":"a","type":{"type":"bigint"}},
           {"name":"b","type":{"type":"varchar"}}
          ]
```

: The value of the type attribute is another dictionary with another type attribute.

If the fields property is provided in the data request, the infer_schema property is implicitly set to false because the fields property already defines the data types.

:

If you use pandas DataFrames and want to check the mapping between data types in Flight and pandas, see Type differences.

The fields property specifies the name and data type of a column and more details. The following example shows the structure of the fields template in Python notation. However, the description is also valid for R.

{
    "name": "fieldname",
    "type": {
        "type": "typename",
        "length": L ,     # integer, depends on type
        "scale": S,       # integer depends on type, only numeric and decimal
        "signed": true or false   # for numeric types
    }
}

The combination of length and scale defines the precision and scale of a numeric or decimal data type.

The property signed influences the type and range of values that are considered to be valid.

• {"type":"TINYINT","signed":true} is mapped to the type int8() in Flight. Valid values are -128 <= signed tinyint <= 127.

• {"type":"TINYINT","signed":False} is mapped to the type uint8() in Flight. Valid values are 0 <= unsigned tinyint <= 255. |

Learn more

• Examples of Flight service data requests with Python
• Examples of Flight service data requests with R

Parent topic: Accessing data sources with Flight service