Flight service in Python notebooks

You can use Flight service and the Apache Arrow Flight protocol to read from and write data to data assets in a project or space in Python. These data assets can be files in the storage associated with your current project or space, or data accessed through a database connection.

Using generated code for Flight service

When you use the code that is generated for you to load data from a file or a connection from the Code snippets pane in a notebook, the generated code uses pyarrow to invoke Flight service. However, the function calls to pyarrow are not visible because they are wrapped in higher level functions provided by another library called itc_utils.

This itc_utils library is pre-installed in all notebook runtime environments provided by IBM for the purpose of reducing the code size and making the code more readable. To achieve this goal, the itc_utils library leverages information from the runtime environment and from ibm_watson_studio_lib library.

Another significant advantage of the generated code is that the data request has special properties, namely 'connection_name', 'connected_data_name', or 'data_name', depending on the kind of asset for which you generate the code. itc_utils converts these properties into an asset_id, plus a project_id or space_id, before creating a flight descriptor.

Example of code that is generated for you to load data:

import itc_utils.flight_service as itcfs

readClient = itcfs.get_flight_client()

MyConnection_data_request = {
    'connection_name': """MyConnection""",
    'interaction_properties': {
        'row_limit': 5000,
        'schema_name': '<schema>',
        'table_name': '<table>'
    }
}

flightInfo = itcfs.get_flight_info(readClient, nb_data_request=MyConnection_data_request)

data_df_1 = itcfs.read_pandas_and_concat(readClient, flightInfo, timeout=240)
data_df_1.head(10)

You can read more about other functions of the itc_utils library in Using itc_utils with your own code.

Learn more

Parent topic: Accessing data sources with Flight service