Accessing data sources with Flight service

About Flight service

Flight service is a library of connectors as a service, which provides a single interface to connect to a variety of data sources. By using Flight service to make data source connections, you do not need to have the specific database drivers for each data source to access data. IBM's Flight service provides a language-agnostic and efficient connector service for read, write, or metadata requests to the data sources.

Required services

Flight service is available only if the common core services are installed. The common core services are not installed separately. Common core services are automatically installed if you install a service that has a dependency on them. For information about which services install the common core services, see Service software requirements.

Services and tools

You can use the Flight service APIs with these services that accept custom Python or R code:

• Analytics Engine powered by Apache Spark
• Watson Studio
• Watson Machine Learning
• Data Product Hub
• watsonx.data

Components of Flight service

Flight service is based on the following components:

• The Flight client API from the open-source Arrow Flight framework.
  • For more information, see the Apache Arrow documentation and Arrow Flight RPC.
• The location and authentication method for the Flight server. Applications don't need to retrieve credentials from a connection asset and submit them to a database driver. Flight service automatically retrieves credentials transparently.
• A library of connectors that Flight service supports to access a variety of data sources. For a full list of connectors supported by Flight service, see Supported connectors in Flight service.
• Data requests that you can use to specify which files or tables you want to access. See Flight data requests.
• The itc_utils library, which simplifies reading data with the Flight client. For examples, see Examples of Flight service with itc_utils and pandas.

Using Flight data requests to access data

You can access data sources by making Flight data requests in two different ways: either by using the generated code that is provided to you in Code snippets in a Python or Jupyter notebook or by writing your own code.

You can make data requests by using the generated code that loads in the Code snippet panel to make data requests in Python or R language to a supported data source with the Flight service APIs. The option to add generated code to load data from a project asset is available in the following tools:

• Python or R notebooks opened in the Notebook editor
• Python scripts and notebooks in JupyterLab

Where possible, always try to use the generated code to load data from a file or connection to a notebook. However, there might be situations in which you need to write your own code, for example:

• The generated code requires changes, for example to use in a production environment
• The functionality to add generated code is not available for the asset
• The tool doesn't support adding generated code, for example RStudio

If you need to write your own code to read from or write data to data assets, refer to the following topics for key concepts, use cases and sample code snippets when using pyarrow to invoke Flight service:

• Flight data requests
• Examples of Flight service data requests with Python
• Examples of Flight service data requests with R
• Flight service in Python notebooks
• Flight service in R notebooks

For more information on using generated code snippets, see Loading data through generated code snippets.

Learn more

• Flight data requests
• Flight service in Python notebooks
• Flight service in R notebooks

Parent topic: Adding data to a project