Loading and accessing data in a notebook (Watson Studio)

You can integrate data into notebooks by accessing the data from a local file, from free data sets, or from a data source connection. You load that data into a data structure or container in the notebook, for example, a pandas.DataFrame, numpy.array, Spark RDD, or Spark DataFrame.

To work with data in a notebook, you can choose between the following options:

Option Recommended method Requirements Details
Add data from a file from your local system. Use generated code for the selected data source by clicking the Code snippets icon (the Code snippets icon) from the toolbar and clicking Read data. The file must exist as a project asset. Add a file from your local system.
Load data from data source connections. Use generated code for the selected data source by clicking the Code snippets icon (the Code snippets icon) from the toolbar and clicking Read data. The connections must exist as project assets. Load data from a data source connection.
Access project assets and metadata programmatically. Use ibm-watson-studio-lib library functions. The data sources must exist as project assets. Use the ibm-watson-studio-lib library to interact with data assets.
Generate your own code to read or write data. Use the Flight Service and the Apache Arrow Flight protocol to read from and write to data assets in a project. The data assets must exist as assets in the project before you can read or write data. Access data sources using the Flight service in Python notebooks or the Flight service in R notebooks.
Create and use feature store data. Use assetframe-lib library functions. The data assets must exist as assets in the project. Use the assetframe-lib library for Python to create and use feature store data.
Access data using an API function or operating system command. For example, use Wget. N/A Access data using an API function or operating system command.
Important: Make sure that the environment in which the notebook is started has enough memory to store the data that you load to the notebook. The environment must have significantly more memory than the total size of the data that is loaded to the notebook. Some data frameworks, like pandas, can hold multiple copies of the data in memory.

Load data from local files

To access data from a local file, you can load the file from within a notebook, or first load the file into your project. From your notebook, you add automatically generated code to access the data by clicking the Code snippets icon (the Code snippets icon) and then clicking Read data. The generated code serves as a quick start to allow you to easily begin working with data sets.

Code is generated for file types such as CSV, JSON, and XLSX. To learn which data structures are generated for which notebook language, see Data load support. For file types for which code generation is not supported, you can insert only the file credentials. With the credentials, you can write your own code to load the file data into a DataFrame or other data structure in a notebook cell.

To add a file from your local system to your notebook:

  1. Open your notebook in edit mode, click the Upload asset to project icon (Shows the Upload asset to project icon) from the toolbar, and then browse a data file or drag it into your notebook sidebar.
  2. Click the Code snippets icon (the Code snippets icon), click Read data and then select the data file from the project. Use the pencil icon if you want to select a different data file.
  3. In the Load as drop-down list, select the load option that you prefer.
  4. Click in an empty code cell in your notebook and then click to insert the generated code. Alternatively, click to copy the generated code to the clipboard and then paste the code into your notebook.

Load data from data source connections

Before you can load data from an IBM data service or from an external data source, you must create or add a connection to your project. See Adding connections to projects.

You add automatically generated code to load data from database connections by clicking the Code snippets icon (the Code snippets icon) from the notebook toolbar and then clicking Read data. To learn which database connections are supported, see Data load support. For database connections that are not supported, you can insert only the database connection credentials. With the credentials, you can write your own code to load the data into a DataFrame or other data structure in a notebook cell.

To load data from an existing data source connection into a data structure in your notebook:

  1. Open your notebook in edit mode, click the Code snippets icon (the Code snippets icon) from the toolbar, click Read data and then select a data source connection from the project. Use the pencil icon if you want to select a different connection.
  2. Select the schema and choose a table. Use the pencil icon if you want to change your selection.
  3. Select the load option. If you select credentials and not a data load option after you choose the schema and table, only metadata will be generated.
  4. Click in an empty code cell in your notebook and then insert code to the cell. Alternatively, click to copy the generated code to the clipboard and then paste the code into your notebook. The generated code serves as a quick start to begin working with a data set or connection. For production systems, carefully review the inserted code to determine whether you should write your own code that better meets your needs.
  5. If necessary, enter your personal credentials for locked data connections that are marked with a key icon (the key symbol for connections with personal credentials). This is a one-time step that permanently unlocks the connection for you. After you have unlocked the connection, the key icon is no longer displayed. See Adding connections to projects.
  6. If no code can be generated for the connection, load the credentials and open the database connection that references your credentials. Write code to load the data.

For a Planning Analytics connection, see Adding data from a Planning Analytics connection.

Use an API function or operating system command to access the data

You can use API functions or operating system commands in your notebook to access data, for example, the Wget command to access data by using the HTTP, HTTPS or FTP protocols.

Learn more

Parent topic: Notebooks and scripts