Data and file size limitation in project-lib and ibm-watson-studio-lib using Python 3.9

Cause

This is caused by the Python requests library that uses a Python SSL library that does not support objects that are larger than 2 GB. This issue is fixed in Python 3.10.

Resolving the problem

2 GB is a hard limit in project-lib for Python 3.9 and can't be resolved.

To save data or upload a file larger than 2 GB in ibm-watson-studio-lib for Python 3.9, you need to move the file or the data to the storage associated with the project as follows:

1. If you upload a file in a notebook using the upload_file function, the file is already available in the file system of your environment runtime and you can skip step 2.
2. If you upload data in a notebook using the save_data function, you need to save the data to a file in the local file system of your environment runtime, for example:

   with open("my_asset.csv", "wb") as file:
       file.write(data)
   

3. Retrieve the path to the mounted project storage:

   wslib.mount.get_base_dir()
   

   Make a note of the path, for example, /project_data/data_asset/.
4. Copy the files to the mounted project storage, for example:

   !cp my_asset.csv /project_data/data_asset/
   

5. Register the files in the mounted project storage as data assets in your project:

   wslib.mount.register_asset("/project_data/data_asset/my_asset.csv", asset_name="my_asset.csv")
   

Parent topic: Troubleshooting problems in Cloud Pak for Data