Creating a library set for your instance

Creating a library set for Python package install

Spark engine bundles a Spark application called customize_instance_app.py that you run to create a library set with your custom packages.

Prerequisites: You must have the permissions to submit a Spark application. See Managing roles and privileges.

1. Prepare a JSON file with the following content:

   {
     "library_set": {
       "action": "add",
       "name": "my_library_set",
       "libraries": {
         "conda": {
           "python": {
             "packages": ["numpy"]
           }
         }
       }
     }
   }

   JSON attribute descriptions:

   • "action": Specifies the action ("add" is the only option supported).
   • "name": Specifies the name of the library set. Use unique names for each set to avoid overwriting.
   • "libraries": Defines a set of libraries. Only "conda" and "pip" package managers are supported.
   • "packages": List of packages to install. For a specific version: package_name==version.
2. Get the IAM token.
3. Pass the JSON file as "arguments" in the following REST API call:

   curl --request POST \
     --url https://<region>.lakehouse.cloud.ibm.com/lakehouse/api/<api_version>/spark_engines/<spark_engine_id>/applications \
     --header 'Authorization: Bearer <token>' \
     --header 'Content-Type: application/json' \
     --header 'AuthInstanceID: <crn_instance>' \
     -d @createLibraryset.json

   Parameters: <region> – Spark instance region; <spark_engine_id> – Native Spark engine ID; <crn_instance> – CRN of the watsonx.data instance; <api_version> – v2 or v3.

4. Track the status of the application by invoking the application status REST API.

Creating a library set for other packages or file download (Script based)

With script based customization, create a Python script using the module naming convention expected by watsonx.data. Implement a Python function that downloads libraries to the predesignated directory: /home/spark/shared/user-libs/<libraryset_name>/custom/<subdir_if_applicable>.

1. Create a Python file named customization_script.py.
2. Implement a function called customize(<install_path>, <params>) in your script.
3. Store the customization_script.py file in IBM Cloud Object Storage or in GitHub.
4. Pass the location of the file to the customization Spark application through "application_details" > "conf" > "spark.submit.pyFiles".
5. Submit the application:

   curl --request POST \
     --url https://<region>.lakehouse.cloud.ibm.com/lakehouse/api/<api_version>/spark_engines/<spark_engine_id>/applications \
     --header 'Authorization: Bearer <token>' \
     --header 'Content-Type: application/json' \
     --header 'AuthInstanceID: <crn_instance>' \
     -d @createLibraryset.json