Customizing Spark applications and notebooks using the AnalyticsEngine CR configuration
You can persist custom Python packages to use in Spark applications and Python notebooks across Spark instances, projects and deployment spaces in the cc-home
volume.
Using Python custom packages
-
Connect to the OpenShift cluster:
oc login OpenShift_URL:port
-
Set the context to the project where Cloud Pak for Data is deployed:
oc project ${PROJECT_CPD_INST_OPERANDS}
-
Set mountCustomizationsFromCCHome to true in the AnalyticsEnginge custom resource:
oc patch AnalyticsEngine analyticsengine-sample --namespace CHANGE_ME_NAMESPACE --type merge --patch '{"spec": {"serviceConfig":{"mountCustomizationsFromCCHome":true}}}'
-
Wait for the AnalyticsEnginge custom resource status to change to Completed.
oc get AnalyticsEngine --namespace CHANGE_ME_NAMESPACE
The AnalyticsEngine custom resource status can take up to several minutes to change to Completed.
-
Get the required platform access token. For details, see Authenticating for programming access.
-
Create a volume instance against the
cc-home-pvc
. For instructions, see Setting up a storage volume to store customizations for the common core services. -
To verify that the new volume instance is available, use the volumes endpoint.
curl -k $CPDHOST/zen-data/v3/service_instances -H "Authorization: ZenApiKey ${TOKEN}" -H 'Content-Type: application/json' | jq '.service_instances[] | .display_name,.id,.metadata.existing_pvc_name'
-
To stop the file server, use the volume_services endpoint.
curl -k -X DELETE $CPDHOST/zen-data/v1/volumes/volume_services/$NAMESPACE::$VSNAME -H "Authorization: ZenApiKey ${TOKEN}"
-
To restart the file server, use the volumes_services endpoint.
curl -k -X POST $CPDHOST/zen-data/v1/volumes/volume_services/$NAMESPACE::$VSNAME -H "Authorization: ZenApiKey ${TOKEN}" -H 'Content-Type: application/json' -d '{}'
-
Upload the custom packages:
- Go to the Cloud Pak for Data navigation menu.
- Select Administration > Storage Volumes.
- From the volumes list, select CCHome.
- Upload the packages in the directory:
_global_/dbdrivers/jdbc/default
.
Important:Packages must be uploaded to the
_global_/dbdrivers/jdbc/default
directory.
Parent topic: Using custom packages