Seamless package management using init scripts

Initialization scripts are shell scripts that run on every cluster node at startup before the Spark driver or executor launch. They are especially useful when your Spark application requires custom packages, environment variables, or perform any one‑time spark runtime configuration. This ensures all nodes are properly configured and ready to run your workload.

Note:

Currently supported only for non‑auto‑scaling jobs.

Applies to:

Spark engine

Apache Gluten accelerated Spark engine

Storing initialization script

Only SWH storage volumes are supported for storing initialization scripts.

You can upload the initialization script to a storage volume in one of the following methods:

  • Method 1: Using user interface - To upload the script from the user interface, do the following steps:
    1. From the Navigation menu, access Administrator > Storage Volumes.
    2. Create a new volume or use an existing one.
    3. Create a folder init_scripts in the volume and upload your scripts in the volume.
  • Method 2: Using API - To upload the script by using API, see Upload file to the volume.

Submitting an Application with Init Scripts

Use the following payload to submit a Spark application that uses an init_scripts script to install packages:
{

"application_details": {

"application": "/myapp/sparkstreaming.py",

"conf": {

"spark.driver.memory": "4G",

"spark.driver.cores": "1",

"spark.executor.memory": "4G",

"spark.executor.cores": "1",

"ae.spark.executor.count": "1",

"spark.app.name": "pw"

}

},

"volumes": [

{

"name": "cpd-instance::app-vol",

"mount_path": "/myapp",

"source_sub_path": ""

},

{

"name": "cpd-instance::volume-inst-4516",

"mount_path": "/init_scripts",

"source_sub_path": "init_scripts"

}

],

"init_scripts": [

"file://init_scripts/install-jar2.sh"

]

}

Viewing init‑script logs

Logs can be found in the home volume of the engine
  1. Open the home volume of the engine.
  2. Navigate to <engine_id>/<application_id>/init_status/.
  3. You find separate log files for the driver and executor init scripts.

Key points

  • init_scripts are always mounted at the init_scripts mount_path.
  • Only the file:// file format is accepted.
  • You cannot mount multiple volumes to the same mount_path.
  • If the init_scripts parameter is present in the payload, you must provide a volume whose mount_path is init_scripts; otherwise the request fails with a 4xx error.