`

When submitting a Spark application in watsonx.data, you can customize the application payload to include the following features:

Applies to :

Spark engine

Apache Gluten accelerated Spark engine

Idempotency keys: It ensures that application submissions are processed exactly once, even in cases of client-server communication failures.
Consider an example where you are submitting a Spark application. Due to a temporary network issue between your client and the server, the request times out and you don’t receive a confirmation. Unsure whether the job was successfully submitted, you retry the submission multiple times. Each retry creates a new application ID, resulting in multiple instances of the same job running potentially consuming unnecessary resources. To avoid this, you can include an Idempotency key in your application payload. This key acts as a unique identifier for the job submission. If you resubmit the same application with the same idempotency key, multiple requests with the same idempotency key are treated as a single submission.
Retry feature: You can specify the following parameters in the payload to control how many times your Spark application should automatically retry submission in case of failure.
- max_retries field in the payload – If specified, the application will be retried up to the specified max_retries upon failures. If you do not specify any value, the application runs exactly once.
- min_retry_interval_in_seconds - The retry interval is calculated in seconds between the start of the failed run and the subsequent retry run.
Maximum runtime controls: You can define a maximum execution time for Spark applications. If the timeout is not specified, jobs continues to run until completion, regardless of how long they take.

Configuring Idempotency key in payload

Use the idempotency_key parameter in the Spark application payload to avoid triggering duplicate job submissions. This ensures that repeated requests with the same key are treated as a single request. Idempotency_key cannot be greater than 64 characters in length.

The following is a sample payload. Specify the idempotency_key parameter value in the <userdefined_key> field while submitting an application.

{
    "idempotency_key": "<userdefined_key>",
    "application_details": {
        "application": "/mnts/myapp/longrunning_with_sleep.py",
        "arguments": [
            ""
        ],
        "conf": {
            "spark.dynamicAllocation.minExecutors": "1",
            "spark.dynamicAllocation.maxExecutors": "4",
            "ae.spark.autoscale.enable": "false",
            "spark.app.name": "PySpark Application",
            "spark.eventLog.enabled": "true"
        }
    },
    "volumes": [
        {
            "name": "cpd-instance::app-vol-inst-6415",
            "mount_path": "/mnts/myapp"
        }
    ]
}

Configuring maximum runtime time in payload

Use the timeout_in_seconds parameter in the payload to specify the maximum execution time while submitting an application.

The following is a sample payload that shows how to configure the timeout_in_seconds while submitting an application. Here, the maximum execution time is 10 seconds. If the timeout is not specified, jobs will continue to run until completion, regardless of how long they take.

{
    "application_details": {
        "application": "cos://<cos_storage_name>.mycos/longrunning_with_sleep.py",
        "conf": {
            "spark.app.name": "cosApp",
            "spark.hadoop.fs.cos.mycos.endpoint": "s3.us-south.cloud-object-storage.appdomain.cloud",
            "spark.hadoop.fs.cos.mycos.secret.key": "**************",
            "spark.hadoop.fs.cos.mycos.access.key": "*********"
        }
    },
    "timeout_in_seconds": "10"
}

Configuring maximum retries applicable

Use max_retries field in the payload to specify Retry feature.

The following is a sample payload that shows how to configure the max_retries and min_retry_interval_in_secondswhile submitting an application.

{
    "application_details": {
      "application":"cos://<cos_storage_name>.mycos/pi5.py",
      "conf": {
        "spark.app.name": "cosApp",
        "spark.hadoop.fs.cos.mycos.endpoint": "s3.us-south.cloud-object-storage.appdomain.cloud",
        "spark.hadoop.fs.cos.mycos.secret.key": "<secret-key>",
        "spark.hadoop.fs.cos.mycos.access.key": "<access-key>"
      }
    },
    "max_retries": "3",
    "min_retry_interval_in_seconds":"30"
}