Spark jobs API syntax, parameters and return codes

You typically submit a Spark job with a cURL command.

The Spark job cURL command syntax is:

curl -k -X POST <V3_JOBS_API_ENDPOINT> -H "Authorization: Bearer <ACCESS_TOKEN>" -d @input.json

Spark jobs cURL options:

• The -k option skips certificate validation as the service instance website uses a self-signed SSL certificate.
• <V3_JOBS_API_ENDPOINT> is the endpoint for the instance that you want to use to submit your Spark job. Note that multiple Analytics Engine Powered by Apache Spark instances can exist on the IBM Cloud Pak for Data server and each instance has its own endpoint for submitting jobs. To get the Spark jobs endpoint for your provisioned instance, see Administering the service instance.
• The -H option is the header parameter. The header parameter is a key value pair. You must send the bearer token (<ACCESS_TOKEN>) in an authorization header. To get the access token for your service instance, see Administering the service instance.

• The -d option defines the payload data to be sent in a POST request to the server. See the example of an input payload below.

  Note: If the application creates a SparkContext, the POST method returns when the SparkSession is created. If the application does not create a SparkContext, the POST method is blocked until the application terminates.

  An example of an input payload for a Python job:

    {
        "template_id": "<template_id>",
        "application_details": {
                "application": "/opt/ibm/spark/examples/src/main/python/wordcount.py",
                "application_arguments": ["/opt/ibm/spark/examples/src/main/resources/people.txt"],
                "conf": {
                        "spark.app.name": "MyJob",
                        "spark.eventLog.enabled": "true"
                        },
                "env": {
                        "SAMPLE_ENV_KEY": "SAMPLE_VALUE"
                        },
                "driver-memory": "4G",
                "driver-cores": 1,
                "executor-memory": "4G",
                "executor-cores": 1,
                "num-executors": 1
        }
    }
  

  An example of an input payload for a Scala job:

    {
        "template_id": "<template_id>",
        "application_details": {
                "application": "/opt/ibm/spark/examples/jars/spark-examples*.jar",
                "application_arguments": ["1"],
                "class": "org.apache.spark.examples.SparkPi",
                "conf": {
                        "spark.app.name": "MyJob",
                        "spark.eventLog.enabled": "true"
                        },
                "env": {
                        "SAMPLE_ENV_KEY": "SAMPLE_VALUE"
                        },
                "driver-memory": "4G",
                "driver-cores": 1,
                "executor-memory": "4G",
                "executor-cores": 1,
                "num-executors": 1
                }
    }
  

  The POST method returns when the application creates a SparkSession. The application continues to run and the status can be retrieved.

  The returned response if your job was successfully submitted:

    {
        "application_id": "<application_id>",
        "state": "RUNNING",
        "start_time": "Monday' 07 June 2021 '14:46:23.237+0000",
        "spark_application_id": "app-20210607144623-0000"
    }
  

Hint:

• Save the returned value of "application_id" to get the status of the job or to stop the job.
• Save the returned value of "spark_application_id" to monitor and analyze the Spark application on the Spark history server.

Spark jobs API parameters

These are the parameters you can use in the Spark jobs API:

Response codes

The Spark jobs API returns the following response codes:

Default Spark configuration parameters and environment variables

The following tables show the Spark configuration parameters and environment variables that are commonly used in Analytics Engine Powered by Apache Spark and their default values.

The following table lists the Spark configuration parameters and their defaults:

The following table lists the environment variables and their defaults:

Parent topic: Submitting Spark jobs