Spark jobs API syntax, parameters and return codes

You typically submit a Spark job with a cURL command.

The Spark job cURL command syntax is:

curl -k -X POST <V4_JOBS_API_ENDPOINT> -H "Authorization: Bearer <ACCESS_TOKEN>" -d @input.json

Spark jobs cURL options:

• The -k option skips certificate validation as the service instance website uses a self-signed SSL certificate.
• <V4_JOBS_API_ENDPOINT> is the endpoint for the instance that you want to use to submit your Spark job. Note that multiple Analytics Engine Powered by Apache Spark instances can exist on the IBM Cloud Pak for Data server and each instance has its own endpoint for submitting jobs. To get the Spark jobs endpoint for your provisioned instance, see Administering the service instance.
• The -H option is the header parameter. The header parameter is a key value pair. You must send the bearer token (<ACCESS_TOKEN>) in an authorization header. To get the access token for your service instance, see Generating an access token.
• The -d option defines the payload data to be sent in a POST request to the server. See the example of an input payload below.

An example of an input payload for a Python job:

{
"application_details": {
        "application": "/opt/ibm/spark/examples/src/main/python/wordcount.py",
        "arguments": ["/opt/ibm/spark/examples/src/main/resources/people.txt"],
        "conf": {
                "spark.app.name": "MyJob",
                "spark.eventLog.enabled": "true",
                "spark.driver.memory": "4G",
                "spark.driver.cores": 1,
                "spark.executor.memory": "4G",
                "spark.executor.cores": 1,
                "ae.spark.executor.count": 1
                },
        "env": {
                "SAMPLE_ENV_KEY": "SAMPLE_VALUE"
                }
        }
}

An example of an input payload for an R job:

{
"application_details": {
        "application": "/opt/ibm/spark/examples/src/main/r/dataframe.R",
        "conf": {
                "spark.app.name": "MyJob",
                "spark.eventLog.enabled": "true",
                "spark.driver.memory": "4G",
                "spark.driver.cores": 1,
                "spark.executor.memory": "4G",
                "spark.executor.cores": 1,
                "ae.spark.executor.count": 1
                },
        "env": {
                "SAMPLE_ENV_KEY": "SAMPLE_VALUE"
                }
        }
}

An example of an input payload for a Scala job:

{
"application_details": {
        "application": "/opt/ibm/spark/examples/jars/spark-examples*.jar",
        "arguments": ["1"],
        "class": "org.apache.spark.examples.SparkPi",
        "conf": {
                "spark.app.name": "MyJob",
                "spark.eventLog.enabled": "true",
                "spark.driver.memory": "4G",
                "spark.driver.cores": 1,
                "spark.executor.memory": "4G",
                "spark.executor.cores": 1,
                "ae.spark.executor.count": 1
                },
        "env": {
                "SAMPLE_ENV_KEY": "SAMPLE_VALUE"
                }
        }
}

The returned response if your job was successfully submitted:

{
"application_id": "<application_id>",
"state": "ACCEPTED"
}

Hint:

• Save the returned value of "application_id" to get the status of the job or to stop the job.
• Save the returned value of "spark_application_id" to monitor and analyze the Spark application on the Spark history server.