Spark jobs API syntax, parameters and return codes

You typically submit a Spark job with a cURL command.

The Spark job cURL command syntax is:

curl -k -X POST <V4_JOBS_API_ENDPOINT> -H "Authorization: ZenApiKey <TOKEN> -d @input.json

Replace the variables as follows:

• <V4_JOBS_API_ENDPOINT>: The endpoint for the instance that you want to use to submit your Spark job. Note that multiple Analytics Engine Powered by Apache Spark instances can exist on the IBM Cloud Pak for Data server and each instance has its own endpoint for submitting jobs. To get the Spark jobs endpoint for your provisioned instance, see Managing Analytics Engine powered by Apache Spark instances.
• <TOKEN>: To get the access token for your service instance, see Generating an API authorization token.

An example of an input payload for a Python job:

{
  "application_details": {
    "application": "/opt/ibm/spark/examples/src/main/python/wordcount.py",
    "arguments": [
      "/opt/ibm/spark/examples/src/main/resources/people.txt"
    ],
    "conf": {
      "spark.app.name": "MyJob",
      "spark.eventLog.enabled": "true",
      "spark.driver.memory": "4G",
      "spark.driver.cores": 1,
      "spark.executor.memory": "4G",
      "spark.executor.cores": 1,
      "ae.spark.executor.count": 1
    },
    "env": {
      "SAMPLE_ENV_KEY": "SAMPLE_VALUE"
    }
  }
}

An example of an input payload for an R job:

{
  "application_details": {
    "application": "/opt/ibm/spark/examples/src/main/r/dataframe.R",
    "conf": {
      "spark.app.name": "MyJob",
      "spark.eventLog.enabled": "true",
      "spark.driver.memory": "4G",
      "spark.driver.cores": 1,
      "spark.executor.memory": "4G",
      "spark.executor.cores": 1,
      "ae.spark.executor.count": 1
    },
    "env": {
      "SAMPLE_ENV_KEY": "SAMPLE_VALUE"
    }
  }
}

An example of an input payload for a Scala job:

{
  "application_details": {
    "application": "/opt/ibm/spark/examples/jars/spark-examples*.jar",
    "arguments": [
      "1"
    ],
    "class": "org.apache.spark.examples.SparkPi",
    "conf": {
      "spark.app.name": "MyJob",
      "spark.eventLog.enabled": "true",
      "spark.driver.memory": "4G",
      "spark.driver.cores": 1,
      "spark.executor.memory": "4G",
      "spark.executor.cores": 1,
      "ae.spark.executor.count": 1
    },
    "env": {
      "SAMPLE_ENV_KEY": "SAMPLE_VALUE"
    }
  }
}

The returned response if your job was successfully submitted:

{
  "application_id": "<application_id>",
  "state": "ACCEPTED"
}

Hint:

• Save the returned value of "application_id" to get the status of the job or to stop the job.
• Save the returned value of "spark_application_id" to monitor and analyze the Spark application on the Spark history server.

{
  "application_details": {
    "application": "/opt/ibm/spark/examples/src/main/python/wordcount.py",
    "arguments": [
      "/opt/ibm/spark/examples/src/main/resources/people.txt"
    ],
    "runtime": {
      "spark_version": "3.4"
    }
  }
}

{
  "volumes": [
    {
      "name": "cpd-instance::myapp-vol",
      "mount_path": "/my-app"
    }
  ],
  "application_details": {
    "application": "/my-app/python-spark-pi.py",
    "packages":
      "org.apache.spark:spark-sql-kafka-0-10_2.12:3.4.0,org.apache.spark:spark-streaming-kafka-0-10_2.12:3.4.0",
    "conf": {
      "spark.app.name": "MyJob",
      "spark.eventLog.enabled": "true",
      "spark.driver.cores": 4,
      "spark.driver.memory": "8G",
      "spark.executor.memory": "2G",
      "spark.executor.cores": 4,
      "ae.spark.driver.log.level": "ERROR",
      "ae.spark.executor.log.level": "WARN"
    }
  }
}