API syntax for submitting Spark runtime

curl -k -X POST --url https://<cpd_host_name>/lakehouse/api/<api_version>/spark_engines/<spark_engine_id>/applications \ -H "Authorization: ZenApiKey <TOKEN> -d @input.json

Replace the variables as follows:

• <cpd_host_name>: The hostname of your IBM Software Hub.
• <api_version>: When using the v2 API, set the <api_version> parameter to v2; for the v3 API, set it to v3.
• <spark_engine_id> : The Engine ID of the native Spark engine.
• <token> : The bearer token. For more information about generating the token, see Generating a bearer token.

Note:

The POST method returns after the initial validation of the application. The runtime request is processed asynchronously; first the Spark context is created and then the application is executed. The current status of the application can be fetched by using the GET method.

An example of an input payload for a Python runtime:

{"application_details":{"application":"/opt/ibm/spark/examples/src/main/python/wordcount.py","arguments":["/opt/ibm/spark/examples/src/main/resources/people.txt"],"conf":{"spark.app.name":"MyJob","spark.eventLog.enabled":"true","spark.driver.memory":"4G","spark.driver.cores":1,"spark.executor.memory":"4G","spark.executor.cores":1,"ae.spark.executor.count":1},"env":{"SAMPLE_ENV_KEY":"SAMPLE_VALUE"}}}

An example of an input payload for a Scala runtime:

{"application_details":{"application":"/opt/ibm/spark/examples/jars/spark-examples*.jar","arguments":["1"],"class":"org.apache.spark.examples.SparkPi","conf":{"spark.app.name":"MyJob","spark.eventLog.enabled":"true","spark.driver.memory":"4G","spark.driver.cores":1,"spark.executor.memory":"4G","spark.executor.cores":1,"ae.spark.executor.count":1},"env":{"SAMPLE_ENV_KEY":"SAMPLE_VALUE"}}}

The returned response if your runtime was successfully submitted:

{
  "application_id": "<application_id>",
  "state": "ACCEPTED"
}

Hint:

• Save the returned value of "application_id" to get the status of the runtime or to stop the runtime.
• Save the returned value of "application_id" to monitor and analyze the Spark application on the Spark history server.