Supported Spark versions and application languages

Supported Spark versions

IBM Cloud Pak for Data supports the following Spark runtime versions to run Spark workloads.

Supported application languages

The following examples show you sample payloads for submitting Spark job for different languages.

• Payload for submitting a Spark job with Python 3.10:

  {
    "application_details": {
      "application": "<your application_file_path>",
      "arguments": [
        "<your_application_arguments>"
      ],
      "conf": {
        "spark.app.name": "MyJob",
        "spark.eventLog.enabled": "true"
      },
      "env": {
        "RUNTIME_PYTHON_ENV": "python310"
      }
    }
  }
  

• Payload for submitting a Spark Scala job:

  {
    "application_details": {
      "application": "/opt/ibm/spark/examples/jars/spark-examples*.jar",
      "arguments": [
        "1"
      ],
      "class": "org.apache.spark.examples.SparkPi",
      "conf": {
        "spark.app.name": "MyJob",
        "spark.eventLog.enabled": "true",
        "spark.driver.memory": "4G",
        "spark.driver.cores": 1,
        "spark.executor.memory": "4G",
        "spark.executor.cores": 1,
        "ae.spark.executor.count": 1
      },
      "env": {
        "SAMPLE_ENV_KEY": "SAMPLE_VALUE"
      }
    }
  }
  

• Payload for submitting an R 4.2 Spark job:

  {
    "application_details": {
      "application": "/opt/ibm/spark/examples/src/main/r/dataframe.R",
      "class": "org.apache.spark.examples.SparkPi",
      "conf": {
        "spark.app.name": "MyJob",
        "spark.eventLog.enabled": "true",
        "spark.driver.memory": "4G",
        "spark.driver.cores": 1,
        "spark.executor.memory": "4G",
        "spark.executor.cores": 1,
        "ae.spark.executor.count": 1
      },
      "env": {
        "SAMPLE_ENV_KEY": "SAMPLE_VALUE"
      }
    }
  }
  

Parent topic: Submitting Spark jobs