支持的Spark版本和应用程序语言

支持的 Spark 版本

IBM Cloud Pak for Data 支持以下 Spark 运行时版本来运行 Spark 工作负载。

支持的应用程序语言

以下示例展示了为不同语言提交Spark作业的有效负载模板。

• 使用 Python 提交Spark作业的有效负载 3.10:

  {
    "application_details": {
      "application": "<your application_file_path>",
      "arguments": [
        "<your_application_arguments>"
      ],
      "conf": {
        "spark.app.name": "MyJob",
        "spark.eventLog.enabled": "true"
      },
      "env": {
        "RUNTIME_PYTHON_ENV": "python310"
      }
    }
  }
  

• 提交 Spark Scala 作业的负载：

  {
    "application_details": {
      "application": "/opt/ibm/spark/examples/jars/spark-examples*.jar",
      "arguments": [
        "1"
      ],
      "class": "org.apache.spark.examples.SparkPi",
      "conf": {
        "spark.app.name": "MyJob",
        "spark.eventLog.enabled": "true",
        "spark.driver.memory": "4G",
        "spark.driver.cores": 1,
        "spark.executor.memory": "4G",
        "spark.executor.cores": 1,
        "ae.spark.executor.count": 1
      },
      "env": {
        "SAMPLE_ENV_KEY": "SAMPLE_VALUE"
      }
    }
  }
  

• 提交R 4.2 Spark作业的有效负载：

  {
    "application_details": {
      "application": "/opt/ibm/spark/examples/src/main/r/dataframe.R",
      "class": "org.apache.spark.examples.SparkPi",
      "conf": {
        "spark.app.name": "MyJob",
        "spark.eventLog.enabled": "true",
        "spark.driver.memory": "4G",
        "spark.driver.cores": 1,
        "spark.executor.memory": "4G",
        "spark.executor.cores": 1,
        "ae.spark.executor.count": 1
      },
      "env": {
        "SAMPLE_ENV_KEY": "SAMPLE_VALUE"
      }
    }
  }