Spark 作业 API 语法、参数和返回码

您通常使用 cURL 命令提交 Spark 作业。

Spark 作业 cURL 命令语法为:

curl -k -X POST <V4_JOBS_API_ENDPOINT> -H "Authorization: ZenApiKey <TOKEN> -d @input.json

按如下所示替换变量:

• <V4_JOBS_API_ENDPOINT>: 要用于提交 Spark 作业的实例的端点。 请注意，Analytics Engine Powered by Apache Spark 服务器上可以存在多个 IBM Cloud Pak for Data 实例，每个实例都有自己的端点用于提交作业。 要获取已配置实例的 Spark 作业端点，请参阅管理 Analytics Engine powered by Apache Spark 实例。
• <TOKEN>: 要获取服务实例的访问令牌，请参阅 生成 API 授权令牌。

Python 作业的输入有效内容示例：

{
  "application_details": {
    "application": "/opt/ibm/spark/examples/src/main/python/wordcount.py",
    "arguments": [
      "/opt/ibm/spark/examples/src/main/resources/people.txt"
    ],
    "conf": {
      "spark.app.name": "MyJob",
      "spark.eventLog.enabled": "true",
      "spark.driver.memory": "4G",
      "spark.driver.cores": 1,
      "spark.executor.memory": "4G",
      "spark.executor.cores": 1,
      "ae.spark.executor.count": 1
    },
    "env": {
      "SAMPLE_ENV_KEY": "SAMPLE_VALUE"
    }
  }
}

R 作业的输入有效内容示例:

{
  "application_details": {
    "application": "/opt/ibm/spark/examples/src/main/r/dataframe.R",
    "conf": {
      "spark.app.name": "MyJob",
      "spark.eventLog.enabled": "true",
      "spark.driver.memory": "4G",
      "spark.driver.cores": 1,
      "spark.executor.memory": "4G",
      "spark.executor.cores": 1,
      "ae.spark.executor.count": 1
    },
    "env": {
      "SAMPLE_ENV_KEY": "SAMPLE_VALUE"
    }
  }
}

Scala 作业的输入有效内容示例：

{
  "application_details": {
    "application": "/opt/ibm/spark/examples/jars/spark-examples*.jar",
    "arguments": [
      "1"
    ],
    "class": "org.apache.spark.examples.SparkPi",
    "conf": {
      "spark.app.name": "MyJob",
      "spark.eventLog.enabled": "true",
      "spark.driver.memory": "4G",
      "spark.driver.cores": 1,
      "spark.executor.memory": "4G",
      "spark.executor.cores": 1,
      "ae.spark.executor.count": 1
    },
    "env": {
      "SAMPLE_ENV_KEY": "SAMPLE_VALUE"
    }
  }
}

如果提交作业成功，那么返回的响应为：

{
  "application_id": "<application_id>",
  "state": "ACCEPTED"
}

提示:

• 保存返回的 "application_id" 值以获取作业的状态或停止作业。
• 保存返回值 "spark_application_id" 以监视和分析 Spark 历史记录服务器上的 Spark 应用程序。

{
  "application_details": {
    "application": "/opt/ibm/spark/examples/src/main/python/wordcount.py",
    "arguments": [
      "/opt/ibm/spark/examples/src/main/resources/people.txt"
    ],
    "runtime": {
      "spark_version": "3.4"
    }
  }
}

{
  "volumes": [
    {
      "name": "cpd-instance::myapp-vol",
      "mount_path": "/my-app"
    }
  ],
  "application_details": {
    "application": "/my-app/python-spark-pi.py",
    "packages":
      "org.apache.spark:spark-sql-kafka-0-10_2.12:3.4.0,org.apache.spark:spark-streaming-kafka-0-10_2.12:3.4.0",
    "conf": {
      "spark.app.name": "MyJob",
      "spark.eventLog.enabled": "true",
      "spark.driver.cores": 4,
      "spark.driver.memory": "8G",
      "spark.executor.memory": "2G",
      "spark.executor.cores": 4,
      "ae.spark.driver.log.level": "ERROR",
      "ae.spark.executor.log.level": "WARN"
    }
  }
}