Submitting Spark application by using native Spark engine

You can submit a Spark application by running a CURL command. Complete the following steps to submit a Python application.

Procedure

Create a storage volume to store the Spark application and related output.
- Option1: Create a storage volume in Cloud Pak for Data cluster. To create storage volume in Cloud Pak for Data cluster, see Creating a storage volume.
- Option2: Create Cloud Object Storage. To create Cloud Object Storage and a bucket, see Creating a storage bucket.
If you use Cloud Object Storage, register the Cloud Object Storage in watsonx.data, register Cloud Object Storage bucket. To register Cloud Object Storage bucket, see Adding bucket catalog pair.
Upload the Spark application to the storage volume.
- If you use Cloud Pak for Data storage volume, see Uploading data.
- If you use Cloud Object Storage, see Uploading data.
If your Spark application resides in Cloud Pak for Data storage volume, specify the parameter values and run the following CURL command to submit the application.
```
curl --request POST \
  --url https://<cpd_host_name>/lakehouse/api/v2/spark_engines/<spark_engine_id>/applications \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --header 'LhInstanceId: <instance_id>' \
  --data '{
  "application_details": {
    "application": "/myapp/<python file name>"
  },
  "volumes": [
    {
      "name": "cpd-instance::my-vol-1",
      "mount_path": "/myapp"
    }
  ]
}'
```
Parameter values:
- <cpd_host_name>: The hostname of your Cloud Pak for Data cluster.
- <spark_engine_id> : The Engine ID of the native Spark engine.
- <token> : The bearer token. For more information about generating the token, see Generating a bearer token.
- <instance_id> : The instance ID from the watsonx.data cluster instance URL. For example, 1609968577169454.
- <python file name> : The Spark application file name. It must be available in the storage volume.
- <my-vol-1> : The display name of the storage volume.
- <python file name> : The Spark application file name.
Submitting an application by accessing the watsonx.data catalog
Use the following command to access data from a catalog associated with the Spark engine and to perform some basic operations on that catalog.
Example:
```
curl --request POST \
  --url https://<cpd_host_name>/lakehouse/api/v2/spark_engines/<spark_engine_id>/applications \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --header 'LhInstanceId: <instance_id>' \
  --data '{
  "application_details": {
"conf": {
"spark.hadoop.wxd.apiKey":"ZenApiKey <encoded key>"
},
"application": "s3a://<application-bucket-name>/iceberg.py"
}
}
```
Parameter values:
- <cpd_host_name>: The hostname of your Cloud Pak for Data cluster.
- <spark_engine_id> : The Engine ID of the native Spark engine.
- <token> : The bearer token. For more information about generating the token, see Generating a bearer token.
- <instance_id> : The instance ID from the watsonx.data cluster instance URL. For example, 1609968977179454.
- <user-authentication-string> : The value must be in the format : echo -n "<username>:<your Zen API key>" | base64. The Zen API Key here is the API key of the user accessing the Object store bucket. To generate API key, log in into the watsonx.data console and navigate to Profile > Profile and Settings > API Keys and generate a new API key.
  Note: If you generate a new API key, your old API key becomes invalid.
- <application-bucket-name> : The display name of the storage volume.
If your Spark application resides in Cloud Object Storage and you want to submit the application without using DAS, specify the parameter values and run the following curl command. The following example shows the command to submit read.py application.
Example 1:
```
curl --request POST \
  --url https://<cpd_host_name>/lakehouse/api/v2/spark_engines/<spark_engine_id>/applications \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --header 'LhInstanceId: <instance_id>' \
  --data '{
  "application_details": {
    "application": "s3a://<s3_bucket_name>/cos-read.py",
    "conf": {
        "spark.hadoop.fs.s3a.bucket.<s3_bucket_name>.endpoint": "<cos_endpoint>",
        "spark.hadoop.fs.s3a.bucket.<s3_bucket_name>.access.key": "<s3 bucket HMAC access key>",
        "spark.hadoop.fs.s3a.bucket.<s3_bucket_name>.secret.key": "<s3 bucket  HMAC secret key>",
        "spark.app.name": "reader-app"
    }
  }
}'
```
Parameter values:
- <cpd_host_name>: The hostname of your Cloud Pak for Data cluster.
- <spark_engine_id> : The Engine ID of the native Spark engine.
- <token> : The bearer token. For more information about generating the token, see Generating a bearer token.
- <instance_id> : The instance ID from the watsonx.data cluster instance URL. For example, 1609968977179454.
- <COS_bucket_name> : The name of the Cloud Object Storage.
- <cos_endpoint>: The public endpoint of the Cloud Object Storage bucket. For example, s3.direct.us-south.cloud-object-storage.appdomain.cloud.
- <Cloud Object storage HMAC access key> : The access key for Cloud Object storage. For more information, see Create HMAC credentials using the CLI.
- <Cloud Object storage HMAC secret key> : The secret key for Cloud Object storage. For more information, see Create HMAC credentials using the CLI.

If your Spark application resides in ADLS (Gen1 or Gen2) and you want to submit the application without using DAS, specify the parameter values and run the following curl command. The following example shows the command to submit read.py application.

Example 1 :

curl --request POST \
  --url https://<cpd_host_name>/lakehouse/api/v2/spark_engines/<spark_engine_id>/applications \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --header 'LhInstanceId: <instance_id>' \
  --data '{
  "application_details": {
    "application": "abfss://<storage_account>@<storage_container>.dfs.core.windows.net/adls-read.py",
    "conf": {
        "spark.hadoop.fs.azure.account.auth.type.<storage_account>.dfs.core.windows.net", "OAuth",
        "spark.hadoop.fs.azure.account.oauth.provider.type.<storage_account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
        "spark.hadoop.fs.azure.account.oauth2.client.id.<storage_account>.dfs.core.windows.net", "<application_id>",
        "spark.hadoop.fs.azure.account.oauth2.client.secret.<storage_account>.dfs.core.windows.net","<secret>",
        "spark.hadoop.fs.azure.account.oauth2.client.endpoint.<storage_account>.dfs.core.windows.net", "https://login.microsoftonline.com/<directory_id>/oauth2/token",
        "spark.hadoop.fs.azure.createRemoteFileSystemDuringInitialization", "false",
        "spark.app.name": "reader-app",
    }
  }
}'

Parameter values:

<cpd_host_name>: The hostname of your Cloud Pak for Data cluster.
<spark_engine_id> : The Engine ID of the native Spark engine.
<token> : The bearer token. For more information about generating the token, see Generating a bearer token.
<instance_id> : The instance ID from the watsonx.data cluster instance URL. For example, 1609968977179454.
<storage_account> : The name of the azure storage account.
<storage_container> : The name of the Azure storage container.
<application_id> : The Application ID of the ServicePrincipal.
<secret>: The Client Secret of the ServicePrincipal. For more information, see Create a service principal.
<directory_id> : The Directory ID of the ServicePrincipal. For more information, see Create a service principal.

If your Spark application resides in ADLS(Gen1 or Gen2) and you want to submit the application by using DAS, specify the parameter values and run the following curl command. The following example shows the command to submit read.py application.

Example 1 :

curl --request POST \
  --url https://<cpd_host_name>/lakehouse/api/v2/spark_engines/<spark_engine_id>/applications \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --header 'LhInstanceId: <instance_id>' \
  --data '{
  "application_details": {
    "application": "abfss://<storage_account>@<storage_container>.dfs.core.windows.net/adls-read.py",
    "conf": {
        "spark.hadoop.fs.azure.account.auth.type.<storage_account>.dfs.core.windows.net", "OAuth",
        "spark.hadoop.fs.azure.account.oauth.provider.type.<storage_account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
        "spark.hadoop.fs.azure.account.oauth2.client.id.<storage_account>.dfs.core.windows.net", "<application_id>",
        "spark.hadoop.fs.azure.account.oauth2.client.secret.<storage_account>.dfs.core.windows.net","<secret>",
        "spark.hadoop.fs.azure.account.oauth2.client.endpoint.<storage_account>.dfs.core.windows.net", "https://login.microsoftonline.com/<directory_id>/oauth2/token",
        "spark.hadoop.fs.azure.createRemoteFileSystemDuringInitialization", "false",
        "spark.app.name": "reader-app",
    }
  }
}'

Parameter values:

<cpd_host_name>: The hostname of your Cloud Pak for Data cluster.
<spark_engine_id> : The Engine ID of the native Spark engine.
<token> : The bearer token. For more information about generating the token, see Generating a bearer token.
<instance_id> : The instance ID from the watsonx.data cluster instance URL. For example, 1609968977179454.
<storage_account> : The name of the azure storage account.
<storage_container> : The name of the Azure storage container.
<application_id> : The Application ID of the ServicePrincipal.
<secret>: The Client Secret of the ServicePrincipal. For more information, see Create a service principal.
<directory_id> : The Directory ID of the ServicePrincipal. For more information, see Create a service principal.

If your Spark application resides in Google Cloud Storage, specify the parameter values and run the following curl command. The following example shows the command to submit gcs-read.py application.
Example :
```
curl --request POST   --url https://<region>.lakehouse.cloud.ibm.com/lakehouse/api/v2/spark_engines/<spark_engine_id>/applications   --header 'Authorization: Bearer <token>'   --header 'Content-Type: application/json'   --header 'AuthInstanceID: <crn_instance>'   --data '{
  "application_details": {
    "application": "gs://{bucket_name}//gcs-read.py",
    "conf": {
        "spark._jsc.hadoopConfiguration().set("google.cloud.auth.service.account.json.keyfile","<json_keyfile>")
        "spark.app.name":"GCSFilesRead"
    }
  }
}
```
Parameter values:
- <region>: The region where the Spark instance is provisioned..
- <spark_engine_id> : The Engine ID of the native Spark engine.
- <token> : The bearer token. For more information about generating the token, see Generating a bearer token.
- <crn_instance> : The instance ID from the watsonx.data cluster instance URL. For example, 1609968977179454.
- <json_keyfile> : The path to the json keyfile generated.
After you submit the Spark application, you receive a confirmation message with the application ID and Spark version. Save it for reference.
Log in to the watsonx.data cluster, access the Engine details page. In the Applications tab, use the application ID to list the application and you can track the stages. For more information, see View and manage applications.