View and edit native Spark engine details

You can use the IBM® watsonx.data UI or API to view and edit the native Spark details.

About this task

Viewing native Spark engine details

You can view the native Spark engine details in list and topology views.

Log in to the watsonx.data cluster. Go to the Infrastructure manager page.
Click the name of Spark engine (either from list or topology view). Engine information window opens.

In the Details tab, you can view the following details:

Field	Description
Display name	The Spark engine name.
Engine ID	The unique identifier of the Spark instance.
Description	The description of the engine.
Tags	The tag that is specified at the time of registering an engine.
Default Spark version	The Spark runtime version that is used by default for any application that is submitted to the Spark engine.
Volume	The storage volume associated with the Spark engine.
Type	The engine type. Here, IBM Analytics Engine (Spark).
watsonx.data application endpoint	The endpoint is used at the time of application submission. To submit an application by using API, see API Docs.
Spark engine endpoint	The native Spark endpoint.
Default Spark Configuration	The Spark configuration properties that are applied to any application that is submitted to the Spark engine.

Editing Spark details from Console UI

You can edit the Spark details in list and topology views.

Log in to the watsonx.data cluster. Go to the Infrastructure manager page.
Click the name of Spark engine (either from list or topology view). Engine information window opens.
In the Details tab, click Edit.
In the Display name field, enter the display name for the Spark engine.
In the Description field, enter the description of the engine or edit the existing description.
In the Tags field, select the tags from the list or start typing to define a new tag.
In the Default Spark version field, select the Spark runtime version that must be considered for processing the applications.
In the Default Spark configuration field, click Edit configuration link to update the default Spark configuration. For more information about different properties, see Available Properties.
- Enter the key-value pair for the Spark configuration that applies to all applications.
- Click Apply.
Click Save and click the name of Spark engine (either from list or topology view). Engine information window opens.

Editing Spark details by using API

Use the following curl command to edit the default Spark configuration and Spark version.
Note: You can update the curl command to include or modify the description and tag also.
```
curl --request PATCH \
  --url https://<cpd_host_name>/lakehouse/api/v2/spark_engines/<spark_engine_id> \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/merge-patch+json' \
  --header 'LhInstanceId: <instance_id>' \
  --data '{
  "engine_details": {
    "default_config": <map_of_spark_properties>,
    "default_version": "<spark_version>"
  }
}'
```
Use the curl command to update the Spark engine details like tags, description, default configuration, and Spark version.
Following are the details of the parameter values to be used in the curl command.
- <cpd_host_name>: The hostname of your IBM Software Hub.
- <spark_engine_id> : The Engine ID of the native Spark engine.
- <token> : The bearer token. For more information about generating the token, see Generating a bearer token.
- <instance_id> : The instance ID from the watsonx.data cluster instance URL. For example, 1709968977177454.
- <map_of_spark_properties> : Specify the Spark properties in the form of key-value pair ("<property_name>": "<property_value>") separated by comma.
  - <property_name>: The default configuration property name. For more information about different properties, see Available Properties.
  - <property_value>: The value that must be configured for the property. For more information about different properties, see Available Properties.
- <spark_version>: The Spark runtime. Possible value is 3.4.
Example:
```
curl --request PATCH \
  --url https://<cpd_host_name>/lakehouse/api/v2/spark_engines/<spark_engine_id> \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/merge-patch+json' \
  --header 'LhInstanceId: 1709968977177454' \
  --data '{
  "engine_details": {
    "default_config": {
      "spark.driver.cores": "1",
      "spark.driver.memory": "4g"
    },
    "default_version": "3.4"
  }
}'
```
Note: To add new properties to the default_config parameter, or update existing properties, specify the property name and value. To delete a property, specify the property name and the set the value as NULL.

Related API: For information on related API, see