Spark job failure due to error in payload

When you try to submit a Spark application by using API and if the JSON payload includes an error, the job fails without generating logs.

Symptoms

The Spark job fails.

Resolving the problem

To resolve this, you must run the following GET API call to retrieve the response from the failed Spark job.

curl -k --request  GET  --url "https://$wxd_host_name/lakehouse/api/v2/spark_engines/$engine_id/applications/$app_id" \n
--header "Authorization: Bearer $token" --header "LhInstanceId: $wxd_instance_id"
Parameter values
  • wxd_host_name : The hostname of your IBM Software Hub.
  • engine_id : The Engine ID of the native Spark engine.
  • app_id : The Spark application identifier.
  • token : The bearer token. For information, see Generating a bearer token.
  • wxd_instance_id : The instance ID from the watsonx.data cluster instance URL. For example, 1609968577169454.