Debug the Spark application

The Engine home (storage volume that you create at the time of provisioning the Spark engine) enables you to debug the Spark application and monitor the logs that the Spark application generates. The administrator can access the engine home and debug the Spark application.

Applies to :

Spark engine

Apache Gluten accelerated Spark engine

Before you begin

Before you can monitor Spark applications, ensure that you have completed the following prerequisites:
Required permissions
To debug Spark runtime, you must be the User of engine or user of the storage volume associated with the engine.

Procedure

  1. Log in to Cloud Pack for Data. Click Administration > Storage volumes. The Engine home (storage volume that you create at the time of provisioning the Spark engine) is already listed.
  2. Click the File browser tab. Search for the engine by using the Spark ID (generated at the time of provisioning Spark engine). The log folder with the Spark ID is listed.
  3. Open the folder to see the logs associated with the application that you submitted. You can search by using the application ID that you received at the time of application submission.
  4. To view the results of the application, open the Spark driver log file. You can also download the file to view the details of execution.