Monitoring and debugging Spark applications from Spark labs

Before you begin

  1. Install a desktop version of Visual Studio Code.
  2. Ensure that you have public-private SSH key pair to establish an SSH connection with the Spark lab. For more information about generating the key, open watsonx.data extension in Visual Studio Code, go to Details tab, see Set up SSH on your machine section.
  3. Install the extension Remote - SSH from Visual Studio Code marketplace.

About this task

Debugging the Spark application from Spark labs
The Spark lab also allows you to debug the Spark application that you submit. To do that:
  1. Go to Visual Studio Code > Extensions.
  2. Browse for the debugging tool to debug the code. For each Spark application type (Python, Java, Scalar, R), you need to choose the official extension to debug. For example, if you submit a Spark application that is written in Python language, install Python extension.
  3. After you install the debugging tool extension, open the file that you want to debug and click Run and Debug from the Visual Studio Code.
  4. The Visual Studio Code window prompts for the language of the Spark application code and default configuration.
  5. Select the language and provide the default configuration based on your Spark application type (Python, Java, Scalar, R).
  6. Click Run and Debug. The debugging process starts and you can view the result in the Terminal.
Accessing Spark UI from Spark labs
The Spark user interface (UI) allows you to monitor various aspects of running a Spark application. For more information, see Spark user interface. Expose Spark UI to access it from Spark labs. To do that:

  1. Go to Visual Studio Code > Terminal and select the Ports tab.

  2. Click Forward a Port. Type 4440 and press Enter. You can now access Spark UI from Spark labs. Open a web browser and enter the URL in the format - localhost:4440. The Spark UI opens, which allows you to inspect Spark applications in the Spark labs. You can view the following details :

    • The Event timeline displays a graphical view of the timeline and the events.
    • Different stages of execution, the storage used, the Spark environment, and executor (memory and driver) details.