Spark user interface

The Spark user interface (Spark UI) allows you to track the following aspects of a running Spark application.

  • Current running stage
  • Number of tasks in a stage
  • Reason for a longer running stage
  • Whether the executors in the application are used optimally
  • Inspect memory consumption of the driver and executor

Use the Spark history server to inspect stages of a completed Spark application. To access the Spark history server, see the Access Spark history server.

Procedure

  1. Log in to the watsonx.data console.
  2. From the navigation menu, select Infrastructure manager.
  3. Click the name of the Spark engine (from list view or topology view). The engine information window opens.
  4. In the Applications tab, select an application and click the application ID link. The Spark UI opens. You can view the following details:
    • The Event Timeline displays a graphical view of the timeline and the events.
    • The Active Jobs displays the details such as job ID, duration and tasks involved in the application.