Spark web interfaces
Apache Spark provides a suite of web user interfaces (UIs) that you can use to monitor the status and resource consumption of your Spark cluster.
- Master web UI
- Worker web UI
- Application web UI
Some sections of the web UIs only appear if relevant information is available. For instance, the master web UI omits the driver information section if a driver has not been running in the cluster.
http://host_IP:ui_port- host_IP
- The IP address for the z/OS® system on which Spark runs
- ui_port
- The Spark web UI port
number
For more information about where to find the port numbers, see Configuring networking for Apache Spark.
Master web UI
- Master URL and REST URL
- CPUs and memory available to the Spark cluster
- Worker status and its allotted resources
- Information about the active and completed applications, such as their status, allotted resources, and duration
- Information about the active and completed drivers, such as their status and allotted resources
You can use the master web UI to identify the amount of CPU and memory resources that are allotted to the Spark cluster and to each application. Spark sees each SMT-enabled zIIP as having two cores.
The master web UI also provides an overview of the applications. For instance, if an application is in the WAITING state, it likely does not have sufficient resources to run.
Figure 1 shows an example of the master web UI.
Worker web UI
The worker web UI provides an overview of the executors and drivers that are spawned by the worker process. It displays information about the allotted resources, status, and provides links to log files for each of these child processes. You can use this web UI to see how many executors are currently running and the amount of resources allotted to each.
Figure 2 shows an example of the worker web UI.
Application web UI
- The Jobs tab displays a summary page of all jobs in the Spark application and a detailed page for each job. The summary page shows high-level information, such as the status, duration, and progress of all jobs and the overall event timeline. When you click on a job on the summary page, you see the detailed page for that job. The detailed page further shows the event timeline, DAG visualization, and all stages of the job.
- The Stage tab displays a summary page that shows the current state of all stages of all jobs in the Spark application, and, when you click on a stage, a detailed page for that stage. The details page shows the event timeline, DAG visualization, and all tasks for the stage.
- If the application has persisted RDDs, the Storage tab displays information about the RDDs. The summary page shows the storage levels, sizes and partitions of all RDDs, and the detailed page shows the sizes and using executors for all partitions in an RDD.
- The Environment tab displays the values for the different environment and configuration variables, including Java™, Spark, and system properties.
- The Executors tab displays summary information about the executors that were created for the application, including memory and disk usage and task and shuffle information. The Storage Memory column shows the amount of memory used and reserved for caching data.
- If the application executes Spark SQL queries, the SQL tab displays information, such as the duration, jobs, and physical and logical plans for the queries.
- The web UI includes a Streaming tab if the application uses Spark streaming. This tab displays scheduling delay and processing time for each micro-batch in the data stream, which can be useful for troubleshooting the streaming application.
From a resource monitoring perspective, the Executors tab provides information about the amount of memory, disk, and cores used by each executor for the application.
From a performance monitoring perspective, the Executors tab displays garbage collection (GC) time and shuffle information for each executor. The Storage tab displays the percentage of RDD data that can be cached in memory.
Figure 3 shows an example of the worker web UI.
