Accessing the Spark history server
The Spark history server is a web UI where you can view the status of running and completed Spark jobs on a provisioned instance of Analytics Engine powered by Apache Spark. If you want to analyse how different stages of your Spark job performed, you can view the details in the Spark history server UI.
The history server shows only the running and the completed or completed but failed Spark jobs. It doesn’t show the jobs for which the Spark application couldn’t be started because, for example, the wrong arguments were passed or the number of passed arguments is less than the expected number.
To access the link to the Spark history server for your provisioned instance:
- From the Navigation menu on the IBM Cloud Pak for Data web user interface, click Services > Instances, find the instance and click it to view the instance details.
- Copy and save the links to the Spark history server and the view history server endpoints. You also need an access token for the service instance.
- Start the history server with the saved Spark history server endpoint and access token. See Starting history server.
- Paste the view history server endpoint in a new tab in the same Cloud Pak for Data browser window to view the history server UI.
- Stop the history server when you no longer need it to release unnecessary resources. See Stopping the history server.