Spark application states

When you submit Spark batch applications or launch Spark notebook applications, the applications and drivers go through many states.

A Spark application includes a driver program and executors, and runs various parallel operations in the cluster. During Spark batch application submission or when the notebook submits the Spark notebook application, the Spark driver is submitted to the Spark master before the application submission process is started.

A Spark driver can have the following states:
Table 1. Spark driver states
State Description
Submitted Indicates that the driver is submitted and accepted by the Spark master.
Running Indicates that driver execution is in progress.
Killed Indicates that the driver execution was stopped (killed) manually.
Failed Indicates that driver execution failed. Check the spark-submit command syntax for any errors.
Error Indicates errors during driver execution. Check the logs for more details.
Relaunching Indicates that the driver is relaunching. A driver can reach the Relaunching state only from the Error, Failed, or Reclaimed state. The Relaunching state applies only to Spark version 1.6.1 or higher.
Reclaimed Indicates that the driver was reclaimed. The Reclaimed state applies only to Spark version 1.6.1 or higher. For Spark version 1.5.2, when the driver is reclaimed the state is Killed.
Finished Indicates that driver execution is complete. For Spark executors, note that this state is EXITED when the executor tasks finish without errors.
Unknown When the Spark master is in the high availability recovery process, indicates that the driver status reporting process is not yet started.
A Spark application can have the following states:
Table 2. Spark application states
State Description
Waiting Indicates that application execution is waiting, because the application cannot acquire the required resources. The application remains in this state until more resources become available.
Running Indicates that application execution is in progress.
Killed Indicates that the application execution was stopped (killed) manually.
Failed Indicates that application execution failed. Check the logs for any errors and for more details.
Reclaimed Indicates that the application was reclaimed. The Reclaimed state applies only to Spark version 1.6.1 or higher. For Spark version 1.5.2, when the application is reclaimed the state is Killed.
Finished Indicates that application execution is complete. For Spark executors, note that this state is EXITED when the executor tasks finish without errors.
Unknown When the Spark master is in the high availability recovery process, indicates that the application status reporting process is not yet started.