Spark applications

You can manage Spark applications that are associated with an instance group.

Spark application workflow
Spark applications include a driver program and executors, and runs various parallel operations in the cluster. There are two types of Spark applications: Spark notebook applications and Spark batch applications.
Spark application access based on user role
Before submitting Spark batch applications or launching notebooks that submit Spark notebook applications to an instance group, understand user roles that impact application access and control.
Spark application states
When you submit Spark batch applications or launch Spark notebook applications, the applications and drivers go through many states.
Submitting Spark batch applications
Submit Spark workload by submitting Spark batch applications by using the cluster management console, RESTful APIs, or the CLI.
Submitting Spark applications that access an Amazon Simple Storage Service (Amazon S3) file system
If you have an Amazon Simple Storage Service (Amazon S3) cloud storage file system enabled, you can configure IBM Spectrum Conductor to access your Amazon S3 file system when submitting Spark applications. This configuration involves adding the Amazon S3-specific access files to your instance group and setting Spark submission parameters.
Scheduling Spark batch applications
Schedule Spark batch applications, enabling Spark batch applications to run on an instance group according to a schedule. You can schedule Spark batch applications to run at a particular time or to run periodically at specific intervals.
Monitoring Spark applications
Monitor Spark applications to check status and to view information about running tasks and resource usage.
Debugging Spark applications
If an application is in the Error state, you can debug the application from the Instance Groups page in the cluster management console.
Stopping Spark batch applications
If required, stop a Spark batch application. You may want to stop a Spark batch application, for example, to fix an error that does not let an application run correctly.
Creating shared Spark batch applications
Create shared Spark batch applications where multiple users can submit Spark jobs to an application and use the same Resilient Distributed Datasets (RDDs).
Spark application performance tuning
Use performance tuning methodologies and approaches to enhance Spark application performance in your cluster.

Parent topic: Workload