Run a Spark application on LSF
Use the bsub command to submit the Spark application to LSF.
Procedure
-
Use the bsub command to submit the Spark application to LSF
through the lsf-spark-submit.sh connector script.
bsub bsub_options lsf-spark-submit.sh spark-submit_options
For example, the following job requests 16 tasks from two hosts, eight tasks on each host, in interactive mode, and specifies hostA as the first execution host (that is, where the driver program starts). In the background, LSF starts the Spark driver and worker using the LSF blaunch framework.
bsub -m "hostA! others" -n16 -R "span[ptile=8]" -I lsf-spark-submit.sh --class "SimpleApp" target/scala-2.10/simple-project_2.10-1.0.jar ../myfile.txt
-
To view the job in the Spark GUI, navigate to the URL of the first execution host using a web
browser.
http://first_execution_hostname:port_number
where first_execution_hostname is the host name of the first execution host and port_number is the port number set using the SPARK_MASTER_PORT environment variable in the lsf-spark-submit.sh connector script file.
For example, http://hostA:7077