Run a Spark application on LSF

Use the bsub command to submit the Spark application to LSF.

Procedure

  1. Use the bsub command to submit the Spark application to LSF through the lsf-spark-submit.sh connector script.

    bsub bsub_options lsf-spark-submit.sh spark-submit_options

    For example, the following job requests 16 tasks from two hosts, eight tasks on each host, in interactive mode, and specifies hostA as the first execution host (that is, where the driver program starts). In the background, LSF starts the Spark driver and worker using the LSF blaunch framework.

    bsub -m "hostA! others" -n16 -R "span[ptile=8]" -I lsf-spark-submit.sh --class "SimpleApp" target/scala-2.10/simple-project_2.10-1.0.jar ../myfile.txt

  2. To view the job in the Spark GUI, navigate to the URL of the first execution host using a web browser.

    http://first_execution_hostname:port_number

    where first_execution_hostname is the host name of the first execution host and port_number is the port number set using the SPARK_MASTER_PORT environment variable in the lsf-spark-submit.sh connector script file.

    For example, http://hostA:7077