Setting up an external Spark client
StacyPedersen 270006DB3S Visits (6317)
Author: Jenna Lau Caruso
In IBM Spectrum Conductor with Spark 2.2.1, users can now download the files that are required to configure an external Spark client. This allows users to submit Spark applications from a host outside of their IBM Spectrum Conductor with Spark cluster. A Spark client allows users to submit Spark applications to a Spark instance group from the command line, by using the spark-submit command. Within the cluster, the deploy directory of a Spark instance group acts as a client on each host where Spark has been depl
A Spark client has two major components:
Depending on your Spark instance group’s configuration, you might also require some additional components:
You might occasionally need to download new versions of your external client files. If your Spark instance group is updated to a new package version, you will need to refresh the Spark binary package and the Spark instance group configuration package. If your Spark instance group is modified with configuration changes, you will need to refresh the Spark instance group configuration package.
To set up an external Spark client in the cluster management console, select the Manage > Set up an external client option from the Spark instance group overview page, and then follow the on-screen instructions:
Alternatively, you can use RESTful APIs to download and configure the Spark client, making it possible to script the download and configuration process. For more information on the Spark client RESTful APIs, access the RESTful API documentation in your cluster:
Once you have configured your external Spark client, you are ready to run Spark applications. Applications can be run in either client or cluster mode. For applications that are submitted in cluster mode, the Spark driver will be run on a host inside the cluster. For applications submitted in client mode, the Spark driver will be run on the external client host.
For more information see: