Question & Answer
Question
Can you please provide details around spark client mode vs cluster mode and how it is used in Spectrum? Does spectrum always use client mode?
The PYSPARK_SUBMIT_ARGS variable seems to submit with client mode when we open Jupyter Notebooks. Does this just dictate where the driver runs or is there any influence on whether the job is utilizing all slots available to the SIG? What would be the impact if we submitted using cluster mode?
[{"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SS4H63","label":"IBM Spectrum Conductor"},"Component":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB77","label":"Automation Platform"}}]
Log InLog in to view more of this document
This document has the abstract of a technical article that is available to authorized users once you have logged on. Please use Log in button above to access the full document. After log in, if you do not have the right authorization for this document, there will be instructions on what to do next.
Was this topic helpful?
Document Information
Modified date:
24 December 2019
UID
ibm11163890