Using _BPX_JOBNAME to assign job names to Spark processes

When a new z/OS® UNIX process is started, it runs in a z/OS UNIX initiator (a BPXAS address space). By default, this address space has an assigned job name of userIDx, where userID is the user ID that started the process, and x is a decimal number. You can use the _BPX_JOBNAME environment variable to set the job name of the new process. Assigning a unique job name to each Spark process helps to identify the purpose of the process and makes it easier to group processes into a WLM service class.

The following example shows a portion of a script that assigns different job names for the master and worker processes before starting them:

_BPX_JOBNAME='ODASM1A' /usr/lpp/IBM/izoda/spark/spark23x/sbin/start-master.sh
sleep 5
export _BPX_SHAREAS=NO
_BPX_JOBNAME='ODASW1A' /usr/lpp/IBM/izoda/spark/spark23x/sbin/start-slave.sh spark://127.0.0.1:7077

A system programmer can set the job name of the executor address space by setting spark.executorEnv._BPX_JOBNAME in the spark-defaults.conf configuration file as in the following example:

spark.executorEnv._BPX_JOBNAME    ODASX1A

If the job name of the executor is set in this manner, all Spark executors will have the same job name.

If spark.executorEnv._BPX_JOBNAME is specified in the spark-defaults.conf file, the _BPX_JOBNAME environment variable takes precedence. Only the executor job name or job name prefix specified in the spark-defaults.conf file is honored; any such settings that are specified in an application or on the command line are ignored.

Tip: If you use the Jupyter Kernel Gateway with Apache Toree to access Spark, it is good practice to use the same naming convention for the Toree instances. For instance, if you set _BPX_JOBNAME=ODASD for the Toree instances and set spark.zos.executor.jobname.prefix=ODASX for the executors and you have client authentication enabled, you can then classify the driver and the executors for user Elpida's application by using transaction name and user ID qualifiers, such as TN=ODAS* and UI=Elpida.

In addition to the master, worker, driver, and executors, your Spark environment may have additional processes, such as the Spark history server. You can also use _BPX_JOBNAME to assign unique job names to those processes.

Setting _BPX_JOBNAME requires appropriate privileges. For more information about _BPX environment variables, see z/OS UNIX System Services Planning.