Define the routing of the log output

When using started tasks to start and stop Spark processes, the routing of the log output is dependent upon a combination of settings in the $SPARK_CONF_DIR/log4j.properties file and the values for the STDERR or STDOUT DD cards in the master and worker procedures.

For example, using the default values in the $SPARK_CONF_DIR/log4j.properties file, as shown in the following sample, will route the log output to the value in the STDERR DD card:

log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err

Using the default values for both the $SPARK_CONF_DIR/log4j.properties file and the STDERR DD card of //STDERR DD SYSOUT=&SOUT directs both the JZOS output and the Spark output to SYSOUT.

Changing the log4j.appender.console.target value to System.out will route the log output to the value in the STDOUT DD card.

Alternatively, to route the Spark log output to a USS file in the $SPARK_LOG_DIR directory, if you are using the default values for the $SPARK_CONF_DIR/log4j.properties file, you must specify the full path for the log file by using the value that is set for $SPARK_LOG_DIR on the STDERR DD card. The following examples show the specifications for the master and the worker log files for a cluster that is denoted as Cluster1 where the value of $SPARK_LOG_DIR is /var/spark/logs. You must specify a unique file name for the master and a unique file name for the worker. If you are starting more than one cluster, you must specify unique file names for each master and worker across all clusters.

The following example is for the master procedure.

//STDERR   DD PATH='/var/spark/logs/azkmstrCluster1.err',
//         PATHOPTS=(OWRONLY,OCREAT,OTRUNC),PATHMODE=SIRWXU

The following example is for the worker procedure.

//STDERR   DD PATH='/var/spark/logs/azkwrkrCluster1.err',
//         PATHOPTS=(OWRONLY,OCREAT,OTRUNC),PATHMODE=SIRWXU

By default, the master and worker log files are created with PATHMODE=SIRWXU, which is equivalent to z/OS UNIX file access attributes of “700” (“rwx------“). As such, the user ID that starts/stops the Spark cluster (SPARKID in these examples) has full access to the log files, and no access for anyone else. You may change this behavior by modifying the PATHMODE parameter value. For more information about the PATHMODE parameter, see "PATHMODE parameter" in z/OS MVS JCL Reference.

When using started tasks to start and stop Spark processes, the log files are not rotated as when using other methods for starting the master and worker. It is recommended that you not specify OAPPEND for the PATHOPTS to avoid file growth.