Troubleshooting issues with Apache Spark

Symptom: Spark scripts fail with the following error message:

env: FSUM6196 bash: not executable: EDC5129I No such file or directory

Cause: Apache Spark expects to find the bash shell in the user's PATH environment variable, but bash cannot be found when the spark-script attempts to invoke bash.

Response: Ensure that the PATH environment variable includes the directory where the bash executable is installed. For example, if bash is located at /usr/bin/bash-4.2/bin/bash, ensure that PATH includes/usr/bin/bash-4.2/bin.

Symptom: Spark scripts fail with the following error message:

failed to launch org.apache.spark.deploy.master.Master: 
/usr/lpp/IBM/Spark/bin/spark-class 77: FSUM7332 syntax error: got (, expecting done

Cause: Apache Spark expects to find the env command in /usr/bin, but it cannot be found. Either the /usr/bin/env symbolic link is missing or it is not pointing to /bin/env. It is possible that creation of this symbolic link was missed during Spark setup or that the symbolic link was lost after a system IPL.

Response: Ensure that /usr/bin/env exists and is a symbolic link to /bin/env, and that the symbolic link persists across system IPLs. For more information, see Verifying the env command path.

Symptom: When starting the master, the following error occurs:

bash-4.2$ $SPARK_HOME/sbin/start-master.sh
starting org.apache.spark.deploy.master.Master, 
logging to /u/user/Billy/logs/spark--org.apache.spark.deploy.master.Master-1-ALPS.out
failed to launch org.apache.spark.deploy.master.Master:
full log in /u/user/Billy/logs/spark--org.apache.spark.deploy.master.Master-1-ALPS.out

Cause: Apache Spark polls for a number of seconds to repeatedly check to see if the master started successfully. If your system is under heavy load, this message might appear, but it generally means that the check finished polling for success before the master startup completed.

Response: Check the master log, or issue the ps command and look for the master process to definitively see whether or not the master started successfully.

Symptom: You specified --num-executors 2 on a spark-submit, but only one executor was started.

Cause: The --num-executors parameter is only valid in YARN mode, which is not supported on z/OS®. Instead, the number of executors is determined by your resource settings.

Response: For more information about resource settings, see Configuring memory and CPU options.

Symptom: When running a shell script, it displays unreadable characters on the screen, such as:

./start-master.sh: line 1: syntax error near unexpected token `$'\101\123\124\105\122^''

Cause: Incorrect file encoding or downlevel bash shell.

Response: Ensure that the file encoding is in EBCDIC, not ASCII, and is not tagged as ASCII. You can check the tagging of a file by issuing the ls -T shell command. Also, ensure that your bash shell level is 4.2.53. You can check the bash level by issuing the bash -version command.

Symptom: Spark-shell fails with the following error message:

java.lang.ExceptionInInitializerError …. Scala signature 
package has wrong version  expected: 5.0  found: 45.0 in scala.package

Cause: Your JVM is likely running with the wrong default encoding.

Response: Ensure that you have the following environment variable set:

IBM_JAVA_OPTIONS=-Dfile.encoding=ISO8859-1

For more information about setting environment variables, see Setting up a user ID for use with IBM z/OS Platform for Apache Spark.

Symptom: The Spark master fails to start and gives the following error:

JVMJ9VM015W Initialization error for library j9gc28(2): 
Failed to instantiate compressed references metadata; 200M requested
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Cause: The master JVM could not obtain enough memory to start. Memory is most likely constrained by your ASSIZEMAX setting.

Response: For more information about setting the ASSIZEMAX parameter, see Configuring memory and CPU options.

Symptom: The Spark master and worker started successfully; however, the Spark application is not making any progress, and the following error appears in the executor log:

JVMJ9VM015W Initialization error for library j9gc28(2): 
Failed to instantiate heap; 20G requested
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Cause: The executor JVM could not obtain enough memory to start. Memory is most likely constrained by your MEMLIMIT setting or IEFUSI exit.

Response: For more information about setting the MEMLIMIT parameter or using the IEFUSI exit, see Configuring memory and CPU options.