PI50564: CLASSNOTFOUNDEXCEPTION ERROR WHEN RUNNING PYSPARK, SPARK-SHELL OR SPARK-SUBMIT ON A NODE WHICH HAS SPARK CLIENTS INSTALLED

APAR status

• Closed as fixed if next.

Error description

• ClassNotFoundException Error when running pyspark, spark-shell
  or spark-submit on a node which has spark clients installed
  
  
  Error:
  [spark@cdtsbimn0106 ?]$ /usr/bin/pyspark
  Exception in thread "main" java.lang.NoClassDefFoundError:
  org/apache/spark/launcher/Main
  Caused by: java.lang.ClassNotFoundException:
  org.apache.spark.launcher.
  Main
           at
  java.net.URLClassLoader$1.run(URLClassLoader.java:217)
           at java.security.AccessController.doPrivileged(Native
  Method)
           at
  java.net.URLClassLoader.findClass(URLClassLoader.java:205)
           at
  java.lang.ClassLoader.loadClass(ClassLoader.java:323)
           at
  sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
           at
  java.lang.ClassLoader.loadClass(ClassLoader.java:268)
  Could not find the main class: org.apache.spark.launcher.Main.
  Program
  will exit.
  [spark@cdtsbimn0106 ?]$ /usr/iop/4.1.0.0/spark/bin/spark-submit
  Exception in thread "main" java.lang.NoClassDefFoundError:
  org/apache/spark/launcher/Main
  Caused by: java.lang.ClassNotFoundException:
  org.apache.spark.launcher.
  Main
  

Local fix

• manuall set JAVA_HOME environment variable and point to java
  which got installed with IOP
  

Problem summary

• See error description
  

Problem conclusion

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• n/a
  

Fix information

Applicable component levels

• R425 PSY

     UP