Configuring z/OS Spark for WML for z/OS

WML for z/OS leverages the high-performance, general execution engine technology of z/OS Spark, a core component of IBM® Open Data Analytics for z/OS®. Built on Apache Spark, z/OS Spark is capable of large-scale data processing and in-memory computing. You must install and configure z/OS Spark for WML for z/OS.

1. Locate the create-spark-runtime.sh script in the <install_dir_zos>/imlpython/bin directory.
2. Run the create-spark-runtime.sh script as shown below:

   ./create-spark-runtime.sh

   The script will gather needed information and then perform a sequence of tasks to create and configure a Spark runtime environment for WML for z/OS. The script will stop in the event of a fatal error. You must fix the error and rerun the script.

3. When prompted, respond by either making selections or entering requested information:
   1. Before configuring z/OS Spark, the script checks if the Spark installation is at the required PTF (build) level, which is 2.2.0.12 or 2.3.3. If the required level is not satisfied, the script will stop and the configuration process will fail. In this case, apply the required Spark PTF and rerun the script.
   2. Confirm or decline to enable Spark client authentication.
      Important: z/OS Spark includes a client authentication option for securing all connections to the Spark master and REST ports. You can enable, which is the default, or disable the option. If you decide to keep the default setting and enable client authentication, you must also enable security for Jupyter kernel gateway and update the setup of certain system resources. See Customizing your environment for z/OS Spark, Installing and configuring WML for z/OS base on z/OS, and Configuring client authentication for z/OS Spark for details.
   3. Enter IP address for your Spark master.
   4. Enter port number for your Spark master (or enter Y to accept default port 7077).
   5. Enter port number for your Spark master REST API (or enter Y to accept default port 6066).
   6. Enter port number for your Spark web UI (or enter Y to accept default port 8080).

   The IP addresses and port numbers that you entered or accepted are saved in the $IML_HOME/spark/conf/spark-defaults.conf and $IML_HOME/spark/conf/spark-env.sh files.

   After gathering all required information, the script starts the Spark master and worker processes. If the processes are successfully started, you will see a message similar to the following example:

   Starting Spark master...
   Spark master started successfully
   Starting Spark worker...
   Spark worker started successfully
   Congratulations! You have successfully configured and started Spark. 
        Check the parameters used for Spark under $IML_HOME/spark/conf

   If any error occurs, you will be directed to the Spark master and worker configuration logs. Review the logs, fix the error, and start the processes manually.

   1. Issue the following command to start the Spark master process:

      start_master

   2. Issue the following command to start the Spark worker process:

      start_slave

   If needed, add the start_master and start_slave commands to the $IML_HOME/.profile file for the <mlz_setup_userid>.

   By default, the Spark master and worker processes use SPARKM1A and SPARKW1A as their respective job names. You can change these job names by editing the $IML_HOME/.profile file.

   If you've enabled Spark client authentication, the script will not start the Spark master and worker processes for you. You will see a message in the following example:

   Congratulations! You have successfully configured Spark. 
   Customize your system to use AT-TLS for client authentication. 
   Follow instructions at https://www.ibm.com/support/knowledgecenter/
   SS3H8V_1.1.0/com.ibm.izoda.v1r1.azka100/topics/azkic_c_configclientauth.htm. 
   Run start-master.sh to start Spark master and spark-slave.sh to start Spark 
   worker: spark-slave.sh spark://$MLZ_SPARK_HOST:$MLZ_SPARK_PORT.

   You must customize your system to use AT-TLS for client authentication by following instructions in both Configuring client authentication for z/OS Spark and Configuring client authentication for z/OS Spark. After you complete your system setup, start the Spark master and worker manually.

4. Verify that z/OS Spark is successfully configured and started on your system.

   Issue the following command to retrieve the name of the Spark example jar file:

   ls -ls $SPARK_HOME/examples/jars  | grep examples

   Issue the following command to verify the availability of Spark master and Spark worker:

   $SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi 
   --master spark://<host_IP_address>:<sparkMaster-port> 
   $SPARK_HOME/examples/jars/<spark-examples_xxx.jar>

   where xxx is the version of Spark jar file. Spark is properly configured and functions normally if you see a response in the following example:

   Pi is roughly 3.13742.