Installing IBM Open Data Analytics for z/OS
You can install IBM® Open Data Analytics for z/OS® (IzODA) by using CBPDO or, alternatively, SystemPac or ServerPac.
Before you begin
- IBM z/OS V2.1 or later
- The minimum required Java™ level is IBM 64-Bit SDK for z/OS Java Technology Edition V8, Service Refresh 7, FP 10. However, if the RELEASE file in the Spark installation directory indicates that the product was built with a later Java level, IBM urges you to use that Java level.
- Bourne Again Shell (bash) version 4.2.53 or version 4.3.48.
- IzODA changes the level of Apache Spark. For more information, see Migrating to a new version of Apache Spark.
- IzODA changes the default z/OS Spark installation directory to /usr/lpp/IBM/zspark/spark/sparknnn (for instance, /usr/lpp/IBM/zspark/spark/spark32x).
- IzODA uses UTF-8 encoding. For details, see Setting up a user ID for use with z/OS Spark and Network port configurations.
- As of the December 2018 release, the Spark REST server port is disabled. You can enable connections to the REST port (such as when using cluster deploy mode) in your local spark-defaults.conf file, but the port will not function properly until you complete the setup to secure and enable the REST port.. For details, see Configuring networking for Apache Spark.
- IzODA introduces client authentication, which is enabled by default and requires additional setup. Apache Spark will not function properly until you complete the setup for client authentication or disable the client authentication function. For details, see Configuring z/OS Spark client authentication.
- IzODA changes the way you assign job names to executor and driver processes. IzODA no longer honors the specification of spark.executorEnv._BPX_JOBNAME on the command line or in an application. For details, see Assigning job names to Spark processes.
- If the PTF for APAR PI93605 is installed, Spark master and worker daemons will
perform environment verification during initialization and will fail to start if the verification
fails. The reason for termination can be found in the daemon's log. You can disable this feature by
setting the
spark.zos.environment.verify
tofalse
in spark-defaults.conf. - z/OS IzODA Livy (Livy) is delivered through Anaconda (HANA11) and introduced in APAR (PH11339). The SMP/E APPLY process installs the Livy package into the Anaconda directory, but does not make them available for use. See z/OS IzODA Livy Installation and Customization for instructions on how to get started with using the z/OS IzODA Livy package.
- APAR PI89136 changes the level of Apache Spark to 2.2.0. For more information, see Migrating to a new version of Apache Spark.
- If you specify an incorrect job name prefix, Spark worker daemon will fail rather than ignoring the error. For more information, see Assigning job names to Spark processes.
- If client authentication is enabled, and you submit an application to the Spark master port in cluster-deploy-mode, then the Spark driver will run under the ID of the user who did the submit.
About this task
Open Data Analytics for z/OS is supplied in a Custom-Built Product Delivery Offering (CBPDO, 5751-CS3). For installation instructions, see Program Directory for IBM Open Data Analytics for z/OS.
You can also install Open Data Analytics for z/OS with a SystemPac or ServerPac. For information about the various z/OS product installation offerings, see z/OS Planning for Installation.
Service updates for Open Data Analytics for z/OS are provided as PTFs that perform a full replacement of the product. Therefore, you can use a PTF to update your existing installation or to perform a new installation.
IBM recommends that you mount your z/OS Spark file system from the same system on which the Spark cluster will be run.
Procedure
Complete the following steps to install Open Data Analytics for z/OS on your system.