Installing IBM Open Data Analytics for z/OS

You can install IBM® Open Data Analytics for z/OS® (IzODA) by using CBPDO or, alternatively, SystemPac or ServerPac.

Before you begin

Ensure that the following software requirements for Open Data Analytics for z/OS have been met:
  • IBM z/OS V2.1 or later
  • The minimum required Java™ level is IBM 64-Bit SDK for z/OS Java Technology Edition V8, Service Refresh 7, FP 10. However, if the RELEASE file in the Spark installation directory indicates that the product was built with a later Java level, IBM urges you to use that Java level.
  • Bourne Again Shell (bash) version 4.2.53 or version 4.3.48.
For the latest list of requirements, see the information in the Preventive Service Planning (PSP) bucket.
Migration notes: If you already use IBM z/OS Platform for Apache Spark, note the following differences in Open Data Analytics for z/OS:
  • IzODA changes the level of Apache Spark. For more information, see Migrating to a new version of Apache Spark.
  • IzODA changes the default z/OS Spark installation directory to /usr/lpp/IBM/zspark/spark/sparknnn (for instance, /usr/lpp/IBM/zspark/spark/spark32x).
  • IzODA uses UTF-8 encoding. For details, see Setting up a user ID for use with z/OS Spark and Network port configurations.
  • As of the December 2018 release, the Spark REST server port is disabled. You can enable connections to the REST port (such as when using cluster deploy mode) in your local spark-defaults.conf file, but the port will not function properly until you complete the setup to secure and enable the REST port.. For details, see Configuring networking for Apache Spark.
  • IzODA introduces client authentication, which is enabled by default and requires additional setup. Apache Spark will not function properly until you complete the setup for client authentication or disable the client authentication function. For details, see Configuring z/OS Spark client authentication.
  • IzODA changes the way you assign job names to executor and driver processes. IzODA no longer honors the specification of spark.executorEnv._BPX_JOBNAME on the command line or in an application. For details, see Assigning job names to Spark processes.
  • If the PTF for APAR PI93605 is installed, Spark master and worker daemons will perform environment verification during initialization and will fail to start if the verification fails. The reason for termination can be found in the daemon's log. You can disable this feature by setting the spark.zos.environment.verify to false in spark-defaults.conf.
  • z/OS IzODA Livy (Livy) is delivered through Anaconda (HANA11) and introduced in APAR (PH11339). The SMP/E APPLY process installs the Livy package into the Anaconda directory, but does not make them available for use. See z/OS IzODA Livy Installation and Customization for instructions on how to get started with using the z/OS IzODA Livy package.
Additional migration note: If you are installing the PTF for APAR PI89136, note the following changes that are introduced by the APAR:
  • APAR PI89136 changes the level of Apache Spark to 2.2.0. For more information, see Migrating to a new version of Apache Spark.
  • If you specify an incorrect job name prefix, Spark worker daemon will fail rather than ignoring the error. For more information, see Assigning job names to Spark processes.
  • If client authentication is enabled, and you submit an application to the Spark master port in cluster-deploy-mode, then the Spark driver will run under the ID of the user who did the submit.
Note: Open Data Analytics for z/OS currently has some restrictions on Apache Spark functionalities. For a list of restrictions, see Restrictions.
Note: A new service (PTF) level for Open Data Analytics for z/OS (FMID HSPK120) might provide a new version of Apache Spark. Before installing a new PTF, see Migrating to a new version of Apache Spark.

About this task

Open Data Analytics for z/OS is supplied in a Custom-Built Product Delivery Offering (CBPDO, 5751-CS3). For installation instructions, see Program Directory for IBM Open Data Analytics for z/OS.

You can also install Open Data Analytics for z/OS with a SystemPac or ServerPac. For information about the various z/OS product installation offerings, see z/OS Planning for Installation.

Service updates for Open Data Analytics for z/OS are provided as PTFs that perform a full replacement of the product. Therefore, you can use a PTF to update your existing installation or to perform a new installation.

IBM recommends that you mount your z/OS Spark file system from the same system on which the Spark cluster will be run.

Procedure

Complete the following steps to install Open Data Analytics for z/OS on your system.

  1. Choose the most appropriate method for installing Open Data Analytics for z/OS.
  2. Use the information in Program Directory for IBM Open Data Analytics for z/OS to install Open Data Analytics for z/OS on your system.

Results

Open Data Analytics for z/OS is installed on your z/OS system.

What to do next

Before you use Open Data Analytics for z/OS for the first time, follow the customization instructions in Customization.