Summary of changes for IBM Open Data Analytics for z/OS Installation and Customization Guide

New

The following information is new.

August 2022

• Changes throughout the manual for software prerequisite of z/OS Java Version 8, SR7 FP10 or higher

December 2021

• Changes throughout the manual for Spark version 2.4.8.2

September 2021

• Changes throughout the manual for the removal of support for Spark version 2.3.4
• Changes throughout the manual for Spark version 2.4.8.0

June 2021

• Added a section to Troubleshooting issues with Apache Spark warning not to use SHAREPORT when assigning TCPIP PORT definitions to Spark
• Changes throughout the manual for Spark version 2.3.4.10
• Changes throughout the manual for Spark version 2.4.7.4

March 2021

• Changes throughout the manual for Spark version 2.3.4.8
• Changes throughout the manual for Spark version 2.4.7.2

November 2020

• Changes throughout the manual for Spark version 2.4.7

August 2020

• Changes throughout the manual for Spark version 2.4.6
• A new section, Using _BPX_ACCT_DATA to assign accounting information to Spark processes

June 2020

• A new section, Event log directory and file permissions

Prior to June 2020

• Changes throughout Setting up started tasks to start and stop Spark processes for History Server and Shuffle service support, including a new section, Procedures for each Spark cluster (March 2020)
• Changes throughout the manual for Spark version 2.3.4, including changes to Using the Spark configuration workflow and Upgrading Spark configuration workflows. (November 2019)
• The default value for the globalmax parameter is increased from 5000 to 50000. (September 2019)
• The newest supported version of Bourne Again Shell (Bash) is 4.3.48. (September 2019)
• A new section, Configuring Spark web interfaces. (September 2019)
• A new section, Restricting the ability to start or stop the Spark cluster. (September 2019)
• Updated descriptions for configuration workflows in Using the Spark configuration workflow. (September 2019)
• Sample configuration and AT-TLS policy rules for z/OS Spark client authentication and Sample z/OS IzODA Livy AT-TLS policy rules have extensive changes. Note that this information has replaced Appendix C, "z/OS IzODA Livy Pagent policies." (September 2019)
• A new chapter, Planning for installation, provides tips on planning the installation and deployment of IzODA. (June 2019)
• New information for z/OS IzODA Livy support (June 2019):
  • Introductory description of Livy in Introduction to IBM Open Data Analytics for z/OS
  • An additional migration note in Installing IBM Open Data Analytics for z/OS
  • z/OS IzODA Livy Installation and Customization
  • Verifying the z/OS IzODA Livy installation
  • z/OS IzODA Livy Pagent policies
• Added Spark worker fails with ICH408I message with NEWJOBNAME insert to Troubleshooting. (June 2019)
• The system messages were moved from this guide to the new Open Data Analytics for z/OS System Messages. (March 2019)
• A new section, Upgrading Spark configuration workflows, which includes Assigning an owner to new or changed steps. (March 2019)
• The JDBC Gateway is a Data Service distributed application server that allows direct connectivity to JDBC data sources. See Installing the JDBC Gateway. (March 2019)
• New sample policy for when AT-TLS is used as the Spark client authentication method, in Sample z/OS IzODA Livy AT-TLS policy rules. (December 2018)
• A new table of Spark configuration options, Table 5. (December 2018)
• New entries in Table 1. (December 2018)
• The Spark REST server port is disabled. See the migration notes in Installing IBM Open Data Analytics for z/OS for more information. (December 2018)
• New section, Other Apache Spark attributes, that describes increasing the parallelism of your Spark applications and allowing multiple Spark applications to run simultaneously. (December 2018)
• This version has received editorial updates. (September 2018)
• A new section, Using the Spark configuration workflow. (September 2018)
• New blog link locations in the What to Do Next part of Verifying the IBM Open Data Analytics for z/OS product. (September 2018)
• Verifying the env command path is updated to include a fixed APAR number. (June 2018)
• Updating the Apache Spark configuration files is updated to provide clarification about Apache Derby configuration. (June 2018)
• A note is updated in the task, Creating the Apache Spark working directories, to provide guidance on temporary file system usage. (June 2018)
• A new network port for the PySpark daemon is added to Configuring networking for Apache Spark. (June 2018)
• Step 3b in Configuring memory and CPU options is updated to clarify the amount of native memory that is required. (June 2018)
• A new property, spark.python.daemon.port, is added to Spark properties specific to the z/OS environment. (June 2018)
• An appendix is added for Data Service server messages and codes. (June 2018)
• Automating the starting of tasks is updated to provide clarification about the sample procedures that are included in IBM Open Data Analytics for z/OS. (April 2018)
• A new topic, Define the routing of the log output, is added for using started tasks. (April 2018)
• The following enhancements are available when customizing the Data Service server (April 2018):
  • You can now manually create the Global registry log stream. See Creating the Global Registry log stream.
  • Using a virtual table rule, you can read a subset of a generation data group. See Configuring generation data set retrieval.
  • Delimited data can now be used with virtual tables. See Configuring delimited data support.
• The following topics are updated to introduce the environment verification function (March 2018):
  • A new migration note is added to Installing IBM Open Data Analytics for z/OS.
  • The Spark property, spark.zos.environment.verify is added to Spark properties specific to the z/OS environment.
• A note is added to the table, Table 1, for the SPARK_MASTER_PORT configuration property. (March 2018)
• Step 4, in Creating and configuring digital certificates and key rings, is updated to include an additional command. (March 2018)
• A note and a new step are added to the task, Configuring additional authorities and permissions for the Spark cluster. (March 2018)
• A note is added to Creating jobs to start and stop Spark processes to provide clarification. (March 2018)
• The following topics introduce the task, Setting up started tasks to start and stop Spark processes (March 2018):
  • Set and export common environment variables
  • Define the RACF started profile for started tasks
  • WLM configuration
  • Stopping the started tasks
  • Canceling the started tasks
  • Automating the starting of tasks
• Using the IBM Open Data Analytics for z/OS Spark Configuration Checker is added to introduce the Configuration Checker tool. (March 2018)
• Step 4, in Securing Spark web interfaces, is updated to include an additional option. (March 2018)
• Troubleshooting issues with Apache Spark includes new troubleshooting information. (March 2018)
• An appendix is added for IBM Open Data Analytics for z/OS system messages. (March 2018)
• A note and performance considerations are added to Installing IBM Open Data Analytics for z/OS. (December 2017)
• Verifying the env command path includes an updated task description. (December 2017)
• An important note is added to Creating jobs to start and stop Spark processes. (December 2017)
• The following topics introduce enhanced job name specification options (December 2017):
  • Installing IBM Open Data Analytics for z/OS
  • Assigning job names to Spark processes
  • Using the spark.zos.executor.jobname.template
  • Using the spark.zos.driver.jobname.template
  • Spark properties specific to the z/OS environment
• The following topics introduce a new client authentication method, Trusted Partner (December 2017):
  • Configuring z/OS Spark client authentication
  • Creating and configuring digital certificates and key rings
  • Defining the AT-TLS policy rules
  • Configuring additional authorities and permissions for the Spark cluster
  • Starting the Spark cluster
  • The spark-defaults.conf configuration file option, spark.zos.master.authenticate.method is introduced. For more information, see Spark properties specific to the z/OS environment.
• A checklist for customizing the Data Service server is added. See Preparing to customize. (December 2017)
• APF-authorizing LOAD library data sets contains updated information about required APF authorizations. (December 2017)
• Configuring additional authorities and permissions for the Spark cluster and Verifying the IBM Open Data Analytics for z/OS product include updated content and code samples. (December 2017)
• The task, Securing Spark web interfaces, is added. (December 2017)
• Troubleshooting issues with Apache Spark includes an updated error message when Spark scripts fail. (December 2017)
• Migrating to a new version of Apache Spark includes updated migration actions. (December 2017)
• Sample z/OS IzODA Livy AT-TLS policy rules is updated to include a new sample policy. (December 2017)
• All of the links to external Apache Spark websites are updated for Spark version 2.2.0. (December 2017)
• All instances of WLM APAR OA50845 are updated to the latest WLM APAR OA52611. (December 2017)
• Creating and configuring digital certificates and key rings and Defining security authorization for Policy Agent include updated code samples. (September 2017)
• Configuring additional authorities and permissions for the Spark cluster provides further clarification on Spark permissions. (September 2017)
• The following topics include updated job names (September 2017):
  • Using _BPX_JOBNAME to assign job names to Spark processes
  • Overview of WLM classification
  • Defining WLM service classes for Spark
  • Defining WLM report classes for Spark
  • Defining WLM classification rules for Spark
  • Interactive performance reports with Monitor III
  • Long-term reporting with the Postprocessor
• Verifying the IBM Open Data Analytics for z/OS product adds two new installation verification procedures to verify the IBM Open Data Analytics for z/OS product. (September 2017)