PH03469: SPARK 2.2.0.8

Obtain the fix for this APAR.

APAR status

• Closed as new function.

Error description

• Spark 2.2.0.8
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * Users of IBM Open Data Analytics for z/OS - IzODA Spark -    *
  * Release 110                                                  *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * Spark Master REST port default change                        *
  * - Change the default setting of Spark Master REST port to be *
  * disabled.                                                    *
  *                                                              *
  * AT-TLS security for the Spark Master REST port               *
  * - AT-TLS security for the Spark Master REST port is now      *
  * supported.                                                   *
  *                                                              *
  * AT-TLS security for the Spark Shuffle port                   *
  * - AT-TLS security for the Spark Shuffle port is now          *
  * supported.                                                   *
  *                                                              *
  * Fixes for Spark Dynamic Workflow                             *
  * - Long path names are allowed for started task support       *
  *                                                              *
  * - Owner and group of $HOME/.profile                          *
  *    is set to SPARKID:SPKGRP and                              *
  *    permissions 700 when we create the .profile               *
  *                                                              *
  * - Various doc bugs                                           *
  *                                                              *
  * - Ensure the home dir is correct for spark ID                *
  *                                                              *
  * Support for scheduling multiple Spark applications           *
  * - New keyword support to enable having multiple Spark        *
  * applications active                                          *
  *                                                              *
  * Spark dynamic allocation and Spark Shuffle server            *
  * - Spark dynamic allocation and Spark Shuffle server is now   *
  * supported                                                    *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  

Problem conclusion

• Spark Master REST port default change
  - Prior to this APAR, Spark's default behavior had the Spark
  Master REST port enabled. After this APAR is applied, the only
  way to enable the Spark Master REST port will be to explicitly
  enable it in the spark-defaults.conf configuration file.
  
  Support AT-TLS security for the Spark Master REST port
  - Prior to this APAR, AT-TLS security for the Spark Master REST
  port was not supported.  After this APAR is applied, AT-TLS
  security for the Spark Master REST port can be used.
  
  Support AT-TLS security for the Spark Shuffle port
  - Prior to this APAR, AT-TLS security for the Spark Shuffle port
  was not supported.  After this APAR is applied, AT-TLS security
  for the Spark Shuffle port can be used.
  
  Fixes for Spark Dynamic Workflow
  - Long path names are allowed for started task support
  - Prior to this APAR,
    the owner and group of
    $HOME/.profile and  permissions 700 may
    not have been set correctly when we create
    the .profile
  . After this APAR is applied,
    the dynamic workflow will ensure
    owner and group of $HOME/.profile
    is set to SPARKID:SPKGRP and
    permissions 700 when we create the .profile
  .
  
  - Prior to this APAR,  the home dir for the spark ID may not
  have been set correctly. After this APAR is applied,  the
  dynamic workflow will ensure the home dir is correct for spark
  ID.
  
  
  Support for scheduling multiple Spark applications
  - Prior to this APAR, multiple Spark applications could only be
  active if each application was started with spark-defaults.conf
  settings that ensured that all resources in the system were not
  used.  After this APAR, with the
  spark.zos.master.app.alwaysScheduleApps setting set to 'true',
  multiple applications can be active simultaneously, allowing WLM
  to manage the resources used by the application.
  
  Support for Spark dynamic allocation and Spark Shuffle server
  - Prior to this APAR, Spark dynamic allocation and Spark shuffle
  server were not supported.  After this apar is applied, they are
  supported.
  
  The following publication is updated:
  
  
  
  1. IBM Open Data Analytics for z/OS Installation and
  Customization Guide (SC27-9033)
  
  
  
  Refer to the latest (December 2018) IBM Open Data Analytics for
  
  z/OS Installation and Customization Guide to see the
  
  documentation updates.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UI60037

Modules/Macros

• AZKSPUPX AZKSP220
  

Fix information

• Fixed component name

  Z/OS SPARK

• Fixed component ID

  5655AAB01

Applicable component levels

• R120 PSY UI60037

     UP18/12/11 P F812  

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.