What's new and changed in Execution Engine for Apache Hadoop

Installing or upgrading Execution Engine for Apache Hadoop

Ready to install or upgrade Execution Engine for Apache Hadoop?

Related documentation:

• Execution Engine for Apache Hadoop
• Installing Execution Engine for Apache Hadoop
• Upgrading Execution Engine for Apache Hadoop

Refresh 16 of Cloud Pak for Data Version 3.5

A new version of Execution Engine for Apache Hadoop was released in October 2022.

Assembly version: 3.5.11

This release includes the following changes:

Security fixes

This release includes fixes for the following security issues:

CVE-2022-33099, CVE-2022-32208, CVE-2022-32206, CVE-2022-29824, CVE-2022-28737, CVE-2022-28736, CVE-2022-28735, CVE-2022-28734, CVE-2022-28733, CVE-2022-27782, CVE-2022-27776, CVE-2022-27774, CVE-2022-25314, CVE-2022-25313, CVE-2022-24765, CVE-2022-22576, CVE-2022-2097, CVE-2022-2068, CVE-2022-1586, CVE-2022-1292

CVE-2021-46822, CVE-2021-46195, CVE-2021-45078, CVE-2021-43618, CVE-2021-4209, CVE-2021-40528, CVE-2021-40330, CVE-2021-3997, CVE-2021-39537, CVE-2021-3697, CVE-2021-3696, CVE-2021-3695, CVE-2021-3468, CVE-2021-3408, CVE-2021-31879, CVE-2021-3114, CVE-2021-25317, CVE-2021-24032, CVE-2021-21300, CVE-2021-20193

CVE-2020-35512, CVE-2020-35507, CVE-2020-35496, CVE-2020-35495, CVE-2020-35494, CVE-2020-35493, CVE-2020-21674, CVE-2020-17049, CVE-2020-16156, CVE-2020-13790, CVE-2020-12413, CVE-2020-12401

CVE-2019-9937, CVE-2019-9936, CVE-2019-9923, CVE-2019-9674, CVE-2019-9077, CVE-2019-9075, CVE-2019-9074, CVE-2019-9071, CVE-2019-8906, CVE-2019-8905, CVE-2019-6110, CVE-2019-2201, CVE-2019-19244, CVE-2019-17543, CVE-2019-14250, CVE-2019-12972, CVE-2019-12900

CVE-2018-6872, CVE-2018-20839, CVE-2018-20673, CVE-2018-20671, CVE-2018-20657, CVE-2018-20651, CVE-2018-20623, CVE-2018-20406, CVE-2018-20002, CVE-2018-19932, CVE-2018-19211, CVE-2018-18701, CVE-2018-18607, CVE-2018-18606, CVE-2018-18605, CVE-2018-18484, CVE-2018-18483, CVE-2018-18309, CVE-2018-17985, CVE-2018-17794, CVE-2018-17360, CVE-2018-16428, CVE-2018-15919, CVE-2018-12934, CVE-2018-12699, CVE-2018-12698, CVE-2018-12697, CVE-2018-12641, CVE-2018-1121, CVE-2018-1000880, CVE-2018-1000879, CVE-2018-1000876, CVE-2018-1000654, CVE-2018-1000021

Refresh 15 of Cloud Pak for Data Version 3.5

A new version of Execution Engine for Apache Hadoop was released in July 2022.

Assembly version: 3.5.10

This release includes the following changes:

Security fixes

This release includes fixes for the following security issues:

CVE-2018-25032, CVE-2020-10543, CVE-2020-10878, CVE-2020-12723, CVE-2020-17541, CVE-2021-20197, CVE-2021-3487, CVE-2021-3580, CVE-2021-3634, CVE-2021-3712, CVE-2021-3733, CVE-2021-3737, CVE-2021-3800, CVE-2021-38185, CVE-2021-3981, CVE-2021-41617, CVE-2021-4189, CVE-2021-42694, CVE-2021-44569, CVE-2021-44570, CVE-2021-44571, CVE-2021-44573, CVE-2021-44574, CVE-2021-44575, CVE-2021-44576, CVE-2021-44577, CVE-2021-45960, CVE-2021-46143, CVE-2022-1271, CVE-2022-21426, CVE-2022-21434, CVE-2022-21443, CVE-2022-21476, CVE-2022-21496, CVE-2022-22822, CVE-2022-22823, CVE-2022-22824, CVE-2022-22825, CVE-2022-22826, CVE-2022-22827, CVE-2022-25313, CVE-2022-26280, CVE-2022-29155

Refresh 14 of Cloud Pak for Data Version 3.5

A new version of Execution Engine for Apache Hadoop was released in May 2022.

Assembly version: 3.5.9

This release includes the following changes:

Security fixes

This release includes fixes for the following security issues:

CVE-2021-38297, CVE-2021-39293, CVE-2021-23177, CVE-2022-25236, CVE-2022-23772, CVE-2021-45346, CVE-2022-23806, CVE-2021-41771, CVE-2022-23773, CVE-2021-31566, CVE-2022-23852, CVE-2021-3999, CVE-2022-25315, CVE-2022-23308, CVE-2021-23841, CVE-2022-23218, CVE-2021-34558, CVE-2022-23219, CVE-2021-44716, CVE-2022-25235, CVE-2021-41772

Refresh 12 of Cloud Pak for Data Version 3.5

A new version of Execution Engine for Apache Hadoop was released in March 2022.

Assembly version: 3.5.8

This release includes the following changes:

Security fixes

This release includes fixes for the following security issues:

CVE-2019-20386, CVE-2020-16135, CVE-2020-1712, CVE-2021-3521

Refresh 10 of Cloud Pak for Data Version 3.5

A new version of Execution Engine for Apache Hadoop was released in January 2022.

Assembly version: 3.5.7

This release includes the following changes:

Bug fixes

• Issue: The DSX-HI pod log no longer provides up-to-update logging of jobs submitted to Hadoop.

  Resolution: The issue is now resolved.

Security fixes

This release includes fixes for the following security issues:

CVE-2021-36086, CVE-2021-35578, CVE-2020-14155, CVE-2021-3445, CVE-2021-35550, CVE-2021-35556, CVE-2019-18218, CVE-2021-20232, CVE-2021-36087, CVE-2021-22946, CVE-2021-22898, CVE-2021-35588, CVE-2021-3200, CVE-2020-10001, CVE-2021-36085, CVE-2020-12399, CVE-2020-35448, CVE-2021-35564, CVE-2018-16429, CVE-2021-20284, CVE-2019-13751, CVE-2018-20673, CVE-2019-17594, CVE-2021-35586, CVE-2021-35942, CVE-2021-35567, CVE-2021-22876, CVE-2021-3733, CVE-2021-3114, CVE-2021-3572, CVE-2021-22925, CVE-2020-12762, CVE-2021-33197, CVE-2019-5827, CVE-2021-33503, CVE-2019-13750, CVE-2021-28153, CVE-2020-13435, CVE-2021-35565, CVE-2021-35603, CVE-2021-20231, CVE-2021-35559, CVE-2021-36084, CVE-2021-35561, CVE-2019-19603, CVE-2020-14145, CVE-2018-12700, CVE-2019-20838, CVE-2020-24370, CVE-2021-20266, CVE-2021-22947, CVE-2021-33560, CVE-2021-3426, CVE-2019-17595, CVE-2017-18018, CVE-2021-27645, CVE-2021-23840, CVE-2021-33574, CVE 2021-45105, CVE 2021-45046

Refresh 9 of Cloud Pak for Data Version 3.5

A new version of Execution Engine for Apache Hadoop was released in October 2021.

Assembly version: 3.5.6

This release includes the following changes:

Security fixes

This release includes fixes for the following security issues:

CVE-2021-2388, CVE-2021-2341, CVE-2021-2369

Refresh 7 of Cloud Pak for Data Version 3.5

Assembly version: 3.5.3

Security fixes

This release includes fixes for the following security issues:

CVE-2015-2059, CVE-2020-12399, CVE-2020-25648, CVE-2020-25692

Refresh 2 of Cloud Pak for Data Version 3.5

A new version of Execution Engine for Apache Hadoop was released in January 2021.

Assembly version: 3.5.1

Bug fixes

• Issue: RStudio® Server with R 3.6 includes a new version of Sparklyr. Sparklyr doesn't work with the HadoopLibUtils library that's provided to connect to remote Hadoop clusters using Livy. The following error occurs: Error: Livy connections now require the Spark version to be specified.

  Resolution: It now works as expected.

Initial release of Cloud Pak for Data Version 3.5

A new version of Execution Engine for Apache Hadoop was released as part of Cloud Pak for Data Version 3.5.

Assembly version: 3.5.0

This release includes the following changes:

New features

Integration with IBM Spectrum® Conductor with Spark clusters

IBM Spectrum Conductor with Spark is now supported. You can integrate IBM Spectrum Conductor with Spark and Watson™ Studio by using Jupyter Endpoint Gateway endpoints. Users can open a notebook in Watson Studio to access Jupyter Endpoint Gateway instances that are running on IBM Spectrum Conductor with Spark. For details, see Spectrum environments.

New configurations that allow you to use your own certificates

The configurations convert DSXHI to do the following customizations:

• Provide a custom Keystore to generate the required .crt.
• Provide any custom truststore (CACERTS), where DSXHI certificates will be added.
• Provide options to either add the host certificate to the truststore yourself or have DSXHI add it.

For details, see Installing the Execution Engine for Apache Hadoop service on Apache Hadoop clusters or on Spectrum Conductor clusters.

Support for additional types of security

Execution Engine for Apache Hadoop supports:

• The JSON Web Tokens to Kerberos delegation token provider, which provides authentication to HiveServer2, HDFS, and HMS resources. For details, see Using delegation token endpoints.
• The updated versions for Jupyter Endpoint Gateway 2.3 and Knox 1.4.

Improved validation

The system_check.py scripts were introduced to validate your Hadoop configuration.