IBM Packages for Apache Spark

Exploit the big data analytics capabilities of Apache Spark with this new package for IBM platforms.

Overview

The IBM Packages for Apache Spark ™ are an integrated, highly performant, and manageable Apache Spark runtime, tuned for solving analytics problems on IBM platforms.

The IBM Development Package for Apache Spark and IBM z/OS ® Platform for Apache Spark provide a fast, general purpose cluster computing platform for large-scale data processing, powered by the latest IBM SDK, Java ™ Technology Edition, Version 8.

Downloads

These are the available downloads of the IBM Packages for Apache Spark.

The IBM Packages for Apache Spark are regularly updated with security fixes. It is strongly recommended that you download and install all version updates when they become available.

Version 2.1.0

IBM z/OS Platform

Looking for the IBM z/OS Platform for Apache Spark? Get in touch on that page to order our supported version.


Installation

If you are upgrading from an earlier version, back up your configuration files as a precaution.

Linux

Run the installer, review and accept the license by following the instructions on the master and each worker in your compute cluster. This places the IBM Development Package for Spark in the install directory. In a console, change to the install directory and run setenv.sh to configure your shell for launching Apache Spark .

z/OS

The installation instructions for the z/OS package are available in the document Installing IBM z/OS Package for Apache Spark .

Support

The latest v2.x releases are packaged and tested with Scala 2.11.8 and Apache Hadoop 2.7.3.

Configurations using other versions of Scala and Hadoop are not supported.

The following table shows the operating systems that are verified for each platform architecture.

Operating systemHardwareBitness
z/OS 2.1z Systems64
SUSE Linux Enterprise Server 12z Systems64
Red Hat Enterprise Linux 7.2Power Systems - Little Endian64
Red Hat Enterprise Linux 7.2x8664
Red Hat Enterprise Linux 6.7x8664
CentOS 7.2x8664
Ubuntu 16.04 LTSx8664
Ubuntu 16.04 LTSz Systems64
Ubuntu 14.04 LTSx8664
Ubuntu 14.04 LTSPower Systems - Little Endian64

Technical support

Use dW Answers to ask questions and share your expertise. Our development team would like to hear your feedback. Include the "ibmjdk" and "spark" tags to help us find your questions quickly.

The IBM Development Package for Apache Spark might be embedded in other IBM products. If you have a licence for one of these products that includes support you can also report problems by using the standard IBM product support process.


Trademarks and attributions

Apache Spark, Spark, Apache, and the Spark logo are trademarks of The Apache Software Foundation. The IBM Development Package for Apache Spark is not formally related to or endorsed by the official Apache Spark open source project.

Java, JavaScript and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Microsoft, Windows, Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Apple, Macintosh, and Mac OS are trademarks of Apple Inc., registered in the United States and other countries.

Other company, product and service names may be trademarks or service marks of others.


Change history

  
22 May 2017Removed 1.6 downloads.
28 February 2017Added version 1.6.3.1 for Linux platforms only.
27 February 2017Added version 2.1.0.1 for Linux platforms only.
11 January 2017Added version 2.1.0.0 for Linux platforms only.
21 November 2016Added version 2.0.2.0 for Linux platforms only.
11 November 2016Added version 1.6.3.0 for Linux platforms only.
18 October 2016Added version 2.0.1.0 for Linux platforms only.
14 September 2016Added support for Ubuntu 16.04 LTS on z Systems.
07 September 2016 Release version 1.6.2.1 for Linux platforms only.
16 August 2016 Added version 2.0.0 for Linux platforms only.
01 July 2016 Release version 1.6.2.0 for Linux platforms only.
29 April 2016 Release version 1.6.1.1 for Linux platforms only.
18 March 2016 IBM z/OS Platform for Apache Spark add Spark Job Server and Spark Kernel files.
11 March 2016 Release version 1.6.1.0 for Linux platforms only.
27 January 2016 Release version 1.6.0.1 for Linux platforms only.
19 January 2016 Release version 1.6.0.0 for Linux platforms only.
15 December 2015IBM z/OS Platform for Apache Spark added.
25 November 2015GA release. New download page created.