Mainframes

Mainframe data becomes more accessible with Apache Spark

Share this post:

The rise of digital business has led to a proliferation of data types and sources – from location data to social media content – stored in disparate formats in an enterprise’s data centers or the public cloud. This data can be extremely valuable to organizations that want to engage their customers in real-time in new, more personalized ways to improve customer service and generate more revenue.

The problem is, companies either can’t access this data, or it takes them hours to days to do so, by which point it’s no longer as relevant or useful. In the Cognitive Era, where data is the new natural resource and computer systems are able to understand, reason and learn, organizations need fast, more secure access to a wide range of data, including the rich data sets on the IBM z Systems mainframe, running z/O 2.1 or later to better serve customers and beat their competition. They need instant analysis of this data to act on market intelligence, instead of simply reacting to market trends.

This week at Strata + Hadoop World in San Jose, IBM unveiled the z/OS Platform for Apache Spark. The platform allows developers and data scientists to access and analyze data in-place on z Systems with the Apache Spark open-source analytics framework – potentially saving their organizations time and money and limiting risk. Previously, they would have had to move data off the platform in batches and analyze it that way — a process that would take hours to days.

z Systems handles critical data and transactions for many of the world’s major banks, insurers, retailers and transport companies. It features the industry’s fastest commercial microprocessor and can score predictive models within a transaction in 2 milliseconds or less.[1] z Systems also meets the highest levels of security to help data remain protected.[2]

To further advance the new capabilities, we’ve also partnered with DataFactz, Zementis and Rocket Software to create customized solutions using IBM z/OS Platform for Apache Spark:

  • DataFactZ is a new partner that is working with IBM to develop Spark analytics based on Spark SQL and MLlib for data and transactions processed on the mainframe.
  • Rocket Software has been a long-standing IBM partner and this collaboration extends to z/OS Apache Spark. For example, the new Rocket Launchpad solution will allow clients to try the platform using data on z/OS.
  • Zementis is complementing its in-transaction predictive analytics offering for z/OS with a standards-based execution engine for Apache Spark. The solution helps allow users to deploy and execute advanced predictive models that can help them anticipate end users’ needs, compute risk or detect fraud in real-time at the point of greatest impact, while processing a transaction.

By enabling Spark natively on the mainframe, clients will be able to take even greater advantage of z Systems’ world-class capabilities of speed, scale and security as they transform into cognitive businesses.

You can find more details on the new z/OS Platform for Apache Spark in this overview, which includes a use case demo of what the new offering can do for an organization.

[1] World’s fastest microprocessor is based on 5.5GHz core processor speed; Based on IBM tests of I/O processing on the z13s. that show results in 2 milliseconds or less.

[2] Based on Common Criteria EAL5+ security rating for z Systems mainframe

More stories

IBM z15 sets a new cloud security standard

Encryption, Mainframes, Multicloud

From time to time, we invite industry thought leaders to share their opinions and insights on current technology trends to the IBM Systems IT Infrastructure blog. The opinions in these posts are their own, and do not necessarily reflect the views of IBM. On September 12, 2019, in New York City, IBM set a new ...read more


Address customer data privacy and protection concerns with encryption everywhere

Data centers, Data security, Mainframes

Data privacy and protection remain top boardroom and C-suite issues. The data breach threat still looms large: 59 percent of businesses experienced a data breach caused by a vendor or third party in 2018[1]. As organizations migrate workloads to hybrid multicloud environments, they must ensure that the data within these environments is effectively protected. Consumers ...read more


Mentoring, music & mainframes

Academic initiatives, Mainframes, Modern data platforms

A story of DJs who power the mainframe community From time to time, we invite industry thought leaders to share their opinions and insights on current technology trends to the IT Infrastructure blog. The opinions in these blogs are their own, and do not necessarily reflect the views of IBM. Since my inception into the ...read more