Software-defined computing

Spark Summit 2015: Is Spark the real deal for enterprise environments?

Share this post:

If you are wondering whether Spark is “for real” as an enterprise solution, the Spark Summit 2015, held recently in San Francisco, strongly suggested that we will be seeing a lot more of it in enterprise environments in the coming months. A major talking point was the commitment IBM showed to Spark at this event.  In addition to IBM, there were  speakers from several large organizations including NASA/JPL, NBC, Netflix, Beth Israel Deaconess Medical Center, Capital One, Thomson Reuters, the CIA, Toyota, and the US Patent & Trademark Office.

The Spark Summit itself is certainly growing. With over 2000 attendees, the Summit was the biggest by far to date and testimony to Spark’s growing momentum. The plenary sessions were typically full to capacity, and three parallel breakout tracks — developer, data science, and applications and use cases – were also well attended. Likewise, the solution expo often appeared to be bursting at the seams and the attendance was probably as large as the venue could handle.

IBM was a platinum sponsor of the event. Beth Smith, General Manager, Analytics Platform, IBM Analytics, gave a keynote address in which she made several important announcements. These included plans to:

  • Donate the IBM SystemML machine learning technology to the Spark open source ecosystem
  • Educate more than one million data scientists and data engineers on Spark
  • Establish a Spark Technology Center in San Francisco to foster innovation

Copies of all Spark Summit presentation materials, along with video, are available from the Spark Summit 2015 website.

On the first evening of the Summit, IBM hosted a Spark Community Event. While guests enjoyed cocktails and snacks, they were able to view several IBM Spark demos. One of these showed a complex genomics sequencing workflow running on an IBM Software Defined Infrastructure platform including IBM Platform LSF, IBM Spectrum ScaleIBM Platform Symphony and, to integrate Spark into the workflow, IBM Platform Application Service Controller.

The centerpiece of the Community Event was a lively hour-long panel discussion of Spark and its impact on industry and education.  The end-user customer voice on this panel was USAA, a Platform Symphony customer. The USAA representative was Robert Dohner, an Associate Actuary who discussed the role of Spark in his organization. The panel was moderated by Edd Dumbill, VP, Strategy & Marketing at Silicon Valley Data Science and included speakers from education and industry.

The overall message of the whole event was probably best summarized by a memorable line from entrepreneur and venture capitalist Ben Horwitz of Andreessen Horowitz.  Speaking of the impact that IBM’s announcements would have on Spark in the enterprise environment, Ben quipped “Think of IBM as a rabbi” who has pronounced that Spark “is kosher for use in your company… Spark is now blessed by the enterprise rabbi.”

For more information about how IBM Software Defined Infrastructure solutions can help with Spark deployment for the enterprise, please visit the IBM Platform Application Service Controller web page.

More stories

Introducing Spectrum Storage for AI with NVIDIA DGX for deep learning

AI, Cloud object storage, Deep learning...

Artificial intelligence, machine learning and deep learning projects are happening in every industry and research team across the globe. According to IDC[1], by 2019 40 percent of digital transformation initiatives will use AI services; by 2021, 75 percent of commercial enterprise apps will use AI, over 90 percent of consumers will interact with customer support ...read more


Guard your virtual environment with IBM Spectrum Protect Plus

Cloud object storage, Data centers, DevOps...

The server virtualization market is expected to grow to approximately 8 billion dollars by 2023. This would mean a compound annual growth rate (CAGR) of 7 percent between 2017 and 2023. Protection of these virtual environments is therefore a significant and critical component of an organization’s data protection strategy. This growth in virtual environments necessitates ...read more


Spectrum Scale, SDI and motorcycles: Scale up, scale out!

IBM Systems Lab Services, Services, Software...

You might wonder if I’ve completely gone off the rails when I mention my passion for motorcycle riding in the same breath as Spectrum Scale and software-defined infrastructure (SDI). Trust me, I’ll make a case for why I put them together. In the early days, a motorcycle was nothing more than a bicycle with a ...read more