IBM Spectrum Conductor with Spark is a complete enterprise-grade multi-tenant solution for Apache Spark. It implements the concept of IBM Spectrum Conductor to address the requirements of users needing to adopt the Apache Spark technology and to integrate it into their environments.
IBM Spectrum Conductor for Spark enables your organization to achieve the following results, here are five things to know:
- Enables you to deploy Spark efficiently, effectively, and with confidence.
IBM Spectrum Conductor with Spark enables organizations to run multiple instances and different versions of Spark simultaneously in a shared environment.
- Improves time to results through efficient resource scheduling.
IBM Spectrum Conductor with Spark the resource orchestrator (EGO) acts as the cluster manager with Apache Spark, enabling Spark applications to benefit from resource sharing across the cluster. Organizations can run Spark natively on a shared infrastructure without the dependency of Hadoop. Hence, helping reduce application wait time, and increase time to results.
- Increases resource utilization, resulting in better cost containment.
Apache Spark uses a coarse-grained or fine-grained resource scheduling policy. The Spark application requests a static number of resources and holds them in its lifecycle, which means each application gets more or fewer resources as it scales up and down. Based on the fine-grained scheduling policy, applications share resources at a very fine granularity, especially many applications running in a cluster concurrently. Fine grain, dynamic allocation of resources maximizes efficiency of Spark instances sharing a common resource pool. Hence extending beyond Spark and eliminates cluster sprawl.
- Eliminates resource silos tied to different Spark instances.
IBM Spectrum Conductor with Spark enables organizations to run multiple instances and different versions of Spark simultaneously in a shared environment. This capability helps users manage Spark lifecycles in the face of frequent updates to open source Spark distributions. Different groups can run their own version of Spark and it is not necessary for all Spark instances to be upgraded at the same time.
- Enhances security through role-based access control.
IBM Platform Conductor for Spark allocates resources so service levels are met while preserving security isolation between Spark instances using role-based access controls which regulate access to the resources based on the roles of individual users within an enterprise.
The ultimate aim of a software-defined infrastructure is to yield an application and data-aware environment that captures workload requirements, provides policy-based automation across data center environments, and includes analytics to optimize in real time.
For more information on the IBM Spectrum Conductor for Spark and IBM Spectrum Conductor with Spark, please also visit:
Dino E. Quintero
Project Leader for Cloud, Analytics, and HPC Solutions,
Digital Services Group