Now available: IBM Platform Conductor for Spark!

Body

We are thrilled to announce the availability of IBM Platform Conductor for Spark V1.1.

As a new product offering, Platform Conductor for Spark (Platform Conductor) can help address many of the challenges that organization face in deploying Apache Spark. At a glance, Platform Conductor offers the following benefits:

• Runs Spark natively on shared infrastructure without the overhead of Hadoop. It also reduces application wait time, creating faster time to results.
• With proven architecture for enterprise-class workload management, monitoring, reporting, and security, it helps to significantly reduce administration costs.
• Maximizes resource utilization through fine-grained dynamic allocation of resources for Spark instances sharing a common resource pool. It extends beyond Spark and eliminates cluster sprawl.
• Provides an end-to-end IBM-supported Spark solution that features:
  • An IBM Spark distribution.
  • The IBM Platform Management Console with the Platform Enterprise Grid Orchestrator (EGO) resource scheduler.
  • IBM Spectrum Scale File Placement Optimizer (FPO).

Platform Conductor is supported on Linux and Linux on POWER. For more information on Platform Conductor, see the IBM Knowledge Center.

So here's what you can expect from Platform Conductor for Spark V1.1:

Granular and dynamic resource management with Spark on EGO

Platform Conductor improves performance and efficiency with Platform EGO, a proven solution for granular and dynamic resource allocation. By integrating Spark on EGO, Platform Conductor leverages the resource scheduling capabilities of EGO for Spark applications within the cluster.
 

Spark multitenancy

Platform Conductor provides multitenancy by enabling you to run multiple instances of Spark, including different versions, in a shared environment. Different groups can run their own instances of Spark and it is not necessary for all Spark instances to be upgraded in lockstep.

With Platform Conductor installed, you can use the built-in Spark version, one that is prepackaged to include Apache Spark and the Apache Zeppelin notebook. As other versions become available, we post updated Spark versions on IBM Fix Central. To leverage any new functionalities available in the later versions, you only need to download and import these versions to Platform Conductor.
 

Third-party notebook samples on IBM BlueMix DevOps

Besides the built-in Zeppelin notebook, you can also use other notebooks such as iPython, by integrating them with Platform Conductor. Find notebook samples that you can readily integrate at pc4spark on IBM BlueMix DevOps Services.
 

Monitoring and reporting leveraging ELK

Platform Conductor provides data analytics by integrating the ELK stack (Elasticsearch, Logstash, and Kibana). With this integration, Platform Conductor enables you to search, analyze, and visualize Spark application data for efficient monitoring.
 

Consolidated framework for monitoring, deployment, and reporting

With the Platform Management Console (PMC), Platform Conductor simplifies management by providing a single point of access to manage the cluster and to deploy, manage, and monitor Spark instance groups. For ease of use, the PMC also embeds Kibana.
 

IBM Spectrum Scale

Platform Conductor incorporates IBM Spectrum Scale FPO storage management technology as a POSIX-compliant and more space-efficient alternative to HDFS (which is also supported if you prefer that option). When using IBM Spectrum Scale, Platform Conductor becomes an end-to-end IBM-supported Spark solution.
 

IBM Knowledge Center

Platform Conductor documentation is available in the IBM Knowledge Center, which you can view either as an installable local framework or online. The local IBM Knowledge Center is installed with Platform Conductor. When you access help from the PMC, you typically access the installed version of the IBM Knowledge Center. If you have access to the Internet, you can switch from the local version to the online version, which always has the most up-to-date content. 

For complete details on the release, see the Release Notes in the IBM Knowledge Center.

To try out Platform Conductor, download an evaluation version from our Service Management Connect page. If you have any questions, post them in our forum!

Third-party images from:
http://spark.apache.org/

https://www.elastic.co/