What's new and changed in IBM Spectrum Conductor?
IBM® Spectrum Conductor 2.3.0 includes various new functionality and enhancements to existing functions.
Product rebranding
New for 2.3.0, the product name has changed from IBM Spectrum Conductor with Spark to IBM Spectrum Conductor.
Apache Spark 2.3.1
- Apache Spark 2.3.1
- Apache Spark 2.2.0
- Apache Spark 2.1.1
- Apache Spark 1.6.1
For more information, see the Spark 2.3.1 release notes at Spark Release 2.3.1.
Anaconda management
IBM Spectrum Conductor now offers better integration of Anaconda with Spark workload and Jupyter notebooks that includes enterprise security. You can manage an existing Anaconda distribution or create a new one to deploy to your cluster. You can also share an Anaconda distribution cluster that is deployed across multiple Spark instance groups to save disk space.
IBM Spectrum Conductor Anaconda version 5.1.0 Python 3 is built in to IBM Spectrum Conductor 2.3.0.
Integration with IBM Cloud Private
You can now deploy IBM Spectrum Conductor as a Helm chart in IBM Cloud Private to quickly launch and run an IBM Spectrum Conductor master container in the Kubernetes cluster. This evaluation version includes autonomous Spark instance group image management, secure Helm chart support, enables Spark master high-viability, and the use of Livy and Spark master services.
Installation, upgrading, and configuration
- You can now install IBM Spectrum Conductor as non-root
- For production clusters, log in with root or sudo to root permission. For evaluation clusters, you can now install as any user, who becomes the cluster administrator.
- egoinstallfixes command
- You can now install and manage IBM Spectrum
Conductor fixes by using the new
egoinstallfixes command. This command offers options to complete these tasks:
- Install fixes, which includes checking the system, backing up the current files, and then installing the specified fix packages on top of your existing IBM Spectrum Conductor cluster.
- Roll back the most recent fix (either by fix package name or by build number) so that your host returns to its original state before you install the fix.
- Check fix packages, without installing them. This preliminary checking includes determining if your existing cluster is compatible with the fix you want to install, verifying that the user account has appropriate permissions, and listing files that will be overwritten by and added by the fix.
- New EGO configuration to allow LIM to start MELIM and PIM processes as a cluster administrator:
- A new EGO_ELIM_RUNAS_CLUSTER_ADMIN parameter within the ego.conf configuration file allows the LIM daemon to start MELIM and PIM processes as a cluster administrator user. If enabled, all processes also can run as this administrative user. By default, this setting is not enabled.
- Installing or upgrading with other products
- IBM Spectrum
Conductor
2.3.0 supports:
- IBM Spectrum Symphony 7.2.1
- IBM Spectrum LSF® Standard Edition for Symphony 10.1.0.2
Prepaid licensing and monitoring
IBM Spectrum Conductor now supports prepaid usage licensing for compute workloads that extend to the cloud. Prepaid usage licensing enables you to prepay for hours used by a host's cores (core hours) and consume those hours however you want. To audit prepaid consumption when your cluster bursts to the cloud, you must configure IBM Spectrum Conductor to upload usage metrics through an external load index monitor (ELIM) to a metering service in IBM Cloud Private. For more information on downloading and installing prepaid usage monitoring, see Installing prepaid usage monitoring for cloud.
Security
- SSL support extended to include the local RS
- In addition to SSL communication between the master repository server (RS) and the RS client, IBM Spectrum Conductor now supports SSL between the local RS and the (master) RS, and between the local RS and the RS client. If you use multiple repository servers, the RS client connects to RS first, and then the system redirects the RS client to the local RS. In this case, you edit the egosc service profile (for example RSmyregion.xml) for this configuration. The steps are very similar to setting up SSL support between RS and the RS client.
Spark instance group enhancements
- GPU monitoring enhancements
- With GPU monitoring enabled, you can now see for the amount of memory and GPU utilization that is used and the total available for all Spark instance groups. For Spark applications, you can now see the amount of memory and GPU utilization that is used and the total available for the applications. You can also see the current values (total, average, maximum, and minimum) across all GPU devices that are used by the applications.
- GPU computing with R
- You can now write a Spark application that uses a Resilient Distributed Dataset (RDD) API in R, in addition to Python and Scala, to create a new GPU RDD; whose tasks run on GPU resources in your cluster. An R GPU RDD API example is also provided.
- Configuring additional parameters and environment variables
- If necessary, you can set additional parameters and environment variables when you are configuring a Spark instance group. If the parameter or variable is invalid, it is ignored.
Notebook enhancements
- Jupyter notebook language kernels
- Inside the Jupyter notebook, you can see three new kernel options that are called Spark Python (Spark Cluster Mode), Spark R (Spark Cluster Mode), and Spark Scala (Spark Cluster Mode). These kernels have a Spark context that is ready to use and that runs outside of the Jupyter notebook service. Select one of these kernels if you want to utilize Spark when you submit jobs from the notebook. With these options, all the monitoring features of Spark applications are available, as they are Spark cluster applications instead of Spark client mode applications.
- Kernel culling for Jupyter notebooks
- Kernel culling is now configured by default for Jupyter 5.4.0 notebooks (or later) from the Notebook Management page in the cluster management console in order to reclaim kernel resources. With kernel culling, you can specify timeout environment variables for the Jupyter kernel so that the kernel does not idle and resources are reclaimed within a specific time frame.
- Jupyter 5.4.0
- IBM Spectrum Conductor 2.3.0 includes a Jupyter 5.4.0 notebook package that works for Linux and Linux on POWER. Jupyter 5.4.0 supports built-in Spark versions that are included in IBM Spectrum Conductor version 2.3.0 except Spark 1.6.1.
Reporting
- Explorer Reports
- With Explorer reports, IBM Spectrum Conductor additionally aggregates Elasticsearch data to make it consumable to be used for the Spark Charge Back report and then puts the data back into Elasticsearch.
- Data loader enhancements
- The new Spark resource usage aggregation data loader sparkresusageaggloader aggregates resource usage for cores, slots, and memory for both GPU and CPU by application ID, user name, and top consumer across each Spark instance group, across one hour for one application. This data loader is disabled by default.
RESTful API references now available
Now you can view RESTful application programming interface (API) documentation to configure and maintain IBM Spectrum Conductor through RESTful APIs.
Tutorial
The tutorial has been updated for IBM Spectrum Conductor 2.3.0. The tutorial is short, easy to follow, and provides step-by-step instructions on how to create and manage a Spark instance group. The tutorial now covers Anaconda management.