Running Apache Livy 0.5.0 with IBM Spectrum Conductor 2.3.0
Kevin Doyle 2700017RUN Visits (8387)
Since IBM Spectrum Conductor version 22.214.171.124, we integrated Apache Livy 0.4.0 as an application instance. Now with IBM Spectrum Conductor version 2.3.0, we integrate with Apache Livy 0.5.0! Why use the latest version of this integration? If you want to use Apache Spark 2.3.1 with Apache Livy, you need the 0.5.0 version.
What is the benefit of this latest version? You can now create multiple application instances that run different Livy instances on the same host. The port is no longer hardcoded in the application template and you can specify the port to use. To learn more about how to use Apache Livy 0.5.0 with IBM Spectrum Conductor 2.3.0, see our sample project.
Apache Livy can be used to integrate with IBM Watson Studio Local (formerly IBM Data Science Experience Local). For more details on how to use IBM Watson Studio Local for your notebooks and have IBM Spectrum Conductor running the Spark jobs in the backend, see our previous blog.
Have any comments or questions? Reach out to us on Slack!