z/OS IzODA Livy Installation and Customization

Before you can use z/OS IzODA Livy, you must customize your environment for it and its dependent products.

Before you begin

Follow the instructions in Installing IBM Open Data Analytics for z/OS to install Open Data Analytics for z/OS on your system.

About this task

Once installed, the default program location for z/OS IzODA Livy is /usr/lpp/IBM/izoda/anaconda/extras/apache-livy.

Complete the tasks in this topic to customize your environment for z/OS Livy. You will need access to a TSO session and an OMVS session (preferably through a Putty terminal).

About z/OS IzODA Livy

The Livy server is a REST service used in conjunction with Spark that allows users to submit Spark jobs without having the Spark client installed. It provides two modes of job submission for the end users:

Batch mode: For submitting Spark application files such as Scala JARs and Python .py files. Conceptually this is equivalent to users using spark-submit for application submission.
Session mode: For submitting Scala or Python application code interactively. Conceptually this is equivalent to users using interactive shell such as spark-shell or pyspark shell.

After the Spark job is received, the Livy server submits the application to the Spark cluster on behalf of the user.

Installing z/OS IzODA Livy

Complete this task to install the z/OS IzODA Livy package onto your system using Anaconda. IBM recommends the System Programmer to perform the install of Livy into the Anaconda base environment.

Determine the Anaconda environment that you will be using for the install. The System Programmer will be using the base environment (already created).
- If you are not the System Programmer, IBM recommends first creating your own conda environment for the install. For example:
```
conda create -n myenv python
```
Activate your environment. The System Programmer will use the base environment.
```
conda activate myenv
```
Enter the following:
```
conda install apache-livy
```

Setting up a user ID for use with z/OS IzODA Livy

Complete this task to set up a user ID for use with z/OS IzODA Livy. For this task, you can either create a new user ID to use for z/OS IzODA Livy, or you can use an existing user ID.

Choose or create an appropriate user ID for use with z/OS IzODA Livy. Specifically, this is the user ID under which the Livy server is started, known as the Livy server user ID in this documentation.

The Livy server will be submitting Spark application to the Spark cluster on behalf of the Livy end users. Therefore, the Livy server user ID must have the appropriate authorities for submitting Spark application to the Spark cluster as any other Spark end user. If z/OS Spark client authentication is enabled, refer to Configuring z/OS Spark client authentication for details about setting up the Spark end user ID with the appropriate authorities.

Before starting the Livy server, you must have the appropriate authorities to either access or create (if they do not already exist) the directories that are specified by the following environment variables, or the defaults taken when not specified:

LIVY_CONF_DIR
LIVY_LOG_DIR
LIVY_PID_DIR

IBM also recommends you set the LIVY_HOME environment variable to the Livy installation directory.