Machine learning models on a remote Apache Hadoop cluster in Jupyter Python

Start a Jupyter notebook

Open or create a Jupyter notebook. In the Select Runtime drop down, choose the Hadoop Environment. This starts a JEG kernel securely with your user identity and therefore the access to the data and resources on the Hadoop cluster is restricted based on your authorization.

If you've upgraded to Cloud Pak for Data versions 4.7.0, 4.7.1, or 4.7.2, you must run the following commands as the first cell after the kernel starts:

from pyspark.sql import SparkSession
from pyspark import SparkContext
spark = SparkSession.builder.getOrCreate()
sc = SparkContext.getOrCreate()

Using the shell action, you can run commands on the YARN nodemanager. For example, you can use the following command:

!hostname -f
!hdfs dfs -ls /user

Work with data

You can read data from an HDFS file or Hive table by using the spark session.

	df_data_1 = spark.read.format(
	    "org.apache.spark.sql.execution.datasources.csv.CSVFileFormat").option(
	    "header", "true").option("inferSchema", "true").load("hdfs:///user/cars.csv)

	df_data_1 = spark.sql("select * from cars")

hdfs_model_dir = "/user/{your username}/.dsxhi/models/{modelname}/{modelversion}/model.tar.gz"
wsl_model_path = "/project_data/data_asset/model.tar.gz"
hadoop_lib_utils.download_hdfs_file(webhdfs_endpoint, hdfs_model_dir, wsl_model_path)

Start a Jupyter notebook

Open or create a Jupyter notebook. In the Select Runtime drop down, choose the Default Python 3.10 environment. Note that you should not choose a Hadoop environment.

Create a remote Livy session

You should import the hadoop_lib_utils python library and call the get_dsxhi_info utility method to get information about the registered Hadoop systems on the Watson Studio cluster, the available services, and the Python environments that are pushed to HDFS. Use the following command:

HI_SYSTEMS = hadoop_lib_utils.get_dsxhi_info(showSummary=True)

You can create a config object to specify additional spark configurations, such as YARN queue, driver memory, executor memory, and number of executors. Use the following command:

myConfig={
 "queue": "default",
 "driverMemory": "1G",
 "numExecutors": 2
};

You should call the setup_livy_sparkmagic function to set up the authentication, Hadoop cluster information and additional configs for the Livy connection. If you want to use a pushed Python environment, you can specify it in the imageId parameter:

HI_CONFIG = hadoop_lib_utils.setup_livy_sparkmagic(
  system="systemName	",
  livy="livyspark2",
  addlConfig=myConfig,
  imageId="jupyter-231n-py")

# (Re-)load sparkmagic to apply the new configs.
%reload_ext sparkmagic.magics
To create a Livy session, run
session_name = mysession
livy_endpoint = HI_CONFIG['LIVY']
%spark add -s $session_name -l python -k -u $livy_endpoint

This starts a YARN application securely with your user identity and therefore the access to the data and resources on the Hadoop cluster is restricted based on your authorization. If you see the call return a 500 error, you should check the Resource Manager UI on the Hadoop cluster. If the YARN application was created successfully, you can ignore the 500 error.

Once a Livy session is established, to run code on the Hadoop cluster, the first line in the cell should include %%spark -s $session_name. Cells that do not start with %%spark run locally on the Watson Studio cluster.

Work with data

You can read data from an HDFS file or Hive table by using the spark session.

%%spark -s $session_name
df_data_1 = spark.read.format(
    "org.apache.spark.sql.execution.datasources.csv.CSVFileFormat").option(
    "header", "true").option("inferSchema", "true").load("hdfs:///user/cars.csv)

%%spark -s $session_name
df_data_1 = spark.sql("select * from cars")

%%spark -s $session_name
hi_core_utils.write_model_to_hdfs(model=model, model_name="modelname")

hdfs_model_dir = "/user/{your username}/.dsxhi/models/{modelname}/{modelversion}/model.tar.gz"
wsl_model_path = "/project_data/data_asset/model.tar.gz"
hadoop_lib_utils.download_hdfs_file(webhdfs_endpoint, hdfs_model_dir, wsl_model_path)