Execution Engine for Apache Hadoop on Cloud Pak for Data

Description

The Execution Engine for Apache Hadoop services integrates the Watson Studio service with your remote Apache Hadoop cluster.

Data scientists can use this service for the following tasks:

Browse remote Hadoop data through connections
Cleanse and transform remote Hadoop data with Data Refinery
Run Data Refinery and jobs on the Hadoop Spark cluster
Run a notebook session on the remote Hadoop system
Access Hadoop systems with basic utilities from RStudio and Jupyter notebooks

The Execution Engine for Apache Hadoop includes:

Services that establish secure connections between Watson Studio and Hadoop
Integration with Hadoop for Refinery and Notebook
A high availability configuration to the remote Hadoop system
Utilities that connect Watson Studio and Hadoop

The service requires a service user who has the necessary privileges to submit requests on behalf of the Watson Studio users to WebHDFS, WebHCAT, Spark, and YARN. The service generates a secure URL for each Watson Studio cluster that is integrated with the Hadoop cluster.

Hadoop ecosystem services

Watson Studio interacts with a Hadoop cluster through the following services:

Service	Purpose
WebHDFS	Browse and preview HDFS data
Jupyter Enterprise Gateway	Submit jobs to JEG on the Hadoop cluster.
Livy for Spark	Submit jobs to Spark on the Hadoop cluster.

Quick links

Install: Install the service
Use: Work with the service
What's new: See a list of new features
Known issues: View limitations
Administer: Manage and maintain the service

Integrated services

Table 1. Prerequisite services. This service requires the following prerequisite services to be installed.
Service	Capability
Watson™ Studio	Prepare, analyze, and model data in a collaborative environment with tools for data scientists, developers, and domain experts.

Table 2. Related services. The following related services are often used with this service and provide complementary features, but they are not required.
Service	Capability
Watson Machine Learning	Build, train, and deploy machine learning models with a full range of tools.
RStudio® Server Runtimes	Access the RStudio IDE.