High availability for Execution Engine for Apache Hadoop

You can use a secondary node when an active node goes down by enabling high availability.

To enable high availability, you need to add two nodes of a system when you register a remote cluster. After high availability is enabled and an active edge node goes down, complete the following steps to activate high availability:

Data Access via WebHDFS: Data browse and preview tools accessing WebHDFS are reconnected to the next available edge node.
Interactive notebooks: Any active Livy Sessions that live on the failed node must be restarted and run again.
Data Refinery: Any running jobs must be resubmitted. Any new jobs started is sent to the active edge node.
Remote jobs: Any jobs that are running on this remote environment must be resubmitted. Any new jobs started is sent to the Active edge node.

Load Balancing with multiple Execution Engine for Apache Hadoop Edge nodes

WebHDFS transfers are allocated with round robin, balancing the network traffic between Watson Studio and the Hadoop edge nodes.
Livy Sessions are allocated with sticky sessions, following an active and passive approach. All Livy sessions are run on the same Execution Engine edge node until a failure is detected, at which point all new sessions are allocated on the next available Execution Engine edge node.
Similar to Livy, JEG sessions are allocated with sticky sessions and follow an active and passive approach. All JEG sessions are run on the same Execution Engine edge node until a failure is detected, at which point all new sessions are allocated on the next available Execution Engine edge node.