Integrating an existing IBM Spectrum Symphony cluster with open source YARN

Integrate open source YARN with an IBM® Spectrum Symphony cluster to enable the EGO-YARN framework.

Before you begin

Note: The steps in this topic outline how to integrate with YARN automatically, using an integration script ($EGO_TOP/integration/yarn/4.0/etc/egoyarn-integration.sh). You can also complete this integration manually, as described in Getting started with integrated YARN.

Ensure that open-source YARN is installed on the nodes in your cluster.
If you did not set JAVA_HOME before installing IBM Spectrum Symphony, set the JAVA_HOME environment variable in the $EGO_TOP/integration/yarn/conf/egoyarn-env.sh file to the directory where Java is installed. For example:
```
export JAVA_HOME=/usr/java/latest 
```
See egoyarn-env.sh reference for more details.
If you did not set HADOOP_YARN_HOME before installing YARN, set the environment variable in $EGO_TOP/integration/yarn/conf/egoyarn-env.sh to the directory where YARN is installed. For example:
```
export HADOOP_YARN_HOME=/usr/hadoop-2.7.2
```
See egoyarn-env.sh reference for more details.
Ensure that the hosts which run YARN workload have sufficient resources.
Ensure that the execution user for the YARN resource manager and node managers has write-permissions to the $HADOOP_CONF_DIR/yarn-site.xml configuration file. Set this up because when the YARN resource manager and node managers restart, they rewrite yarn-site.xml.
Ensure that your host names do not map to the common IPv4 loopback ID address (127.0.0.1). Use a different IP address for your hosts.
If you are using Apache HDFS (Hadoop Distributed File System), ensure that your setup is complete and functional.

Ensure that open source YARN is installed on the nodes in your IBM Spectrum Symphony cluster.

About this task

You must manually integrate IBM Spectrum Symphony and open source YARN when:

You install IBM Spectrum Symphony without setting HADOOP_YARN_HOME.
You install IBM Spectrum Symphony first and then install open source YARN.

Procedure

Configure all EGO-YARN hosts in $EGO_TOP/integration/yarn/conf/egohosts.
Set the HADOOP_YARN_HOME environment variable. For example:

export HADOOP_YARN_HOME=/opt/hadoop-2.7.2
Run integration script $EGO_TOP/integration/yarn/4.0/etc/egoyarn-integration.sh using one of the following commands:
- To integrate EGO-YARN on hosts listed in egohosts, run the following command from a host that has been configured with passwordless SSH for logging into other hosts in the cluster:
  ./egoyarn-integration.sh remote
- To integrate EGO-YARN on local hosts, run:
  ./egoyarn-integration.sh local
Restart the IBM Spectrum Symphony cluster.
Start the EGOYARN service:

egosh service start EGOYARN