YARN in IBM Spectrum Symphony

When YARN is integrated with IBM® Spectrum Symphony, you gain a robust setup that takes advantage of resource scheduling functionality both in YARN and EGO.

A EGO-YARN environment provides the following benefits:
  • An integrated installation so that when you install IBM Spectrum Symphony Advanced Edition, your existing YARN cluster is compatible with IBM Spectrum Symphony. Your YARN configuration, APIs, logging, and security mechanisms (authentication and authorization) maintain the same, so that your environment is consistent after integrating with IBM Spectrum Symphony.
  • Dynamic resource sharing between YARN workload and IBM Spectrum Symphony workload. Open-source YARN does not support sharing hosts between YARN workload and other types of workload. A EGO-YARN environment leverages EGO's allocation and scheduling mechanisms to enable resource sharing between different types of workload.
  • Graceful reclaiming of resources from YARN, triggered by a non-YARN consumer.
  • Multidimensional scheduling for host sharing with any consumer on the same multidimensional resource plan. Multidimensional scheduling enables multitenant policy sharing.
  • Pre-startup and dynamic startup support for node managers.
  • High availability (HA) for YARN resource manager entities. Resource managers can automatically recover from failures in a EGO-YARN environment. You can set up HA failover support using basic IBM Spectrum Symphony HA, using Apache ZooKeeper.
  • Dynamic detection of host metrics, removing the need to configure host metrics statically.
  • A dedicated resource group for the resource manager and the application server. With EGO-YARN, a resource group can contains more reliable computing resources than a typical resource group, thereby, securing stability for the resource manger and application server.
  • Access to the YARN GUI from the cluster management console for resource scheduling.
  • Access to the IBM Spectrum Symphony command line to submit YARN workload. For example, you can run commands from $EGO_TOP/integration/yarn/4.0/etc, rather than from /opt/hadoop-2.7.2/bin.