Configuring CGroups in YARN

CGroups is a mechanism for aggregating/partitioning sets of tasks, and all their future children, into hierarchical groups with specialized behavior. CGroups is a Linux kernel feature and was merged into kernel version 2.6.24. From a YARN perspective, this allows containers to be limited in their resource usage. A good example of this is CPU usage. Without CGroups, it becomes hard to limit container CPU usage. Currently, CGroups is used only for limiting CPU usage.

Before you begin

To help avoid CGroups stability issues it is recommended to check your Linux kernel version. The Linux version should be RHEL 7.2 with kernel-3.10.0-327.44.2 or RHEL 7.3.

Procedure

To enable CGroups:

  1. Create a CGroups directory on all NodeManager nodes.
    For RHEL6:
    1. Create /cgroup/cpu on each node manager: mkdir -p /cgroup/cpu.
    For RHEL7:
    1. Create /sys/fs/cgroup/cpu,cpuacct/yarn/ on each node manager: mkdir -p /sys/fs/cgroup/cpu,cpuacct/yarn/
    2. Change the owner of the directory: chown -R yarn:hadoop /sys/fs/cgroup/cpu,cpuacct/yarn
  2. Log into Ambari.
  3. Navigate to YARN > Configs.
  4. Click Enable CPU Isolation.
  5. Confirm that the following properties are set: 
    yarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor
yarn.nodemanager.linux-container-executor.resources-handler.class=org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler
yarn.nodemanager.linux-container-executor.cgroups.hierarchy=/yarn
yarn.nodemanager.linux-container-executor.cgroups.mount = true
yarn.nodemanager.linux-container-executor.cgroups.mount-path=/cgroup
yarn.nodemanager.linux-container-executor.group=hadoop
  6. Save the configuration.
  7. On RHEL7, change the following property as shown before restarting services: 
    yarn.nodemanager.linux-container-executor.cgroups.mount=false
    Because the CGroups path has been mounted in RHEL7, there is no need to have YARN mount it again.

Results

You are now able to use CGroups to limit CPU usage. The following settings are related to limiting resource usage of YARN containers. You may adjust these according to the resource requirements on your system:
  • Update this setting to adjust the CPU usage of all YARN containers: 
    yarn.nodemanager.resource.percentage-physical-cpu-limit
  • Update this setting to determine how spare CPU is used: 
    yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage
    When set to true, containers cannot use more CPU than allocated, even when it is available. When set to false, containers can use spare CPU when it is available.
Note: If running in non-secure mode, by default the LinuxContainerExecutor runs all jobs as user nobody. When running a job the exception below may occur:
Requested user nobody is not whitelisted and has id 99,which is below the minimum allowed 200
To resolve this, do any of the following options as required:
  1. Add the yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user property and set it to the desired user.
  2. Configure the LinuxContainerExecutor to run jobs as the user submitting the job by adding property yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users and setting it to false.
  3. set min.user.id to a lower value in /etc/hadoop/conf/container-executor.cfg in all NodeManagers.

What to do next

For more information about YARN CGroups see http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html.