LSB_RESOURCE_ENFORCE

Syntax

LSB_RESOURCE_ENFORCE="[memory]|[cpu]|[gpu]"

Description

Controls resource enforcement through the Linux cgroup memory, CPU set subsystem, and GPU access on Linux systems with cgroup support. Memory and CPU set enforcement for Linux cgroups is supported on Red Hat Enterprise Linux (RHEL) 6.2 or above, SuSe Linux Enterprise Linux 11 SP2 or above. GPU enforcement for Linux cgroup device subsystems is supported on Red Hat Enterprise Linux 6.2 or above, and SUSE Linux Enterprise Linux 11 SP2 or later.

You can specify memory, cpu, or gpu as the resource. You may also specify two or three of these resources in any order.

LSF can impose strict host-level memory and swap limits on systems that support Linux cgroups. These limits cannot be exceeded. All LSF job processes are controlled by the Linux cgroup system. If job processes on a host use more memory than the defined limit, the job will be immediately killed by the Linux cgroup memory subsystem. Memory is enforced on a per job or per host basis, not per task. If the host OS is Red Hat Enterprise Linux 6.3 or above, cgroup memory limits are enforced, and LSF is notified to terminate the job. Additional notification is provided to users through specific termination reasons displayed by bhist –l.

To enable memory enforcement, configure LSB_RESOURCE_ENFORCE="memory".

Note:
  • If LSB_RESOURCE_ENFORCE="memory" is configured, all existing LSF memory limit related parameters such as LSF_HPC_EXTENSIONS="TASK_MEMLIMIT", LSF_HPC_EXTENSIONS="TASK_SWAPLIMIT", LSB_JOB_MEMLIMIT and LSB_MEMLIMIT_ENFORCE are ignored.
  • LSB_RESOURCE_ENFORCE="memory" must be specified for host-based memory and swap limit enforcement with the bsub -hl command option to take effect. If no memory or swap limit is specified for the job (the merged limit for the job, queue, and application profile, if specified), or LSB_RESOURCE_ENFORCE="memory" is not specified, a host-based memory limit is not set for the job. When LSB_RESOURCE_ENFORCE="memory" is configured and memory and swap limits are specified for the job, but -hl is not specified, memory and swap limits are calculated and enforced as a multiple of the number of tasks running on the execution host.

LSF can also enforce CPU affinity binding on systems that support the Linux cgroup cpuset subsystem. When CPU affinity binding through Linux cgroups is enabled, LSF will create a cpuset to contain job processes if the job has affinity resource requirements, so that the job processes cannot escape from the allocated CPUs. Each affinity job cpuset includes only the CPU and memory nodes that LSF distributes. Linux cgroup cpusets are only created for affinity jobs.

To enable CPU enforcement, configure LSB_RESOURCE_ENFORCE="cpu".

If you are enabling memory and CPU enforcement through the Linux cgroup memory cpsuset subsystems after upgrading an existing LSF cluster, make sure that the following parameters are set in lsf.conf:
  • LSF_PROCESS_TRACKING=Y
  • LSF_LINUX_CGROUP_ACCT=Y

Different LSF hosts in the cluster can use different versions of cgroup as long as each individual LSF host is only running one version of cgroup. If you have both versions of cgroup enabled in a host, you must disable one of the versions. For example, hostA can use cgroup v1 and hostB can use cgroup v2 as long as each host is only running one version of cgroup.

To enable GPU access through Linux cgroups, configure LSB_RESOURCE_ENFORCE="gpu". LSF creates devices to contain job processes if the job has GPU resource requirements so that the job processes cannot escape from the allocated GPUs. Each GPU job device includes only the GPUs that LSF distributes. Linux cgroup devices are only created for GPU jobs.

Note: When GPU enforcement is enabled, the GPUs that are contained in one device cgroup are reallocated new GPU IDs, beginning with 0. CUDA Version 7.0 or newer supports cgroup completely.

Starting in Fix Pack 14, additionally, LSF leverages cgroups to enforce MIG device isolation. Set this enforcement by configuring LSB_RESOURCE_ENFORCE="gpu" in the lsf.conf file.

Jobs can specify how job processes are to be bound to these computing elements. LSF uses the environment variable CUDA_VISIBLE_DEVICES to tell user applications which GPUs are allocated. It is possible for user applications to escape from the allocated GPUs by changing CUDA_VISIBLE_DEVICES directly to use other GPUs.

For example, the following command submits a job with one exclusive thread GPU requirement: 

bsub -R "rusage[ngpus_excl_t=1]"./myapp

LSF creates a device that contains one exclusive thread GPU and attaches the process ID of the application ./myapp to this device. The device serves as a strict container for job processes, so that the application ./myapp cannot use other GPUs.

Examples

For a parallel job with three tasks and a memory limit of 100 MB, such as the following:

bsub -n 3 -M 100 –R "span[ptile=2]" blaunch ./mem_eater

The application mem_eater keeps increasing the memory usage. LSF will kill the job if it consumes more than 200 MB total memory on one host. For example, if hosta runs two tasks and hostb runs one task, the job will only be killed if total memory on exceeds 200 MB on either hosta or hostb. If one of the tasks consumes more than 100 MB memory but less than 200 MB, and the other task doesn’t consume any memory, the job will not be killed. That is, LSF does not support per task memory enforcement for cgroups.

For a job with affinity requirement: 

bsub -R "affinity[core:membind=localonly]"./myapp

LSF will create a cpuset which contains one core and attach the process ID of the application ./myapp to this cpuset. The cpuset serves as a strict container for job processes, so that the application ./myapp cannot bind to other CPUs. LSF will add all memory nodes into the cpu set to ensure that the job can access all memory nodes on the host, and will make sure job processes will access preferred memory nodes first.

Default

Not defined. Resource enforcement through the Linux cgroup system is not enabled.