Affinity binding based on Linux cgroup cpuset subsystem

LSF can enforce CPU binding on systems that support the Linux cgroup cpuset subsystem. When CPU affinity binding through Linux cgroups is enabled, LSF will create a cpuset to contain job processes if the job has affinity resource requirements, so that the job processes cannot escape from the allocated CPUs. Each affinity job cpuset includes only the CPU and memory nodes that LSF distributes. Linux cgroup cpusets are only created for affinity jobs.

With this feature, LSF collects processor topology from hosts, including NUMA nodes, sockets, cores, and hyperthreads. Users can submit jobs specifying how processes of a job should be bound to these computing elements. LSF uses the system call sched_setaffinity() to bind CPUs. It is possible for user applications to escape from the bound CPUs by calling sched_setaffinity() directly to bind to other CPUs.

For example, submit a job with core affinity requirement and localprefer memory binding:

bsub -R "affinity[core:membind=localprefer]"./myapp

LSF will create a cpuset which contains one core and attach the process ID of the application ./myapp to this cpuset. The cpuset serves as a strict container for job processes, so that the application ./myapp cannot bind to other CPUs.

In this example, the memory binding policy is localprefer. When membind=localprefer, or it is not specified, LSF adds all memory nodes to the cpuset to make sure the job can access all memory nodes on the host, and will make sure job processes will access preferred memory nodes first. If the memory binding policy is localonly, LSF only adds the memory nodes that the LSF scheduler distributes to the cpuset, and myapp only uses those memory nodes, not all memory nodes.

To enable the cpuset enforcement feature, configure LSB_RESOURCE_ENFORCE="cpu" in lsf.conf.