Processor binding for LSF job processes

Processor binding for LSF job processes takes advantage of the power of multiple processors and multiple cores to provide hard processor binding functionality for sequential LSF jobs and parallel jobs that run on a single host.

Rapid progress of modern processor manufacture technologies has enabled the low-cost deployment of LSF on hosts with multicore and multithread processors. The default soft affinity policy enforced by the operating system scheduler may not give optimal job performance. For example, the operating system scheduler may place all job processes on the same processor or core leading to poor performance. Frequently switching processes as the operating system schedules and reschedules work between cores can cause cache invalidations and cache miss rates to grow large.

Restriction: Processor binding is supported on hosts running Linux with kernel version 2.6 or higher.

For multi-host parallel jobs, LSF sets two environment variables (LSB_BIND_JOB and LSB_BIND_CPU_LIST) but does not attempt to bind the job to any host.

When processor binding for LSF job processes is enabled on supported hosts, job processes of an LSF job are bound to a processor according to the binding policy of the host. When an LSF job is completed (exited or done successfully) or suspended, the corresponding processes are unbound from the processor.

When a suspended LSF job is resumed, the corresponding processes are bound again to a processor. The process is not guaranteed to be bound to the same processor it was bound to before the job was suspended.

The processor binding affects the whole job process group. All job processes forked from the root job process (the job RES) are bound to the same processor.

Processor binding for LSF job processes does not bind daemon processes.

If processor binding is enabled, but the execution hosts do not support processor affinity, the configuration has no effect on the running processes. Processor binding has no effect on a single-processor host.

Processor, core, and thread-based CPU binding

By default, the number of CPUs on a host represents the number of cores a machine has. For LSF hosts with multiple cores, threads, and processors, ncpus can be defined by the cluster administrator to consider one of the following:
  • Processors
  • Processors and cores
  • Processors, cores, and threads

Globally, this definition is controlled by the parameter EGO_DEFINE_NCPUS in lsf.conf or ego.conf. The default behavior for ncpus is to consider the number of cores (EGO_DEFINE_NCPUS=cores).

Note: When PARALLEL_SCHED_BY_SLOT=Y in lsb.params, the resource requirement string keyword ncpus refers to the number of slots instead of the number of CPUs, however lshosts output will continue to show ncpus as defined by EGO_DEFINE_NCPUS in lsf.conf.

Binding job processes randomly to multiple processors, cores, or threads, may affect job performance. Processor binding configured with LSF_BIND_JOB in lsf.conf or BIND_JOB in lsb.applications, detects the EGO_DEFINE_NCPUS policy to bind the job processes by processor, core, or thread (PCT).

For example, if the PCT policy for the host is set to processor (EGO_DEFINE_NCPUS=procs) and the binding option is set to BALANCE, the first job process is bound to the first physical processor, the second job process is bound to the second physical processor and so on.

If the PCT policy for the host is set to core level (EGO_DEFINE_NCPUS=cores) and the binding option is set to BALANCE, the first job process is bound to the first core on the first physical processor, the second job process is bound to the first core on the second physical processor, the third job process is bound to the second core on the first physical processor, and so on.

If the PCT policy for the host is set to thread level (EGO_DEFINE_NCPUS=threads) and the binding option is set to BALANCE, the first job process is bound to the first thread on the first physical processor, the second job process is bound to the first thread on the second physical processor, the third job process is bound to the second thread on the first physical processor, and so on.

Note: BIND_JOB and LSF_BIND_JOB are deprecated in LSF Standard Edition and LSF Advanced Edition. You should enable LSF CPU and memory affinity scheduling in with the AFFINITY parameter in lsb.hosts. If both BIND_JOB and affinity scheduling are enabled, affinity scheduling takes effect, and BIND_JOB is disabled. If both LSF_BIND_JOB and affinity scheduling are enabled, affinity scheduling takes effect, and LSF_BIND_JOB is disabled. BIND_JOB and LSF_BIND_JOB are the only affinity options available in LSF Express Edition.

BIND_JOB=BALANCE

The BIND_JOB=BALANCE option instructs LSF to bind the job that is based on the load of the available processors/cores/threads. For each slot:
  • If the PCT level is set to processor, the lowest loaded physical processor runs the job.
  • If the PCT level is set to core, the lowest loaded core on the lowest loaded processor runs the job.
  • If the PCT level is set to thread, the lowest loaded thread on the lowest loaded core on the lowest loaded processor runs the job.

If there is a single 2 processor quad core host and you submit a parallel job with –n 2 –R”span[hosts=1]” when the PCT level is core, the job is bound to the first core on the first processor and the first core on the second processor:

After submitting another three jobs with -n 2 -R"span[hosts=1]":

If PARALLEL_SCHED_BY_SLOT=Y is set in lsb.params, the job specifies a maximum and minimum number of job slots instead of processors. If the MXJ value is set to 16 for this host (there are 16 job slots on this host), LSF can dispatch more jobs to this host. Another job submitted to this host is bound to the first core on the first processor and the first core on the second processor:

BIND_JOB=PACK

The BIND_JOB=PACK option instructs LSF to try to pack all the processes onto a single processor. If this cannot be done, LSF tries to use as few processors as possible. Email is sent to you after job dispatch and when job finishes. If no processors/cores/threads are free (when the PCT level is processor/core/thread level), LSF tries to use the BALANCE policy for the new job.

LSF depends on the order of processor IDs to pack jobs to a single processor.

If PCT level is processor, there is no difference between BALANCE and PACK.

This option binds jobs to a single processor where it makes sense, but does not oversubscribe the processors/cores/threads. The other processors are used when they are needed. For instance, when the PCT level is core level, if we have a single four processor quad core host and we had bound 4 sequential jobs onto the first processor, the 5th-8th sequential job is bound to the second processor.

If you submit three single-host parallel jobs with -n 2 -R"span[hosts=1]" when the PCT level is core level, the first job is bound to the first and seconds cores of the first processor, the second job is bound to the third and fourth cores of the first processor. Binding the third job to the first processor oversubscribes the cores in the first processor, so the third job is bound to the first and second cores of the second processor:

After JOB1 and JOB2 finished, if you submit one single-host parallel jobs with -n 2 -R"span[hosts=1], the job is bound to the third and fourth cores of the second processor:

BIND_JOB=ANY

BIND_JOB=ANY binds the job to the first N available processors/cores/threads with no regard for locality. If the PCT level is core, LSF binds the first N available cores regardless of whether they are on the same processor or not. LSF arranges the order based on APIC ID.

If PCT level is processor (default value after installation), there is no difference between ANY and BALANCE.

For example, with a single 2-processor quad core host and the following table is the relationship of APIC ID and logic processor/core id:
APC ID Processor ID Core ID
0 0 0
1 0 1
2 0 2
3 0 3
4 1 0
5 1 1
6 1 2
7 1 3

If the PCT level is core level and you submit two jobs to this host with -n 3 -R "span[hosts=1]", then the first job is bound to the first, second, and third core of the first physical processor, the second job is bound to the fourth core of the first physical processor and the first, second core in the second physical processor.

BIND_JOB=USER

The BIND_JOB=USER parameter binds the job to the value of $LSB_USER_BIND_JOB as specified in the user submission environment. This allows the Administrator to delegate binding decisions to the actual user. This value must be one of Y, N, NONE, BALANCE, PACK, or ANY. Any other value is treated as ANY.

BIND_JOB=USER_CPU_LIST

The BIND_JOB=USER_CPU_LIST parameter binds the job to the explicit logic CPUs specified in environment variable $LSB_USER_BIND_CPU_LIST. LSF does not check that the value is valid for the execution host(s). It is the user's responsibility to correctly specify the CPU list for the hosts they select.

The correct format of $LSB_USER_BIND_CPU_LIST is a list which may contain multiple items, separated by comma, and ranges. For example, 0,5,7,9-11.

If the value's format is not correct or there is no such environment variable, jobs are not bound to any processor.

If the format is correct and it cannot be mapped to any logic CPU, the binding fails. But if it can be mapped to some CPUs, the job is bound to the mapped CPUs. For example, with a two-processor quad core host and the logic CPU ID is 0-7:

  1. If user1 specifies 9,10 into $LSB_USER_BIND_CPU_LIST, his job is not bound to any CPUs.
  2. If user2 specifies 1,2,9 into $LSB_USER_BIND_CPU_LIST, his job is bound to CPU 1 and 2.

If the value's format is not correct or it does not apply for the execution host, the related information is added to the email sent to users after job dispatch and job finish.

If user specifies a minimum and a maximum number of processors for a single-host parallel job, LSF may allocate processors between these two numbers for the job. In this case, LSF binds the job according to the CPU list specified by the user.

BIND_JOB=NONE

The BIND_JOB=NONE parameter is functionally equivalent to the the former BIND_JOB=N parameter where the processor binding is disabled.

Feature interactions

  • Existing CPU affinity featuresProcessor binding of LSF job processes will not take effect on a management host with the following parameters configured.
    • MBD_QUERY_CPUS
    • LSF_DAEMONS_CPUS
    • EGO_DAEMONS_CPUS
  • Job requeue, rerun, and migration

    When a job is requeued, rerun or migrated, a new job process is created. If processor binding is enabled when the job runs, the job processes will be bound to a processor.

  • bctrld restart sbd

    bctrld restart sbd restarts a new sbatchd. If a job process has already been bound to a processor, after sbatchd is restarted, processor binding for the job processes are restored.

  • badmin reconfig

    If the BIND_JOB parameter is modified in an application profile, badmin reconfig only affects pending jobs. The change does not affect running jobs.

  • LSF multicluster capability job forwarding model

    In the LSF multicluster capability environment, the behavior is similar to the current application profile behavior. If the application profile name specified in the submission cluster is not defined in the execution cluster, the job is rejected. If the execution cluster has the same application profile name, but does not enable processor binding, the job processes are not bound at the execution cluster.