SCHED_PER_JOB_SORT

Syntax

SCHED_PER_JOB_SORT=Y/N

Description

Enable this parameter to use the per-job sorting feature in scheduler. This feature allows LSF to schedule jobs accurately in one scheduling cycle by adding extra load to scheduler.

For example, jobs 1 to 3 are submitted to LSF as follows:
  • bsub -R "order[slots]" job1
  • bsub -R "order[slots]" job2
  • bsub -R "order[slots]" job3
Each job was requested to be dispatched to the host with the most unused slots. The request is not completely satisfied in one scheduling cycle if per-job sorting is disabled. For example:
  • host1 3 slots
  • host2 3 slots
  • host3 3 slots

Jobs 1 to 3 are dispatched to the same host (for example, host1) in one scheduling cycle because the hosts are only sorted once in one scheduling cycle. If the per-job sorting feature is enabled, the candidate hosts are sorted again before the job is actually scheduled. Therefore, these three jobs are dispatched to each host separately. This result is exactly what you want but the cost is that scheduler performance maybe significantly lower. For example, if there are 5000 hosts in the cluster and the management host is on the machine with 24GB memory and 2 physical CPUs with 2 cores with 6 threads, each job with ! in the ORDER[] section consumes about an extra 10ms of time in one scheduling cycle. If there are many jobs with! in the ORDER[] section that are controlled by LSB_MAX_JOB_DISPATCH_PER_SESSION, then a lot of extra time is consumed in one scheduling cycle.

To get the accurate schedule result without impacting scheduler performance, you can set JOB_ACCEPT_INTERVAL as a non-zero value. This is because the hosts do not need to be sorted again by the per-job sorting feature.

Default

N

See also

JOB_ACCEPT_INTERVAL in lsb.params and lsb.queues.

LSB_MAX_JOB_DISPATCH_PER_SESSION in lsf.conf.