Absolute priority scheduling (APS)

Absolute job priority scheduling (APS) provides a mechanism to control the job dispatch order to prevent job starvation. APS provides administrators with detailed yet straightforward control of the job selection process. When configured in a queue, APS sorts pending jobs for dispatch according to a job priority value calculated based on several configurable job-related factors. Each job priority weighting factor can contain sub-factors. Factors and sub-factors can be independently assigned a weight.

  • APS sorts only the jobs. Job scheduling is still based on configured LSF scheduling policies. LSF attempts to schedule and dispatch jobs by their order in the APS queue, but the dispatch order is not guaranteed.
  • The job priority is calculated for pending jobs across multiple queues that are based on the sum of configurable factor values. Jobs are then ordered based on the calculated APS value.
  • You can adjust the following values for APS factors:
    • A weight for scaling each job-related factor and sub-factor
    • Limits for each job-related factor and sub-factor
    • A grace period for each factor and sub factor
  • To configure absolute priority scheduling (APS) across multiple queues, define APS queue groups. When you submit a job to any queue in a group, the job's dispatch priority is calculated by using the formula that is defined with the APS_PRIORITY parameter in the group's parent queue in the lsb.queues file.
  • Administrators can also set a static system APS value for a job. A job with a system APS priority is guaranteed to have a higher priority than any calculated value. Jobs with higher system APS settings have priority over jobs with lower system APS settings.
  • Administrators can use the ADMIN factor to manually adjust the calculated APS value for individual jobs.

Scheduling priority factors

To calculate the job priority, APS divides job-related information into several categories. Each category becomes a factor in the calculation of the scheduling priority. You can configure the weight, limit, and grace period of each factor to get the wanted job dispatch order.

LSF uses the weight of each factor to sum the value of each factor.

Factor weight
The weight of a factor expresses the importance of the factor in the absolute scheduling priority. The factor weight is multiplied by the value of the factor to change the factor value. A positive weight increases the importance of the factor, and a negative weight decreases the importance of a factor. Undefined factors have a weight of zero, which causes the factor to be ignored in the APS calculation.
Factor limit
The limit of a factor sets the minimum and maximum absolute value of each weighted factor. Factor limits must be positive values.
Factor grace period
Each factor can be configured with a grace period. The factor is only counted as part of the APS value when the job was pending for a long time and it exceeds the grace period.

APS_PRIORITY syntax

APS_PRIORITY=WEIGHT[[factor, value] [sub factor, value]...]...] LIMIT[[factor, value] [sub factor, value]...]...] GRACE_PERIOD[[factor, value] [sub factor, value]...]...]

Factors and sub factors

Factors Sub factors Metric
FS (user-based fair share factor) The existing fair share feature tunes the dynamic user priority The fair share factor automatically adjusts the APS value based on dynamic user priority.

The FAIRSHARE parameter must be defined in the queue. The FS factor is ignored for non-fair-share queues.

The FS factor is influenced by the following fair share parameters that are defined in the lsb.queues or lsb.params file:

  • CPU_TIME_FACTOR
  • FWD_JOB_FACTOR
  • RUN_TIME_FACTOR
  • RUN_JOB_FACTOR
  • HIST_HOURS
RSRC (resource factors) PROC Requested tasks are the maximum of bsub -n min_task, max_task, the min of bsub -n min, or the value of the TASKLIMIT parameter in the lsb.queues file.
MEM Total real memory requested (in MB or in units set in the LSF_UNIT_FOR_LIMITS parameter in the lsf.conf file).

Memory requests appearing to the right of a || symbol in a usage string are ignored in the APS calculation.

For multi-phase memory reservation, the APS value is based on the first phase of reserved memory.
SWAP Total swap space requested (in MB or in units set in the LSF_UNIT_FOR_LIMITS parameter in the lsf.conf file).

As with MEM, swap space requests appearing to the right of a || symbol in a usage string are ignored.
WORK (job attributes) JPRIORITY The job priority that is specified by:
  • Default that is specified by half of the value of the MAX_USER_PRIORITY parameter in the lsb.params file
  • Users with bsub -sp or bmod -sp
  • Automatic priority escalation with the JOB_PRIORITY_OVER_TIME parameter in the lsb.params file

If the TRACK_ELIGIBLE_PENDINFO parameter in the lsb.params file is set to Y or y, LSF increases the job priority for pending jobs as long as it is eligible for scheduling. LSF does not increase the job priority for ineligible pending jobs.
QPRIORITY The priority of the submission queue.
APP   Set the priority factor at the application profile level by specifying the PRIORITY parameter in the lsb.applications file. The APP_PRIORITY factor is added to the calculated APS value to change the factor value. The APP_PRIORITY factor applies to the entire job.
USER   Set the priority factor for users by specifying the PRIORITY parameter in the User section of the lsb.users file. The USER_PRIORITY factor is added to the calculated APS value to change the factor value. The USER_PRIORITY factor applies to the entire job.
UG   Set the priority factor for user groups by specifying the PRIORITY parameter in the UserGroup section of the lsb.users file. The UG_PRIORITY factor is added to the calculated APS value to change the factor value. The UG_PRIORITY factor applies to the entire job. LSF uses the priority of the user group as specified in the bsub -G option.
ADMIN   Administrators use bmod -aps to set this sub factor value for each job. A positive value increases the APS. A negative value decreases the APS. The ADMIN factor is added to the calculated APS value to change the factor value. The ADMIN factor applies to the entire job. You cannot configure separate weight, limit, or grace period factors. The ADMIN factor takes effect as soon as it is set.

Where LSF gets the job information for each factor

Factor or sub factor Gets job information from...
MEM The value for jobs that are submitted with -R "rusage[mem]"

For compound resource requirements submitted with -R "n1*{rusage[mem1]} + n2*{rusage[mem2]}" the value of MEM depends on whether resources are reserved per slot.

  • If RESOURCE_RESERVE_PER_TASK=N, then MEM=mem1+mem2
  • If RESOURCE_RESERVE_PER_TASK=Y, then MEM=n1*mem1+n2*mem2

For alternative resource requirements, use a plug-in that considers all alternatives and uses the maximum value for the resource under consideration (SWP or MEM).
SWAP The value for jobs that are submitted with the -R "rusage[swp]" option

For compound and alternative resource requirements, SWAP is determined in the same manner as MEM.
PROC The value of n for jobs that are submitted with the bsub -n command (min_task, max_task), or the value of the TASKLIMIT parameter in the lsb.queues file

Task limits can be specified at the job-level (bsub -n), the application-level (TASKLIMIT), and at the queue-level (TASKLIMIT). Job-level limits (bsub -n) override application-level TASKLIMIT, which overrides queue-level TASKLIMIT. Job-level limits must fall within the maximum and minimum limits of the application profile and the queue.

Compound resource requirements by their nature express the number of processors a job requires. The minimum number of processors that are requested by way of job-level (bsub -n), application-level (TASKLIMIT), and queue-level (TASKLIMIT) must be equal and possibly greater than the number of processors that are requested through the resource requirement. If the final term of the compound resource requirement does not specify a number of processors, then the relationship is equal to or greater than. If the final term of the compound resource requirement does specify a number of processors, then the relationship is equal to, and the maximum number of processors that are requested must be equal to the minimum requested. LSF checks only that the default value supplied in TASKLIMIT (the first value of a pair or middle value of three values) is a multiple of a block. Maximum or minimum TASKLIMIT does not need to be a multiple of the block value.

Alternative resource requirements might not specify the number of processors a job requires.

The minimum number of processors that are requested by way of job-level (bsub -n command), application-level (the TASKLIMIT parameter in the lsb.applications file), and queue-level (the TASKLIMIT parameter in the lsb.queues file) must be less than or equal the minimum that is implied through the resource requirement.

The maximum number of processors that are requested by way of job-level (the bsub -n command), application-level (the TASKLIMIT parameter in the lsb.applications file), and queue-level (the TASKLIMIT parameter in the lsb.queues file) must be equal to or greater than the maximum implied through the resource requirement. Any alternative that does not specify the number of processors is assumed to request the range from minimum to maximum, or request the default number of processors.
JPRIORITY The dynamic priority of the job, which is updated every scheduling cycle and escalated by interval that is defined in the JOB_PRIORITY_OVER_TIME parameter defined in the lsb.params file
QPRIORITY The priority of the job submission queue
FS The fair share priority value of the submission user
APP The priority of the application profile
USER The priority of the user
UG The priority of the user group