About resource requirements
Resource requirements define which hosts a job can run on. Each job has its resource requirements and hosts that match the resource requirements are the candidate hosts. When LSF schedules a job, it uses the load index values of all the candidate hosts. The load values for each host are compared to the scheduling conditions. Jobs are only dispatched to a host if all load values are within the scheduling thresholds.
By default, if a job has no resource requirements, LSF places it on a host of the same type as the submission host (that is, type==local). However, if a job has string or Boolean resource requirements specified and the host type has not been specified, LSF places the job on any host (that is, type==any) that satisfies the resource requirements.
To override the LSF defaults, specify resource requirements explicitly. Resource requirements can be set for queues, for application profiles, or for individual jobs.
To best place a job with optimized performance, resource requirements can be specified for each application. This way, you do not have to specify resource requirements every time you submit a job. The LSF administrator may have already configured the resource requirements for your jobs, or you can put your executable name together with its resource requirements into your personal remote task list.
The bsub command automatically uses the resource requirements of the job from the remote task lists.
A resource requirement is an expression that contains resource names and operators.
Compound resource requirements
In some cases different resource requirements may apply to different parts of a parallel job. The first execution host, for example, may require more memory or a faster processor for optimal job scheduling. Compound resource requirements allow you to specify different requirements for some slots within a job in the queue-level, application-level, or job-level resource requirement string.
Compound resource requirement strings can be set by the application-level or queue-level RES_REQ parameter, or used with bsub -R when a job is submitted. bmod -R accepts compound resource requirement strings for pending jobs but not running jobs.
Special rules take effect when compound resource requirements are merged with resource requirements defined at more than one level. If a compound resource requirement is used at any level (job, application, or queue) the compound multi-level resource requirement combinations described later in this chapter apply.
The same resource requirement can be used within each component expression (simple resource requirement). For example, suppose static strings resource res1 and res2 are defined. We permit a resource requirement such as:
"4*{select[io] same[res1]} + 4*{select[compute] same[res1]}"
With this resource requirement, there are two simple sub-expressions, R1 and R2. For each of these sub-expressions, all slots must come from hosts with equal values of res1. However, R1 may occupy hosts of a different value than those occupied by R2.
You can specify a global same requirement that takes effect over multiple sub-expressions of a compound resource requirement string. For example,
"{4*{select[io]} + 4*{select[compute]}} same[res1]"
This syntax allows users to express that both sub-expressions must reside on hosts that have a common value for res1.
In general, there may be more than two sub-expressions in a compound resource requirement. The global same will apply to all of them.
Arbitrary nesting of brackets is not permitted. For example, you cannot have a global same apply to only two of three sub-expressions of a compound resource requirement. However, each sub-expression can have its own local same as well as a global same for the compound expression as a whole. For example, the following is permitted:
"{4*{same[res1]} + 4*{same[res1]}} same[res2]"
In addition, a compound resource requirement expression with a global same may be part of a larger alternative resource requirement string.
A compound resource requirement expression with a global same can be used in the following instances:
- Submitting a job: bsub -R "rsrc_req_string" <other_bsub_options> a.out
- Configuring application profile (lsb.applications): RES_REQ = "rsrc_req_string"
- Queue configuration (lsb.queues): RES_REQ = "rsrc_req_string"
Syntax:
- A single compound resource requirement:
"{ compound_rsrc_req } same[ same_str ]"
- A compound resource requirement within an alternative resource requirement:
"{{ compound_rsrc_req } same[ same_str ]} || {R}"
- A compound resource requirement within an alternative resource requirement with delay:
"{R} || {{ compound_rsrc_req } same[ same_str ]}@D"
where D is a positive integer.
- Compound resource requirements cannot contain the || operator. Compound resource requirements cannot be defined (included) in any multiple -R options.
- Compound resource requirements cannot contain the compute unit (cu) keywords balance or excl, but works normally with other cu keywords (including pref, type, maxcus, and usablecuslots).
- Resizable jobs can have compound resource requirements, but only the portion of the job represented by the last term of the compound resource requirement is eligible for automatic resizing. When using bresize release to release slots, you can only release slots represented by the last term of the compound resource requirement. To release slots in earlier terms, run bresize release repeatedly to release slots in subsequent last terms.
- Compound resource requirements cannot be specified in the definition of a guaranteed resource pool.
- Resource allocation for parallel jobs using compound resources is done for each compound resource term in the order listed instead of considering all possible combinations. A host rejected for not satisfying one resource requirement term will not be reconsidered for subsequent resource requirement terms.
- Compound resource requirements were introduced in LSF Version 7 Update 5, and are not compatible with earlier versions of LSF.
Alternative resource requirements
In some circumstances more than one set of resource requirements may be acceptable for a job to be able to run. LSF provides the ability to specify alternative resource requirements.
An alternative resource requirement consists of two or more individual simple or compound resource requirements. Each separate resource requirement describes an alternative. When a job is submitted with alternative resource requirements, the alternative resource picked must satisfy the mandatory first execution host. If none of the alternatives can satisfy the mandatory first execution host, the job will PEND.
Alternative resource requirement strings can be specified at the application-level or queue-level RES_REQ parameter, or used with bsub -R when a job is submitted. bmod -R also accepts alternative resource requirement strings for pending jobs.
The rules for merging job, application, and queue alternative resource requirements are the same as for compound resource requirements.
- Multiple bsub -R commands
- TS jobs, including those with the tssub command
- Hosts from HPC integrations that use toplib, including CPUset and Blue Gene.
- Compute unit (cu) sections specified with balance or excl keywords.
If a job with alternative resource requirements specified is re-queued, it will have all alternative resource requirements considered during scheduling. If a @D delay time is specified, it is interpreted as waiting, starting from the original submission time. For a restart job, @D delay time starts from the restart job submission time.
Resource requirements in application profiles
See Resource requirements for information about how resource requirements in application profiles are resolved with queue-level and job-level resource requirements.
Re-sizable jobs and resource requirements
In general, resize allocation requests for re-sizable jobs use the resource requirements of the running job. When the resource requirement string for a job is modified with bmod -R, the new string takes effects for a job resize request. The resource requirement of the allocation request is merged from resource requirements specified at the queue, job, and application levels.