Compute unit string

A cu string specifies the network architecture-based requirements of parallel jobs. cu sections are accepted by bsub -R, and by bmod -R for non-running jobs.

Compute unit strings are supported in compound and alternative resource requirements except for the excl and balance keywords.

Syntax

The cu string supports the following syntax:

cu[type=cu_type]

Indicates the type of compute units the job can run on. Types are defined by COMPUTE_UNIT_TYPES in lsb.params. If the type keyword is not specified, the default set by COMPUTE_UNIT_TYPES is assumed.

cu[pref=bestfit | maxavail | minavail | config]

Indicates the compute unit scheduling preference, grouping hosts by compute unit before applying a first-fit algorithm to the sorted hosts. For resource reservation, the default pref=config is always used.

Compute units are ordered as follows:

  • bestfit attempts to place the job in as few compute units as possible while preferring to use compute units with fewer free slots to reduce fragmentation in the cluster. This scheduling preference does not work with the cu[balance] keyword.
  • config lists compute units in the order they appear in the ComputeUnit section of lsb.hosts. If pref is not specified, pref=config is assumed.
  • maxavail lists compute units with more free slots first. Should compute units have equal numbers of free slots, they appear in the order listed in the ComputeUnit section of lsb.hosts.
  • minavail lists compute units with fewer free slots first. Should compute units have equal numbers of free slots, they appear in the order listed in the ComputeUnit section of lsb.hosts.

Free slots include all available slots that are not occupied by running jobs.

When pref is used with the keyword balance, balance takes precedence.

Hosts accept jobs separated by the time interval set by JOB_ACCEPT_INTERVAL in lsb.params; jobs submitted closer together than this interval will run on different hosts regardless of the pref setting.

cu[maxcus=number]

Indicates the maximum number of compute units a job can run over. Jobs may be placed over fewer compute units if possible.

When used with bsub -n min, max a job is allocated the first combination satisfying both min and maxcus, while without maxcus a job is allocated as close to max as possible.

cu[usablecuslots=number]

Specifies the minimum number of slots a job must use on each compute unit it occupies. number is a non-negative integer value.

When more than one compute unit is used by a job, the final compute unit allocated can provide less than number slots if less are needed.

usablecuslots and balance cannot be used together.

cu[balance]

Indicates that a job should be split evenly between compute units, with a difference in compute unit slot allocation of at most 1. A balanced allocation spans the fewest compute units possible.

When used with bsub -n min, max the value of max is disregarded.

balance and usablecuslots cannot be used together.

When balance and pref are both used, balance takes precedence. The keyword pref is only considered if there are multiple balanced allocations spanning the same number of compute units. In this case pref is considered when choosing the allocation. balance cannot be used with the pref=bestfit scheduling preference.

When balance is used with span[ptile=X] (for X>1) a balanced allocation is one split evenly between compute units, with a difference in compute unit host allocation of at most 1.

balance cannot be used in compound and alternative resource requirements.

cu[excl]

Indicates that jobs must use compute units exclusively. Exclusivity applies to the compute unit granularity that is specified by type.

Compute unit exclusivity must be enabled by EXCLUSIVE=CU[cu_type] in lsb.queues.

excl cannot be used in compound and alternative resource requirements.

Resizable jobs

Autoresizable jobs can be submitted with compute unit resource requirements. The maxcus keyword is enforced across the job's entire allocation as it grows, while the balance and usablecuslots keywords only apply to the initial resource allocation.

Examples

  • bsub -n 11,60 -R "cu[maxcus=2:type=enclosure]" myjob

    Spans the fewest possible compute units for a total allocation of at least 11 slots using at most 2 compute units of type enclosure. In contrast, without maxcus:

  • bsub -n 11,60 myjob

    In this case, the job is allocated as close to 60 slots as possible, with a minimum of 11 slots.

  • bsub -n 64 -R "cu[balance:maxcus=4:type=enclosure]" myjob

    Spans the fewest possible compute units for a balanced allocation of 64 slots using 4 or less compute units of type enclosure. Possible balanced allocations (in order of preference) are:

    • 64 slots on 1 enclosure
    • 32 slots on 2 enclosures
    • 22 slots on 1 enclosure and 21 slots on 2 enclosures
    • 16 slots on 4 enclosures
  • bsub -n 64 -R "cu[excl:maxcus=8:usablecuslots=10]" myjob

    Allocates 64 slots over 8 or less compute units in groups of 10 or more slots per compute unit (with one compute unit possibly using less than 10 slots). The default compute unit type set in COMPUTE_UNIT_TYPES is used, and are used exclusively by myjob.

  • bsub -n 58 -R "cu[balance:type=rack:usablecuslots=20]" myjob

    Provides a balanced allocation of 58 slots with at least 20 slots in each compute unit of type rack. Possible allocations are 58 slots in 1 rack or 29 slots in 2 racks.

    Jobs submitted with balance requirements choose compute units based on the pref keyword secondarily, as shown in the following examples where cu1 has 5 available slots and cu2 has 19 available slots.

  • bsub -n 5 -R "cu[balance:pref=minavail]"

    Runs the job on compute unit cu1 where there are the fewest available slots.

  • bsub -n 5 -R "cu[balance:pref=maxavail]"

    Runs the job on compute unit cu2 where there are the most available slots. In both cases the job is balanced over the fewest possible compute units.

  • bsub -n 11,60 -R "cu[maxcus=2:type=enclosure]" -app resizable -ar myjob

    An autoresizable job that spans the fewest possible compute units for a total allocation of at least 11 slots using at most 2 compute units of type enclosure. If the autoresizable job grows, the entire job still uses at most 2 compute units of type enclosure.

  • bsub -n 64 -R "cu[balance:maxcus=4:type=enclosure]" -app resizable -ar myjob

    An autoresizable job that spans the fewest possible compute units for a balanced allocation of 64 slots using 4 or less compute units of type enclosure. If the autoresizable job grows, each subsequent allocation is a balanced allocation. The entire job (that is, the total of the initial and subsequent job allocations) still uses at most 4 compute units of type enclosure, but the job as a whole might not be a balanced allocation.

  • bsub -n 64 -R "cu[excl:maxcus=8:usablecuslots=10]" -app resizable -ar myjob

    An autoresizable job that allocates 64 slots over 8 or less compute units in groups of 10 or more slots per compute unit (with one compute unit possibly using less than 10 slots). If the autoresizable job grows, each subsequent allocation allocates in groups of 10 or more slots per compute unit (with one compute unit possible using less than 10 slots) and the entire job (that is, the total of the initial and subsequent job allocations) still uses at most 8 compute units. Since each subsequent allocation might have one compute unit that uses less than 10 slots, the entire job might have more than one compute unit that uses less than 10 slots. The default compute unit type set in COMPUTE_UNIT_TYPES is used, and are used exclusively by myjob.

CU string in application profiles

See Resource requirements for information about how resource requirements in application profiles are resolved with queue-level and job-level resource requirements.