Remote Resources

You can allow the submission forward policy to consider remote resource availability before forwarding jobs. This allows jobs to be forwarded more successfully, but may result in the submission cluster only running local jobs.

Configure remote resource availability

About this task

Submission cluster scheduler considers whether remote resources exist, and only forwards jobs to a queue with free slots or space in the MultiCluster pending job threshold (IMPT_JOBBKLG).

To enable a submission forward policy and consider remote resource availability, define MC_PLUGIN_REMOTE_RESOURCE=y in lsf.conf.

When MC_PLUGIN_REMOTE_RESOURCE is defined, the following resource requirements are supported: -R "type==type_name", -R "same[type] and -R "defined(boolean resource_name)".

Note: Both remote resource availability and remote queue workload are considered by the scheduler when the parameter MC_PLUGIN_SCHEDULE_ENHANCE is defined.

The submission cluster scheduler considers whether remote resources exist, and only forwards jobs to a queue with free slots or space in the MultiCluster pending job threshold (IMPT_JOBBKLG or IMPT_SLOTBKLG). Host type and user-defined boolean resources for a host are automatically passed from the execution cluster to the submission cluster.

In some cases, you may want to forward jobs based on numeric or string resources on the execution cluster. For example, different versions of the same application may be installed on different nodes, which are naturally represented as string or numeric resources. Numeric and string resources are only passed back if they are defined in MC_RESOURCE_MATCHING_CRITERIA in lsb.params. The remote execution cluster makes the submission cluster aware of what resources (and their values) are listed so that the submission cluster can make better job forward decision. Mapping information for the remote receive queue to the execution cluster is also sent back to the submission cluster so that the submission cluster knows which remote queue can access which execution cluster.

Based on MC forwarding concepts, the job is forwarded through a channel between the send queue and the receive queue. One send queue can correspond to multiple receive queues belonging to one or more remote clusters. The forward scheduling finds the best destination queue among several candidate receive queues. Job forward scheduling is usually based on the ownership of resources visible to receive queues instead of whole remote cluster.

Remote job modification

In LSF MC forwarding mode, you can modify almost all attributes for pending jobs (in PEND and PSUSP status) and some attributes for started jobs (which include jobs in RUN, SSUSP, USUSP and WAIT status) from the submission cluster with the bmod command.

After LSB_MOD_ALL_JOBS in lsf.conf is set to Y, you can modify the following attributes of a running job:

  • cpu limit ([-c cpu_limit[/host_spec] | -cn])

  • Memory limit ([-M mem_limit | -Mn])

  • Rerunnable attribute ([-r | -rn])

  • Run limit ([-W [hour:]minute[/host_name | /host_model] | -Wn])

  • Swap limit ([-v swap_limit | -vn])

  • Standard output/error ([-o out_file | -on] [-oo out_file | -oon] [-e err_file | -en][-eo err_file | -en])