Running parallel jobs

LSF provides a generic interface to parallel programming packages so that any parallel package can be supported by writing shell scripts or wrapper programs.

How LSF runs parallel jobs
When LSF runs a job, the LSB_HOSTS variable is set to the names of the hosts running the batch job. For a parallel batch job, LSB_HOSTS contains the complete list of hosts that LSF has allocated to that job.
Preparing your environment to submit parallel jobs to LSF
Submitting a parallel job
Starting parallel tasks with LSF utilities
For simple parallel jobs you can use LSF utilities to start parts of the job on other hosts. Because LSF utilities handle signals transparently, LSF can suspend and resume all components of your job without additional programming.
Job slot limits for parallel jobs
A job slot is the basic unit of processor allocation in LSF. A sequential job uses one job slot. A parallel job that has N components (tasks) uses N job slots, which can span multiple hosts.
Specify a minimum and maximum number of tasks
By default, when scheduling a parallel job, the number of slots allocated on each host will not exceed the number of CPUs on that host even though host MXJ is set greater than number of CPUs. When submitting a parallel job, you can also specify a minimum number and a maximum number of tasks.
Restricting job size requested by parallel jobs
Specifying a list of allowed job sizes (number of tasks) in queues or application profiles enables LSF to check the requested job sizes when submitting, modifying, or switching jobs.
About specifying a first execution host
In general, the first execution host satisfies certain resource requirements that might not be present on other available hosts.
Compute units
Compute units are similar to host groups, with the added feature of granularity allowing the construction of cluster-wide structures that mimic network architecture. Job scheduling using compute unit resource requirements optimizes job placement based on the underlying system architecture, minimizing communications bottlenecks. Compute units are especially useful when running communication-intensive parallel jobs spanning several hosts. Compute units encode cluster network topology for jobs with a lot of communication between processes. For example, compute units can help minimize network latency and take advantage of fast interconnects by placing all job tasks in the same rack instead of making several network hops.
Control processor allocation across hosts
Sometimes you need to control how the selected processors for a parallel job are distributed across the hosts in the cluster.
Run parallel processes on homogeneous hosts
Parallel jobs run on multiple hosts. If your cluster has heterogeneous hosts some processes from a parallel job may for example, run on Solaris. However, for performance reasons you may want all processes of a job to run on the same type of host instead of having some processes run on one type of host and others on another type of host.
Limit the number of processors allocated
Use the TASKLIMIT parameter in lsb.queues or lsb.applications to limit the number of tasks that can be allocated to a parallel job.
Limit the number of allocated hosts
Use the HOSTLIMIT_PER_JOB parameter in lsb.queues to limit the number of hosts that a job can use. For example, if a user submits a parallel job using bsub -n 1,4096 -R "span[ptile=1]", this job requests 4096 hosts from the cluster. If you specify a limit of 20 hosts per job, a user submitting a job requesting 4096 hosts will only be allowed to use 20 hosts.
Reserve processors
Reserve memory for pending parallel jobs
Backfill scheduling
By default, a reserved job slot cannot be used by another job. To make better use of resources and improve performance of LSF, you can configure backfill scheduling.
How deadline constraint scheduling works for parallel jobs
Deadline constraint scheduling is enabled by default.
Optimized preemption of parallel jobs
You can configure preemption for parallel jobs to reduce the number of jobs suspended in order to run a large parallel job.
Controlling CPU and memory affinity
IBM® Spectrum LSF can schedule jobs that are affinity aware. This allows jobs to take advantage of different levels of processing units (NUMA nodes, sockets, cores, and threads). Affinity scheduling is supported only on Linux and Power 7 and Power 8 hosts. Affinity scheduling is supported in LSF Standard Edition and LSF Advanced Edition. Affinity scheduling is not supported on LSF Express Edition.
Processor binding for LSF job processes
Processor binding for LSF job processes takes advantage of the power of multiple processors and multiple cores to provide hard processor binding functionality for sequential LSF jobs and parallel jobs that run on a single host.
Running parallel jobs with blaunch
Learn how to configure and use the blaunch command for launching parallel and distributed applications within LSF. Task geometry allows for flexibility in how tasks are grouped for execution on system nodes. A typical LSF parallel job launches its tasks across multiple hosts. By default you can enforce limits on the total resources used by all the tasks in the job.
Running MPI workload through IBM Parallel Environment Runtime Edition
IBM Spectrum LSF integrates with the IBM Parallel Environment Runtime Edition (IBM PE Runtime Edition) program product - Version 1.3 or later to run PE jobs through the IBM Parallel Operating Environment (POE). The integration enables network-aware scheduling, allowing an LSF job to specify network resource requirements, collect network information, and schedule the job according to the requested network resources.

Parent topic: Administering IBM Spectrum LSF parallel workload