Job dependency scheduling

Sometimes, whether a job should start depends on the result of another job. For example, a series of jobs could process input data, run a simulation, generate images based on the simulation output, and finally, record the images on a high-resolution film output device. Each step can only be performed after the previous step finishes successfully, and all subsequent steps must be aborted if any step fails.

About job dependency scheduling

Some jobs may not be considered complete until some post-job processing is performed. For example, a job may need to exit from a post-execution job script, clean up job files, or transfer job output after the job completes.

In LSF, any job can be dependent on other LSF jobs. When you submit a job, you use bsub -w to specify a dependency expression, usually based on the job states of preceding jobs.

LSF will not place your job unless this dependency expression evaluates to TRUE. If you specify a dependency on a job that LSF cannot find (such as a job that has not yet been submitted), your job submission fails.

Syntax

bsub -w 'dependency_expression'

The dependency expression is a logical expression that is composed of one or more dependency conditions.
  • To make dependency expression of multiple conditions, use the following logical operators:
    • && (AND)
    • || (OR)
    • ! (NOT)
  • Use parentheses to indicate the order of operations, if necessary.
  • Enclose the dependency expression in single quotes (') to prevent the shell from interpreting special characters (space, any logic operator, or parentheses). If you use single quotes for the dependency expression, use double quotes for quoted items within it, such as job names.
  • Job names specify only your own jobs, unless you are an LSF administrator.
  • Use double quotes (") around job names that begin with a number.
  • In Windows, enclose the dependency expression in double quotes (") when the expression contains a space. For example:
    • bsub -w "exit(678, 0)" requires double quotes in Windows.
    • bsub -w 'exit(678,0)' can use single quotes in Windows.
  • In the job name, specify the wildcard character (*) at the end of a string to indicate all jobs whose name begins with the string. For example, if you use jobA* as the job name, it specifies jobs named jobA, jobA1, jobA_test, jobA.log, etc.
    Note:

    Wildcard characters can only be used at the end of job name strings within the job dependency expression.

Multiple jobs with the same name

By default, if you use the job name to specify a dependency condition, and more than one of your jobs has the same name, all of your jobs that have that name must satisfy the test.

To change this behavior, set JOB_DEP_LAST_SUB in lsb.params to 1. Then, if more than one of your jobs has the same name, the test is done on the one submitted most recently.