Specify a dependency on the start or submission of specific jobs
Use the Starts dependency type to specify a dependency on the start of a job, or the Is submitted dependency type to specify a dependency on the submission of a job.
By default, when you establish a dependency on a job to complete successfully, dependent jobs are not submitted until dependencies are satisfied. For example, job124 depends on job 123. Job 123 is submitted, and job 124 is not submitted until job 123 completes.
In some cases, you might have a data preparation job that takes a long time, followed by a computation job. If you have a busy cluster and your second job gets submitted after the first job is completed, your second job might be waiting a long time in the queue to be scheduled. For these kinds of cases, you can specify to LSF Process Manager to pre-submit dependent jobs, reducing the time that a job waits in the queue to be scheduled.
Examples:
- You want your job to be started as early as possible.
For example, you have job 123 followed by job 124. You want job 124 to be submitted as early as possible. In this case, you pre-submit job 124 and specify the Is submitted dependency. Job 124 will be submitted right after job 123 is submitted to LSF.
- The next job cannot start until the execution host is known for the previous job.
For example, job124 needs to run on the same host as the preceding job. You pre-submit job 124 and specify the Starts dependency. Job 124 will be submitted right after job 123 started to run in LSF.
Requirements to pre-submit dependent jobs
- Only LSF jobs, job arrays, job scripts, job array scripts, template jobs, proxy job events, and proxy job array events can be pre-submitted.
- Only jobs, job scripts, job arrays, job array scripts, and template jobs can be preceding jobs to the dependent job to be pre-submitted.
- The jobs to be pre-submitted must be direct links. They cannot be more than one link away.
- The dependencies of all predecessors must be logically connected with AND.
- In the Flow Editor, the Event type in the Job Event Definition for the preceding jobs to the other job must be set to Starts or Is submitted.
- If you specify dependent jobs to be pre-submitted and the condition is never met, it is possible for the flow to be “stuck”. To handle this situation, define an overrun exception handler to kill the last job if it runs or pends for more than a certain period.
Examples
Example: Simple flow, pre-submission with "Starts" dependency
In this flow, you can specify only J2 and J4 to pre-submit J5. As a result, J5 is submitted right after J2 and J4 start to run in LSF.
J1 and J3 cannot be considered because they are more than one link away from J5.
Example: Simple flow, pre-submission with "Is submitted" dependency
In this flow, you can specify only J2 and J4 to pre-submit J5. As a result, J5 is submitted right after J2 and J4 are submitted to LSF.
J1 and J3 cannot be considered because they are more than one link away from J5.
Example: Pre-submission not possible
In this flow, you cannot specify any pre-submission because jobs must be logically connected with AND.
Example: Complex flow with "Starts" dependency
In this flow, you can specify only J1 to pre-submit J5. As a result, J5 will be submitted right after J1 starts to run in LSF.
J3 and J4 cannot be considered because they are more than one link away from J5, and there is also a logical OR.
