Use LSF job dependencies to specify the job execution flow during job submission. LSF will hold those jobs, periodically evaluating the dependency conditions. When the dependency condition becomes true, LSF will start looking for resources for the job.
As more and more jobs are added into a dependency flow, or when dependency conditions are complicated, it can be hard to monitor a flow with the bjobs command. Use bjdepinfo to see a hierarchical view of job dependency relationships.
% bsub -J "pre-process 1" … Job <8021> is submitted to default queue <normal>. % bsub -J "pre-process 2" … Job <8022> is submitted to default queue <normal>. % bsub -J "execution[1-4]" -w "done(8021) && done(8022)" … Job <8023> is submitted to default queue <normal>. % bsub -J "post-process" -w "ended(8023)” … Job <8024> is submitted to default queue <normal>. % bjobs JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 8021 userA RUN normal hostA hostB *process 1 Oct 1 14:59 8022 userA RUN normal hostA hostB *process 2 Oct 1 14:59 8023 userA PEND normal hostA *cution[1] Oct 1 14:59 8023 userA PEND normal hostA *cution[2] Oct 1 14:59 8023 userA PEND normal hostA *cution[3] Oct 1 14:59 8023 userA PEND normal hostA *cution[4] Oct 1 14:59 8024 userA PEND normal hostA *t-process Oct 1 15:00
Use the –p option to get the job parent’s status. Use the -l option to see its dependency conditions and pending reasons. If the job has grandparents, -r level reaches grandparents recursively up to the level specified.
% bjdepinfo -lp 8023 The dependency condition of job <8023> is not satisfied: done(8021) && done(8022) JOBID PARENT PARENT_STATUS PARENT_NAME LEVEL 8023 8021 RUN *process 1 1 8023 8022 RUN *process 2 1
Query a job with the –c option to get the status of its children and to find the grandparents of specific jobs in the flow. For example, for jobs 8021 and 8022 in the previous example,-r 10 reaches grandchildren recursively to the level specified (in this case, 10). In the following example, job 8024 is the grandchild of job 8022.
% bjdepinfo -c -r 10 8022 JOBID CHILD CHILD_STATUS CHILD_NAME LEVEL 8022 8023[1] RUN *cution[1] 1 8022 8023[2] RUN *cution[2] 1 8022 8023[3] RUN *cution[3] 1 8022 8023[4] RUN *cution[4] 1 8023 8024 PEND *t-process 2