Example data requirements job query
Use the bdata cache command to check the progress of a job with data requirements.
The following job requests three data files for staging, listing in the -data option:
bsub -o %J.out -data "hostA:/proj/std/model.tar
hostA:/proj/user1/case02341.dat hostB:/data/human/seq342138.dna" /share/bin/job_copy.sh
Job <1962> is submitted to default queue <normal>.
The job_copy.sh script contains the necessary bstage in commands to copy the required files to the job current working directory:
cat /share/bin/job_copy.sh
#!/bin/sh
bstage in -src hostA:/proj/std/model.tar -dst $LSB_JOBID/model.tar
bstage in -src hostA:/proj/user1/case02341.dat -dst $LSB_JOBID/case02341.dat
bstage in -src hostB:/data/human/seq342138.dna -dst $LSB_JOBID/seq342138.dna
The job remains pending while the data files are being staged:
bjobs -p 1962
JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
1962 user1 PEND normal submit *b_copy.sh Sep 20 16:31
Job is waiting for its data requirement to be satisfied;
After a few seconds, check the job data requirement status in the data manager:
bdata cache 1962@cluster1
Job <1962@cluster1> has the following file records in LSF data manager:
-----------------------------------------------------------------------------
INPUT:
hostA:/proj/std/model.tar
HASH STATUS REF_JOB XFER_JOB GRACE
ee4b497dc35b* STAGING 1962@cluster1 1963@cluster1 -
-----------------------------------------------------------------------------
INPUT:
hostA:/proj/user1/case02341.dat
HASH STATUS REF_JOB XFER_JOB GRACE
497de4c3b5eb* STAGING 1962@cluster1 1965@cluster1 -
-----------------------------------------------------------------------------
INPUT:
hostB:/data/human/seq342138.dna
HASH STATUS REF_JOB XFER_JOB GRACE
bb846e6d24c1* STAGING 1962@cluster1 1966@cluster1 -
After all files are staged, jobs are no longer pending:
bdata cache 1962@cluster1
INPUT:
hostA:/proj/std/model.tar
-----------------------------------------------------------------------------
HASH STATUS REF_JOB XFER_JOB GRACE
ee4b497dc35b* TRANSFERRED 1962@cluster1 1963@cluster1 -
INPUT:
hostA:/proj/user1/case02341.dat
-----------------------------------------------------------------------------
HASH STATUS REF_JOB XFER_JOB GRACE
497de4c3b5eb* TRANSFERRED 1962@cluster1 1965@cluster1 -
INPUT:
hostB:/data/human/seq342138.dna
-----------------------------------------------------------------------------
HASH STATUS REF_JOB XFER_JOB GRACE
bb846e6d24c1* TRANSFERRED 1962@cluster1 1966@cluster1 -
bjobs 1962
JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
1962 user1 RUN normal submit compute *in/job.sh Sep 20 16:31