Submitting direct data staging jobs with CSM

Submit direct data staging jobs (for example, IBM CAST burst buffer) with CSM for IBM POWER.

About this task

Submit a direct data staging job and observe LSF behavior.

Procedure

  1. Use the bsub -stage option to submit a job with data staging options.

    bsub -nnodes num_nodes -stage " [storage= min_size [ , max_size ]] [:in=path_to_stage_in_script ] [:out=path_to_stage_out_script ] -q data_queue command

    For example,

    bsub -nnodes 1 -stage "storage=5:in=/u/usr1/mystagein.pl:out=/home/mystagein.pl" -q bbq myjob

  2. Use the bjobs -data job_id option to view the data options for the submitted job.
  3. Manage data staging details for the job.

    After you submit a job with data staging options, LSF automatically submits a stage in job to the transfer queue, and later submits a stage out job also to the transfer queue. Stage out jobs are always submitted.

    LSF can export the following environment variables in the user scripts to the stage in/out transfer jobs:

    LSF_STAGE_JOBID
    The original job's base ID.
    LSF_STAGE_JOBINDEX
    The original job's array index.
    LSF_STAGE_USER
    The original job's submission user.
    LSF_STAGE_HOSTS
    Allocated host list for the original job.
    LSF_STAGE_STORAGE_MINSIZE
    Minimum SSD storage space.
    LSF_STAGE_STORAGE_MAXSIZE
    Target maximum SSD storage space.
    LSF_STAGE_USER_STAGE_IN
    User stage in script.
    LSF_STAGE_USER_STAGE_OUT
    User stage out script.
    LSF_STAGE_IN_STATUS
    Stage in job exit code.
    LSF_STAGE_JOB_STATUS
    The original job exit code.
    LSF_STAGE_HOSTFILE
    A file that contains all hosts that are reserved for the data job, including the names of the launch node.
    CSM_ALLOCATION_ID
    CSM allocation ID of the original job.
    Note: If the administrator stage in script exits with code 125, LSF kills the pending user job and triggers a stage out. Then, this job does not go back to the system to wait to be rescheduled.
  4. View the data staging details for the job.
    1. Use the bhist -l option to view the original data staging job and the stage in and stage out transfer jobs.
    2. Use the CSM command csm_allocation_query_details -a allocation_id to view the state transition of the allocation.