Use direct data staging (bsub -stage)

Use the bsub -stage command to specify options for direct data staging (for example, IBM CAST burst buffer).

The bsub -stage command, which is a standard LSF command, is the latest method that is available for job data management. This command option allows a user to specify options for direct data staging (including the amount of storage required on execution host, and the stage in or stage out scripts to be run). Direct data staging jobs use a fast storage layer, such as IBM CAST burst buffer, between LSF servers and a parallel file system. The burst buffer allows overlapping direct data staging jobs to copy files between submission hosts and execution hosts (bsub -f), and to stage in and stage out data for a variety of workloads, including checkpoint/restart, scratch volume, and extended memory I/O workloads.

Using direct data staging allows LSF to consider storage that is local to an execution host. LSF provides two-level stage in and stage out operations. The LSB_STAGE_IN_EXEC and LSB_STAGE_OUT_EXEC options in the lsf.conf file specify scripts that are created by the cluster administrator and executed by root. The stage in and stage out scripts specified during job submission are passed into the system level scripts. These scripts can set up and clean up a job's use of local storage. The stage in and stage out scripts run separately from the job allocation. This method of moving data files allows LSF to take advantage of execution hosts with high performance local storage and allow more flexibility when creating stage in and stage out scripts that are customized for the site.