Job Packs

Use LSF job packs to speed up the submission of a large number of jobs. With job packs, you can submit jobs by submitting a single file containing multiple job requests.

Job packs overview

Grouping jobs into packs maintains performance: while LSF is processing a job pack,mbatchd is blocked from processing other requests. Limiting the number of jobs in each pack ensures a reasonable response time for other job submissions. Job pack size is configurable.

If the cluster configuration is not consistent, and LSF receives a job pack that exceeds the job pack size defined in lsf.conf, it will be rejected.

The job packs feature supports all bsub options in the job submission file except for:

-I -Ip -Is -IS -ISp -ISs -IX -XF -K -jsdl -h -V -pack

About job packs

Enable / disable
Job packs are disabled by default. You must enable the feature before you can run bsub -pack.
Job submission rate
Using job packs to submit multiple jobs at once, instead of submitting the jobs individually minimizes system overhead and improves the overall job submission rate.
Job submission file
Create a job submission file that defines each job request. You specify all the bsub options individually for each job, so unlike chunk jobs or job arrays, the jobs in this file do not need to have anything in common. To submit the jobs to LSF, you submit the file using the bsub -pack option.
Job pack
LSF parses the file contents and submits the job requests, sending multiple requests at one time. Each group of jobs submitted together is called a job pack. The job submission file can contain any number of job requests, and LSF will group them into job packs automatically.
Job request
After the job pack is submitted, each job request in the pack is handled by LSF as if it was submitted individually with the bsub command.
For example:
  • If BSUB_CHK_RESREQ is enabled, LSF checks the syntax of the resource requirement string, instead of scheduling the job.
  • If -is or -Zs is specified, LSF copies the command file to the spool directory, and this may affect the job submission rate.
  • The job request cannot be submitted to mbatchd if the pending job or slots thresholds have been reached (MAX_PEND_JOBS and MAX_PEND_SLOTS in lsb.params or lsb.users).
  • If BSUB_QUIET is enabled, LSF will not print information about successful job submission.
Job submission errors
By default, if any job request in a file cannot be submitted to mbatchd, LSF assumes the job submission file has become corrupt, and does not process any more requests from the file (the jobs already submitted to mbatchd successfully do continue to run). Optionally, you can modify the configuration and change this. If you do, LSF processes every request in the file and attempts to submit all the jobs, even if some previous job submissions have failed.

For example, the job submission file may contain job requests from many users, but the default behavior is that LSF stops processing requests after one job fails because the pending job threshold for the user has been reached. If you change the configuration, processing of the job submission file can continue, and job requests from other users can run.

mesub
By default, LSF runs mesub as usual for all jobs in the file. Optionally, you can modify configuration and change this. If you do, LSF processes the jobs in the file without running any mesub, even if there are esubs configured at the application level (-a option of bsub), or using LSB_ESUB_METHOD in lsf.conf, or through a named esub executable under LSF_SERVERDIR.

The esub is never executed.

Enable and configure job packs

  1. Edit lsf.conf.

    These parameters will be ignored if defined in the environment instead of the lsf.conf file.

  2. Define the parameter LSB_MAX_PACK_JOBS=100.

    Do this to enable the feature and set the job pack size. We recommend 100 as the initial pack size.

    If the value is 1, jobs from the file are submitted individually, as if submitted directly using the bsub command.

    If the value is 0, job packs are disabled.

  3. Optionally, define the parameter LSB_PACK_MESUB=N.

    Do this if you want to further increase the job submission rate by preventing the execution of any mesub during job submission.

    This parameter only affects the jobs submitted using job packs, it does not affect jobs submitted in the usual way.

  4. Optionally, define the parameter LSB_PACK_SKIP_ERROR=Y.

    Do this if you want LSF to process all requests in a job submission file, and continue even if some requests have errors.

  5. Restart mbatchd to make your changes take effect.

Submit job packs

  1. Prepare the job submission file.

    Prepare a text file containing all the jobs you want to submit. Each line in the file is one job request. For each request, the syntax is identical to the bsub command line (without the word "bsub").

    For example:

    #This file contains 2 job requests.
-R "select[mem>200] rusage[mem=100]" job1.sh
-R "select[swap>400] rusage[swap=200]" job2.sh
#end

    The job submission file has the following limitations:

    • The following bsub options are not supported:

      -I -Ip -Is -IS -ISp -ISs -IX -XF -K -jsdl -h -V -pack

    • Terminal Services jobs are not supported.
    • I/O redirection is not supported.
    • Blank lines and comment lines (beginning with #) are ignored. Comments at the end of a line are not supported.
    • Backslash (\) is NOT considered a special character to join two lines.
    • Shell scripting characters are treated as plain text, they will not be interpreted.
    • Matched pairs of single and double quotations are supported, but they must have space before and after. For example, -J "job1" is supported, -J"job1" is not, and -J "job"1 is not.

    For job dependencies, a job name is recommended instead of job ID to specify the dependency condition. A job request will be rejected if the job name or job ID of the job it depends on does not already exist.

  2. Submit the job submission file.

    Use the bsub -pack option to submit all the jobs in a file.

    bsub -pack job_submission_file

    where job_submission_file is the full path to the job submission file. Do not put any other bsub options in the command line, they must be included in each individual job request in the file.

The -pack option is not supported in a job script.

Performance metrics

If you enable performance metric collection, every job submitted in a job pack is counted individually, except for the Job submission requests metric. Each job pack counts as just one job submission request.