lsb.applications

The lsb.applications file defines application profiles. Use application profiles to define common parameters for the same type of jobs, including the execution requirements of the applications, the resources they require, and how they should be run and managed.

This file is optional. Use the DEFAULT_APPLICATION parameter in the lsb.params file to specify a default application profile for all jobs. LSF does not automatically assign a default application profile.

This file is installed by default in the LSB_CONFDIR/cluster_name/configdir directory.

Changing lsb.applications configuration

After you change the lsb.applications file, run the badmin reconfig command to reconfigure the mbatchd daemon. Configuration changes apply to pending jobs only. Running jobs are not affected.

lsb.applications structure

Each application profile definition begins with the line Begin Application and ends with the line End Application. The application name must be specified. All other parameters are optional.

Example

Begin Application
NAME         = catia
DESCRIPTION  = CATIA V5
CPULIMIT     = 24:0/hostA      # 24 hours of host hostA
FILELIMIT    = 20000
DATALIMIT    = 20000           # jobs data segment limit
CORELIMIT    = 20000
TASKLIMIT    = 5               # job processor limit
REQUEUE_EXIT_VALUES = 55 34 78
End Application

See the lsb.applications template file for other application profile examples.

#INCLUDE

Syntax

#INCLUDE "path-to-file"

Description

Inserts a configuration setting from another file to the current location. Use this directive to dedicate control of a portion of the configuration to other users or user groups by providing write access for the included file to specific users or user groups, and to ensure consistency of configuration file settings in different clusters (if you are using the LSF multicluster capability).

For more information, see Shared configuration file content.

#INCLUDE can be inserted anywhere in the local configuration file.

Default

Not defined.

ABS_RUNLIMIT

Syntax

ABS_RUNLIMIT=y | Y

Description

If set, absolute (wall-clock) run time is used instead of normalized run time for all jobs submitted with the following values:
  • Run time limit or run time estimate specified by the -W or -We option of bsub
  • RUNLIMIT queue-level parameter in lsb.queues
  • RUNLIMIT application-level parameter in lsb.applications
  • ESTIMATED_RUNTIME parameter in lsb.applications

The runtime estimates and limits are not normalized by the host CPU factor.

Default

Not defined. Run limit and runtime estimate are normalized.

BIND_JOB

BIND_JOB specifies the processor binding policy for sequential and parallel job processes that run on a single host. On Linux execution hosts that support this feature, job processes are hard bound to selected processors.

Syntax

BIND_JOB=NONE | BALANCE | PACK | ANY | USER | USER_CPU_LIST

Description

Note: BIND_JOB is deprecated in LSF Standard Edition and LSF Advanced Edition. You should enable LSF CPU and memory affinity scheduling in with the AFFINITY parameter in lsb.hosts. If both BIND_JOB and affinity scheduling are enabled, affinity scheduling takes effect, and LSF_BIND_JOB is disabled. BIND_JOB and LSF_BIND_JOB are the only affinity options available in LSF Express Edition.

If processor binding feature is not configured with the BIND_JOB parameter in an application profile in lsb.applications, the LSF_BIND_JOB configuration setting lsf.conf takes effect. The application profile configuration for processor binding overrides the lsf.conf configuration.

For backwards compatibility:
  • BIND_JOB=Y is interpreted as BIND_JOB=BALANCE
  • BIND_JOB=N is interpreted as BIND_JOB=NONE

Supported platforms

Linux with kernel version 2.6 or higher

Default

Not defined. Processor binding is disabled.

CHKPNT_DIR

Syntax

CHKPNT_DIR=chkpnt_dir

Description

Specifies the checkpoint directory for automatic checkpointing for the application. To enable automatic checkpoint for the application profile, administrators must specify a checkpoint directory in the configuration of the application profile.

If CHKPNT_PERIOD, CHKPNT_INITPERIOD or CHKPNT_METHOD was set in an application profile but CHKPNT_DIR was not set, a warning message is issued and those settings are ignored.

The checkpoint directory is the directory where the checkpoint files are created. Specify an absolute path or a path relative to the current working directory for the job. Do not use environment variables in the directory path.

If checkpoint-related configuration is specified in the queue, application profile, and at job level:
  • Application-level and job-level parameters are merged. If the same parameter is defined at both job-level and in the application profile, the job-level value overrides the application profile value.
  • The merged result of job-level and application profile settings override queue-level configuration.

To enable checkpointing of jobs with the LSF multicluster capability, define a checkpoint directory in an application profile (CHKPNT_DIR, CHKPNT_PERIOD, CHKPNT_INITPERIOD, CHKPNT_METHOD in lsb.applications) of both submission cluster and execution cluster. LSF uses the directory specified in the execution cluster.

Checkpointing is not supported if a job runs on a leased host.

The file path of the checkpoint directory can contain up to 4000 characters for UNIX and Linux, or up to 255 characters for Windows, including the directory and file name.

Default

Not defined

CHKPNT_INITPERIOD

Syntax

CHKPNT_INITPERIOD=init_chkpnt_period

Description

Specifies the initial checkpoint period in minutes. CHKPNT_DIR must be set in the application profile for this parameter to take effect. The periodic checkpoint specified by CHKPNT_PERIOD does not happen until the initial period has elapse.

Specify a positive integer.

Job-level command line values override the application profile configuration.

If administrators specify an initial checkpoint period and do not specify a checkpoint period (CHKPNT_PERIOD), the job will only checkpoint once.

If the initial checkpoint period of a job is specified, and you run bchkpnt to checkpoint the job at a time before the initial checkpoint period, the initial checkpoint period is not changed by bchkpnt. The first automatic checkpoint still happens after the specified number of minutes.

Default

Not defined

CHKPNT_PERIOD

Syntax

CHKPNT_PERIOD=chkpnt_period

Description

Specifies the checkpoint period for the application in minutes. CHKPNT_DIR must be set in the application profile for this parameter to take effect. The running job is check-pointed automatically every checkpoint period.

Specify a positive integer.

Job-level command line values override the application profile and queue level configurations. Application profile level configuration overrides the queue level configuration.

Default

Not defined

CHKPNT_METHOD

Syntax

CHKPNT_METHOD=chkpnt_method

Description

Specifies the checkpoint method. CHKPNT_DIR must be set in the application profile for this parameter to take effect. Job-level command line values override the application profile configuration.

Default

Not defined

CHUNK_JOB_SIZE

Syntax

CHUNK_JOB_SIZE=integer

Description

Note: This parameter is deprecated and might be removed in a future version of LSF.

Chunk jobs only. Allows jobs submitted to the same application profile to be chunked together and specifies the maximum number of jobs allowed to be dispatched together in a chunk. Specify a positive integer greater than or equal to 1.

All of the jobs in the chunk are scheduled and dispatched as a unit, rather than individually.

Specify CHUNK_JOB_SIZE=1 to disable job chunking for the application. This value overrides chunk job dispatch configured in the queue.

Use the CHUNK_JOB_SIZE parameter to configure application profiles that chunk small, short-running jobs. The ideal candidates for job chunking are jobs that have the same host and resource requirements and typically take 1 to 2 minutes to run.

Job chunking can have the following advantages:
  • Reduces communication between sbatchd and mbatchd and reduces scheduling overhead in mbschd.
  • Increases job throughput in mbatchd and CPU utilization on the execution hosts.

However, throughput can deteriorate if the chunk job size is too big. Performance may decrease on profiles with CHUNK_JOB_SIZE greater than 30. You should evaluate the chunk job size on your own systems for best performance.

With the job forwarding model for the LSF multicluster capability, this parameter does not affect jobs that are forwarded to a remote cluster.

LSF ignores this parameter does not chunk jobs under the following conditions:
  • CPU limit greater than 30 minutes (CPULIMIT parameter in lsb.queues or lsb.applications)
  • Run limit greater than 30 minutes (RUNLIMIT parameter in lsb.queues or lsb.applications)
  • Runtime estimate greater than 30 minutes (ESTIMATED_RUNTIME parameter in lsb.applications)

Default

Not defined

CONTAINER

Syntax

CONTAINER=podman[image(image_name) options(podman_run_options)]

CONTAINER=docker[image(image_name) options(docker_run_options) container_io() job_pre_post()]

CONTAINER=nvidia-docker[image(image_name) options(docker_run_options)]

CONTAINER=shifter[image(image_name) options(container_options)]

CONTAINER=singularity[image(image_name) options(container_options)]

CONTAINER=apptainer[image(image_name) options(apptainer_options]

CONTAINER=enroot[image(image_name) options(enroot_start_options)]

Description

Enables LSF to use a supported container for jobs that are submitted to this application profile. This parameter uses the following keywords:

podman | docker | nvidia-docker | shifter | singularity | apptainer | enroot
Required. Use one of these keywords to specify the type of container to use for jobs that are submitted to this application profile.
image
Required. This keyword specifies the image name that is used in running jobs.

For Podman, Docker, NVIDIA Docker, and Enroot jobs, use the $LSB_CONTAINER_IMAGE environment variable to allow users to specify the image name for the container jobs at job submission time. At job submission time, users can specify a specific image name that is in the specified repository server by specifying a value for the $LSB_CONTAINER_IMAGE environment variable.

options
Optional. This keyword specifies the job run options for the container.

To enable a pre-execution script to run, specify an at sign (@) and a full file path to the script, which the execution host must be able to access. Before the container job runs, LSF runs this script with LSF administrator privileges. While the script is running, the jobs' environment variables are passed to the script. When the script finishes running, the output is used as container startup options. The script must provide this output on one line. The method of processing the container job depends on the result of the pre-execution script:

  • If the pre-execution script failed, the container job exits with the same exit code from the script. In addition, an external status message is sent to inform the user that the job exited because of script execution failure.
  • If the execution of the script is successful but the output contains more than 512 options, LSF only keeps the first 512 options, and the remaining options are ignored.
  • If the execution of the script is successful and the output is valid, the output is part of the container job running options. The position of the output from the script in the options is exactly where the user configured the script in the options field.
For Docker and NVIDIA Docker containers, this keyword specifies the Docker job run options for the docker run command, which are passed to the job container.
Note:
  • Before you specify the Docker job run options, make sure that these options work with the docker run command in the command line.
  • The --cgroup-parent and --user (or -u) options are reserved for LSF internal use. Do not use these options in the options keyword configuration, otherwise the job fails.

    If you specified a pre-execution script and the output of this script contains --cgroup-parent, --user, or -u, the container job also fails.

  • The -w and --ulimit options are automatically set for LSF. Do not use these options in the options keyword configuration because these specifications override the LSF settings.
  • The -v option is automatically used by LSF to mount the working directories that LSF requires, such as the current working directory, job spool directory, destination file for the bsub -f command, tmp directory, the LSF_TOP, and the checkpoint directory on demand.
  • You can configure the --rm option in the options keyword configuration to automatically remove containers after the job is done.
  • You can enable LSF to automatically assign a name to a Docker container when it creates the Docker container. To enable this feature, set the ENABLE_CONTAINER_NAME parameter to True in the lsfdockerlib.py file.

    The container name uses the following naming convention:

    • Normal jobs and blaunch parallel job containers: <cluster_name>.job.<job_id>
    • Array jobs and array blaunch parallel job containers: <cluster_name>.job.<job_id>.<job_index>
    • blaunch parallel job task containers: <cluster_name>.job.<job_id>.task.<task_id>
    • Array blaunch parallel job task containers: <cluster_name>.job.<job_id>.<job_index>.task.<task_id>
  • Limitation: If you use the -d option, LSF incorrectly gets the status of the Docker jobs as DONE.
For Shifter containers, this keyword specifies the Shifter job run options for the shifter command, which are passed to the job container.
Note:
  • Run shifter --help in the command line to view the options that the shifter command supports.
  • Before you specify the Shifter job run options, make sure that these options work with the shifter command in the command line.
  • The $LD_LIBRARY_PATH directory is cleaned according to the setuid bit that Shifter uses to work. Therefore, for programs that depend on $LD_LIBRARY_PATH to work (such as openmpi), ensure that the setuid bit can be properly set inside the container by adding LD_LIBRARY_PATH to the siteEnvAppend section of the udiRoot.conf file.
For Singularity or Apptainer containers, this keyword specifies the Singularity or Apptainer job run options for the singularity exec or or apptainer exec command, which are passed to the job container.
Note:
  • Run singularity exec --help or apptainer exec --help in the command line to view the options that the singularity or apptainer command supports.
  • Before you specify the Singularity or Apptainer job run options, make sure that these options work with the singularity exec or apptainer exec command in the command line.
  • The $LD_LIBRARY_PATH directory is cleaned according to the setuid bit that Singularity or Apptainer uses to work. Therefore, for programs that depend on $LD_LIBRARY_PATH to work (such as openmpi), ensure that the setuid bit can be properly set inside the container by adding LD_LIBRARY_PATH to the ld.so.conf file and run the ldconfig command.
For Podman containers, this keyword specifies the Podman job run options for the podman run command, which are passed to the job container.
Note:
  • Before you specify the Podman job run options, make sure that these options work with the podman run command in the command line.
  • The --user (or -u) option is reserved for LSF internal use. Do not use these options in the options keyword configuration, otherwise the job fails.

    If you specified a pre-execution script and the output of this script contains --user, or -u, the container job also fails.

  • The -w and --ulimit options are automatically set for LSF. Do not use these options in the options keyword configuration because these specifications override the LSF settings.
  • The -v option is automatically used by LSF to mount the working directories that LSF requires, such as the current working directory, job spool directory, destination file for the bsub -f command, tmp directory, the LSF_TOP, and the checkpoint directory on demand.
  • You can configure the --rm option in the options keyword configuration to automatically remove containers after the job is done.
  • Limitation: If you use the -d option, LSF incorrectly gets the status of the Docker jobs as DONE.
For Enroot containers, this keyword specifies the Enroot job run options for the enroot start command, which are passed to the job container.
Note: Before you specify the Enroot job run options, make sure that these options work with the enroot start command in the command line.

Examples

To specify an Ubuntu image for use with container jobs without specifying any optional keywords:
Begin Application
NAME = podmanapp
CONTAINER = podman[image(repository.example.com:5000/file/path/ubuntu:latest)]
DESCRIPTION = Podman User Service
End Application
Begin Application
NAME = dockerapp
CONTAINER = docker[image(repository.example.com:5000/file/path/ubuntu:latest)]
DESCRIPTION = Docker User Service
End Application
Begin Application
NAME = ndockerapp
CONTAINER = nvidia-docker[image(repository.example.com:5000/file/path/ubuntu:latest)]
DESCRIPTION = NVIDIA Docker User Service
End Application
Begin Application
NAME = shifterapp
CONTAINER = shifter[image(ubuntu:latest)]
DESCRIPTION = Shifter User Service
End Application
Begin Application
NAME = singapp
CONTAINER = singularity[image(/file/path/ubuntu.img)]
DESCRIPTION = Singularity User Service
End Application
Begin Application
NAME = apptainer
CONTAINER = apptainer[image(/share/apptainer/images/ubuntu_latest.sif)]
DESCRIPTION = Apptainer User Service
End Application
Begin Application
NAME = enrootapp
CONTAINER = enroot[image(repository.example.com:5000/file/path/ubuntu:latest)]
DESCRIPTION = Enroot User Service
End Application
To specify a pre-execution script in the /share/usr/ directory, which generates the container startup options:
Begin Application
NAME = dockerappoptions
CONTAINER = docker[image(repository.example.com:5000/file/path/ubuntu:latest) options(@/share/usr/docker-options.sh)]
DESCRIPTION = Docker User Service with pre-execution script for options
End Application
Begin Application
NAME = ndockerappoptions
CONTAINER = nvidia-docker[image(repository.example.com:5000/file/path/ubuntu:latest) options(@/share/usr/docker-options.sh)]
DESCRIPTION = NVIDIA Docker User Service with pre-execution script for options
End Application
Begin Application
NAME = shifterappoptions
CONTAINER = shifter[image(ubuntu:latest) options(@/share/usr/shifter-options.sh)]
DESCRIPTION = Shifter User Service
End Application
Begin Application
NAME = singappoptions
CONTAINER = singularity[image(/file/path/ubuntu.img) options(@/share/usr/sing-options.sh)]
DESCRIPTION = Singularity User Service
End Application
Begin Application
NAME = apptainer
CONTAINER = apptainer[image(/share/apptainer/images/ubuntu_latest.sif) options(@/share/usr/apptainer-options.sh)]
DESCRIPTION = Apptainer User Service
End Application
Begin Application
NAME = enrootappoptions
CONTAINER = enroot[image(repository.example.com:5000/file/path/ubuntu:latest) options(@/share/usr/enroot-options.sh)]
DESCRIPTION = Enroot User Service with pre-execution script for options
End Application
  • For sequential jobs, specify the following CONTAINER parameter value for LSF to automatically remove containers after the job is done:

    CONTAINER=docker[image(image-name) options(--rm)]

  • For parallel jobs, the network and IPC must work across containers to make blaunch work. The execution user ID and user name mapping file must be mounted to the container for blaunch authentication.

    Therefore, specify the following CONTAINER parameter value for LSF to configure the container IPC and network parameters so that blaunch can work across multiple containers, to configure the container password file for blaunch authentication, and automatically remove containers after the job is done:

    CONTAINER=docker[image(image-name) options(--rm --network=host --ipc=host -v --runtime=nvidia /path/to/my/passwd:/etc/passwd)]

    The passwd file must be in the standard format for UNIX and Linux password files, such as the following format:

    user1:x:10001:10001:::
    user2:x:10002:10002:::
  • To allow users to specify image names for Podman, Docker, NVIDIA Docker, and Enroot container jobs at job submission time, use the $LSB_CONTAINER_IMAGE environment variable as the image name when specifying the image keyword.
    For example, when defining the CONTAINER parameter for the udocker application profile, add the $LSB_CONTAINER_IMAGE environment variable to the image specification:
    Begin Application
    NAME = udockerGPU
    CONTAINER = docker[image(repository.example.com:5000/$LSB_CONTAINER_IMAGE) \
                options(--rm --net=host --ipc=host -v --runtime=nvidia /gpfs/u/fred:/data )]
    DESCRIPTION = Docker User Service
    End Application
    Specify a container image name (such as ubuntu) at the job submission time by setting the $LSB_CONTAINER_IMAGE environment using one of the following methods:
    • Specify the $LSB_CONTAINER_IMAGE environment variable according to your shell environment:
      • In csh or tcsh:

        setenv LSB_CONTAINER_IMAGE ubuntu

      • In sh, ksh, or bash:

        export LSB_CONTAINER_IMAGE=ubuntu

    • Use the bsub -env option:

      bsub -env LSB_CONTAINER_IMAGE=ubuntu -app udocker a.out -in in.dat -out out.dat

    • Use an esub script to set the LSB_CONTAINER_IMAGE environment variable, then call the esub with the bsub command.
      For example, create an esub.docker script in the $LSF_SERVERDIR directory with the following contents:
      #!/bin/sh
      exec 1>&2
      echo "LSB_CONTAINER_IMAGE=∖"$1∖"" >> $LSB_SUB_MODIFY_ENVFILE
      Submit a job to call the esub.docker script by running the following command:
      bsub -a "docker(ubuntu)" -app udocker a.out -in in.dat -out out.dat

Default

Undefined

CORELIMIT

Syntax

CORELIMIT=integer

Description

The per-process (soft) core file size limit for all of the processes belonging to a job from this application profile (see getrlimit(2)). Application-level limits override any default limit specified in the queue, but must be less than the hard limit of the submission queue. Job-level core limit (bsub -C) overrides queue-level and application-level limits.

By default, the limit is specified in KB. Use LSF_UNIT_FOR_LIMITS in lsf.conf to specify a larger unit for the limit (MB, GB, TB, PB, or EB).

Default

Unlimited

CPU_FREQUENCY

Syntax

CPU_FREQUENCY=[float_number][unit]

Description

Specifies the CPU frequency for an application profile. All jobs submit to the application profile require the specified CPU frequency. Value is a positive float number with units (GHz, MHz, or KHz). If no units are set, the default is GHz.

This value can also be set using the command bsub –freq.

The submission value will overwrite the application profile value, and the application profile value will overwrite the queue value.

Default

Not defined (Nominal CPU frequency is used)

CPULIMIT

Syntax

CPULIMIT=[[hour:]minute[/host_name | /host_model]

Description

Limits the total CPU time the job can use. This parameter is useful for preventing runaway jobs or jobs that use up too many resources.

When the total CPU time for the whole job has reached the limit, a SIGXCPU signal is sent to all processes belonging to the job. If the job has no signal handler for SIGXCPU, the job is killed immediately. If the SIGXCPU signal is handled, blocked, or ignored by the application, then after the grace period expires, LSF sends SIGINT, SIGTERM, and SIGKILL to the job to kill it.

If a job dynamically spawns processes, the CPU time used by these processes is accumulated over the life of the job.

Processes that exist for fewer than 30 seconds may be ignored.

By default, jobs submitted to the application profile without a job-level CPU limit (bsub -c) are killed when the CPU limit is reached. Application-level limits override any default limit specified in the queue.

The number of minutes may be greater than 59. For example, three and a half hours can be specified either as 3:30 or 210.

If no host or host model is given with the CPU time, LSF uses the default CPU time normalization host defined at the queue level (DEFAULT_HOST_SPEC in lsb.queues) if it has been configured, otherwise uses the default CPU time normalization host defined at the cluster level (DEFAULT_HOST_SPEC in lsb.params) if it has been configured, otherwise uses the host with the largest CPU factor (the fastest host in the cluster).

On Windows, a job that runs under a CPU time limit may exceed that limit by up to SBD_SLEEP_TIME. This is because sbatchd periodically checks if the limit has been exceeded.

On UNIX systems, the CPU limit can be enforced by the operating system at the process level.

You can define whether the CPU limit is a per-process limit enforced by the OS or a per-job limit enforced by LSF with LSB_JOB_CPULIMIT in lsf.conf.

Default

Unlimited

DATALIMIT

Syntax

DATALIMIT=integer

Description

The per-process (soft) data segment size limit (in KB) for all of the processes belonging to a job running in the application profile (see getrlimit(2)).

By default, jobs submitted to the application profile without a job-level data limit (bsub -D) are killed when the data limit is reached. Application-level limits override any default limit specified in the queue, but must be less than the hard limit of the submission queue.

Default

Unlimited

DESCRIPTION

Syntax

DESCRIPTION=text

Description

Description of the application profile. The description is displayed by bapp -l.

The description should clearly describe the service features of the application profile to help users select the proper profile for each job.

The text can include any characters, including white space. The text can be extended to multiple lines by ending the preceding line with a backslash (\). The maximum length for the text is 512 characters.

DJOB_COMMFAIL_ACTION

Syntax

DJOB_COMMFAIL_ACTION="KILL_TASKS|IGNORE_COMMFAIL"

Description

Defines the action LSF should take if it detects a communication failure with one or more remote parallel or distributed tasks. If defined with "KILL_TASKS", LSF tries to kill all the current tasks of a parallel or distributed job associated with the communication failure. If defined with "IGNORE_COMMFAIL", failures will be ignored and the job continues. If not defined, LSF terminates all tasks and shuts down the entire job.

This parameter only applies to the blaunch distributed application framework.

When defined in an application profile, the LSB_DJOB_COMMFAIL_ACTION variable is set when running bsub -app for the specified application.

Default

Not defined. Terminate all tasks, and shut down the entire job.

DJOB_DISABLED

Syntax

DJOB_DISABLED=Y | N

Description

Disables the blaunch distributed application framework.

Default

Not defined. Distributed application framework is enabled.

DJOB_ENV_SCRIPT

Syntax

DJOB_ENV_SCRIPT=script_name

Description

Defines the name of a user-defined script for setting and cleaning up the parallel or distributed job environment.

The specified script must support a setup argument and a cleanup argument. The script is executed by LSF with the setup argument before launching a parallel or distributed job, and with argument cleanup after the job is finished.

The script runs as the user, and is part of the job.

If a full path is specified, LSF uses the path name for the execution. Otherwise, LSF looks for the executable from $LSF_BINDIR.

This parameter only applies to the blaunch distributed application framework.

When defined in an application profile, the LSB_DJOB_ENV_SCRIPT variable is set when running bsub -app for the specified application.

The command path can contain up to 4094 characters for UNIX and Linux, or up to 255 characters for Windows, including the directory, file name, and expanded values for %J (job_ID) and %I (index_ID).

If DJOB_ENV_SCRIPT=openmpi_rankfile.sh is set in lsb.applications, LSF creates a host rank file and sets the environment variable LSB_RANK_HOSTFILE.

Default

Not defined.

DJOB_HB_INTERVAL

Syntax

DJOB_HB_INTERVAL=seconds

Description

Value in seconds used to calculate the heartbeat interval between the task RES and job RES of a parallel or distributed job.

This parameter only applies to the blaunch distributed application framework.

When DJOB_HB_INTERVAL is specified, the interval is scaled according to the number of tasks in the job:

max(DJOB_HB_INTERVAL, 10) + host_factor

where

host_factor = 0.01 * number of hosts allocated for the job

Default

Not defined. Interval is the default value of LSB_DJOB_HB_INTERVAL.

DJOB_RESIZE_GRACE_PERIOD

Syntax

DJOB_RESIZE_GRACE_PERIOD = seconds

Description

When a resizable job releases resources, the LSF distributed parallel job framework terminates running tasks if a host has been completely removed. A DJOB_RESIZE_GRACE_PERIOD defines a grace period in seconds for the application to clean up tasks itself before LSF forcibly terminates them.

Default

No grace period.

DJOB_RU_INTERVAL

Syntax

DJOB_RU_INTERVAL=seconds

Description

Value in seconds used to calculate the resource usage update interval for the tasks of a parallel or distributed job.

This parameter only applies to the blaunch distributed application framework.

When DJOB_RU_INTERVAL is specified, the interval is scaled according to the number of tasks in the job:

max(DJOB_RU_INTERVAL, 10) + host_factor

where

host_factor = 0.01 * number of hosts allocated for the job

Default

Not defined. Interval is the default value of LSB_DJOB_RU_INTERVAL.

DJOB_TASK_BIND

Syntax

DJOB_TASK_BIND=Y | y | N | n

Description

For CPU and memory affinity scheduling jobs launched with the blaunch distributed application framework.

To enable LSF to bind each task to the proper CPUs or NUMA nodes you must use blaunch to start tasks. You must set DJOB_TASK_BIND=Y in lsb.applications or LSB_DJOB_TASK_BIND=Y in the submission environment before submitting the job. When set, only the CPU and memory bindings allocated to the task itself will be set in each task's environment.

If DJOB_TASK_BIND=N or LSB_DJOB_TASK_BIND=N, or they are not set, each task will have the same CPU or NUMA node binding on one host.

If you do not use blaunch to start tasks, and use another MPI mechanism such as IBM® Spectrum LSF MPI or IBM Parallel Environment, you should not set DJOB_TASK_BIND or set it to N.

Default

N

DOCKER_IMAGE_AFFINITY

Syntax

DOCKER_IMAGE_AFFINITY=Y | y | N | n

Description

When scheduling Docker-based containerized jobs, setting this parameter to y or Y enables LSF to give preference for execution hosts that already have the requested Docker image. This reduces network bandwidth and the job start time because the execution host does not have to pull the Docker image from the repository and the job can immediately start on the execution host.

When this feature is enabled, LSF considers Docker image location information when scheduling Docker jobs. Docker image affinity interacts with host preference and order[] string requests in the following manner:

  • If host preference is specified, the host preference is honored first. Among hosts with the same preference level, hosts with the requested Docker image are given higher priority.
  • If the order[] string is specified, the hosts with the requested Docker image have a higher priority first. Among hosts that all have the requested Docker image, the order[] string is then honored.

The CONTAINER parameter must be defined for this parameter to work with this application profile.

Default

Not defined.

ELIGIBLE_PEND_TIME_LIMIT

Syntax

ELIGIBLE_PEND_TIME_LIMIT=[hour:]minute

Description

Specifies the eligible pending time limit for a job.

LSF sends the application-level eligible pending time limit configuration to IBM Spectrum LSF RTM, which handles the alarm and triggered actions such as user notification (for example, notifying the user that submitted the job and the LSF administrator) and job control actions (for example, killing the job). IBM Spectrum LSF RTM compares the job's eligible pending time to the eligible pending time limit, and if the job is in an eligible pending state for longer than this specified time limit, IBM Spectrum LSF RTM triggers the alarm and actions. This parameter works without IBM Spectrum LSF RTM, but LSF does not take any other alarm actions.

In the job forwarding model for the LSF multicluster capability, the job's eligible pending time limit is ignored in the execution cluster, while the submission cluster merges the job's queue-, application-, and job-level eligible pending time limit according to local settings.

The eligible pending time limit is in the form of [hour:]minute. The minutes can be specified as a number greater than 59. For example, three and a half hours can either be specified as 3:30, or 210.

The job-level eligible pending time limit (bsub -eptl) overrides the application-level limit specified here, and the application-level limit overrides the queue-level limit (ELIGIBLE_PEND_TIME_LIMIT in lsb.queues).

Default

Not defined.

ENV_VARS

Syntax

ENV_VARS="name='value'[,name1='value1'] [,name2='value2',... ]"

Description

ENV_VARS defines application-specific environment variables that will be used by jobs for the application. Use this parameter to define name/value pairs as environment variables. These environment variables are also used in the pre/post-execution environment.

You can include spaces within the single quotation marks when defining a value. Commas and double quotation marks are reserved by LSF and cannot be used as part of the environment variable name or value. If the same environment variable is named multiple times in ENV_VARS and given different values, the last value in the list will be the one which takes effect. LSF does not allow environment variables to contain other environment variables to be expanded on the execution side. Do not redefine LSF environment variables in ENV_VARS.

To define a NULL environment variable, use single quotes with nothing inside. For example:

ENV_VARS="TEST_CAR=''"

Any variable set in the user’s environment will overwrite the value in ENV_VARS. The application profile value will overwrite the execution host environment value.

After changing the value of this parameter, run badmin reconfig to have the changes take effect. The changes apply to pending jobs only. Running jobs are not affected.

Default

Not defined.

ESTIMATED_RUNTIME

Syntax

ESTIMATED_RUNTIME=[hour:]minute[/host_name | /host_model]

Description

This parameter specifies an estimated run time for jobs associated with an application. LSF uses the ESTIMATED_RUNTIME value for scheduling purposes only, and does not kill jobs that exceed this value unless the jobs also exceed a defined RUNLIMIT. The format of runtime estimate is same as the RUNLIMIT parameter.

The job-level runtime estimate specified by bsub -We overrides the ESTIMATED_RUNTIME setting in an application profile. The ESTIMATED_RUNTIME setting in an application profile overrides the ESTIMATED_RUNTIME setting in the queue and cluster.

The following LSF features use the ESTIMATED_RUNTIME value to schedule jobs:
  • Job chunking
  • Advance reservation
  • SLA
  • Slot reservation
  • Backfill
  • Allocation planner

Default

Not defined

EXEC_DRIVER

Syntax

Podman and Docker jobs: EXEC_DRIVER=context[user(user_name)] starter[/file_path_serverdir/docker-starter.py] controller[/file_path/to/serverdir/docker-control.py] monitor[/file_path/to/serverdir/docker-monitor.py]

Enroot jobs: EXEC_DRIVER=starter[/file_path_serverdir/enroot-starter.py]

Replace file_path/to/serverdir with the actual file path of the LSF_SERVERDIR directory.

Description

Optional for Enroot jobs. Specifies the execution driver framework for Podman, Docker, or Enroot container jobs in this application profile. This parameter uses the following keyword:

user
Optional for Docker jobs and ignored for Enroot jobs. This keyword specifies the user account for starting scripts. The configured value is a user name instead of a user ID. For Docker jobs, this user must be a member of the Docker user group. For Podman jobs, the user name must be set to "default".

By default, this is the LSF primary administrator by default.

Note: This cannot be the root user.

LSF includes three execution driver scripts that are used to start a job (docker-starter.py), monitor the resource of a job (docker-monitor.py), and send a signal to a job (docker-control.py). These scripts are located in the LSF_SERVERDIR directory. Change the owner of the script files to the context user and change the file permissions to 700 or 500 before using them in the EXEC_DRIVER parameter.

The starter script is required. For Docker container jobs, the monitor and control scripts are required if the cgroupfs driver is systemd, but are optional if the cgroupfs driver is cgroupfs. For Podman container jobs, the monitor script is optional while the control script is required. For Enroot container jobs, the starter script is required while all other scripts are ignored.

Interaction with the CONTAINER parameter for Podman, Docker, or Enroot jobs

For Podman, Docker, or Enroot jobs, the EXEC_DRIVER parameter interacts with the following keywords in the CONTAINER parameter:

  • image, which specifies the image name ($LSB_CONTAINER_IMAGE environment variable) is supported when specifying the script names.
  • options with runtime options and the option script is supported.

Example

Begin Application
NAME = podmanapp
CONTAINER = podman[image(repository.example.com:5000/file/path/ubuntu:latest) \
            options(--rm --network=host --ipc=host -v /path/to/my/passwd:/etc/passwd)]
EXEC_DRIVER = context[user(default)] starter[/path/to/driver/docker-starter.py] \
              controller[/path/to/driver/docker-control.py] \
              monitor[/path/to/driver/docker-monitor.py]
DESCRIPTION = Podman User Service
End Application
Begin Application
NAME = dockerapp
CONTAINER = docker[image(repository.example.com:5000/file/path/ubuntu:latest) \
            options(--rm --network=host --ipc=host -v /path/to/my/passwd:/etc/passwd)]
EXEC_DRIVER = context[user(user-name)] starter[/path/to/driver/docker-starter.py] \
              controller[/path/to/driver/docker-control.py] \
              monitor[/path/to/driver/docker-monitor.py]
DESCRIPTION = Docker User Service
End Application
Begin Application
NAME = enrootapp
CONTAINER = enroot[image(repository.example.com:5000/file/path/ubuntu:latest) \
            options(--mount /mydir:/mydir2]
EXEC_DRIVER = starter[/path/to/driver/enroot-starter.py]
DESCRIPTION = Enroot User Service
End Application

Default

Undefined for Podman and Docker jobs.

starter[$LSF_SERVERDIR/enroot-starter.py] for Enroot jobs

FILELIMIT

Syntax

FILELIMIT=integer

Description

The per-process (soft) file size limit (in KB) for all of the processes belonging to a job running in the application profile (see getrlimit(2)). Application-level limits override any default limit specified in the queue, but must be less than the hard limit of the submission queue.

Default

Unlimited

GPU_REQ

Specify GPU requirements together in one statement.

Syntax

GPU_REQ = "[num=num_gpus[/task | host]] [:mode=shared | exclusive_process] [:mps=yes[,shared][,nocvd]| no | per_socket[,shared][,nocvd] | per_gpu[,shared][,nocvd]] [:aff=yes | no] [:j_exclusive=yes | no][:block=yes | no] [:gpack=yes | no] [:gvendor=amd | nvidia] [:gmodel=model_name[#mem_size]] [:gtile=tile_num|'!'] [:gmem=mem_value] [:glink=yes] [:mig=GI_size[/CI_size]]"

Description

The GPU_REQ parameter takes the following arguments:
num=num_gpus[/task | host]
The number of physical GPUs required by the job. By default, the number is per host. You can also specify that the number is per task by specifying /task after the number.

If you specified that the number is per task, the configuration of the ngpus_physical resource in the lsb.resources file is set to PER_TASK, or the RESOURCE_RESERVE_PER_TASK=Y parameter is set in the lsb.params file, this number is the requested count per task.

mode=shared | exclusive_process
The GPU mode when the job is running, either shared or exclusive_process. The default mode is shared.

The shared mode corresponds to the Nvidia or AMD DEFAULT compute mode. The exclusive_process mode corresponds to the Nvidia EXCLUSIVE_PROCESS compute mode.

Note: Do not specify exclusive_process when you are using AMD GPUs (that is, when gvendor=amd is specified).
mps=yes[,nocvd][,shared] | per_socket[,shared][,nocvd] | per_gpu[,shared][,nocvd] | no
Enables or disables the Nvidia Multi-Process Service (MPS) for the GPUs that are allocated to the job. Using MPS effectively causes the EXCLUSIVE_PROCESS mode to behave like the DEFAULT mode for all MPS clients. MPS always allows multiple clients to use the GPU through the MPS server.
Note: To avoid inconsistent behavior, do not enable mps when you are using AMD GPUs (that is, when gvendor=amd is specified). If the result of merging the GPU requirements at the cluster, queue, application, and job levels is gvendor=amd and mps is enabled (for example, if gvendor=amd is specified at the job level without specifying mps=no, but mps=yes is specified at the application, queue, or cluster level), LSF ignores the mps requirement.

MPS is useful for both shared and exclusive process GPUs, and allows more efficient sharing of GPU resources and better GPU utilization. See the Nvidia documentation for more information and limitations.

When using MPS, use the EXCLUSIVE_PROCESS mode to ensure that only a single MPS server is using the GPU, which provides additional insurance that the MPS server is the single point of arbitration between all CUDA process for that GPU.

You can also enable MPS daemon sharing by adding the share keyword with a comma and no space (for example, mps=yes,shared enables MPS daemon sharing on the host). If sharing is enabled, all jobs that are submitted by the same user with the same resource requirements share the same MPS daemon on the host, socket, or GPU.

LSF starts MPS daemons on a per-host, per-socket, or per-GPU basis depending on value of the mps keyword:

  • If mps=yes is set, LSF starts one MPS daemon per host per job.

    When share is enabled (that is, if mps=yes,shared is set), LSF starts one MPS daemon per host for all jobs that are submitted by the same user with the same resource requirements. These jobs all use the same MPS daemon on the host.

    When the CUDA_VISIBLE_DEVICES environment variable is disabled (that is, if mps=yes,nocvd is set), LSF does not set the CUDA_VISIBLE_DEVICES<number> environment variables for tasks, so LSF MPI does not set CUDA_VISIBLE_DEVICES for the tasks. LSF just sets the CUDA_VISIBLE_DEVICES<number> environment variables for tasks, not CUDA_VISIBLE_DEVICES. LSF MPI converts the CUDA_VISIBLE_DEVICES<number> environment variables into CUDA_VISIBLE_DEVICES and sets that for the tasks.

  • If mps=per_socket is set, LSF starts one MPS daemon per socket per job. When enabled with share (that is, if mps=per_socket,shared is set), LSF starts one MPS daemon per socket for all jobs that are submitted by the same user with the same resource requirements. These jobs all use the same MPS daemon for the socket.
  • If mps=per_gpu is set, LSF starts one MPS daemon per GPU per job. When enabled with share (that is, if mps=per_gpu,shared is set), LSF starts one MPS daemon per GPU for all jobs that are submitted by the same user with the same resource requirements. These jobs all use the same MPS daemon for the GPU.
Important: Using EXCLUSIVE_THREAD mode with MPS is not supported and might cause unexpected behavior.
j_exclusive=yes | no
Specifies whether the allocated GPUs can be used by other jobs. When the mode is set to exclusive_process, the j_exclusive=yes option is set automatically.
aff=yes | no
Specifies whether to enforce strict GPU-CPU affinity binding. If set to no, LSF relaxes GPU affinity while maintaining CPU affinity. By default, aff=yes is set to maintain strict GPU-CPU affinity binding.
Note: The aff=yes setting conflicts with block=yes (distribute allocated GPUs as blocks when the number of tasks is greater than the requested number of GPUs). This is because strict CPU-GPU binding allocates GPUs to tasks based on the CPU NUMA ID, which conflicts with the distribution of allocated GPUs as blocks. If aff=yes and block=yes are both specified in the GPU requirements string, the block=yes setting takes precedence and strict CPU-GPU affinity binding is disabled (that is, aff=no is automatically set).
block=yes | no
Specifies whether to enable block distribution, that is, to distribute the allocated GPUs of a job as blocks when the number of tasks is greater than the requested number of GPUs. If set to yes, LSF distributes all the allocated GPUs of a job as blocks when the number of tasks is bigger than the requested number of GPUs. By default, block=no is set so that allocated GPUs are not distributed as blocks.

For example, if a GPU job requests to run on a host with 4 GPUs and 40 tasks, block distribution assigns GPU0 for ranks 0-9, GPU1 for ranks 10-19, GPU2 for tanks 20-29, and GPU3 for ranks 30-39.

Note: The block=yes setting conflicts with aff=yes (strict CPU-GPU affinity binding). This is because strict CPU-GPU binding allocates GPUs to tasks based on the CPU NUMA ID, which conflicts with the distribution of allocated GPUs as blocks. If block=yes and aff=yes are both specified in the GPU requirements string, the block=yes setting takes precedence and strict CPU-GPU affinity binding is disabled (that is, aff=no is automatically set).
gpack=yes | no
For shared mode jobs only. Specifies whether to enable pack scheduling. If set to yes, LSF packs multiple shared mode GPU jobs to allocated GPUs. LSF schedules shared mode GPUs as follows:
  1. LSF sorts the candidate hosts (from largest to smallest) based on the number of shared GPUs that already have running jobs, then by the number of GPUs that are not exclusive.

    If the order[] keyword is defined in the resource requirements string, after sorting order[], LSF re-sorts the candidate hosts by the gpack policy (by shared GPUs that already have running jobs first, then by the number of GPUs that are not exclusive). The gpack policy sort priority is higher than the order[] sort.

  2. LSF sorts the candidate GPUs on each host (from largest to smallest) based on the number of running jobs.

After scheduling, the shared mode GPU job packs to the allocated shared GPU that is sorted first, not to a new shared GPU.

If Docker attribute affinity is enabled, the order of candidate hosts are sorted by Docker attribute affinity before sorting by GPUs.

By default, gpack=no is set so that pack scheduling is disabled.

gvendor=amd | nvidia
Specifies the GPU vendor type. LSF allocates GPUs with the specified vendor type.

Specify amd to request AMD GPUs, or specify nvidia to request Nvidia GPUs.

By default, LSF requests Nvidia GPUs.

gmodel=model_name[-mem_size]
Specifies GPUs with the specific model name and, optionally, its total GPU memory. By default, LSF allocates the GPUs with the same model, if available.

The gmodel keyword supports the following formats:

gmodel=model_name
Requests GPUs with the specified brand and model name (for example, TeslaK80).
gmodel=short_model_name
Requests GPUs with a specific brand name (for example, Tesla, Quadro, NVS, ) or model type name (for example, K80, P100).
gmodel=model_name-mem_size
Requests GPUs with the specified brand name and total GPU memory size. The GPU memory size consists of the number and its unit, which includes M, G, T, MB, GB, and TB (for example, 12G).

To find the available GPU model names on each host, run the lsload –gpuload, lshosts –gpu, or bhosts -gpu commands. The model name string does not contain space characters. In addition, the slash (/) and hyphen (-) characters are replaced with the underscore character (_). For example, the GPU model name “Tesla C2050 / C2070” is converted to “TeslaC2050_C2070” in LSF.

gmem=mem_value

Specify the GPU memory on each GPU required by the job. The format of mem_value is the same to other resource value (for example, mem or swap) in the rusage section of the job resource requirements (-R).

gtile=! | tile_num
Specifies the number of GPUs per socket. Specify an number to explicitly define the number of GPUs per socket on the host, or specify an exclamation mark (!) to enable LSF to automatically calculate the number, which evenly divides the GPUs along all sockets on the host. LSF guarantees the gtile requirements even for affinity jobs. This means that LSF might not allocate the GPU's affinity to the allocated CPUs when the gtile requirements cannot be satisfied.

If the gtile keyword is not specified for an affinity job, LSF attempts to allocate enough GPUs on the sockets that allocated GPUs. If there are not enough GPUs on the optimal sockets, jobs cannot go to this host.

If the gtile keyword is not specified for a non-affinity job, LSF attempts to allocate enough GPUs on the same socket. If this is not available, LSF might allocate GPUs on separate GPUs.

nvlink=yes
Obsolete in LSF, Version 10.1 Fix Pack 11. Use the glink keyword instead. Enables the job enforcement for NVLink connections among GPUs. LSF allocates GPUs with NVLink connections in force.
glink=yes
Enables job enforcement for special connections among GPUs. LSF must allocate GPUs with the special connections that are specific to the GPU vendor.

If the job requests AMD GPUs, LSF must allocate GPUs with the xGMI connection. If the job requests Nvidia GPUs, LSF must allocate GPUs with the NVLink connection.

Do not use glink together with the obsolete nvlink keyword.

By default, LSF can allocate GPUs without special connections when there are not enough GPUs with these connections.

mig=GI_size[/CI_size]
Specifies Nvidia Multi-Instance GPU (MIG) device requirements.

Specify the requested number of GPU instances for the MIG job. Valid GPU instance sizes are 1, 2, 3, 4, 7.

Optionally, specify the requested number of compute instances after the specified GPU instance size and a slash character (/). The requested compute instance size must be less than or equal to the requested GPU instance size. In addition, Nvidia MIG does not support the following GPU/compute instance size combinations: 4/3, 7/5, 7/6. If this is not specified, the default compute instance size is 1.

The syntax of the GPU requirement in the -gpu option is the same as the syntax in the LSB_GPU_REQ parameter in the lsf.conf file and the GPU_REQ parameter in the lsb.queues and lsb.applications files.
Note: The bjobs output does not show aff=yes even if you specify aff=yes in the bsub -gpu option.

If the GPU_REQ_MERGE parameter is defined as Y or y in the lsb.params file and a GPU requirement is specified at multiple levels (at least two of the default cluster, queue, application profile, or job level requirements), each option of the GPU requirement is merged separately. Job level overrides application level, which overrides queue level, which overrides the default cluster GPU requirement. For example, if the mode option of the GPU requirement is defined on the -gpu option, and the mps option is defined in the queue, the mode of job level and the mps value of queue is used.

If the GPU_REQ_MERGE parameter is not defined as Y or y in the lsb.params file and a GPU requirement is specified at multiple levels (at least two of the default cluster, queue, application profile, or job level requirements), the entire GPU requirement string is replaced. The entire job level GPU requirement string overrides application level, which overrides queue level, which overrides the default GPU requirement.

The esub parameter LSB_SUB4_GPU_REQ modifies the value of the -gpu option.

LSF selects the GPU that meets the topology requirement first. If the GPU mode of the selected GPU is not the requested mode, LSF changes the GPU to the requested mode. For example, if LSF allocates an exclusive_process GPU to a job that needs a shared GPU, LSF changes the GPU mode to shared before the job starts and then changes the mode back to exclusive_process when the job finishes.

The GPU requirements are converted to rusage resource requirements for the job. For example, num=2 is converted to rusage[ngpus_physical=2]. Use the bjobs, bhist, and bacct commands to see the merged resource requirement.

There might be complex GPU requirements that the bsub -gpu option and GPU_REQ parameter syntax cannot cover, including compound GPU requirements (for different GPU requirements for jobs on different hosts, or for different parts of a parallel job) and alternate GPU requirements (if more than one set of GPU requirements might be acceptable for a job to run). For complex GPU requirements, use the bsub -R command option, or the RES_REQ parameter in the lsb.applications or lsb.queues file to define the resource requirement string.

Important: You can define the mode, j_exclusive, and mps options only with the -gpu option, the LSB_GPU_REQ parameter in the lsf.conf file, or the GPU_REQ parameter in the lsb.queues or lsb.applications files. You cannot use these options with the rusage resource requirement string in the bsub -R command option or the RES_REQ parameter in the lsb.queues or lsb.applications files.

Default

Not defined

See also

  • LSB_GPU_REQ
  • bsub -gpu

HOST_POST_EXEC

Syntax

HOST_POST_EXEC=command

Description

Enables host-based post-execution processing at the application level. The HOST_POST_EXEC command runs on all execution hosts after the job finishes. If job based post-execution POST_EXEC was defined at the queue-level/application-level/job-level, the HOST_POST_EXEC command runs after POST_EXEC of any level.

Host-based post-execution commands can be configured at the queue and application level, and run in the following order:
  1. The application-level command
  2. The queue-level command.

The supported command rule is the same as the existing POST_EXEC for the queue section. See the POST_EXEC topic for details.

Note:

The host-based post-execution command cannot be executed on Windows platforms. This parameter cannot be used to configure job-based post-execution processing.

Default

Not defined.

HOST_PRE_EXEC

Syntax

HOST_PRE_EXEC=command

Description

Enables host-based pre-execution processing at the application level. The HOST_PRE_EXEC command runs on all execution hosts before the job starts. If job based pre-execution PRE_EXEC was defined at the queue-level/application-level/job-level, the HOST_PRE_EXEC command runs before PRE_EXEC of any level.

Host-based pre-execution commands can be configured at the queue and application level, and run in the following order:
  1. The queue-level command
  2. The application-level command.

The supported command rule is the same as the existing PRE_EXEC for the queue section. See the PRE_EXEC topic for details.

Note:

The host-based pre-execution command cannot be executed on Windows platforms. This parameter cannot be used to configure job-based pre-execution processing.

Default

Not defined.

JOB_CWD

Syntax

JOB_CWD=directory

Description

Current working directory (CWD) for the job in the application profile. The path can be absolute or relative to the submission directory. The path can include the following dynamic patterns (which are case sensitive):

  • %J - job ID
  • %JG - job group (if not specified, it will be ignored)
  • %I - job index (default value is 0)
  • %EJ - execution job ID
  • %EI - execution job index
  • %P - project name
  • %U - user name
  • %G - user group
  • %H - first execution host name

Unsupported patterns are treated as text.

If this parameter is changed, then any newly submitted jobs with the -app option will use the new value for CWD if bsub -cwd is not defined.

JOB_CWD supports all LSF path conventions such as UNIX, UNC and Windows formats. In the mixed UNIX /Windows cluster it can be specified with one value for UNIX and another value for Windows separated by a pipe character (|).

JOB_CWD=unix_path|windows_path

The first part of the path must be for UNIX and the second part must be for Windows. Both paths must be full paths.

Default

Not defined.

JOB_CWD_TTL

Syntax

JOB_CWD_TTL=hours

Description

Specifies the time-to-live (TTL) for the current working directory (CWD) of a job. LSF cleans created CWD directories after a job finishes based on the TTL value. LSF deletes the CWD for the job if LSF created that directory for the job. The following options are available:

  • 0 - sbatchd deletes CWD when all process related to the job finish.
  • 2147483647 - Never delete the CWD for a job.
  • 1 to 2147483646 - Delete the CWD for a job after the timeout expires.

The system checks the directory list every 5 minutes with regards to cleaning and deletes only the last directory of the path to avoid conflicts when multiple jobs share some parent directories. TTL will be calculated after the post-exec script finishes. When LSF (sbatchd) starts, it checks the directory list file and deletes expired CWDs.

If the value for this parameter is not set in the application profile, LSF checks to see if it is set at the cluster-wide level. If neither is set, the default value is used.

Default

Not defined. The value of 2147483647 is used, meaning the CWD is not deleted.

JOB_INCLUDE_POSTPROC

Syntax

JOB_INCLUDE_POSTPROC=Y | N

Description

Specifies whether LSF includes the post-execution processing of the job as part of the job. When set to Y:
  • Prevents a new job from starting on a host until post-execution processing is finished on that host
  • Includes the CPU and run times of post-execution processing with the job CPU and run times
  • sbatchd sends both job finish status (DONE or EXIT) and post-execution processing status (POST_DONE or POST_ERR) to mbatchd at the same time

The variable LSB_JOB_INCLUDE_POSTPROC in the user environment overrides the value of JOB_INCLUDE_POSTPROC in an application profile in lsb.applications. JOB_INCLUDE_POSTPROC in an application profile in lsb.applications overrides the value of JOB_INCLUDE_POSTPROC in lsb.params.

For CPU and memory affinity jobs, if JOB_INCLUDE_POSTPROC=Y, LSF does not release affinity resources until post-execution processing has finished, since slots are still occupied by the job during post-execution processing.

For SGI cpusets, if JOB_INCLUDE_POSTPROC=Y, LSF does not release the cpuset until post-execution processing has finished, even though post-execution processes are not attached to the cpuset.

Default

N. Post-execution processing is not included as part of the job, and a new job can start on the execution host before post-execution processing finishes.

JOB_POSTPROC_TIMEOUT

Syntax

JOB_POSTPROC_TIMEOUT=minutes

Description

Specifies a timeout in minutes for job post-execution processing. The specified timeout must be greater than zero

If post-execution processing takes longer than the timeout, sbatchd reports that post-execution has failed (POST_ERR status). On UNIX and Linux, it kills the entire process group of the job's pre-execution processes. On Windows, only the parent process of the pre-execution command is killed when the timeout expires, the child processes of the pre-execution command are not killed.

If JOB_INCLUDE_POSTPROC=Y, and sbatchd kills the post-execution processes because the timeout has been reached, the CPU time of the post-execution processing is set to 0, and the job’s CPU time does not include the CPU time of post-execution processing.

JOB_POSTPROC_TIMEOUT defined in an application profile in lsb.applications overrides the value in lsb.params. JOB_POSTPROC_TIMEOUT cannot be defined in user environment.

When running host-based post execution processing, set JOB_POSTPROC_TIMEOUT to a value that gives the process enough time to run.

Default

Not defined. Post-execution processing does not time out.

JOB_PREPROC_TIMEOUT

Syntax

JOB_PREPROC_TIMEOUT=minutes

Description

Specify a timeout in minutes for job pre-execution processing. The specified timeout must be an integer greater than zero. If the job's pre-execution processing takes longer than the timeout, LSF kills the job's pre-execution processes, kills the job with a pre-defined exit value of 98, and then requeues the job to the head of the queue. However, if the number of pre-execution retries has reached the pre-execution retry threshold, LSF suspends the job with PSUSP status instead of requeuing it.

JOB_PREPROC_TIMEOUT defined in an application profile in lsb.applications overrides the value in lsb.params. JOB_PREPROC_TIMEOUT cannot be defined in the user environment.

On UNIX and Linux, sbatchd kills the entire process group of the job's pre-execution processes.

On Windows, only the parent process of the pre-execution command is killed when the timeout expires, the child processes of the pre-execution command are not killed.

Default

Not defined. Pre-execution processing does not time out. However, when running host-based pre-execution processing, you cannot use the infinite value or it will fail. You must configure a reasonable value.

JOB_SIZE_LIST

Syntax

JOB_SIZE_LIST=default_size [size ...]

Description

A list of job sizes (number of tasks) that are allowed on this application.

When submitting a job or modifying a pending job that requests a job size by using the -n or -R options for bsub and bmod, the requested job size must be a single fixed value that matches one of the values that JOB_SIZE_LIST specifies, which are the job sizes that are allowed on this application profile. LSF rejects the job if the requested job size is not in this list. In addition, when using bswitch to switch a pending job with a requested job size to another queue, the requested job size in the pending job must also match one of the values in JOB_SIZE_LIST for the new queue.

The first value in this list is the default job size, which is the assigned job size request if the job was submitted without requesting one. The remaining values are the other job sizes allowed in the queue, and may be defined in any order.

When defined in both a queue (lsb.queues) and an application profile, the job size request must satisfy both requirements. In addition, JOB_SIZE_LIST overrides any TASKLIMIT parameters defined at the same level. Job size requirements do not apply to queues and application profiles with no job size lists, nor do they apply to other levels of job submissions (that is, host level or cluster level job submissions).

Note: An exclusive job may allocate more slots on the host then is required by the tasks. For example, if JOB_SIZE_LIST=8 and an exclusive job requesting -n8 runs on a 16 slot host, all 16 slots are assigned to the job. The job runs as expected, since the 8 tasks specified for the job matches the job size list.

Valid values

A space-separated list of positive integers between 1 and 2147483646.

Default

Undefined

JOB_STARTER

Syntax

JOB_STARTER=starter [starter] ["%USRCMD"] [starter]

Description

Creates a specific environment for submitted jobs prior to execution. An application-level job starter overrides a queue-level job starter.

starter is any executable that can be used to start the job (i.e., can accept the job as an input argument). Optionally, additional strings can be specified.

By default, the user commands run after the job starter. A special string, %USRCMD, can be used to represent the position of the user’s job in the job starter command line. The %USRCMD string and any additional commands must be enclosed in quotation marks (" ").

Example

JOB_STARTER=csh -c "%USRCMD;sleep 10"
In this case, if a user submits a job
bsub myjob arguments
the command that actually runs is:
csh -c "myjob arguments;sleep 10"

Default

Not defined. No job starter is used,

LOCAL_MAX_PREEXEC_RETRY

Syntax

LOCAL_MAX_PREEXEC_RETRY=integer

Description

The maximum number of times to attempt the pre-execution command of a job on the local cluster.

When this limit is reached, the default behavior of the job is defined by the LOCAL_MAX_PREEXEC_RETRY_ACTION parameter in lsb.params, lsb.queues, or lsb.applications.

Valid values

0 < MAX_PREEXEC_RETRY < INFINIT_INT

INFINIT_INT is defined in lsf.h.

Default

Not defined. The number of preexec retry times is unlimited

See also

LOCAL_MAX_PREEXEC_RETRY_ACTION in lsb.params, lsb.queues, and lsb.applications.

LOCAL_MAX_PREEXEC_RETRY_ACTION

Syntax

LOCAL_MAX_PREEXEC_RETRY_ACTION=SUSPEND | EXIT

Description

The default behavior of a job when it reaches the maximum number of times to attempt its pre-execution command on the local cluster (LOCAL_MAX_PREEXEC_RETRY in lsb.params, lsb.queues, or lsb.applications).

  • If set to SUSPEND, the job is suspended and its status is set to PSUSP.
  • If set to EXIT, the job exits and its status is set to EXIT. The job exits with the same exit code as the last pre-execution fail exit code.

This parameter is configured cluster-wide (lsb.params), at the queue level (lsb.queues), and at the application level (lsb.applications). The action specified in lsb.applications overrides lsb.queues, and lsb.queues overrides the lsb.params configuration.

Default

Not defined. If not defined in lsb.queues or lsb.params, the default action is SUSPEND.

See also

LOCAL_MAX_PREEXEC_RETRY in lsb.params, lsb.queues, and lsb.applications.

MAX_JOB_PREEMPT

Syntax

MAX_JOB_PREEMPT=integer

Description

The maximum number of times a job can be preempted. Applies to queue-based preemption only.

Valid values

0 < MAX_JOB_PREEMPT < INFINIT_INT

INFINIT_INT is defined in lsf.h.

Default

Not defined. The number of preemption times is unlimited.

MAX_JOB_REQUEUE

Syntax

MAX_JOB_REQUEUE=integer

Description

The maximum number of times to requeue a job automatically.

Valid values

0 < MAX_JOB_REQUEUE < INFINIT_INT

INFINIT_INT is defined in lsf.h.

Default

Not defined. The number of requeue times is unlimited

MAX_PREEXEC_RETRY

Syntax

MAX_PREEXEC_RETRY=integer

Description

Use REMOTE_MAX_PREEXEC_RETRY instead. This parameter is only maintained for backwards compatibility.

Job forwarding model for the LSF multicluster capability only. The maximum number of times to attempt the pre-execution command of a job from a remote cluster.

If the job's pre-execution command fails all attempts, the job is returned to the submission cluster.

Valid values

0 < MAX_PREEXEC_RETRY < INFINIT_INT

INFINIT_INT is defined in lsf.h.

Default

5

MAX_TOTAL_TIME_PREEMPT

Syntax

MAX_TOTAL_TIME_PREEMPT=integer

Description

The accumulated preemption time in minutes after which a job cannot be preempted again, where minutes is wall-clock time, not normalized time.

Setting this parameter in lsb.applications overrides the parameter of the same name in lsb.queues and in lsb.params.

Valid values

Any positive integer greater than or equal to one (1)

Default

Unlimited

MEMLIMIT

Syntax

MEMLIMIT=integer

Description

The per-process (soft) process resident set size limit for all of the processes belonging to a job running in the application profile.

Sets the maximum amount of physical memory (resident set size, RSS) that may be allocated to a process.

By default, the limit is specified in KB. Use LSF_UNIT_FOR_LIMITS in lsf.conf to specify a larger unit for the limit (MB, GB, TB, PB, or EB).

By default, jobs submitted to the application profile without a job-level memory limit are killed when the memory limit is reached. Application-level limits override any default limit specified in the queue, but must be less than the hard limit of the submission queue.

LSF has two methods of enforcing memory usage:
  • OS Memory Limit Enforcement
  • LSF Memory Limit Enforcement

OS memory limit enforcement

OS memory limit enforcement is the default MEMLIMIT behavior and does not require further configuration. OS enforcement usually allows the process to eventually run to completion. LSF passes MEMLIMIT to the OS, which uses it as a guide for the system scheduler and memory allocator. The system may allocate more memory to a process if there is a surplus. When memory is low, the system takes memory from and lowers the scheduling priority (re-nice) of a process that has exceeded its declared MEMLIMIT. Only available on systems that support RLIMIT_RSS for setrlimit().

Not supported on:
  • Sun Solaris 2.x
  • Windows

LSF memory limit enforcement

To enable LSF memory limit enforcement, set LSB_MEMLIMIT_ENFORCE in lsf.conf to y. LSF memory limit enforcement explicitly sends a signal to kill a running process once it has allocated memory past MEMLIMIT.

You can also enable LSF memory limit enforcement by setting LSB_JOB_MEMLIMIT in lsf.conf to y. The difference between LSB_JOB_MEMLIMIT set to y and LSB_MEMLIMIT_ENFORCE set to y is that with LSB_JOB_MEMLIMIT, only the per-job memory limit enforced by LSF is enabled. The per-process memory limit enforced by the OS is disabled. With LSB_MEMLIMIT_ENFORCE set to y, both the per-job memory limit enforced by LSF and the per-process memory limit enforced by the OS are enabled.

Available for all systems on which LSF collects total memory usage.

Default

Unlimited

MEMLIMIT_TYPE

Syntax

MEMLIMIT_TYPE=JOB [PROCESS] [TASK]

MEMLIMIT_TYPE=PROCESS [JOB] [TASK]

MEMLIMIT_TYPE=TASK [PROCESS] [JOB]

Description

A memory limit is the maximum amount of memory a job is allowed to consume. Jobs that exceed the level are killed. You can specify different types of memory limits to enforce. Use any combination of JOB, PROCESS, and TASK.

By specifying a value in the application profile, you overwrite these three parameters: LSB_JOB_MEMLIMIT, LSB_MEMLIMIT_ENFORCE, LSF_HPC_EXTENSIONS (TASK_MEMLIMIT).

Note: A task list is a list in LSF that keeps track of the default resource requirements for different applications and task eligibility for remote execution.
  • PROCESS: Applies a memory limit by OS process, which is enforced by the OS on the server host (where the job is running). When the memory allocated to one process of the job exceeds the memory limit, LSF kills the job.
  • TASK: Applies a memory limit based on the task list file. It is enforced by LSF. LSF terminates the entire parallel job if any single task exceeds the limit setting for memory and swap limits.
  • JOB: Applies a memory limit identified in a job and enforced by LSF. When the sum of the memory allocated to all processes of the job exceeds the memory limit, LSF kills the job.
  • PROCESS TASK: Enables both process-level memory limit enforced by OS and task-level memory limit enforced by LSF.
  • PROCESS JOB: Enables both process-level memory limit enforced by OS and job-level memory limit enforced by LSF.
  • TASK JOB: Enables both task-level memory limit enforced by LSF and job-level memory limit enforced by LSF.
  • PROCESS TASK JOB: Enables process-level memory limit enforced by OS, task-level memory limit enforced by LSF, and job-level memory limit enforced by LSF.

Default

Not defined. The memory limit-level is still controlled by LSF_HPC_EXTENSIONS=TASK_MEMLIMIT, LSB_JOB_MEMLIMIT, LSB_MEMLIMIT_ENFORCE

MIG

Syntax

MIG=minutes

Description

Enables automatic job migration and specifies the migration threshold for checkpointable or rerunnable jobs, in minutes.

LSF automatically migrates jobs that have been in the SSUSP state for more than the specified number of minutes. A value of 0 specifies that a suspended job is migrated immediately. The migration threshold applies to all jobs running on the host.

Job-level command line migration threshold overrides threshold configuration in application profile and queue. Application profile configuration overrides queue level configuration.

When a host migration threshold is specified, and is lower than the value for the job, the queue, or the application, the host value is used.

Members of a chunk job can be migrated. Chunk jobs in WAIT state are removed from the job chunk and put into PEND state.

Does not affect jobs with the that are forwarded to a remote cluster.

Default

Not defined. LSF does not migrate checkpointable or rerunnable jobs automatically.

NAME

Syntax

NAME=string

Description

Required. Unique name for the application profile.

Specify any ASCII string up to 60 characters long. You can use letters, digits, underscores (_), dashes (-), periods (.) or spaces in the name. The application profile name must be unique within the cluster.

Note: If you want to specify the ApplicationVersion in a JSDL file, include the version when you define the application profile name. Separate the name and version by a space, as shown in the following example:
NAME=myapp 1.0

Default

You must specify this parameter to define an application profile. LSF does not automatically assign a default application profile name.

NETWORK_REQ

Syntax

NETWORK_REQ="network_res_req"

network_res_req has the following syntax:

[type=sn_all | sn_single] [:protocol=protocol_name[(protocol_number)][,protocol_name[(protocol_number)]] [:mode=US | IP] [:usage=dedicated | shared] [:instance=positive_integer]

Description

Note: This parameter is deprecated and might be removed in a future version of LSF.

For LSF IBM Parallel Environment (PE) integration. Specifies the network resource requirements for a PE job.

If any network resource requirement is specified in the job, queue, or application profile, the job is treated as a PE job. PE jobs can only run on hosts where IBM PE pnsd daemon is running.

The network resource requirement string network_res_req has the same syntax as the bsub -network option.

The -network bsub option overrides the value of NETWORK_REQ defined in lsb.queues or lsb.applications. The value of NETWORK_REQ defined in lsb.applications overrides queue-level NETWORK_REQ defined in lsb.queues.

The following IBM LoadLeveller job command file options are not supported in LSF:
  • collective_groups
  • imm_send_buffers
  • rcxtblocks
The following network resource requirement options are supported:
type=sn_all | sn_single
Specifies the adapter device type to use for message passing: either sn_all or sn_single.
sn_single

When used for switch adapters, specifies that all windows are on a single network

sn_all

Specifies that one or more windows are on each network, and that striped communication should be used over all available switch networks. The networks specified must be accessible by all hosts selected to run the PE job. See the Parallel Environment Runtime Edition for AIX: Operation and Use guide (SC23-6781-05) for more information about submitting jobs that use striping.

If mode is IP and type is specified as sn_all or sn_single, the job will only run on InfiniBand (IB) adapters (IPoIB). If mode is IP and type is not specified, the job will only run on Ethernet adapters (IPoEth). For IPoEth jobs, LSF ensures the job is running on hosts where pnsd is installed and running. For IPoIB jobs, LSF ensures the job is running on hosts where pnsd is installed and running, and that IB networks are up. Because IP jobs do not consume network windows, LSF does not check if all network windows are used up or if the network is already occupied by a dedicated PE job.

Equivalent to the PE MP_EUIDEVICE environment variable and -euidevice PE flag See the Parallel Environment Runtime Edition for AIX: Operation and Use guide (SC23-6781-05) for more information. Only sn_all or sn_single are supported by LSF. The other types supported by PE are not supported for LSF jobs.

protocol=protocol_name[(protocol_number)]
Network communication protocol for the PE job, indicating which message passing API is being used by the application. The following protocols are supported by LSF:
mpi

The application makes only MPI calls. This value applies to any MPI job regardless of the library that it was compiled with (PE MPI, MPICH2).

pami

The application makes only PAMI calls.

lapi

The application makes only LAPI calls.

shmem

The application makes only OpenSHMEM calls.

user_defined_parallel_api

The application makes only calls from a parallel API that you define. For example: protocol=myAPI or protocol=charm.

The default value is mpi.

LSF also supports an optional protocol_number (for example, mpi(2), which specifies the number of contexts (endpoints) per parallel API instance. The number must be a power of 2, but no greater than 128 (1, 2, 4, 8, 16, 32, 64, 128). LSF will pass the communication protocols to PE without any change. LSF will reserve network windows for each protocol.

When you specify multiple parallel API protocols, you cannot make calls to both LAPI and PAMI (lapi, pami) or LAPI and OpenSHMEM (lapi, shmem) in the same application. Protocols can be specified in any order.

See the MP_MSG_API and MP_ENDPOINTS environment variables and the -msg_api and -endpoints PE flags in the Parallel Environment Runtime Edition for AIX: Operation and Use guide (SC23-6781-05) for more information about the communication protocols that are supported by IBM PE.

mode=US | IP

The network communication system mode used by the communication specified communication protocol: US (User Space) or IP (Internet Protocol). A US job can only run with adapters that support user space communications, such as the IB adapter. IP jobs can run with either Ethernet adapters or IB adapters. When IP mode is specified, the instance number cannot be specified, and network usage must be unspecified or shared.

Each instance on the US mode requested by a task running on switch adapters requires an adapter window. For example, if a task requests both the MPI and LAPI protocols such that both protocol instances require US mode, two adapter windows will be used.

The default value is US.

usage=dedicated | shared

Specifies whether the adapter can be shared with tasks of other job steps: dedicated or shared. Multiple tasks of the same job can share one network even if usage is dedicated.

The default usage is shared.

instances=positive_integer

The number of parallel communication paths (windows) per task made available to the protocol on each network. The number actually used depends on the implementation of the protocol subsystem.

The default value is 1.

If the specified value is greater than MAX_PROTOCOL_INSTANCES in lsb.params or lsb.queues, LSF rejects the job.

LSF_PE_NETWORK_NUM must be defined to a non-zero value in lsf.conf for NETWORK_REQ to take effect. If LSF_PE_NETWORK_NUM is not defined or is set to 0, NETWORK_REQ is ignored with a warning message.

Example

The following network resource requirement string specifies that the requirements for an sn_all job (one or more windows are on each network, and striped communication should be used over all available switch networks). The PE job uses MPI API calls (protocol), runs in user-space network communication system mode, and requires 1 parallel communication path (window) per task.

NETWORK_REQ = "protocol=mpi:mode=us:instance=1:type=sn_all"

Default

No default value, but if you specify no value (NETWORK_REQ=""), the job uses the following: protocol=mpi:mode=US:usage=shared:instance=1 in the application profile.

NICE

Syntax

NICE=integer

Description

Adjusts the UNIX scheduling priority at which jobs from the application execute.

A value of 0 (zero) maintains the default scheduling priority for UNIX interactive jobs. This value adjusts the run-time priorities for batch jobs to control their effect on other batch or interactive jobs. See the nice(1) manual page for more details.

On Windows, this value is mapped to Windows process priority classes as follows:
  • nice>=0 corresponds to a priority class of IDLE
  • nice<0 corresponds to a priority class of NORMAL

LSF on Windows does not support HIGH or REAL-TIME priority classes.

When set, this value overrides NICE set at the queue level in lsb.queues.

Default

Not defined.

NO_PREEMPT_INTERVAL

Specifies the number of minutes a preemptable job can run before it is preempted. If the uninterrupted run time of a preemptable job is longer than the specified time, it can be preempted.

Syntax

NO_PREEMPT_INTERVAL=minutes

The value of minutes is wall-clock time, not normalized time.

Description

The NO_PREEMPT_INTERVAL=0 parameter allows immediate preemption of jobs as soon as they start or resume running.

For example, if a job A needs to preempt other candidate preemptable jobs (B, C, and D), the NO_PREEMPT_INTERVAL parameter determines which job is preempted:
  • Run time of job B and job C is less than the NO_PREEMPT_INTERVAL parameter: job B and C are not preempted.
  • Run time of job D is greater than or equal to the NO_PREEMPT_INTERVAL parameter: job D is preempted.

Setting this parameter in lsb.applications overrides the parameter of the same name in the lsb.queues and in lsb.params files.

Default

0

NO_PREEMPT_FINISH_TIME

Syntax

NO_PREEMPT_FINISH_TIME=minutes | percentage

Description

Prevents preemption of jobs that will finish within the specified number of minutes or the specified percentage of the estimated run time or run limit.

Specifies that jobs due to finish within the specified number of minutes or percentage of job duration should not be preempted, where minutes is wall-clock time, not normalized time. Percentage must be greater than 0 or less than 100% (between 1% and 99%).

For example, if the job run limit is 60 minutes and NO_PREEMPT_FINISH_TIME=10%, the job cannot be preempted after it runs 54 minutes or longer.

If you specify percentage for NO_PREEMPT_FINISH_TIME, requires a run time (bsub -We or ESTIMATED_RUNTIME in lsb.applications), or run limit to be specified for the job (bsub -W, or RUNLIMIT in lsb.queues, or RUNLIMIT in lsb.applications)

NO_PREEMPT_RUN_TIME

Syntax

NO_PREEMPT_RUN_TIME=minutes | percentage

Description

Prevents preemption of jobs that have been running for the specified number of minutes or the specified percentage of the estimated run time or run limit.

Specifies that jobs that have been running for the specified number of minutes or longer should not be preempted, where minutes is wall-clock time, not normalized time. Percentage must be greater than 0 or less than 100% (between 1% and 99%).

For example, if the job run limit is 60 minutes and NO_PREEMPT_RUN_TIME=50%, the job cannot be preempted after it running 30 minutes or longer.

If you specify percentage for NO_PREEMPT_RUN_TIME, requires a run time (bsub -We or ESTIMATED_RUNTIME in lsb.applications), or run limit to be specified for the job (bsub -W, or RUNLIMIT in lsb.queues, or RUNLIMIT in lsb.applications)

PEND_TIME_LIMIT

Syntax

PEND_TIME_LIMIT=[hour:]minute

Description

Specifies the pending time limit for a job.

LSF sends the application-level pending time limit configuration to IBM Spectrum LSF RTM, which handles the alarm and triggered actions such as user notification (for example, notifying the user that submitted the job and the LSF administrator) and job control actions (for example, killing the job). IBM Spectrum LSF RTM compares the job's pending time to the pending time limit, and if the job is pending for longer than this specified time limit, IBM Spectrum LSF RTM triggers the alarm and actions. This parameter works without IBM Spectrum LSF RTM, but LSF does not take any other alarm actions.

In the job forwarding model for the LSF multicluster capability, the job's pending time limit is ignored in the execution cluster, while the submission cluster merges the job's queue-, application-, and job-level pending time limit according to local settings.

The pending time limit is in the form of [hour:]minute. The minutes can be specified as a number greater than 59. For example, three and a half hours can either be specified as 3:30, or 210.

The job-level pending time limit (bsub -ptl) overrides the application-level limit specified here, and the application-level limit overrides the queue-level limit (PEND_TIME_LIMIT in lsb.queues).

Default

Not defined.

PERSISTENT_HOST_ORDER

Syntax

PERSISTENT_HOST_ORDER=Y | yes | N | no

Description

Applies when migrating parallel jobs in the LSF multicluster capability. Setting PERSISTENT_HOST_ORDER=Y ensures that jobs are restarted on hosts based on alphabetical names of the hosts, preventing them from being restarted on the same hosts that they ran on before migration.

Default

PERSISTENT_HOST_ORDER=N. Migrated jobs in LSF multicluster capability could run on the same hosts that they ran on before.

PLAN

For use when the ALLOCATION_PLANNER parameter is enabled. Used to identify the jobs that are candidates for planning.

Syntax

PLAN = Y | N | "<key>[value] ..."

Description

LSF requires that the ALLOCATION_PLANNER parameter is enabled in order to use PLAN=Y.

Also defined at the cluster and queue levels. The precedence is: application, queue, global. For example, application level setting overrides the queue level setting.

The following key-value pairs are supported:

Table 1. Key-Value pairs for PLAN
key value Default Description
DELAY positive integer - Number of minutes to delay before considering making a plan for a job following the job's submission time.
MAX_JOBS positive integer - Maximum number of jobs that can have a plan concurrently.
Note:

The PLAN parameter replaces the existing SLOT_RESERVE parameter and RESOURCE_RESERVE parameter when the ALLOCATION_PLANNER parameter is enabled.

Default

N

POST_EXEC

Syntax

POST_EXEC=command

Description

Enables post-execution processing at the application level. The POST_EXEC command runs on the execution host after the job finishes. Post-execution commands can be configured at the job, application, and queue levels.

If both application-level (POST_EXEC in lsb.applications) and job-level post-execution commands are specified, job level post-execution overrides application-level post-execution commands. Queue-level post-execution commands (POST_EXEC in lsb.queues) run after application-level post-execution and job-level post-execution commands.

The POST_EXEC command uses the same environment variable values as the job, and runs under the user account of the user who submits the job.

When a job exits with one of the application profile’s REQUEUE_EXIT_VALUES, LSF requeues the job and sets the environment variable LSB_JOBPEND. The post-execution command runs after the requeued job finishes.

When the post-execution command is run, the environment variable LSB_JOBEXIT_STAT is set to the exit status of the job. If the execution environment for the job cannot be set up, LSB_JOBEXIT_STAT is set to 0 (zero).

The command path can contain up to 4094 characters for UNIX and Linux, or up to 255 characters for Windows, including the directory, file name, and expanded values for %J (job_ID) and %I (index_ID).

For UNIX:
  • The pre- and post-execution commands run in the /tmp directory under /bin/sh -c, which allows the use of shell features in the commands. The following example shows valid configuration lines:
    PRE_EXEC= /usr/share/lsf/misc/testq_pre >> /tmp/pre.out
    
    POST_EXEC= /usr/share/lsf/misc/testq_post | grep -v "Hey!"
    
  • LSF sets the PATH environment variable to
    PATH="/bin /usr/bin /sbin /usr/sbin"
    
  • The stdin, stdout, and stderr are set to /dev/null
  • To allow UNIX users to define their own post-execution commands, an LSF administrator specifies the environment variable $USER_POSTEXEC as the POST_EXEC command. A user then defines the post-execution command:
    setenv USER_POSTEXEC /path_name
    
    Note: The path name for the post-execution command must be an absolute path. This parameter cannot be used to configure host-based post-execution processing.
For Windows:
  • The pre- and post-execution commands run under cmd.exe /c
  • The standard input, standard output, and standard error are set to NULL
  • The PATH is determined by the setup of the LSF Service
Note:

For post-execution commands that execute on a Windows Server 2003, x64 Edition platform, users must have read and execute privileges for cmd.exe.

Default

Not defined. No post-execution commands are associated with the application profile.

PREEMPT_DELAY

Syntax

PREEMPT_DELAY=seconds

Description

Preemptive jobs will wait the specified number of seconds from the submission time before preempting any low priority preemptable jobs. During the grace period, preemption will not be trigged, but the job can be scheduled and dispatched by other scheduling policies.

This feature can provide flexibility to tune the system to reduce the number of preemptions. It is useful to get better performance and job throughput. When the low priority jobs are short, if high priority jobs can wait a while for the low priority jobs to finish, preemption can be avoided and cluster performance is improved. If the job is still pending after the grace period has expired, the preemption will be triggered.

The waiting time is for preemptive jobs in the pending status only. It will not impact the preemptive jobs that are suspended.

The time is counted from the submission time of the jobs. The submission time means the time mbatchd accepts a job, which includes newly submitted jobs, restarted jobs (by brestart) or forwarded jobs from a remote cluster.

When the preemptive job is waiting, the pending reason is:

The preemptive job is allowing a grace period before preemption.

If you use an older version of bjobs, the pending reason is:

Unknown pending reason code <6701>;

The parameter is defined in lsb.params, lsb.queues (overrides lsb.params), and lsb.applications (overrides both lsb.params and lsb.queues).

Run badmin reconfig to make your changes take effect.

Default

Not defined (if the parameter is not defined anywhere, preemption is immediate).

PRE_EXEC

Syntax

PRE_EXEC=command

Description

Enables pre-execution processing at the application level. The PRE_EXEC command runs on the execution host before the job starts. If the PRE_EXEC command exits with a non-zero exit code, LSF requeues the job to the front of the queue.

Pre-execution commands can be configured at the application, queue, and job levels and run in the following order:
  1. The queue-level command
  2. The application-level or job-level command. If you specify a command at both the application and job levels, the job-level command overrides the application-level command; the application-level command is ignored.

The PRE_EXEC command uses the same environment variable values as the job, and runs under the user account of the user who submits the job.

The command path can contain up to 4094 characters for UNIX and Linux, or up to 255 characters for Windows, including the directory, file name, and expanded values for %J (job_ID) and %I (index_ID).

For UNIX:
  • The pre- and post-execution commands run in the /tmp directory under /bin/sh -c, which allows the use of shell features in the commands. The following example shows valid configuration lines:
    PRE_EXEC= /usr/share/lsf/misc/testq_pre >> /tmp/pre.out
    
    POST_EXEC= /usr/share/lsf/misc/testq_post | grep -v "Hey!"
    
  • LSF sets the PATH environment variable to
    PATH="/bin /usr/bin /sbin /usr/sbin"
    
  • The stdin, stdout, and stderr are set to /dev/null
For Windows:
  • The pre- and post-execution commands run under cmd.exe /c
  • The standard input, standard output, and standard error are set to NULL
  • The PATH is determined by the setup of the LSF Service
Note:

For pre-execution commands that execute on a Windows Server 2003, x64 Edition platform, users must have read and execute privileges for cmd.exe. This parameter cannot be used to configure host-based pre-execution processing.

Default

Not defined. No pre-execution commands are associated with the application profile.

PROCESSLIMIT

Syntax

PROCESSLIMIT=integer

Description

Limits the number of concurrent processes that can be part of a job.

By default, jobs submitted to the application profile without a job-level process limit are killed when the process limit is reached. Application-level limits override any default limit specified in the queue.

SIGINT, SIGTERM, and SIGKILL are sent to the job in sequence when the limit is reached.

Default

Unlimited

PRIORITY

Syntax

PRIORITY =integer

Description

Specifies a priority that is used as a factor when calculating the job priority for absolute job priority scheduling (APS).

Valid values

Specify an integer between 0 and 2147483646.

Default

Not defined.

If APS is enabled for users, user groups, or application profiles, the default value is 0.

RC_ACCOUNT

Assigns an account name (tag) to hosts borrowed through LSF resource connector, so that they cannot be used by other user groups, users, or jobs.

Syntax

RC_ACCOUNT=account_name

Description

When a job is submitted to an application profile with the RC_ACCOUNT parameter specified, hosts borrowed to run the job are tagged with the value of the RC_ACCOUNT parameter. The borrowed host cannot be used by other applications that have a different value for the RC_ACCOUNT parameter (or that don't have the RC_ACCOUNT parameter defined at all).

After the borrowed host joins the cluster, use the lshosts -s command to view the value of the RC_ACCOUNT parameter for the host.

Example

RC_ACCOUNT=project1

Default

No account defined for the application profile

RC_RECLAIM_ACTION

Controls how the LSF resource connector takes action on jobs that are running on a host when that host is reclaimed.

Syntax

RC_RECLAIM_ACTION = REQUEUE | TERMINATE

Description

Specify one of the following actions:

  • REQUEUE: Requeue the jobs that are running on a host that is reclaimed.
  • TERMINATE: Terminate the jobs that are running on a host that is reclaimed.

Default

TERMINATE for interactive jobs.

REQUEUE for all other jobs.

REMOTE_MAX_PREEXEC_RETRY

Syntax

REMOTE_MAX_PREEXEC_RETRY=integer

Description

Job forwarding model for the LSF multicluster capability only. The maximum number of times to attempt the pre-execution command of a job from a remote cluster.

If the job's pre-execution command fails all attempts, the job is returned to the submission cluster.

Valid values

up to INFINIT_INT defined in lsf.h.

Default

5

REQUEUE_EXIT_VALUES

Syntax

REQUEUE_EXIT_VALUES=[exit_code ...] [EXCLUDE(exit_code ...)]

Description

Enables automatic job requeue and sets the LSB_EXIT_REQUEUE environment variable. Use spaces to separate multiple exit codes. Application-level exit values override queue-level values. Job-level exit values (bsub -Q) override application-level and queue-level values.

exit_code has the following form:
"[all] [~number ...] | [number ...]"

The reserved keyword all specifies all exit codes. Exit codes are typically between 0 and 255. Use a tilde (~) to exclude specified exit codes from the list.

Jobs are requeued to the head of the queue. The output from the failed run is not saved, and the user is not notified by LSF.

Define an exit code as EXCLUDE(exit_code) to enable exclusive job requeue, ensuring the job does not rerun on the samehost. Exclusive job requeue does not work for parallel jobs.

For jobs with the LSF multicluster capability forwarded to a remote execution cluster, the exit values specified in the submission cluster with the EXCLUDE keyword are treated as if they were non-exclusive.

You can also requeue a job if the job is terminated by a signal.

If a job is killed by a signal, the exit value is 128+signal_value. The sum of 128 and the signal value can be used as the exit code in the parameter REQUEUE_EXIT_VALUES.

For example, if you want a job to rerun if it is killed with a signal 9 (SIGKILL), the exit value would be 128+9=137. You can configure the following requeue exit value to allow a job to be requeue if it was kill by signal 9:

REQUEUE_EXIT_VALUES=137

In Windows, if a job is killed by a signal, the exit value is signal_value. The signal value can be used as the exit code in the parameter REQUEUE_EXIT_VALUES.

For example, if you want to rerun a job after it was killed with a signal 7 (SIGKILL), the exit value would be 7. You can configure the following requeue exit value to allow a job to requeue after it was killed by signal 7:

REQUEUE_EXIT_VALUES=7

You can configure the following requeue exit value to allow a job to requeue for both Linux and Windows after it was killed:

REQUEUE_EXIT_VALUES=137 7

If mbatchd is restarted, it does not remember the previous hosts from which the job exited with an exclusive requeue exit code. In this situation, it is possible for a job to be dispatched to hosts on which the job has previously exited with an exclusive exit code.

You should configure REQUEUE_EXIT_VALUES for interruptible backfill queues (INTERRUPTIBLE_BACKFILL=seconds).

Example

REQUEUE_EXIT_VALUES=30 EXCLUDE(20)

means that jobs with exit code 30 are requeued, jobs with exit code 20 are requeued exclusively, and jobs with any other exit code are not requeued.

Default

Not defined. Jobs are not requeued.

RERUNNABLE

Syntax

RERUNNABLE=yes | no

Description

If yes, enables automatic job rerun (restart) for any job associated with the application profile.

Rerun is disabled when RERUNNABLE is set to no. The yes and no arguments are not case-sensitive.

Members of a chunk job can be rerunnable. If the execution host becomes unavailable, rerunnable chunk job members are removed from the job chunk and dispatched to a different execution host.

Job level rerun (bsub -r) overrides the RERUNNABLE value specified in the application profile, which overrides the queue specification. bmod -rn to make rerunnable jobs non-rerunnable overrides both the application profile and the queue.

Default

Not defined.

RES_REQ

Syntax

RES_REQ=res_req

Description

Resource requirements used to determine eligible hosts. Specify a resource requirement string as usual. The resource requirement string lets you specify conditions in a more flexible manner than using the load thresholds.

The following resource requirement sections are supported:
  • select
  • rusage
  • order
  • span
  • same
  • cu
  • affinity

Resource requirement strings can be simple (applying to the entire job), compound (applying to the specified number of slots), or can contain alternative resources (alternatives between 2 or more simple and/or compound). When a compound resource requirement is set at the application-level, it will be ignored if any job-level resource requirements (simple or compound) are defined.

Compound and alternative resource requirements follow the same set of rules for determining how resource requirements are going to be merged between job, application, and queue level. In the event no job-level resource requirements are set, the compound or alternative application-level requirements interact with queue-level resource requirement strings in the following ways:

  • When a compound resource requirement is set at the application level, it will be ignored if any job level resource requirements (simple or compound) are defined.
  • If no queue-level resource requirement is defined or a compound or alternative queue-level resource requirement is defined, the application-level requirement is used.
  • If a simple queue-level requirement is defined, the application-level and queue-level requirements combine as follows:

    section compound/alternative application and simple queue behavior
    select both levels satisfied; queue requirement applies to all terms
    same queue level ignored
    order

    span

    application-level section overwrites queue-level section (if a given level is present); queue requirement (if used) applies to all terms
    rusage
    • both levels merge
    • queue requirement if a job-based resource is applied to the first term, otherwise applies to all terms
    • if conflicts occur the application-level section overwrites the queue-level section.

    For example: if the application-level requirement is num1*{rusage[R1]} + num2*{rusage[R2]} and the queue-level requirement is rusage[RQ] where RQ is a job resource, the merged requirement is num1*{rusage[merge(R1,RQ)]} + num2*{rusage[R2]}


Compound or alternative resource requirements do not support the cu section, or the || operator within the rusage section.

Alternative resource strings use the || operator as a separator for each alternative resource.

Multiple -R strings cannot be used with multi-phase rusage resource requirements.

For internal load indices and duration, jobs are rejected if they specify resource reservation requirements at the job or application level that exceed the requirements specified in the queue.

By default, memory (mem) and swap (swp) limits in select[] and rusage[] sections are specified in MB. Use LSF_UNIT_FOR_LIMITS in lsf.conf to specify a larger unit for these limits (GB, TB, PB, or EB).

Resource requirement strings in select sections must conform to a more strict syntax. The strict resource requirement syntax only applies to the select section. It does not apply to the other resource requirement sections (order, rusage, same, span, cu, or affinity). LSF rejects resource requirement strings where an rusage section contains a non-consumable resource.

select section

For simple resource requirements, the select section defined at the application, queue, and job level must all be satisfied.

rusage section

The rusage section can specify additional requests. To do this, use the OR (||) operator to separate additional rusage strings. The job-level rusage section takes precedence.

Note:

Compound resource requirements do not support use of the || operator within the component rusage simple resource requirements. Multiple rusage strings cannot be used with multi-phase rusage resource requirements.

When both job-level and application-level rusage sections are defined using simple resource requirement strings, the rusage section defined for the job overrides the rusage section defined in the application profile. The rusage definitions are merged, with the job-level rusage taking precedence. Any queue-level requirements are then merged with that result.

For example:
Application-level RES_REQ:
RES_REQ=rusage[mem=200:lic=1] ...
For the job submission:
bsub -R "rusage[mem=100]" ...

the resulting requirement for the job is

rusage[mem=100:lic=1]

where mem=100 specified by the job overrides mem=200 specified by the application profile. However, lic=1 from application profile is kept, since job does not specify it.

Application-level RES_REQ threshold:
RES_REQ = rusage[bwidth =2:threshold=5] ...
For the job submission:
bsub -R "rusage[bwidth =1:threshold=6]" ...

the resulting requirement for the job is

rusage[bwidth =1:threshold=6]
Application-level RES_REQ with decay and duration defined:
RES_REQ=rusage[mem=200:duration=20:decay=1] ...
For a job submission with no decay or duration:
bsub -R "rusage[mem=100]" ...
the resulting requirement for the job is:
rusage[mem=100:duration=20:decay=1]

Application-level duration and decay are merged with the job-level specification, and mem=100 for the job overrides mem=200 specified by the application profile. However, duration=20 and decay=1 from application profile are kept, since job does not specify them.

Application-level RES_REQ with resource reservation method:
RES_REQ=rusage[mem=200/host:duration=20:decay=1] ...
For a job submission with no decay or duration:
bsub -R'rusage[mem=100/task]' ...
the resulting requirement for the job is:
rusage[mem=100/task:duration=20:decay=1]
Application-level RES_REQ with multi-phase job-level rusage:
RES_REQ=rusage[mem=(200 150):duration=(10 10):decay=(1),swap=100] ...
For a multi-phase job submission:
bsub -app app_name -R "rusage[mem=(600 350):duration=(20 10):decay=(0 1)]" ...
the resulting requirement for the job is:
rusage[mem=(600 350):duration=(20 10):decay=(0 1),swap=100]

The job-level values for mem, duration and decay override the application-level values. However, swap=100 from the application profile is kept, since the job does not specify swap.

Application-level RES_REQ with multi-phase application-level rusage:
RES_REQ=rusage[mem=(200 150):duration=(10 10):decay=(1)] ...
For a job submission:
bsub -app app_name -R "rusage[mem=200:duration=15:decay=0]" ...
the resulting requirement for the job is:
rusage[mem=200:duration=15:decay=0]

Job-level values override the application-level multi-phase rusage string.

Note: The merged application-level and job-level rusage consumable resource requirements must satisfy any limits set by the parameter RESRSV_LIMIT in lsb.queues, or the job will be rejected.

order section

For simple resource requirements the order section defined at the job-level overrides any application-level order section. An application-level order section overrides queue-level specification. The order section defined at the application level is ignored if any resource requirements are specified at the job level. If the no resource requirements include an order section, the default order r15s:pg is used.

The command syntax is:

[!][-]resource_name [: [-]resource_name]

For example:

bsub -R "order[!ncpus:mem]" myjob

"!" only works with consumable resources because resources can be specified in the rusage[] section and their value may be changed in schedule cycle (for example, slot or memory). In LSF scheduler, slots under RUN, SSUSP, USUP and RSV may be freed in different scheduling phases. Therefore, the slot value may change in different scheduling cycles.

span section

For simple resource requirements the span section defined at the job-level overrides an application-level span section, which overrides a queue-level span section.

Note: Define span[hosts=-1] in the application profile or in bsub -R resource requirement string to disable the span section setting in the queue.

same section

For simple resource requirements all same sections defined at the job-level, application-level, and queue-level are combined before the job is dispatched.

cu section

For simple resource requirements the job-level cu section overrides the application-level, and the application-level cu section overrides the queue-level.

affinity section

For simple resource requirements the job-level affinity section overrides the application-level, and the application-level affinity section overrides the queue-level.

Default

select[type==local] order[r15s:pg]

If this parameter is defined and a host model or Boolean resource is specified, the default type is any.

RESIZABLE_JOBS

Syntax

RESIZABLE_JOBS = [Y|N|auto]

Description

N|n: The resizable job feature is disabled in the application profile. Under this setting, all jobs attached to this application profile are not resizable. All bresize and bsub -ar commands will be rejected with a proper error message.

Y|y: Resize is enabled in the application profile and all jobs belonging to the application are resizable by default. Under this setting, users can run bresize commands to cancel pending resource allocation requests for the job or release resources from an existing job allocation, or use bsub to submit an autoresizable job.

auto: All jobs belonging to the application will be autoresizable.
Restriction: The auto option is only supported in RESIZABLE_JOBS under lsb.applications and not in lsb.params

Resizable jobs must be submitted with an application profile that defines RESIZABLE_JOBS as either auto or Y. If an application defines RESIZABLE_JOBS=auto, but an administrator changes it to N and reconfigures LSF, jobs without the job-level auto resizable attribute are no longer autoresizable. For running jobs that are in the middle of the notification stage, LSF lets the current notification complete and stops scheduling. Changing RESIZABLE_JOBS configuration does not affect jobs with the job-level autoresizable attribute (This behavior is the same as for exclusive jobs, bsub -x, and the EXCLUSIVE parameter at the queue level).

Resizable jobs can have alternative and compound resource requirements. When using bresize release to release slots from compound resource requirements, you can only release slots represented by the last term of the compound resource requirement. To release slots in earlier terms, run bresize release repeatedly to release slots in subsequent last terms.

Default

If the parameter is undefined, the default value is N.

See also

RESIZABLE_JOBS in lsb.params

RESIZE_NOTIFY_CMD

Syntax

RESIZE_NOTIFY_CMD = notification_command

Description

Defines an executable command to be invoked on the first execution host of a job when a resize event occurs. The maximum length of notification command is 4 KB.

Default

Not defined. No resize notification command is invoked.

RESUME_CONTROL

Syntax

RESUME_CONTROL=signal | command
Remember: Unlike the JOB_CONTROLS parameter in lsb.queues, the RESUME_CONTROL parameter does not require square brackets ([ ]) around the action.
  • signal is a UNIX signal name. The specified signal is sent to the job. The same set of signals is not supported on all UNIX systems. To display a list of the symbolic names of the signals (without the SIG prefix) supported on your system, use the kill -l command.
  • command specifies a /bin/sh command line to be invoked. Do not quote the command line inside an action definition. Do not specify a signal followed by an action that triggers the same signal. For example, do not specify RESUME_CONTROL=bresume. This causes a deadlock between the signal and the action.

Description

Changes the behavior of the RESUME action in LSF.
  • The contents of the configuration line for the action are run with /bin/sh -c so you can use shell features in the command.
  • The standard input, output, and error of the command are redirected to the NULL device, so you cannot tell directly whether the command runs correctly. The default null device on UNIX is /dev/null.
  • The command is run as the user of the job.
  • All environment variables set for the job are also set for the command action. The following additional environment variables are set:
    • LSB_JOBPGIDS — a list of current process group IDs of the job
    • LSB_JOBPIDS —a list of current process IDs of the job
  • If the command fails, LSF retains the original job status.

The command path can contain up to 4094 characters for UNIX and Linux, or up to 255 characters for Windows, including the directory, file name, and expanded values for %J (job_ID) and %I (index_ID).

Default

  • On UNIX, by default, RESUME sends SIGCONT.
  • On Windows, actions equivalent to the UNIX signals have been implemented to do the default job control actions. Job control messages replace the SIGINT and SIGTERM signals, but only customized applications are able to process them.

RTASK_GONE_ACTION

Syntax

RTASK_GONE_ACTION="[KILLJOB_TASKDONE | KILLJOB_TASKEXIT] [IGNORE_TASKCRASH]"

Description

Defines the actions LSF should take if it detects that a remote task of a parallel or distributed job is gone.

This parameter only applies to the blaunch distributed application framework.

IGNORE_TASKCRASH

A remote task crashes. LSF does nothing. The job continues to launch the next task.

KILLJOB_TASKDONE

A remote task exits with zero value. LSF terminates all tasks in the job.

KILLJOB_TASKEXIT

A remote task exits with non-zero value. LSF terminates all tasks in the job.

Environment variable

When defined in an application profile, the LSB_DJOB_RTASK_GONE_ACTION variable is set when running bsub -app for the specified application.

You can also use the environment variable LSB_DJOB_RTASK_GONE_ACTION to override the value set in the application profile.

Example

RTASK_GONE_ACTION="IGNORE_TASKCRASH KILLJOB_TASKEXIT"

Default

Not defined. LSF does nothing.

RUNLIMIT

Syntax

RUNLIMIT=[hour:]minute[/host_name | /host_model]

Description

The default run limit. The name of a host or host model specifies the runtime normalization host to use.

By default, jobs that are in the RUN state for longer than the specified run limit are killed by LSF. You can optionally provide your own termination job action to override this default.

Jobs submitted with a job-level run limit (bsub -W) that is less than the run limit are killed when their job-level run limit is reached. Jobs submitted with a run limit greater than the maximum run limit are rejected. Application-level limits override any default limit specified in the queue.

Note: If you want to provide an estimated run time for scheduling purposes without killing jobs that exceed the estimate, define the ESTIMATED_RUNTIME parameter in the application profile, or submit the job with -We instead of a run limit.

The run limit is in the form of [hour:]minute. The minutes can be specified as a number greater than 59. For example, three and a half hours can either be specified as 3:30, or 210.

The run limit you specify is the normalized run time. This is done so that the job does approximately the same amount of processing, even if it is sent to host with a faster or slower CPU. Whenever a normalized run time is given, the actual time on the execution host is the specified time multiplied by the CPU factor of the normalization host then divided by the CPU factor of the execution host.

If ABS_RUNLIMIT=Y is defined in lsb.params or in the application profile, the runtime limit is not normalized by the host CPU factor. Absolute wall-clock run time is used for all jobs submitted to an application profile with a run limit configured.

Optionally, you can supply a host name or a host model name defined in LSF. You must insert ‘/’ between the run limit and the host name or model name. (See lsinfo(1) to get host model information.)

If no host or host model is given, LSF uses the default runtime normalization host defined at the queue level (DEFAULT_HOST_SPEC in lsb.queues) if it has been configured; otherwise, LSF uses the default CPU time normalization host defined at the cluster level (DEFAULT_HOST_SPEC in lsb.params) if it has been configured; otherwise, the host with the largest CPU factor (the fastest host in the cluster).

For jobs with the LSF multicluster capability, if no other CPU time normalization host is defined and information about the submission host is not available, LSF uses the host with the largest CPU factor (the fastest host in the cluster).

Jobs submitted to a chunk job queue are not chunked if RUNLIMIT is greater than 30 minutes.

Default

Unlimited

RUNTIME

This parameter is deprecated. Use ESTIMATED_RUNTIME instead.

STACKLIMIT

Syntax

STACKLIMIT=integer

Description

The per-process (soft) stack segment size limit for all of the processes belonging to a job from this queue (see getrlimit(2)). Application-level limits override any default limit specified in the queue, but must be less than the hard limit of the submission queue.

By default, the limit is specified in KB. Use LSF_UNIT_FOR_LIMITS in lsf.conf to specify a larger unit for the limit (MB, GB, TB, PB, or EB).

Default

Unlimited

SUCCESS_EXIT_VALUES

Syntax

SUCCESS_EXIT_VALUES=[exit_code …]

Description

Specifies exit values used by LSF to determine if job was done successfully. Use spaces to separate multiple exit codes. Job-level success exit values specified with the LSB_SUCCESS_EXIT_VALUES environment variable override the configration in application profile.

Use SUCCESS_EXIT_VALUES for applications that successfully exit with non-zero values so that LSF does not interpret non-zero exit codes as job failure.

exit_code should be the value between 0 and 255. Use spaces to separate exit code values.

If both SUCCESS_EXIT_VALUES and REQUEUE_EXIT_VALUES are defined with the same exit code, REQUEUE_EXIT_VALUES will take precedence and the job will be set to PEND state and requeued.

Default

0

SUSPEND_CONTROL

Syntax

SUSPEND_CONTROL=signal | command | CHKPNT
Remember: Unlike the JOB_CONTROLS parameter in lsb.queues, the SUSPEND_CONTROL parameter does not require square brackets ([ ]) around the action.
  • signal is a UNIX signal name (for example, SIGTSTP). The specified signal is sent to the job. The same set of signals is not supported on all UNIX systems. To display a list of the symbolic names of the signals (without the SIG prefix) supported on your system, use the kill -l command.
  • command specifies a /bin/sh command line to be invoked.
    • Do not quote the command line inside an action definition.
    • Do not specify a signal followed by an action that triggers the same signal. For example, do not specify SUSPEND_CONTROL=bstop. This causes a deadlock between the signal and the action.
  • CHKPNT is a special action, which causes the system to checkpoint the job. The job is checkpointed and then stopped by sending the SIGSTOP signal to the job automatically.

Description

Changes the behavior of the SUSPEND action in LSF.
  • The contents of the configuration line for the action are run with /bin/sh -c so you can use shell features in the command.
  • The standard input, output, and error of the command are redirected to the NULL device, so you cannot tell directly whether the command runs correctly. The default null device on UNIX is /dev/null.
  • The command is run as the user of the job.
  • All environment variables set for the job are also set for the command action. The following additional environment variables are set:
    • LSB_JOBPGIDS - a list of current process group IDs of the job
    • LSB_JOBPIDS - a list of current process IDs of the job
    • LSB_SUSP_REASONS - an integer representing a bitmap of suspending reasons as defined in lsbatch.h The suspending reason can allow the command to take different actions based on the reason for suspending the job.
    • LSB_SUSP_SUBREASONS - an integer representing the load index that caused the job to be suspended
  • If the command fails, LSF retains the original job status.

When the suspending reason SUSP_LOAD_REASON (suspended by load) is set in LSB_SUSP_REASONS, LSB_SUSP_SUBREASONS is set to one of the load index values defined in lsf.h.

Use LSB_SUSP_REASONS and LSB_SUSP_SUBREASONS together in your custom job control to determine the exact load threshold that caused a job to be suspended.

  • If an additional action is necessary for the SUSPEND command, that action should also send the appropriate signal to the application. Otherwise, a job can continue to run even after being suspended by LSF. For example, SUSPEND_CONTROL=bkill $LSB_JOBPIDS; command

The command path can contain up to 4094 characters for UNIX and Linux, or up to 255 characters for Windows, including the directory, file name, and expanded values for %J (job_ID) and %I (index_ID).

Default

  • On UNIX, by default, SUSPEND sends SIGTSTP for parallel or interactive jobs and SIGSTOP for other jobs.
  • On Windows, actions equivalent to the UNIX signals have been implemented to do the default job control actions. Job control messages replace the SIGINT and SIGTERM signals, but only customized applications are able to process them.

SWAPLIMIT

Syntax

SWAPLIMIT=integer

Description

Limits the amount of total virtual memory limit for the job.

This limit applies to the whole job, no matter how many processes the job may contain. Application-level limits override any default limit specified in the queue.

The action taken when a job exceeds its SWAPLIMIT or PROCESSLIMIT is to send SIGQUIT, SIGINT, SIGTERM, and SIGKILL in sequence. For CPULIMIT, SIGXCPU is sent before SIGINT, SIGTERM, and SIGKILL.

By default, the limit is specified in KB. Use LSF_UNIT_FOR_LIMITS in lsf.conf to specify a larger unit for the limit (MB, GB, TB, PB, or EB).

Default

Unlimited

TASKLIMIT

Syntax

TASKLIMIT=[minimum_limit [default_limit]] maximum_limit

Description

Note: TASKLIMIT replaces PROCLIMIT as of LSF 9.1.3.

Maximum number of tasks that can be allocated to a job. For parallel jobs, the maximum number of tasks that can be allocated to the job.

Queue level TASKLIMIT has the highest priority over application level TASKLIMIT and job level TASKLIMIT. Application level TASKLIMIT has higher priority than job level TASKLIMIT. Job-level limits must fall within the maximum and minimum limits of the application profile and the queue.

Note: If you also defined JOB_SIZE_LIST in the same application profile where you defined TASKLIMIT, the TASKLIMIT parameter is ignored.

Optionally specifies the minimum and default number of job tasks. All limits must be positive numbers greater than or equal to 1 that satisfy the following relationship:

1 <= minimum <= default <= maximum

In the job forwarding model for the LSF multicluster capability, the local cluster considers the receiving queue's TASKLIMIT on remote clusters before forwarding jobs. If the receiving queue's TASKLIMIT definition in the remote cluster cannot satisfy the job's task requirements, the job is not forwarded to that remote queue.

Default

Unlimited, the default number of tasks is 1

TERMINATE_CONTROL

Syntax

TERMINATE_CONTROL=signal | command | CHKPNT
Remember: Unlike the JOB_CONTROLS parameter in lsb.queues, the TERMINATE_CONTROL parameter does not require square brackets ([ ]) around the action.
  • signal is a UNIX signal name (for example, SIGTERM). The specified signal is sent to the job. The same set of signals is not supported on all UNIX systems. To display a list of the symbolic names of the signals (without the SIG prefix) supported on your system, use the kill -l command.
  • command specifies a /bin/sh command line to be invoked.
    • Do not quote the command line inside an action definition.
    • Do not specify a signal followed by an action that triggers the same signal. For example, do not specify TERMINATE_CONTROL=bkill. This causes a deadlock between the signal and the action.
  • CHKPNT is a special action, which causes the system to checkpoint the job. The job is checkpointed and killed automatically.

Description

Changes the behavior of the TERMINATE action in LSF.
  • The contents of the configuration line for the action are run with /bin/sh -c so you can use shell features in the command.
  • The standard input, output, and error of the command are redirected to the NULL device, so you cannot tell directly whether the command runs correctly. The default null device on UNIX is /dev/null.
  • The command is run as the user of the job.
  • All environment variables set for the job are also set for the command action. The following additional environment variables are set:
    • LSB_JOBPGIDS — a list of current process group IDs of the job
    • LSB_JOBPIDS —a list of current process IDs of the job

The command path can contain up to 4094 characters for UNIX and Linux, or up to 255 characters for Windows, including the directory, file name, and expanded values for %J (job_ID) and %I (index_ID).

Default

  • On UNIX, by default, TERMINATE sends SIGINT, SIGTERM and SIGKILL in that order.
  • On Windows, actions equivalent to the UNIX signals have been implemented to do the default job control actions. Job control messages replace the SIGINT and SIGTERM signals, but only customized applications are able to process them. Termination is implemented by the TerminateProcess() system call.

THREADLIMIT

Syntax

THREADLIMIT=integer

Description

Limits the number of concurrent threads that can be part of a job. Exceeding the limit causes the job to terminate. The system sends the following signals in sequence to all processes belongs to the job: SIGINT, SIGTERM, and SIGKILL.

By default, jobs submitted to the queue without a job-level thread limit are killed when the thread limit is reached. Application-level limits override any default limit specified in the queue.

The limit must be a positive integer.

Default

Unlimited

USE_PAM_CREDS

Applies PAM limits to this application.

Syntax

USE_PAM_CREDS=y | n | [limits] [session]

Description

USE_PAM_CREDS is only supported on Linux systems. If the execution host does not have PAM configured and this parameter is enabled, the job fails.

If USE_PAM_CREDS is set to y or limits, LSF can apply PAM limits to an application when its job is dispatched to a Linux host using PAM LSF. The LSF job does not run within the PAM session.

If USE_PAM_CREDS is set to session:
  • If a job is started on the first execution host, the job RES opens a PAM session for the user and forks a RES process within that session. This RES process becomes the user's job.
  • If a task is launched by the blaunch command or an API, the task RES opens a PAM session for the user and executes a RES process within that session. This RES process becomes the user's task.

The limits keyword can be defined together with the session keyword.

If LSF limits are more restrictive than PAM limits, LSF limits are used, otherwise PAM limits are used. PAM limits are system resource limits defined in the limits.conf file.

For parallel jobs, PAM sessions are only launched on the first execution host if USE_PAM_CREDS=y or USE_PAM_CREDS=limits is defined. PAM sessions are launched on all tasks if USE_PAM_CREDS=session or USE_PAM_CREDS=limits session is defined.

Note: When configuring Linux PAM to be used with LSF, you must configure Linux PAM so that it does not ask users for their passwords because jobs are not usually interactive.
Depending on the USE_PAM_CREDS parameter setting, LSF assumes that the following Linux PAM services are created:
  • If USE_PAM_CREDS is set to y or limits, LSF assumes that the Linux PAM service "lsf" is created.
  • If USE_PAM_CREDS is set to session, LSF assumes that the Linux PAM service "lsf-<clustername>" is created.
  • If USE_PAM_CREDS is set to limits session, LSF assumes that the Linux PAM services "lsf" and "lsf-<clustername>" are created.
It is also assumed that the "lsf" service is used in conjunction with the /etc/security/limits.conf file.

The job sbatchd daemon checks the lsf service, and the job or task RES daemon checks the lsf-<clustername> service.

Overrides MEMLIMIT_TYPE=Process.

Overridden (for CPU limit only) by LSB_JOB_CPULIMIT=y.

Overridden (for memory limits only) by LSB_JOB_MEMLIMIT=y.

The USE_PAM_CREDS value in lsb.applications overrides the USE_PAM_CREDS value in lsb.queues.

Default

n. USE_PAM_CREDS is disabled.

WATCHDOG

Syntax

WATCHDOG=script[file/path/to/script] init[init_delay] period[start_interval]

Description

Enables LSF to use the watchdog feature to regularly run external scripts that check application data, logs, and other information. LSF can use these scripts to pass on the job information.

This parameter uses the following keywords:

script
Required. This keyword specifies the file path to the external watchdog script to check the application data and other information. This file must have the proper permissions for the job submission user to execute the script.
init
Optional. This keyword specifies the delay to start the watchdog script after the job starts, in seconds. Specify a number larger than 30 seconds. The default value is 60 seconds.
period
Optional. This keyword specifies the interval in which to start the watchdog script after the previous time that the watchdog script started, in seconds. Specify a number larger than 30 seconds. The default value is 60 seconds.

All job environment variables are available to the watchdog scripts. In addition, the following LSF job-level resource consumption environment variables are available to the watchdog scripts:

  • LSB_GPU_ALLOC_INFO
  • LSB_JOB_AVG_MEM
  • LSB_JOB_CPU_TIME
  • LSB_JOB_MAX_MEM
  • LSB_JOB_MEM
  • LSB_JOB_NTHREAD
  • LSB_JOB_PGIDS
  • LSB_JOB_PIDS
  • LSB_JOB_RUN_TIME
  • LSB_JOB_SWAP

Default

Not defined.

Automatic time-based configuration

Use if-else constructs and time expressions to define time windows in the file. Configuration defined within in a time window applies only during the specified time period; configuration defined outside of any time window applies at all times. After editing the file, run badmin reconfig to reconfigure the cluster.

Time expressions in the file are evaluated by LSF every 10 minutes, based on the mbatchd daemon start time. When an expression evaluates true, LSF changes the configuration in real time, without restarting mbatchd, providing continuous system availability.

Time-based configuration also supports LSF multicluster capability configuration in terms of shared configuration for groups of clusters (using the #include parameter). You can include a common configuration file by using the time-based feature in local configuration files.

Example

Begin application
NAME=app1
#if time(16:00-18:00 EDT)
CPULIMIT=180/hostA
#else
CPULIMIT=60/hostA
#endif
End application
In this example, for two hours every day, the configuration is the following:
Begin application
NAME=app1
CPULIMIT=180/hostA
End application
The rest of the time, the configuration is the following:
Begin application
NAME=app1
CPULIMIT=60/hostA
End application

Specifying the time zone is optional. If you do not specify a time zone, LSF uses the local system time zone. LSF supports all standard time zone abbreviations.