Environment variable reference

Reference for LSF environment variables.

BSUB_BLOCK

The BSUB_BLOCK variable is set when you submit a job for synchronous job execution with the bsub -K option.

Description

If set, tells NIOS that the job is running in batch mode.

Default

Not defined

Notes

Synchronous execution means you have to wait for the job to finish before you can continue.

Where defined

Set internally

BSUB_CHK_RESREQ

When the BSUB_CHK_RESREQ variable is set, bsub checks the syntax of the resource requirement selection string without actually submitting the job for scheduling and dispatch.

Syntax

BSUB_CHK_RESREQ=any_value

Description

Use BSUB_CHK_RESREQ to check the compatibility of your existing resource requirement select strings against the strict syntax.

The bsub command only checks the select section of the resource requirement. Other sections in the resource requirement string are not checked.

Default

Not defined

Where defined

From the command line

Example

BSUB_CHK_RESREQ=1

BSUB_QUIET

Syntax

BSUB_QUIET=value

Description

Controls the printed output information about job submissions.

In Fix Pack 13 and earlier:

If BSUB_QUIET is set to any value, messages are suppressed; the bsub command will not print any information about submitted jobs.
If BSUB_QUIET is not set, the bsub command prints the default message of Job job_ID is submitted to default queue queue_type will be printed. For example, if the most recently submitted job ID is 100, and the queue is normal, then the command prints:
```
Job 100 is submitted to default queue normal
```

Starting in Fix Pack 14:

If set to BSUB_QUIET=JOBID, the bsub command prints the JOBID value in the output. For example, if the most recently submitted job ID is 100, bsub prints:
```
100
```
If set to BSUB_QUIET=QUEUE, the bsub command prints the JOBID and QUEUE values in the output, with a single space to separating the values. For example, if the most recently submitted job ID is 100, and the queue is normal, then bsub prints:
```
100 normal
```
If BSUB_QUIET is set to any other value, messages are suppressed; that is, if BSUB_QUIET=any_other_value, the bsub command will not print any information about submitted jobs.
If BSUB_QUIET is not set, the bsub command prints the default message of Job job_ID is submitted to default queue queue_type will be printed. For example, if the most recently submitted job ID is 100, and the queue is normal, then the command prints:
```
Job 100 is submitted to default queue normal
```

Default

Not defined

Where defined

From the command line

Examples

BSUB_QUIET=JOBID

BSUB_QUIET=QUEUE

BSUB_QUIET=any_other_value

BSUB_QUIET=100

BSUB_QUIET2

Syntax

BSUB_QUIET2=any_value

Description

Suppresses the printing of information about job completion when a job is submitted with the bsub -K option.

If set, bsub will not print information about job completion to stdout. For example, when this variable is set, the message <<Job is finished>> will not be written to stdout.

If BSUB_QUIET and BSUB_QUIET2 are both set, no job messages will be printed to stdout.

Default

Not defined

Where defined

From the command line

Example

BSUB_QUIET2=1

BSUB_STDERR

Syntax

BSUB_STDERR=y

Description

Redirects LSF messages for bsub to stderr.

By default, when this parameter is not set, LSF messages for bsub are printed to stdout.

When this parameter is set, LSF messages for bsub are redirected to stderr.

Default

Not defined

Where defined

From the command line on UNIX. For example, in csh:

setenv BSUB_STDERR Y

From the Control Panel on Windows, as an environment variable

CLEARCASE_DRIVE

Syntax

CLEARCASE_DRIVE=drive_letter:

Description

Optional, Windows only.

Defines the virtual drive letter for a Rational ClearCase view to the drive. This is useful if you wish to map a Rational ClearCase view to a virtual drive as an alias.

If this letter is unavailable, Windows attempts to map to another drive. Therefore, CLEARCASE_DRIVE only defines the default drive letter to which the Rational ClearCase view is mapped, not the final selected drive letter. However, the PATH value is automatically updated to the final drive letter if it is different fromCLEARCASE_DRIVE.

Notes

CLEARCASE_DRIVE is not case sensitive.

Where defined

From the command line

Examples

CLEARCASE_DRIVE=F:

CLEARCASE_DRIVE=f:

CLEARCASE_MOUNTDIR

Syntax

CLEARCASE_MOUNTDIR=path

Description

Optional.

Defines the Rational ClearCase mounting directory.

Default

/vobs

Notes

CLEARCASE_MOUNTDIR is used if any of the following conditions apply:

A job is submitted from a UNIX environment but run in a Windows host.
The Rational ClearCase mounting directory is not the default /vobs

Where defined

From the command line

Example

CLEARCASE_MOUNTDIR=/myvobs

CLEARCASE_ROOT

Syntax

CLEARCASE_ROOT=path

Description

The path to the Rational ClearCase view.

In Windows, this path must define an absolute path starting with the default Rational ClearCase drive and ending with the view name without an ending backslash (\).

Notes

CLEARCASE_ROOT must be defined if you want to submit a batch job from a Rational ClearCase view.

For interactive jobs, use the bsub -I command to submit the job.

Where defined

In the job starter, or from the command line

Example

In UNIX:

CLEARCASE_ROOT=/view/myview

In Windows:

CLEARCASE_ROOT=F:\myview

DAEMON_WRAP_DEBUG

LSF integration for Rational ClearCase. Enables daemon wrapper debugging.

Syntax

DAEMON_WRAP_DEBUG=1

Description

When DAEMON_WRAP_DEBUG is set to 1, LSF logs the Rational ClearCase stdout and stderr streams to /tmp/daemons.wrap.log. Use one of the following shell commands to set the DAEMON_WRAP_DEBUG environment variable:

In csh or tcsh:
```
setenv DAEMON_WRAP_DEBUG 1
```
In sh, ksh, or bash:
```
export DAEMON_WRAP_DEBUG=1
```

ELIM_ABORT_VALUE

Syntax

ELIM_ABORT_VALUE

Description

Used when writing an elim executable to test whether the elim should run on a particular host. If the host does not have or share any of the resources listed in the environment variable LSF_RESOURCES, your elim should exit with $ELIM_ABORT_VALUE.

When the MELIM finds an elim that exited with ELIM_ABORT_VALUE, the MELIM marks the elim and does not restart it on that host.

Where defined

Set by the management elim (MELIM) on the host when the MELIM invokes the elim executable

JOB_GPU_VENDOR

Syntax

JOB_GPU_VENDOR=amd | nvidia

Description

Specifies the GPU vendor type that is allowed for GPU jobs. LSF rejects any jobs that are submitted with -gpu properties that conflict with this vendor type.

When defined as amd, the GPU job only runs on AMD GPUs.

When defined as nvidia, the GPU job only runs on Nvidia GPUs.

Default

nvidia

LS_EXEC_T

Syntax

LS_EXEC_T=START | END | CHKPNT | JOB_CONTROLS

Description

Indicates execution type for a job. LS_EXEC_T is set to:

START or END for a job when the job begins executing or when it completes execution
CHKPNT when the job is check-pointed
JOB_CONTROLS when a control action is initiated

Where defined

Set by sbatchd during job execution

LS_JOBPID

Description

The process ID of the job.

Where defined

During job execution, sbatchd sets LS_JOBPID to be the same as the process ID assigned by the operating system.

LS_SUBCWD

Description

The current working directory (cwd) of the submission host where the remote task command was executed.

The current working directory can be up to 4094 characters long for UNIX and Linux or up to 255 characters for Windows.

This is the flow:

LSF looks for the PWD environment variable. If it finds it, sets LS_SUBCWD to PWD.
If the PWD environment variable does not exist, LSF looks for the CWD environment variable. If it finds CWD, sets LS_SUBCWD to CWD.
If the CWD environment variable does not exist, LSF calls the getwd() system function to retrieve the current working directory path name. LSF sets LS_SUBCWD to the value that is returned.

Where defined

Set by sbatchd

LSB_AFFINITY_HOSTFILE

Syntax

LSB_AFFINITY_HOSTFILE=file_path

Description

Path to the NUMA CPU and memory affinity binding decision file.

On the first execution host, LSF sbatchd will create a binding decision file per job under the same location as $LSB_DJOB_HOSTFILE. The binding decision file has a format similar to the job host file; that is, one task per line, where each line includes: host_name cpu_id_list NUMA_node_id_list memory_policy.

For memory policy, a value of 1 means localonly, 2 means localprefer, as specified in the affinity resource requirement membind parameter.

Comma (,) is the only supported delimiter for the list of CPU IDs and the list of NUMA node IDs.

The following host file represents a job with six tasks:

Host1 and Host2 each have two tasks bound to CPUs {0,1,2,3} and {4,5,6,7}, and NUMA nodes 0 and 1 respectively with a membind=localprefer policy.
Host3 and Host4 each have one task bound to CPUs {0,1,2,3} and NUMA node 0, again with a membind=localprefer policy.

Host1 0,1,2,3 0 2
Host1 4,5,6,7 1 2
Host2 0,1,2,3 0 2
Host2 4,5,6,7 1 2
Host3 0,1,2,3 0 2
Host4 0,1,2,3 0 2

Default

Not defined

Where defined

Set during job execution.

LSB_BHIST_HOURS

Usage

LSB_BHIST_HOURS=hours

Description

The multiple-line format of bhist considers job events from the last LSB_BHIST_HOURS.

Use LSB_BHIST_HOURS with the command bhist -t to display job events before the current time for a longer or shorter period than the default one week.

Default

Not defined

If not defined, the multiple-line format for bhist considers job events from the last week.

LSB_BIND_CPU_LIST

Syntax

LSB_BIND_CPU_LIST=cpu_list

Description

LSB_BIND_CPU_LIST contains allocated CPUs on each host. LSF will combine all of the allocated CPUs for all the tasks that ended up on the host (for the given job), and set this environment variable to the entire list before launching tasks. The following example corresponds to the tasks specified in the example for LSB_AFFINITY_HOSTFILE:

LSB_BIND_CPU_LIST on Host1: 0,1,2,3,4,5,6,7
LSB_BIND_CPU_LIST on Host2: 0,1,2,3,4,5,6,7
LSB_BIND_CPU_LIST on Host3: 0,1,2,3 
LSB_BIND_CPU_LIST on Host4: 0,1,2,3

Default

Not defined

Where defined

Set during job execution.

LSB_BIND_MEM_LIST

Syntax

LSB_BIND_MEM_LIST=memory_node_list

Description

LSB_BIND_MEM_LIST contains allocated memory nodes on each host. If the job is submitted with a memory affinity requirement, LSF will combine all of the allocated NUMA node IDs for all the tasks that ended up on the host (for the given job), and set this environment variable to the entire list before launching tasks. The following example corresponds to the tasks specified in the example for LSB_AFFINITY_HOSTFILE:

LSB_BIND_MEM_LIST on Host1: 0,1
LSB_BIND_MEM_LIST on Host2: 0,1
LSB_BIND_MEM_LIST on Host3: 0
LSB_BIND_MEM_LIST on Host4: 0

Default

Not defined

Where defined

Set during job execution.

LSB_BIND_MEM_POLICY

Syntax

LSB_BIND_MEM_POLICY=localprefer | localonly

Description

For jobs submitted with a NUMA memory affinity resource requirement, LSF sets the memory binding policy in LSB_BIND_MEM_POLICY environment variable, as specified in the membind affinity resource requirement parameter: either localprefer or localonly.

Default

Not defined

Where defined

Set during job execution.

LSB_BHOSTS_FORMAT

This parameter can be set from the command line or from lsf.conf.

See LSB_BHOSTS_FORMAT in lsf.conf.

LSB_BJOBS_FORMAT

This parameter can be set from the command line or from lsf.conf.

See LSB_BJOBS_FORMAT in lsf.conf.

LSB_BMGROUP_ALLREMOTE_EXPAND

Syntax

This parameter can be set from the command line or from lsf.conf.

See LSB_BMGROUP_ALLREMOTE_EXPAND in lsf.conf.

LSB_BQUEUES_FORMAT

This parameter can be set from the command line or from lsf.conf.

See LSB_BQUEUES_FORMAT in lsf.conf.

LSB_BSUB_ERR_RETRY

Syntax

LSB_BSUB_ERR_RETRY=RETRY_CNT[number] ERR_TYPE[error1 error2 ...]

Description

In some cases, jobs can benefit from being automatically retried in the case of failing for a particular error. When specified, LSB_BSUB_ERR_RETRY automatically retries jobs that exit with a particular reason, up to the number of times specified by RETRY_CNT.

Only the following error types (ERR_TYPE) are supported:

BAD_XDR: Error during XDR.
MSG_SYS: Failed to send or receive a message.
INTERNAL: Internal library error.

The number of retries (RETRY_CNT) can be a minimum of 1 to a maximum of 50.

Considerations when setting this parameter:

Users may experience what seems like a lag during job submission while the job is retried automatically in the background.
Users may see a job submitted more than once, with no explanation (no error is communicated to the user; the job keeps getting submitted until it succeeds or reaches its maximum retry count). In this case, the job ID also changes each time the error is retried.

Default

LSB_BUSERS_FORMAT

This parameter can be set from the command line or from lsf.conf.

See LSB_BUSERS_FORMAT in lsf.conf.

LSB_CHKPNT_DIR

Syntax

LSB_CHKPNT_DIR=checkpoint_dir/job_ID

Description

The directory containing files related to the submitted checkpointable job.

Valid values

The value of checkpoint_dir is the directory you specified through the -k option of bsub when submitting the checkpointable job.

The value of job_ID is the job ID of the checkpointable job.

Where defined

Set by LSF, based on the directory you specified when submitting a checkpointable job with the -k option of bsub.

LSB_CONTAINER_IMAGE_UPDATE

Syntax

LSB_CONTAINER_IMAGE_UPDATE=Y | y

Description

If defined, LSF updates the Docker image that is specified in the job related application file before execution.

Default

If undefined, the Docker image does not update.

LSB_DATA_CACHE_TOP

Contains a string defining the location of the data management staging area relative to the compute node. The value of this variable is equivalent to STAGING_AREA in lsf.datamanager.

LSB_DATA_META_FILE

For jobs submitted with data requirements, this variable contains a string defining the location of the job's metadata file relative to the compute node. The value of this variable is equivalent to the value of STAGING_AREA/work/cluster_name/jobID/stgin.meta.

LSB_DATA_PROVENANCE

Syntax

LSB_DATA_PROVENANCE=Y | y

Description

Enables data provenance for LSF jobs, which allows you to trace files that are generated by LSF jobs. Data provenance information also allows you to reproduce your data results when using the same job input and steps.

You must use IBM® Spectrum Scale as the file system to support the extended attribute specification of files that is required for data provenance. These attributes are attached to the files, which allows LSF to maintain the files with the provenance data.

Run the predefined post execution script tag.sh with this environment variable in the job submission command to use data provenance for a particular job. For example, bsub -e 'LSB_DATA_PROVENANCE=y' -Ep 'tag.sh' myjob

This environment variable is automatically enabled in the esub.dprov application.

Default

Where defined

Set by esub.dprov or bsub at job submission.

LSB_DEBUG

This parameter can be set from the command line or from lsf.conf. See LSB_DEBUG in lsf.conf.

LSB_DEBUG_CMD

This parameter can be set from the command line or from lsf.conf. See LSB_DEBUG_CMD in lsf.conf.

LSB_DEBUG_MBD

This parameter can be set from the command line with badmin mbddebug or from lsf.conf.

See LSB_DEBUG_MBD in lsf.conf.

LSB_DEBUG_SBD

This parameter can be set from the command line with badmin sbddebug or from lsf.conf.

See LSB_DEBUG_SBD in lsf.conf.

LSB_DEBUG_SCH

This parameter can be set from the command line or from lsf.conf. See LSB_DEBUG_SCH in lsf.conf.

LSB_DEFAULT_JOBGROUP

Syntax

LSB_DEFAULT_JOBGROUP=job_group_name

Description

The name of the default job group.

When you submit a job to LSF without explicitly specifying a job group, LSF associates the job with the specified job group. LSB_DEFAULT_JOBGROUP overrides the setting of DEFAULT_JOBGROUP in lsb.params. The bsub -g job_group_name option overrides both LSB_DEFAULT_JOBGROUP and DEFAULT_JOBGROUP.

If you submit a job without the -g option of bsub, but you definedLSB_DEFAULT_JOBGROUP, then the job belongs to the job group specified in LSB_DEFAULT_JOBGROUP.

Job group names must follow this format:

Job group names must start with a slash character (/). For example, LSB_DEFAULT_JOBGROUP=/A/B/C is correct, but LSB_DEFAULT_JOBGROUP=A/B/C is not correct.
Job group names cannot end with a slash character (/). For example, LSB_DEFAULT_JOBGROUP=/A/ is not correct.
Job group names cannot contain more than one slash character (/) in a row. For example, job group names like LSB_DEFAULT_JOBGROUP=/A//B or LSB_DEFAULT_JOBGROUP=A////B are not correct.
Job group names cannot contain spaces. For example, LSB_DEFAULT_JOBGROUP=/A/B C/D is not correct.
Project names and user names used for macro substitution with %p and %u cannot start or end with slash character (/).
Project names and user names used for macro substitution with %p and %u cannot contain spaces or more than one slash character (/) in a row.
Project names or user names containing slash character (/) will create separate job groups. For example, if the project name is canada/projects, LSB_DEFAULT_JOBGROUP=/%p results in a job group hierarchy /canada/projects.

Where defined

From the command line

Example

LSB_DEFAULT_JOBGROUP=/canada/projects

Default

Not defined

LSB_DEFAULTPROJECT

Syntax

LSB_DEFAULTPROJECT=project_name

Description

The name of the project to which resources consumed by a job will be charged.

Default

Not defined

Notes

Project names can be up to 511 characters long.

If the LSF administrator defines a default project in the lsb.params configuration file, the system uses this as the default project. You can change the default project by setting LSB_DEFAULTPROJECT or by specifying a project name with the -P option of bsub.

If you submit a job without the -P option of bsub, but you defined LSB_DEFAULTPROJECT, then the job belongs to the project specified in LSB_DEFAULTPROJECT.

If you submit a job with the -P option of bsub, the job belongs to the project specified through the -P option.

Where defined

From the command line, or through the -P option of bsub

Example

LSB_DEFAULTPROJECT=engineering

LSB_DEFAULTQUEUE

Syntax

LSB_DEFAULTQUEUE=queue_name

Description

Defines the default LSF queue.

Default

mbatchd decides which is the default queue. You can override the default by defining LSB_DEFAULTQUEUE.

Notes

If the LSF administrator defines a default queue in the lsb.params configuration file, then the system uses this as the default queue. Provided you have permission, you can change the default queue by setting LSB_DEFAULTQUEUE to a valid queue (see bqueues for a list of valid queues).

Where defined

From the command line

LSB_DEFAULT_USERGROUP

Syntax

LSB_DEFAULT_USERGROUP=user_group_name

Description

Defines the default user group, if not specified at bsub submission time.

The user group serves as the charge account in LSF. Specify a user group using the bsub -G option. LSF provides the LSB_DEFAULT_USERGROUP environment variable to convey a user group by default at bsub submission time. If LSB_DEFAULT_USERGROUP is not set in the shell where bsub is invoked, specify a user group using either the bsub -G user_group option or by adding a #BSUB -G user_group directive to the start of the job script.

Default

Not defined

Where defined

From the command line

LSB_DJOB_COMMFAIL_ACTION

Syntax

LSB_DJOB_COMMFAIL_ACTION="KILL_TASKS"

Description

Defines the action LSF should take if it detects a communication failure with one or more remote parallel or distributed tasks. If defined, LSF will try to kill all the current tasks of a parallel or distributed job associated with the communication failure. If not defined, the job RES notifies the task RES to terminate all tasks, and shut down the entire job.

Default

Terminate all tasks, and shut down the entire job

Valid values

KILL_TASKS

Where defined

Set by the system based on the value of the parameter DJOB_COMMFAIL_ACTION in lsb.applications when running bsub -app for the specified application

LSB_DJOB_ENV_SCRIPT

Syntax

LSB_DJOB_ENV_SCRIPT=script_name

Description

Defines the name of a user-defined script for setting and cleaning up the parallel or distributed job environment. This script will be executed by LSF with the argument setup before launching a parallel or distributed job, and with argument cleanup after the parallel job is finished.

The script will run as the user, and will be part of the job.

If a full path is specified, LSF will use the path name for the execution. Otherwise, LSF will look for the executable from $LSF_BINDIR.

Where defined

Set by the system to the value of the parameter DJOB_ENV_SCRIPT in lsb.applications when running bsub -app for the specified application

LSB_DJOB_HB_INTERVAL

Syntax

LSB_DJOB_HB_INTERVAL=seconds

Description

Defines the time interval between heartbeat messages sent by the remote execution tasks to the head node. If the head node does not receive a heartbeat message from one task within two intervals, LSF will take action according to how LSB_DJOB_COMMFAIL_ACTION is specified. Heartbeat message sending cannot be disabled. LSB_DJOB_HB_INTERVAL can be set as an environment variable of bsub. If defined, it will overwrite DJOB_HB_INTERVAL configuration in the application profile. If neither parameter is defined, LSF uses the default value to report heartbeat messages. The default value is calculated with the following formula:

MAX(60, number_of_execution_hosts * 0.12)

Valid values must be positive integers.

LSB_DJOB_RU_INTERVAL

Syntax

LSB_DJOB_RU_INTERVAL=seconds

Description

Defines the time interval that LSF reports parallel job resource usage (rusage) on each execution host. A value 0 seconds means disable the resource update. LSB_DJOB_RU_INTERVAL can be set as an environment variable of bsub. If defined, it overwrites the DJOB_RU_INTERVAL configuration in application profile. If neither LSB_DJOB_RU_INTERVAL nor DJOB_RU_INTERVAL is defined, LSF will use the default value to report resource usage. The default value is calculate with the following formula:

MAX(60, number_of_execution_hosts * 0.3)

Valid values are non-negative integers.

LSB_DJOB_PE_NETWORK

Description

Network resource information for IBM Parallel Environment (PE) jobs submitted with the bsub -network option, or to a queue (defined in lsb.queues) or an application profile (defined in lsb.applications) with the NETWORK_REQ parameter defined.

Where defined

Set by sbatchd before a job is dispatched.

LSB_DJOB_NUMPROC

Syntax

LSB_DJOB_NUMPROC=num

Description

The number of processors (slots) allocated to the job.

Default

Not defined

Where defined

Set by sbatchd before starting a job on the execution host.

LSB_DJOB_RANKFILE

Syntax

LSB_DJOB_RANKFILE=file_path

Description

When a job is submitted (bsub -hostfile) or modified (bmod -hostfile) with a user-specified host file , the LSB_DJOB_RANKFILE environment variable is generated from the user-specified host file. If a job is not submitted with a user-specified host file then LSB_DJOB_RANKFILE points to the same file as LSB_DJOB_HOSTFILE.

Duplicate host names are combined, along with the total number of slots for a host name and the results are used for scheduling (LSB_DJOB_HOSTFILE groups the hosts together) and for LSB_MCPU_HOSTS. LSB_MCPU_HOSTS represents the job allocation.

The esub parameter LSB_SUB4_HOST_FILE reads and modifies the value of the -hostfile option.

A host name repeated sequentially will be combined to one host name with a summary of slots but host name order will be maintained.

For example, if the user specified host file contains:

host01
host01  2
host02
host01
host02
host02
host03

then the bjobs and bhist commands will show the following allocation summary for the job:

USER-SPECIFIED HOST FILE:
HOST            SLOTS
host01            3
host02            1
host01            1
host02            2
host03            1

Default

By default, points to LSB_DJOB_HOSTFILE.

LSB_DJOB_TASK_BIND

Syntax

LSB_DJOB_TASK_BIND=Y | y | N | n

Description

For CPU and memory affinity scheduling jobs launched with the blaunch distributed application framework.

If LSB_DJOB_TASK_BIND=Y in the submission environment before submitting the job, you must use blaunch to start tasks to enable LSF to bind each task to the proper CPUs or NUMA nodes Only the CPU and memory bindings allocated to the task itself will be set in each tasks environment.

If LSB_DJOB_TASK_BIND=N, or it is not set, each task will have the same CPU or NUMA node binding on one host.

If you do not use blaunch to start tasks, and use another MPI mechanism such as IBM Platform MPI or IBM Parallel Environment, you should not set DJOB_TASK_BIND or set it to N.

Default

Where defined

Set by sbatchd before a job is dispatched.

LSB_DOCKER_IMAGE_AFFINITY

Syntax

LSB_DOCKER_IMAGE_AFFINITY=Y|y|N|n

Description

When scheduling Docker-based containerized jobs, setting this parameter to y or Y enables LSF to give preference for execution hosts that already have the requested Docker image. This reduces network bandwidth and the job start time because the execution host does not have to pull the Docker image from the repository and the job can immediately start on the execution host.

When this feature is enabled, LSF considers Docker image location information when scheduling Docker jobs. Docker image affinity interacts with host preference and order[] string requests in the following manner:

If host preference is specified, the host preference is honored first. Among hosts with the same preference level, hosts with the requested Docker image are given higher priority.
If the order[] string is specified, the hosts with the requested Docker image have a higher priority first. Among hosts that all have the requested Docker image, the order[] string is then honored.

Default

Not defined.

LSB_ECHKPNT_METHOD

This parameter can be set as an environment variable or set in lsf.conf. See LSB_ECHKPNT_METHOD in lsf.conf.

LSB_ECHKPNT_METHOD_DIR

This parameter can be set as an environment variable or set in lsf.conf. See LSB_ECHKPNT_METHOD_DIR in lsf.conf.

LSB_ECHKPNT_KEEP_OUTPUT

This parameter can be set as an environment variable or set in lsf.conf. See LSB_ECHKPNT_KEEP_OUTPUT in lsf.conf.

LSB_EFFECTIVE_RSRCREQ

Syntax

LSB_EFFECTIVE_RSRCREQ=res_req_string

Description

The job effective resource requirement. If the job level resource requirement was changed by bmod -R for a running job, the changed effective resource requirement is not available via this environment variable.

Where defined

Set by bsub at job submission.

LSB_ERESTART_USRCMD

Syntax

LSB_ERESTART_USRCMD=command

Description

Original command used to start the job.

This environment variable is set by erestart to pass the job's original start command to a custom erestart method erestart.method_name. The value of this variable is extracted from the job file of the checkpointed job.

If a job starter is defined for the queue to which the job was submitted, the job starter is also included in LSB_ERESTART_USRCMD. For example, if the job starter is /bin/sh -c "%USRCMD" in lsb.queues, and the job name is myapp -d, LLSB_ERESTART_USRCMD will be set to /bin/sh -c "myapp -d".

Where defined

Set by erestart as an environment variable before a job is restarted

LSB_EXEC_RUSAGE

Syntax

LSB_EXEC_RUSAGE="resource_name1 resource_value1 resource_name2 resource_value2..."

Description

Indicates which rusage string is satisfied to permit the job to run. This environment variable is necessary because the OR (||) operator specifies alternative rusage strings for running jobs.

Valid values

resource_value1, resource_value2,... refer to the resource values on resource_name1, resource_name2,... respectively.

Default

Not defined

Where defined

Set by LSF after reserving a resource for the job.

LSB_EXECHOSTS

Description

A list of hosts on which a batch job will run, for a cluster with LSF multicluster capability.

Where defined

Set by sbatchd

LSB_EXIT_IF_CWD_NOTEXIST

Syntax

LSB_EXIT_IF_CWD_NOTEXIST=Y|y|N|n

Description

Indicates that the job will exit if the current working directory specified by bsub -cwd or bmod -cwd is not accessible on the execution host.

Default

Not defined

Where defined

From the command line

LSB_EXIT_PRE_ABORT

Description

The queue-level or job-level pre_exec_command can exit with this value if the job is to be aborted instead of being requeued or executed

Where defined

Set by sbatchd

LSB_EXIT_REQUEUE

Syntax

LSB_EXIT_REQUEUE="exit_value1 exit_value2..."

Description

Contains a list of exit values found in the queue's REQUEUE_EXIT_VALUES parameter defined in lsb.queues.

Valid values

Any positive integer

Default

Not defined

Notes

If LSB_EXIT_REQUEUE is defined, a job will be requeued if it exits with one of the specified values.

LSB_EXIT_REQUEUE is not defined if the parameter REQUEUE_EXIT_VALUES is not defined.

Where defined

Set by the system based on the value of the parameter REQUEUE_EXIT_VALUES in lsb.queues

Example

LSB_EXIT_REQUEUE="7 31"

LSB_FRAMES

Syntax

LSB_FRAMES=start_number,end_number,step

Description

Determines the number of frames to be processed by a frame job.

Valid values

The values of start_number, end_number, and step are positive integers. Use commas to separate the values.

Default

Not defined

Notes

When the job is running, LSB_FRAMES will be set to the relative frames with the format LSB_FRAMES=start_number,end_number,step.

From the start_number, end_number, andstep, the frame job can know how many frames it will process.

Where defined

Set by sbatchd

Example

LSB_FRAMES=10,20,1

LSB_HMS_TIME_FORMAT

This parameter can be set from the command line or from lsf.conf.

See LSB_HMS_TIME_FORMAT in lsf.conf.

LSB_HOSTS

Syntax

LSB_HOSTS="host_name..."

Description

A list of hosts selected by LSF to run the job.

Notes

If a job is run on a single processor, the system sets LSB_HOSTS to the name of the host used.

For parallel jobs, the system sets LSB_HOSTS to the names of all the hosts used.

Where defined

Set by sbatchd when the job is executed. LSB_HOSTS is set only when the list of host names is less than 4096 bytes.

LSB_INTERACTIVE

Syntax

LSB_INTERACTIVE=Y

Description

Indicates an interactive job. When you submit an interactive job using bsub -I, the system sets LSB_INTERACTIVE to Y.

Valid values

LSB_INTERACTIVE=Y (if the job is interactive)

Default

Not defined (if the job is not interactive)

Where defined

Set by sbatchd

LSB_JOBAFF_REQ

Description

Affinity preferences for the job as specified by the bsub -jobaff command option.

Where defined

From the command line.

LSB_JOB_INCLUDE_POSTPROC

Syntax

LSB_JOB_INCLUDE_POSTPROC=Y|y|N|n

Description

Enables the post-execution processing of the job to be included as part of the job. For interactive jobs, the interactive job will be finished before the post-execution completes.

LSB_JOB_INCLUDE_POSTPROC in the user environment overrides the value of JOB_INCLUDE_POSTPROC in lsb.params and lsb.applications.

Default

Not defined

Where defined

From the command line.

LSB_JOBEXIT_INFO

Syntax

LSB_JOBEXIT_INFO="SIGNAL signal_value signal_name"

Description

Contains information about signal that caused a job to exit.

Applies to post-execution commands. Post-execution commands are set with POST_EXEC in lsb.queues.

When the post-execution command is run, the environment variable LSB_JOBEXIT_INFO is set if the job is signaled internally. If the job ends successfully, or the job is killed or signaled externally, LSB_JOBEXIT_INFO is not set.

Examples

LSB_JOBEXIT_INFO="SIGNAL -1 SIG_CHKPNT" LSB_JOBEXIT_INFO="SIGNAL -14 SIG_TERM_USER"
LSB_JOBEXIT_INFO="SIGNAL -23 SIG_KILL_REQUEUE"

Default

Not defined

Where defined

Set by sbatchd

LSB_JOBEXIT_STAT

Syntax

LSB_JOBEXIT_STAT=exit_status

Description

Indicates a job's exit status.

Applies to post-execution commands. Post-execution commands are set with POST_EXEC in lsb.queues.

When the post-execution command is run, the environment variable LSB_JOBEXIT_STAT is set to the exit status of the job. Refer to the man page for the wait(2) command for the format of this exit status.

The post-execution command is also run if a job is requeued because the job’s execution environment fails to be set up, or if the job exits with one of the queue's REQUEUE_EXIT_VALUES. The LSB_JOBPEND environment variable is set if the job is requeued. If the job's execution environment could not be set up, LSB_JOBEXIT_STAT is set to 0.

Valid values

Any positive integer

Where defined

Set by sbatchd

LSB_JOBFILENAME

Syntax

LSB_JOBFILENAME=file_name

Description

The path to the batch executable job file that invokes the batch job. The batch executable job file is a /bin/sh script on UNIX systems or a .BAT command script on Windows systems.

LSB_JOBGROUP

Syntax

LSB_JOBGROUP=job_group_name

Description

The name of the job group associated with the job. When a job is dispatched, if it belongs to a job group, the runtime variable LSB_JOBGROUP is defined as its group. For example, if a dispatched job belongs to job group /X, LSB_JOBGROUP=/X.

Where defined

Set during job execution based on bsub options or the default job group defined in DEFAULT_JOBGROUP in lsb.params and the LSB_DEFAULT_JOBGROUP environment variable.

Default

Not defined

LSB_JOBID

Syntax

LSB_JOBID=job_ID

Description

The job ID assigned by sbatchd. This is the ID of the job assigned by LSF, as shown by bjobs.

Valid values

Any positive integer

Where defined

Set by sbatchd, defined by mbatchd

LSB_JOBINDEX

Syntax

LSB_JOBINDEX=index

Description

Contains the job array index.

Valid values

Any integer greater than zero but less than the maximum job array size.

Notes

LSB_JOBINDEX is set when each job array element is dispatched. Its value corresponds to the job array index. LSB_JOBINDEX is set for all jobs. For non-array jobs, LSB_JOBINDEX is set to zero (0).

Where defined

Set during job execution based on bsub options.

Example

You can use LSB_JOBINDEX in a shell script to select the job command to be performed based on the job array index.

For example:

if [$LSB_JOBINDEX -eq 1]; then cmd1 fi if [$LSB_JOBINDEX -eq 2]; then cmd2 fi

LSB_JOBINDEX_STEP

Syntax

LSB_JOBINDEX_STEP=step

Description

Step at which single elements of the job array are defined.

Valid values

Any integer greater than zero but less than the maximum job array size

Default

Notes

LSB_JOBINDEX_STEP is set when a job array is dispatched. Its value corresponds to the step of the job array index. This variable is set only for job arrays.

Where defined

Set during job execution based on bsub options.

Example

The following is an example of an array where a step of 2 is used:

array[1-10:2] elements:1 3 5 7 9

If this job array is dispatched, then LSB_JOBINDEX_STEP=2.

LSB_JOBNAME

Syntax

LSB_JOBNAME=job_name

Description

The name of the job defined by the user at submission time.

Default

The job's command line

Notes

The name of a job can be specified explicitly when you submit a job. The name does not have to be unique. If you do not specify a job name, the job name defaults to the actual batch command as specified on the bsub command line.

The job name can be up to 4094 characters long for UNIX and Linux or up to 255 characters for Windows.

Where defined

Set by sbatchd

Example

When you submit a job using the -J option of bsub, for example:

% bsub -J "myjob" job

sbatchd sets LSB_JOBNAME to the job name that you specified:

LSB_JOBNAME=myjob

LSB_JOBPEND

Description

Set if the job is requeued.

Where defined

Set by sbatchd for POST_EXEC only

LSB_JOBPGIDS

Description

A list of the current process group IDs of the job.

Where defined

The process group IDs are assigned by the operating system, and LSB_JOBPGIDS is set by sbatchd.

LSB_JOBPIDS

Description

A list of the current process IDs of the job.

Where defined

The process IDs are assigned by the operating system, and LSB_JOBPIDS is set by sbatchd.

LSB_MAILSIZE

Syntax

LSB_MAILSIZE=value

Description

Gives an estimate of the size of the batch job output when the output is sent by email. It is not necessary to configure LSB_MAILSIZE_LIMIT.

LSF sets LSB_MAILSIZE to the size in kilobytes (KB) of the job output, allowing the custom mail program to intercept output that is larger than desired.

LSB_MAILSIZE is not recognized by the LSF default mail program. To prevent large job output files from interfering with your mail system, use LSB_MAILSIZE_LIMIT to explicitly set the maximum size in kilobytes of the email containing the job information.

Valid values

A positive integer: If the output is being sent by email, LSB_MAILSIZE is set to the estimated mail size in kilobytes.
-1: If the output fails or cannot be read, LSB_MAILSIZE is set to -1 and the output is sent by email using LSB_MAILPROG if specified in lsf.conf.
Not defined: If you use the -o or -e options of bsub, the output is redirected to an output file. Because the output is not sent by email in this case, LSB_MAILSIZE is not used and LSB_MAILPROG is not called.
If the -N option is used with the -o option of bsub,LSB_MAILSIZE is not set.

Where defined

Set by sbatchd when the custom mail program specified by LSB_MAILPROG in lsf.conf is called.

LSB_MAX_NUM_PROCESSORS

Syntax

LSB_MAX_NUM_PROCESSORS=integer

Description

The maximum number of processors requested when the job is submitted.

Where defined

Set during job execution based on bsub options. For example, for a job submitted with -n 2,4, the maximum number of processors requested is four.

LSB_MCPU_HOSTS

Syntax

LSB_MCPU_HOSTS="host_nameA num_processors1 host_nameB num_processors2..."

Description

Contains a list of the hosts and the number of CPUs used to run a job.

Valid values

num_processors1, num_processors2,... refers to the number of CPUs used on host_nameA, host_nameB,..., respectively

Default

Not defined

Notes

The environment variables LSB_HOSTS and LSB_MCPU_HOSTS both contain the same information, but the information is presented in different formats. LSB_MCPU_HOSTS uses a shorter format than LSB_HOSTS. As a general rule, sbatchd sets both these variables. However, for some parallel jobs, LSB_HOSTS is not set.

For parallel jobs, several CPUs are used, and the length of LSB_HOSTS can become very long. sbatchd needs to spend a lot of time parsing the string. If the size of LSB_HOSTS exceeds 4096 bytes, LSB_HOSTS is ignored, and sbatchd sets only LSB_MCPU_HOSTS.

To verify the hosts and CPUs used for your dispatched job, check the value of LSB_HOSTS for single CPU jobs, and check the value of LSB_MCPU_HOSTS for parallel jobs.

Where defined

Set by sbatchd before starting a job on the execution host

Example

When the you submit a job with the -m and -n options of bsub, for example,

% bsub -m "hostA hostB" -n 6 job

sbatchd sets the environment variables LSB_HOSTS and LSB_MCPU_HOSTS as follows:

LSB_HOSTS= "hostA hostA hostA hostB hostB hostB"
LSB_MCPU_HOSTS="hostA 3 hostB 3"

Both variables are set in order to maintain compatibility with earlier versions.

LSB_NQS_PORT

This parameter can be defined in lsf.conf or in the services database such as /etc/services.

See LSB_NUM_NIOS_CALLBACK_THREADS in lsf.conf for more details.

LSB_NTRIES

Syntax

LSB_NTRIES=integer

Description

The number of times that LSF libraries attempt to contact mbatchd or perform a concurrent jobs query.

For example, if this parameter is not defined, when you type bjobs, LSF keeps displaying batch system not responding if mbatchd cannot be contacted or if the number of pending jobs exceeds MAX_PEND_JOBS specified in lsb.params or lsb.users.

If this parameter is set to a value, LSF only attempts to contact mbatchd the defined number of times and then quits. LSF will wait for a period of time equal to SUB_TRY_INTERVAL specified in lsb.params before attempting to contact mbatchd again.

Valid values

Any positive integer

Default

INFINIT_INT. Any value rather than a positive integer will be treated as INFINIT_INT. (The default is to continue the attempts to contact mbatchd).

LSB_OLD_JOBID

Syntax

LSB_OLD_JOBID=job_ID

Description

The job ID of a job at the time it was checkpointed.

When a job is restarted, it is assigned a new job ID and LSB_JOBID is replaced with the new job ID. LSB_OLD_JOBID identifies the original ID of a job before it is restarted.

Valid values

Any positive integer

Where defined

Set by sbatchd, defined by mbatchd

LSB_OUTDIR

String containing the full path to the output directory specified by the bsub -outdir option when a job is submitted. If an output directory wasn't specified, this variable contains the path to the job submission directory.

The LSF data management bstage out command uses this directory as the default destination for jobs being transferred out.

LSB_OUTPUT_TARGETFAILED

Syntax

LSB_OUTPUT_TARGETFAILED=Y

Description

Indicates that LSF cannot access the output file specified for a job submitted the bsub -o option.

Valid values

Set to Y if the output file cannot be accessed; otherwise, it is not defined.

Where defined

Set by sbatchd during job execution

LSB_PROJECT_NAME

Syntax

LSB_PROJECT_NAME=project_name

Description

To enable the use of project names for accounting purposes inside third party tools that launch jobs under LSF using environment variables, the environment variable, LSB_PROJECT_NAME is used.

When bsub -P is used to specify a project name, LSB_PROJECT_NAME is set with the project name.

Default

Not defined

Notes

Project names can be up to 511 characters long.

There are three ways to set the project name for a job. The order of priority for each is:

If you submit a job with the -P option of bsub, the job belongs to the project specified through the -P option (sets LSB_PROJECT_NAME).
If you submit a job without the -P option of bsub, but you defined LSB_DEFAULTPROJECT, then the job belongs to the project specified in LSB_DEFAULTPROJECT.
If the LSF administrator defines DEFAULT_PROJECT in the lsb.params configuration file, the system uses this as the default project name (unless LSB_DEFAULTPROJECT is set or bsub -P is used).

Where defined

From the command line, or through the -P option of bsub

LSB_QUEUE

Syntax

LSB_QUEUE=queue_name

Description

The name of the queue from which the job is dispatched.

Where defined

Set by sbatchd

LSB_RANK_HOSTFILE

Syntax

LSB_RANK_HOSTFILE=file_path

Description

Path to the OpenMPI host rank file specified by the DJOB_ENV_SCRIPT option in lsb.applications.

If DJOB_ENV_SCRIPT=openmpi_rankfile.sh is set in lsb.applications, LSF creates a host rank file and sets the environment variable LSB_RANK_HOSTFILE.

On the first execution host, LSF sbatchd will create a host rank file per job under the same location as $LSB_DJOB_HOSTFILE.

Default

Not defined

Where defined

Set during job execution.

LSB_REMOTEINDEX

Syntax

LSB_REMOTEINDEX=index

Description

The job array index of a remote LSF multicluster capability job. LSB_REMOTEINDEX is set only if the job is an element of a job array.

Valid values

Any integer greater than zero, but less than the maximum job array size

Where defined

Set by sbatchd

LSB_REMOTEJID

Syntax

LSB_REMOTEJID=job_ID

Description

The job ID of a remote LSF multicluster capability job.

Where defined

Set by sbatchd, defined by mbatchd

LSB_JOB_REPORT_MAIL

Syntax

LSB_JOB_REPORT_MAIL=Y | N | ERROR

Description

End users can set the value for this environment variable to N in the job submission environment to disable email notification for only that particular job and not email notification for all jobs. In this case, there is no need to restart sbatchd. Mail generated from mbatchd is sent as usual. If LSB_JOB_REPORT_MAIL=Y, job-related email is sent.

End users can set the value for this environment variable to ERROR in the job submission environment to send mail only when the job exits (that is, when the job is under Exit status). This parameter only affects email sent by sbatchd. This ensures that an email notification is only sent on a job error.

Where defined

From the command line.

LSB_RESTART

Syntax

LSB_RESTART=Y

Description

Indicates that a job has been restarted or migrated.

Valid values

Set to Y if a checkpointable job has been restarted or migrated; otherwise, it is undefined.

Notes

If a checkpointable job is submitted with the -k option of the bsub command, the LSB_RESTART variable is set to Y when the job is restarted. If the bmig command is used to migrate a job, the LSB_RESTART variable is set to Y when the migrated job is restarted.

If the job is not a restarted job, then the LSB_RESTART variable is not set.

Where defined

Set by sbatchd during job execution

LSB_RESTART_PGID

Syntax

LSB_RESTART_PGID=pgid

Description

The process group ID of the checkpointed job when the job is restarted.

Notes

When a checkpointed job is restarted, the operating system assigns a new group process ID to the job. LSF sets LSB_RESTART_PGID to the new group process ID.

Where defined

Set during restart of a checkpointed job.

LSB_RESTART_PID

Syntax

LSB_RESTART_PID=pid

Description

The process ID of the checkpointed job when the job is restarted.

Notes

When a checkpointed job is restarted, the operating system assigns a new process ID to the job. LSF sets LSB_RESTART_PID to the new process ID.

Where defined

Defined during restart of a checkpointed job

LSB_RTASK_GONE_ACTION

Syntax

LSB_RTASK_GONE_ACTION=task_action ...

Description

Defines the actions LSF should take if it detects that a remote task of a parallel job is gone. Where task_action is:

IGNORE_TASKCRASH: A remote task crashes. The job RES does nothing.
KILLJOB_TASKDONE: A remote task exits with zero value. The job RES notifies the task RES to terminate all tasks in the job.
KILLJOB_TASKEXIT: A remote task exits with non-zero value. The job RES notifies the task RES to terminate all tasks in the job.

Where defined

Set by the system based on the value of the parameter RTASK_GONE_ACTION in lsb.applications when running bsub -app for the specified application.

LSB_SUB_ADDITIONAL

Usage

LSB_SUB_ADDTIONAL=esub executables

Description

String that contains the application name or names of the esub executables requested by the user. This is the only option that an esub cannot change or add at job submission.

Default

Not defined

Where defined

Set by LSF on the submission host before running esub.

LSB_SUB_CLUSTER

Description

Name of submission cluster (for a cluster with LSF multicluster capability only)

Where defined

Set on the submission environment and passed to the execution cluster environment. The parameter will only be valid in an LSF multicluster capability environment. For jobs on a local cluster, the parameter is not set when using any daemon wrappers such as job starter, post, pre, or eexec scripts.

LSB_SUB_COMMANDNAME

Description

The job command line.

If set, enables esub to use the variable LSB_SUB_COMMAND_LINE in the esub job parameter file specified by the $LSB_SUB_PARM_FILE environment variable. The LSB_SUB_COMMAND_LINE variable carries the value of the bsub command argument, and is used when esub runs.

Where defined

Set by esub before a job is submitted.

LSB_SUB_COMMAND_LINE

Description

The job command line.

The job command line can be up to 4094 characters long for UNIX and Linux or up to 255 characters for Windows.

Where defined

Set by esub before a job is submitted.

LSB_SUB_EXTSCHED_PARAM

Description

Value of external scheduling options specified by bsub -extsched, or queue-level MANDATORY_EXTSCHED or DEFAULT_EXTSCHED.

Where defined

Set by esub before a job is submitted.

LSB_SUB_JOB_ACTION_WARNING_TIME

Description

Value of job warning time period specified by bsub -wt.

Where defined

Set by esub before a job is submitted.

LSB_SUB_JOB_ERR

Syntax

LSB_SUB_JOB_ERR=lsb_err_number

Description

The error number for an externally-submitted job defined by mbatchd if the job submission failed. This is available to the external post-submission scripts (epsub) to determine the reason for the job submission failure.

If the job submission is successful, this value is LSB_NO_ERROR (or 0).

Valid values

Any non-negative integer.

A value of LSB_NO_ERROR (or 0) indicates that the job submission is successful.

Where defined

Set by bsub or bmod, defined by mbatchd.

LSB_SUB_JOB_ID

Syntax

LSB_SUB_JOB_ID=job_ID | -1

Description

The job ID assigned by mbatchd. This is available to the external post-submission scripts (epsub).

Valid values

Any positive integer, or -1 to indicate that no job ID was received from mbatchd.

Where defined

Set by bsub or bmod, defined by mbatchd.

LSB_SUB_JOB_QUEUE

Syntax

LSB_SUB_JOB_QUEUE=queue_name

Description

The name of the final queue from which the job is dispatched, which includes any queue modifications made by esub. This is available to the external post-submission scripts (epsub).

Where defined

Set by bsub or bmod, defined by mbatchd.

LSB_SUB_JOB_WARNING_ACTION

Description

Value of job warning action specified by bsub -wa.

Where defined

Set by esub before a job is submitted.

LSB_SUB_MEM_SWAP_HOST_LIMIT

Syntax

LSB_SUB_MEM_SWAP_HOST_LIMIT=Y | SUB_RESET

Description

Controls per-job host-based memory and swap limit enforcement on hosts that support Linux cgroups. LSB_RESOURCE_ENFORCE="memory" must be specified in lsf.conf and a memory or swap limit must be specified for the job for host-based memory and swap limit enforcement to take effect.

Where defined

Set by esub at job submission.

LSB_SUB_PARM_FILE

Syntax

LSB_SUB_PARM_FILE=file_name

Description

Points to a temporary file that LSF uses to store the bsub options entered in the command line. An esub reads this file at job submission and either accepts the values, changes the values, or rejects the job. Job submission options are stored as name-value pairs on separate lines in the format option_name=value. A typical use of this file is to control job submission options.

Where defined

Set by LSF on the submission host before running esub. Not defined when lsrun or lsgrun are used for interactive remote execution.

LSB_SUB_RES_REQ

Syntax

LSB_SUB_RES_REQ=res_req_string

Description

The job level resource requirement for the job submission. If the job level resource requirement was changed by bmod -R for a running job, the changed resource requirement is not available via this environment variable.

Where defined

Set by bsub at job submission.

LSB_SUB_USER

Syntax

LSB_SUB_USER=user_name

Description

The user name for the job submission.

Where defined

Set by bsub at job submission.

LSB_SUBK_SHOW_JOBID

This parameter can be set from the command line or from lsf.conf.

See LSB_SUBK_SHOW_JOBID in lsf.conf.

LSB_SUCCESS_EXIT_VALUES

Syntax

LSB_SUCCESS_EXIT_VALUES=[exit_code …]

Description

Specifies the exit values that indicate successful execution for applications that successfully exit with non-zero values. Use spaces to separate multiple exit codes. exit_code should be the value between 0 and 255.

User-defined LSB_SUCCESS_EXIT_VALUES overrides application profile level specification of SUCCESS_EXIT_VALUES in lsb.applications.

LSB_SUSP_REASONS

Syntax

LSB_SUSP_REASONS=integer

Description

An integer representing suspend reasons. Suspend reasons are defined in lsbatch.h.

This parameter is set when a job goes to system-suspended (SSUSP) or user-suspended status (USUSP). It indicates the exact reason why the job was suspended.

To determine the exact reason, you can test the value of LSB_SUSP_REASONS against the symbols defined in lsbatch.h.

Where defined

Set during job execution

Syntax

LSB_SUSP_SUBREASONS=integer

Description

An integer representing the load index that caused a job to be suspended.

When the suspending reason SUSP_LOAD_REASON (suspended by load) is set in LSB_SUSP_REASONS, LSB_SUSP_SUBREASONS set to one of the load index values defined in lsf.h.

Use LSB_SUSP_SUBREASONS and LSB_SUSP_SUBREASONS together in you custom job control to determine the exact load threshold that caused a job to be suspended.

Load index values are defined in lsf.h.

Load Index	Value
R15S	0
R1M	1
R15M	2
UT	3
PG	4
IO	5
LS	6
IT	7
TMP	8
SWP	9
MEM	10

Default

Not defined

Where defined

Set during job execution.

LSB_SWITCH_ABORT_VALUE

Usage

LSB_SWITCH_ABORT_VALUE=exit_code

Description

Indicates the value that eswitch should exit with if LSF is to reject the job switch request.

Notes

If eswitch exits with the value LSB_SWITCH_ABORT_VALUE, LSF will reject the job submission. For example, to reject a job in a in Bourne shell eswitch:

exit $LSB_SWITCH_ABORT_VALUE

Default

Not defined

Where defined

Set by LSF on the submission host before running eswitch.

LSB_SWITCH_MODIFY_FILE

Description

Points to the file that the eswitch uses to modify the bswitch target queue values.

When the eswitch runs, LSF checks the specified file and applies changes so that the job switches to the revised queue. The job switch request is rejected when eswitch encounters any parameters in the file defined for LSB_SWITCH_MODIFY_FILE, other than LSB_SWITCH_QUEUE.

Where defined

Set by eswitch before a job is switched.

LSB_SWITCH_PARM_FILE

Syntax

LSB_SWITCH_PARM_FILE=file_name

Description

Points to a temporary file that LSF uses to store the bswitch options entered in the command line. An eswitch reads this file at the job bswitch level and either accepts or changes the values. Job switch options are stored as name-value pairs on separate lines in the format option_name=value. A typical use of this file is to control job switch options.

Where defined

Set by LSF on the submission host before running the eswitch.

LSB_TASK_GEOMETRY

Usage

setenv LSB_TASK_GEOMETRY "{(task_ID,...) ...}"

Description

Specifies task geometry for your jobs. LSB_TASK_GEOMETRY overrides any process group or command file placement options. This environment variable is checked for all parallel jobs. If set, users submit a parallel job (a job that requests more than 1 slot) and LSF attempts to shape LSB_MCPU_HOSTS accordingly.

LSB_TASK_GEOMETRY replaces LSB_PJL_TASK_GEOMETRY, which is kept for compatibility with earlier versions

Where defined

From the command line.

LSB_UNIT_FOR_JOBS_DISPLAY

This parameter can be set from the command line or from lsf.conf.

See LSB_UNIT_FOR_JOBS_DISPLAY in lsf.conf.

LSB_UNIXGROUP

Description

Specifies the UNIX user group of the submitting user.

Notes

This variable is useful if you want pre-execution or post-execution processing to use the user group of the user who submitted the job, and not sys(1).

Where defined

Set during job execution

LSB_USER_BIND_CPU_LIST

The binding requested at job submission takes effect when LSF_BIND_JOB=USER_CPU_LIST in lsf.conf or BIND_JOB=USER_CPU_LIST in an application profile in lsb.applications. LSF makes sure that the value is in the correct format, but does not check that the value is valid for the execution hosts.

The correct format is a list which may contain multiple items, separated by comma, and ranges. For example: 0,5,7,9-11.

LSB_USER_BIND_JOB

The binding requested at job submission takes effect when LSF_BIND_JOB=USER in lsf.conf or BIND_JOB=USER in an application profile in lsb.applications. This value must be one of Y, N, NONE, BALANCE, PACK, or ANY. Any other value is treated as ANY.

LSF_AC_JOB_NOTIFICATION

Syntax

LSF_AC_JOB_NOTIFICATION="[exit ][done][start][suspend]"

Description

Requests that the user be notified when the job reaches any of the specified states.

Use a space to separate multiple job states. The notification request is logged in JOB_NEW events.

Where defined

Set before job submission.

LSF_CMD_LOGDIR

This parameter can be set from the command line or from lsf.conf.

See LSF_CMD_LOGDIR in lsf.conf.

LSF_DEBUG_CMD

This parameter can be set from the command line or from lsf.conf.

See LSF_DEBUG_CMD in lsf.conf.

LSF_DEBUG_LIM

This parameter can be set from the command line or from lsf.conf.

See LSF_DEBUG_LIM in lsf.conf.

LSF_DEBUG_RES

This parameter can be set from the command line or from lsf.conf.

See LSF_DEBUG_RES in lsf.conf.

LSF_EAUTH_AUX_DATA

Syntax

LSF_EAUTH_AUX_DATA=path/file_name

Description

Used in conjunction with LSF daemon authentication, specifies the full path to the temporary file on the local file system that stores auxiliary authentication information (such as credentials required by a remote host for use during job execution). Provides a way for eauth -c, mbatchd, and sbatchd to communicate the location of auxiliary authentication data. Set internally by the LSF libraries in the context of eauth.

For Kerberos authentication, used for forwarding credentials to the execution host.

LSF_EAUTH_AUX_PASS

Syntax

LSF_EAUTH_AUX_PASS=yes

Description

Enables forwarding of credentials from a submission host to an execution host when daemon authentication is enabled. LSF_EAUTH_AUX_PASS=yes indicates that a credential can be added to the execution context of a job. Set to yes by bsub during job submission or by bmod during job modification so that eauth -c can forward credentials.

LSF_EAUTH_CLIENT

Syntax

LSF_EAUTH_CLIENT=mbatchd|sbatchd|pam|res|user

Description

Used with LSF daemon authentication, specifies the LSF daemon, command, or user that invokes eauth -c. Used when writing a customized eauth executable to set the context for the call to eauth. Set internally by the LSF libraries or by the LSF daemon, command, or user calling eauth -c.

LSF_EAUTH_SERVER

Syntax

LSF_EAUTH_SERVER=mbatchd|sbatchd|pam|res|user

Description

Used with LSF daemon authentication, specifies the daemon that invokes eauth -s. Used when writing a customized eauth executable to set the context for the call to eauth. Set internally by the LSF libraries or by the LSF daemon calling eauth -s.

LSF_EAUTH_UID

Syntax

LSF_EAUTH_UID=user_ID

Description

Specifies the user account under which eauth -s runs. Set by the LSF daemon that executes eauth.

LSF_EXECUTE_DOMAIN

Syntax

LSF_EXECUTE_DOMAIN=domain_name setenv LSF_EXECUTE_DOMAIN domain_name

Description

If UNIX or Windows user account mapping is enabled, specifies the preferred Windows execution domain for a job submitted by a UNIX user. The execution domain must be one of the domains listed in LSF_USER_DOMAIN.

LSF_EXECUTE_DOMAIN is defined in the user environment (.cshrc or .profile) or from the command line. Specify only one domain.

Use this parameter in conjunction with the bsub, lsrun, and lsgrun commands to bypass the order of the domains listed in LSF_USER_DOMAIN and run the job using the specified domain. If you do not have a Windows user account in the execution domain, LSF tries to run the job using one of the other domains defined by LSF_USER_DOMAIN. Once you submit a job with an execution domain defined, you cannot change the execution domain for that particular job.

LSF_INTERACTIVE_STDERR

This parameter can be defined in lsf.conf.

See LSF_INTERACTIVE_STDERR in lsf.conf for more details.

LSF_INVOKE_CMD

Syntax

LSF_INVOKE_CMD=invoking_command_name

Description

Indicates the name of the last LSF command that invoked an external executable (for example, esub or eexec).

External executables get called by different LSF commands, such as bsub, bmod, or lsrun.

Default

Not defined

Where defined

Set internally within by LSF.

LSF_JOB_STARTER

Syntax

LSF_JOB_STARTER=binary

Description

Specifies an executable program that has the actual job as an argument.

Default

Not defined

Interactive Jobs

If you want to run an interactive job that requires some preliminary setup, LSF provides a job starter function at the command level. A command-level job starter allows you to specify an executable file that will run prior to the actual job, doing any necessary setup and running the job when the setup is complete.

If LSF_JOB_STARTER is properly defined, RES will invoke the job starter (rather than the job itself), supplying your commands as arguments.

Batch Jobs

A job starter can also be defined at the queue level using the LSF_JOB_STARTER parameter, although this can only be done by the LSF administrator.

Where defined

From the command line

Example: UNIX

The job starter is invoked from within a Bourne shell, making the command-line equivalent:

/bin/sh -c "$LSF_JOB_STARTER command [argument...]"

where command [argument...] are the command line arguments you specified in lsrun or lsgrun.

If you define LSF_JOB_STARTER as follows:

setenv LSF_JOB_STARTER "/bin/csh -c"

and run a simple C-shell job:

lsrun "&apos;a.out; echo hi&apos;"

The following will be invoked to correctly start the job:

/bin/sh -c "/bin/csh -c &apos;a.out; echo hi&apos;"

Example: Windows

RES runs the job starter, passing it your commands as arguments:

LSF_JOB_STARTER command [argument...]

If you define LSF_JOB_STARTER as follows:

set LSF_JOB_STARTER=C:\cmd.exe /C

and run a simple DOS shell job:

C:\> lsrun dir /p

then the following will be invoked to correctly start the job:

C:\cmd.exe /C dir /p

LSF_LD_LIBRARY_PATH

Description

When LSF_LD_SECURITY=Y in lsf.conf, contains the value of the LD_LIBRARY_PATH environment variable, which is removed from the job environment during job initialization to ensure enhanced security against users obtaining root privileges. LSF_LD_LIBRARY_PATH allows the LD_LIBRARY_PATH environment variable to be put back before the job runs.

Where defined

For jobs submitted using bsub -Is or bsub -Ip only.

LSF_LD_PRELOAD

Description

When LSF_LD_SECURITY=Y in lsf.conf, contains the value of the LD_PRELOAD environment variable, which is removed from the job environment during job initialization to ensure enhanced security against users obtaining root privileges. LSF_LD_PRELOAD allows the LD_PRELOAD environment variable to be put back before the job runs.

Where defined

For jobs submitted using bsub -Is or bsub -Ip only.

LSF_LIM_API_NTRIES

Syntax

LSF_LIM_API_NTRIES=integer

Description

Defines the number of times LSF commands will retry to communicate with the LIM API when LIM is not available. LSF_LIM_API_NTRIES is ignored by LSF and EGO daemons and EGO commands. The LSF_LIM_API_NTRIES environment variable overrides the value of LSF_LIM_API_NTRIES in lsf.conf.

Valid values

1 to 65535

Where defined

From the command line or from lsf.conf

Default

Not defined. If not defined in lsf.conf. LIM API exits without retrying.

LSF_LIM_DEBUG

This parameter can be set from the command line or from lsf.conf.

See LSF_LIM_DEBUG in lsf.conf.

LSF_LOGDIR

This parameter can be set from the command line or from lsf.conf.

See LSF_LOGDIR in lsf.conf.

LSF_LSLOAD_FORMAT

This parameter can be set from the command line or from lsf.conf.

See LSF_LSLOAD_FORMAT in lsf.conf.

LSF_LSHOSTS_FORMAT

This parameter can be set from the command line or from lsf.conf.

See LSF_LSHOSTS_FORMAT in lsf.conf.

LSF_MASTER

Description

Set by the LIM to identify the management host. The value is Y on the management host and N on all other hosts. An elim executable can use this parameter to check the host on which the elim is currently running.

Used when the external load indices feature is enabled.

When defined

Set by the LIM when it starts the parent external load information manager (MELIM).

LSF_NIOS_DEBUG

This parameter can be set from the command line or from lsf.conf.

See LSF_NIOS_DEBUG in lsf.conf.

LSF_NIOS_ERR_LOGDIR

This parameter can be set from the command line or from lsf.conf.

See LSF_NIOS_ERR_LOGDIR in lsf.conf.

LSF_NIOS_DIE_CMD

Syntax

LSF_NIOS_DIE_CMD=command

Description

If set, the command defined by the LSF_NIOS_DIE_CMD environment variable is executed for a pending job before NIOS exits with a failure.

You can use the %J special character to specify the command. %J is replaced by the job ID.

Default

bkill %J

Where defined

From the command line.

LSF_NIOS_IGNORE_SIGWINDOW

Syntax

LSF_NIOS_IGNORE_SIGWINDOW=any_value

Description

If defined, the NIOS will ignore the SIGWINDOW signal.

Default

Not defined

Notes

When the signal SIGWINDOW is defined, some tasks appear to die when they receive the SIGWINDOW while inputting and outputting. By defining LSF_NIOS_IGNORE_SIGWINDOW, these tasks are given the chance to ignore the signal.

Where defined

From the command line.

LSF_NIOS_PEND_TIMEOUT

Syntax

LSF_NIOS_PEND_TIMEOUT=minutes

Description

Applies only to interactive batch jobs.

Maximum amount of time that an interactive batch job can remain pending.

If this parameter is defined, and an interactive batch job is pending for longer than the specified time, LSF executes the command as specified by the LSF_NIOS_DIE_CMD environment variable and returns the output of that command.

Valid values

Any integer greater than zero.

Default

Not defined

LSF_NIOS_PORT_RANGE

Syntax

LSF_NIOS_PORT_RANGE=min_port_number-max_port_number

Description

Defines a range of listening ports for NIOS to use.

Example

LSF_NIOS_PORT_RANGE=5000-6000

Default

Not defined. LSF randomly assigns a NIOS port number.

LSF_RESOURCES

Syntax

LSF_RESOURCES=dynamic_external_resource_name...

Description

Space-separated list of dynamic external resources. When the LIM starts a parent external load information manager (MELIM) on a host, the LIM checks the resource mapping defined in the ResourceMap section of lsf.cluster.cluster_name. Based on the mapping (default, all, or a host list), the LIM sets LSF_RESOURCES to the list of resources expected on the host and passes the information to the MELIM.

Used when the external load indices feature is enabled.

When defined

Set by the MELIM on the host when the MELMELIMIM invokes the elim executable.

LSF_SUBMIT_TO_CLUSTER

Syntax

LSF_SUBMIT_TO_CLUSTER=cluster_name

Description

For forwarded jobs when using the LSF multicluster capability. Defines the submission cluster, and submits jobs to this submission cluster instead of the local cluster. This enables forwarded jobs to submit jobs back to the submission cluster.

Where defined

From the forwarded job submission script.

Default

Not defined. The forwarded job script submits jobs to the local cluster.

LSF_TS_LOGON_TIME

Syntax

LSF_TS_LOGON_TIME=milliseconds

Description

Specifies the time to create a Windows terminal service session. Configure LSF_TS_LOGON_TIME according to the load on your network environment.

The default, 30000 milliseconds, is suitable for most environments. If you set LSF_TS_LOGON_TIME too small, LSF tries multiple times before it succeeds in making a terminal service session with the terminal service server, which can cause the job wait a long time before it runs. For a congested network, set LSF_TS_LOGON_TIME=1000000.

Where defined

From the command line.

Default

30000 milliseconds

LSF_USE_HOSTEQUIV

Syntax

LSF_USE_HOSTEQUIV=y|Y

Description

Used for authentication purposes. If LSF_USE_HOSTEQUIV is defined, RES and mbatchd call the ruserok(3) function to decide if a user is allowed to run remote jobs. LSF trusts all hosts configured in the LSF cluster that are defined in hosts.equiv, or in .rhosts in the user’s home directory.

The ruserok(3) function checks in the /etc/hosts.equiv file and the user's $HOME/.rhosts file to decide if the user has permission to execute remote jobs.

If LSF_USE_HOSTEQUIV is not defined, all normal users in the cluster can execute remote jobs on any host.

If LSF_ROOT_REX is set, root can also execute remote jobs with the same permission test as for normal users.

Default

Not defined

LSF_USER_DOMAIN

Syntax

LSF_USER_DOMAIN=domain_name | .

Description

Set during LSF installation or setup. If you modify this parameter in an existing cluster, you probably have to modify passwords and configuration files also.

Windows or mixed UNIX and Windows clusters only.

Enables default user mapping, and specifies the LSF user domain. The period (.) specifies local accounts, not domain accounts.

A user name specified without a domain is interpreted (on a Windows host) as belonging to the LSF user domain
A user name specified with the domain name of the LSF user domain is not valid
In a mixed cluster, this parameter defines a two-way, one-to-one user map between UNIX user accounts and Windows user accounts belonging to the specified domain, as long as the accounts have the same user name. This means jobs submitted by the Windows user account can run on a UNIX host, and jobs submitted by the UNIX account can run on any Windows host that is available to the Windows user account.

If this parameter is not defined, the default user mapping is not enabled. You can still configure user mapping at the user or system level. User account mapping is required to run cross-platform jobs in a UNIX and Windows mixed cluster.

Where defined

lsf.conf

Default

If you upgrade from LSF 4.0.1 or earlier, the default is the existing LSF user domain.
For a new, Windows-only cluster, this parameter is not defined (no LSF user domain, no default user mapping).
For a new, mixed UNIX and Windows cluster, the default is the domain that the Windows installation account belongs to. This can be modified during LSF installation.

NOCHECKVIEW_POSTEXEC

LSF integration for Rational ClearCase. Setting the NOCHECKVIEW_POSTEXEC environment variable to a value of 1 configures the daemon wrapper to skip running the checkView function during the post-execution phase of a job.

Syntax

NOCHECKVIEW_POSTEXEC=1

Description

Use one of the following shell commands to set the NOCHECKVIEW_POSTEXEC environment variable:

In csh or tcsh:
```
setenv NOCHECKVIEW_POSTEXEC 1
```
In sh, ksh, or bash:
```
export NOCHECKVIEW_POSTEXEC=1
```

OMP_NUM_THREADS

Syntax

OMP_NUM_THREADS=num_cpus

Description

LSF will set the environment variable OMP_NUM_THREADS to the total number of CPUs allocated to each task if the job requests more than one per task (that is, the job is a multithreaded job).

If you set OMP_NUM_THREADS before submitting your job, LSF does not change it, but passes it to each task.

If OMP_NUM_THREADS is not set in the job submission environment, and the job has an affinity resource request, and number_pu * number_sub_tasks is greater than one, then OMP_NUM_THREADS is set to number_pu * number_sub_tasks. If number_pu * number_sub_tasks is less than or equal to one, then OMP_NUM_THREADS is not set.

For example, for a job submitted with bsub -n 2 -R "affinity[core(3)*4]", OMP_NUM_THREADS will be set to 12 (3 multiplied by 4)

If the job has no affinity resource request, the openmp esub is specified (bsub -a openmp), and OMP_NUM_THREADS is set according to the span[] resource requirements option:

If -R "span[hosts=x]", then OMP_NUM_THREADS is set
If -R "span[ptile=y]", then OMP_NUM_THREADS is not set

For example, for a job submitted with bsub -a openmp -n 4 -R "span[ptile=2]", OMP_NUM_THREADS will be set to 2.

If the job does not specify the openmpesub, the OMP_NUM_THREADS is not set.

Default

Not defined

Where defined

Set during job execution.

RM_CPUTASK`n`

Syntax

RM_CPUTASKn=cpu_list

Description

The RM_CPUTASKn environment variable is the IBM Parallel Environment equivalent of LSB_BIND_CPU_LIST. Each RM_CPUTASK represents the CPU allocation for one task, as opposed to having a single variable which amalgamates them all. For example, a job with 6 tasks running on 4 hosts could have the following values for RM_CPUTASK (the example corresponds to the tasks specified in the example for LSB_AFFINITY_HOSTFILE):

On Host1:

RM_CPUTASK1=0,1,2,3
RM_CPUTASK2=4,5,6,7

On Host2:

RM_CPUTASK1=0,1,2,3
RM_CPUTASK2=4,5,6,7

On Host3:

RM_CPUTASK1=0,1,2,3

On Host4:

RM_CPUTASK1=0,1,2,3

Default

Not defined

Where defined

Set during job execution.

RM_MEM_AFFINITY

Syntax

RM_MEM_AFFINITY=Yes | No

Description

The RM_MEM_AFFINITY environment variable informs the IBM Parallel Environment memory binding policy. If a local-only policy (membind=localonly affinity resource requirement parameter) has been set, RM_MEM_AFFINITY will be set to Yes. If a local preference policy (membind=localprefer affinity resource requirement parameter) has been set, RM_MEM_AFFINITY will be set to No.

Note: IBM Parallel Environment does not distinguish between a local-prefer memory policy and no policy at all.

Default

Not defined

Where defined

Set during job execution.

TASKMAN_EXEC_RUSAGE

Syntax

TASKMAN_EXEC_RUSAGE="resource_name1 resource_value1 resource_name2 resource_value2..."

Description

Indicates to taskman which rusage string is satisfied to permit the task to run. This environment variable is necessary because the OR (||) operator specifies alternative rusage strings for running tasks that are launched inside LSF jobs.

Valid values

resource_value1, resource_value2,... refer to the resource values on resource_name1, resource_name2,... respectively.

Default

Not defined

Where defined

Set by LSF after reserving a resource for the job.

Environment variable reference

BSUB_BLOCK

Description

Default

Notes

Where defined

See also

BSUB_CHK_RESREQ

Syntax

Description

Default

Where defined

Example

BSUB_QUIET

Syntax

Description

Default

Where defined

Examples

BSUB_QUIET2

Syntax

Description

Default

Where defined

Example

BSUB_STDERR

Syntax

Description

Default

Where defined

CLEARCASE_DRIVE

Syntax

Description

Notes

Where defined

Examples

See also

CLEARCASE_MOUNTDIR

Syntax

Description

Default

Notes

Where defined

Example

See also

CLEARCASE_ROOT

Syntax

Description

Notes

Where defined

Example

See also

DAEMON_WRAP_DEBUG

Syntax

Description

ELIM_ABORT_VALUE

Syntax

Description

Where defined

JOB_GPU_VENDOR

Syntax

Description

Default

LS_EXEC_T

Syntax

Description

Where defined

LS_JOBPID

Description

Where defined

LS_SUBCWD

Description

Where defined

LSB_AFFINITY_HOSTFILE

Syntax

Description

Default

Where defined

LSB_BHIST_HOURS

Usage