Environment variable reference
Reference for LSF environment variables.
BSUB_BLOCK
The BSUB_BLOCK
variable is set when you submit a job for synchronous job
execution with the bsub
-K option.
Description
If set, tells NIOS that the job is running in batch mode.
Default
Not defined
Notes
Synchronous execution means you have to wait for the job to finish before you can continue.
Where defined
Set internally
See also
The -K option of bsub
BSUB_CHK_RESREQ
When the BSUB_CHK_RESREQ
variable is set, bsub checks
the syntax of the resource requirement selection string without actually submitting the job for
scheduling and dispatch.
Syntax
BSUB_CHK_RESREQ=any_value
Description
Use BSUB_CHK_RESREQ
to check the compatibility of your existing resource
requirement select strings against the strict syntax.
The bsub command only checks the select section of the resource requirement. Other sections in the resource requirement string are not checked.
Default
Not defined
Where defined
From the command line
Example
BSUB_CHK_RESREQ=1
BSUB_QUIET
Syntax
BSUB_QUIET=value
Description
Controls the printed output information about job submissions.
- If
BSUB_QUIET
is set to any value, messages are suppressed; the bsub command will not print any information about submitted jobs. - If
BSUB_QUIET
is not set, the bsub command prints the default message of Job job_ID is submitted to default queue queue_type will be printed. For example, if the most recently submitted job ID is 100, and the queue is normal, then the command prints:Job 100 is submitted to default queue normal
- If set to BSUB_QUIET=JOBID, the bsub command prints
the JOBID value in the output. For example, if the most recently submitted job
ID is 100, bsub prints:
100
- If set to BSUB_QUIET=QUEUE, the bsub command prints
the JOBID and QUEUE values in the output, with a single
space to separating the values. For example, if the most recently submitted job ID is 100, and the
queue is normal, then bsub prints:
100 normal
- If
BSUB_QUIET
is set to any other value, messages are suppressed; that is, if BSUB_QUIET=any_other_value, the bsub command will not print any information about submitted jobs. - If
BSUB_QUIET
is not set, the bsub command prints the default message of Job job_ID is submitted to default queue queue_type will be printed. For example, if the most recently submitted job ID is 100, and the queue is normal, then the command prints:Job 100 is submitted to default queue normal
Default
Not defined
Where defined
From the command line
Examples
BSUB_QUIET=JOBID
BSUB_QUIET=QUEUE
BSUB_QUIET=any_other_value
BSUB_QUIET=100
BSUB_QUIET2
Syntax
BSUB_QUIET2=any_value
Description
Suppresses the printing of information about job completion when a job is submitted with the bsub -K option.
If set, bsub will not print information about job completion to stdout. For example, when this variable is set, the message <<Job is finished>> will not be written to stdout.
If BSUB_QUIET
and BSUB_QUIET2
are both set, no job messages
will be printed to stdout.
Default
Not defined
Where defined
From the command line
Example
BSUB_QUIET2=1
BSUB_STDERR
Syntax
BSUB_STDERR=y
Description
Redirects LSF messages for bsub to stderr.
By default, when this parameter is not set, LSF messages for bsub are printed to stdout.
When this parameter is set, LSF messages for bsub are redirected to stderr.
Default
Not defined
Where defined
From the command line on UNIX. For example, in csh:
setenv BSUB_STDERR Y
From the Control Panel on Windows, as an environment variable
CLEARCASE_DRIVE
Syntax
CLEARCASE_DRIVE=drive_letter:
Description
Optional, Windows only.
Defines the virtual drive letter for a Rational ClearCase view to the drive. This is useful if you wish to map a Rational ClearCase view to a virtual drive as an alias.
If this letter is unavailable, Windows attempts to map to another drive. Therefore,
CLEARCASE_DRIVE
only defines the default drive letter to which the Rational ClearCase view is mapped, not the final selected drive letter. However,
the PATH value is automatically updated to the final drive letter if it is
different fromCLEARCASE_DRIVE
.
Notes
CLEARCASE_DRIVE
is not case sensitive.
Where defined
From the command line
Examples
CLEARCASE_DRIVE=F:
CLEARCASE_DRIVE=f:
See also
CLEARCASE_MOUNTDIR
and CLEARCASE_ROOT
CLEARCASE_MOUNTDIR
Syntax
CLEARCASE_MOUNTDIR=path
Description
Optional.
Defines the Rational ClearCase mounting directory.
Default
/vobs
Notes
CLEARCASE_MOUNTDIR
is used if any of the following conditions apply:
- A job is submitted from a UNIX environment but run in a Windows host.
- The Rational ClearCase mounting directory is not the default /vobs
Where defined
From the command line
Example
CLEARCASE_MOUNTDIR=/myvobs
See also
CLEARCASE_DRIVE
and CLEARCASE_ROOT
CLEARCASE_ROOT
Syntax
CLEARCASE_ROOT=path
Description
The path to the Rational ClearCase view.
In Windows, this path must define an absolute path starting with the default Rational ClearCase drive and ending with the view name without an ending backslash (\).
Notes
CLEARCASE_ROOT
must be defined if you want to submit a batch job from a Rational ClearCase view.
For interactive jobs, use the bsub -I command to submit the job.
Where defined
In the job starter, or from the command line
Example
CLEARCASE_ROOT=/view/myview
CLEARCASE_ROOT=F:\myview
See also
CLEARCASE_DRIVE
, CLEARCASE_MOUNTDIR
, and
LSF_JOB_STARTER
DAEMON_WRAP_DEBUG
LSF integration for Rational ClearCase. Enables daemon wrapper debugging.
Syntax
DAEMON_WRAP_DEBUG=1
Description
DAEMON_WRAP_DEBUG
is set to 1, LSF logs
the Rational ClearCase
stdout and stderr streams to
/tmp/daemons.wrap.log. Use one of the following shell commands to set the
DAEMON_WRAP_DEBUG
environment variable: - In csh or tcsh:
setenv DAEMON_WRAP_DEBUG 1
- In sh, ksh, or bash:
export DAEMON_WRAP_DEBUG=1
ELIM_ABORT_VALUE
Syntax
ELIM_ABORT_VALUE
Description
Used when writing an elim executable to test whether the elim should run on a particular host. If the host does not have or share any of the resources listed in the environment variable LSF_RESOURCES, your elim should exit with $ELIM_ABORT_VALUE.
When the MELIM finds an elim that exited with ELIM_ABORT_VALUE, the MELIM marks the elim and does not restart it on that host.
Where defined
Set by the management elim (MELIM) on the host when the MELIM invokes the elim executable
JOB_GPU_VENDOR
Syntax
JOB_GPU_VENDOR=amd | nvidia
Description
Specifies the GPU vendor type that is allowed for GPU jobs. LSF rejects any jobs that are submitted with -gpu properties that conflict with this vendor type.
When defined as amd, the GPU job only runs on AMD GPUs.
When defined as nvidia, the GPU job only runs on Nvidia GPUs.
Default
nvidia
LS_EXEC_T
Syntax
LS_EXEC_T=START | END | CHKPNT | JOB_CONTROLS
Description
Indicates execution type for a job. LS_EXEC_T
is set to:
- START or END for a job when the job begins executing or when it completes execution
- CHKPNT when the job is check-pointed
- JOB_CONTROLS when a control action is initiated
Where defined
Set by sbatchd during job execution
LS_JOBPID
Description
The process ID of the job.
Where defined
During job execution, sbatchd sets LS_JOBPID
to be the same
as the process ID assigned by the operating system.
LS_SUBCWD
Description
The current working directory (cwd) of the submission host where the remote task command was executed.
The current working directory can be up to 4094 characters long for UNIX and Linux or up to 255 characters for Windows.
- LSF looks
for the
PWD
environment variable. If it finds it, setsLS_SUBCWD
toPWD
. - If the
PWD
environment variable does not exist, LSF looks for theCWD
environment variable. If it finds CWD, setsLS_SUBCWD
toCWD
. - If the
CWD
environment variable does not exist, LSF calls the getwd() system function to retrieve the current working directory path name. LSF setsLS_SUBCWD
to the value that is returned.
Where defined
Set by sbatchd
LSB_AFFINITY_HOSTFILE
Syntax
LSB_AFFINITY_HOSTFILE=file_path
Description
Path to the NUMA CPU and memory affinity binding decision file.
On the first execution host, LSF
sbatchd will create a binding decision file per job under the same location as
$LSB_DJOB_HOSTFILE. The binding decision file has a format similar to the job
host file; that is, one task per line, where each line includes:
host_name
cpu_id_list
NUMA_node_id_list
memory_policy
.
For memory policy, a value of 1 means localonly, 2 means localprefer, as specified in the affinity resource requirement membind parameter.
Comma (,) is the only supported delimiter for the list of CPU IDs and the list of NUMA node IDs.
- Host1 and Host2 each have two tasks bound to CPUs
{0,1,2,3}
and{4,5,6,7}
, and NUMA nodes 0 and 1 respectively with a membind=localprefer policy. - Host3 and Host4 each have one task bound to CPUs
{0,1,2,3}
and NUMA node 0, again with a membind=localprefer policy.
Host1 0,1,2,3 0 2
Host1 4,5,6,7 1 2
Host2 0,1,2,3 0 2
Host2 4,5,6,7 1 2
Host3 0,1,2,3 0 2
Host4 0,1,2,3 0 2
Default
Not defined
Where defined
Set during job execution.
LSB_BHIST_HOURS
Usage
LSB_BHIST_HOURS=hours
Description
The multiple-line format of bhist considers job events from the last
LSB_BHIST_HOURS
.
Use LSB_BHIST_HOURS
with the command bhist -t to display job
events before the current time for a longer or shorter period than the default one week.
Default
Not defined
If not defined, the multiple-line format for bhist considers job events from the last week.
LSB_BIND_CPU_LIST
Syntax
LSB_BIND_CPU_LIST=cpu_list
Description
LSB_BIND_CPU_LIST
contains allocated CPUs on each host. LSF will
combine all of the allocated CPUs for all the tasks that ended up on the host (for the given job),
and set this environment variable to the entire list before launching tasks. The following example
corresponds to the tasks specified in the example for
LSB_AFFINITY_HOSTFILE
:LSB_BIND_CPU_LIST on Host1: 0,1,2,3,4,5,6,7
LSB_BIND_CPU_LIST on Host2: 0,1,2,3,4,5,6,7
LSB_BIND_CPU_LIST on Host3: 0,1,2,3
LSB_BIND_CPU_LIST on Host4: 0,1,2,3
Default
Not defined
Where defined
Set during job execution.
LSB_BIND_MEM_LIST
Syntax
LSB_BIND_MEM_LIST=memory_node_list
Description
LSB_BIND_MEM_LIST
contains allocated memory nodes on each host. If the job is
submitted with a memory affinity requirement, LSF will
combine all of the allocated NUMA node IDs for all the tasks that ended up on the host (for the
given job), and set this environment variable to the entire list before launching tasks. The
following example corresponds to the tasks specified in the example for
LSB_AFFINITY_HOSTFILE
:LSB_BIND_MEM_LIST on Host1: 0,1
LSB_BIND_MEM_LIST on Host2: 0,1
LSB_BIND_MEM_LIST on Host3: 0
LSB_BIND_MEM_LIST on Host4: 0
Default
Not defined
Where defined
Set during job execution.
LSB_BIND_MEM_POLICY
Syntax
LSB_BIND_MEM_POLICY=localprefer | localonly
Description
For jobs submitted with a NUMA memory affinity resource requirement, LSF sets
the memory binding policy in LSB_BIND_MEM_POLICY
environment variable, as specified
in the membind affinity resource requirement parameter: either
localprefer or localonly.
Default
Not defined
Where defined
Set during job execution.
LSB_BHOSTS_FORMAT
This parameter can be set from the command line or from lsf.conf.
See LSB_BHOSTS_FORMAT
in lsf.conf.
LSB_BJOBS_FORMAT
This parameter can be set from the command line or from lsf.conf.
See LSB_BJOBS_FORMAT
in lsf.conf.
LSB_BMGROUP_ALLREMOTE_EXPAND
Syntax
This parameter can be set from the command line or from lsf.conf.
See LSB_BMGROUP_ALLREMOTE_EXPAND
in lsf.conf.
LSB_BQUEUES_FORMAT
This parameter can be set from the command line or from lsf.conf.
See LSB_BQUEUES_FORMAT
in lsf.conf.
LSB_BSUB_ERR_RETRY
Syntax
LSB_BSUB_ERR_RETRY=RETRY_CNT[number] ERR_TYPE[error1 error2 ...]
Description
In some cases, jobs can benefit from being automatically retried in the case of failing for a
particular error. When specified, LSB_BSUB_ERR_RETRY
automatically retries jobs
that exit with a particular reason, up to the number of times specified by RETRY_CNT.
- BAD_XDR: Error during XDR.
- MSG_SYS: Failed to send or receive a message.
- INTERNAL: Internal library error.
The number of retries (RETRY_CNT) can be a minimum of 1 to a maximum of 50.
- Users may experience what seems like a lag during job submission while the job is retried automatically in the background.
- Users may see a job submitted more than once, with no explanation (no error is communicated to the user; the job keeps getting submitted until it succeeds or reaches its maximum retry count). In this case, the job ID also changes each time the error is retried.
Default
N
LSB_BUSERS_FORMAT
This parameter can be set from the command line or from lsf.conf.
See LSB_BUSERS_FORMAT
in lsf.conf.
LSB_CHKPNT_DIR
Syntax
LSB_CHKPNT_DIR=checkpoint_dir/job_ID
Description
The directory containing files related to the submitted checkpointable job.
Valid values
The value of checkpoint_dir is the directory you specified through the -k option of bsub when submitting the checkpointable job.
The value of job_ID is the job ID of the checkpointable job.
Where defined
Set by LSF, based on the directory you specified when submitting a checkpointable job with the -k option of bsub.
LSB_CONTAINER_IMAGE_UPDATE
Syntax
LSB_CONTAINER_IMAGE_UPDATE=Y | y
Description
If defined, LSF updates the Docker image that is specified in the job related application file before execution.
Default
If undefined, the Docker image does not update.
LSB_DATA_CACHE_TOP
Contains a string defining the location of the data management staging area relative to the compute node. The value of this variable is equivalent to STAGING_AREA in lsf.datamanager.
See also
lsf.datamanager
LSB_DATA_META_FILE
For jobs submitted with data requirements, this variable contains a string defining the location of the job's metadata file relative to the compute node. The value of this variable is equivalent to the value of STAGING_AREA/work/cluster_name/jobID/stgin.meta.
See also
LSB_OUTDIR
, the bsub -data option
LSB_DATA_PROVENANCE
Syntax
LSB_DATA_PROVENANCE=Y | y
Description
Enables data provenance for LSF jobs, which allows you to trace files that are generated by LSF jobs. Data provenance information also allows you to reproduce your data results when using the same job input and steps.
You must use IBM® Spectrum Scale as the file system to support the extended attribute specification of files that is required for data provenance. These attributes are attached to the files, which allows LSF to maintain the files with the provenance data.
Run the predefined post execution script tag.sh with this environment
variable in the job submission command to use data provenance for a particular job. For example,
bsub -e 'LSB_DATA_PROVENANCE=y' -Ep 'tag.sh' myjob
This environment variable is automatically enabled in the esub.dprov application.
Default
N
Where defined
Set by esub.dprov or bsub at job submission.
LSB_DEBUG
This parameter can be set from the command line or from lsf.conf. See
LSB_DEBUG
in lsf.conf.
LSB_DEBUG_CMD
This parameter can be set from the command line or from lsf.conf. See
LSB_DEBUG_CMD
in lsf.conf.
LSB_DEBUG_MBD
This parameter can be set from the command line with badmin mbddebug or from lsf.conf.
See LSB_DEBUG_MBD
in lsf.conf.
LSB_DEBUG_SBD
This parameter can be set from the command line with badmin sbddebug or from lsf.conf.
See LSB_DEBUG_SBD
in lsf.conf.
LSB_DEBUG_SCH
This parameter can be set from the command line or from lsf.conf. See
LSB_DEBUG_SCH
in lsf.conf.
LSB_DEFAULT_JOBGROUP
Syntax
LSB_DEFAULT_JOBGROUP=job_group_name
Description
The name of the default job group.
When you submit a job to LSF
without explicitly specifying a job group, LSF
associates the job with the specified job group. LSB_DEFAULT_JOBGROUP
overrides the
setting of DEFAULT_JOBGROUP
in lsb.params. The bsub
-g
job_group_name option overrides both LSB_DEFAULT_JOBGROUP
and
DEFAULT_JOBGROUP
.
If you submit a job without the -g option of bsub, but you
definedLSB_DEFAULT_JOBGROUP
, then the job belongs to the job group specified in
LSB_DEFAULT_JOBGROUP
.
- Job group names must start with a slash character (/). For example, LSB_DEFAULT_JOBGROUP=/A/B/C is correct, but LSB_DEFAULT_JOBGROUP=A/B/C is not correct.
- Job group names cannot end with a slash character (/). For example, LSB_DEFAULT_JOBGROUP=/A/ is not correct.
- Job group names cannot contain more than one slash character (/) in a row. For example, job group names like LSB_DEFAULT_JOBGROUP=/A//B or LSB_DEFAULT_JOBGROUP=A////B are not correct.
- Job group names cannot contain spaces. For example, LSB_DEFAULT_JOBGROUP=/A/B C/D is not correct.
- Project names and user names used for macro substitution with %p and %u cannot start or end with slash character (/).
- Project names and user names used for macro substitution with %p and %u cannot contain spaces or more than one slash character (/) in a row.
- Project names or user names containing slash character (/) will create separate job groups. For example, if the project name is canada/projects, LSB_DEFAULT_JOBGROUP=/%p results in a job group hierarchy /canada/projects.
Where defined
From the command line
Example
LSB_DEFAULT_JOBGROUP=/canada/projects
Default
Not defined
See also
DEFAULT_JOBGROUP
in lsb.params, the -g
option of bsub
LSB_DEFAULTPROJECT
Syntax
LSB_DEFAULTPROJECT=project_name
Description
The name of the project to which resources consumed by a job will be charged.
Default
Not defined
Notes
Project names can be up to 511 characters long.
If the LSF administrator defines a default project in the lsb.params configuration file, the system uses this as the default project. You can change the default project by setting LSB_DEFAULTPROJECT or by specifying a project name with the -P option of bsub.
If you submit a job without the -P option of bsub, but you defined LSB_DEFAULTPROJECT, then the job belongs to the project specified in LSB_DEFAULTPROJECT.
If you submit a job with the -P option of bsub, the job belongs to the project specified through the -P option.
Where defined
From the command line, or through the -P option of bsub
Example
LSB_DEFAULTPROJECT=engineering
See also
DEFAULT_PROJECT
in lsb.params, the -P
option of bsub
LSB_DEFAULTQUEUE
Syntax
LSB_DEFAULTQUEUE=queue_name
Description
Defines the default LSF queue.
Default
mbatchd decides which is the default queue. You can override the default by
defining LSB_DEFAULTQUEUE
.
Notes
If the LSF administrator defines a default queue in the lsb.params configuration file, then the system uses this as the default queue. Provided you have permission, you can change the default queue by setting LSB_DEFAULTQUEUE to a valid queue (see bqueues for a list of valid queues).
Where defined
From the command line
See also
DEFAULT_QUEUE
in lsb.params
LSB_DEFAULT_USERGROUP
Syntax
LSB_DEFAULT_USERGROUP=user_group_name
Description
Defines the default user group, if not specified at bsub submission time.
The user group serves as the charge account in LSF.
Specify a user group using the bsub -G option. LSF
provides the LSB_DEFAULT_USERGROUP
environment variable to convey a user group by
default at bsub submission time. If LSB_DEFAULT_USERGROUP
is not
set in the shell where bsub is invoked, specify a user group using either the
bsub -G user_group option or by adding a #BSUB
-G user_group directive to the start of the job script.
Default
Not defined
Where defined
From the command line
LSB_DJOB_COMMFAIL_ACTION
Syntax
LSB_DJOB_COMMFAIL_ACTION="KILL_TASKS"
Description
Defines the action LSF should
take if it detects a communication failure with one or more remote parallel or distributed tasks. If
defined, LSF will
try to kill all the current tasks of a parallel or distributed job associated with the communication
failure. If not defined, the job RES
notifies the task RES
to
terminate all tasks, and shut down the entire job.
Default
Terminate all tasks, and shut down the entire job
Valid values
KILL_TASKS
Where defined
Set by the system based on the value of the parameter DJOB_COMMFAIL_ACTION
in
lsb.applications when running bsub -app for the specified
application
See also
DJOB_COMMFAIL_ACTION
in lsb.applications
LSB_DJOB_ENV_SCRIPT
Syntax
LSB_DJOB_ENV_SCRIPT=script_name
Description
Defines the name of a user-defined script for setting and cleaning up the parallel or distributed job environment. This script will be executed by LSF with the argument setup before launching a parallel or distributed job, and with argument cleanup after the parallel job is finished.
The script will run as the user, and will be part of the job.
If a full path is specified, LSF will use the path name for the execution. Otherwise, LSF will look for the executable from $LSF_BINDIR.
Where defined
Set by the system to the value of the parameter DJOB_ENV_SCRIPT
in
lsb.applications when running bsub -app for the specified
application
See also
DJOB_ENV_SCRIPT
in lsb.applications
LSB_DJOB_HB_INTERVAL
Syntax
LSB_DJOB_HB_INTERVAL=seconds
Description
Defines the time interval between heartbeat messages sent by the remote execution tasks to the
head node. If the head node does not receive a heartbeat message from one task within two intervals,
LSF will take action according to how LSB_DJOB_COMMFAIL_ACTION
is specified.
Heartbeat message sending cannot be disabled. LSB_DJOB_HB_INTERVAL
can be set as an
environment variable of bsub. If defined, it will overwrite
DJOB_HB_INTERVAL
configuration in the application profile. If neither parameter is
defined, LSF uses
the default value to report heartbeat messages. The default value is calculated with the following
formula:
MAX(60, number_of_execution_hosts * 0.12)
Valid values must be positive integers.
LSB_DJOB_RU_INTERVAL
Syntax
LSB_DJOB_RU_INTERVAL=seconds
Description
Defines the time interval that LSF
reports parallel job resource usage (rusage
) on each execution host. A value 0
seconds means disable the resource update. LSB_DJOB_RU_INTERVAL
can be set as an
environment variable of bsub. If defined, it overwrites the
DJOB_RU_INTERVAL
configuration in application profile. If neither
LSB_DJOB_RU_INTERVAL
nor DJOB_RU_INTERVAL
is defined, LSF will
use the default value to report resource usage. The default value is calculate with the following
formula:
MAX(60, number_of_execution_hosts * 0.3)
Valid values are non-negative integers.
LSB_DJOB_PE_NETWORK
Description
Network resource information for IBM Parallel Environment (PE) jobs submitted with the bsub -network option, or to a queue (defined in lsb.queues) or an application profile (defined in lsb.applications) with the NETWORK_REQ parameter defined.
Where defined
Set by sbatchd before a job is dispatched.
LSB_DJOB_NUMPROC
Syntax
LSB_DJOB_NUMPROC=num
Description
The number of processors (slots) allocated to the job.
Default
Not defined
Where defined
Set by sbatchd before starting a job on the execution host.
See Also
LSB_MCPU_HOSTS
LSB_DJOB_RANKFILE
Syntax
LSB_DJOB_RANKFILE=file_path
Description
When a job is submitted (bsub -hostfile) or modified (bmod
-hostfile) with a user-specified host file , the LSB_DJOB_RANKFILE
environment variable is generated from the user-specified host file. If a job is not submitted with
a user-specified host file then LSB_DJOB_RANKFILE
points to the same file as
LSB_DJOB_HOSTFILE
.
Duplicate host names are combined, along with the total number of slots for a host name and the
results are used for scheduling (LSB_DJOB_HOSTFILE
groups the hosts together) and
for LSB_MCPU_HOSTS
. LSB_MCPU_HOSTS
represents the job
allocation.
The esub parameter LSB_SUB4_HOST_FILE
reads and modifies the
value of the -hostfile option.
A host name repeated sequentially will be combined to one host name with a summary of slots but host name order will be maintained.
host01
host01 2
host02
host01
host02
host02
host03
USER-SPECIFIED HOST FILE:
HOST SLOTS
host01 3
host02 1
host01 1
host02 2
host03 1
Default
By default, points to LSB_DJOB_HOSTFILE
.
LSB_DJOB_TASK_BIND
Syntax
LSB_DJOB_TASK_BIND=Y | y | N | n
Description
For CPU and memory affinity scheduling jobs launched with the blaunch distributed application framework.
If LSB_DJOB_TASK_BIND=Y in the submission environment before submitting the job, you must use blaunch to start tasks to enable LSF to bind each task to the proper CPUs or NUMA nodes Only the CPU and memory bindings allocated to the task itself will be set in each tasks environment.
If LSB_DJOB_TASK_BIND=N, or it is not set, each task will have the same CPU or NUMA node binding on one host.
If you do not use blaunch to start tasks, and use another MPI mechanism such as IBM Platform MPI or IBM
Parallel Environment, you should not set DJOB_TASK_BIND
or set it to N.
Default
N
Where defined
Set by sbatchd before a job is dispatched.
LSB_DOCKER_IMAGE_AFFINITY
Syntax
LSB_DOCKER_IMAGE_AFFINITY=Y|y|N|nDescription
When scheduling Docker-based containerized jobs, setting this parameter to y or Y enables LSF to give preference for execution hosts that already have the requested Docker image. This reduces network bandwidth and the job start time because the execution host does not have to pull the Docker image from the repository and the job can immediately start on the execution host.
When this feature is enabled, LSF considers Docker image location information when scheduling Docker jobs. Docker image affinity interacts with host preference and order[] string requests in the following manner:
- If host preference is specified, the host preference is honored first. Among hosts with the same preference level, hosts with the requested Docker image are given higher priority.
- If the order[] string is specified, the hosts with the requested Docker image have a higher priority first. Among hosts that all have the requested Docker image, the order[] string is then honored.
Default
Not defined.
LSB_ECHKPNT_METHOD
This parameter can be set as an environment variable or set in lsf.conf. See
LSB_ECHKPNT_METHOD
in lsf.conf.
LSB_ECHKPNT_METHOD_DIR
This parameter can be set as an environment variable or set in lsf.conf. See
LSB_ECHKPNT_METHOD_DIR
in lsf.conf.
LSB_ECHKPNT_KEEP_OUTPUT
This parameter can be set as an environment variable or set in lsf.conf. See LSB_ECHKPNT_KEEP_OUTPUT in lsf.conf.
LSB_EFFECTIVE_RSRCREQ
Syntax
LSB_EFFECTIVE_RSRCREQ=res_req_string
Description
The job effective resource requirement. If the job level resource requirement was changed by bmod -R for a running job, the changed effective resource requirement is not available via this environment variable.
Where defined
Set by bsub at job submission.
LSB_ERESTART_USRCMD
Syntax
LSB_ERESTART_USRCMD=command
Description
Original command used to start the job.
This environment variable is set by erestart to pass the job's original start command to a custom erestart method erestart.method_name. The value of this variable is extracted from the job file of the checkpointed job.
If a job starter is defined for the queue to which the job was submitted, the job starter is also
included in LSB_ERESTART_USRCMD
. For example, if the job starter is
/bin/sh -c "%USRCMD" in lsb.queues, and the job
name is myapp -d, LLSB_ERESTART_USRCMD
will be set to
/bin/sh -c "myapp -d"
.
Where defined
Set by erestart as an environment variable before a job is restarted
See also
LSB_ECHKPNT_METHOD
, erestart, and
echkpnt
LSB_EXEC_RUSAGE
Syntax
LSB_EXEC_RUSAGE="resource_name1 resource_value1 resource_name2 resource_value2..."
Description
Indicates which rusage string is satisfied to permit the job to run. This environment variable is necessary because the OR (||) operator specifies alternative rusage strings for running jobs.
Valid values
resource_value1, resource_value2,... refer to the resource values on resource_name1, resource_name2,... respectively.
Default
Not defined
Where defined
Set by LSF after reserving a resource for the job.
LSB_EXECHOSTS
Description
A list of hosts on which a batch job will run, for a cluster with LSF multicluster capability.
Where defined
Set by sbatchd
LSB_EXIT_IF_CWD_NOTEXIST
Syntax
LSB_EXIT_IF_CWD_NOTEXIST=Y|y|N|n
Description
Indicates that the job will exit if the current working directory specified by bsub -cwd or bmod -cwd is not accessible on the execution host.
Default
Not defined
Where defined
From the command line
LSB_EXIT_PRE_ABORT
Description
The queue-level or job-level pre_exec_command can exit with this value if the job is to be aborted instead of being requeued or executed
Where defined
Set by sbatchd
See also
See PRE_EXEC
in lsb.queues, or the -E
option of bsub
LSB_EXIT_REQUEUE
Syntax
LSB_EXIT_REQUEUE="exit_value1 exit_value2..."
Description
Contains a list of exit values found in the queue's REQUEUE_EXIT_VALUES
parameter defined in lsb.queues.
Valid values
Any positive integer
Default
Not defined
Notes
If LSB_EXIT_REQUEUE
is defined, a job will be requeued if it exits with one of
the specified values.
LSB_EXIT_REQUEUE
is not defined if the parameter
REQUEUE_EXIT_VALUES
is not defined.
Where defined
Set by the system based on the value of the parameter REQUEUE_EXIT_VALUES
in
lsb.queues
Example
LSB_EXIT_REQUEUE="7 31"
See also
REQUEUE_EXIT_VALUES
in lsb.queues
LSB_FRAMES
Syntax
LSB_FRAMES=start_number,end_number,step
Description
Determines the number of frames to be processed by a frame job.
Valid values
The values of start_number, end_number, and step are positive integers. Use commas to separate the values.
Default
Not defined
Notes
When the job is running, LSB_FRAMES
will be set to the relative frames with the
format
LSB_FRAMES=start_number,end_number,step.
From the start_number, end_number, and step, the frame job can know how many frames it will process.
Where defined
Set by sbatchd
Example
LSB_FRAMES=10,20,1
LSB_HMS_TIME_FORMAT
This parameter can be set from the command line or from lsf.conf.
See LSB_HMS_TIME_FORMAT
in lsf.conf.
LSB_HOSTS
Syntax
LSB_HOSTS="host_name..."
Description
A list of hosts selected by LSF to run the job.
Notes
If a job is run on a single processor, the system sets LSB_HOSTS
to the name of
the host used.
For parallel jobs, the system sets LSB_HOSTS
to the names of all the hosts
used.
Where defined
Set by sbatchd when the job is executed. LSB_HOSTS
is set
only when the list of host names is less than 4096 bytes.
See also
LSB_MCPU_HOSTS
LSB_INTERACTIVE
Syntax
LSB_INTERACTIVE=Y
Description
Indicates an interactive job. When you submit an interactive job using bsub
-I, the system sets LSB_INTERACTIVE
to Y.
Valid values
LSB_INTERACTIVE=Y (if the job is interactive)
Default
Not defined (if the job is not interactive)
Where defined
Set by sbatchd
LSB_JOBAFF_REQ
Description
Affinity preferences for the job as specified by the bsub -jobaff command option.
Where defined
From the command line.
LSB_JOB_INCLUDE_POSTPROC
Syntax
LSB_JOB_INCLUDE_POSTPROC=Y|y|N|n
Description
Enables the post-execution processing of the job to be included as part of the job. For interactive jobs, the interactive job will be finished before the post-execution completes.
LSB_JOB_INCLUDE_POSTPROC
in the user environment overrides the value of
JOB_INCLUDE_POSTPROC
in lsb.params and
lsb.applications.
Default
Not defined
Where defined
From the command line.
LSB_JOBEXIT_INFO
Syntax
LSB_JOBEXIT_INFO="SIGNAL signal_value signal_name"
Description
Contains information about signal that caused a job to exit.
Applies to post-execution commands. Post-execution commands are set with
POST_EXEC
in lsb.queues.
When the post-execution command is run, the environment variable
LSB_JOBEXIT_INFO
is set if the job is signaled internally. If the job ends
successfully, or the job is killed or signaled externally, LSB_JOBEXIT_INFO
is not
set.
Examples
LSB_JOBEXIT_INFO="SIGNAL -1 SIG_CHKPNT" LSB_JOBEXIT_INFO="SIGNAL -14 SIG_TERM_USER"
LSB_JOBEXIT_INFO="SIGNAL -23 SIG_KILL_REQUEUE"
Default
Not defined
Where defined
Set by sbatchd
LSB_JOBEXIT_STAT
Syntax
LSB_JOBEXIT_STAT=exit_status
Description
Indicates a job's exit status.
Applies to post-execution commands. Post-execution commands are set with
POST_EXEC
in lsb.queues.
When the post-execution command is run, the environment variable
LSB_JOBEXIT_STAT
is set to the exit status of the job. Refer to the man page for
the wait(2) command for the format of this exit status.
The post-execution command is also run if a job is requeued because the job’s execution
environment fails to be set up, or if the job exits with one of the queue's
REQUEUE_EXIT_VALUES
. The LSB_JOBPEND
environment variable is set
if the job is requeued. If the job's execution environment could not be set up,
LSB_JOBEXIT_STAT
is set to 0.
Valid values
Any positive integer
Where defined
Set by sbatchd
LSB_JOBFILENAME
Syntax
LSB_JOBFILENAME=file_name
Description
The path to the batch executable job file that invokes the batch job. The batch executable job file is a /bin/sh script on UNIX systems or a .BAT command script on Windows systems.
LSB_JOBGROUP
Syntax
LSB_JOBGROUP=job_group_name
Description
The name of the job group associated with the job. When a job is dispatched, if it belongs to a
job group, the runtime variable LSB_JOBGROUP
is defined as its group. For example,
if a dispatched job belongs to job group /X, LSB_JOBGROUP=/X.
Where defined
Set during job execution based on bsub options or the default job group
defined in DEFAULT_JOBGROUP
in lsb.params and the
LSB_DEFAULT_JOBGROUP
environment variable.
Default
Not defined
LSB_JOBID
Syntax
LSB_JOBID=job_ID
Description
The job ID assigned by sbatchd. This is the ID of the job assigned by LSF, as shown by bjobs.
Valid values
Any positive integer
Where defined
Set by sbatchd, defined by mbatchd
See also
LSB_REMOTEJID
LSB_JOBINDEX
Syntax
LSB_JOBINDEX=index
Description
Contains the job array index.
Valid values
Any integer greater than zero but less than the maximum job array size.
Notes
LSB_JOBINDEX
is set when each job array element is dispatched. Its value
corresponds to the job array index. LSB_JOBINDEX
is set for all jobs. For non-array
jobs, LSB_JOBINDEX
is set to zero (0).
Where defined
Set during job execution based on bsub options.
Example
You can use LSB_JOBINDEX
in a shell script to select the job command to be
performed based on the job array index.
if [$LSB_JOBINDEX -eq 1]; then cmd1 fi if [$LSB_JOBINDEX -eq 2]; then cmd2 fi
See also
LSB_JOBINDEX_STEP
and LSB_REMOTEINDEX
LSB_JOBINDEX_STEP
Syntax
LSB_JOBINDEX_STEP=step
Description
Step at which single elements of the job array are defined.
Valid values
Any integer greater than zero but less than the maximum job array size
Default
1
Notes
LSB_JOBINDEX_STEP
is set when a job array is dispatched. Its value corresponds
to the step of the job array index. This variable is set only for job arrays.
Where defined
Set during job execution based on bsub options.
Example
array[1-10:2] elements:1 3 5 7 9
If this job array is dispatched, then LSB_JOBINDEX_STEP=2.
See also
LSB_JOBINDEX
LSB_JOBNAME
Syntax
LSB_JOBNAME=job_name
Description
The name of the job defined by the user at submission time.
Default
The job's command line
Notes
The name of a job can be specified explicitly when you submit a job. The name does not have to be unique. If you do not specify a job name, the job name defaults to the actual batch command as specified on the bsub command line.
The job name can be up to 4094 characters long for UNIX and Linux or up to 255 characters for Windows.
Where defined
Set by sbatchd
Example
% bsub -J "myjob" job
LSB_JOBNAME=myjob
LSB_JOBPEND
Description
Set if the job is requeued.
Where defined
Set by sbatchd for POST_EXEC
only
See also
LSB_JOBEXIT_STAT
, REQUEUE_EXIT_VALUES
, and
POST_EXEC
LSB_JOBPGIDS
Description
A list of the current process group IDs of the job.
Where defined
The process group IDs are assigned by the operating system, and LSB_JOBPGIDS
is
set by sbatchd.
See also
LSB_JOBPGIDS
LSB_JOBPIDS
Description
A list of the current process IDs of the job.
Where defined
The process IDs are assigned by the operating system, and LSB_JOBPIDS
is set by
sbatchd.
LSB_MAILSIZE
Syntax
LSB_MAILSIZE=value
Description
Gives an estimate of the size of the batch job output when the output is sent by email. It is not
necessary to configure LSB_MAILSIZE_LIMIT
.
LSF sets
LSB_MAILSIZE
to the size in kilobytes (KB) of the job output, allowing the custom
mail program to intercept output that is larger than desired.
LSB_MAILSIZE
is not recognized by the LSF
default mail program. To prevent large job output files from interfering with your mail system, use
LSB_MAILSIZE_LIMIT
to explicitly set the maximum size in kilobytes of the email
containing the job information.
Valid values
- A positive integer
- If the output is being sent by email,
LSB_MAILSIZE
is set to the estimated mail size in kilobytes. - -1
- If the output fails or cannot be read,
LSB_MAILSIZE
is set to -1 and the output is sent by email usingLSB_MAILPROG
if specified in lsf.conf. - Not defined
- If you use the -o or -e options of bsub,
the output is redirected to an output file. Because the output is not sent by email in this case,
LSB_MAILSIZE
is not used andLSB_MAILPROG
is not called.If the -N option is used with the -o option of bsub,
LSB_MAILSIZE
is not set.
Where defined
Set by sbatchd when the custom mail program specified by
LSB_MAILPROG
in lsf.conf is called.
LSB_MAX_NUM_PROCESSORS
Syntax
LSB_MAX_NUM_PROCESSORS=integer
Description
The maximum number of processors requested when the job is submitted.
Where defined
Set during job execution based on bsub options. For example, for a job submitted with -n 2,4, the maximum number of processors requested is four.
LSB_MCPU_HOSTS
Syntax
LSB_MCPU_HOSTS="host_nameA num_processors1 host_nameB num_processors2..."
Description
Contains a list of the hosts and the number of CPUs used to run a job.
Valid values
num_processors1, num_processors2,... refers to the number of CPUs used on host_nameA, host_nameB,..., respectively
Default
Not defined
Notes
The environment variables LSB_HOSTS
and LSB_MCPU_HOSTS
both
contain the same information, but the information is presented in different formats.
LSB_MCPU_HOSTS
uses a shorter format than LSB_HOSTS
. As a general
rule, sbatchd sets both these variables. However, for some parallel jobs,
LSB_HOSTS
is not set.
For parallel jobs, several CPUs are used, and the length of LSB_HOSTS
can become
very long. sbatchd needs to spend a lot of time parsing the string. If the size
of LSB_HOSTS
exceeds 4096 bytes, LSB_HOSTS
is ignored, and
sbatchd sets only LSB_MCPU_HOSTS
.
To verify the hosts and CPUs used for your dispatched job, check the value of
LSB_HOSTS
for single CPU jobs, and check the value of
LSB_MCPU_HOSTS
for parallel jobs.
Where defined
Set by sbatchd before starting a job on the execution host
Example
% bsub -m "hostA hostB" -n 6 job
LSB_HOSTS
and
LSB_MCPU_HOSTS
as
follows:LSB_HOSTS= "hostA hostA hostA hostB hostB hostB"
LSB_MCPU_HOSTS="hostA 3 hostB 3"
Both variables are set in order to maintain compatibility with earlier versions.
See also
LSB_HOSTS
LSB_NQS_PORT
This parameter can be defined in lsf.conf or in the services database such as /etc/services.
See LSB_NUM_NIOS_CALLBACK_THREADS in lsf.conf for more details.
LSB_NTRIES
Syntax
LSB_NTRIES=integer
Description
The number of times that LSF libraries attempt to contact mbatchd or perform a concurrent jobs query.
For example, if this parameter is not defined, when you type bjobs, LSF keeps displaying batch system not responding if mbatchd cannot be contacted or if the number of pending jobs exceeds MAX_PEND_JOBS specified in lsb.params or lsb.users.
If this parameter is set to a value, LSF only attempts to contact mbatchd the defined number of times and then quits. LSF will wait for a period of time equal to SUB_TRY_INTERVAL specified in lsb.params before attempting to contact mbatchd again.
Valid values
Any positive integer
Default
INFINIT_INT. Any value rather than a positive integer will be treated as INFINIT_INT. (The default is to continue the attempts to contact mbatchd).
LSB_OLD_JOBID
Syntax
LSB_OLD_JOBID=job_ID
Description
The job ID of a job at the time it was checkpointed.
When a job is restarted, it is assigned a new job ID and LSB_JOBID
is replaced
with the new job ID. LSB_OLD_JOBID
identifies the original ID of a job before it is
restarted.
Valid values
Any positive integer
Where defined
Set by sbatchd, defined by mbatchd
See also
LSB_JOBID
LSB_OUTDIR
String containing the full path to the output directory specified by the bsub -outdir option when a job is submitted. If an output directory wasn't specified, this variable contains the path to the job submission directory.
The LSF data management bstage out command uses this directory as the default destination for jobs being transferred out.
LSB_OUTPUT_TARGETFAILED
Syntax
LSB_OUTPUT_TARGETFAILED=Y
Description
Indicates that LSF cannot access the output file specified for a job submitted the bsub -o option.
Valid values
Set to Y if the output file cannot be accessed; otherwise, it is not defined.
Where defined
Set by sbatchd during job execution
LSB_PROJECT_NAME
Syntax
LSB_PROJECT_NAME=project_name
Description
To enable the use of project names for accounting purposes inside third party tools that launch
jobs under LSF using
environment variables, the environment variable, LSB_PROJECT_NAME
is used.
When bsub -P is used to specify a project name,
LSB_PROJECT_NAME
is set with the project name.
Default
Not defined
Notes
Project names can be up to 511 characters long.
- If you submit a job with the -P option of bsub, the job
belongs to the project specified through the -P option (sets
LSB_PROJECT_NAME
). - If you submit a job without the -P option of bsub, but you
defined
LSB_DEFAULTPROJECT
, then the job belongs to the project specified inLSB_DEFAULTPROJECT
. - If the LSF
administrator defines
DEFAULT_PROJECT
in the lsb.params configuration file, the system uses this as the default project name (unlessLSB_DEFAULTPROJECT
is set or bsub -P is used).
Where defined
From the command line, or through the -P option of bsub
See also
LSB_DEFAULTPROJECT environment variable, DEFAULT_PROJECT in lsb.params, the -P option of bsub
LSB_QUEUE
Syntax
LSB_QUEUE=queue_name
Description
The name of the queue from which the job is dispatched.
Where defined
Set by sbatchd
LSB_RANK_HOSTFILE
Syntax
LSB_RANK_HOSTFILE=file_path
Description
Path to the OpenMPI host rank file specified by the DJOB_ENV_SCRIPT option in lsb.applications.
If DJOB_ENV_SCRIPT=openmpi_rankfile.sh is set in
lsb.applications, LSF creates a host rank file and sets the environment
variable LSB_RANK_HOSTFILE
.
On the first execution host, LSF sbatchd will create a host rank file per job under the same location as $LSB_DJOB_HOSTFILE.
Default
Not defined
Where defined
Set during job execution.
LSB_REMOTEINDEX
Syntax
LSB_REMOTEINDEX=index
Description
The job array index of a remote LSF multicluster
capability job.
LSB_REMOTEINDEX
is set only if the job is an element of a job array.
Valid values
Any integer greater than zero, but less than the maximum job array size
Where defined
Set by sbatchd
See also
LSB_JOBINDEX
and MAX_JOB_ARRAY_SIZE
in
lsb.params
LSB_REMOTEJID
Syntax
LSB_REMOTEJID=job_ID
Description
The job ID of a remote LSF multicluster capability job.
Where defined
Set by sbatchd, defined by mbatchd
See also
LSB_JOBID
LSB_JOB_REPORT_MAIL
Syntax
LSB_JOB_REPORT_MAIL=Y | N | ERROR
Description
End users can set the value for this environment variable to N in the job submission environment to disable email notification for only that particular job and not email notification for all jobs. In this case, there is no need to restart sbatchd. Mail generated from mbatchd is sent as usual. If LSB_JOB_REPORT_MAIL=Y, job-related email is sent.
End users can set the value for this environment variable to ERROR in the job submission environment to send mail only when the job exits (that is, when the job is under Exit status). This parameter only affects email sent by sbatchd. This ensures that an email notification is only sent on a job error.
Where defined
From the command line.
LSB_RESTART
Syntax
LSB_RESTART=Y
Description
Indicates that a job has been restarted or migrated.
Valid values
Set to Y if a checkpointable job has been restarted or migrated; otherwise, it is undefined.
Notes
If a checkpointable job is submitted with the -k option of the bsub command, the LSB_RESTART variable is set to Y when the job is restarted. If the bmig command is used to migrate a job, the LSB_RESTART variable is set to Y when the migrated job is restarted.
If the job is not a restarted job, then the LSB_RESTART variable is not set.
Where defined
Set by sbatchd during job execution
See also
LSB_RESTART_PGID
and LSB_RESTART_PID
LSB_RESTART_PGID
Syntax
LSB_RESTART_PGID=pgid
Description
The process group ID of the checkpointed job when the job is restarted.
Notes
When a checkpointed job is restarted, the operating system assigns a new group process ID to the
job. LSF sets
LSB_RESTART_PGID
to the new group process ID.
Where defined
Set during restart of a checkpointed job.
See also
LSB_RESTART
LSB_RESTART_PID
Syntax
LSB_RESTART_PID=pid
Description
The process ID of the checkpointed job when the job is restarted.
Notes
When a checkpointed job is restarted, the operating system assigns a new process ID to the job.
LSF sets LSB_RESTART_PID
to the new process ID.
Where defined
Defined during restart of a checkpointed job
See also
LSB_RESTART_PID
and LSB_RESTART
LSB_RTASK_GONE_ACTION
Syntax
LSB_RTASK_GONE_ACTION=task_action ...
Description
- IGNORE_TASKCRASH
- A remote task crashes. The job
RES
does nothing. - KILLJOB_TASKDONE
- A remote task exits with zero value. The job
RES
notifies the taskRES
to terminate all tasks in the job. - KILLJOB_TASKEXIT
- A remote task exits with non-zero value. The job
RES
notifies the taskRES
to terminate all tasks in the job.
Where defined
Set by the system based on the value of the parameter RTASK_GONE_ACTION
in
lsb.applications when running bsub -app for the specified
application.
See also
RTASK_GONE_ACTION
in lsb.applications
LSB_SUB_ADDITIONAL
Usage
LSB_SUB_ADDTIONAL=esub executables
Description
String that contains the application name or names of the esub executables requested by the user. This is the only option that an esub cannot change or add at job submission.
Default
Not defined
Where defined
Set by LSF on the submission host before running esub.
LSB_SUB_CLUSTER
Description
Name of submission cluster (for a cluster with LSF multicluster capability only)
Where defined
Set on the submission environment and passed to the execution cluster environment. The parameter will only be valid in an LSF multicluster capability environment. For jobs on a local cluster, the parameter is not set when using any daemon wrappers such as job starter, post, pre, or eexec scripts.
LSB_SUB_COMMANDNAME
Description
The job command line.
If set, enables esub to use the variable LSB_SUB_COMMAND_LINE
in the esub job parameter file specified by the
$LSB_SUB_PARM_FILE
environment variable. The LSB_SUB_COMMAND_LINE
variable carries the value of the bsub command argument, and is used when
esub runs.
Where defined
Set by esub before a job is submitted.
LSB_SUB_COMMAND_LINE
Description
The job command line.
The job command line can be up to 4094 characters long for UNIX and Linux or up to 255 characters for Windows.
Where defined
Set by esub before a job is submitted.
LSB_SUB_EXTSCHED_PARAM
Description
Value of external scheduling options specified by bsub -extsched, or
queue-level MANDATORY_EXTSCHED
or DEFAULT_EXTSCHED
.
Where defined
Set by esub before a job is submitted.
LSB_SUB_JOB_ACTION_WARNING_TIME
Description
Value of job warning time period specified by bsub -wt.
Where defined
Set by esub before a job is submitted.
LSB_SUB_JOB_ERR
Syntax
LSB_SUB_JOB_ERR=lsb_err_number
Description
The error number for an externally-submitted job defined by mbatchd if the job submission failed. This is available to the external post-submission scripts (epsub) to determine the reason for the job submission failure.
If the job submission is successful, this value is LSB_NO_ERROR (or 0).
Valid values
Any non-negative integer.
A value of LSB_NO_ERROR (or 0) indicates that the job submission is successful.
Where defined
Set by bsub or bmod, defined by mbatchd.
See also
LSB_NO_ERROR
LSB_SUB_JOB_ID
Syntax
LSB_SUB_JOB_ID=job_ID | -1
Description
The job ID assigned by mbatchd. This is available to the external post-submission scripts (epsub).
Valid values
Any positive integer, or -1 to indicate that no job ID was received from mbatchd.
Where defined
Set by bsub or bmod, defined by mbatchd.
See also
LSB_JOBID
LSB_SUB_JOB_QUEUE
Syntax
LSB_SUB_JOB_QUEUE=queue_name
Description
The name of the final queue from which the job is dispatched, which includes any queue modifications made by esub. This is available to the external post-submission scripts (epsub).
Where defined
Set by bsub or bmod, defined by mbatchd.
LSB_SUB_JOB_WARNING_ACTION
Description
Value of job warning action specified by bsub -wa.
Where defined
Set by esub before a job is submitted.
LSB_SUB_MEM_SWAP_HOST_LIMIT
Syntax
LSB_SUB_MEM_SWAP_HOST_LIMIT=Y | SUB_RESET
Description
Controls per-job host-based memory and swap limit enforcement on hosts that support Linux cgroups. LSB_RESOURCE_ENFORCE="memory" must be specified in lsf.conf and a memory or swap limit must be specified for the job for host-based memory and swap limit enforcement to take effect.
Where defined
Set by esub at job submission.
LSB_SUB_PARM_FILE
Syntax
LSB_SUB_PARM_FILE=file_name
Description
Points to a temporary file that LSF uses to store the bsub options entered in the command line. An esub reads this file at job submission and either accepts the values, changes the values, or rejects the job. Job submission options are stored as name-value pairs on separate lines in the format option_name=value. A typical use of this file is to control job submission options.
Where defined
Set by LSF on the submission host before running esub. Not defined when lsrun or lsgrun are used for interactive remote execution.
LSB_SUB_RES_REQ
Syntax
LSB_SUB_RES_REQ=res_req_string
Description
The job level resource requirement for the job submission. If the job level resource requirement was changed by bmod -R for a running job, the changed resource requirement is not available via this environment variable.
Where defined
Set by bsub at job submission.
LSB_SUB_USER
Syntax
LSB_SUB_USER=user_name
Description
The user name for the job submission.
Where defined
Set by bsub at job submission.
LSB_SUBK_SHOW_JOBID
This parameter can be set from the command line or from lsf.conf.
See LSB_SUBK_SHOW_JOBID
in lsf.conf.
LSB_SUCCESS_EXIT_VALUES
Syntax
LSB_SUCCESS_EXIT_VALUES=[exit_code …]
Description
Specifies the exit values that indicate successful execution for applications that successfully exit with non-zero values. Use spaces to separate multiple exit codes. exit_code should be the value between 0 and 255.
User-defined LSB_SUCCESS_EXIT_VALUES
overrides application profile level
specification of SUCCESS_EXIT_VALUES
in lsb.applications.
LSB_SUSP_REASONS
Syntax
LSB_SUSP_REASONS=integer
Description
An integer representing suspend reasons. Suspend reasons are defined in lsbatch.h.
This parameter is set when a job goes to system-suspended (SSUSP) or user-suspended status (USUSP). It indicates the exact reason why the job was suspended.
To determine the exact reason, you can test the value of LSB_SUSP_REASONS
against the symbols defined in lsbatch.h.
Where defined
Set during job execution
See also
LSB_SUSP_SUBREASONS
LSB_SUSP_SUBREASONS
Syntax
LSB_SUSP_SUBREASONS=integer
Description
An integer representing the load index that caused a job to be suspended.
When the suspending reason SUSP_LOAD_REASON
(suspended by load) is set in
LSB_SUSP_REASONS
, LSB_SUSP_SUBREASONS
set to one of the load index
values defined in lsf.h.
Use LSB_SUSP_SUBREASONS
and LSB_SUSP_SUBREASONS
together in you
custom job control to determine the exact load threshold that caused a job to be suspended.
Load Index | Value |
---|---|
R15S | 0 |
R1M | 1 |
R15M | 2 |
UT | 3 |
PG | 4 |
IO | 5 |
LS | 6 |
IT | 7 |
TMP | 8 |
SWP | 9 |
MEM | 10 |
Default
Not defined
Where defined
Set during job execution.
See also
LSB_SUSP_REASONS
LSB_SWITCH_ABORT_VALUE
Usage
LSB_SWITCH_ABORT_VALUE=exit_code
Description
Indicates the value that eswitch should exit with if LSF is to reject the job switch request.
Notes
If eswitch exits with the value LSB_SWITCH_ABORT_VALUE, LSF will reject the job submission. For example, to reject a job in a in Bourne shell eswitch:
exit $LSB_SWITCH_ABORT_VALUE
Default
Not defined
Where defined
Set by LSF on the submission host before running eswitch.
See also
LSB_SWITCH_PARM_FILE
LSB_SWITCH_MODIFY_FILE
Description
Points to the file that the eswitch uses to modify the bswitch target queue values.
When the eswitch runs, LSF checks
the specified file and applies changes so that the job switches to the revised queue. The job switch
request is rejected when eswitch encounters any parameters in the file defined
for LSB_SWITCH_MODIFY_FILE
, other than LSB_SWITCH_QUEUE
.
Where defined
Set by eswitch before a job is switched.
LSB_SWITCH_PARM_FILE
Syntax
LSB_SWITCH_PARM_FILE=file_name
Description
Points to a temporary file that LSF uses to store the bswitch options entered in the command line. An eswitch reads this file at the job bswitch level and either accepts or changes the values. Job switch options are stored as name-value pairs on separate lines in the format option_name=value. A typical use of this file is to control job switch options.
Where defined
Set by LSF on the submission host before running the eswitch.
LSB_TASK_GEOMETRY
Usage
setenv LSB_TASK_GEOMETRY "{(task_ID,...) ...}"
Description
Specifies task geometry for your jobs. LSB_TASK_GEOMETRY
overrides any process
group or command file placement options. This environment variable is checked for all parallel jobs.
If set, users submit a parallel job (a job that requests more than 1 slot) and LSF
attempts to shape LSB_MCPU_HOSTS
accordingly.
LSB_TASK_GEOMETRY
replaces LSB_PJL_TASK_GEOMETRY
, which is kept
for compatibility with earlier versions
Where defined
From the command line.
LSB_UNIT_FOR_JOBS_DISPLAY
This parameter can be set from the command line or from lsf.conf.
See LSB_UNIT_FOR_JOBS_DISPLAY
in lsf.conf.
LSB_UNIXGROUP
Description
Specifies the UNIX user group of the submitting user.
Notes
This variable is useful if you want pre-execution or post-execution processing to use the user group of the user who submitted the job, and not sys(1).
Where defined
Set during job execution
LSB_USER_BIND_CPU_LIST
The binding requested at job submission takes effect when LSF_BIND_JOB=USER_CPU_LIST in lsf.conf or BIND_JOB=USER_CPU_LIST in an application profile in lsb.applications. LSF makes sure that the value is in the correct format, but does not check that the value is valid for the execution hosts.
The correct format is a list which may contain multiple items, separated by comma, and ranges.
For example: 0,5,7,9-11
.
LSB_USER_BIND_JOB
The binding requested at job submission takes effect when LSF_BIND_JOB=USER in lsf.conf or BIND_JOB=USER in an application profile in lsb.applications. This value must be one of Y, N, NONE, BALANCE, PACK, or ANY. Any other value is treated as ANY.
LSF_AC_JOB_NOTIFICATION
Syntax
LSF_AC_JOB_NOTIFICATION="[exit ][done ][start ][suspend]"
Description
Requests that the user be notified when the job reaches any of the specified states.
Use a space to separate multiple job states. The notification request is logged in JOB_NEW events.
Where defined
Set before job submission.
LSF_CMD_LOGDIR
This parameter can be set from the command line or from lsf.conf.
See LSF_CMD_LOGDIR
in lsf.conf.
LSF_DEBUG_CMD
This parameter can be set from the command line or from lsf.conf.
See LSF_DEBUG_CMD
in lsf.conf.
LSF_DEBUG_LIM
This parameter can be set from the command line or from lsf.conf.
See LSF_DEBUG_LIM
in lsf.conf.
LSF_DEBUG_RES
This parameter can be set from the command line or from lsf.conf.
See LSF_DEBUG_RES
in lsf.conf.
LSF_EAUTH_AUX_DATA
Syntax
LSF_EAUTH_AUX_DATA=path/file_nameDescription
Used in conjunction with LSF daemon authentication, specifies the full path to the temporary file on the local file system that stores auxiliary authentication information (such as credentials required by a remote host for use during job execution). Provides a way for eauth -c, mbatchd, and sbatchd to communicate the location of auxiliary authentication data. Set internally by the LSF libraries in the context of eauth.
For Kerberos authentication, used for forwarding credentials to the execution host.
LSF_EAUTH_AUX_PASS
Syntax
LSF_EAUTH_AUX_PASS=yesDescription
Enables forwarding of credentials from a submission host to an execution host when daemon authentication is enabled. LSF_EAUTH_AUX_PASS=yes indicates that a credential can be added to the execution context of a job. Set to yes by bsub during job submission or by bmod during job modification so that eauth -c can forward credentials.
LSF_EAUTH_CLIENT
Syntax
LSF_EAUTH_CLIENT=mbatchd|sbatchd|pam|res|userDescription
Used with LSF daemon authentication, specifies the LSF daemon, command, or user that invokes eauth -c. Used when writing a customized eauth executable to set the context for the call to eauth. Set internally by the LSF libraries or by the LSF daemon, command, or user calling eauth -c.
LSF_EAUTH_SERVER
Syntax
LSF_EAUTH_SERVER=mbatchd|sbatchd|pam|res|userDescription
Used with LSF daemon authentication, specifies the daemon that invokes eauth -s. Used when writing a customized eauth executable to set the context for the call to eauth. Set internally by the LSF libraries or by the LSF daemon calling eauth -s.
LSF_EAUTH_UID
Syntax
LSF_EAUTH_UID=user_IDDescription
Specifies the user account under which eauth -s runs. Set by the LSF daemon that executes eauth.
LSF_EXECUTE_DOMAIN
Syntax
LSF_EXECUTE_DOMAIN=domain_name setenv LSF_EXECUTE_DOMAIN domain_nameDescription
If UNIX or Windows user account mapping is enabled, specifies the preferred Windows execution
domain for a job submitted by a UNIX user. The execution domain must be one of the domains listed in
LSF_USER_DOMAIN
.
LSF_EXECUTE_DOMAIN
is defined in the user environment
(.cshrc or .profile) or from the command line. Specify
only one domain.
Use this parameter in conjunction with the bsub, lsrun, and
lsgrun commands to bypass the order of the domains listed in
LSF_USER_DOMAIN
and run the job using the specified domain. If you do not have a
Windows user account in the execution domain, LSF tries
to run the job using one of the other domains defined by LSF_USER_DOMAIN
. Once you
submit a job with an execution domain defined, you cannot change the execution domain for that
particular job.
LSF_INTERACTIVE_STDERR
This parameter can be defined in lsf.conf.
See LSF_INTERACTIVE_STDERR
in lsf.conf for more
details.
LSF_INVOKE_CMD
Syntax
LSF_INVOKE_CMD=invoking_command_nameDescription
Indicates the name of the last LSF command that invoked an external executable (for example, esub or eexec).
External executables get called by different LSF commands, such as bsub, bmod, or lsrun.
Default
Not defined
Where defined
Set internally within by LSF.
LSF_JOB_STARTER
Syntax
LSF_JOB_STARTER=binary
Description
Specifies an executable program that has the actual job as an argument.
Default
Not defined
Interactive Jobs
If you want to run an interactive job that requires some preliminary setup, LSF provides a job starter function at the command level. A command-level job starter allows you to specify an executable file that will run prior to the actual job, doing any necessary setup and running the job when the setup is complete.
If LSF_JOB_STARTER
is properly defined, RES
will invoke the job
starter (rather than the job itself), supplying your commands as arguments.
Batch Jobs
A job starter can also be defined at the queue level using the LSF_JOB_STARTER
parameter, although this can only be done by the LSF
administrator.
Where defined
From the command line
Example: UNIX
/bin/sh -c "$LSF_JOB_STARTER command [argument...]"
where command [argument...] are the command line arguments you specified in lsrun or lsgrun.
LSF_JOB_STARTER
as
follows:setenv LSF_JOB_STARTER "/bin/csh -c"
lsrun "'a.out; echo hi'"
/bin/sh -c "/bin/csh -c 'a.out; echo hi'"
Example: Windows
LSF_JOB_STARTER command [argument...]
LSF_JOB_STARTER
as
follows:set LSF_JOB_STARTER=C:\cmd.exe /C
C:\> lsrun dir /p
C:\cmd.exe /C dir /p
See also
JOB_STARTER
in lsb.queues
LSF_LD_LIBRARY_PATH
Description
When LSF_LD_SECURITY=Y in lsf.conf, contains the
value of the LD_LIBRARY_PATH
environment variable, which is removed from the job
environment during job initialization to ensure enhanced security against users obtaining root
privileges. LSF_LD_LIBRARY_PATH
allows the LD_LIBRARY_PATH
environment variable to be put back before the job runs.
Where defined
For jobs submitted using bsub -Is or bsub -Ip only.
See also
LSF_LD_PRELOAD
and LSF_LD_SECURITY
in
lsf.conf
LSF_LD_PRELOAD
Description
When LSF_LD_SECURITY=Y in lsf.conf, contains the
value of the LD_PRELOAD
environment variable, which is removed from the job
environment during job initialization to ensure enhanced security against users obtaining root
privileges. LSF_LD_PRELOAD
allows the LD_PRELOAD
environment
variable to be put back before the job runs.
Where defined
For jobs submitted using bsub -Is or bsub -Ip only.
See also
LSF_LD_LIBRARY_PATH
and LSF_LD_SECURITY
in
lsf.conf
LSF_LIM_API_NTRIES
Syntax
LSF_LIM_API_NTRIES=integer
Description
Defines the number of times LSF
commands will retry to communicate with the LIM API when LIM is not available. LSF_LIM_API_NTRIES
is ignored by LSF and
EGO daemons and EGO
commands. The LSF_LIM_API_NTRIES
environment variable overrides the value of
LSF_LIM_API_NTRIES
in lsf.conf.
Valid values
1 to 65535
Where defined
From the command line or from lsf.conf
Default
Not defined. If not defined in lsf.conf. LIM API exits without retrying.
LSF_LIM_DEBUG
This parameter can be set from the command line or from lsf.conf.
See LSF_LIM_DEBUG
in lsf.conf.
LSF_LOGDIR
This parameter can be set from the command line or from lsf.conf.
See LSF_LOGDIR
in lsf.conf.
LSF_LSLOAD_FORMAT
This parameter can be set from the command line or from lsf.conf.
See LSF_LSLOAD_FORMAT
in lsf.conf.
LSF_LSHOSTS_FORMAT
This parameter can be set from the command line or from lsf.conf.
See LSF_LSHOSTS_FORMAT
in lsf.conf.
LSF_MASTER
Description
Set by the LIM to identify the management host. The value is Y on the management host and N on all other hosts. An elim executable can use this parameter to check the host on which the elim is currently running.
Used when the external load indices feature is enabled.
When defined
Set by the LIM when it starts the parent external load information manager (MELIM).
See also
LSF_RESOURCES
LSF_NIOS_DEBUG
This parameter can be set from the command line or from lsf.conf.
See LSF_NIOS_DEBUG
in lsf.conf.
LSF_NIOS_ERR_LOGDIR
This parameter can be set from the command line or from lsf.conf.
See LSF_NIOS_ERR_LOGDIR
in lsf.conf.
LSF_NIOS_DIE_CMD
Syntax
LSF_NIOS_DIE_CMD=command
Description
If set, the command defined by the LSF_NIOS_DIE_CMD
environment variable is
executed for a pending job before NIOS exits with a failure.
You can use the %J special character to specify the command. %J is replaced by the job ID.
Default
bkill %J
Where defined
From the command line.
LSF_NIOS_IGNORE_SIGWINDOW
Syntax
LSF_NIOS_IGNORE_SIGWINDOW=any_value
Description
If defined, the NIOS will ignore the SIGWINDOW
signal.
Default
Not defined
Notes
When the signal SIGWINDOW
is defined, some tasks appear to die when they receive
the SIGWINDOW
while inputting and outputting. By defining
LSF_NIOS_IGNORE_SIGWINDOW
, these tasks are given the chance to ignore the
signal.
Where defined
From the command line.
LSF_NIOS_PEND_TIMEOUT
Syntax
LSF_NIOS_PEND_TIMEOUT=minutes
Description
Applies only to interactive batch jobs.
Maximum amount of time that an interactive batch job can remain pending.
If this parameter is defined, and an interactive batch job is pending for longer than the
specified time, LSF executes the command as specified by the LSF_NIOS_DIE_CMD
environment variable and returns the output of that command.
Valid values
Any integer greater than zero.
Default
Not defined
LSF_NIOS_PORT_RANGE
Syntax
LSF_NIOS_PORT_RANGE=min_port_number-max_port_number
Description
Defines a range of listening ports for NIOS to use.
Example
LSF_NIOS_PORT_RANGE=5000-6000
Default
Not defined. LSF randomly assigns a NIOS port number.
LSF_RESOURCES
Syntax
LSF_RESOURCES=dynamic_external_resource_name...
Description
Space-separated list of dynamic external resources. When the LIM starts a
parent external
load information manager (MELIM) on a host, the LIM
checks the resource mapping defined in the ResourceMap
section of
lsf.cluster.cluster_name. Based on the mapping
(default, all, or a host list), the LIM sets LSF_RESOURCES
to the list of resources expected on the host and
passes the information to the MELIM.
Used when the external load indices feature is enabled.
When defined
Set by the MELIM on the host when the MELMELIMIM invokes the elim executable.
See also
LSF_MASTER
LSF_SUBMIT_TO_CLUSTER
Syntax
LSF_SUBMIT_TO_CLUSTER=cluster_name
Description
For forwarded jobs when using the LSF multicluster capability. Defines the submission cluster, and submits jobs to this submission cluster instead of the local cluster. This enables forwarded jobs to submit jobs back to the submission cluster.
Where defined
From the forwarded job submission script.
Default
Not defined. The forwarded job script submits jobs to the local cluster.
LSF_TS_LOGON_TIME
Syntax
LSF_TS_LOGON_TIME=milliseconds
Description
Specifies the time to create a Windows terminal service session. Configure
LSF_TS_LOGON_TIME
according to the load on your network environment.
The default, 30000 milliseconds, is suitable for most environments. If you set
LSF_TS_LOGON_TIME
too small, LSF tries
multiple times before it succeeds in making a terminal service session with the terminal service
server, which can cause the job wait a long time before it runs. For a congested network, set
LSF_TS_LOGON_TIME=1000000.
Where defined
From the command line.
Default
30000 milliseconds
LSF_USE_HOSTEQUIV
Syntax
LSF_USE_HOSTEQUIV=y|Y
Description
Used for authentication purposes. If LSF_USE_HOSTEQUIV
is defined,
RES
and mbatchd call the ruserok(3)
function to decide if a user is allowed to run remote jobs. LSF trusts
all hosts configured in the LSF
cluster that are defined in hosts.equiv, or in .rhosts in
the user’s home directory.
The ruserok(3) function checks in the /etc/hosts.equiv file and the user's $HOME/.rhosts file to decide if the user has permission to execute remote jobs.
If LSF_USE_HOSTEQUIV
is not defined, all normal users in the cluster can execute
remote jobs on any host.
If LSF_ROOT_REX
is set, root can also execute remote jobs with the same
permission test as for normal users.
Default
Not defined
See also
LSF_ROOT_REX
and LSF_AUTH
in lsf.conf
LSF_USER_DOMAIN
Syntax
LSF_USER_DOMAIN=domain_name | .
Description
Set during LSF installation or setup. If you modify this parameter in an existing cluster, you probably have to modify passwords and configuration files also.
Windows or mixed UNIX and Windows clusters only.
Enables default user mapping, and specifies the LSF user domain. The period (.) specifies local accounts, not domain accounts.
- A user name specified without a domain is interpreted (on a Windows host) as belonging to the LSF user domain
- A user name specified with the domain name of the LSF user domain is not valid
- In a mixed cluster, this parameter defines a two-way, one-to-one user map between UNIX user accounts and Windows user accounts belonging to the specified domain, as long as the accounts have the same user name. This means jobs submitted by the Windows user account can run on a UNIX host, and jobs submitted by the UNIX account can run on any Windows host that is available to the Windows user account.
If this parameter is not defined, the default user mapping is not enabled. You can still configure user mapping at the user or system level. User account mapping is required to run cross-platform jobs in a UNIX and Windows mixed cluster.
Where defined
lsf.conf
Default
- If you upgrade from LSF 4.0.1 or earlier, the default is the existing LSF user domain.
- For a new, Windows-only cluster, this parameter is not defined (no LSF user domain, no default user mapping).
- For a new, mixed UNIX and Windows cluster, the default is the domain that the Windows installation account belongs to. This can be modified during LSF installation.
NOCHECKVIEW_POSTEXEC
LSF
integration for Rational ClearCase. Setting the
NOCHECKVIEW_POSTEXEC
environment variable to a value of 1
configures the daemon wrapper to skip running the checkView
function during the
post-execution phase of a job.
Syntax
NOCHECKVIEW_POSTEXEC=1
Description
NOCHECKVIEW_POSTEXEC
environment variable: - In csh or tcsh:
setenv NOCHECKVIEW_POSTEXEC 1
- In sh, ksh, or bash:
export NOCHECKVIEW_POSTEXEC=1
OMP_NUM_THREADS
Syntax
OMP_NUM_THREADS=num_cpus
Description
LSF will
set the environment variable OMP_NUM_THREADS
to the total number of CPUs allocated
to each task if the job requests more than one per task (that is, the job is a multithreaded job).
If you set OMP_NUM_THREADS
before submitting your job, LSF does
not change it, but passes it to each task.
If OMP_NUM_THREADS
is not set in the job submission environment, and the job has
an affinity resource request, and number_pu * number_sub_tasks
is greater than one, then OMP_NUM_THREADS
is set to number_pu *
number_sub_tasks. If number_pu *
number_sub_tasks is less than or equal to one, then
OMP_NUM_THREADS
is not set.
For example, for a job submitted with bsub -n 2 -R "affinity[core(3)*4]",
OMP_NUM_THREADS
will be set to 12 (3 multiplied by 4)
OMP_NUM_THREADS
is set according to the span[] resource
requirements option:- If -R "span[hosts=x]", then
OMP_NUM_THREADS
is set - If -R "span[ptile=y]", then
OMP_NUM_THREADS
is not set
For example, for a job submitted with bsub -a openmp -n 4 -R
"span[ptile=2]", OMP_NUM_THREADS
will be set to 2.
If the job does not specify the openmpesub, the
OMP_NUM_THREADS
is not set.
Default
Not defined
Where defined
Set during job execution.
RM_CPUTASKn
Syntax
RM_CPUTASKn=cpu_list
Description
The RM_CPUTASKn
environment variable is the IBM Parallel Environment equivalent of LSB_BIND_CPU_LIST
. Each
RM_CPUTASK
represents the CPU allocation for one task, as opposed to having a
single variable which amalgamates them all. For example, a job with 6 tasks running on 4 hosts could
have the following values for RM_CPUTASK
(the example corresponds to the tasks
specified in the example for LSB_AFFINITY_HOSTFILE
):
RM_CPUTASK1=0,1,2,3
RM_CPUTASK2=4,5,6,7
RM_CPUTASK1=0,1,2,3
RM_CPUTASK2=4,5,6,7
RM_CPUTASK1=0,1,2,3
RM_CPUTASK1=0,1,2,3
Default
Not defined
Where defined
Set during job execution.
RM_MEM_AFFINITY
Syntax
RM_MEM_AFFINITY=Yes | No
Description
RM_MEM_AFFINITY
environment variable informs the IBM
Parallel Environment memory binding policy. If a local-only policy
(membind=localonly affinity resource requirement parameter) has been set,
RM_MEM_AFFINITY
will be set to Yes
. If a local preference policy
(membind=localprefer affinity resource requirement parameter) has been set,
RM_MEM_AFFINITY
will be set to No
. Default
Not defined
Where defined
Set during job execution.
TASKMAN_EXEC_RUSAGE
Syntax
TASKMAN_EXEC_RUSAGE="resource_name1 resource_value1 resource_name2 resource_value2..."
Description
Indicates to taskman which rusage string is satisfied to permit the task to run. This environment variable is necessary because the OR (||) operator specifies alternative rusage strings for running tasks that are launched inside LSF jobs.
Valid values
resource_value1, resource_value2,... refer to the resource values on resource_name1, resource_name2,... respectively.
Default
Not defined
Where defined
Set by LSF after reserving a resource for the job.