LSF_HPC_EXTENSIONS

Syntax

LSF_HPC_EXTENSIONS="extension_name ..."

Description

Enables LSF HPC extensions.

After adding or changing LSF_HPC_EXTENSIONS, use badmin mbdrestart and badmin hrestart to reconfigure your cluster.

Valid values

The following extension names are supported:

CUMULATIVE_RUSAGE: When a parallel job script runs multiple commands, resource usage is collected for jobs in the job script, rather than being overwritten when each command is executed.

DISP_RES_USAGE_LIMITS: bjobs displays resource usage limits configured in the queue as well as job-level limits.

HOST_RUSAGE: For parallel jobs, reports the correct rusage based on each host’s usage and the total rusage being charged to the execution host. This host rusage breakdown applies to the blaunch framework, the pam framework, and vendor MPI jobs. For a running job, you will see run time, memory, swap, utime, stime, and pids and pgids on all hosts that a parallel job spans. For finished jobs, you will see memory, swap, utime, and stime on all hosts that a parallel job spans. The host-based rusage is reported in the JOB_FINISH record of lsb.acct and lsb.stream, and the JOB_STATUS record of lsb.events if the job status is done or exit. Also for finished jobs, bjobs -l shows CPU time, bhist -l shows CPU time, and bacct -l shows utime, stime, memory, and swap. In the MultiCluster lease model, the parallel job must run on hosts that are all in the same cluster. If you use the jobFinishLog API, all external tools must use jobFinishLog built with LSF 9.1 or later, or host-based rusage will not work. If you add or remove this extension, you must restart mbatchd, sbatchd, and res on all hosts. The behaviour used to be controlled by HOST_RUSAGE prior to LSF 9.1.

NO_CUMULATIVE_RUSAGE: Turn on this parameter if you do not want to collect the total resource usage for jobs that are run in a parallel job script that runs multiple commands. Instead, the resource usage is overwritten when each command is executed.

NO_HOST_RUSAGE: Turn on this parameter if you do not want to see host-based job resource usage details. However, mbatchd will continue to report job rusage with bjobs for all running jobs even if you configure NO_HOST_RUSAGE and restart all the daemons.

LSB_HCLOSE_BY_RES: If res is down, host is closed with a message

Host is closed because RES is not available.

The status of the closed host is closed_Adm. No new jobs are dispatched to this host, but currently running jobs are not suspended.

RESERVE_BY_STARTTIME: LSF selects the reservation that gives the job the earliest predicted start time.

By default, if multiple host groups are available for reservation, LSF chooses the largest possible reservation based on number of slots.

SHORT_EVENTFILE: Compresses long host name lists when event records are written to lsb.events and lsb.acct for large parallel jobs. The short host string has the format:

number_of_hosts*real_host_name

Tip:

When SHORT_EVENTFILE is enabled, older daemons and commands (pre-LSF Version 7) cannot recognize the lsb.acct and lsb.events file format.

For example, if the original host list record is

6 "hostA" "hostA" "hostA" "hostA" "hostB" "hostC"

redundant host names are removed and the short host list record becomes

3 "4*hostA" "hostB" "hostC"

When LSF_HPC_EXTENSIONS="SHORT_EVENTFILE" is set, and LSF reads the host list from lsb.events or lsb.acct, the compressed host list is expanded into a normal host list.

SHORT_EVENTFILE affects the following events and fields:

JOB_START in lsb.events when a normal job is dispatched
- numExHosts (%d)
- execHosts (%s)
JOB_CHUNK in lsb.events when a job is inserted into a job chunk
- numExHosts (%d)
- execHosts (%s)
JOB_FORWARD in lsb.events when a job is forwarded to a MultiCluster leased host
- numReserHosts (%d)
- reserHosts (%s)
JOB_FINISH record in lsb.acct
- numExHosts (%d)
- execHosts (%s)

SHORT_PIDLIST: Shortens the output from bjobs to omit all but the first process ID (PID) for a job. bjobs displays only the first ID and a count of the process group IDs (PGIDs) and process IDs for the job.

Without SHORT_PIDLIST, bjobs -l displays all the PGIDs and PIDs for the job. With SHORT_PIDLIST set, bjobs -l displays a count of the PGIDS and PIDs.

TASK_MEMLIMIT: Enables enforcement of a memory limit (bsub -M, bmod -M, or MEMLIMIT in lsb.queues) for individual tasks in a parallel job. If any parallel task exceeds the memory limit, LSF terminates the entire job.

TASK_SWAPLIMIT: Enables enforcement of a virtual memory (swap) limit (bsub -v, bmod -v, or SWAPLIMIT in lsb.queues) for individual tasks in a parallel job. If any parallel task exceeds the swap limit, LSF terminates the entire job.

Example JOB_START events in lsb.events:

For a job submitted with

bsub -n 64 -R "span[ptile=32]" blaunch sleep 100

Without SHORT_EVENTFILE, a JOB_START event like the following is logged in lsb.events:

"JOB_START" "9.12" 1389121640 602 4 0 0 60.0 64 "HostA" 
"HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA"
"HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA"
"HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA"
"HostA" "HostA" "HostA" "HostA" "HostB" "HostB" "HostB" "HostB" "HostB"
"HostB" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB"
"HostB" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB"
"HostB" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB
"" "" 0 "" 0 "" 2147483647 4 "select[type == local] order[r15s:pg]
span[ptile=32] " "" -1 "" -1 0 "" -1 0

With SHORT_EVENTFILE, a JOB_START event would be logged in lsb.events with the number of execution hosts (numExHosts field) changed from 64 to 2 and the execution host list (execHosts field) shortened to "32*hostA" and "32*hostB":

"JOB_START" "9.12" 1389304047 703 4 0 0 60.0 2 "32* HostA " "32* HostB " "" "" 0
"" 0 "" 2147483647 4 "select[type == local] order[r15s:pg] " "" -1 "" -1 0 "" -1 0

Example JOB_FINISH records in lsb.acct:

For a job submitted with

bsub -n 64 -R "span[ptile=32]" blaunch sleep 100

Without SHORT_EVENTFILE, a JOB_FINISH event like the following is logged in lsb.acct:

"JOB_FINISH" "9.12" 1389121646 602 33793 33816578 64 1389121640 0 0 
1389121640 "user1" "normal" "span[ptile=32]" "" "" "HostB" 
"/scratch/user1/logdir" "" "" "" "1389121640.602" 0 64 "HostA" "HostA" 
"HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA" 
"HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA" 
"HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA" "HostA" 
"HostA" "HostA" "HostA" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB" 
"HostB" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB" 
"HostB" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB" 
"HostB" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB" "HostB" 64 60.0 ""
"blaunch sleep 100" 0.073956 0.252925 93560 0 -1 0 0 46526 15 0 3328 0 -1 0 0 0
6773 720 -1 "" "default" 0 64 "" "" 0 0 0 "" "" "" "" 0 "" 0 "" -1 "/user1"
"" "" "" -1 "" "" 4112 "" 1389121640 "" "" 0 2 HostA 0 0 0 0 0 HostB 0 0 0 0
0 -1 0 0 "select[type == local] order[r15s:pg] span[ptile=32] " "" -1 "" -1 0
"" 0 0 "" 6 "/scratch/user1/logdir" 0 "" 0.000000

With SHORT_EVENTFILE, a JOB_FINISH event like the following would be logged in lsb.acct with the number of execution hosts (numExHosts field) changed from 64 to 2 and the execution host list (execHosts field) shortened to "32*hostA" and "32*hostB":

"JOB_FINISH" "9.12" 1389304053 703 33793 33554434 64 1389304041 0 0  
1389304047 "user1" "normal" "" "" "" "HostB" "/scratch/user1/LSF/conf"
 "" "" "" "1389304041.703" 0 2 "32*HostA" "32*HostB" 64 60.0 "" 
"blaunch sleep 100" 0.075956 0.292922 93612 0 -1 0 0 46466 0 0 0 0 -1 0 
0 0 9224 478 -1 "" "default" 0 64 "" "" 0 0 0 "" "" "" "" 0 "" 0 "" -1 "/user1"
"" "" "" -1 "" "" 4112 "" 1389304047 "" "" 0 2 HostB 0 0 0 0 0 HostA 0 0 0 
0 0 -1 0 0 "select[type == local] order[r15s:pg] " "" -1 "" -1 0 "" 0 0 "" 6 
"/scratch/user1/LSF/conf" 0 "" 0.000000

Example bjobs -l output without SHORT_PIDLIST:

bjobs -l displays all the PGIDs and PIDs for the job:

bjobs -l
Job <109>, User <user3>, Project <default>, Status <RUN>, Queue <normal>, Inte
                     ractive mode, Command <./myjob.sh>
Mon Jul 21 20:54:44 2009: Submitted from host <hostA>, CWD <$HOME/LSF/jobs;

 RUNLIMIT 
 10.0 min of hostA

 STACKLIMIT CORELIMIT MEMLIMIT
   5256 K    10000 K    5000 K
Mon Jul 21 20:54:51 2009: Started on <hostA>;
Mon Jul 21 20:55:03 2009: Resource usage collected.
                     MEM: 2 Mbytes;  SWAP: 15 Mbytes
                     PGID: 256871;  PIDs: 256871 
                     PGID: 257325;  PIDs: 257325 257500 257482 257501 257523 
                     257525 257531
 SCHEDULING PARAMETERS:
           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem
 loadSched   -     -     -     -       -     -    -     -     -      -      -  
 loadStop    -     -     -     -       -     -    -     -     -      -      -  
                    cpuspeed    bandwidth
 loadSched          -            -
 loadStop           -            -
<< Job <109> is done successfully. >>

Example bjobs -l output with SHORT_PIDLIST:

bjobs -l displays a count of the PGIDS and PIDs:

bjobs -l
Job <109>, User <user3>, Project <default>, Status <RUN>, Queue <normal>, Inte
                     ractive mode, Command <./myjob.sh>
Mon Jul 21 20:54:44 2009: Submitted from host <hostA>, CWD <$HOME/LSF/jobs;

 RUNLIMIT                
 10.0 min of hostA

 STACKLIMIT CORELIMIT MEMLIMIT
   5256 K    10000 K    5000 K
Mon Jul 21 20:54:51 2009: Started on <hostA>;
Mon Jul 21 20:55:03 2009: Resource usage collected.
                     MEM: 2 Mbytes;  SWAP: 15 Mbytes
                     PGID(s):  256871:1 PID, 257325:7 PIDs

 SCHEDULING PARAMETERS:
           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem
 loadSched   -     -     -     -       -     -    -     -     -      -      -  
 loadStop    -     -     -     -       -     -    -     -     -      -      - 
                    cpuspeed    bandwidth
 loadSched          -            -
 loadStop           -            -

Default

Set to "CUMULATIVE_RUSAGE HOST_RUSAGE LSB_HCLOSE_BY_RES SHORT_EVENTFILE" at time of installation for the PARALLEL configuration template. If otherwise undefined, then "HOST_RUSAGE CUMULATIVE_RUSAGE".