bhist

-a

Displays information about both finished and unfinished jobs.

This option overrides -d, -p, -s, and -r.

-aff

Displays historical job information about jobs with CPU and memory affinity resource requirement for each task in the job. If the job is pending, the requested affinity resources are displayed. For running jobs, the effective and combined affinity resource allocation is also displayed, along with a table headed AFFINITY. The table shows detailed memory and CPU binding information for each task, one line for each allocated processor unit. For finished jobs (EXIT or DONE state), the affinity requirements for the job, and the effective and combined affinity resource requirement details are displayed.

Use this option only with the -l option.

-b

Brief format.

-cname

In IBM® Spectrum LSF Advanced Edition, includes the cluster name for execution cluster hosts and host groups in output.

-d

Displays information about finished jobs.

-data

Displays historical information for jobs with data requirements (for example, jobs that are submitted with -data). The bhist -data option acts as a filter to show only jobs with data requirements.

bhist -data
Summary of time in seconds spent in various states:
JOBID  USER  JOB_NAME  PEND    PSUSP  RUN  USUSP  SSUSP  UNKWN  TOTAL
1962   user1 *1000000  410650  0      0     0     0      0      410650

-e

Displays information about exited jobs.

-hostfile

If a job was submitted with the bsub -hostfile option or modified with the bmod -hostfile option to point to a user-specified host file, the bhist -l -hostfile option, shows the user-specified host file path. The -hostfile option also shows the contents of the host file.

-gpu

bhist -l -gpu shows the following information on GPU job allocation after the job finishes:

Use this option only with the -l option.

Host Name

The name of the host.

GPU IDs on the host

Each GPU is shown as a separate line.

TASK and ID

List of job tasks and IDs using the GPU (separated by comma if used by multiple tasks)

MODEL

Contains the GPU brand name and model type name.

MTOTAL

The total GPU memory size.

GPU Compute Capability

MRSV

GPU memory reserved by the job

SOCKET

socket ID of the GPU located at

NVLINK

Indicates if the GPU has NVLink connections with other GPUs allocated for the job (ranked by GPU ID and including itself). The connection flag of each GPU is a character separated by “/” with the next GPU:

A “Y” indicates there is a direct NVLINK connection between two GPUs.

An “N” shows there is no direct NVLINK connection with that GPU.

A “-” shows the GPU is itself.

If the job exited abnormally due to a GPU-related error or warning, the error or warning message displays. If LSF could not get GPU usage information from DCGM, a hyphen (-) displays.

-l

Long format.

If the job was submitted with the bsub -K option, the -l option displays Synchronous execution.

If you submitted a job by using the OR (||) expression to specify alternative resources, this option displays the successful Execution rusage string with which the job ran.

If you submitted a job with multiple resource requirement strings by using the bsub -R option for the order, same, rusage, and select sections, the bhist -l command displays a single, merged resource requirement string for those sections, as if they were submitted by using a single -R option.

Jobs submitted with an esub (or epsub) using bsub -a (or modified using bmod -a), will show the latest esubs used for execution in bhist -l output, first with the default and then user esubs. If a user-specified esub script is the same as the default esub script, the duplicate esubs will show as one entry. If a job is submitted with an esub containing parameters, the esub and its parameters will be shown in bhist -l as well, and the format of the esub is the same as that specified in the job submission.

The long format includes the following information:

• Job exit codes.
• Exit reasons for terminated jobs.
• Job exceptions (for example, if job run time exceeds the runtime estimate, a job exception of runtime_est_exceeded is displayed).
• Resizable job information.
• SSH X11 forwarding information (-XF).
• Specified and execution current working directory (CWD). The full path is shown, including directory pattern values.
• Detailed information about jobs with data requirements. The heading DATA REQUIREMENTS is displayed followed by a list of the files or tags that are requested by the job, and any modifications made to the data requirements.
• User-specified reason for killing, stopping, or resuming the job.
• The host that issued the request to kill, stop, or resume the job.
• Job-level pending time limits and eligible pending time limits.
• Memory and CPU usage information, such as CPU efficiency, CPU peak usage, and memory efficiency values.
• The amount of time that the job spent in an eligible and ineligible pending state after the job started, if the TRACK_ELIGIBLE_PENDINFO parameter in the lsb.params file is set to Y or y.
• Applicable loaning information. This information shows whether the job uses resources that are borrowed from the GSLA resource pool, and from which pool the resources are borrowed.
• Account name for LSF resource connector.
• Starting in Fix Pack 14, the assigned resource names for each job.
• Changes to pending jobs as a result of the following bmod command options:
  • Absolute priority scheduling (-aps | -apsn)
  • Auto-resizable job attribute (-ar | -arn)
  • Current working directory (-cwd)
  • Data requirements (-data | -datan)
  • Post-execution command (-Ep | -Epn)
  • Job description (-Jd | -Jdn)
  • Checkpoint options (-k | -kn)
  • Migration threshold (-mig | -mign)
  • Job resize notification command (-rnc | -rncn)
  • User limits (-ul | -uln)
  • Runtime estimate (-We | -Wen)

The bhist -l command displays the effective GPU requirements string for a GPU allocation.

bhist -l
Job <204>, User <user1>, Project <default>, Command <blaunch sleep 60>
Wed Jul 12 22:40:54: Submitted from host <hosta>, to Queue <normal>, CWD </
                     scratch/user1>, 8 Task(s),Requested Resources <span[ptile=4] 
.....................rusage[ngpus_physical=4]>,Specified Hosts <hostb>, 
                     <hosta!>, Requested GPU <num=4:mode=shared:j_exclusive=yes>;
Wed Jul 12 22:40:55: Dispatched 8 Task(s) on Host(s) <hosta> <hosta> <h
                     hosta> <hosta> <hostb> <hostb> <hostb>
                     <hostb>, Allocated 8 Slot(s) on Host(s) <hosta> <h
                     hosta> <hosta> <hosta> <hostb> <hostb>
                     <hostb> <hostb>, Effective RES_REQ <select[type ==
                      any] order[r15s:pg] rusage[ngpus_physical=4.00] span[ptil
                     e=4] >;
Wed Jul 12 22:40:56: Starting (Pid 116194);
Wed Jul 12 22:40:56: External Message "hostb:gpus=0,3,1,2;hosta:gpus=0,1,2,3;
                     EFFECTIVE GPU REQ: num=4:mode=shared:mps=no:j_exclusive=yes;"
                     was posted from "user1" to message box 0;

-p

Displays information about pending jobs.

-r

Displays information about running jobs.

-s

Displays information about suspended jobs.

-t

Displays job events chronologically, including energy aware scheduling events JOB_PROV_HOST and HOST_POWER_STATUS.

By default, displays only records from the last week. For different time periods, use the -t option with the -T option.

Use LSB_BHIST_HOURS with -t to display job events before the current time for a longer or shorter period than the default one week.

-w

Wide format. Displays the information in a wide format.

-UF

Displays unformatted job detail information.

This option makes it easy to write scripts for parsing keywords on bhist. The results of this option have no wide control for the output. Each line starts from the beginning of the line. The resource usage message lines that end without any separator have a semicolon added to separate their different parts. The first line and all lines that start with the time stamp are displayed unformatted in a single line. The output has no line length and format control.

-app application_profile_name

Displays information about jobs that are submitted to the specified application profile.

-C start_time,end_time

Displays jobs that completed or exited during the specified time interval. Specify the times in the format yyyy/mm/dd/HH:MM. Do not specify spaces in the time interval string. This option overrides the -r, -s, -p, and -a options.

For more information about the time syntax, see Time interval format.

-D start_time,end_time

Displays jobs that are dispatched during the specified time interval. Specify the times in the format yyyy/mm/dd/HH:MM. Do not specify spaces in the time interval string.

Must be used with the -a option since it finds results only in running jobs.

For more information about the time syntax, see Time interval format.

-f logfile_name | -f -

Searches the specified event log, which is useful for offline analysis. Specify either an absolute or a relative path.

The specified file path can contain up to 4094 characters for UNIX, or up to 255 characters for Windows.

Specify the -f - option to force the bhist command to use the lsb.events log file. If you are using IBM Spectrum LSF Explorer (LSF Explorer) to load event log records, the -f - option (or any -f argument that specifies a log file) forces the bhist command to bypass LSF Explorer.

Note: The bhist -cname, -t, and -T options always bypass LSF Explorer and instead get the information from the lsb.events file.

For more details on using LSF Explorer to load event log records, refer to the LSF_QUERY_ES_SERVERS and LSF_QUERY_ES_FUNCTIONS parameters in the lsf.conf file.

-G user_group

Displays jobs that are associated with a user group that is submitted with the bsub -G command for the specified user group. The -G option does not display jobs from subgroups within the specified user group.

The -G option cannot be used together with the -u option. You can specify only a user group name. The keyword all is not supported for the -G option.

-J job_name

Displays the jobs that have the specified job name.

The job name can be up to 4094 characters long. Job names are not unique.

The wildcard character (*) can be used anywhere within a job name, but it cannot appear within an array index. For example, the pattern job* returns jobA and jobarray[1]. The *AAA*[1] pattern returns the first element in job arrays with names that contain AAA. However, the pattern job1[*] does not return anything since the wildcard is within the array index.

-Jd "job_description"

Displays the jobs that have the specified job description.

The job description can be up to 4094 characters long. Job descriptions are not unique.

The wildcard character (*) can be used anywhere within a job description.

-Lp ls_project_name

Displays information about jobs that belong to the specified LSF License Scheduler project.

-m "host_name..."

Displays jobs that are dispatched to the specified host.

-n number_logfiles | -n min_logfile, max_logfile | -n 0

Searches the specified number of event logs, starting with the current event log and working through the most recent logs in consecutive order. Specify 0 to specify all the event log files in $(LSB_SHAREDIR)/cluster_name/logdir, up to a maximum of 100 files.

If you delete a file, you break the consecutive numbering, and older files are inaccessible to the bhist command. For example, if you specify 3, LSF searches the lsb.events, lsb.events.1, and lsb.events.2 files. If you specify 4, LSF searches the lsb.events, lsb.events.1, lsb.events.2, and lsb.events.3 files. However, if the lsb.events.2file is missing, both searches include only the lsb.events and lsb.events.1 files.

-n number_blocks (LSF Explorer only)

If you are using LSF Explorer to load event log records, use the -n option to control how many job records (number of block records) that LSF Explorer returns. The block size is configured in LSF Explorer. For more details on using LSF Explorer to load event log records, refer to the LSF_QUERY_ES_SERVERS and LSF_QUERY_ES_FUNCTIONS parameters in the lsf.conf file.

-N host_name | -N host_model | -N cpu_factor

Normalizes CPU time by the specified CPU factor, or by the CPU factor of the specified host or host model.

If you use the bhist command directly on an event log, you must specify a CPU factor.

Use the lsinfo command to get host model and CPU factor information.

-P project_name

Displays information about jobs that belong to the specified project.

-q queue_name

Displays information about jobs that are submitted to the specified queue.

-S start_time,end_time

Displays information about jobs that are submitted during the specified time interval. Specify the times in the format yyyy/mm/dd/HH:MM. Do not specify spaces in the time interval string.

Must be used with the -a option since it finds results only in running jobs.

For more information about the time syntax, see Time interval format.

-T start_time,end_time

Used together with the -t option.

Displays information about job events within the specified time interval. Specify the times in the format yyyy/mm/dd/HH:MM. Do not specify spaces in the time interval string.

For more information about the time syntax, see Time interval format.

-u user_name | -u all

Displays information about jobs that are submitted by the specified user, or by all users if the keyword all is specified. To specify a Windows user account, include the domain name in uppercase letters and use a single backslash in a Windows command (DOMAIN_NAME\user_name), or a double backslash in a UNIX command (DOMAIN_NAME\\user_name).

job_ID | "job_ID[index]" ...

Searches all event log files and displays only information about the specified jobs. If you specify a job array, displays all elements chronologically.

You specify job ID when you know exactly which jobs you want, so do not specify any other options that control job selection (-a, -d, -e, -p, -r, -s, -D, -S, -T, -app, -G, -J, -Jd, -Lp, -M, -q, -u). If you specify an illogical combination of selection criteria, the system does not return any matching jobs.

In LSF multicluster capability job forwarding mode, you can use the local job ID and cluster name to retrieve the job details from the remote cluster. Use the following query syntax:

bhist submission_job_id@submission_cluster_name

For job arrays, use the following query syntax:

bhist "submission_job_id[index]"@submission_cluster_name

The advantage of using src_job_id@src_cluster_name instead of bhist -l job_id is that you can use src_job_id@src_cluster_name as an alias to query a local job in the execution cluster without knowing the local job ID in the execution cluster. The bhist command output is identical no matter which job ID you use (local job ID or src_job_id@src_cluster_name).

You can use the bhist 0 option to find all historical jobs in your local cluster, but the bhist 0@submission_cluster_name option is not supported.

-h

Prints command usage to stderr and exits.

-V

Prints release version to stderr and exits.

Use the -l -aff option to display historical job information about CPU and memory affinity resource requirements for job tasks. A table with the heading AFFINITY is displayed containing the detailed affinity information for each task, one line for each allocated processor unit. CPU binding and memory binding information are shown in separate columns in the display.

HOST

The host the task is running on.

TYPE

Requested processor unit type for CPU binding. One of numa, socket, core, or thread.

LEVEL

Requested processor unit binding level for CPU binding. One of numa, socket, core, or thread. If no CPU binding level is requested, a dash (-) is displayed.

EXCL

Requested processor unit binding level for exclusive CPU binding. One of numa, socket, or core. If no exclusive binding level is requested, a dash (-) is displayed.

IDS

List of physical or logical IDs of the CPU allocation for the task.

The list consists of a set of paths, represented as a sequence of integers separated by slash characters (/), through the topology tree of the host. Each path identifies a unique processing unit that is allocated to the task. For example, a string of the form 3/0/5/12 represents an allocation to thread 12 in core 5 of socket 0 in NUMA node 3. A string of the form 2/1/4represents an allocation to core 4 of socket 1 in NUMA node 2. The integers correspond to the node ID numbers displayed in the topology tree from the bhosts -aff command.

POL

Requested memory binding policy. Either local or pref. If no memory binding is requested, a dash (-) is displayed.

NUMA

ID of the NUMA node that the task memory is bound to. If no memory binding is requested, a dash (-) is displayed.

SIZE

Amount of memory that is allocated for the task on the NUMA node.

For example, the following job starts six tasks with the following affinity resource requirements:

bsub -n 6 -R"span[hosts=1] rusage[mem=100]affinity[core(1,same=socket,
exclusive=(socket,injob)):cpubind=socket:membind=localonly:distribute=pack]" myjob
Job <6> is submitted to default queue <normal>.

bhist -l -aff 6

Job <6>, User <user1>, Project <default>, Command <myjob>
Thu Feb 14 14:13:46: Submitted from host <hostA>, to Queue <normal>, CWD <$HO
                     ME>, 6 Task(s), Requested Resources <span[hos
                     ts=1] rusage[mem=100]affinity[core(1,same=socket,exclusive
                     =(socket,injob)):cpubind=socket:membind=localonly:distribu
                     te=pack]>;
Thu Feb 14 14:15:07: Dispatched 6 Task(s) on Host(s) <hostA> <hostA> <hostA>
                     <hostA> <hostA> <hostA>; Allocated <6> Slot(s) on Host(s)
                     <hostA> <hostA> <hostA> <hostA> <hostA> <hostA>;
                     Effective RES_REQ <select[type == local] order[r15s:pg] 
                     rusage[mem=100.00] span[hosts=1] affinity [core(1,same=
                     socket,exclusive=(socket,injob))*1:cpubind=socket:membind=
                     localonly:distribute=pack] >;

AFFINITY:
                    CPU BINDING                          MEMORY BINDING
                    ------------------------             --------------------
HOST                TYPE   LEVEL  EXCL   IDS             POL   NUMA SIZE
hostA               core   socket socket /0/0/0          local 0    16.7MB
hostA               core   socket socket /0/1/0          local 0    16.7MB
hostA               core   socket socket /0/2/0          local 0    16.7MB
hostA               core   socket socket /0/3/0          local 0    16.7MB
hostA               core   socket socket /0/4/0          local 0    16.7MB
hostA               core   socket socket /0/5/0          local 0    16.7MB

Thu Feb 14 14:15:07: Starting (Pid 3630709);
Thu Feb 14 14:15:07: Running with execution home </home/jsmith>, Execution CWD
                     </home/jsmith>, Execution Pid <3630709>;
Thu Feb 14 14:16:47: Done successfully. The CPU time used is 0.0 seconds;
Thu Feb 14 14:16:47: Post job process done successfully;

MEMORY USAGE:
MAX MEM: 2 Mbytes;  AVG MEM: 2 Mbytes; MEM EFFICIENCY: 100%

CPU USAGE:

CPU PEAK: 4.24 ;  CPU PEAK DURATION: 54 second(s)
CPU AVERAGE EFFICIENCY: 99.55%;  CPU PEAK EFFICIENCY: 106.02%

Summary of time in seconds spent in various states by  Thu Feb 14 14:16:47
  PEND     PSUSP    RUN      USUSP    SSUSP    UNKWN    TOTAL
  81       0        100      0        0        0        181

Use the -l -data option to display detailed information about jobs with data requirements. The heading DATA REQUIREMENTS is displayed followed by a list of the files or tags that are requested by the job, and any modifications made to the data requirements.

bhist -data -l 84046
Job <84046>, User <user1>, Project <default>, Command <bstage out -src /home/
                     user1/data2; bstage out -src /home/user1/data2 -dst /tmp; 
                     datajob1.sh>
Mon Aug 18 15:06:57: Submitted from host <hostA>, to Queue <normal>, CWD </sc
                     ratch/user1/workspace/simulation/data/>, Data Require
                     ment Requested;

DATA REQUIREMENTS:

 FILE: hostA:/home/user1/data2
 SIZE: 40 MB
 MODIFIED: Thu Aug 14 17:01:57

 FILE: hostA:/home/user1/data3
 SIZE: 45 MB
 MODIFIED: Fri Aug 15 16:32:45
Mon Aug 18 15:07:07: Dispatched to <hostB>, Effective RES_REQ <select[type == 
                     local] order[r15s:pg] >;
Mon Aug 18 15:07:07: Starting (Pid 16128);
Mon Aug 18 15:07:07: Running with execution home </home/user1>, Execution CWD <
                     /scratch/user1/workspace/simulation/data/>, Execution
                     Pid <16128>;
Mon Aug 18 15:08:47: Done successfully. The CPU time used is 0.4 seconds;
Mon Aug 18 15:08:47: Post job process done successfully;

MEMORY USAGE:
MAX MEM: 3 Mbytes;  AVG MEM: 2 Mbytes; MEM EFFICIENCY: 100%


CPU USAGE:

CPU PEAK: 4.24 ;  CPU PEAK DURATION: 54 second(s)
CPU AVERAGE EFFICIENCY: 99.55%;  CPU PEAK EFFICIENCY: 106.02%
      
Summary of time in seconds spent in various states by  Mon Aug 18 15:08:47
  PEND     PSUSP    RUN      USUSP    SSUSP    UNKWN    TOTAL
  10       0        100      0        0        0        110

Use the -l -hostfile option to display a user-specified host file that was submitted with a job or added to a job with the bmod command.

bhist -l -hostfile  1976
Job <1976>, User <user1>, Project <default>, Command <my_data_job>
Fri Sep 20 16:31:17: Submitted from host <hostA>, to 
           Queue <normal>, CWD <$HOME/source/user1/work>, 
           Host file </home/user4/myhostfile>;

Summary of time in seconds spent in various states by  Wed Sep 25 10:50:37
PEND     PSUSP    RUN      USUSP    SSUSP    UNKWN    TOTAL
21305      0        0        0        0        0      21305
USER-SPECIFIED HOST FILE:
HOST            SLOTS
host01            3
host02            1
host01            1
host02            2
host03            1