What's new in IBM Spectrum LSF Version 10.1 Fix Pack 14
The following topics summarize the new and changed behavior in LSF 10.1 Fix Pack 14.
Download this Fix Pack from IBM Fix Central. For more information, see Getting fixes from IBM Fix Central.
Release date: June 2023.
Operating system versions
When a specific release or version of an operating system reaches end of life (EOL) or its end of support date, the operating system vendor no longer supports and releases updates for the product, including automatic fixes, updates, and online technical assistance.
LSF, in turn, is not tested on EOL operating systems. If you have extended support to continue to use EOL operating systems, you can use these operating system versions with LSF. Any issue with using LSF will be supported only if it can be reproduced on a supported operating system.
- Ubuntu 20.04 is newly certified for Linux kernel on IBM Power LE.
- Ubuntu 22.04 is newly certified for Linux x64 kernel and Linux kernel on IBM Power LE.
Also note for Ubuntu 22.04 support, the LSF installer has been refreshed on IBM Passport Advantage. If you have not or cannot run the new LSF installer, update the
profile.lsf
andcshrc.lsf
environment shell files to use LSF on Ubuntu 22.04, according to the steps in the Fix Pack 14 readme file.
New LSF rate limiter
mbatchd
daemon, preventing stress on the cluster,
and improving performance.
The
rate
limiter is
supported on Linux.- Leverage the rate limiter to tune performance
- The rate
limiter acts as a
gatekeeper between commands and the
mbatchd
daemon.Furthermore, to allow an administrator to temporarily block non-administrator and non-root users, hosts, or both, from performing
mbatchd
daemon operations when using the rate limiter, the badmin command has been extended to support badmin lsfproxyd block. Administrators can run this command to temporarily stop abusive or misbehaving users from interacting with the LSF cluster, and to avoid performance impact on other users. - Enable diagnostic logs to monitor the rate limiter
- Similar to using diagnostic logs to enable
mbatchd
to write query source information to a log file, you can now do the same for the new LSFrate limiter:lsfproxyd
. The new query_info.queryproxylog.hostname log file shows information about the source oflsproxyd
requests, allowing you to troubleshoot problems. The log file shows who issued these requests, where the requests came from, the data size of the query, the batch operation code, whether the request was rejected or accepted, and the time that thelsfproxyd
daemon receives and processes the requests. Configuration for thelsfproxyd
daemon logging is also the same as formbatchd
diagnostic logs.
Mark dynamic hosts as expired and remove them from the cluster
LSF administrators can mark dynamic hosts that are no longer available, as expired, using the new lsadmin expire command. Furthermore, if you have configured the LSF_DYNAMIC_HOST_TIMEOUT parameter, in the lsf.conf configuration file, then the hosts marked as expired will be automatically removed from the LSF cluster, cleaning up the environment so extra hosts do not affect cluster performance. In Fix Pack 14, the LSF_DYNAMIC_HOST_TIMEOUT has been enhanced to support more cleanup timeout options.
Global job IDs for using an LSF multicluster environment
Global job IDs allow an LSF multicluster environment to use the same job IDs between the forwarding and forwarded clusters, keeping the IDs uniform. These global job IDs are unique. To guarantee unique job IDs, LSF introduces indexes for clusters, so that each job submitted from the cluster includes an index to the ending digits of the job ID. To configure global job IDs for your LSF multicluster environment, see Global job IDs for forwarding and forwarded clusters using LSF multicluster capability.
IBM Spectrum LSF Data Manager supports cross-cluster user account mapping for a multicluster environment
LSF Data Manager now supports cross-cluster user account mapping for a multicluster environment, allowing cross-cluster job submission and execution for a multicluster environment which has different user accounts assigned to different hosts. If you have LSF Data Manager installed, and enabled it for multicluster, you can additionally enable this cross-cluster user account mapping, configure your lsb.users configuration file on the remote and local clusters in your multicluster environment.
Support MIG device isolation
LSF now leverages cgroups to enforce Nvidia Multi-Instance GPU (MIG) device isolation. Set this enforcement by configuring LSB_RESOURCE_ENFORCE="gpu" in the lsf.conf configuration file.
Support DCGM version 3.1.8
Fix Pack 14 supports Nvidia Data Center GPU Manager (DCGM) version 3.1.8. Note, however, that Nvidia Multi-Instance GPU (MIG) integration with DCGM does not work with LSF.
Higher priority GPU jobs can now preempt shared-mode GPU jobs if there are multiple jobs running on the GPU
Prior to Fix Pack 14, higher priority GPU jobs with j_exclusive=yes
or
mode=exclusive_process
settings could not preempt shared-mode GPU jobs, if there
were multiple jobs running on the GPU. With Fix Pack 14, this restriction is removed so that higher
priority jobs can preempt other jobs in this scenario. For details about submitting jobs that
require GPU resources, see the LSB_GPU_NEW_SYNTAX parameter for the lsf.conf
configuration file.
Support for key LSF daemons to restart upon failure
The existing /usr/lib/systemd/system/lsfd.service file is used to start,
stop, or restart key LSF
daemons (LIM
, RES
, and sbatchd
) as a group, not
for each daemon. For example, if only the LIM
daemon fails (core dumps or
abnormally exits), the lsfd.service file cannot restart just the one
daemon.
Fix Pack 14 introduces enhanced behavior for the lsfd.service file: there is now a child service file for each LSF daemon (lsfd-lim.service, lsfd-res.service, and lsfd-sbd.service), to handle each daemon separately. If any of the key LSF daemons fail, each daemon can now automatically restart.
Note that if LSF is already installed on a server host, to leverage this enhanced behavior for the lsfd.service file, apply Fix Pack 14 and follow the steps in the Fix Pack 14 readme file (SPCLSF-866 section).
The systemctl command manages Linux systemd
services,
including starting, stopping, and restarting the LSF
daemons. For more information about the systemctl command, see LSF system daemon commands. Note that once you use
systemctl commands to control these daemons, do not use the existing
lsadmin limrestart, lsadmin resrestart, or badmin
hrestart control commands. Use the systemctl commands instead.
Automatically create cgroups for a job, without affinity[] requirements
Previously, LSF
created a cgroup for a job only when you explicitly specify
affinity[] requirements. With the new LSF_CGROUP_CORE_AUTO_CREATE parameter in the
lsf.conf configuration file, you can now enable LSF to
automatically create Linux cgroups for a job. With this setting, LSF
automatically adds the "affinity[core(1)]"
resource requirement string to the
current job's resource requirement whenever jobs are submitted.
Assign resource values to be passed into dispatched jobs
Administrators can now assign a specific resource value when defining and mapping shared
resources in a cluster. This way, when jobs are dispatched, they are associated with the assigned
resource requirement. For example, consider an FPGA device as a shared hardware resource, where you
have three types of FPGA devices: card1
, card2
, and
card3
; you can assign card1
, card2
, and
card3
to the fpga
resource using the ResourceMap
section of the lsf.cluster.cluster_name configuration file
(see lsf.cluster for an example
ResourceMap
section). The file has now been extended to enable LSF to
assign specific values to resources, rather than only accept the number of resources.
Note
that in Docker or Podman, the characters !@#$%^&*()+-.
are not allowed in
environment variable names. To use this feature within a Docker or Podman environment, ensure that
your host names and resource names do not contain these characters. Specifically, for hostnames, you
can ensure that the LSF_STRIP_DOMAIN parameter in the lsf.conf
configuration file is set to remove all periods (.
) from the hostnames if a period
exists, and manually ensure that all resource names do not include unsupported characters.
- Running the bjobs -l or bhist -l command now shows the names assigned to a resource, which dispatches with jobs. The information will be in the header of the job in this format. For example, Assigned resource <fpga___hostname> with names <card1 card2 card3>
- The bhosts and lshosts commands now support a new -sl option to show the resource names assignments.
Support to display all field names for -o option in bhosts, bjobs, bqueues, and busers commands
You can now specify a new all keyword for the -o custom output option when running the bhosts, bjobs, bqueues, or busers commands. Specifying -o "all" indicates for LSF to display all (versus specific) field names when showing the output for these commands.
New BSUB_QUIET values for bsub command job submission output
The BSUB_QUIET
environment variable, which controls job submission printed
output for the bsub command, now supports new values to print the job ID only,
and to print both the job ID and queue type. See BSUB_QUIET for details on these new BSUB_QUIET
values.
New host memory output fields for the bhosts command and the -o option
- available_mem displays the total currently available memory on the server (if processes outside of LSF uses memory on that server, this value decreases).
- reserved_mem displays the currently reserved memory on a host.
- total_mem displays the total memory available and reserved by LSF on a host.
More CPU and memory usage details added to bjobs, bhist, and bacct commands
- You can now review the average CPU efficiency, CPU peak usage, duration to get to the CPU peak
for a job, and CPU peak efficiency values, and see this level of detail when you run the bjobs
-l, bhist
-l, or bacct
-l commands. Refer to the to the
CPU USAGE
andAccounting information about this job
sections in the output for these commands to see the new CPU usage details. Additionally, you can see memory usage efficiency values in the output. - You can also see the average CPU efficiency, or the duration to reach the CPU peak, for a job by
running the bjobs
-o command with these new field name options:
- bjobs -o average_cpu_efficiency
- bjobs -o cpu_peak_reached_duration
- The
cpu_efficiency
field name for the bjobs -o command now has a more descriptive name:cpu_peak_efficiency
, which differentiates it from the other CPU usage field names. To see the value, run bjobs -o cpu_peak_efficiency if you are using Fix Pack 14 or later.
Support for the LSB_CLEAN_PDONE_TIMEOUT parameter for the lsb.params file
LSF cleans finished jobs in PDONE state; however, some jobs can remain in DONE state and never move into PDONE state and remain in mbatchd core memory, increasing the lsb.event file size unnecessarily. Use the LSB_CLEAN_PDONE_TIMEOUT parameter to indicate the maximum amount of time to wait, before cleaning out the finished jobs that remain in DONE state that cannot move to PDONE state. LSF multiplies the value of LSB_CLEAN_PDONE_TIMEOUT by the number of CLEAN_PERIOD seconds (defined in the lsb.params configuration file), to determine the number of seconds to wait before the mbatchd daemon cleans those jobs from core memory.
You can run the bparams -a or bparams -l command to view the LSB_CLEAN_PDONE_TIMEOUT parameter as a configurable parameter in the lsb.params file.
New MAX_PREEXEC_FORWARDING_RETRY parameter for the lsb.params file
Fix Pack 14 introduces a new MAX_PREEXEC_FORWARDING_RETRY parameter to the lsb.params configuration file, which at the submission cluster, controls the maximum number of times to attempt forwarding a job to a remote cluster, if the job has been rejected due to the pre-execution command of the job.
If the job's pre-execution command fails all attempts, the job is recalled by the submission cluster.
If the job fails all forwarding attempts, the job is suspended or terminated based on the LOCAL_MAX_PREEXEC_RETRY_ACTION setting. If the suspended job resumes, the counter will be reset
You can run the bparams -a or bparams -l command to view the MAX_PREEXEC_FORWARDING_RETRY parameter as a configurable parameter in the lsb.params file.
New MAX_NUM_JOBS_KILL_OVER_RUNLIMIT parameter for the lsb.params configuration file
If many (such as thousands of) jobs reach their run limit, it takes time for the
mbatchd
daemon to kill those jobs using the
KILL_JOBS_OVER_RUNLIMIT parameter. Fix Pack 14 introduces a new parameter to
the lsb.params file, called MAX_NUM_JOBS_KILL_OVER_RUNLIMIT, which controls the maximum number of
jobs that can be killed by KILL_JOBS_OVER_RUNLIMIT in one cycle. This prevents
the mbatchd
daemon from potentially hanging when there are too many jobs to handle.
New default values for the LSF_DATA_SCP_CMD and LSF_DATA_SSH_CMD parameters within the lsf.conf configuration file
LSF_DATA_SCP_CMD="scp -q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o BatchMode=yes -r -p"
LSF_DATA_SSH_CMD="ssh -q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o BatchMode=yes"
Previously, the LSF_DATA_SCP_CMD and LSF_DATA_SSH_CMD
parameters affected job transfer scripts. As of Fix Pack 14, however, these parameters no
longer affect job transfer scripts; they affect the bstage command when the
staging area is not mounted on the execution host of the user's job instead. For an LSF
administrator to change how job transfer scripts use scp
or ssh
,
they must modify the job transfer scripts directly.
New
RC_ACCOUNT and VM_TYPE fields for the
JOB_FINISH2
LSF event
The JOB_FINISH2
LSF event contains details about LSF
resource connector jobs. The LSF
lsb.stream file can then capture and stream actions about the
JOB_FINISH2
event. To provide more details in the JOB_FINISH2
event logs, LSF now
includes the RC_ACCOUNT and VM_TYPE fields with the
JOB_FINISH2
event. For details on how to enable these new fields, see Viewing LSF resource connector job events.
LSF resource connector supports Amazon Web Services (AWS) EC2 Fleet
The LSF resource connector for Amazon Web Services (AWS) now uses an Amazon EC2 Fleet API to create groups (or a fleet) of instances. EC2 Fleet is an AWS feature that extends the existing spot fleet, which gives you a unique ability to create fleets of EC2 instances composed of a combination of EC2 on-demand, reserved, and spot instances, by using a single API. For configuration details, see Using Amazon EC2 Fleet.
IBM Cloud Gen 2 provider capacity error codes added to the LSB_RC_TEMPLATE_REQUEST_DELAY parameter in the lsf.conf file
The LSB_RC_TEMPLATE_REQUEST_DELAY parameter in the lsf.conf configuration file allows you to define the number of minutes that LSF waits before repeating a request for a template, if the ebrokerd daemon encounters provider errors in a previous request. The parameter has been enhanced for the VPC Gen 2 (IBM Cloud Gen 2) API, to include several IBM Cloud Gen 2 capacity error codes.