What's new in IBM Spectrum LSF Version 10.1 Fix Pack 14

The following topics summarize the new and changed behavior in LSF 10.1 Fix Pack 14.

Download this Fix Pack from IBM Fix Central. For more information, see Getting fixes from IBM Fix Central.

Release date: June 2023.

Operating system versions

When a specific release or version of an operating system reaches end of life (EOL) or its end of support date, the operating system vendor no longer supports and releases updates for the product, including automatic fixes, updates, and online technical assistance.

LSF, in turn, is not tested on EOL operating systems. If you have extended support to continue to use EOL operating systems, you can use these operating system versions with LSF. Any issue with using LSF will be supported only if it can be reproduced on a supported operating system.

Before applying Fix Pack 14, ensure that you use one of the supported operating systems. Here are the highlights of LSF operating system changes as of Fix Pack 14:
  • Ubuntu 20.04 is newly certified for Linux kernel on IBM Power LE.
  • Ubuntu 22.04 is newly certified for Linux x64 kernel and Linux kernel on IBM Power LE.

    Also note for Ubuntu 22.04 support, the LSF installer has been refreshed on IBM Passport Advantage. If you have not or cannot run the new LSF installer, update the profile.lsf and cshrc.lsf environment shell files to use LSF on Ubuntu 22.04, according to the steps in the Fix Pack 14 readme file.

New LSF rate limiter

LSF now supports a rate limiter, to limit the rate of excessive requests to the mbatchd daemon, preventing stress on the cluster, and improving performance. The rate limiter is supported on Linux.
Leverage the rate limiter to tune performance
The rate limiter acts as a gatekeeper between commands and the mbatchd daemon.

Furthermore, to allow an administrator to temporarily block non-administrator and non-root users, hosts, or both, from performing mbatchd daemon operations when using the rate limiter, the badmin command has been extended to support badmin lsfproxyd block. Administrators can run this command to temporarily stop abusive or misbehaving users from interacting with the LSF cluster, and to avoid performance impact on other users.

Enable diagnostic logs to monitor the rate limiter
Similar to using diagnostic logs to enable mbatchd to write query source information to a log file, you can now do the same for the new LSFrate limiter: lsfproxyd. The new query_info.queryproxylog.hostname log file shows information about the source of lsproxyd requests, allowing you to troubleshoot problems. The log file shows who issued these requests, where the requests came from, the data size of the query, the batch operation code, whether the request was rejected or accepted, and the time that the lsfproxyd daemon receives and processes the requests. Configuration for the lsfproxyd daemon logging is also the same as for mbatchd diagnostic logs.

Mark dynamic hosts as expired and remove them from the cluster

LSF administrators can mark dynamic hosts that are no longer available, as expired, using the new lsadmin expire command. Furthermore, if you have configured the LSF_DYNAMIC_HOST_TIMEOUT parameter, in the lsf.conf configuration file, then the hosts marked as expired will be automatically removed from the LSF cluster, cleaning up the environment so extra hosts do not affect cluster performance. In Fix Pack 14, the LSF_DYNAMIC_HOST_TIMEOUT has been enhanced to support more cleanup timeout options.

Global job IDs for using an LSF multicluster environment

Global job IDs allow an LSF multicluster environment to use the same job IDs between the forwarding and forwarded clusters, keeping the IDs uniform. These global job IDs are unique. To guarantee unique job IDs, LSF introduces indexes for clusters, so that each job submitted from the cluster includes an index to the ending digits of the job ID. To configure global job IDs for your LSF multicluster environment, see Global job IDs for forwarding and forwarded clusters using LSF multicluster capability.

IBM Spectrum LSF Data Manager supports cross-cluster user account mapping for a multicluster environment

LSF Data Manager now supports cross-cluster user account mapping for a multicluster environment, allowing cross-cluster job submission and execution for a multicluster environment which has different user accounts assigned to different hosts. If you have LSF Data Manager installed, and enabled it for multicluster, you can additionally enable this cross-cluster user account mapping, configure your lsb.users configuration file on the remote and local clusters in your multicluster environment.

Support MIG device isolation

LSF now leverages cgroups to enforce Nvidia Multi-Instance GPU (MIG) device isolation. Set this enforcement by configuring LSB_RESOURCE_ENFORCE="gpu" in the lsf.conf configuration file.

Support DCGM version 3.1.8

Fix Pack 14 supports Nvidia Data Center GPU Manager (DCGM) version 3.1.8. Note, however, that Nvidia Multi-Instance GPU (MIG) integration with DCGM does not work with LSF.

Higher priority GPU jobs can now preempt shared-mode GPU jobs if there are multiple jobs running on the GPU

Prior to Fix Pack 14, higher priority GPU jobs with j_exclusive=yes or mode=exclusive_process settings could not preempt shared-mode GPU jobs, if there were multiple jobs running on the GPU. With Fix Pack 14, this restriction is removed so that higher priority jobs can preempt other jobs in this scenario. For details about submitting jobs that require GPU resources, see the LSB_GPU_NEW_SYNTAX parameter for the lsf.conf configuration file.

Support for key LSF daemons to restart upon failure

The existing /usr/lib/systemd/system/lsfd.service file is used to start, stop, or restart key LSF daemons (LIM, RES, and sbatchd) as a group, not for each daemon. For example, if only the LIM daemon fails (core dumps or abnormally exits), the lsfd.service file cannot restart just the one daemon.

Fix Pack 14 introduces enhanced behavior for the lsfd.service file: there is now a child service file for each LSF daemon (lsfd-lim.service, lsfd-res.service, and lsfd-sbd.service), to handle each daemon separately. If any of the key LSF daemons fail, each daemon can now automatically restart.

Note that if LSF is already installed on a server host, to leverage this enhanced behavior for the lsfd.service file, apply Fix Pack 14 and follow the steps in the Fix Pack 14 readme file (SPCLSF-866 section).

The systemctl command manages Linux systemd services, including starting, stopping, and restarting the LSF daemons. For more information about the systemctl command, see LSF system daemon commands. Note that once you use systemctl commands to control these daemons, do not use the existing lsadmin limrestart, lsadmin resrestart, or badmin hrestart control commands. Use the systemctl commands instead.

Automatically create cgroups for a job, without affinity[] requirements

Previously, LSF created a cgroup for a job only when you explicitly specify affinity[] requirements. With the new LSF_CGROUP_CORE_AUTO_CREATE parameter in the lsf.conf configuration file, you can now enable LSF to automatically create Linux cgroups for a job. With this setting, LSF automatically adds the "affinity[core(1)]" resource requirement string to the current job's resource requirement whenever jobs are submitted.

Assign resource values to be passed into dispatched jobs

Administrators can now assign a specific resource value when defining and mapping shared resources in a cluster. This way, when jobs are dispatched, they are associated with the assigned resource requirement. For example, consider an FPGA device as a shared hardware resource, where you have three types of FPGA devices: card1, card2, and card3; you can assign card1, card2, and card3 to the fpga resource using the ResourceMap section of the lsf.cluster.cluster_name configuration file (see lsf.cluster for an example ResourceMap section). The file has now been extended to enable LSF to assign specific values to resources, rather than only accept the number of resources.

Note that in Docker or Podman, the characters !@#$%^&*()+-. are not allowed in environment variable names. To use this feature within a Docker or Podman environment, ensure that your host names and resource names do not contain these characters. Specifically, for hostnames, you can ensure that the LSF_STRIP_DOMAIN parameter in the lsf.conf configuration file is set to remove all periods (.) from the hostnames if a period exists, and manually ensure that all resource names do not include unsupported characters.

With Fix Pack 14, you can assign which resources to use, not only how many of each resource. LSF commands provide the following enhanced output and usage:
  • Running the bjobs -l or bhist -l command now shows the names assigned to a resource, which dispatches with jobs. The information will be in the header of the job in this format. For example, Assigned resource <fpga___hostname> with names <card1 card2 card3>
  • The bhosts and lshosts commands now support a new -sl option to show the resource names assignments.

Support to display all field names for -o option in bhosts, bjobs, bqueues, and busers commands

You can now specify a new all keyword for the -o custom output option when running the bhosts, bjobs, bqueues, or busers commands. Specifying -o "all" indicates for LSF to display all (versus specific) field names when showing the output for these commands.

New BSUB_QUIET values for bsub command job submission output

The BSUB_QUIET environment variable, which controls job submission printed output for the bsub command, now supports new values to print the job ID only, and to print both the job ID and queue type. See BSUB_QUIET for details on these new BSUB_QUIET values.

New host memory output fields for the bhosts command and the -o option

The bhosts -o command shows hosts and their static and dynamic resources, in a customized output format. With the -o option, you specify one or more field names to display in the command output, and Fix Pack 14 supports three new field names:
  • available_mem displays the total currently available memory on the server (if processes outside of LSF uses memory on that server, this value decreases).
  • reserved_mem displays the currently reserved memory on a host.
  • total_mem displays the total memory available and reserved by LSF on a host.

More CPU and memory usage details added to bjobs, bhist, and bacct commands

Fix Pack 14 provides more detailed information for CPU and memory usage for running jobs:
  • You can now review the average CPU efficiency, CPU peak usage, duration to get to the CPU peak for a job, and CPU peak efficiency values, and see this level of detail when you run the bjobs -l, bhist -l, or bacct -l commands. Refer to the to the CPU USAGE and Accounting information about this job sections in the output for these commands to see the new CPU usage details. Additionally, you can see memory usage efficiency values in the output.
  • You can also see the average CPU efficiency, or the duration to reach the CPU peak, for a job by running the bjobs -o command with these new field name options:
    • bjobs -o average_cpu_efficiency
    • bjobs -o cpu_peak_reached_duration
  • The cpu_efficiency field name for the bjobs -o command now has a more descriptive name: cpu_peak_efficiency, which differentiates it from the other CPU usage field names. To see the value, run bjobs -o cpu_peak_efficiency if you are using Fix Pack 14 or later.

Support for the LSB_CLEAN_PDONE_TIMEOUT parameter for the lsb.params file

LSF cleans finished jobs in PDONE state; however, some jobs can remain in DONE state and never move into PDONE state and remain in mbatchd core memory, increasing the lsb.event file size unnecessarily. Use the LSB_CLEAN_PDONE_TIMEOUT parameter to indicate the maximum amount of time to wait, before cleaning out the finished jobs that remain in DONE state that cannot move to PDONE state. LSF multiplies the value of LSB_CLEAN_PDONE_TIMEOUT by the number of CLEAN_PERIOD seconds (defined in the lsb.params configuration file), to determine the number of seconds to wait before the mbatchd daemon cleans those jobs from core memory.

You can run the bparams -a or bparams -l command to view the LSB_CLEAN_PDONE_TIMEOUT parameter as a configurable parameter in the lsb.params file.

New MAX_PREEXEC_FORWARDING_RETRY parameter for the lsb.params file

Fix Pack 14 introduces a new MAX_PREEXEC_FORWARDING_RETRY parameter to the lsb.params configuration file, which at the submission cluster, controls the maximum number of times to attempt forwarding a job to a remote cluster, if the job has been rejected due to the pre-execution command of the job.

If the job's pre-execution command fails all attempts, the job is recalled by the submission cluster.

If the job fails all forwarding attempts, the job is suspended or terminated based on the LOCAL_MAX_PREEXEC_RETRY_ACTION setting. If the suspended job resumes, the counter will be reset

You can run the bparams -a or bparams -l command to view the MAX_PREEXEC_FORWARDING_RETRY parameter as a configurable parameter in the lsb.params file.

New MAX_NUM_JOBS_KILL_OVER_RUNLIMIT parameter for the lsb.params configuration file

If many (such as thousands of) jobs reach their run limit, it takes time for the mbatchd daemon to kill those jobs using the KILL_JOBS_OVER_RUNLIMIT parameter. Fix Pack 14 introduces a new parameter to the lsb.params file, called MAX_NUM_JOBS_KILL_OVER_RUNLIMIT, which controls the maximum number of jobs that can be killed by KILL_JOBS_OVER_RUNLIMIT in one cycle. This prevents the mbatchd daemon from potentially hanging when there are too many jobs to handle.

New default values for the LSF_DATA_SCP_CMD and LSF_DATA_SSH_CMD parameters within the lsf.conf configuration file

As of Fix Pack 14, the default values for the LSF_DATA_SCP_CMD and LSF_DATA_SSH_CMD parameters have changed as follows:
  • LSF_DATA_SCP_CMD="scp -q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o BatchMode=yes -r -p"
  • LSF_DATA_SSH_CMD="ssh -q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o BatchMode=yes"

Previously, the LSF_DATA_SCP_CMD and LSF_DATA_SSH_CMD parameters affected job transfer scripts. As of Fix Pack 14, however, these parameters no longer affect job transfer scripts; they affect the bstage command when the staging area is not mounted on the execution host of the user's job instead. For an LSF administrator to change how job transfer scripts use scp or ssh, they must modify the job transfer scripts directly.

New RC_ACCOUNT and VM_TYPE fields for the JOB_FINISH2 LSF event

The JOB_FINISH2 LSF event contains details about LSF resource connector jobs. The LSF lsb.stream file can then capture and stream actions about the JOB_FINISH2 event. To provide more details in the JOB_FINISH2 event logs, LSF now includes the RC_ACCOUNT and VM_TYPE fields with the JOB_FINISH2 event. For details on how to enable these new fields, see Viewing LSF resource connector job events.

LSF resource connector supports Amazon Web Services (AWS) EC2 Fleet

The LSF resource connector for Amazon Web Services (AWS) now uses an Amazon EC2 Fleet API to create groups (or a fleet) of instances. EC2 Fleet is an AWS feature that extends the existing spot fleet, which gives you a unique ability to create fleets of EC2 instances composed of a combination of EC2 on-demand, reserved, and spot instances, by using a single API. For configuration details, see Using Amazon EC2 Fleet.

IBM Cloud Gen 2 provider capacity error codes added to the LSB_RC_TEMPLATE_REQUEST_DELAY parameter in the lsf.conf file

The LSB_RC_TEMPLATE_REQUEST_DELAY parameter in the lsf.conf configuration file allows you to define the number of minutes that LSF waits before repeating a request for a template, if the ebrokerd daemon encounters provider errors in a previous request. The parameter has been enhanced for the VPC Gen 2 (IBM Cloud Gen 2) API, to include several IBM Cloud Gen 2 capacity error codes.