Storage

Celebrating a silver anniversary: IBM Spectrum LSF

Share this post:

This week we celebrate the release of our second agile update to IBM Spectrum LSF 10. And it’s our silver anniversary… 25 years of IBM Spectrum LSF!

As we previously discussed in this blog post, IBM Spectrum LSF 10 focused primarily on architectural changes to support future growth in performance and scalability. We’ve seen steady adoption of LSF 10, and client feedback has been very positive, with peak submission rates trebling and peak dispatch more than doubling with LSF 10 when compared to LSF 9.1.3.

lsf 10 chart, IBM Spectrum LSF 10

Advancing IBM Spectrum LSF 10

In this update, we’re primarily focused on usability.

When we added the “-o” option in LSF 9 to customize the output of bjobs, clients loved it — and it generated a whole slew of requests for additional fields and additional support with other commands.  So in this update we’ve added numerous additional fields and added “-o” to most of the query commands — we didn’t quite get them all completed with this update, so expect some more in the next update! We also added a -json option for this who like their output in JSON format, making it easy to be consumed by something else.

“Is my job running? Is my job running? Is my job running?” Folks often have scripts that constantly loop over the bjobs output waiting for specific job/job states before doing something else. Apart from this putting unnecessary load on the network, it’s just burning CPU cycles on the client side.  What’s more, these looping scripts are often easily broken by unexpected messages back from the scheduler — so not very environmentally friendly or robust.  To help make things simpler, we’ve introduced a new “bwait” command that effectively makes the “Is my job running” loop redundant.  The user can issue a bwait with a list of jobs and dependencies and the script will block until it receives a call back from the scheduler telling it when the condition is true.

To make life easier for the administrator, we’ve also added some cognitive capabilities to autotune the scheduler.  We’ve added a couple of example scripts for how to run TensorFlow and Caffe as Spectrum LSF jobs – though you would think these cognitive frameworks can figure that out for themselves!

Finally, we’ve added formal support for NVIDIA DCGM on both IBM POWER and x86.  This provides a better experience with GPU workloads with GPU health monitoring, integrated GPU accounting and intelligent topology-aware job placement of applications onto healthy GPUs.

So what’s next? The third update is due in July and will contain, amongst other things, additional GPGPU support and further enhancements to support containerized workloads.

We’re only 25 years young and just getting started.  Are you looking to learn more about the complete HPC workload management capabilities of IBM Spectrum LSF? Give it a try! Learn more about IBM Spectrum LSF evaluations here.

More Storage stories

A future of powerful clouds

Hybrid cloud storage, Multicloud, Storage

In a very good way, the future is filled with clouds. In the realm of information technology, this statement is especially true. Already, the majority of organizations worldwide are taking advantage of more than one cloud provider.[1] IBM calls this a “hybrid multicloud” environment – “hybrid” meaning both on- and off-premises resources are involved, and ...read more


Accelerating data for NVIDIA GPUs

AI, Big data & analytics, Storage

These days, most AI and big data workloads need more compute power and memory than one node can provide. As both the number of computing nodes and the horsepower of processors and GPUs increases, so does the demand for I/O bandwidth. What was once a computing challenge can now become an I/O challenge. For those ...read more


Secure, efficient and high-performance deployment on premises and in the cloud

Flash storage, Multicloud, Storage

In Part one of this 2-part post on IBM Storage for hybrid multicloud, I discussed the two pieces of software at the center of our approach. The first is IBM Storage Insights, a free cloud-based service that simplifies your operations. The second is IBM Spectrum Virtualize, the strategic storage software foundation. In Part 2, we’ll ...read more