Storage

Train your AI, feed your head

Share this post:

Machine learning.  Deep learning.  AI, or artificial intelligence.  Since I read the book The Computer and the Brain written by the mathematician John von Neumann, comparisons between the human brain and computers have always fascinated me.  Hardly a day goes by that we don’t hear about the promise of AI, and how it will help to alleviate mundane tasks by making sense of the massive oceans of data bursting storage at the seams worldwide.

Much like a child learning to ride a bicycle using training wheels, deep learning networks too are trainable.  As I recently discovered while tinkering with a well-known deep learning framework, training is very compute intensive and is certainly the type of task that you’d want to schedule intelligently across your HPC infrastructure.

Train smart

So why not apply HPC scheduling best practices to the training of deep learning networks?  IBM Spectrum LSF is a workload scheduler for demanding HPC environments.  Since its inception more than 20 years ago, its scheduling brain has been accelerating workloads throughout the HPC world with intelligent policies to make the best possible use of available resources.  By continuously monitoring the load across the computing environment, IBM Spectrum LSF makes lightning-fast decisions to place the right work in the right place at the right time.  So why not apply these scheduling smarts to the task of training deep learning networks?

IBM Spectrum LSF supports training deep learning frameworks BVLC Caffe and TensorFlow alongside traditional HPC workloads with ease.  Find out more about IBM Spectrum LSF and deep learning framework integrations here.

Not only can IBM Spectrum LSF be used to help accelerate training deep learning networks, but machine learning can also be used to help simplify HPC by specifying memory requirements for IBM Spectrum LSF jobs.  Learn more in this IBM Research paper here.

And to paraphrase Jefferson Airplane in White Rabbit, “Remember what the dormouse said, “Feed your [AI] head, feed your [AI] head.” Do it with some help from IBM Spectrum LSF.

More Storage stories

Powerful new storage for your mission-critical hybrid multicloud

Flash storage, Multicloud, Storage

Cloud computing and mainframes processors are two of the most important contributors in information technology solutions. They are increasingly being linked together to unlock value for clients. 85 percent of companies are already operating in multicloud and by 2021, most existing apps will have migrated to the cloud. [1] At the same time, mainframe utilization ...read more


A conversation about metadata management featuring Forrester Analyst Michele Goetz

AI, Big data & analytics, Storage

I recently had an opportunity to speak with Forrester Principal Analyst Michele Goetz following a research study on the subject of metadata management conducted by Forrester Consulting and commissioned by IBM. Michele’s research covers artificial intelligence technologies and consultancies, semantic technology, data management strategy, data governance and data integration, and includes within its scope the ...read more


IBM is transforming data storage for media and entertainment

Data security, Storage, Tape and virtual tape storage

Disruptions in media and entertainment have been occurring at a rapid pace over the past few years and many organizations are still struggling with optimizing their IT infrastructure for new data requirements. With growth in capacity, new larger file formats, keeping more data online for business opportunities, and the growing use of AI, organizations are ...read more