Meeting the data needs of artificial intelligence

Share this post:

Artificial intelligence (AI) is playing an increasingly critical role in business. By 2020, 30 percent of organizations that fail to apply AI will not be operationally and economically viable, according to one report[1].  And in a survey, 91 percent of infrastructure and operations leaders cite “data” as a main inhibitor of AI initiatives[2]. What does a data professional need to know about AI and its data requirements in order to support his or her organization’s AI efforts?

Many factors have converged in recent years to make AI viable, including the growth of processing power and advances in AI techniques, notably in the area of deep learning (DL). Unlike traditional programming in which a programmer provides the computer with each step that it has to take to accomplish some task, deep learning requires the computer to learn for itself.  In the case of visual object recognition, for example, there is no way to program a computer with the steps needed to recognize a given object which may present itself in different locations, at different angles, in different lighting conditions, perhaps partially obscured by some other object and so forth.  Instead, the computer is trained by being given thousands of examples of images containing the object until it can consistently recognize it.

This kind of training requires lots of data. One recommendation is to start with at least 100,000 examples – and each example can be large: an image or a voice recording, for example. Different stages of training and deployment of a deep learning system have different data and processing requirements.  For the training stage, there may be years of data to process, and it can take weeks or even months to complete.   By contrast with these extended time frames, once deployed, the system may need to respond in seconds.

Obviously, given the data volumes involved, storage capacity is an important consideration during the training stage.  The data may also be in different formats in different systems, so multi-protocol capability may be needed. The data may also be geographically dispersed, an additional factor the storage system needs to handle. Once deployed, fast access to the data becomes particularly important to support the response requirements of users and applications, which typically need answers in seconds.

A system such as IBM Spectrum Scale is perfectly suited to meeting these requirements.  It is a high-performance system that can scale out to handle petabytes or exabytes of data.  It supports a wide variety of protocols for accessing files or objects. For Hadoop applications, it provides direct access to data without having to copy the data to HDFS, as is usually required. Avoiding the overhead of copying data between systems lowers costs by saving space and also speeds time to results.

IBM Spectrum Scale is a software-defined solution that can be deployed on a customer’s choice of platform, or it can be delivered as a complete solution in the form of IBM Elastic Storage Server (ESS).  The capacity and performance capabilities of IBM Spectrum Scale and ESS are well illustrated by the US Department of Energy CORAL project, currently on track to build the world’s fastest supercomputer.  ESS will be providing the 250PB of storage the system requires, with performance requirements that include 2.5 TB/second single stream IOR and the creation of 2.6 million 32K files per second.

IBM Spectrum Scale and IBM Elastic Storage Server undergo constant improvement.  The latest version of IBM Spectrum Scale incorporates enhancements to the install and upgrade process, the GUI, and system health capabilities, along with scalability and performance tuning for Transparent Cloud Tiering up to one billion files, and file audit logging enhancements.

Meanwhile, ESS now offers models incorporating the superior performance of IBM Spectrum Scale version 5.0 with performance improvements designed to meet the requirements of the CORAL supercomputer.  ESS is also bringing out its first hybrid models incorporating both flash and disk storage in a single unit, allowing improved handling of different kinds of data such as video and analytics within a single environment.

Constant improvements, along with decades of experience in the most challenging customer environments, ensure that IBM Spectrum Scale and IBM Elastic Storage Server will continue to lead the way in managing the data that is a key element in the success of any deep learning project. Visit our website to learn more about IBM Spectrum Scale and IBM Elastic Storage Server.

[1] Gartner Predicts 2018: Compute Infrastructure 

[2] Gartner AI State of The Market – and Where HPC intersects

More Storage stories

Data protection modernization: What’s it worth?

Integrated infrastructure, Modern data platforms, Storage

What is data protection modernization worth? For many companies, it’s priceless! To understand why, let’s look at major trends having a significant impact on data protection and how companies are evolving their data protection solutions to address new requirements. Enormous growth in digital content, broad adoption of AI and data analytics, additional regulatory reporting requirements more

IBM Storage builds on leadership for containers

Cloud object storage, Modern data platforms, Real-time analytics...

Next directions to accelerate cloud native Containers are an increasingly important element in the design techniques for enterprises to develop and deliver applications and services with greater speed and agility as part of their cloud-native transformation. IBM Storage has taken a position in enabling the data management and protection of containers with open-source Kubernetes integrations. more

New smart multicloud storage solutions for businesses

Flash storage, Integrated infrastructure, Storage...

Today IBM is announcing a broad spectrum of innovations, enhancements and new features across our entire storage portfolio, aimed at providing leading-edge solutions for 21st century business and technology challenges. I hear from business leaders every day about their aspirations as well as their challenges and pain points. They demand the highest levels of performance, more