Flash storage

The Big Bang means big data with infrastructure storage

Share this post:

From nothingness to sudden, explosive existence—the Big Bang is a widely accepted and well-tested scientific theory. However, there are periods of time that are still a mystery and many questions left to answer.

How are galaxies formed and how do they evolve? Are we alone in the universe? Astronomers around the world are hoping to explain some of these mysteries and achieve technology breakthroughs by analyzing 13-billion-year-old radio signals from outer space, collected by an enormous radio telescope. This isn’t an ordinary telescope, but a vast array of sensitive devices stitched together to create a detailed picture of our universe.

It’s a project that faces many challenges; in particular, developing a reliable, accessible and affordable storage system to handle the vast amount of data that will be generated. If you’re dealing with storage issues in your company, read on to learn how they’re being dealt with in one of the greatest storage missions of the era.

A massive undertaking

Upon completion in 2024, the Square Kilometer Array (SKA) will be the world’s largest and most sensitive radio telescope, employing a network of more than 3,000 dishes to collect approximately one exabyte of radio signal data daily. Analyzing this amount of data, which is equal to two times the current daily traffic on the Internet, will require that significant advancements be made in high-performance, purposeful cognitive infrastructures.

ASTRON, the Netherlands Institute for Radio Astronomy, is helping to facilitate the endeavor by building a streaming analytics platform for the fastest insights that runs on energy-efficient exascale computing technology.

Joining forces with IBM Research, ASTRON launched the five-year DOME project, which aims to address systems engineering challenges related to the SKA.

Managing the budget

Simply accounting for the capacity requirements of the SKA project, the cost for acquiring and running the storage devices will be tremendous. To keep storage costs manageable, the data will need to be placed on different storage tiers such as flash, disk and tape. With this quantity of data, policy driven tiering is simply not enough.

Packing up

The placement of SKA data is a packing job on a massive scale. That’s where IBM Research comes in. There is an expansion underway upon current capabilities to add machine learning to IBM Storage. IBM Storage solutions provide a variety of solutions tailored for different price and performance.

Flash storage is similar to a carry-on bag: it is limited in size but quickly accessible. Flash storage size restrictions are mostly due to its high cost.
Disk storage is similar to checked luggage: it can be accessed shortly and you can readily expand until the cost becomes too high.
Tape storage is similar to the items you’ve shipped: it isn’t needed right away but it might be later. Tape storage is the least-costly option with data integrity to efficiently store large data sets.

Manually deciding on which tier to place the data would take a lot of manpower and policies cannot automatically adopt to changing usage patterns, so IBM and ASTRON have been working on optimizing data placement based on workload requirements such as platforms, runtimes and deployment models, along with storage characteristics such as input/output latency, read/write bandwidth and capacity.

Moving on

To pack up your personal possessions, you might hire a moving company to help. But IBM and ASTRON are enlisting the help of technology to conquer data-centric design, storage and management issues. That technology is IBM Spectrum Scale, which is the storage platform being tested to manage the data spanning flash, disk and tape under a common global namespace. IBM Spectrum Storage already provides a scale-out solution for large-scale workloads with policy-driven tiering between flash, disk and tape. Furthermore, the software-defined storage provides custom metadata and mature performance monitoring to provide a mature toolkit for further development. See this presentation to learn more.

Arriving at the destination

Whether you’re using IT analytics and purposeful architectures to deliver immediate insights to inform your operational decisions, or analyzing radio signal data to discover what happened in the universe after the Big Bang occurred, infrastructure that has been intentionally designed for data can accelerate important technology breakthroughs and change the way we see our universe. Discover new options for innovation and optimization and see how you can support analytics in your company.

More stories

Powerful new storage for your mission-critical hybrid multicloud

Flash storage, Multicloud, Storage

Cloud computing and mainframes processors are two of the most important contributors in information technology solutions. They are increasingly being linked together to unlock value for clients. 85 percent of companies are already operating in multicloud and by 2021, most existing apps will have migrated to the cloud. [1] At the same time, mainframe utilization ...read more

A conversation about metadata management featuring Forrester Analyst Michele Goetz

AI, Big data & analytics, Storage

I recently had an opportunity to speak with Forrester Principal Analyst Michele Goetz following a research study on the subject of metadata management conducted by Forrester Consulting and commissioned by IBM. Michele’s research covers artificial intelligence technologies and consultancies, semantic technology, data management strategy, data governance and data integration, and includes within its scope the ...read more

IBM is transforming data storage for media and entertainment

Data security, Storage, Tape and virtual tape storage

Disruptions in media and entertainment have been occurring at a rapid pace over the past few years and many organizations are still struggling with optimizing their IT infrastructure for new data requirements. With growth in capacity, new larger file formats, keeping more data online for business opportunities, and the growing use of AI, organizations are ...read more