Share this post:
From nothingness to sudden, explosive existence—the Big Bang is a widely accepted and well-tested scientific theory. However, there are periods of time that are still a mystery and many questions left to answer.
How are galaxies formed and how do they evolve? Are we alone in the universe? Astronomers around the world are hoping to explain some of these mysteries and achieve technology breakthroughs by analyzing 13-billion-year-old radio signals from outer space, collected by an enormous radio telescope. This isn’t an ordinary telescope, but a vast array of sensitive devices stitched together to create a detailed picture of our universe.
It’s a project that faces many challenges; in particular, developing a reliable, accessible and affordable storage system to handle the vast amount of data that will be generated. If you’re dealing with storage issues in your company, read on to learn how they’re being dealt with in one of the greatest storage missions of the era.
A massive undertaking
Upon completion in 2024, the Square Kilometer Array (SKA) will be the world’s largest and most sensitive radio telescope, employing a network of more than 3,000 dishes to collect approximately one exabyte of radio signal data daily. Analyzing this amount of data, which is equal to two times the current daily traffic on the Internet, will require that significant advancements be made in high-performance, purposeful cognitive infrastructures.
ASTRON, the Netherlands Institute for Radio Astronomy, is helping to facilitate the endeavor by building a streaming analytics platform for the fastest insights that runs on energy-efficient exascale computing technology.
Joining forces with IBM Research, ASTRON launched the five-year DOME project, which aims to address systems engineering challenges related to the SKA.
Managing the budget
Simply accounting for the capacity requirements of the SKA project, the cost for acquiring and running the storage devices will be tremendous. To keep storage costs manageable, the data will need to be placed on different storage tiers such as flash, disk and tape. With this quantity of data, policy driven tiering is simply not enough.
The placement of SKA data is a packing job on a massive scale. That’s where IBM Research comes in. There is an expansion underway upon current capabilities to add machine learning to IBM Storage. IBM Storage solutions provide a variety of solutions tailored for different price and performance.
• Flash storage is similar to a carry-on bag: it is limited in size but quickly accessible. Flash storage size restrictions are mostly due to its high cost.
• Disk storage is similar to checked luggage: it can be accessed shortly and you can readily expand until the cost becomes too high.
• Tape storage is similar to the items you’ve shipped: it isn’t needed right away but it might be later. Tape storage is the least-costly option with data integrity to efficiently store large data sets.
Manually deciding on which tier to place the data would take a lot of manpower and policies cannot automatically adopt to changing usage patterns, so IBM and ASTRON have been working on optimizing data placement based on workload requirements such as platforms, runtimes and deployment models, along with storage characteristics such as input/output latency, read/write bandwidth and capacity.
To pack up your personal possessions, you might hire a moving company to help. But IBM and ASTRON are enlisting the help of technology to conquer data-centric design, storage and management issues. That technology is IBM Spectrum Scale, which is the storage platform being tested to manage the data spanning flash, disk and tape under a common global namespace. IBM Spectrum Storage already provides a scale-out solution for large-scale workloads with policy-driven tiering between flash, disk and tape. Furthermore, the software-defined storage provides custom metadata and mature performance monitoring to provide a mature toolkit for further development. See this presentation to learn more.
Arriving at the destination
Whether you’re using IT analytics and purposeful architectures to deliver immediate insights to inform your operational decisions, or analyzing radio signal data to discover what happened in the universe after the Big Bang occurred, infrastructure that has been intentionally designed for data can accelerate important technology breakthroughs and change the way we see our universe. Discover new options for innovation and optimization and see how you can support analytics in your company.