Share this post:
IBM empowers software-defined storage to manage unstructured data for cloud, big data, analytics, objects and more
The Internet of Things — the digitalization of our environment — contributes to an ever-growing need for corporations and organizations to create, analyze, and store more data than ever before. And for this trend, there is no end in sight. Those who can keep up and adopt new technologies to meet their growing demands in data management will thrive.
To help with all of this IoT data, the new release of IBM Spectrum Scale has a new feature, Transparent Cloud Tiering. It’s a versatile all-in-one tool for data management, able to handle cloud data at scale with its ability to perform tiering and archiving. I had the opportunity to speak with IBM scientist Alessandro Sorniotti, one of the inventors who contributed several unique capabilities of the technology, to explain how he brought his idea to fruition and how it will help IBM’s clients.
Please give us a rough timeline of the project.
Alessandro Sorniotti, IBM Research-Zurich
Alessandro Sorniotti (AS): I think the whole thing started in 2010 as a question: “How can we use object stores in a secure and reliable way?” Object stores are services offered by most major cloud providers, including IBM. They are like big boxes of data. Usually they have rather simple interfaces and offer a low-cost option to store huge amounts of data. So, we asked ourselves how can we improve the security and reliability of these boxes.
We started to develop a bit of technology around that goal, mostly aimed at various problems like, “How do you encrypt data that goes to the cloud and where do you store the keys?” or “How do you manage data so it’s also available if one of the providers is not online?” This was around 2012 and during this phase we wrote some papers, participated in an EU project, registered some patents, and built the multicloud storage toolkit, which creates a single reliable and secure virtual object interface out of one or more real storage providers.
Armed with all that, we then started to put out some feelers to see whether there was actually a demand for our technology as an integrated toolkit within our storage products.
We successfully showed working prototypes of various IBM storage solutions (Spectrum Scale, Spectrum Virtualize, and others) using the toolkit to tier or backup data to different clouds. We demonstrated this at CeBIT, which generated a lot of interest, and we soon realized that we were on to something.
Encouraged by the success of our prototypes, we then started to work closely with various development units. And now it seems we’re getting close to the finish line of this interesting journey, as the next release of Spectrum Scale will have transparent cloud tiering functionality for the first time. We are really busy at the moment finalizing the product, but you can clearly see how we went from some complex ideas, to papers and patents, to prototypes, to the final product in a short period.
Throughout this entire journey, what, for you, were the most important and most exciting milestones?
AS: In terms of milestones, one of the most exciting times was the initial phase. We were bubbling with ideas like, “We could use these cloud storage providers by taking the data, splitting it into small chunks, and then putting up one chunk per storage provider. In this way, you could tolerate two out of five going down but still get your data from the other three.” And other such fancy ideas; this really was an agile process. A lot of prototypes, very fast development cycles, with a lot of fun along the way.
In the second phase, we weren’t working on the technology itself but on its integration.
The third exciting phase was: “Hey, now it’s really becoming a product.” So we had to shift gears. We couldn’t just have prototypes and, if they crashed, just shrug it off. Now everything had to work reliably, or at least fail gracefully with an acceptable recovery path. And if anything broke down, we had to find a fix for it. For 9 months we basically took off our researcher hats and put on the hardhats of developers in the trenches. A lot of theory had to turn into practice. It’s almost like being an architect who learns to lay bricks or plaster walls.
What are the advantages of IBM Spectrum Scale Transparent Cloud Tiering compared to other solutions?
AS: First of all, with this new feature, customers are able to move data from a file system to the cloud provider of their choice, and they can do this natively. Previously, it was highly complex; the systems of several different vendors had to be made to work together. In most cases there was quite a bit of work involved in making them do so. But now Spectrum Scale features this native capability. It’s like a coffee machine that also makes cappuccino without having to buy an add-on to do so. It just comes in the box.
Secondly, you have the choice of which provider to use – no more cloud vendor lock-in.
Moreover, the technology is highly scalable in the sense that we take advantage of the distributed nature of Spectrum Scale in such a way that potentially each Spectrum Scale node could start moving data to the cloud, thus filling the biggest of network pipes. Data movement is never impaired by a single node’s lack of capacity.
Another advantage comes from Cleversafe, a company IBM acquired last year, which is now called IBM Cloud Object Storage. They are a leader in the object storage space, and our toolkit is the perfect glue to combine current IBM storage solutions with this new acquisition.
Now that you’re in the open beta stage of IBM Spectrum Scale Transparent Cloud Tiering, what do you expect from this phase and what kind of feedback have you got so far?
AS: So far, the open beta has been very exciting. We have counted more than 100 unique downloads. Users can register on a webpage, download it and try it out – very simple. Mostly we’ve received positive feedback. We especially appreciate it when customers give us feedback on how to improve the product.
For instance, a large banking client told us, “If you don’t have this particular way of storing encryption keys, we can’t use it, because we have our own corporate rules.” This was exactly the type of feedback we were hoping for. I look forward to responding to this client shortly to tell them that the feature is now available.
So the product launch of IBM Spectrum Scale Transparent Cloud Tiering is predicted for its next release?
AS: Yes, general availability was announced today. I’m a researcher at heart, but it’s incredibly satisfying to bring an idea to market and see it get in the hands of our clients.