Steel and Bridges: Evolving our Big Data Platform

Share this post:

Andrew Carnegie, born in Scotland, came to the US in 1848. He started work in a factory and eventually built one of the single most successful businesses in US history, Carnegie Steel (later became known as US Steel, based on a few mergers). Carnegie is a tremendous book, if you want a deep look into how he transformed from factory worker, to steel magnate, to philanthropist. Hint: it was a lot of hard work and smarts. However, one thing stuck with me from this book: his insights on strategy.

As he was building the steel company, he had a bad experience where he was mistreated by a railroad owner. At that point, he decided that he needed to either (a) own all components of the steel value-chain (from ore to tracks to railroads to bridges) or (b) build a set of partnerships for the long term, with folks that he could rely upon (and vice versa). This is why Carnegie Steel started investing in (owning sometimes, partnerships in other cases) most components from ore to bridges (railroads, sleeping cars, oil derricks, depots, etc). It may seem obvious now, as we have seen many companies emulate these strategies (think of media companies, auto companies, etc). However, this notion of owning or partnering throughout the value chain was his strategic insight, at a time when no one else saw that opportunity.

We made two significant announcements this morning, much along the lines of Carnegie’s spirit for building a business for the long term. Like Carnegie, we believe that building the Big Data Platform for the enterprise requires us to be ambidexterous: in some cases build, in others buy, and in even others partner.


First, we announced our intention to acquire Vivisimo. With strong roots at Carnegie Mellon, Vivisimo has been in business for over a decade. While they have had tremendous success as a company, you could also argue that Raul, Jerome, and Chris were ahead of their time, as we are now at the dawn of the Big Data Era. Vivisimo brings an immensely talented team, along with very sophisticated federated discovery and data navigation technology, to IBM. They have built these capabilities on top of their core strength in indexing disparate data sources, across an enterprise. Think about it: With our Big Data Platform and the Vivisimo capabilities, clients can access all of their data through a common lense, whether it is stored in hadoop, a warehouse, ERP, CRM, etc, etc…In short, this is an extremely powerful analytical capability.

To date, Big Data in IBM has been one of our most incredible organic development projects in recent memory. Our core platform is 100% IBM technology, built in IBM labs (Silicon Valley and Research), delivering tremendous client value. That being said, we are thrilled to welcome Vivisimo to the family, as the combination of our organic innovation, along with their people and technology, will keep us well ahead of others in providing a Big Data Platform to clients everywhere.

Hadoop Distributions

I’ve stated in these pages, since May of last year, that IBM is not in the Hadoop distribution business. Many people have questioned how I can assert that, based on what they perceive. Hopefully today’s announcement will change that perception once and for all. We announced today that we will be enabling our Big Data Platform on other Hadoop distributions. The first partner that we are announcing in this area is Cloudera. Our objective is to be distribution agnostic. Said another way, we believe in client choice. If a client wants IBM to package Apache Hadoop as part of our Platform, we are willing and ready. If the client prefers the Cloudera distribution underneath our Platform, we will now be ready. If a client prefers another distribution underneath our Platform…well, we have nothing to announce on that point at this time. But suffice it to say, we believe in serving client choice, when it comes to the open source components of our Big Data Platform. I’ve been working with the Cloudera team for nearly 1.5 years at this point and have utmost respect for what Mike, Jeff, Kirk, and others are doing there. We look forward to a great partnership, serving our joint clients.


We will remain very active in Big Data. We have great client references (see some here and here, see the full reference book and others here), we now have over 100 partners in our ecosystem, we’ve been named a Leader in Big Data, and we are hiring engineers that want to be a part of the largest (and best) development team in Big Data.

If you are interested in partnering with us, please reach out. We will be over 200 strong in the ecosystem by the end of the year. If you have interesting technology that complements our Platform, please reach out. We are hosting a Big Data Partner conference call on April 26 at Noon est, to announce some new offerings for our Big Data Channel. Please join us if you are a current partner or are just thinking about it. Register here.

We are still in the early days of the Big Data Era, but we’ll need more steel, bridges, sleeping cars, and depots to serve our enterprise clients.

More stories

Why we added new map tools to Netcool

I had the opportunity to visit a number of telecommunications clients using IBM Netcool over the last year. We frequently discussed the benefits of have a geographically mapped view of topology. Not just because it was nice “eye candy” in the Network Operations Center (NOC), but because it gives an important geographically-based view of network […]

Continue reading

How to streamline continuous delivery through better auditing

IT managers, does this sound familiar? Just when everything is running smoothly, you encounter the release management process in place for upgrading business applications in the production environment. You get an error notification in one of the workflows running the release management process. It can be especially frustrating when the error is coming from the […]

Continue reading

Want to see the latest from WebSphere Liberty? Join our webcast

We just released the latest release of WebSphere Liberty, It includes many new enhancements to its security, database management and overall performance. Interested in what’s new? Join our webcast on January 11, 2017. Why? Read on. I used to take time to reflect on the year behind me as the calendar year closed out, […]

Continue reading