Inside System Storage -- by Tony Pearson

This week I am in Moscow, Russia for today's "Edge Comes to You" event. Although we had over 20 countries represented at the Edge2012 conference in Orlando, Florida earlier this month, IBM realizes that not everyone can travel to the United States. So, IBM has created the "Edge Comes to You" events where a condensed subset of the agenda is presented. Over the next four months, these events are planned in about two dozen other countries.

This is my first time in Russia, and the weather was very nice. With over 11 million people, Moscow is the 6th largest city in the world, and boasts having the largest community of billionaires. With this trip, I have now been to all five of the so-called BRICK countries (Brazil, Russia, India, China and Korea) in the past five years!

The venue was the [Info Space Transtvo Conference Center] not far from the Kremlin. While Barack Obama was making friends with Vladimir Putin this week at the G2012 Summit in Mexico, I was making friends with the lovely ladies at the check-in counter.

If it looks like some of the letters are backwards, that is not an illusion. The Russian language uses the [Cyrillic alphabet]. The backwards N ("И"), backwards R ("Я"), the number 3 ("З), and what looks like the big blue staple logo from Netapp ("П"), are actually all characters in this alphabet.

Having spent eight years in a fraternity during college, I found these not much different from the Greek alphabet. Once you learn how to pronounce each of the 33 characters, you can get by quite nicely in Moscow. I successfully navigated my way through Moscow's famous subway system, and ordered food on restaurant menus.

The conference coordinators were Tatiana Eltekova (left) and Natalia Grebenshchikova (right). Business is booming in Russia, and IBM just opened ten new branch offices throughout the country this month. So these two ladies in the marketing department have been quite busy lately.

I especially liked all the attention to detail. For example, the signage was crisp and clean, and the graphics all matched the Powerpoint charts of each presentation.

Moscow is close to the North pole, similar in latitude as Juneau, Alaska; Edinburgh, Scottland; Copenhagen, Denmark; and Stockholm, Sweden.

As a result, it is daylight for nearly 18 hours a day. The first part of the day, from 8:00am to 4:30pm, was "Technical Edge", a condensed version of the 4.5 day event in Orlando, Florida. I gave three of the five keynote presentations:

• Game Change on a Smarter Planet: A New Era in IT, discussing Smarter Computing and Expert-Integrated systems, based on what Rod Adkins presented in Orlando.
  
  
• A New Approach to Storage, explaining IBM Smarter Storage for Smarter Computing, IBM's new approach to the way storage is designed and deployed for our clients
  
  
• IBM Watson: How it Works and What it Means for Society Beyond Winning Jeopardy! explaining how IBM Watson technologies are being used in Healthcare and Financial Services, based on what I presented in Orlando.

(Note: I do not speak Russian fluently enough to give a technical presentation, so I did then entire presentation in English, and had real-time translators convert to Russian for me. The audience wore headphones. However, I was able to sprinkly a few Russian phrases, such as "доброе утро", "Я не понимаю по-русский" and "спасибо".)

After the keynote sessions, I was interviewed by a journalist for [Storage News] magazine. The questions covered a variety of topics, from the implications of [Big Data analytics] to the future of storage devices that employ [Phase Change Memory]. I look forward to reading the article when it gets published!

The afternoon had break-out sessions in three separate rooms. Each room hosted seven topics, giving the attendees plenty to choose from for each time slot. I presented one of these break-out sessions, Big Data Cloud Storage Technology Comparison. The title was already printed in all the agendas, so we went with it, but I would have rather called it "Big Data Storage Options". In this session, I explained Hadoop, InfoSphere BigInsights, internal and external storage options.

I spent some time comparing Hadoop File System (HDFS) with IBM's own General Parallel File System (GPFS) which now offers Hadoop interfaces in a Shared-Nothing Cluster (SNC) configuration. IBM GPFS is about twice as fast as HDFS for typical workloads.

At the end of the Technical Edge event, there was a prize draw. Business cards were drawn at random, and three lucky attendees won a complete four-volume set of my book series "Inside System Storage"! Sadly, these got held up in customs, so we provided a "certificate" to redeem them for the books when they arrive to the IBM office.

The second part of the day, from 5:00pm to 8pm, was "Executive Edge", a condensed version of the 2 day event in Orlando, designed for CIOs and IT leaders. Having this event in the evening allowed busy executives to come over after they spend the day in the office. I presented IBM Storage Strategy in the Smarter Computing Era, similar to my presentation in Orlando.

Both events were well-attended. Despite fighting jet lag across 11 time zones, I managed to hang in there for the entire day. I got great feedback and comments from the attendees. I look forward to hearing how the other "Edge Comes to You" events fare in the other countries. I would like to thank Tatiana and Natalia for their excellent work organizing and running this event!

technorati tags: IBM, Moscow, Russia, Edge, ECTY, Cyrillic, Tatiana Eltekova, Natalia Grebenshchikova, Smarter Storage, Smarter Computing, Smarter Planet, Big Data, Cloud, IBM Watson, Jeopardy, Hadoop, HDFS, InfoSphere, BigInsights, GPFS, GPFS-SNC

Continuing my coverage of the 30th annual [Data Center Conference]. Here is a recap of the Tuesday morning sessions:

Wells Fargo: Data Center Lessons Learned from the Wachovia Acquisition

This was the next in their "Mastermind Interview" series. The analyst interviewed Scott Dillon, EVP and Head of Technology Infrastructure Services for Wells Fargo bank. Some 13 years ago, Wells Fargo merged with Norwest, and three years ago, Wells Fargo merged again, this time with Wachovia bank. Today, the new merged Wells Fargo manages 1.2 Trillion USD in assets, some 12,000 ATMs, and 9,000 branch offices within two miles of 50 percent of the US population.

On the technical side, Scott's team has to deal with 10,000 IT changes per month, spanning 85 discrete businesses that Wells Fargo is involved in. To help drive the consolidation, they formed a culture group called "One Wells Fargo".

Often, Wells Fargo and Wachovia used different applications for the same function. The consolidation team took the A-or-B-but-not-C approach, which means they would either choose the existing application that Wells Fargo was already using (A), or the one that Wachovia was already using (B), but not look for a replacement (C). They also wanted to avoid re-platforming any apps during the merger. This simplified the process of developing target operating models (TOMs).

Before each application cut-over, the consolidation team did dry-run, dress rehearsals and walkthroughs over the phone to ensure smooth success. They wanted a Wachovia account holder to be able to walk into the bank on one day, and then come back the next day as a Wells Fargo account holder, into the same branch office but now with Wells Fargo signage, with minimal disruption.

Wells Fargo also adopted a test-to-learn approach of choosing small test markets to see how well the transition would work before tackling larger, more complicated markets. For example, they started in Colorado, where Wells Fargo has a huge presence, but Wachovia had a small presence.

This was first and foremost a business merger, not just an IT merger. Each decision to 6-18 months to act on, and the IT team spent the last three years working every weekend to make this a reality.

A Satirical Look at Business and Technology

Comedian Bob Hirschfeld presented a light-hearted look at the IT industry. Bob actually attended sessions on Monday at this conference so his satire was exceptionally hard-hitting. He took jabs at the latest IT job requirements, padding on light poles, IBM Watson, social media's impact on dictators, various industry acronyms, virtualization, the various reasons why printer ink is so expensive, and the evil masterminds behind Powerpoint.

Storing Big Data takes a Village

Two analysts co-presented this session on the 12 dimensions of information management that revolve around the volume, variety and velocity of "Big Data".

In the past, it took a while to gather data, and a while to process the data, so annual, quarterly and monthly reports were common. Today, with high-velocity streams like Twitter, especially during cultural events or natural disasters, data is produced and analyzed quickly. It is important to sort the steady-state from the anomalies.

Myth 1: All data fits nicely into relational databases. The analysts feel the concept of putting everything into one big data base is dead. Some data sets are so complicated that traditional database joins would cause smoke to come out of the sides of the servers. Instead, new technologies have emerged, including NoSQL, Cassandra, Hadoop, Columnar databases, and In-memory databases. XML has helped to bring together disparate data formats.

Companies need to adapt to this new reality of Business Analytics. Here is a poll of the audience on how many are in what stage of adaptation:

Myth 2: Everyone will do Big Data with commodity hardware. Businesses want commmercial offerings that don't fail every day. (For example, instead of using open-source Hadoop, consider IBM's [InfoSphere BigInsights] commercial product based on Hadoop designed for the Enterprise).

Myth 3: Big Data is too big for backup. Certainly, traditional full-plus-incremental approaches fail to scale, but that is not the only option you have. Consider disk replication, snapshots, and integrated disk-and-tape blended solutions that adopt a more progressive backup methodology.

Capacity forecasting can be difficult with Big Data. Scale-out NAS systems, including IBM SONAS and the various me-too competitive offerings, were originally focused on High Performance Computing (HPC) and the Media & Entertainment (M&E) industries, are now ready for prime-time and appropriate for other use cases.

It's like the game of Clue, but instead of Professor Plum with the candlestick in the library, it was Chuck with the Cluster in the Closet. To avoid shadow IT creating huge Hadoop Clusters in your closets, encourage the use of Cloud Computing for "sandbox" projects. IBM, Amazon and others offer hosted MapReduce engines for this purpose.

What type of storage do you plan to use for Big Data? The top five, weighted from a list during a poll of the audience were: (78) traditional disk arrays, (71) Scale-out NAS, (46) pre-configured appliances, (30) Hadoop clusters, and (23) Cloud Storage.

Big Data is about doing things differently. Do your employees understand analytical techniques? Your company may need to start thinking about policies for capturing Big Data, storing it correctly, and analyzing it for insights and patterns needed to stay competitive.

It was good to mix reality with a bit of humor. Some of these conference attendees take themselves too seriously, and it is good to be reminded that IT is just part of the overall business operation.

technorati tags: IBM, Wells Fargo, Wachovia, Scott Dillon, , Bob Hirschfeld, Big Data, SONAS, NoSQL, Cassandra, Hadoop, Columnar databases, business analytics

Continuing my coverage of the 30th annual [Data Center Conference]. Here is a recap of the other Monday morning keynote sessions:

Driving Innovation to Achieve Dramatic Improvements

What is Innovation? It is a process that starts with one or more ideas, that results in change, that creates value. Easier said than done!

Innovation drives business growth. The analyst indicated that the IT infrastructure can either be in the way to impede business growth, neutral to enable growth, or contributing to business growth. Companies often find downtime as an inhibitor to business growth. The analyst gave these typical numbers.

Category	Unplanned downtime (hours per year)	Planned Downtime (hours per year)
Important	61	<200
Business Critical	26	<50
Mission Critical	5	<12

A big inhibitor to change is "cultural inertia", which states that the way things are prevent what they could be. Change requires both rewards and measures. Employees are often uncomfortable with change. Motivation should be with carrots not sticks.

(I often joke that the only people who are comfortable with change are babies with soiled diapers and prisoners on death row!)

The impedence to change is further amplified by leadership because what got them into their positions was their history of success, and often leaders perpetuate what worked for them in the past.

"There is nothing so useless as doing efficiently that which should not be done at all."
--- Peter Drucker

Nothing lasts forever, and companies should not try to avoid the inevitable. Innovators need to see themselves as change agents. the analyst feels that less than 10 percent of IT will adopt innovation to enact dramatic change. The analyst took a poll of the audience asking: Why isn't your IT Infrastructure and Operations more innovative? Over 800 attendees responded. Here were the results:

The analyst suggests treating Innovation like a team sport, with small 2-5 person teams. Search for breakthrough opportunities by setting audacious goals to inspire innovative thinking. What approach are most people doing today? Here are some polling results:

The analyst suggest it is more important to establish a culture of innovation first, and process second. Skunkworks projects are back in favor. IT folks should avoid the worship of so-called "best practices" as a reason to avoid change in trying something different. To think "outside-the-box" you need to get outside the box, or office, or cubicle, or wherever you work that prevents you from interacting with your internal or external customers. Customers can bring great insights on new approaches to take.

One new approach, born in the Cloud and now coming to the Enterprise is the concept of [DevOps], which consists of promoting collaboration between the "Appplication Development" half of IT, with the "Operations" half. If you never had heard of DevOps before, you are not alone, most of the attendees at this conference hadn't either. Here are the poll results:

Some companies have instituted a "Fresh Eyes" program, asking new-hires and early-tenure employees questions like: What surprised you the most when you joined the company? Was there anything that didn't make sense to you? Do you have any ideas to improve the way we do things?

"In a time of crisis we all have the potential to morph up to a new level and do things we never thought possible"
– Stuart Wilde

Why wait for a crisis?

Facebook: Efficient Infrastructure at a Massive Scale

Frank Frankovsky, the Director of Hardware and Design and Supply Chain at Facebook, was sitting right next to me in the audience. I didn't know this until it was his turn to speak, and he jumps up and walks to the stage! For those who live under a rock and/or are over 40 years old, Facebook is a social media site that allows people to maintain personal profiles, share photos, news and messages, play games, and create groups to organize events. They now support over 800 million accounts, a healthy percentage of the 1.9 billion people on the internet today.

Started in 2004, Facebook was originally hosted on standard server and storage hardware in colocation facilities. Facebook saved 38 percent costs by bringing their operations in-house, building their own servers from parts, and using no third-party software. Facebook has the advantage of owning their entire software stack, leveraging open source as much as possible. They even re-wrote their own PHP compiler, which they pronounce "Hip-Hop", short for high-performance-PHP.

Facebook can stand up a new data center in less than 10 months, from breaking ground to serving users. Most of Facebook's data centers sport a PUE less than 1.5, but their newest one in Prineville, Oregon is down to an amazing 1.07 level for a 7.5 Megawatt facility! How did they do it? Here are a few of their tricks:

• Use Scale-Out architecture. Having lots of small servers, scattered in various data centers, allows them to survive a server failure, as well as having the luxury to shut down a datacenter when needed for maintenance reasons.
• Free Cooling. Instead of air-conditioning, they pump in cold air from the outside, and send the heated exhaust back outdoors. Frank does not believe servers should be treated like humans, so their data centers run uncomfortably hot. The 50-year climate data is used to determine data center locations that have the optimal "free cooling" opportunities.
• Eliminate UPS and PDU energy losses. Rather than running 480 VAC power through UPS that represent a 6 to 12 percent loss, and then PDU that introduce another 3 percent loss getting down to 208 or 120 VAC, Frank's team builds servers that feed direclty off the 480 VAC from the power company. For backup power, they use 48VDC batteries. One set of batteries can backup six racks of servers.
• Target 6 to 8 KW per rack. Low-density racks are easier to keep cool.
• Build their own IT equipment. Rather than buying commercially-available servers, Frank's team builds 1.5 U servers based on Intel "Westmere" chipset. 1.5U allows for larger fan radius than standard 1U pizza box format. (IBM's iDataPlex uses 2U fans for the same reason!) Facebook has a "Vanity free" design philosophy, so no fancy plastic bezels. In most cases, the covers are left off. Most (65 percent) of their servers are web front-ends. They plan new IT equipment based on Intel's "Sandy Bridge" chipset.
• Use SATA drives. They buy the largest SATA drives available, directly from manufacturers, in direct-attach storage (DAS) in their servers. Data is organized in a Hadoop cluster, and they have developed their own internal "Haystack" for photo storage. Despite the floods in Thailand, Facebook has secured all the SATA disk they plan to buy for 2012 from their suppliers.
• Use Solid-State drives. Their Database tier uses 100 percent Solid-State drives.

Frank is also a founder for the [Open Compute Project], which takes an "Open source" approach to IT hardware.

Facebook does not bother with hypervisors. Instead, they have adapted their own software to make full use of the CPU natively. This eliminates the "I/O Tax" penalty associated with VMware and other hypervisors.

Of course, not everyone owns their entire software stack, and can build their own servers! It was nice to hear how a company without such limitations can innovate to their advantage.

technorati tags: IBM, Innovation, Peter Drucker, Stuart Wilde, Facebook, Frank Frankovsky, Hadoop, Open Compute