This blog is for the open exchange of ideas relating to IBM Systems, storage and storage networking hardware, software and services.
(Short URL for this blog: ibm.co/Pearson )
Tony Pearson is a Master Inventor, Senior IT Architect and Event Content Manager for [IBM Systems for IBM Systems Technical University] events. With over 30 years with IBM Systems, Tony is frequent traveler, speaking to clients at events throughout the world.
Lloyd Dean is an IBM Senior Certified Executive IT Architect in Infrastructure Architecture. Lloyd has held numerous senior technical roles at IBM during his 19 plus years at IBM. Lloyd most recently has been leading efforts across the Communication/CSI Market as a senior Storage Solution Architect/CTS covering the Kansas City territory. In prior years Lloyd supported the industry accounts as a Storage Solution architect and prior to that as a Storage Software Solutions specialist during his time in the ATS organization.
Lloyd currently supports North America storage sales teams in his Storage Software Solution Architecture SME role in the Washington Systems Center team. His current focus is with IBM Cloud Private and he will be delivering and supporting sessions at Think2019, and Storage Technical University on the Value of IBM storage in this high value IBM solution a part of the IBM Cloud strategy. Lloyd maintains a Subject Matter Expert status across the IBM Spectrum Storage Software solutions. You can follow Lloyd on Twitter @ldean0558 and LinkedIn Lloyd Dean.
Tony Pearson's books are available on Lulu.com! Order your copies today!
Safe Harbor Statement: The information on IBM products is intended to outline IBM's general product direction and it should not be relied on in making a purchasing decision. The information on the new products is for informational purposes only and may not be incorporated into any contract. The information on IBM products is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. The development, release, and timing of any features or functionality described for IBM products remains at IBM's sole discretion.
Tony Pearson is a an active participant in local, regional, and industry-specific interests, and does not receive any special payments to mention them on this blog.
Tony Pearson receives part of the revenue proceeds from sales of books he has authored listed in the side panel.
Tony Pearson is not a medical doctor, and this blog does not reference any IBM product or service that is intended for use in the diagnosis, treatment, cure, prevention or monitoring of a disease or medical condition, unless otherwise specified on individual posts.
The developerWorks Connections platform will be sunset on December 31, 2019. On January 1, 2020, this blog will no longer be available. More details available on our FAQ.
I'm down here in Australia, where the government is a bit stalled for the past two weeks at the moment, known formally as being managed by the [Caretaker government]. Apparently, there is a gap between the outgoing administration and the incoming administration, and the caretaker government is doing as little as possible until the new regime takes over. They are still counting votes, including in some cases dummy ballots known as "donkey votes", the Australian version of the hanging chad. Three independent parties are also trying to decide which major party they will support to finalize the process.
While we are on the topic of a government stalled, I feel bad for the state of Virginia in the United States. Apparently, one of their supposedly high-end enterprise class EMC Symmetrix DMX storage systems, supporting 26 different state agencies in Virginia, crashed on August 25th and now more than a week later, many of those agencies are still down, including the Department of Motor Vehicles and the Department of Taxation and Revenue.
Many of the articles in the press on this event have focused on what this means for the reputation of EMC. Not surprisingly, EMC says that this failure is unprecedented, but really this is just one in a long series of failures from EMC. It reminds me of the last time EMC had a public failure with a dual-controller CLARiiON a few months ago that stopped another company from their operations. There is nothing unique in the physical equipment itself, all IT gear can break or be taken down by some outside force, such as a natural disaster. The real question, though, is why haven’t EMC and the State Government been able to restore operations many days after the hardware was fixed?
In the Boston Globe, Zeus Kerravala, a data storage analyst at Yankee Group in Boston, is quoted as saying that such a high-profile breakdown could undermine EMC’s credibility with large businesses and government agencies. “I think it’s extremely important for them,’’ said Kerravala. “When you see a failure of this magnitude, and their inability to get a customer like the state of Virginia up and running almost immediately, all companies ought to look at that and raise their eyebrows.’’
Was the backup and disaster recovery solution capable of the scale and service level requirements needed by vital state
agencies? Had they tested their backups to ensure they were running correctly, and had they tested their recovery plans? Were they monitoring the success of recent backup operations?
Eventually, the systems will be back up and running, fines and penalties will be paid, and perhaps the guy who chose to go with EMC might feel bad enough to give back that new set of golf clubs, or whatever ridiculously expensive gift EMC reps might offer to government officials these days to influence the purchase decision making process.
(Note: I am not accusing any government employee in particular working at the state of Virginia of any wrongdoing, and mention this only as a possibility of what might have happened. I am sure the media will dig into that possibility soon enough during their investigations, so no sense in me discussing that process any further.)
So what lessons can we learn from this?
Lesson 1: You don't just buy technology, you also are choosing to work with a particular vendor
IBM stands behind its products. Choosing a product strictly on its speeds and feeds misses the point. A study IBM and Mercer Consulting Group conducted back in 2007 found that only 20 percent of the purchase decision for storage was from the technical capabilities. The other 80 percent were called "wrapper attributes", such as who the vendor was, their reputation, the service, support and warranty options.
Lesson 2: Losing a single disk system is a disaster, so disaster recovery plans should apply
IBM has a strong Business Continuity and Recovery Services (BCRS) services group to help companies and government agencies develop their BC/DR plans. In the planning process, various possible incidents are identified, recovery point objectives (RPO) and recovery time objectives (RTO) and then appropriate action plans are documentede on how to deal with them. For example, if the state of Virginia had an RPO of 48 hours, and an RTO of 5 days, then when the failure occurred on August 25, they could have recovered up to August 23 level data(48 hours prior to the incident) and be up and running by August 30 (five days after the incident). I don't personally know what RPO and RTO they planned for, but certainly it seems like they missed it by now already.
Lesson 3: BC/DR Plans only work if you practice them often enough
Sadly, many companies and government agencies make plans, but never practice them, so they have no idea if the plans will work as expected, or if they are fundamentally flawed. Just as we often have fire drills that force everyone to stop what they are doing and vacate the office building, anyone with an IT department needs to practice BC/DR plans often enough so that you can ensure the plan itself is solid, but also so that the people involved know what to do and their respective roles in the recovery process.
Lesson 4: This can serve as a wake-up call to consider Cloud Computing as an alternative option
Are you still doing IT in your own organization? Do you feel all of the IT staff have been adequately trained for the job? If your biggest disk system completely failed, not just a minor single or double drive failure, but a huge EMC-like failure, would your IT department know how to recover in less than five days? Perhaps this will serve as a wake-up call to consider alternative IT delivery options. The advantage of big Cloud Service Providers (Microsoft, Google, Yahoo, Amazon, SalesForce.com and of course, IBM) is that they are big enough to have worked out all the BC/DR procedures, and have enough resources to switch over to in case any individual disk system fails.
Continuing my romp through Australia and New Zealand, today I presented in Hobart, the second city on my seven-city tour. Hobart is on a separate island called Tasmania, just south of the main Australian continent. The island is heart-shaped, and Hobart is in the lower right ventricle.
Hobart boasts the second deepest harbour in the Southern Hemisphere (yesterday's Sydney Harbour being the first). It is quite cold here, but at least the skies are clear.
I stayed in the [Henry Jones Art Hotel], named after the famous owner of the IXL Jam Company. When I arrived, they presented me with a list of 18 known convicts that shared my last name: PEARSON. I checked and made sure I was not on the list. Then it was explained to me that here in Australia, everyone values their criminal ancestors, as this is how the country was formed. The names were from registry archives from the 19th century.
In keeping with the concept of an art hotel, each of the rooms were unique, which is a nice way of saying that they fit whatever they could into the spaces available. It's been a while since I stayed at a hotel with the phone at one end of the room, but the electrical outlet at the other. The thermostat was hidden in the bathroom, and I had to master some 16 different ropes to put down the shades, as the bright light from the [Cenotaph] was keeping me awake. I was able to take pictures of some of the art sculptures from the balcony.
This was a smaller event than Sydney, with only about two dozen attendees. This makes sense, as Hobart population is only about 250,000 people. Tasmania island hold about 1 million people overall, concentrated mostly along the center line of the island.
As we had done in Sydney, Anna Wells presented IBM strategy and products, Adam Beames, system administrator for Tennis Australia (shown here in the picture at left) presented his experiences transforming their datacenter, and I presented the future trends in storage.
In appreciation for Adam's presentations in Sydney and Hobart, I presented him with a copy of my book, [Inside System Storage: Volume I], available from my publisher, Lulu.com, in paperback, hard cover, and now also in eBook format for those with Kindle, Nook or other digital book readers. See panel at right on this blog for ordering information.
This week and next, I am down under in Australia and New Zealand for a seven-city Storage Optimisation Breakfast series of presentations to clients and prospects. My first city for this seven-city tour was Sydney, Australia.
Here is the view from my room at the [Shangri-La hotel], including the famous [Sydney Opera House] and Circular Quay, from which to take a water taxi or ride the Manly Ferry. [Sydney harbour] is the deepest harbour in the Southern Hemisphere, allowing boats of all sizes to enter. This section of the city is known as "The Rocks".
Sydney is a very modern metropolis. The last time I was in Sydney was in May 2007 to teach an IBM Top Gun class. My post back then on [Dealing with Jet Lag] is as relevant now as it was back then. In addition to being 9 hours off-shifted from last week in Dallas, Texas, I also have to deal with the colder climate, about 40 degrees F cooler down here. The weather is crisp and clear, it is Winter going into Spring down here as the seasons are flipped below the equator.
Many of the buildings are recognizable from the movie ["The Matrix"] which was filmed here. We joked that this seven-city trip was also similar to [The Adventures of Priscilla, Queen of the Desert], in that both journeys started in Sydney. If you haven't seen the latter, I highly recommend it to get to learn more about Australia as a country.
(Completely useless trivia: Actor Hugo Weaving appeared in both movies. While most people associate him with Australia, where he has lived since 1976, he actually was born in Nigeria, and traveled extensively because his father worked in the computer industry.)
Here I am standing next to our banner.
The line-up for each event is simple. After all the attendees sit down for breakfast, we have the following three sessions:
First, Anna Wells, local IBM Executive for Storage Sales in Australia and New Zealand presents IBM's strategy for storage, and how IBM plans to address Storage Efficiency, Data Protection and Service Delivery. She then highlights various products that are currently available to help meet customer needs, including XIV and the SAN Volume Controller (SVC).
Second, we have a client or two share their success story. We will have different speakers at the different locations.
Third, I present on future trends that will impact the storage marketplace. With only 40 minutes for my section, I decided to focus on just three specific trends, with a mix of some colorful analogies to help emphasize my key points.
We had a great turn-out for our first event in Sydney, lots of clients and prospects came out for this. There is a lot of enthusiasm for IBM's vision, thought leadership, and broad portfolio of storage solutions.
This week I am down under, starting my 7-city Storage Optimisation Breakfast roadshow on Tuesday in Sydney, Australia. I can't be at two places at once, and it seems whenever I am one place, lots of my coworkers are somewhere else at another conference or event. For those at [VMworld 2010] conference in San Francisco this week, IBM is a Platinum Sponsor and hosting a variety of presentations and activities. Here are some things to look forward to:
Session ID SP9638 - Getting the MAX from your Virtualization Investment
Monday 1:30pm, Moscone South Room 309
Speaker: Bob Zuber, IBM System x Program Director
Speaker: Clod Barrera Distinguished Engineer and Chief Technical Strategist
Clod and I just finished Solutions University 2010 in Dallas, and here he is going to VMworld! You already know that virtualization is beneficial. Exploit virtualization to its MAXimum and move beyond virtualization 101 where you have virtualized web, file/print, and DHCP type workloads. Now it is time to take virtualization to the next step and virtualize business infrastructure applications such as ERP, Messaging, CRM, and Database. With IBM solutions you can take the virtualization journey to build a smarter data center through; 1) Consolidation, 2) Management, 3) Automation and 4) Optimization. Attend this session and learn the key considerations for virtualizing mission-critical workloads and the best practices for a virtual data center that delivers a REAL return on your investment.
Session ID TA8065 - Storage Best Practices, Performance Tuning and Troubleshooting
Speaker: Duane Fafard, Senior XIV Storage Architect, IBM
Monday 10:30 AM Moscone South Room 301
Wednesday 03:00 PM Moscone West Room 2005
The industry has solved many of the challenges of virtualization applications by delivering innovative server solutions that automatically migrate load to available resources, but the complete environment requires both the network and the storage to be part of the equation. Designing, managing, and troubleshooting intricate storage environments in today’s age have become more and more complex. This session will discuss storage best practices, performance challenges, and resolving issues in the storage area network using native tools within the environment. With the techniques learned in this session, the storage administrator will be able to use these best practices to design proper storage solutions and pinpoint troubled areas quickly and accurately.
Session ID SS1012 - Expert Panel: How Smarter Systems can Address your Business Challenges
Wednesday, 12-1pm, Room 135
This is IBM's "Super Session". At IBM, we know that all business challenges such as sprawling IT infrastructure, poor performance and rising management costs are solvable on a smarter planet. With Smarter Systems, IBM can help you increase utilization and flexibility, reduce complexity and cost, respond to business changes swiftly and effectively, and enable end-to-end resiliency and security. Alex Yost, Vice President and Business Line Executive for IBM System x and BladeCenter hosts a panel of Virtualization experts:
James Northington, Vice President and Business Line Executive, IBM System x
Donn Bullock, Vice President of Sales, Mainline Information Systems, Inc.
Dylan Larson, Director of Advanced Software and Server Technologies, Intel Data Center Group
Richard, McAniff, Chief Development Officer and Member of the Office of the President, VMware
Siddhartha (Sid) Chatterjee, Ph.D, Vice President, Strategy & Partnerships, IBM Systems Software
David Guzman, Chief Information Officer and Senior Vice President, Global Technology Solution, Acxiom
Next week is [VMworld 2010], so I thought today would be a good day to write a blog post about reporting and managing virtual guest images.
As the original lead architect for IBM Tivoli Storage Productivity Center, I am no stranger to reporting and management tools. Needless to say, if you have lots of virtual guest images, it makes sense to deploy reporting and management software. I had never heard of Veeam before, but I decided to check out Veeam Reporter 4.0, an enterprise-level reporting solution specifically designed for large Virtual Infrastructure (VI3) and vSphere virtual environments that allows you to automatically discover and collect information about your VMware virtual environment.
Their 90-page User Guide offered these helpful "First Steps" on page 9 which I used as the master plan for my evaluation.
Install Veeam Reporter 4.0
The instructions appeared fairly straightforward: Download [the latest version] of the application. Unpack the downloaded archive and run the VeeamReporter.exe file. Then follow the installation wizard steps. What could go wrong?
I should have known better. Like IBM Tivoli Storage Productivity Center, Veeam Reporter is designed to be installed on its own server-class machine with its own application web server and database. I wasn't going to stand up a new server in our lab just for this contest, so I decided to just install it on my Windows XP SP3, which Veeam had listed as a supported operating system level. I ran into a series of installation issues, including installing IIS, installing SQL server, and installing the SRSS component. I am more familiar with IBM's WebSphere Application Server and DB2 combination used in IBM's own products, and have experience with Apache and MySQL on a standard LAMP stack, so my lack of experience with IIS and SQL server made the installation more difficult. Many thanks to all the support personnel at Veeam, Microsoft, and my internal IT department to finally get all of this working.
It appears you can set this up as a client/server environment, where the Veeam Reporter server runs IIS and SQL Server, and then you have a browser on your client machine point to that server. In my cases, I have client browser and server all on one machine.
Create and run a collection job
This step also seemed fairly standard for reporting tools. Once you launch Veeam Reporter 4.0 for the first time, you need to retrieve data from your virtual infrastructure to be able to generate reports. To start the created collection job, select it and click the Start button on the toolbar. If you have a vCenter server in your VI environment, we recommend that you create a job for it to immediately collect data for all objects in its hierarchy. After that, you will be able to select VI objects that were engaged in the performed job using the Workspace, and generate reports for it.
I signed up for this contest August 7, but step 1 above took me two weeks to resolve all the installation iissues. I wanted to get my blog post entry for the contest BEFORE the start of VMworld. Since I am in Dallas, Texas this week for the IBM Storage Solutions University, I had to go through several firewalls for my laptop to tunnel through and get to my VMware Center back in Tucson.
Click on the graphic above to see larger view.
I was able to create and run a collection job. I have a WMware ESX 3.5 host running five guest images and 14 datastores. This seemed to be enough to evaluate the basic features of this reporting tool. Veeam Reporter let's you run the collection process manually, or set a "periodic" schedule to collect data every hour.
Generate reports manually or create a reporting job
Finally, I get to the fun part: To generate report manually, click the Workspace tab, select a necessary VI object from the tree view, date and collection job session, choose reports and click the Create Report button.
At this point, I am reminded of a famous poem:
To see a world in a grain of sand
And a heaven in a wild flower,
Hold infinity in the palm of your hand
And eternity in an hour.
- William Blake
When evaluating products, try to imagine what the reports would look like with hundreds of virtual guest images. Certainly, I can see some potential, even though I had rather limited data to work with. In theory, the tool can create Visio output files, but you need to have Microsoft Visio installed. I have only "Visio Viewer" so I was unable to create any visio files with this product.
The reports can be exported to PDF, Word or Excel formats. Here is an example of an Excel spreadsheet export. While it has 14 bars for the 14 datastores, there are no labels, and the misleading details link in the lower right corner is non-functional. The only way for me to figure out what each referred to was to go back to my vCenter client, which kind of defeats the purpose of having a separate reporting tool.
This same report exported to PDF spanned across four pages, leaving the re-assembly to be done with a pair of scissors and celophane tape.
When you create reports, you can use SRSS or Veeam's internal proprietary format. Only SRSS reports can be put on the dashboard, so I recommend that.
Customize your dashboard
The fourth and final step is to configure your own dashboard: To add reports to the Dashboard, you should first create and save them using Workspace of Veeam Reporter 4.0. Keep in mind that you can add to the Dashboard only saved SSRS-based reports. To customize the Dashboard, click the Dashboard tab and then click the Edit Dashboard button. Customize the layout by dragging blue borders from the right and the bottom of the screen. Then, drag reports from the Reports list and drop them onto the created cells.
The "Free Edition" only allows you to put a single report on the dashboard, so as in step 3, you have to use your imagination of what the potential of the full license would looke like with multiple reports are on a single pane of glass.
(FTC Disclosure: I work for IBM, the leader in server virtualization worldwide, and the number #1 reseller of VMware. In this post, I review [Veeam Reporter 4.0] as my official entry for their blogging contest. IBM and Veeam do not have any business relationshiop that I know of, other than both being VMware business partners, so I am treating them here as an Independent Software Vendor (ISV). Veeam has not compensated me in any manner for this review, this review is not to be taken as an endorsement of Veeam or its products, and I was not provided any full or evaluator license keys. The review is based entirely on my experience using the "Free Edition" available to all for download. None of this blog post was pre-reviewed by anyone from Veeam. IBM, of course, also offers similar software, which I mention below for comparison purposes.)
At this point, you might be thinking, "Doesn't IBM offer something like this?" Of course it does! IBM is the leader in infrastructure reporting, monitoring and management software. Last October, [IBM unveiled IBM Systems Director VMcontrol] software. Not only does IBM Systems Director VMcontrol provide similar support for your VMware environment, it also manages Microsoft Hyper-V and Xen deployments, PowerVM on POWER-based serves, and even z/VM guest images on the System z mainframes. Combined with the rest of the IBM Systems Director, you can manage all of your physical and virtual servers with a single tool from a single pane of glass. How cool is that?
I would like to think Doug Hazelman, Senior Director of Product Strategy at Veeam, for organizing this awesome blogging contest. If you liked this blog post, click here to [vote for me] to get counted for this contest.