Tony Pearson is a Master Inventor and Senior IT Architect for the IBM Storage product line at the
IBM Systems Client Experience Center in Tucson Arizona, and featured contributor
to IBM's developerWorks. In 2016, Tony celebrates his 30th year anniversary with IBM Storage. He is
author of the Inside System Storage series of books. This blog is for the open exchange of ideas relating to storage and storage networking hardware, software and services.
(Short URL for this blog: ibm.co/Pearson )
My books are available on Lulu.com! Order your copies today!
Safe Harbor Statement: The information on IBM products is intended to outline IBM's general product direction and it should not be relied on in making a purchasing decision. The information on the new products is for informational purposes only and may not be incorporated into any contract. The information on IBM products is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. The development, release, and timing of any features or functionality described for IBM products remains at IBM's sole discretion.
Tony Pearson is a an active participant in local, regional, and industry-specific interests, and does not receive any special payments to mention them on this blog.
Tony Pearson receives part of the revenue proceeds from sales of books he has authored listed in the side panel.
Tony Pearson is not a medical doctor, and this blog does not reference any IBM product or service that is intended for use in the diagnosis, treatment, cure, prevention or monitoring of a disease or medical condition, unless otherwise specified on individual posts.
Continuing my week in Chicago, for the IBM Storage Symposium 2008, we had sessions that focused on individual products. IBM System Storage SAN Volume Controller (SVC) was a popular topic.
SVC - Everything you wanted to know, but were afraid to ask!
Bill Wiegand, IBM ATS, who has been working with SAN Volume Controller since it was first introduced in 2003. answered some frequently asked questions about IBM System Storage SAN Volume Controller.
Do you have to upgrade all of your HBAs, switches and disk arrays to the recommended firmware levels before upgrading SVC? No. These are recommended levels, but not required. If you do plan to update firmware levels, focus on the host end first, switches next, and disk arrays last.
How do we request special support for stuff not yet listed on the Interop Matrix?
Submit an RPQ/SCORE, same as for any other IBM hardware.
How do we sign up for SVC hints and tips? Go to the IBM
[SVC Support Site] and select the "My Notifications" under the "Stay Informed" box on the right panel.
When we call IBM for SVC support, do we select "Hardware" or "Software"?
While the SVC is a piece of hardware, there are very few mechanical parts involved. Unless there are sparks,
smoke, or front bezel buttons dangling from springs, select "Software". Most of the questions are
related to the software components of SVC.
When we have SVC virtualizing non-IBM disk arrays, who should we call first?
IBM has world-renown service, with some of IT's smartest people working the queues. All of the major storage vendors play nice
as part of the [TSAnet Agreement when a mutual customer is impacted.
When in doubt, call IBM first, and if necessary, IBM will contact other vendors on your behalf to resolve.
What is the difference between livedump and a Full System Dump?
Most problems can be resolved with a livedump. While not complete information, it is generally enough,
and is completely non-disruptive. Other times, the full state of the machine is required, so a Full System Dump
is requested. This involves rebooting one of the two nodes, so virtual disks may temporarily run slower on that
What does "svc_snap -c" do?The "svc_snap" command on the CLI generates a snap file, which includes the cluster error log and trace files from all nodes. The "-c" parameter includes the configuration and virtual-to-physical mapping that can be useful for
disaster recovery and problem determination.
I just sent IBM a check to upgrade my TB-based license on my SVC, how long should I wait for IBM to send me a software license key?
IBM trusts its clients. No software license key will be sent. Once the check clears, you are good to go.
During migration from old disk arrays to new disk arrays, I will temporarily have 79TB more disk under SVC management, do I need to get a temporary TB-based license upgrade during the brief migration period?
Nope. Again, we trust you. However, if you are concerned about this at all, contact IBM and they will print out
a nice "Conformance Letter" in case you need to show your boss.
How should I maintain my Windows-based SVC Master Console or SSPC server?
Treat this like any other Windows-based server in your shop, install Microsoft-recommended Windows updates,
run Anti-virus scans, and so on.
Where can I find useful "How To" information on SVC?
Specify "SAN Volume Controller" in the search field of the
[IBM Redbooks vast library of helpful books.
I just added more managed disks to my managed disk group (MDG), can I get help writing a script to redistribute the extents to improve wide-striping performance?
Yes, IBM has scripting tools available for download on
[AlphaWorks]. For example, svctools will take
the output of the "lsinfo" command, and generate the appropriate SVC CLI to re-migrate the disks around to optimize
performance. Of course, if you prefer, you can use IBM Tivoli Storage Productivity Center instead for a more
Any rules of thumb for sizing SVC deployments?
IBM's Disk Magic tool includes support for SVC deployments. Plan for 250 IOPS/TB for light workloads,
500 IOPS/TB for average workloads, and 750 IOPS/TB for heavy workloads.
Can I migrate virtual disks from one manage disk group (MDG) to another of different extent size?
Yes, the new Vdisk Mirroring capability can be used to do this. Create the mirror for your Vdisk between the
two MDGs, wait for the copy to complete, and then split the mirror.
Can I add or replace SVC nodes non-disruptively? Absolutely, see the Technotes
[SVC Node Replacement page.
Can I really order an SVC EE in Flamingo Pink? Yes. While my blog post that started all
this [Pink It and Shrink It] was initially just some Photoshop humor, the IBM product manager for SVC accepted this color choice as an RPQ option.
The default color remains Raven Black.
Continuing my week in Chicago, for the IBM Storage Symposium 2008, I attended two presentations on XIV.
XIV Storage - Best Practices
Izhar Sharon, IBM Technical Sales Specialist for XIV, presented best practices using XIV in various environments.He started out explaining the innovative XIV architecture: a SATA-based disk system from IBM can outperformFC-based disk systems from other vendors using massive parallelism. He used a sports analogy:
"The men's world record for running 800 meters was set in 1997 by Wilson Kipketer of Denmark in a time of 1:41.11.
However, if you have eight men running, 100 meters each, they will all cross the finish line in about 10 seconds."
Since XIV is already self-tuning, what kind of best practices are left to present? Izhar presented best practicesfor software, hosts, switches and storage virtualization products that attach to the XIV. Here's some quickpoints:
Use as many paths as possible.
IBM does not require you to purchase and install multipathing software as other competitors might. Instead, theXIV relies on multipathing capabilities inherent to each operating system.For multipathing preference, choose Round-Robin, which is now available onAIX and VMware vSphere 4.0, for example. Otherwise, fixed-path is preferred over most-recently-used (MRU).
Encourage parallel I/O requests.
XIV architecture does not subscribe to the outdated notion of a "global cache". Instead, the cache is distributed across the modules, to reduce performance bottlenecks. Each HBA on the XIV can handle about 1400requests. If you have fewer than 1400 hosts attached to the XIV, you can further increase parallel I/O requests by specifying a large queue depth in the host bus adapter (HBA).An HBA queue depth of 64 is a good start. Additional settings mightbe required in the BIOS, operating system or application for multiple threads and processes.
For sequential workloads, select host stripe size less than 1MB. For random, select host stripe size larger than 1MB. Set rr_min_io between ten(10) and the queue depth(typically 64), setting it to half of the queue depth is a good starting point.
If you have long-running batch jobs, consider breaking them up into smaller steps and run in parallel.
Define fewer, larger LUNs
Generally, you no longer need to define many small LUNs, a practice that was often required on traditionaldisk systems. This means that you can now define just 1 or 2 LUNs per application, and greatly simplifymanagement. If your application must have multiple LUNs in order to do multiple threads or concurrent I/O requests, then, by all means, define multiple LUNs.
Modern Data Base Management Systems (DBMS) like DB2 and Oracle already parallelize their I/O requests, sothere is no need for host-based striping across many logical volumes. XIV already stripes the data for you.If you use Oracle Automated Storage Management (ASM), use 8MB to 16MB extent sizes for optimal performance.
For those virtualizing XIV with SAN Volume Controller (SVC), define manage disks as 1632GB LUNs, in multiple of six LUNs per managed disk group (MDG), to balance across the six interface modules. Define SVC extent size to 1GB.
XIV is ideal for VMware. Create big LUNs for your VMFS that you can access via FCP or iSCSI.
Organize data to simplify Snapshots.
You no longer need to separate logs from databases for performance reasons. However, for some backup productslike IBM Tivoli Storage Manager (TSM) for Advanced Copy Services (ACS), you might want to keep them separatefor snapshot reasons. Gernally, putting all data for an application on one big LUNgreatly simplifies administration and snapshot processing, without losing performance.If you define multiple LUNs for an application, simply put them into the same "consistencygroup" so that they are all snapshot together.
OS boot image disks can be snapshot before applying any patches, updates or application software, so that ifthere are any problems, you can reboot to the previous image.
Employ sizing tools to plan for capacity and performance.
The SAP Quicksizer tool can be used for new SAP deployments, employing either the user-based orthroughput-based sizing model approach. The result is in mythical unit called "SAPS", which represents0.4 IOPS for ERP/OLTP workloads, and 0.6 IOPS for BI/BW and OLAP workloads.
If you already have SAP or other applications running, use actual I/O measurements. IBM Business Partners and field technical sales specialists have an updated version of Disk Magic that can help size XIV configurations fromPERFMON and iostat figures.
Lee La Frese, IBM STSM for Enteprise Storage Performance Engineering, presented internal lab test results forthe XIV under various workloads, based on the latest hardware/software levels [announced two weeks ago]. Three workloadswere tested:
Web 2.0 (80/20/40) - 80 percent READ, 20 percent WRITE, 40 percent cache hits for READ.YouTube, FlickR, and the growing list at [GoWeb20] are applications with heavy read activity, but because of[long-tail effects], may not be as cache friendly.
Social Networking (50/50/50) - 50 percent READ, 50 percent WRITE, 50 percent cache hits for READ.Lotus Connections, Microsoft Sharepoint, and many other [social networking] usage are more write intensive.
Database (70/30/50) - 70 percent READ, 30 percent WRITE, 50 percent cache hits for READ.The traditional workload characteristics for most business applications, especially databases like DB2 andOracle on Linux, UNIX and Windows servers.
The results were quite impressive. There was more than enough performance for tier 2 application workloads,and most tier 1 applications. The performance was nearly linear from the smallest 6-module to the largest 15-module configuration. Some key points:
A full 15-module XIV overwhelms a single SVC 8F4 node-pair. For a full XIV, consider 4 to 8 nodes 8F4 models, or 2 to 4 nodes of an 8G4. For read-intensive cache-friendly workloads, an SVC in front of XIV was able to deliver over 300,000 IOPS.
A single node TS7650G ProtecTIER can handle 6 to 9 XIV modules. Two nodes of TS7650G were needed to drivea full 15-module XIV. A single node TS7650 in front of XIV was able to ingest 680 MB/sec on the seventh day with17 percent per-day change rate test workload using 64 virtual drives. Reading the data back got over 950 MB/sec.
For SAP environments where response time 20-30 msec are acceptable, the 15-module XIV delivered over 60,000 IOPS. Reducing this down to 25,000-30,000 cut the msec response time to a faster 10-15 msec.
These were all done as internal lab tests. Your mileage may vary.
Not surprisingly, XIV was quite the popular topic here this week at the Storage Symposium. There were many moresessions, but these were the only two that I attended.
Well, it's Tuesday again, and that means IBM announcements!
We've got a variety of storage-related items today, so here's my quick recap:
DS5020 and EXP520 disk systems
[IBM System Storage DS5020]
provides the functional replacement for DS4700 disk systems. These are combined controller
and 16 drives in a compact 3U package.
The EXP520 expansion drawer provides additional 16 drives per 3U drawer. A DS5020 can
support upo to six additional EXP520, for a total of 112 drives per system.
The DS5020 supports both 8 Gbps FC as well as 1GbE iSCSI.
New Remote Support Manager (DS-RSM model RS2)
The [IBM System Storage DS-RSM Model
RS2] supports of up to 50 disk systems, any mix of DS3000, DS4000 and DS5000 series.
It includes "call home" support, which is really "email home", sending error alerts to IBM
if there are any problems. The RSM also allows IBM to dial-in to perform diagnostics before
arrival, reducing the time needed to resolve a problem. The model RS2 is a beefier model
with more processing power than the prior generation RS1.
New Ethernet Switches
With the increased interest in iSCSI protocol, and the new upcoming Fibre Channel over
Convergence Enhanced Ethernet (FCoCEE), IBM's re-entrance into the ethernet switch market
has drawn a lot of interest.
The [IBM Ethernet Switch r-
series] offers 4-slot, 8-slot, 16-slot, and 32-slot models. Each slot can handle either
16 10GbE ports, or 48 1GbE ports. This means up to 1,536 ports.
The [c-series] now offers a
24-port model. This is either 24 copper and 4 fiber optic, or 24 fiber optic.
The "hybrid fiber" SFP fiber optic can handle either single or multi-mode, eliminating the
need to commit to one or the other, providing greater data center flexibility.
The [IBM Ethernet Switch B24X]
offers 24 fiber optic (that can handle 10GbE or 1GbE) and 4 copper (10/100/1000 MbE RJ45)
Storage Optimization and Integration Services
[IBM Storage Optimization and
Integration Services] are available. IBM service consultants use IBM's own
Storage Enterprise Resource Planner (SERP) software to evaluate your environment and provide
recommendations on how to improve your information infrastructure. This can be especially
helpful if you are looking at deploying server virtualization like VMware or Hyper-V.
As people look towards deploying a dynamic infrastructure, these new offerings can be a
Over on his Backup Blog, fellow blogger Scott Waterhouse from EMC has a post titled
[Backup Sucks: Reason #38]. Here is an excerpt:
Unfortunately, we have not been able to successfully leverage economies of scale in the world of backup and recovery. If it costs you $5 to backup a given amount of data, it probably costs you $50 to back up 10 times that amount of data, and $500 to back up 100 times that amount of data.
If anybody can figure out how to get costs down to $40 for 10 times the amount of data, and $300 for 100 times the amount of data, they will have an irrefutable advantage over anybody that has not been able to leverage economies of scale.
I suspect that where Scott mentions we in the above excerpt, he is referring to EMC in general, with products like
Legato. Fortunately, IBM has scalable backup solutions, using either a hardware approach, or one purely with software.
The hardware approach involves using deduplication hardware technology as the storage pool for IBM Tivoli Storage Manager (TSM). Using this approach, IBM Tivoli Storage Manager would receive data from dozens, hundreds or even thousands
of client nodes, and the backup copies would be sent to an IBM TS7650 ProtecTIER data deduplication appliance, IBM TS7650G gateway, or IBM N series with A-SIS. In most cases, companies have standardized on the operating systems and applications used on these nodes, and multiple copies of data reside across employee laptops. As a result, as you have more nodes backing up, you are able to achieve benefits of scale.
Perhaps your budget isn't big enough to handle new hardware purchases at this time, in this economy. Have no fear,
IBM also offers deduplication built right into the IBM Tivoli Storage Manager v6 software itself. You can use sequential access disk storage pool for this. TSM scans and identifies duplicate chunks of data in the backup copies, and also archive and HSM data, and reclaims the space when found.
If your company is using a backup software product that doesn't scale well, perhaps now is a good time to switch over to IBM Tivoli Storage Manager. TSM is perhaps the most scalable backup software product in the marketplace, giving IBM an "irrefutable advantage" over the competition.
Back in June, I mentioned this blog was [Moving to MyDeveloperWorks] which is based on IBM Lotus Connections.
Finally, the move is complete for all bloggers. If you are having problems with the redirects, you might need to unsubscribe and re-subscribe in your RSS feed reader. Here are the new links for several IBM bloggers that have moved over:
This week, I was in the Phoenix area presenting at TechData's TechSelect University. TechData is one of IBM's IT distributors,
and TechSelect is their community of 440 resellers and 20 vendors. This year they celebrate their 10 year anniversary of this event. I covered three particular topics, and I was videotaped for those who were not able to attend my session. (There were very few empty seats at my sessions)
IBM Business Partners now realize that the "killer app" for storage is combining the IBM System Storage SAN Volume Controller with entry-level or midrange disk storage systems for an awesome solution. Solutions based on either the Entry Edition or the standard hardware models can compete well with a variety of robust features, including thin provisioning, vDisk mirroring, FlashCopy, Metro and Global Mirror. This has the advantage that the SVC can extend these functions not just to newly purchased disk capacity, but also existing storage capacity. The newly purchased capacity can be DS3400, DS4700 or the new DS5000 models. This is great "investment protection" for small and medium sized businesses.
LTO-4 drives and automation
The Linear Tape Open (LTO) consortium--consisting of IBM, HP and Quantum--has proven wildly successful, ending the
vendor-lockin from SDLT tape. I presented the latest LTO-4 offerings, including the TS2240, TS2340, TS2900, TS3100
and TS3200. The LTO consortium has already worked out a technology roadmap for LTO-5 and LTO-6. The LTO-4 drives
support WORM cartridges and on-board hardware-based encryption. The encryption keys can be managed with IBM Tivoli Key Lifecycle Manager (TKLM).
SAN and FCoCEE switches
IBM has agreements with Brocade, Cisco and Juniper Networks for various networking gear. I focused on entry-level switches for SAN fabrics, the SAN24B-4 and Cisco 9124, as well as new equipment for Convergence Enhanced Ethernet (CEE),
including IBM's Converged Network Adapater (CNA) for System x servers, and the SAN32B switch that has 24 10GbE CEE ports and 8 FC ports that support 8/4/2 and 4/2/1 SFP transceivers. FCoE Clients that want to deploy Fibre Channel over CEE (FCoCEE) today have everything the need to get started.
The venue was the
[Sheraton Wild Horse Pass Resort and Spa] in Chandler, just south of Phoenix. This compound includes [Rawhide], an 1800's era Western Town attraction, a rodeo arena, and a casino still under construction.
Dinners were held nearby at the infamous
[Rustler's Rooste] Steakhouse on South mountain.
This week, scientists at IBM Research and the California Institute of
Technology announced a scientific advancement that could be a major
breakthrough in enabling the semiconductor industry to pack more power
and speed into tiny computer chips, while making them more energy
efficient and less expensive to manufacture. IBM is a leader in
solid-state technology, and this scientific breakthrough shows promise.
But first, a discussion of how solid-state chips are made in the first place. Basically, a round thin wafer is etched using [photolithography]
with lots of tiny transistor circuits. The same chip is repeated over
and over on a single wafer, and once the wafer is complete, it is
chopped up into little individual squares. Wikipedia has a nice article
on [semiconductor device fabrication], but I found this
[YouTube video] more illuminating.
Up until now, the industry was able to get features down to 22
nanometers, and were hitting physical limitations to get down to
anything smaller. The new development from IBM and Caltech is to use
self-assembling DNA strands, folded into specific shapes using other
strands that act as staples, and then using these folded structures as
scaffolding to place in nanotubes. The result? Features as small as 6 nanometers. How cool is that?
While NAND Flash Solid-State Drives are available today, this new
technique can help develop newer, better technologies like Phase Change
Continuing my week in Chicago, at the IBM System x and BladeCenter Technical Conference, I attended an
awesome session that summarized IBM's Linux directions. Pat Byers presented the global forces that are
forcing customers to re-evaluate the TCO of their operating system choices, the need for rapid integration
in an ever-changing business climate, government stimulus packages, and technology that has enabled much
better solutions than we had during the last economic turn-down in 2001-2003.
IBM has been committed to Linux for over 10 years now. I was part of the initial IBM team in the 1990s to work on Linux for the mainframe. In various roles, I helped get Linux attachment tested for disk and tape systems, and helped get Linux selected as an operating system platform of choice for our storage management software.
Today, Linux-based server generate $7 Billion US dollars in revenues. For UNIX customers, Linux provides greater flexibility for hardware platform. For Windows customers, Linux provides better security and reliability.
Initially, Linux was used for simple infrastructure applications, edge-of-the-network and Web-based workloads.
This evolved to Application and Data serving, Enterprise applications like ERP, CRM and SCM. Today,
Linux is well positioned to help IBM make our world a smarter planet, able to handle business-critical applications. It is the only operating system to scale to the full capability of the biggest IBM System x3950M2 server.
Pat gave an examples of IBM's work with Linux helping clients.
City of Stockholm
The city of Stockholm, Sweden introduced congestion pricing to reduce traffic.
IBM helped them deploy systems to collect tariffs from 300,000 vehicles a day, with real-time scanning and recognition of vehicle license plates, Web-accessible payment processing, and analytics for metrics and reporting. This configuration was able to
[reduce traffic by 25 percent in the first month].
IBM helped [ConAgra Foods] switch their SAP environment from a monolithic Solaris on SPARC deployment, to a more distributed one using Novell SUSE Linux on x86. The result? Six times faster performance at 75 percent lower total cost of ownership!
IBM's strategy has been to focus on working with two of the major Linux distributors: Red Hat and Novell. It also works with [Asianux] which is like the UnitedLinux for Asia, internationalized for Japan, Korea, and China. It handles special requests for other distributions, from CentOS to Ubuntu, as needed on a case by case basis.
IBM's Linux Technology Center of 600 employees help to enable IBM products for Linux, make Linux a better operating system, expand Linux's reach, and help drive collaboration and innovation. In fact, IBM is the #3 corporate contributor to the open source Linux kernel, behind Red Hat (#1) and Novell (#2). For most IBM products, IBM tests with Linux as rigorously as it does Microsoft Windows. IBM offers complete RTS/ServicePac and SupportLine service and support contracts for Red Hat and Novell Linux.
At the IBM Solutions Center this week, several booths used Linux bootable USB sticks to run their software.
[Novell SUSE Studio] was developed to help
customize Linux to the specific needs for independent vendors.
Both Red Hat and Novell offer distributions in four categories:
Standard - for small entry-level servers, with support for a few virtual guests
Advanced Platform - for bigger servers, and support for many or unlimited number of virtual guests
High Performance Computing - HPC and Analytics for large grid deployments
Real Time - for real time processing, such as with
[IBM WebSphere Real Time], where
sub-second response time is critical.
A key difference between Red Hat and Novell appears to be on their strategy towards server virtualization.
Red Hat wants to position itself as the hypervisor of choice, for both servers and desk top virtualization, announcing Kernel-based Virtual Machine
[KVM] on their Red Hat Enterprise Linux (RHEL) 5.4 release, and their new upcoming
RHEV-V, a tight 128MB hypervisor to compete against VMware ESXi. Meanwhile, Novell is focusing SUSE to be
the perfect virtual guest OS, being hypervisor-aware an dhaving consistent terms and licensing when run under any hypervisor, including VMware, Hyper-V, Citrix Xen, KVM or others.
IBM has tons of solutions that are based on Linux, including the IBM Information Server blade, the InfoSphere Balanced Warehouse, SAN Volume Controller (SVC), TS7650 ProtecTIER data deduplication virtual tape library, Grid Medical Archive Solution (GMAS), Scale-out File Services (SoFS), Lotus Foundations, and the IBM Smart Cube.
If you are interested in trying out Linux, IBM offers evaluation copies at no charge for 30 to 90 days. For
more on how to deploy Linux successfully on IBM servers, see the
[IBM Linux Blueprints] landing page.
This week, SHARE conference is being held at the Colorado Convention Center in Denver, Colorado. I covered this conference for 10 years earlier in my career. Now, my colleague Curtis Neal covers these on a regular basis, and is giving the following presentations this week:
IBM Virtual Tape Products: DILIGENT ProtecTIER and TS7700 Update
Wednesday (1:30-2:30pm), Curtis will present IBM's premier virtual tape libraries. The TS7650G ProtecTIER Data Deduplication solution supports distributed systems like Windows, UNIX and Windows. The TS7700 supports the IBM System z mainframe.
SAN Volume Controller Update
Thursday (8:00-9:00am), Curtis will cover the latest features of the IBM System Storage SAN Volume Controller (SVC). SVC has features like thin provisioning, vDisk mirroring, and cascaded FlashCopy support.
IBM System Storage DS5000 Update
Friday (8:00-9:00am), Curtis will cover the DS5100 and DS5300, and probably the new DS5020 model. These are midrange disk systems that provide excellent performance for distributed systems, full disk encryption, and intermix of FC and SATA drives.
Unlike other conferences where people just go once and are never seen again, SHARE brings back the same people back year after year, so that you can maintain relationships across organizations, and can carry on forward-looking strategic discussions.
Eventually, there comes a time to drop support for older, outdated programs that don't meet the latest standards. I had several complain that they could not read my last post on Internet Explorer 6. The post reads fine on more modern browsers like Firefox 3 and even Google's Chrome browser, but not IE6.
Google confirms that warnings are appearing:
[Official: YouTube to stop IE6 support].
My choice is to either stop embedding YouTube videos, some of which are created by my own marketing team specifically on my behalf, or drop support for IE6. I choose the latter. If you are still using IE6, please consider switching to Firefox 3 or Google Chrome instead.