Anthony's Blog: Using System Storage - An Aussie Storage Blog
So lets be honest here. Watching corporate advertising on YouTube is not my favourite thing to do. But watching people I know... People who are passionate and articulate and who are talking about a subject they understand with tremendous depth... that's worth taking your time to check out. Brian Carmody is the XIV Technical Product Manager. There are few people who can explain a deeply technical concept as well as he can. I have spotted him in two videos so far. If you can forgive the (very) cheesy music and the shaky handheld-camera-like graphics, listen to Brian, Yossi Siles and Robert Cancilla talking about XIV.
One of the many popular features of the XIV is the ability to replicate using iSCSI. On XIV Gen3 there are now at least 10 and up to 22 active iSCSI ports on each machine (depending on how many modules you order).
Implementation of the iSCSI connection between two XIVs is a piece of cake. If both XIVs are defined to the XIV GUI (which they should be), you just need to drag and drop links between XIVs to bring the iSCSI mirroring connections alive. If the network gods are with you, the link goes green. But... if the networking gods are against you... the links staysred and then the question is... what to do?
Old fashioned problem diagnosis leads us straight to the ping command. However I routinely find that the ping command works fine (all interfaces respond), but the link stubbornly remains red.
The first possible problem is that iSCSI uses TCP port 3260, so hopefully there are no firewalls blocking that port.
The second possible problem is the MTU size (Maximum Transmission Unit). When we define the iSCSI interfaces on the XIV we set the MTU as a value of up to 4500 bytes. When we establish connections between two XIVs, each XIV will send test packets that are sized to the MTU. If the intervening network does not support that packet size, the packets will be dropped by the network, because the XIV sets the don't fragment flag to ON.
So how to work out what the MTU is? Well the first thing to do is ask your friendly networking team member. But sometimes I find that the intervening networks are controlled by third parties, which means that getting a straight (and reliable) answer can prove difficult. Even worse, some of these third parties charge a fee every time you call them, so there may be hesitation to even get them involved!
One simple trick is to re-use the ping command but play with payload sizes. We can use a command that looks like this:
ping -f -l 1472 192.168.0.1
That command sends a ping with a payload of 1472 bytes to IP address 192.168.0.1. We add the -f parameter to prevent packet fragmentation. What you then do is slowly increase the payload until you no longer get a reply.
This process works fine and is great way to determine the maximum payload size the end to end network will support. However if you're using the payload size to determine the maximum transmission unit, there is a little trick. The MTU is the maximum packet size, but a ping sends a payload wrapped in 28 bytes of IP and ICMP headers. So our example:
ping -f -l 1472 192.168.0.1
sends a 1500 byte frame to the 192.168.0.1 IP address (1472 bytes of payload and 28 bytes of headers.
If this command succeeds, you can use an MTU of 1500 in the XIV GUI or XCLI (rather than an MTU of 1472, which is 28 bytes smaller).
For those who are wondering how I did the networking sniffing to get the screen captures above, I used a brilliant piece of freeware software called Wireshark. My only warning is that your corporate security policies may have rules on sniffing the network. Don't take my blog post as permission to use it # And for the networking geeks among you, yes I know that extra packets could actually be wrapped around our ethernet packet for things like VLAN tags or encapsulation, but hopefully this should not affect our mathematics.
Controlling the background traffic
Final pointer. Having finally gotten the link up and going, you are now free to start replicating volumes. But how much traffic can the cross site link support? The XIV can limit the background copy bandwidth with a parameter called max_initialization_rate. This is useful to stop you flooding the cross site link and annoying your link co-tenants. To display the current setting, open an XCLI window and issue the following command:
For each target you should see three parameters:
<max_initialization_rate value="100"/> <max_resync_rate value="300"/> <max_syncjob_rate value="300"/>
These three settings should be tuned to reflect the possible throughput of the cross site links.
To change the settings use a command like this (change the target name and the rates to suit):
target_config_sync_rates target="Remote_XIV" max_initialization_rate=120 max_syncjob_rate=240 max_resync_rate=240
If you want to see the current throughput rate, open XIV Top on the remote machine. You should see how much write I/O is being sent to the mirror target volumes in MBps.
So hopefully your now better positioned to diagnose iSCSI link issues, maximize your MTU and tune and monitor your link speed.
Questions? Fire away...
With the announcement that you can order an XIV with 3 TB SAS disks, IBM now have some amazing capacity options and some equally clever growth options with XIV Gen3.
As you hopefully know, the XIV consists of modules that each contain 12 disks. An XIV can have 6, 9, 10, 11, 12, 13, 14 or 15 modules (all modules must have the same size disk). You can start at any of those points and then grow without interruption or outage up to 15 modules (that's 243.3 TB!). There is practically no planning required to do a capacity upgrade and the data relocation to re-balance between the nodes is done automatically by the machine (without any end-user intervention).
The useable capacity sizing with 3 TB drives stretches from 84.1 TB with 6 modules to 243.3 TB with 15 modules (these are decimal TB).
However the Capacity on Demand (CoD) options are far more interesting. With CoD you effectively buy a certain amount of capacity up front but also get up to 3 more modules shipped with the machine. You can start using this extra capacity when your business requirements demand it, at which point you will be asked by IBM to purchase it. The advantage here is that you physically get a bigger machine up front with all the performance benefits that bestows, plus you don't have to contact IBM to start using that extra capacity. Lets look at the possible configurations.
So lets take a scenario. You need 100 TB today, but you know this will grow to 130 TB over the next 12 months. So you could purchase an XIV with 9 physical modules (using 3 TB drives), with 7 CoD activations. This means IBM ship a machine that physically has 132 TB and that physically has 108 drives in 9 modules. Your data will be spread over all these drives and all of these modules will be active and working. However you have effectively only paid for 103 TB of that space up front. If you order extra CoD activations, you could also order extra physical modules. As long as you stick to the chart above and have at least one un-activated module, you stay in the CoD program.
When your data requirements exceed 103 TB you just start using the extra space, no license keys or special tasks required. Nice!
So having told you how great it is... are there any disadvantages?
1) You need to actually buy the storage... eventually. Depending on the CoD contract there will be a point when IBM expect you to purchase this extra capacity. The whole point of CoD is that it is like pre-ordering capacity without actually paying for it up front. If your really not certain you need extra capacity, your probably better off not ordering CoD capacity in the first place. Instead order capacity upgrades as you require them.
2) There is nothing to stop you using the storage. Now this is a curious disadvantage because it means that if you have paid for 103 TB and you start using 105 TB, the machine will not tell you off, or yell at you. So is this a good thing or a bad thing? Well I really like the flexibility so I think it is a good thing. Plus there is a nice command called cod_list which displays consumed capacity to help keep you on the path. You can also display it in the GUI. So it just means you need to keep an eye on volume and pool creation to ensure you don't start configuring extra capacity until your prepared to pay for it.
You can also use CoD with 2 TB drives on XIV Gen3 so this is another option. With 2 TB drives, the useable capacities look like this:
Questions? Fire away....
It is that time of the year - IBM Storage announcements time.
The number and depth of announcements is a bit over whelming.... there are so many things to talk about.
The good news is that my fellow bloggers are producing mountains of great content so please check them out:
You can also check out a list of all the announcements here.
It seems like only yesterday that it was November 2010.
It was a very clever decision, taking the mature and very popular SAN Volume Controller (SVC) and packaging it with the latest SAS enclosure technology and offering it as a midrange storage product.
Since IBM started shipping the Storwize V7000, they have sold (up until September 2011):
You can find Storwize V7000s in every major country of the world, in every major industry, in every common environment. A phenomenal success, all achieved in less than a year.
So what does the future hold? Watch this space... some very cool news is really close...
For those of you with Apple iPads, you might consider dropping by the Apple Store and picking up your free IBM XIV Mobile Dashboard.
The IBM XIV Mobile Dashboard application can be used to securely monitor the performance and health of your XIV over a Wi-Fi or 3G link. Having downloaded and installed the Mobile Dashboard you will get a lovely XIV Icon:
When you start the Mobile Dashboard you will have the choice to either run in Demo Mode or to connect to an actual XIV. Demo mode can be accessed by selecting the Demo Mode option deep in the lower right hand corner. So you don't actually need an XIV to give it a test drive.
Once connected you have the choice of viewing volume performance or host performance. If you view (hold) the iPad in portrait mode you get a list of up to 27 volumes or hosts ordered by performance metrics (it defaults to ordering by IOPS). If you view the iPad in landscape mode you will get a more graphical output (as per the examples below). There are no options to perform configuration, the dashboard is intended only for monitoring. This means each panel will show the performance and redundancy state of the XIV.
The volume performance panel is shown by default. The example below shows the output when the iPad is operated in landscape mode. From this panel you can see up to 120 seconds worth of performance for a highlighted volume. Use your finger to rotate the arrow on the blue volume icon to switch the display between IOPS, bandwidth (in megabytes per second or MBps) and latency (in milliseconds or MS). The data redundancy state of the XIV is shown in the upper right hand corner (in this example it is in Full Redundancy, but it could be Rebuilding or Redistributing).
The example above shows the output when the iPad is operated in landscape mode. If you instead rotate the iPad to portrait mode, you will get a list of the performance of up to 27 of your busiest volumes.
Now swipe to the left to navigate to the Hosts panel as shown below.
From this panel you can see up to 120 seconds worth of performance for a highlighted host. Use your finger to rotate the arrow on the purple host icon to switch the display between IOPS, bandwidth (in megabytes per second or MBps) and latency (in milliseconds or MS). The data redundancy state of the XIV is shown in the upper right hand corner (in this example it is in Full Redundancy, but it could potentially also be Rebuilding or Redistributing). Swipe to the right to navigate to the Volumes panel.
The example above shows the output when the iPad is operated in landscape mode. If you instead rotate the iPad to portrait mode, you will get a list of the performance of up to 27 of your busiest hosts.
From either the volumes or the hosts panels you can log off from the mobile dashboard using the icon in the upper right hand-most corner of the display. When you log back on, the last used XIV IP address and username will be displayed (but not the password which will need to be entered again).
I can see some nice use cases here. You get a call regarding performance but you are on the road. Are there any problems with the XIV? You can quickly logon with your iPad and confirm if response times are normal and the redundancy state is Full Redundancy.
A better use case... now you can ask your manager to buy you an iPad, so you can monitor your XIV! Let me know how that goes #
For those of you planning to move to ESXi 5.0, IBM have found an annoying (but not show stopping) issue with the way XCOPY is implemented in the VAAI driver. With ESX/EXi 4.1, IBM supplied the VAAI driver, but with ESXi 5.0 this changed and VMware now manage this themselves. It has since emerged that the way VMware implemented XCOPY in this driver does not totally work with the way IBM implemented XCOPY in both the XIV and the Storwize V7000 and SVC.
This is the current situation with the first three VAAI primitives in ESXi 5.0:
Hardware accelerated locking: Also known as Atomic Test and Set (ATS), this function works fine when ESXi 5.0 detects a volume from an XIV, Storwize V7000 or SVC. In fact the moment ESXi 5.0 detects a LUN from any of these products it uses ATS to confirm that VAAI is possible. So this is goodness.
Hardware accelerated initialization: Also known as write same, this function offloads almost all effort on the part of ESXi to write zeros across disks. This function works fine when ESXi 5.0 works with XIV, Storwize V7000 or SVC. So this is also goodness.
Hardware Accelerated Move: Also known as XCOPY, full copy or clone blocks, this function works fine with XIV, Storwize V7000 and SVC if you clone a virtual machine and place the new copy into the same datastore as the source. This means creating multiple clones of a VMDK inside the one datastore will still be accelerated by VAAI. So far so good, but unfortunately on XIV, if you place the clone in a different datastore on the same XIV, it will not be hardware accelerated. This means the clone is still created, but in the old-fashioned way (reading from the source and writing to the target). As for storage vMotion, it also reverts to working in the old-fashioned way, reading from the source and writing to the target, rather than the hardware accelerated way.
So to be clear, this issue with XCOPY:
How will this be fixed? Well right now it looks like it will be fixed in new firmware on the IBM hardware. Watch this space and I will update you as soon as I have more news to hand.
As for the fourth VAAI primitive, unmap, which is used for space reclamation on thin provisioning capable hardware, please also watch this space on when IBM hardware will support it... BUT in my opinion it does not matter right now, because this new unmap function in ESXi 5.0 can potentially cause issues. This is described here: http://kb.vmware.com/kb/2007427
So until VMware confirm a fix, I recommend that you run the following commands on all ESXi 5.0 boxes which connect to IBM Storage to disable unmap. I tested the following syntax to confirm it works:
First confirm the value of the unmap setting (1 means enabled, 0 means disabled):
~ # esxcli system settings advanced list -o /VMFS3/EnableBlockDelete | grep "Int Value" Int Value: 1 Default Int Value: 1
Then disable it:
~ # esxcli system settings advanced set --int-value 0 --option /VMFS3/EnableBlockDelete
Then confirm it is disabled:
~ # esxcli system settings advanced list -o /VMFS3/EnableBlockDelete | grep "Int Value" Int Value: 0 Default Int Value: 1
I just discovered that you can find freaky blue aliens wearing sun-glasses in release 3.0 of the XIV GUI.
Just go to the Users icon (the padlock) and take the option to Add User Group. I think you will find the new grouped users icon rather amusing. I don't know if I prefer the dude with sunglasses or the two faceless people behind him (or her... or it).
Here are two groups I created, inspired by the icon:
If your running in demo mode and you want to have a closer look, just go to the padlock icon on the XIV Demo2 machine.
You gotta love that icon... #
Actually.... the purpose of groups is to let you group together users in the Application Admin category who manage the same hosts. Application Admin users can only work with hosts that are assigned to them. This limits their access to only the volumes that are mapped to those hosts. A great (and safe) way to hand out user-ids to say SAP developers who want to snapshot as they go. With the added bonus of a cool spaceman icon.
anthonyv 2000004B9K Tags:  ibm interoperation interoperability ssic interop storage 6,215 Views
The eternal question: Which hardware/software combinations are tested and supported? If you use IBM Storage hardware and you need to answer this question you need to be using the IBM System Storage Center, or , which you will find here:
I use this site a lot and rely heavily on the output it creates. I thought I knew the site well, but I recently learnt some really handy tricks that you might find helpful...
1) Export all the data for a single product version.
If you want to download every interoperability test result for a specific product version, you can select the relevant version from the Product Version box of the and then select Export Selected Product Version (xls).
In the example below we want to see all the results for XIV Gen3 which uses XIV Software version 11.
a) Use the scroll bar in the Product Version box to bring up the XIV product versions. Youdon't need to make a selection from the Product Family or Product Model boxes.
b) Select IBM Storage System (11) from the Product version list.
c) Select the option to Export Selected Product Version (xls). A spreadsheet compressed into a ZIP file will be downloaded in your browser.
So that is just two clicks and the result is a giant spreadsheet. Reminds me of when matrices were giant .
2) Changing your selections from an existing search
As you make selections, the webpage leaves what are called cookie crumbs. They will appear at the top of the page and can be seen in the example below, numbered 1 to 6. You can use those cookie crumbs to go backwards at any point, to any point.
3) Start anywhere
It seems human nature to always start at the top and work downwards. But in fact you can start anywhere on the and work in any direction. There are no real restrictions on the combinations you can attempt to build. Every time you make a selection in a different box, the number of configuration results will drop. For instance just click in the Connection Protocol box... or just select IBM AIX 7.1 from the Operating System box. Then work up or down from there.
Hopefully these suggestions will help you work more effectively with the .
I know the walls are coming down... but there are still many organizational barriers that can exist in IT. How about:
Team work and co-operation? Sure it's an option.... but then an option means its optional.... right?
So when vendors come along with plug-ins and products that dare to connect two worlds... is this a unifying force, or is it anti-matter, or do they just get ignored and not used?
A case in point being the IBM Storage Management Console for VMware vCenter which you can download from here. I have written about this plug-in before, but with the release of version 2.6 (that supports vSphere 5.0), I thought I would try something out. Installing the plug-in potentially offloads a lot of storage management from the storage admin to the VMware admin. But what if the storage admin does not WANT to offload this work?
The answer is to give the VMware admin read-only access.
When you configure your IBM storage device to the plug-in, you supply the plug-in with log-in credentials (so it can log into your IBM storage device and collect the required information). If the user-id supplied only has read-only access to the XIV for instance, the plug-in still works... but not for any operations that change resources. You cannot see the pools on the XIV, but you can still see your volumes and any snapshots that have been created (but annoyingly you cannot see mirrors).
This does have one big advantage. You can clearly match the VMware datatstore name to the XIV volume name. You can also identify which XIV supplied the volume.
I also tested this with Storwize V7000 with a user in the Monitor category and got pretty well the same results. A nice bonus is that I could also see the state of the mirrors as well as the flashcopies. In the example below, all of this information would normally not be visible to the VMware admin, so this is very handy stuff.
Of course I get to also visit one-man bands where the same (exhausted) individual manages the VMware servers, the Operating System Guests, the Network, the Firewall, the Exchange server, the SQL servers and pretty well everything else including getting the elevators and coffee machine fixed. For those people, they need all the help they can get.
I have had some reports of clients upgrading to XIV GUI 3.0 and seeing occasional timeouts connecting to their XIVs. This is annoying management issue rather than a data access issue. This appears to only occur when the machine is in a remote location to the person managing the XIV. One simple solution would be to install the GUI on a server local to the XIV and use a remote desktop connection to run the GUI from that server. But there is a way to modify the XIV GUI timeouts and eliminate this issue regardless of where you are:
1) Shutdown the XIV GUI.
2) Edit the following file: xiv-constants.properties
The location for this file in Windows is either:
Open the file in notepad or some other text editor and change:
# Timeouts (seconds) connect=8 command=30
# Timeouts (seconds) connect=30 command=30
I found that with Windows 7 I needed to run the text editor as Administrator to stop the file being opened as read only (meaning I could not save the changes).
3) Start the GUI and confirm there are no more timeouts.
I see this comment occasionally and it makes me bristle:
"Disk and tape traffic must be in separate zones because they have different characteristics. If they are in the same zone this can cause performance problems or have other adverse affects."
True statement or totally wrong?
Well actually kinda wrong... If you have a single HBA with only one port, then separating disk and tape traffic into separate zones does NOTHING to get around the fact that both streams of traffic (random bursty disk traffic and sequential streaming tape traffic) are all coming from or going to the same HBA port. All zoning achieves in this case is stop the disk and tape devices from talking to each other (something that in general they are not interested in doing).
Separating disk and tape traffic is best done with separate HBA ports (whether thats a dual ported HBA or two single port HBAs).
So lets change the wording:
"Disk and tape traffic should ideally be handled by separate HBA ports, because they have different characteristics. If both traffic types use the same HBA port this can cause performance problems or have other adverse affects."
Guess what? This advice has not changed since 2001.
Now that IBM is shipping the XIV Gen3, a new version of the XIV Host Attachment Kit has been released: version 1.7. This new release brings in support the XIV Gen3 (or more specifically, XIV Firmware 11.0). You can download the new HAK from here.
What changed? Well there are the usual lists of enhancements and fixes in the release notes, but what I really like is that the HAK now comes in a portable edition. So instead of installing the HAK locally on each host, it can be run from a mounted network drive or from a portable USB flash drive. Nice!
For every Operating System HAK, there is a portable version. Meaning AIX, HP-UX, Linux and Solaris can all be configured without installing a separate piece of software. The python needed to run the scripts is all encapsulated. This is a really cool enhancement.
I tried it out on a Windows 2008 R2 machine that did not have a fibre channel HBA or any pre-existing HAK. All I did is download the portable kit as a zip and unpack it. Then open a command prompt and go to the bin directory of the unpacked zip.
I ran the xiv_fc_admin -V command since that tells me whether all the pre-reqs have been met without changing anything:
\host_attach\bin>xiv_fc_admin -V Cleaning up previous versions... OK Modifying Disk Timeout settings... OK Multi-Path I/O Feature for Windows 2008 (R2)... NOT OK Configure Multi-Path I/O feature to work with XIV Devices NOT OK Clean-up deprecated XIV MPIO Load Balancing Service... OK Cleaning Depricated XIV Legacy MultiPath Management Agent OK Windows Hotfix 2460971... NOT OK Windows Hotfix 2460971... NOT OK Windows Hotfix 981208... NOT OK Windows Hotfix 979711... NOT OK Lun 0 enabler for Windows... NOT OK
Now that I know that most the pre-reqs were not met, I ran xiv_attach to get the host attachment kit to install the relevant hotfixes (they get provided with the zip so nothing needs to be downloaded):
portable\host_attach\bin>xiv_attach ------------------------------------------------------------------------------- Welcome to the XIV Host Attachment wizard, version 1.7.0. This wizard will assist you to attach this host to the XIV system. The wizard will now validate host configuration for the XIV system. Press [ENTER] to proceed. ------------------------------------------------------------------------------- Please choose a connectivity type, [f]c / [i]scsi : f ------------------------------------------------------------------------------- Please wait while the wizard validates your existing configuration... The wizard needs to configure the host for the XIV system. Do you want to proceed? [default: yes ]: Please wait while the host is being configured... ------------------------------------------------------------------------------- A reboot is required in order to continue. Please reboot the machine and restart the wizard Press [ENTER] to exit.
Now that the xiv_attach command has been run, I ran xiv_fc_admin -V again to see what changes occurred. One hotfix still showed as not OK, but that changed after I rebooted the server.
portable\host_attach\bin>xiv_fc_admin -V Cleaning up previous versions... OK Modifying Disk Timeout settings... OK Multi-Path I/O Feature for Windows 2008 (R2)... OK Configure Multi-Path I/O feature to work with XIV Devices OK Clean-up deprecated XIV MPIO Load Balancing Service... OK Cleaning Depricated XIV Legacy MultiPath Management Agent OK Windows Hotfix 2460971... OK Windows Hotfix 2460971... OK Windows Hotfix 981208... OK Windows Hotfix 979711... NOT OK Lun 0 enabler for Windows... OK
I then turned to a second machine that actually has a fibre channel HBA and is attached to an XIV with release 1.6 of the HAK already installed. This is where things got interesting. Because 1.6 is installed, I could use the already installed 1.6 version or the new portable 1.7 version (unpacked in a folder or on a USB stick). First I ran the portable 1.7 version of xiv_fc_admin -V command to check what issues need to be corrected.
portable\host_attach\bin>xiv_fc_admin -V Cleaning up previous versions... OK Modifying Disk Timeout settings... OK Multi-Path I/O Feature for Windows 2008 (R2)... OK Configure Multi-Path I/O feature to work with XIV Devices... OK Clean-up deprecated XIV MPIO Load Balancing Service... OK Cleaning Depricated XIV Legacy MultiPath Management Agent... NOT OK Windows Hotfix 2460971... OK Windows Hotfix 2460971... OK Lun 0 enabler for Windows... OK
I then ran the command with the -C parameter to get it to correct any detected issues. I could have run xiv_attach, but the xiv_fc_admin -C command shows you what changes it made:
portable\host_attach\bin>xiv_fc_admin -C Cleaning up previous versions... OK Modifying Disk Timeout settings... OK Multi-Path I/O Feature for Windows 2008 (R2)... OK Configure Multi-Path I/O feature to work with XIV Devices OK Clean-up deprecated XIV MPIO Load Balancing Service... OK Cleaning Depricated XIV Legacy MultiPath Management Agent OK Windows Hotfix 2460971... OK Windows Hotfix 2460971... OK Lun 0 enabler for Windows... OK
I then ran xiv_attach to be certain (since that also scans for new devices).
Now that I was happy with the portable version of 1.7, I downloaded the full release of version 1.7 and installed it. There was no requirement to reboot. I could confirm the system version was upgraded with xiv_fc_admin -v
C:\Users\Administrator>xiv_fc_admin -v 1.7.0
There are some great possibilities with the new portable version:
Three quick facts for you:
1) XIV Gen3s are now shipping.
2) Orders for XIV Gen3 are flowing in.
3) The XIV Release GUI is available for everyone to download and use.
Links for the GUI are here. My personal recommendation is that you uninstall the old 2.4.x GUI before installing the new 3.x GUI. Remember you do not need a Gen3 to use this great new GUI. It works just fine with the 2nd Generation XIVs. So get downloading and tell me what you think.
One of the things I like about creating host definitions on XIV, Storwize V7000 or SVC is that for the vast majority of host types, the default choice is the correct choice. This saves me having to specifically tell the storage what operating system each host is running.
However in recent times more and more choices have snuck in. For instance in 184.108.40.206 code the Storwize V7000 and SVC had to add an extra option for OpenVMS hosts (there is an alert related to this here). Fortunately most of these exceptions are (from my perspective) for relatively obscure operating systems (users can start spamming me now).
So you need to warn the XIV if your host is running HP-UX or :
On the SVC or Storwize V7000 you need to change from default if your host is running HP-UX or OpenVMS. There is also TPGS (or Target Port Group Support) for use with Apple OS and Solaris hosts using MPxIO. For more details visit this link for Solaris and this linkfor Apple.
Why does this requirement exist? Or put differently, why can't the storage be smart enough to just detect the host OS and save the storage admin this bother? Sadly it all has to do with how the hosts issues and responds to SCSI commands.
HP-UX for instance uses flat space addressing (rather than peripheral device addressing). If the storage does not respond in the expected way it will see only 8 LUNs per port (which is not many). I have seen some strange things when HP-UX hosts have not been defined correctly. I have seen even stranger things when non HP-UX have been defined as HP-UX!
OpenVMS on the other hand gets very upset if the storage reports itself compliant with a SCSI specification higher than SPC2 (which is now a very old SCSI specification).
The good news is that for AIX, Linux, Windows and VMware hosts, default works just fine.
If your familiar with IBM Redbooks you will be familiar with the group photograph that appears at the start of the book. I was surprised to learn that there are quite a few rules on what can and cannot appear in the photo. So here is a photo that will not be appearing in any of the XIV IBM Redbooks.
The back row is Roger Eriksson, Markus Oscheka, Rainer Pansky, Pete Wendler, Anthony Vandewerdt, Suad Musovich.
The photo was taken outside IBM Mainz in Germany.
Having worked in IT for 25 years I can definitely say that while there are nations that are slowly eliminating the gender bias (often way too slowly), parts of the IT industry lag seriously behind. You only have to read this article by Wendy Schuchart to see we still have some way to go. I just hope the people who organize these sort of events are listening to people like Wendy, but I sadly doubt it. Of course EMC or VMware may blame their external partners, but it is not difficult to discourage this sort of behaviour.
Maybe it's time for some positive discrimination? IBM has a great track history here, way back to the early 30s. But while the sales part of the business appears to attract women and men in fairly even numbers, the more technical roles remain male dominated. I have never been certain why (though I have some ideas).
So if you are responsible in any way in organising IT industry events, please read Wendy's article and share it around. Consider it mandatory reading:
Of course there are worse industries out there. Just look at auto industry trade shows.
For the next few weeks I may not blog as frequently. It's not that I am tired of writing, but I will be writing on a different subject: Helping to update the four XIV Redbooks.
IBM Redbooks are one of the many ways that IBM differentiates itself from its competitors. They are very detailed how-to guides that IBM gives away for free, you don't even need a userid to download them. IBMs customers and business partners use them extensively and IBMs competitors love them too (for different reasons).
With the General Availability date of XIV Gen3 rapidly approaching, the Redbooks of course need to be updated. We have assembled a great team of residents (which due to the timing of the residency, could not include non-IBMers) and have started our chapter updates. Some of the chapters I have already picked up include Monitoring, Volume Copies and SVC migration with XIV. I am re-learning the joys of Adobe FrameMaker and have actually had the pleasure of running I/O to an XIV Gen3 (in fact two of them).
So my only question to all of you is: What do you want to see in the next XIV Redbook?
anthonyv 2000004B9K Tags:  unix storwize putty ssh client svc brocade logging shell v7000 25,352 Views
I love PuTTY, it is one of my favourite pieces of open source software. For those who don't know what PuTTY is, it is a free implementation of Telnet and SSH for Windows and Unix platforms. Personally I use PuTTY for connecting to Storwize V7000s, SVCs, SAN switches and Unix servers in my lab and at clients.
Being someone who does implementation services, I want to always be sure about what I do, what I did and what affect it had. So keeping logs of every time I connect to any device is very important to me. I want to be able to go back in time to any point at any client I have ever worked with (sounds like time travel, but hopefully you get the idea).
I achieve this by changing the default behaviour of Putty so that every new Session I save uses my preferred default settings. Note all the screen captures below use version 0.61 of PuTTY (which came out July 2011).
First start up PuTTY. We are going to change the Default Settings (which I have highlighted):
Now select Logging from the left hand panel and change all the indicated settings (2, 3 and 4). The folder I use is C:\PuttyLogs but you could of course choose a different one. Note the &H_&Y&M&D&T means the session file name will be the host name, year, month, day and time for that session.
Now go back to Session, highlight Default Settings and hit Save:
Now every new session you open and/or save will automatically log your session output to the specified folder. Existing Saved Sessions will not be changed. You will need to update each of these separately.
The Storwize V7000 is a modular storage system. This means you buy it in enclosures that are two rack units (2U) in size. The first enclosure you buy is a control enclosure which contains two node canisters and two power supplies (with integrated batteries).
The purpose of the batteries (that are only found in the control enclosure) is to protect the node canisters from brown-outs (so they can ride through sudden drops in input voltage) and to give the node canisters time to gracefully shutdown should power be lost. Any data in the cache of the Storwize V7000 will be persistently written to the internal solid state drives. So the batteries are only needed if power is lost and only used while the nodes are shutting down. The Storwize V7000 Information Center contains a great write-up on how this works here.
In the image below you can see a control enclosure battery unit on the right, with an arrow to show where it fits into a control enclosure power supply.
On August 9, 2011 IBM announced that the feature code (8001) for the Storwize V7000 battery was going away without replacement. This left some people a little bit concerned as to what was happening with the batteries. We don't want to have a Storwize V7000 without batteries! The answer is that IBM had separate feature codes for the control enclosure power supplies (Feature Code 9801) and the batteries (Feature Code 8001). Since you cannot have one without the other, IBM have decided to remove one of the feature codes (the battery feature code). Thus this change is purely an administration change and the machine will continue to ship with two batteries. Phew!
With all my blogging about the XIV Gen3, I am sure there are 2nd Generation XIV users out there thinking: Whats in it for me? I already bought an XIV! So for all of you, I have some really cool news: Release 3.0 of the GUI is coming and it will work with your machine.
The current XIV GUI (as of this writing) is version 2.4.4 Build 3:
When we start shipping Gen3 XIVs in September we will also release new GUI software at the same time. This new GUI (release 3.0) will work with both 2nd Generation XIVs as well as Gen3 XIVs. I currently have a pre-release (beta) version so the final release may look slightly different. Swish huh?
Any changes? Here are a few I have spotted so far (I am not going to list them all):
Binary volume sizes can now be selected at volume creation time. It even uses my preferred terminology (GiB as well as GB). In addition a re-sizable box shows the space being used in the pool by this new volume (or volumes).
Simplified access to demo mode (no need to know the quirky p10demomode userid):
You can now easily spot the interface modules as they are highlighted.
Improved module health display showing module temperature. I was initially not amused when the temperature came up in Fahrenheit, but all you do is click on the temperature and it changes to Centigrade (and remembers your preference). Which is cool.
Revamped pool creation and information displays. Check out the pool titled Thin Pool, it's wearing a bright yellow belt to show it's a thin provisioned pool. The wording used for pool usage has also been improved. Nice!
There are some clever changes to the statistics panel such as simplified sliders and cool changes to the way you can manage the displayed time scale in multiple windows.
There are lots of other subtle changes too, particularly around multi-systems management and alerting, including a separate Alerts panel and new pop-ups for hardware failures. I could keep going for some time.... You will not be disappointed. They have taken a great GUI and made it even better.
Now in case you're rushing to download it, let me be clear: This new GUI is not available yet and anything I have shown you is subject to change. As soon as you can download it I will let you know.
One of Melbourne's main newspapers is The Age and like many newspapers it can be accessed via a Web Browser, including a mobile website. Most articles that appear on the mobile site have advertisements by Google, as well as featured advertising. Now I cannot blame The Age for using advertising to subsidize their website; after all if I visit their website I may not purchase an actual paper. But I do ponder what mechanism exists for selecting the advertisements. Let look at some examples:
On April 28, 2011 The Age carried a story about a tourist who nearly killed a crocodile by feeding it a brick. So what did Google suggest? A brick matching service of course!
In an article on May 3, 2011 regarding the death of Osama Bin Laden and how he had evaded capture up until his death, what did Google suggest? How about a mobile phone stealth product and home conveyancing! Unsure who they were targeted to?
On August 23, 2011 an article ran about the Federal Labour politician Craig Thomson who is having problems with allegations regarding the use of a credit card to pay for escort services. So what did Google suggest? Cable TV () and Credit Card debt relief! Perhaps they are suggesting that watching the right channels on could prevent embarrassing credit card charges?
Probably the worst example of Google confusion is the story of the bully videothat The Age published on March 21, 2011. Bullying is a terrible thing and should not be made light of, but considering that one of the bullied children reported being abused by being tied up with duct tape, Google's suggestion was rather shocking:
So have you seen worse examples? Curious to see if other websites are having the same issues.
anthonyv 2000004B9K Tags:  heatmap ds8700 cmua00007e ds8800 easy tier storwize stat svc v7000 15 Comments 13,749 Views
The IBM Storage Tier Adviser Tool (known as STAT for short) is a clever piece of software that lets you predict how much business value you would get from adding SSDs to your Storwize V7000, SVC, DS8700 or DS8800. This is because you can add SSDs to all of these products and then have hot spots dynamically and automatically migrated to SSD using IBM's Easy Tier technology (which is offered as a no charge feature).
For clients who have not yet purchased SSDs, or who are unsure which Storage Pools to deploy them into, the STAT tool will help with decision-making.
I recently struck a rather simple problem with the STAT tool after installing it: I kept getting a CMUA00007E error. I downloaded the tool from here and installed it successfully onto my Windows 2008 64-bit lab machine (running on a IBM x3850). The install went fine so I then proceeded to download the heatmap from my Storwize V7000. The heatmap file is automatically generated by Easy Tier and is used as an input file for the STAT tool. You can see an example of where to find the heatmap file in the screen capture below:
I then placed the heatmap into the same folder as the STAT tool and tried to generate a report. It failed with this rather annoying message:
C:\Program Files (x86)\IBM\STAT>stat dpa_heat.78G01A6-2.110823.072309.data CMUA00007E The STAT.exe command failed to produce the heat distribution output.
I initially thought I had a bad heatmap, but since it is a binary file, opening it in a text editor did not tell me anything.
Actually the issue was simple: I did not have write authority to that folder. To get around this I instead started the command prompt as Administrator:
I then re-ran the command:
C:\Program Files (x86)\IBM\STAT>stat dpa_heat.78G01A6-2.110823.072309.data CMUA00019I The STAT.exe command has completed.
Having run the tool I was now able to open the index.html in the STAT folder and see much hot data I have in my lab. Turns out that I don't actually have any hot data right now! Don't tell my manager though, he might try to take my SSDs away. #
Having run the tool once, I did not need to use this trick again. It now runs without starting the command prompt as Administrator.
The IBM XIV Gen3 was announced July 12, 2011 with a planned availability date of September 8, 2011. So far I have written blog articles about the changes to the rack, the layout and the disk drives. So it's now time to head around to the back of the rack!
The first thing I like to point out when showing clients a 2nd Generation XIV is the patch panel. This is a really nice innovation that places all the external connections (Fibre Channel SAN, iSCSI LAN, Management LAN and Remote and Local Support connections) into the one easily accessible place. Users not only like the simplified layout, but also appreciate that they can run cables to the patch panel through the top of the machine or from under the floor.
The big change in the IBM XIV Gen3 is that the developers got very excited about iSCSI connections, provisioning a staggering 22 ports (on a fully provisioned machine). This means the patch panel had to be redesigned to accommodate these extra ports.
In the examples below (taken from the GUI), active ports are coloured white while inactive ports are yellow.
Here is a picture showing the rear view of the rack with the two section patch panel indicated.
As I have explained in previous posts, you can get Visios of the XIV patch panels and the XIV racks from here. If there is another Visio stencil you would like to see, feel free to leave a comment and I will get busy.
Appears this is hardly new, but certainly inventive!
For some reason I am not inclined to open the attachment. Amusingly if you Google
Anyone else out there a lead foot driver when in New York? Even when your not even there?
When your buying a car it's worth opening the boot (that's the trunk for my American friends) and looking at the spare to see what kind it is. Car manufacturers sometimes choose to sacrifice a full spare to save space (or cost). I can understand the motivation, but there is nothing worse than having a flat and then finding that your spare is half the tyre you expected it to be.
With low end fibre channel switches there is a similar challenge when working to achieve the lowest cost and smallest footprint in a 1U form factor. The main way it expresses itself is with power supplies.
The IBM SAN24B-4 (a 24 port switch) is a case in point as it only has one fixed power supply. This of course means that if that power supply fails, then that switch will be down and the entire unit will have to be replaced. If you have dual fabrics (two switches) with dual pathed hosts, then this should not cause an application outage, but it may not be what you expected. You also need to ensure that each switch is connected to a different power rail (and/or UPS) to cater for building power issues.
How to avoid this? Purchase the slightly larger IBM SAN40B-4 (a 40 port switch) which comes with two hot swap power supplies as standard. It's a little more expensive (which makes sense as it has more ports and more hardware) but also offers redundant power and greater scaleability.
Of course in the end you need to select the switch which matches your budget. The SAN24B-4 starts at only 8 ports active while the SAN40B-4 starts at 24 ports active, so the SAN24B-4 will always be a cheaper purchase. The SAN24B-4 is also much smaller, lighter (it weighs 4.35 kg versus 9.34 kg) and uses less power (48 W versus 84 W).
My preference? Well I would always choose to use a switch with dual redundant hot-swappable power supplies, but then I am not the person signing the cheques. What I would suggest is that if you choose the SAN24B-4 then you need to ensure your backing up your switch config (especially if your running single switch fabrics). You could look atSimply-Save-Your-SAN as one way to do this.
And no... that is not my car. By shear synchronicity I was thinking about this issue when I spotted this car in the carpark at Southgate. Timing is everything.
IBM today announced two new disk choices for the Storwize V7000 in the 2.5-inch Small Form Factor:
This means there are now a total of 8 disk choices for Storwize V7000:
One nice thing that the Announce letter does not mention is that you will NOT need to do a firmware update to use these new disks. All you need to do is plug them in.
anthonyv 2000004B9K Tags:  v7000 ibm bladecenter storwize module support brocade switch xiv 21,610 Views
IBM have offered Brocade switch modules for their BladeCenters for many years. One common question I get asked regards which of these Switch Modules are supported with the IBM Storwize V7000 or IBM XIV.
Your first port of call is the System Storage Interoperation center or SSIC. However that will only list the switch modules by the IBM Feature Part Number (for example 26K5601), which may confuse you. So to help you determine which switch module you possess, here is a history of Brocade SAN switch modules for IBM BladeCenter:
2 Gbps Brocade Switch Modules for IBM BladeCenter
There were two switches but they are actually the same switch with different software features. The Enterprise version had extra licenses like Trunking and Extended Fabrics. These products were withdrawn from marketing on May 26, 2006.
4 Gbps Brocade Switch Modules for IBM BladeCenter
IBM offered two models which were physically identical. You could upgrade from the 10 port to the 20 port with a software activation key. These products were withdrawn from marketing on December 31, 2010.
There are actually three models available but I have collapsed them down to two. The 20 port switch is offered as both Enterprise and non-Enterprise. The 42C1828 switch module is the Enterprise version that has more licensed software features such as Trunking, Fabric Watch and Extended Fabrics (and is indicated with an *).
At the moment the SSIC does not list every switch module. However support is available for many configurations via the IBM SCORE request system (sometimes called an RPQ). Your IBM pre-sales storage specialist can raise one of these. Depending on your request you may get a support statement as quickly as overnight.
If your wondering what an IBM FRU number is: a FRU or Field Replacement Unit is the part number used in the IBM spare parts system to replace your part under warranty or maintenance agreement.
If you want the information above in a spreadsheet format, you can find it here.
I was inspired by this article on CBC News regarding the 30th anniversary of the launch of the IBM PC. That's right, on Friday August 12, 2011 the IBM PC turned 30.
Of course there were plenty of alternatives out there, but the IBM PC set standards that changed the industry forever (and IBM!). There is some great material in the IBM archives. Check them out here.
My first computer? An Exidy Sorcerer that I purchased around 1982. There it is in my bedroom (checkout the wood paneling and macrame plant holder!). It had 32KB of RAM plus plugable cartridges and a cassette tape recorder for storage.
I sold it in 1984 to a Doctor who paid far more than I initially did. He was running his whole surgery on a Sorcerer and desperately needed another one for parts. Tells you something about the risks of writing software for a closed platform.
My next computer was a 512 KB Apple Macintosh that I bought in 1985 through the University of Western Australia (UWA). UWA was an all Apple campus with Macs and then Mac SEs in every faculty. The library had Macs you could rent by the hour.
I remember paying $400 Australian for an external floppy disk drive. There was no hard drive and definitely no web browser!
My second employer (also a High School) had IBM JXs running DOS.
And my first computer at IBM was not a PC at all. It was an IBM 3290 Gas Plasma terminal that gave you four mainframe logons at the same time. I still remember that console with great affection. I found an image in Flikr if you want to see what one looked like.
So... what was your first computer?
VMware vSphere 4.1 brings in a brilliant new function to offload storage related workload. This function is called VAAI (vStorage APIs for Array Integration) and requires that your SAN storage supports VAAI and that your ESX or ESXi server has a driver installed to utilize it.
IBM first supported VAAI with the IBM XIV using an IBM supplied VAAI driver. IBM then added support to the Storwize V7000 and SVC, so IBM has now released a new VAAI driver to support all three products at once. You can find the driver, installation guide and release notes at this URL.
I discovered some quirks in the process to update the IBM VAAI driver from version 220.127.116.11 to version 18.104.22.168 on VMware ESXi. The benefit in moving to version 22.214.171.124 is that the updated driver supports both the IBM XIV as well as the Storwize V7000 and IBM SVC.
I downloaded the new driver from here and which uses the following naming convention:
Version 126.96.36.199 is named IBM-ibm_vaaip_module-268846-offline_bundle-395553.zip
The last 6 digits in the file name is what differentiates them. However when I ran the --query command against an ESXi box, I got confused:
vihostupdate.pl --server 10.1.60.10 --username root --password passw0rd -query ---------Bulletin ID--------- -----Installed----- ----------------Summary----------------- ESXi410-201104402-BG 2011-07-01T12:36:32 Updates VMware Tools ESXi410-201104401-SG 2011-07-04T04:59:19 Updates Firmware ESXi410-Update01 2011-07-04T04:59:19 VMware ESXi 4.1 Complete Update 1 IBM-ibm_vaaip_module-268846 2011-07-14T11:59:15 vmware-esx-ibm-vaaip-module: ESX release
Both the uplevel and downlevel VAAI driver files start with:
vihostupdate.pl --server 10.1.60.11 --username root --password passw0rd --scan --bundle IBM-ibm_vaaip_module-268846-offline_bundle-613937.zip The bulletins which apply to but are not yet installed on this ESX host are listed. ---------Bulletin ID--------- ----------------Summary----------------- IBM-ibm_vaaip_module-268846 vmware-esx-ibm-vaaip-module: ESX release
To perform the upgrade I first used vMotion to move all guests off the server I was upgrading. I then placed the server in maintenance mode and installed the new driver:
vicfg-hostops.pl --server 10.1.60.11 --username root --password passw0rd --operation enter vihostupdate.pl --server 10.1.60.11 --username root --password passw0rd --install -bundle IBM-ibm_vaaip_module-268846-offline_bundle-613937.zip
I got the following messages:
Please wait patch installation is in progress ... The update completed successfully, but the system needs to be rebooted for the changes to be effective.
I then rebooted the server and then finally took it out of maintenance mode:
vicfg-hostops.pl --server 10.1.60.11 --username root --password passw0rd --operation reboot vicfg-hostops.pl --server 10.1.60.11 --username root --password passw0rd --operation exit
There are no commands needed to activate VAAI or claim VAAI capable devices in ESXi. You simply need to confirm that the both boxes shown in the example below have the number 1 in them (for hardware accelerated move and for fast init):
To test VAAI I normally do a storage migration (storage vMotion) moving a VMDK between datastores on the same storage device. What you should see is very little VMware to Storage I/O, as I depicted in this blog post and this blog post.
My colleague Alexandre Chabrol from Montpellier Benchmarking Center also helped me out with the ESXCLI commands to control VAAI. We can confirm the state of each of the three VAAI functions and switch them off and on.
esxcfg-advcfg.pl --server 10.1.60.11 --username root --password password -g /DataMover/HardwareAcceleratedMove Value of HardwareAcceleratedMove is 1 esxcfg-advcfg.pl --server 10.1.60.11 --username root --password password -g /DataMover/HardwareAcceleratedInit Value of HardwareAcceleratedInit is 1 esxcfg-advcfg.pl --server 10.1.60.11 --username root --password password -g /VMFS3/HardwareAcceleratedLocking Value of HardwareAcceleratedLocking is 1 esxcfg-advcfg.pl --server 10.1.60.11 --username root --password password -s 0 /DataMover/HardwareAcceleratedMove Value of HardwareAcceleratedMove is 0 esxcfg-advcfg.pl --server 10.1.60.11 --username root --password password -s 1 /DataMover/HardwareAcceleratedMove Value of HardwareAcceleratedMove is 1
Final thought: Most if not all of these commands can be done via the vSphere Client GUI, you do not need to use CLI. But I am surprised how many people like to use the CLI and want to see example syntax. Got a preference yourself? Love to hear about your experiences.
*** Update February 20, 2012 ***
The IBM Storage Device Driver for VMware VAAI was updated to version 188.8.131.52 in February 2012. This new version fixes a rare case where XIV, Storwize V7000, or SVC LUNs are not claimed by the IBM Storage device driver. If you are using version 184.108.40.206 without issue, there is no need to upgrade. I have updated this post to reflect the new version.