The developerWorks Connections platform will be sunset on December 31, 2019. On January 1, 2020, this blog will no longer be available. More details available on our FAQ.

Anthony's Blog: Using System Storage - An Aussie Storage Blog

Matching: vaai

Upgrading to ESXi 5.0? How to confirm VAAI status?

anthonyv Tags: storwize v7000 xiv esxi vaai svc 5.0 ‎ 5 Comments ‎ 23,828 Views

I am getting this question on a very regular basis:

"We have just upgraded to ESXi 5.0 but we cannot find the VAAI driver on the IBM Website"

The answer? There is no vendor supplied driver because no driver is needed. ESXi 5.0 uses a SCSI T10 compliant set of commands that all vendors need to support for VAAI to work.

But of course in the tradition of all answered questions, it leads to another question:

"Once I have upgraded to ESXi 5.0 how can I tell if VAAI is really working?"

The good news is that it is very easy to spot if ESXi 5.0 has detected a VAAI capable LUN. The moment a new LUN is detected by ESXi 5.0 it tries out an Atomic Test and Set command. If that works, you will see that Hardware Acceleration shows as Supported in vCenter. In the screen capture below I have three datastores, two from XIV and one from Storwize V7000, all presented to an ESXi 5.0 server. I dragged the Hardware Acceleration column over from the right hand side to help with the screen capture (in case your vCenter looks different), but you can see the Hardware Acceleration column shows each DataStore as Supported (and did so the moment the volume was detected).

Of course having seen the Hardware Acceleration Supported message only proves that Atomic Test and Set works. To confirm if XCopy (Hardware Accelerated Move) is working, on SVC or Storwize V7000 we can use the Performance monitoring panel. In the example below I first performed a storage vMotion, moving a virtual machine between two Datastores located on the same Storwize V7000 (running 6.3.0.0 firmware). I then performed a clone of the same virtual machine, where the source was on one datastore and the target was placed on another (but both located on the same Storwize V7000). What you can clearly see is that both operations (storage vMotion and cloning) generated no volume traffic, only MDisk traffic. This means that the ESXi server is doing none of the work and the storage is doing all of the work.

For more examples of how to test VAAI, check out section 9.5.5 Testing VAAI, in the new XIV IBM Redbook draft, XIV Storage System: Host Attachment and Interoperabilty which you can find here:
http://www.redbooks.ibm.com/Redbooks.nsf/RedpieceAbstracts/sg247904.html?Open

It documents the following tests:

Test one: Hardware accelerated Initialization or block zeroing
Test two: Hardware accelerated move or full copy

The tests can be performed regardless of whether you have an XIV (on code levels 10.2.4a and above) or a Storwize V7000/SVC (on code levels 6.2.0.0 and above).

If upgrading to ESXi 5.0 with IBM Storage, please also be aware of the following knowledge base articles regarding VAAI support with IBM Storage:

XCOPY and XIV:
http://www-01.ibm.com/support/docview.wss?uid=ssg1S1003946

VAAI unmap with XIV, SVC and Storwize V7000:
http://kb.vmware.com/kb/2007723
and
http://kb.vmware.com/kb/2007427

VAAI issues with ESXi 5.0

anthonyv Tags: xiv xcopy vaai ‎ 16,781 Views

For those of you planning to move to ESXi 5.0, IBM have found an annoying (but not show stopping) issue with the way XCOPY is implemented in the VAAI driver. With ESX/EXi 4.1, IBM supplied the VAAI driver, but with ESXi 5.0 this changed and VMware now manage this themselves. It has since emerged that the way VMware implemented XCOPY in this driver does not totally work with the way IBM implemented XCOPY in both the XIV and the Storwize V7000 and SVC.

This is the current situation with the first three VAAI primitives in ESXi 5.0:

Hardware accelerated locking: Also known as Atomic Test and Set (ATS), this function works fine when ESXi 5.0 detects a volume from an XIV, Storwize V7000 or SVC. In fact the moment ESXi 5.0 detects a LUN from any of these products it uses ATS to confirm that VAAI is possible. So this is goodness.

Hardware accelerated initialization: Also known as write same, this function offloads almost all effort on the part of ESXi to write zeros across disks. This function works fine when ESXi 5.0 works with XIV, Storwize V7000 or SVC. So this is also goodness.

Hardware Accelerated Move: Also known as XCOPY, full copy or clone blocks, this function works fine with XIV, Storwize V7000 and SVC if you clone a virtual machine and place the new copy into the same datastore as the source. This means creating multiple clones of a VMDK inside the one datastore will still be accelerated by VAAI. So far so good, but unfortunately on XIV, if you place the clone in a different datastore on the same XIV, it will not be hardware accelerated. This means the clone is still created, but in the old-fashioned way (reading from the source and writing to the target). As for storage vMotion, it also reverts to working in the old-fashioned way, reading from the source and writing to the target, rather than the hardware accelerated way.

So to be clear, this issue with XCOPY:

Does not affect ESX/ESXi 4.1 at all.
Occurs no matter what version of VAAI compliant XIV, Storwize V7000 or SVC code level your running, or what model of XIV (A14 or 114).
Does not affect Atomic Test and Set or Write Same.
Does not affect clone operations on an SVC or Storwize V7000.
Does not prevent you using cloning on XIV, it just means that this task will not be hardware accelerated if the target datastore is different from the source.
Does not prevent you using storage vMotion, it just means that this task will not be hardware accelerated.

How will this be fixed? Well right now it looks like it will be fixed in new firmware on the IBM hardware. Watch this space and I will update you as soon as I have more news to hand.

As for the fourth VAAI primitive, unmap, which is used for space reclamation on thin provisioning capable hardware, please also watch this space on when IBM hardware will support it... BUT in my opinion it does not matter right now, because this new unmap function in ESXi 5.0 can potentially cause issues. This is described here: http://kb.vmware.com/kb/2007427

So until VMware confirm a fix, I recommend that you run the following commands on all ESXi 5.0 boxes which connect to IBM Storage to disable unmap. I tested the following syntax to confirm it works:

First confirm the value of the unmap setting (1 means enabled, 0 means disabled):

~ # esxcli system settings advanced list -o /VMFS3/EnableBlockDelete | grep "Int Value" 
   Int Value: 1
   Default Int Value: 1

Then disable it:

~ # esxcli system settings advanced set --int-value 0 --option /VMFS3/EnableBlockDelete

Then confirm it is disabled:

 ~ # esxcli system settings advanced list -o /VMFS3/EnableBlockDelete | grep "Int Value" 
   Int Value: 0
   Default Int Value: 1

Updating the IBM VAAI driver in ESXi

anthonyv Tags: esxi ibm storwize xiv vaai v7000 svc ‎ 20,196 Views

VMware vSphere 4.1 brings in a brilliant new function to offload storage related workload. This function is called VAAI (vStorage APIs for Array Integration) and requires that your SAN storage supports VAAI and that your ESX or ESXi server has a driver installed to utilize it.

IBM first supported VAAI with the IBM XIV using an IBM supplied VAAI driver. IBM then added support to the Storwize V7000 and SVC, so IBM has now released a new VAAI driver to support all three products at once. You can find the driver, installation guide and release notes at this URL.

I discovered some quirks in the process to update the IBM VAAI driver from version 1.1.0.1 to version 1.2.0.1 on VMware ESXi. The benefit in moving to version 1.2.0.1 is that the updated driver supports both the IBM XIV as well as the Storwize V7000 and IBM SVC.

I downloaded the new driver from here and which uses the following naming convention:

Version 1.1.0.1 is named IBM-ibm_vaaip_module-268846-offline_bundle-395553.zip
Version 1.2.0.0 is named IBM-ibm_vaaip_module-268846-offline_bundle-406056.zip
Version 1.2.0.1 is named IBM-ibm_vaaip_module-268846-offline_bundle-613937.zip

The last 6 digits in the file name is what differentiates them. However when I ran the --query command against an ESXi box, I got confused:

vihostupdate.pl --server 10.1.60.10 --username root --password passw0rd -query
---------Bulletin ID--------- -----Installed----- ----------------Summary-----------------
ESXi410-201104402-BG          2011-07-01T12:36:32 Updates VMware Tools
ESXi410-201104401-SG          2011-07-04T04:59:19 Updates Firmware
ESXi410-Update01              2011-07-04T04:59:19 VMware ESXi 4.1 Complete Update 1
IBM-ibm_vaaip_module-268846   2011-07-14T11:59:15 vmware-esx-ibm-vaaip-module: ESX release

Both the uplevel and downlevel VAAI driver files start with:
IBM-ibm_vaaip_module-268846
So which one is installed? The 1.1.0.1 version or the 1.2.0.0 version? I ran the following command to confirm if the updated bulletin applies (the one ending in 613937 ). This confirmed my ESXi server was using version 1.1.0.1 and needed an upgrade to version 1.2.0.1.

vihostupdate.pl --server 10.1.60.11 --username root --password passw0rd --scan --bundle IBM-ibm_vaaip_module-268846-offline_bundle-613937.zip
The bulletins which apply to but are not yet installed on this ESX host are listed.

---------Bulletin ID---------   ----------------Summary-----------------
IBM-ibm_vaaip_module-268846     vmware-esx-ibm-vaaip-module: ESX release

To perform the upgrade I first used vMotion to move all guests off the server I was upgrading. I then placed the server in maintenance mode and installed the new driver:

vicfg-hostops.pl --server 10.1.60.11 --username root --password passw0rd --operation enter
vihostupdate.pl --server 10.1.60.11 --username root --password passw0rd --install -bundle IBM-ibm_vaaip_module-268846-offline_bundle-613937.zip

I got the following messages:

Please wait patch installation is in progress ...
The update completed successfully, but the system needs to be rebooted for the changes to be effective.

I then rebooted the server and then finally took it out of maintenance mode:

vicfg-hostops.pl --server 10.1.60.11 --username root --password passw0rd --operation reboot
vicfg-hostops.pl --server 10.1.60.11 --username root --password passw0rd --operation exit

There are no commands needed to activate VAAI or claim VAAI capable devices in ESXi. You simply need to confirm that the both boxes shown in the example below have the number 1 in them (for hardware accelerated move and for fast init):

To test VAAI I normally do a storage migration (storage vMotion) moving a VMDK between datastores on the same storage device. What you should see is very little VMware to Storage I/O, as I depicted in this blog post and this blog post.

My colleague Alexandre Chabrol from Montpellier Benchmarking Center also helped me out with the ESXCLI commands to control VAAI. We can confirm the state of each of the three VAAI functions and switch them off and on.
We use the -g switch to display them, the -s 0 switch to turn them off and the -s 1 switch to turn them on.
In this example I first confirm that VAAI is active for hardware accelerated moves, hardware accelerated inititialization (write zeros) and then hardware assisted locking. I then disable and enable hardware accelerated moves.

esxcfg-advcfg.pl --server 10.1.60.11 --username root --password password -g /DataMover/HardwareAcceleratedMove
Value of HardwareAcceleratedMove is 1

esxcfg-advcfg.pl --server 10.1.60.11 --username root --password password -g /DataMover/HardwareAcceleratedInit
Value of HardwareAcceleratedInit is 1

esxcfg-advcfg.pl --server 10.1.60.11 --username root --password password -g /VMFS3/HardwareAcceleratedLocking
Value of HardwareAcceleratedLocking is 1

esxcfg-advcfg.pl --server 10.1.60.11 --username root --password password -s 0 /DataMover/HardwareAcceleratedMove
Value of HardwareAcceleratedMove is 0

esxcfg-advcfg.pl --server 10.1.60.11 --username root --password password -s 1 /DataMover/HardwareAcceleratedMove
Value of HardwareAcceleratedMove is 1

Final thought: Most if not all of these commands can be done via the vSphere Client GUI, you do not need to use CLI. But I am surprised how many people like to use the CLI and want to see example syntax. Got a preference yourself? Love to hear about your experiences.

*** Update February 20, 2012 ***

The IBM Storage Device Driver for VMware VAAI was updated to version 1.2.0.1 in February 2012. This new version fixes a rare case where XIV, Storwize V7000, or SVC LUNs are not claimed by the IBM Storage device driver. If you are using version 1.2.0.0 without issue, there is no need to upgrade. I have updated this post to reflect the new version.

vriendelijke groeten with VAAI on SVC and Storwize V7000

anthonyv Tags: storwize v7000 vaai svc ‎ 9,779 Views

I recently got a great email from an IBMer in the Netherlands by the name of Jack Tedjai. He sent me two screen shots, taken with the new performance monitor panel (that comes with the SVC and Storwize V7000 6.2 code). He wrote:

I am working on a project to migrate VMware/SRM/DS5100 to SVC Stretch Cluster and one of the goals is to prevent using ISL (4Gbps) and VMware Hypervisor/HBA load during the migration. For the migration we are using VMware Storage vMotion. To minimize the impact of the migration on production, we tested VAAI for Storage vMotion and template deployment and it worked perfectly.

So whats this all about? Well one of the improvements provided with VAAI support is the ability to dramatically offload the I/O processing generated by performing a storage vMotion. Normally a storage vMotion requires an ESX server to issue lots of reads from the source datastore and lots of writes to the target datastore. So there is a lot of I/O flowing from ESX to the SVC, and then from the SVC to its backend disk. What you get is something that looks like the image below. In the top right graph we have traffic from SVC to ESX (host to volume traffic). In the bottom right graph we have traffic from the SVC to its backend disk controllers (DS5100 in this case). This is SVC to MDisk traffic.

When we add VAAI support to the SVC, we suddenly change the picture. Suddenly VMWare does not need to do any of the heavy lifting. There is almost no I/O between VMWare and the SVC (no host to SVC volume traffic) related to the vMotion. The SVC is still doing the work, but it is happening in the background without burning VMWare CPU cycles or HBA ports (in that there is still SVC to MDisk traffic).

This difference translates to: Faster vMotion times, far less SAN I/O and far less VMware CPU being used on this process.

So do VMware support this? They sure do! Check this link here. It currently shows something like this (taken on June 23, 2011):

So what are your next steps?

Upgrade your Storwize V7000 or SVC to version 6.2 code. Download details arehere.
Download and install the VAAI driver onto your ESX servers. You can get it from here. If your already using the XIV VAAI driver you need to upgrade from version 1.1.0.1 to version 1.2. There is an installation guide at the same link.

And the blog title? It means friendly greetings in Dutch. So to Jack (and to all of you), vriendelijke groeten and please keep sending me those screen captures.

VAAI driver download link for XIV

anthonyv Tags: xiv vaai driver ‎ 2 Comments ‎ 9,659 Views

Hi Team!

Back from a short break (for Easter and the School Holidays) to three great pieces of news:

A new series of Doctor Who is screening.
Will and Kates wedding went off without a hitch (its not often I get to yell at the dog to stop barking at possums because there is a Royal Wedding on).
The VAAI driver for XIV has an official download link.

Ok... maybe the Royal Wedding has no place in my blog, but the VAAI link is very appreciated.

Two ways to get to the driver:

Get it directly from here
Go to fix Central and select it from the download list: http://www-933.ibm.com/support/fixcentral/

Remember, your XIV needs to be on 10.2.4a firmware, so you need to be talking to your IBM Service Representative to schedule a concurrent firmware update before you turn the VAAI functions on.

Now if your going, um... what is VAAI and how does it help? Check this blog post out:

https://aussiestorageblog.wordpress.com/2011/03/09/my-first-experience-with-xiv-and-vaai/

If your asking, hey what else will 10.2.4a code bring me?

How about better write performance?
How about QoS?
10.2.4a code also brings the ability to do 'truck' initialization of an async pair (which lets you pre-load an async secondary for faster initial mirroring, or to convert from sync to async without re-mirroring all your data).
It also lets you format a snapshot, which means you can keep a snapshot in place and mapped to a host, but it will not consume any space.

VAAI and XIV - Its here!

anthonyv Tags: 4.1 esx vaai xiv 10.2.4a u1 ‎ 3 Comments ‎ 12,804 Views

I have some great news regarding VAAI support for XIV.

Let me detail the current situation:

VMware has approved the IBM driver for VAAI and we can now release it to the public. The IBM_VAAIP_MODULE plugin will be available shortly from the ibm.com website. When the release URL is available I will update this post. In the meantime you can get the driver from your XIV TA (Technical Assistant) or IBM Account Team. If they have somehow missed the news, get them to talk to their XIV Product Manager (or they can always talk to me!).
The VMware Hardware Compatibility guide found here shows that VMware support the three VAAI primitives with XIV, if you are using ESX 4.1 or ESX 4.1 U1 and your XIV is on firmware release 10.2.4 or higher.
XIV firmware release 10.2.4a is available for install. Installation of this firmware is non-disruptive (concurrent) and will be performed by IBM.
The VAAI driver and installation of the 10.2.4a code are all supplied free of charge.

So what should your plan be?

Ensure VAAI is disabled on your ESX hosts.
Talk to your local XIV TA or IBM Service Representative (SSR) and arrange to have 10.2.4a firmware installed.
When 10.2.4a code is installed, you can then begin installing the VAAI driver on each of your ESX 4.1 servers. You will need to reboot each server to install the driver.

Trying out XIV and VAAI

anthonyv Tags: xiv vaai ‎ 2 Comments ‎ 12,543 Views

As I blogged previously, VAAI support for XIV has two dependencies:

10.2.4a code
Vmware Certified driver

Both of these things are very close to release....
In the meantime I have had the chance to demonstrate the uncertified VAAI driver with XIV 10.2.4 code, just to see what affect it has.

And what is the affect?

Its mindblowing!

VAAI dramatically reduces the amount of work that the vSphere 4.1 server needs to do to get things done.

The XIV implementation of VAAI provides the three fundamentals of VAAI:

Full clone, copying data from one logical unit (LUN based) to another without writing to the ESX server.
Block Zeroing, assigning zeros to large storage areas without actually sending the zeros to the storage system.
Hardware Assisted locking, locking a particular range of blocks in a shared logical unit (providing exclusive access to these blocks), instead of using SCSI reservation that locks the entire logical unit.

To test VAAI with XIV, I did two things: a VMDK migration (a Storage Vmotion) and VMDK cloning. I used the vSphere client to time how long the operation took and XIV Top to see how much IO was being generated by the vSphere server. Now please understand, these numbers and timings are based on a lab environment. The speed and peaks will vary from client to client and install to install.

Firstly the migration: I performed a migration of a VMDK from one data store to another. The migration without VAAI took 42 seconds as can be seen from the screen capture below:

The migration generated a peak of 135 MBps of traffic being written to the target volume as can be seen from XIV Top:

I then turned on VAAI and did the same migration. I won't document the process to install the VAAI driver, as it will be different when the certified version is released. However after the driver is installed, I could turn VAAI on and off by toggling these settings from 0 1 and back again:

I we did another VMDK migration with VAAI enabled. This time the migration took 19 seconds (as opposed to 42 seconds), so an immediate improvement occurred.

When I checked XIV Top, there was no IO at all! In other words the vMotion was done with no apparent load on the vSphere HBAs or the SAN. I feel silly showing this screen capture, but this is what I saw.... nothing.

I then did a VMDK clone. The Data store was on XIV, VAAI was not enabled. There was no other IO running on the ESX server. The clone took 40 seconds (as reported by vCenter):

The clone generated a peak of 230 MBps for around 50 seconds (as reported by XIV Top)

We then again activated VAAI and repeated the clone. Now the clone took 15 seconds (as reported by vCenter), so thats 25 seconds faster (more than 50%).

The clone generated a peak of 2 MBps for around 20 seconds (as reported by XIV Top). Almost no fibre channel IO was thus generated by the clone.

As I have blogged before, I will be repeating this whole exercise once I have real live customers running this configuration, so expect further updates.

VMware vSphere 4.1 VAAI and XIV

anthonyv Tags: vmware vaai xiv ‎ 14,072 Views

I recently listed to a great Podcast on VAAI from Greg Knierieman over at the Storage Monkeys website. You can find the podcast here (the whole site is well worth a visit).
He was talking with Marc Farley (his co-host, from 3PAR), Chad Sakac (from EMC) and Chris Evans (the 'Storage Architect'). The topic was VMWares newly announced VAAI.

To quote from the podcast, VAAI is a set of APIs focused on the VMWare kernel to off-load various functions onto the storage.
To get the newly announced VAAI functions you need to upgrade to the newly announced VSphere 4.1 and you will need to upgrade your storage hardware firmware to a version that supports it (when such a version comes out).
.
Some of the major new functions are:
.
Hardware accelerated locking (to avoid the need for ESX to use SCSI reserves when doing meta-data updates)
Hardware accelerated full copy (to help VMWare clone data without having to do lots of read and writes)
Hardware accelerated zero (to avoid the need to send vast numbers of 'empty' SCSI write I/Os to zero out blocks)
.
Given who was on the call, the conversation focused mainly on what EMC, 3PAR and to some extent what NetApp are doing in regards to this development.
Apart from some (good natured?) digging at HP, no other Vendor was really mentioned.
.
One thing that was mentioned was that some storage hardware architectures will lend themselves far better to VAAI than others.
In particular Chad mentioned that he would expect fullcopy and hardware accelerated zero would work better on V-Max than CLARiiON, due to hardware architecture differences that also benefit 3PAR.
I found that a really interesting observation.
.
What wasn't mentioned on the podcast was XIV.
So to be clear, the architecture of XIV lends itself very very well to the changes required to support VAAI.
.
To give an example of how we have done this with other vendors, XIV firmware 10.2.0a brought in support for Symantec Storage Foundation Thin Reclamation.
XIV support for Symantec's Storage Foundation and Thin Reclamation API means that when data is deleted by a user who uses the thin provisioning aware Veritas File System (VxFS), XIV will immediately free unutilised blocks and reclaim such blocks, rather than leaving them with 'garbage' data that wastes space.
.
So have no doubts, XIVs architecture is very 'friendly' to the sorts of things VMware are trying to achieve with VAAI. To underscore this, Chad also said that a VMware goal was that VMware admins should need "no requisite knowledge of the underlying infrastructure for any task".   The goal is to use policies instead. Given this goal, XIV is also a perfect match.   With XIV there is no need to think about raid types, raid sizes, disk types, disk sizes, LUN allegiances and trespassing, controller workload balancing or hot spot detection prevention or correction.   All of these concerns simply don't exist.
.
The good news is that XIV is working with the VMware Reference Architecture Lab and the statement of direction is that we will announce VAAI support for XIV later this year.   XIV continues to be an excellent choice for VMware environments and when VAAI support is added to XIV, this will only improve.
.
Finally, Chad made a great quote on the podcast. He said: "Never trust any vendor when they talk about what other vendors are doing"
I think this is a really great statement and one that everyone should take to heart.

Feed for Blog Entries

Blogs

▼ Tags

▼ Similar Blogs

Inside System...

Application P...

Asset Managem...

Notes from Ra...

Application I...

▼ Similar Ideation Blogs

IBM PureData-...

IBM Forms Exp...

IBM Tunisia H...

Open Ideas

IBM Algo Fina...

▼ Archive

▼ Blog Authors