Topic
  • 14 replies
  • Latest Post - ‏2012-12-05T16:10:16Z by dsmithpdx
SystemAdmin
SystemAdmin
4779 Posts

Pinned topic vSphere 5.1a - DS 3500 7.83 - cloning and storage vmotion broken?

‏2012-11-01T13:04:54Z |
Hi there,

this could be very specific but I need some help. Following situation: DS3500 (with 3 enclosures, 8GB FC, Firmware 7.83.25.00), Emulex 8GB HBAs in some Dell Blade Systems running ESXi 5.1a with vsphere 5.1a. Standard config (no ALUA) with Multipath (Brocade Switches).

Following is happening (I could belive it started with the DS Firmware or with the Upgrade vom ESXi 5.0 to 5.1):

If we clone or storage vmotion a VM from one LUN (formatted with VMFS 5) to another LUN (also VMFS 5) on the same DS 3500 it gets corrupt an the OS is broken with filesystem errors (some OS even do not start).

After some experiments I came to following status. If the clone or vmotion process is from one LUN that is on SP A of the DS3500 to SP B of the DS3500 everything is fine. But if I clone or vmotion onto a a LUN residing on the same SP (either A or B) it gets broken.

Please, any suggestions where to get this fixed? I triple checked zoning, firmwares, FC errors, cabels, read whatever was to read and got a bit frustrated :).

THX
Oli
Updated on 2012-12-05T16:10:16Z at 2012-12-05T16:10:16Z by dsmithpdx
  • Novikov_Alexander
    Novikov_Alexander
    1404 Posts

    Re: vSphere 5.1a - DS 3500 7.83 - cloning and storage vmotion broken?

    ‏2012-11-01T18:27:03Z  
    Dear Oliver,

    vSphere 5.1a is not supported for DS3500 yet. Use any supported version from compatibility list:

    IBM System Storage Interoperation Center (SSIC)
    http://www-03.ibm.com/systems/support/storage/ssic/interoperability.wss

    Regards,
    Alexander Novikov
    Russia, Moscow
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: vSphere 5.1a - DS 3500 7.83 - cloning and storage vmotion broken?

    ‏2012-11-02T13:42:49Z  
    Once uppon a time i had a horrible week, when i slept all the week at the job. Client requested to install an SQL Cluster on Windows 2008R2 on x3850X5, when SP1 wasn't yet released. Finally we found problem and patch and this patch, was included in SP1.
    Then i swear on the Bible do not install anymore anything until fist ServicePack or Update was released. But did it again :-( I installed Vmware 5.1 on the DS3500 with 7.83.22 FW.
    Oliver, thx! I verified and i have the same problem. BTW there is no problem with thin disks. Problem appears when you work with thick disk and you may guess, problem is in DS3500 hardware acceleration for Vmware. You may disable it
    http://pubs.vmware.com/vsphere-51/topic/com.vmware.vsphere.storage.doc/GUID-88969C21-CB0D-499F-8902-8269E31057A3.html
    and problem gone. It's enough to disable only DataMover.HardwareAcceleratedMove and DataMover.HardwareAcceleratedInit.
    I think downgrade to 5.0u1 won't help, but i'll do it to have possibility to request IBM support.
    I only begin this installation so i'll lose a day to reconfigure hosts and vCenter.
    Strange things, but judging documentation, storage should assist on clone or move operation between different arrays
    http://pubs.vmware.com/vsphere-51/topic/com.vmware.vsphere.storage.doc/GUID-874E15C3-9C19-4EAE-A156-6A631F852C21.html
    "Hardware cloning between arrays, even within the same VMFS datastore, does not work."
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: vSphere 5.1a - DS 3500 7.83 - cloning and storage vmotion broken?

    ‏2012-11-02T19:05:59Z  
    Once uppon a time i had a horrible week, when i slept all the week at the job. Client requested to install an SQL Cluster on Windows 2008R2 on x3850X5, when SP1 wasn't yet released. Finally we found problem and patch and this patch, was included in SP1.
    Then i swear on the Bible do not install anymore anything until fist ServicePack or Update was released. But did it again :-( I installed Vmware 5.1 on the DS3500 with 7.83.22 FW.
    Oliver, thx! I verified and i have the same problem. BTW there is no problem with thin disks. Problem appears when you work with thick disk and you may guess, problem is in DS3500 hardware acceleration for Vmware. You may disable it
    http://pubs.vmware.com/vsphere-51/topic/com.vmware.vsphere.storage.doc/GUID-88969C21-CB0D-499F-8902-8269E31057A3.html
    and problem gone. It's enough to disable only DataMover.HardwareAcceleratedMove and DataMover.HardwareAcceleratedInit.
    I think downgrade to 5.0u1 won't help, but i'll do it to have possibility to request IBM support.
    I only begin this installation so i'll lose a day to reconfigure hosts and vCenter.
    Strange things, but judging documentation, storage should assist on clone or move operation between different arrays
    http://pubs.vmware.com/vsphere-51/topic/com.vmware.vsphere.storage.doc/GUID-874E15C3-9C19-4EAE-A156-6A631F852C21.html
    "Hardware cloning between arrays, even within the same VMFS datastore, does not work."
    > Strange things, but judging documentation, storage should assist on clone or move operation between different arrays
    sorry, should NOT assist.
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: vSphere 5.1a - DS 3500 7.83 - cloning and storage vmotion broken?

    ‏2012-11-06T11:52:41Z  
    > Strange things, but judging documentation, storage should assist on clone or move operation between different arrays
    sorry, should NOT assist.
    As i a supposed, downgrade to 5.0u1 didn't help.
    Strange thing, but issue didn't manifest itself from the beginning. Yesterday, after i have installed 5.0u1, I cloned several VM and no problem. Today i tried to move them and i've got crashed VMs. Now just clone VM to another LUN (which yesterday worked fine) does the same trouble.
    Disabling hardware acceleration helps.
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: vSphere 5.1a - DS 3500 7.83 - cloning and storage vmotion broken?

    ‏2012-11-07T07:41:10Z  
    First of all Pavel thank you very much for your help!!!
    This is what I call "constructive help" not just the usual "it is not supported, so we don't care".

    I'm going to report the problem to vmware maybe there is a slight chance to get somone "more important" to adress this to IBM (to get it fixed).

    No offense, but everytime I got to serios trouble with hardware nobody seems so care (this is not just a IBM problem)
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: vSphere 5.1a - DS 3500 7.83 - cloning and storage vmotion broken?

    ‏2012-11-07T08:35:11Z  
    First of all Pavel thank you very much for your help!!!
    This is what I call "constructive help" not just the usual "it is not supported, so we don't care".

    I'm going to report the problem to vmware maybe there is a slight chance to get somone "more important" to adress this to IBM (to get it fixed).

    No offense, but everytime I got to serios trouble with hardware nobody seems so care (this is not just a IBM problem)
    I can't open request on Vmware because I used our vSphere EnterprisePlus NFR keys at the testing stage. Customer purchased vSphere Standard and it looks like storage hardware assistance will not work with their keys
    www.vmware.com/products/datacenter-virtualization/vsphere/storage-api.html#integration
    I'll try to connect IBM Service, but i am afraid until i'll get it i'll have to install Vmware customer keys and possible repartition storage.

    P.S. Unfortunately i have to state that support at all became very poor. And this "disease" touched mostly all vendors. At least i personally had problems with different support services and many my frends complained too. cost optimization :-(
  • dsmithpdx
    dsmithpdx
    6 Posts

    Re: vSphere 5.1a - DS 3500 7.83 - cloning and storage vmotion broken?

    ‏2012-11-08T18:36:24Z  
    Is there any word from IBM on when ESXi 5.1 will be supported with a DS3500? We really want to do the upgrade, but obviously won't until IBM certifies the DS with it.
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: vSphere 5.1a - DS 3500 7.83 - cloning and storage vmotion broken?

    ‏2012-11-09T16:13:34Z  
    Well, it looks like IBM stopped the distribution of the 7.83 firmware. No comment on the site no information no download ...

    The only version available is the 7.77.34.00 firmware.

    "Hello IBM": What is goning on?
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: vSphere 5.1a - DS 3500 7.83 - cloning and storage vmotion broken?

    ‏2012-11-11T11:31:44Z  
    Well, it looks like IBM stopped the distribution of the 7.83 firmware. No comment on the site no information no download ...

    The only version available is the 7.77.34.00 firmware.

    "Hello IBM": What is goning on?
    https://www-947.ibm.com/support/entry/myportal/docdisplay?lndocid=MIGR-5091874&brandind=5000028
    Data corruption issue using XCOPY in VAAI (VMware ESXI 5.xx) - IBM System Storage DCS3700, DS3512, DS3524
  • dsmithpdx
    dsmithpdx
    6 Posts

    Re: vSphere 5.1a - DS 3500 7.83 - cloning and storage vmotion broken?

    ‏2012-11-12T18:43:34Z  
    https://www-947.ibm.com/support/entry/myportal/docdisplay?lndocid=MIGR-5091874&brandind=5000028
    Data corruption issue using XCOPY in VAAI (VMware ESXI 5.xx) - IBM System Storage DCS3700, DS3512, DS3524
    Well that's a bit disconcerting...we are running 7.83 on our DS3512. Could one now logically conclude that IBM won't support ESXi 5.1 until the next version of the DS firmware is released? It would be nice to hear from IBM on this one! We'd really like to get to 5.1.
  • nabilzahid
    nabilzahid
    3 Posts

    Re: vSphere 5.1a - DS 3500 7.83 - cloning and storage vmotion broken?

    ‏2012-11-14T17:34:20Z  
    Dear Oliver,

    vSphere 5.1a is not supported for DS3500 yet. Use any supported version from compatibility list:

    IBM System Storage Interoperation Center (SSIC)
    http://www-03.ibm.com/systems/support/storage/ssic/interoperability.wss

    Regards,
    Alexander Novikov
    Russia, Moscow
    hi ,
    i am in this situation using vsphere 5.1a , is there any roadmap to update the DS3500 firmware to support vsphere 5.1 platforms ?
    do you have any release date ? we are in a production enviroment and the amount of time we need to get back into vsphere 5.0 is too contrainfull

    thanks in advance .
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: vSphere 5.1a - DS 3500 7.83 - cloning and storage vmotion broken?

    ‏2012-11-15T08:51:27Z  
    @nabilzahid
    as long as you use the 7.77.34.00 firmware you should be ok with 5.1a (even if it's not supported) the problem started with the 7.83 firmware (and appears since vSphere 5.0)
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: vSphere 5.1a - DS 3500 7.83 - cloning and storage vmotion broken?

    ‏2012-11-20T08:16:33Z  
    @nabilzahid
    as long as you use the 7.77.34.00 firmware you should be ok with 5.1a (even if it's not supported) the problem started with the 7.83 firmware (and appears since vSphere 5.0)
    New firmware version 7.83.27.00for DS3500 series appeared.
    It solves problem with data corruption issue using XCOPY in VAAI.
  • dsmithpdx
    dsmithpdx
    6 Posts

    Re: vSphere 5.1a - DS 3500 7.83 - cloning and storage vmotion broken?

    ‏2012-12-05T16:10:16Z  
    New firmware version 7.83.27.00for DS3500 series appeared.
    It solves problem with data corruption issue using XCOPY in VAAI.
    According to the IBM SSIC, ESXi 5.1 is still not officially supported on the DS3500. Does anyone know if firmware 7.83.27.00 will be certified with 5.1 soon? Or will we have to wait for yet another version?