Topic
  • 24 replies
  • Latest Post - ‏2014-10-31T09:43:41Z by NAYABSK
SystemAdmin
SystemAdmin
4779 Posts

Pinned topic DS3512 prefered path changing intermittently issue .

‏2011-04-12T17:07:24Z |
Dear All,

I am facing problem with my storage DS3512 i have configure storage with IBM blade Center H (88524TG) and host server is IBM blade server HX5 (787264G) OS windows 2008 DATA Center x64 everything seems to be working fine but my logical drive changed its preferred path intermittently i have try to update HBA device driver and MPIO driver but still its changing path and also i saw some error on event log "Controller cache not enabled or was internally disabled" or "Logical Drive not on preferred path due to ADT/RDAC failover"

can anyone help me to find out root causes of this issue ?
logs atteched herewith for your reference.

Regards,
Noman Ajaz Ahmed Siddiqui

Attachments

Updated on 2012-11-07T20:30:11Z at 2012-11-07T20:30:11Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2011-04-12T19:00:38Z  
    Hi Noman,
    Please provide the CASD and the DSA (*.xml) for analysis.
    Regards.
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2011-04-12T19:27:19Z  
    Hi Noman,
    Please provide the CASD and the DSA (*.xml) for analysis.
    Regards.
    Already attached in my post.

    Regards,
    Noman Ajaz
  • andrew.scott
    andrew.scott
    1 Post

    Re: DS3512 prefered path changing intermittently issue .

    ‏2011-04-13T06:58:44Z  
    Already attached in my post.

    Regards,
    Noman Ajaz
    How did you get on with this issue?
    Also have a DS3512 with two esx4.1 hosts (x3650 M3) stuck in a continual failover loop.
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2011-04-13T11:29:57Z  
    How did you get on with this issue?
    Also have a DS3512 with two esx4.1 hosts (x3650 M3) stuck in a continual failover loop.
    Its new configuration.
    its not esx4.1 hosts. its windows 2008 Data center x64 with hyper-visor on blade server HX5 and its scalable partition.

    And i have attached my hardware details.
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2011-04-13T11:49:17Z  
    Its new configuration.
    its not esx4.1 hosts. its windows 2008 Data center x64 with hyper-visor on blade server HX5 and its scalable partition.

    And i have attached my hardware details.
    Find in attachment.
  • pavel.alexei
    pavel.alexei
    335 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2011-04-14T08:30:08Z  
    Its new configuration.
    its not esx4.1 hosts. its windows 2008 Data center x64 with hyper-visor on blade server HX5 and its scalable partition.

    And i have attached my hardware details.
    As far i understood you have 4 HX5 blades.
    Judging Storage profile 2 blades works independently and didn't access storage (OTHMKT-BLADE1 and OTHMKT-BLADE2) and from rest 2 you created a two node complex. I am right?
    Did you zone Fiber Switches?
    And what is the purpose of OSLUN1,OSLUN2,OSLUN3?
  • dbilyk
    dbilyk
    1 Post

    Re: DS3512 prefered path changing intermittently issue .

    ‏2011-05-06T20:27:20Z  
    Has any progress been made with this issue? I have been using our DS3512 for the past four months without issue and now suddenly, the preferred controller keeps switching from one to the other. Things have just worked up until now, so much so that I did not suspect anything wrong with the DS3512 until our ESXi VMs suddenly started to bog on heavy I/O access. When I checked the DS3512, that is when I saw the warning that the current controller was not the preferred one. Then, 15 minutes later, things swapped; it now preferred A and was connected to B whereas previously it preferred B and was connected to A. I do not know what is causing this switch back and forth but I suspect it is the cause of our VMs bogging down so heavily.

    Despite operating the DS3512 for four months, I am still a relative n00b. Until now, things just worked. I should have suspected something going together so easily was just hiding an issue that would someday materialize. At least we are still in pre-production.

    If anyone has any suggestions, I would love to hear them.
  • skyron
    skyron
    84 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2011-05-07T04:52:12Z  
    • dbilyk
    • ‏2011-05-06T20:27:20Z
    Has any progress been made with this issue? I have been using our DS3512 for the past four months without issue and now suddenly, the preferred controller keeps switching from one to the other. Things have just worked up until now, so much so that I did not suspect anything wrong with the DS3512 until our ESXi VMs suddenly started to bog on heavy I/O access. When I checked the DS3512, that is when I saw the warning that the current controller was not the preferred one. Then, 15 minutes later, things swapped; it now preferred A and was connected to B whereas previously it preferred B and was connected to A. I do not know what is causing this switch back and forth but I suspect it is the cause of our VMs bogging down so heavily.

    Despite operating the DS3512 for four months, I am still a relative n00b. Until now, things just worked. I should have suspected something going together so easily was just hiding an issue that would someday materialize. At least we are still in pre-production.

    If anyone has any suggestions, I would love to hear them.
    This issue mostly comes up due to either incorrect zoning, or having old fiber card drivers (qlogic/emulex) on the server. If you update them to the latest, especially for ESX/ESXi, it resolves the issue in most cases.
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2011-05-09T17:35:31Z  
    issue resolved.
    thanks for your support
  • itguy4hospice
    itguy4hospice
    1 Post

    Re: DS3512 prefered path changing intermittently issue .

    ‏2011-06-29T22:11:42Z  
    issue resolved.
    thanks for your support
    What part of the previous post (if indeed that's the answer) corrected your issue. I am seeing same on our DS3512
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2011-11-24T17:06:40Z  
    Hi Sorry for digging up an old post, but I'm having the same issue. DS3500 with Windows 2008 Cluster. Logical Drive not on preferred path due to ADT/RDAC failover errors and the drives would go offline.

    What was the fix for it?? thanks.
  • kashif.adeel
    kashif.adeel
    3 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2012-10-18T16:50:25Z  
    issue resolved.
    thanks for your support
    Dear Nouman
    what was the root cause of this issue and how'ld you solved this issue please share.
  • Berniebgbb
    Berniebgbb
    83 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2012-10-19T03:27:12Z  
    Dear Nouman
    what was the root cause of this issue and how'ld you solved this issue please share.
    Hi all,

    To possibly help a few people I though I would not some core configuration steps, as most people have said the majority of the time "path thrashing" is due to incorrect cabling, zoning or host type / multi-pathing setup issues. Each setup has different specific requirements so my comments below are broad and just a quick summary....

    1. Storage Host Port configuration
    Unless you are using SAS connections, you must have a switch fabric (redundant preferred) between the hosts and the storage.

    Cabling for Fibre channel and iSCSI from storage(generally):
    Controller A Port 1 -- Switch 1
    Controller A Port 2 -- Switch 2
    Controller B Port 1 -- Switch 1
    Controller B Port 2 -- Switch 2

    Cabling from Host:
    Host HBA 1 -- Switch 1
    Host HBA 2 -- Switch 2

    Note: To rule out issues; if you are configuring multipathing to dual controller, ALL hosts should be configured for multipathing with visibility to both controllers. Especially if they are accessing the same LUN's. Generally each HBA in the host should have visibility to BOTH controllers in the storage.

    Zoning (Fibre)
    Create Alia's for your Storage Ports and your Host ports.
    For DS3500 setup "single target, single initiator" zoning. So, create a zone for each host port to each storage port, then put all the zones in a config and enable it.(NOTE: This is different for other arrays E.G: V7000)
    The main thing is that each host can see 4 paths. once you have setup the zoning and the storage is running

    Register hosts in storage (depending on connection type and card type)
    Register 2 x HBA's for fibre to the host
    Register 2 x Hardware iSCSI HBA's to the host
    Register 1 x Sofware iSCSI Initiator to the host (you only have 1 x software initiator however pathing can still be through multiple NIC's for redundancy).

    NOTE: If you IQN or WWN for your cards are not showing up under "Auto Discovered" then there is something wrong, don't bother trying to enter them manually.
    Re-Scan from your hosts to help the HBA's register to the storage....see if they are now register under "Auto Discovered", if not you still have problems.

    Host type:
    Make sure you use the correct host type when registering the host.
    For VMware - select VMWARE (or for old firmware LNXCLVMWARE)
    For Microsoft Servers running IBM DSM Software with Microsoft MPIO select Microsoft XXXXXX
    For Microsoft Servers running Microsoft MPIO with Veritas Dynamic Multi-pathing select Microsoft XXXXX (Supporting DMP)
    When Clustering select the cluster host type option.
    Once you have the hosts register you can MAP a LUN, once the LUN is mapped you can check via the HBA BIOS to see if both HBA's can see the target device and LUN.
    You want to make sure all HBA's can see the Array.
    For Software iSCSI, you want to make sure that you can ping ALL host ports of the Array.

    Host Multipathing
    Once you have your mappings right and your LUN/s mapped you need to setup the multipathing on the Host
    For VMware, multipathing is enabled by default, however you should check your pathing policy and the QTy of paths to be sure the zoning is correct

    For Microsoft, firstly make sure you have the latest HBA firmware and StorPort drives, then you will need to install Microsoft MPIO framework (windows 2008) and then the IBM DSM modules. The IBM DSM modules are now packaged separately to Storage Manager software so make sure you install it. Just installing IBM storage manager will NOT install the multipathing software.

    Once the multipathing is install and the LUN is visible you can run command lines with the IBM DSM to confirm pathing visibility utilising the "SMcli" commands...Google SMcli...

    You should see multiple paths for all luns.

    Another item to check is pathing issues due to fabric issues, so check your port error levels on your FC switches....

    Bernie
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2012-10-20T02:25:52Z  
    im also experiencing this behavior after firmware upgrade 7.83. message like "Controller cache not enabled or was internally disabled" or "Logical Drive not on preferred path due to ADT/RDAC failover".
  • Berniebgbb
    Berniebgbb
    83 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2012-10-21T10:34:58Z  
    im also experiencing this behavior after firmware upgrade 7.83. message like "Controller cache not enabled or was internally disabled" or "Logical Drive not on preferred path due to ADT/RDAC failover".
    Rebelinux, can you provide configuration information?

    or even better Support logs for your storage and switches?

    Bernie.
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2012-10-22T13:18:21Z  
    Rebelinux, can you provide configuration information?

    or even better Support logs for your storage and switches?

    Bernie.
    We have one ibm bladecenter h with 8 blade server (4 hs22v & 4 hx5). the chassis have 2 brocade fibre switch connected to one ibm ds3512 with 2 controller and 2 exp3512 enclosures. We use this hardware to support a vmware infrastructure cluster version 5.0.U1

    Recently we upgrade the ds3512 FW from version 7.77.34 to 7.83.22 and the brocade fabric os from 6.3.1 to 6.4.2a. Since then we are experiencing this kind of behavior also i suspect that this kind of errors are produced when there are a lot of IOps on the luns

    here is one of the support files.

    thanks.
  • Novikov_Alexander
    Novikov_Alexander
    1404 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2012-10-22T17:56:55Z  
    We have one ibm bladecenter h with 8 blade server (4 hs22v & 4 hx5). the chassis have 2 brocade fibre switch connected to one ibm ds3512 with 2 controller and 2 exp3512 enclosures. We use this hardware to support a vmware infrastructure cluster version 5.0.U1

    Recently we upgrade the ds3512 FW from version 7.77.34 to 7.83.22 and the brocade fabric os from 6.3.1 to 6.4.2a. Since then we are experiencing this kind of behavior also i suspect that this kind of errors are produced when there are a lot of IOps on the luns

    here is one of the support files.

    thanks.
    Dear Jonathan,

    read section Appendix B. Using a storage subsystem with a VMware ESX Server configuration and fix your configuration:

    Installation and Host Support Guide v10.8 - IBM System Storage DS Storage Manager
    https://www-947.ibm.com/support/entry/myportal/docdisplay?lndocid=migr-5090826

    And lets check your blades and switches. Post output of command supportshow from both FC switches, AMM Service Data, and DSA logs from all blades (use DSA Preboot Edition and post resulted files *.xml.gz).

    Regards,
    Alexander Novikov
    Russia, Moscow
  • jfrench
    jfrench
    9 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2012-10-23T01:11:48Z  
    Dear Jonathan,

    read section Appendix B. Using a storage subsystem with a VMware ESX Server configuration and fix your configuration:

    Installation and Host Support Guide v10.8 - IBM System Storage DS Storage Manager
    https://www-947.ibm.com/support/entry/myportal/docdisplay?lndocid=migr-5090826

    And lets check your blades and switches. Post output of command supportshow from both FC switches, AMM Service Data, and DSA logs from all blades (use DSA Preboot Edition and post resulted files *.xml.gz).

    Regards,
    Alexander Novikov
    Russia, Moscow
    Hello,
    I have been having this issue with a DS3524 since May. I have had several calls with IBM open and my last call had 11 techs on it without any resolution. My setup is using iSCSI with a BladeCenter H and 9 blades.

    Can anyone post a specify fix that worked.

    In our case replacing the Switches/Cables in the blade center did not resolve issue and no IBM tech seems to have any idea on cause or fix.

    Thanks,
    Jeff
  • Berniebgbb
    Berniebgbb
    83 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2012-10-23T12:26:59Z  
    We have one ibm bladecenter h with 8 blade server (4 hs22v & 4 hx5). the chassis have 2 brocade fibre switch connected to one ibm ds3512 with 2 controller and 2 exp3512 enclosures. We use this hardware to support a vmware infrastructure cluster version 5.0.U1

    Recently we upgrade the ds3512 FW from version 7.77.34 to 7.83.22 and the brocade fabric os from 6.3.1 to 6.4.2a. Since then we are experiencing this kind of behavior also i suspect that this kind of errors are produced when there are a lot of IOps on the luns

    here is one of the support files.

    thanks.
    Rebelinux,

    In the past when I have had "not on preferred path" issues specifically under load, it has been due to a partially faulty Fibre cable. It may not be relevant in your case but still worth checking.

    First capture the support logs from your FC switches as they are, then clear (reset) the port error counts on the external FC ports. In theory there should be no problems with the internal ports as they are through the back-plane of the chassis....

    Keep any eye on the error counts, don't expect zero as it is not un-usually to see errors. However compare the error counts across ports and see if any specific ports stands out for high errors. Example: If you are seeing tens of thousands of errors on a specific port then there are issues ;-).

    Don't divert all our energy to this, just worth reviewing.

    Have you had a chance to note down some details on the issue?

    1. What LUN's are effected?
    2. What date's and time's?
    3. Under what load?
    4. What overlaps with issue? (Batch processing period, Backup period, Login Storm period?, "VMware Event", "Host Event" (E.G: Reboot)).

    If there is a pattern, have you had a chance to isolate any infrastructure components? (E.G: Hosts...)

    Can you let us know when you did the firmware update on the storage? and if you have made any other changes to the storage or VMware environment at this time?
    As mentioned, send through the data on the chassis and the switches...

    Bernie.
  • Berniebgbb
    Berniebgbb
    83 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2012-10-23T12:34:17Z  
    • jfrench
    • ‏2012-10-23T01:11:48Z
    Hello,
    I have been having this issue with a DS3524 since May. I have had several calls with IBM open and my last call had 11 techs on it without any resolution. My setup is using iSCSI with a BladeCenter H and 9 blades.

    Can anyone post a specify fix that worked.

    In our case replacing the Switches/Cables in the blade center did not resolve issue and no IBM tech seems to have any idea on cause or fix.

    Thanks,
    Jeff
    jfrench,

    There are so many specifics with respect to a iSCSI solution with Blades with redundant pathing setup there is no simple answer.

    Can you tell us more about your environment and how it is setup?

    1. Storage Setup? Logs
    2. Switch setup? logs
    3. Host setup? Logs
    4. Solution setup? what are you trying to achieve?

    I have found that IBM tech's typically look at what they are responsible for (E.G: Server, Storage, fabric....) and not across the complete environment, and when / if they do it is through many diffident people. However my experience has been that most of the time I get a quick resolution for "known issues", but when it comes to user configurations issues that are nothing to do with hardware or firmware it is a different story....

    Hopefully the forum can help if you provide more info.

    Bernie.
  • jfrench
    jfrench
    9 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2012-10-23T14:36:19Z  
    Rebelinux,

    In the past when I have had "not on preferred path" issues specifically under load, it has been due to a partially faulty Fibre cable. It may not be relevant in your case but still worth checking.

    First capture the support logs from your FC switches as they are, then clear (reset) the port error counts on the external FC ports. In theory there should be no problems with the internal ports as they are through the back-plane of the chassis....

    Keep any eye on the error counts, don't expect zero as it is not un-usually to see errors. However compare the error counts across ports and see if any specific ports stands out for high errors. Example: If you are seeing tens of thousands of errors on a specific port then there are issues ;-).

    Don't divert all our energy to this, just worth reviewing.

    Have you had a chance to note down some details on the issue?

    1. What LUN's are effected?
    2. What date's and time's?
    3. Under what load?
    4. What overlaps with issue? (Batch processing period, Backup period, Login Storm period?, "VMware Event", "Host Event" (E.G: Reboot)).

    If there is a pattern, have you had a chance to isolate any infrastructure components? (E.G: Hosts...)

    Can you let us know when you did the firmware update on the storage? and if you have made any other changes to the storage or VMware environment at this time?
    As mentioned, send through the data on the chassis and the switches...

    Bernie.
    Bernie,
    Thanks for the response but at this point I was just looking to see other people's resolutions for this issue. I am currently dealing with IBM but no resolution has been found yet.
    Jeff
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2012-11-07T20:23:22Z  
    Rebelinux,

    In the past when I have had "not on preferred path" issues specifically under load, it has been due to a partially faulty Fibre cable. It may not be relevant in your case but still worth checking.

    First capture the support logs from your FC switches as they are, then clear (reset) the port error counts on the external FC ports. In theory there should be no problems with the internal ports as they are through the back-plane of the chassis....

    Keep any eye on the error counts, don't expect zero as it is not un-usually to see errors. However compare the error counts across ports and see if any specific ports stands out for high errors. Example: If you are seeing tens of thousands of errors on a specific port then there are issues ;-).

    Don't divert all our energy to this, just worth reviewing.

    Have you had a chance to note down some details on the issue?

    1. What LUN's are effected?
    2. What date's and time's?
    3. Under what load?
    4. What overlaps with issue? (Batch processing period, Backup period, Login Storm period?, "VMware Event", "Host Event" (E.G: Reboot)).

    If there is a pattern, have you had a chance to isolate any infrastructure components? (E.G: Hosts...)

    Can you let us know when you did the firmware update on the storage? and if you have made any other changes to the storage or VMware environment at this time?
    As mentioned, send through the data on the chassis and the switches...

    Bernie.
    I can confirm that disabling VAAI DataMover.HardwareAcceleratedMove and DataMover.HardwareAcceleratedInit resolve my issues of intermittent controller path changes. No more errors seen since 2 weeks. Maybe firmware bug?

    http://pubs.vmware.com/vsphere-51/topic/com.vmware.vsphere.storage.doc/GUID-88969C21-CB0D-499F-8902-8269E31057A3.html
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: DS3512 prefered path changing intermittently issue .

    ‏2012-11-07T20:30:11Z  
    I can confirm that disabling VAAI DataMover.HardwareAcceleratedMove and DataMover.HardwareAcceleratedInit resolve my issues of intermittent controller path changes. No more errors seen since 2 weeks. Maybe firmware bug?

    http://pubs.vmware.com/vsphere-51/topic/com.vmware.vsphere.storage.doc/GUID-88969C21-CB0D-499F-8902-8269E31057A3.html
    also forgot to mention possible data losses. Also we fix this disabling VAAI DataMover.HardwareAcceleratedMove and DataMover.HardwareAcceleratedInit

    http://www.ibm.com/developerworks/forums/thread.jspa?threadID=460553&tstart=0
  • NAYABSK
    NAYABSK
    1 Post

    Re: DS3512 prefered path changing intermittently issue .

    ‏2014-10-31T09:43:41Z  
    also forgot to mention possible data losses. Also we fix this disabling VAAI DataMover.HardwareAcceleratedMove and DataMover.HardwareAcceleratedInit

    http://www.ibm.com/developerworks/forums/thread.jspa?threadID=460553&tstart=0

    Hi All,

     

    I know this thread is very old but i too have the same issue can anybody help or advise me to fix this issue.

     

    Thanks,

    Nayab