Topic
23 replies Latest Post - ‏2012-11-07T20:30:11Z by SystemAdmin
SystemAdmin
SystemAdmin
4779 Posts
ACCEPTED ANSWER

Pinned topic DS3512 prefered path changing intermittently issue .

‏2011-04-12T17:07:24Z |
Dear All,

I am facing problem with my storage DS3512 i have configure storage with IBM blade Center H (88524TG) and host server is IBM blade server HX5 (787264G) OS windows 2008 DATA Center x64 everything seems to be working fine but my logical drive changed its preferred path intermittently i have try to update HBA device driver and MPIO driver but still its changing path and also i saw some error on event log "Controller cache not enabled or was internally disabled" or "Logical Drive not on preferred path due to ADT/RDAC failover"

can anyone help me to find out root causes of this issue ?
logs atteched herewith for your reference.

Regards,
Noman Ajaz Ahmed Siddiqui

Attachments

Updated on 2012-11-07T20:30:11Z at 2012-11-07T20:30:11Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    4779 Posts
    ACCEPTED ANSWER

    Re: DS3512 prefered path changing intermittently issue .

    ‏2011-04-12T19:00:38Z  in response to SystemAdmin
    Hi Noman,
    Please provide the CASD and the DSA (*.xml) for analysis.
    Regards.
    • SystemAdmin
      SystemAdmin
      4779 Posts
      ACCEPTED ANSWER

      Re: DS3512 prefered path changing intermittently issue .

      ‏2011-04-12T19:27:19Z  in response to SystemAdmin
      Already attached in my post.

      Regards,
      Noman Ajaz
      • andrew.scott
        andrew.scott
        1 Post
        ACCEPTED ANSWER

        Re: DS3512 prefered path changing intermittently issue .

        ‏2011-04-13T06:58:44Z  in response to SystemAdmin
        How did you get on with this issue?
        Also have a DS3512 with two esx4.1 hosts (x3650 M3) stuck in a continual failover loop.
        • SystemAdmin
          SystemAdmin
          4779 Posts
          ACCEPTED ANSWER

          Re: DS3512 prefered path changing intermittently issue .

          ‏2011-04-13T11:29:57Z  in response to andrew.scott
          Its new configuration.
          its not esx4.1 hosts. its windows 2008 Data center x64 with hyper-visor on blade server HX5 and its scalable partition.

          And i have attached my hardware details.
          • SystemAdmin
            SystemAdmin
            4779 Posts
            ACCEPTED ANSWER

            Re: DS3512 prefered path changing intermittently issue .

            ‏2011-04-13T11:49:17Z  in response to SystemAdmin
            Find in attachment.
          • pavel.alexei
            pavel.alexei
            334 Posts
            ACCEPTED ANSWER

            Re: DS3512 prefered path changing intermittently issue .

            ‏2011-04-14T08:30:08Z  in response to SystemAdmin
            As far i understood you have 4 HX5 blades.
            Judging Storage profile 2 blades works independently and didn't access storage (OTHMKT-BLADE1 and OTHMKT-BLADE2) and from rest 2 you created a two node complex. I am right?
            Did you zone Fiber Switches?
            And what is the purpose of OSLUN1,OSLUN2,OSLUN3?
  • dbilyk
    dbilyk
    1 Post
    ACCEPTED ANSWER

    Re: DS3512 prefered path changing intermittently issue .

    ‏2011-05-06T20:27:20Z  in response to SystemAdmin
    Has any progress been made with this issue? I have been using our DS3512 for the past four months without issue and now suddenly, the preferred controller keeps switching from one to the other. Things have just worked up until now, so much so that I did not suspect anything wrong with the DS3512 until our ESXi VMs suddenly started to bog on heavy I/O access. When I checked the DS3512, that is when I saw the warning that the current controller was not the preferred one. Then, 15 minutes later, things swapped; it now preferred A and was connected to B whereas previously it preferred B and was connected to A. I do not know what is causing this switch back and forth but I suspect it is the cause of our VMs bogging down so heavily.

    Despite operating the DS3512 for four months, I am still a relative n00b. Until now, things just worked. I should have suspected something going together so easily was just hiding an issue that would someday materialize. At least we are still in pre-production.

    If anyone has any suggestions, I would love to hear them.
    • skyron
      skyron
      84 Posts
      ACCEPTED ANSWER

      Re: DS3512 prefered path changing intermittently issue .

      ‏2011-05-07T04:52:12Z  in response to dbilyk
      This issue mostly comes up due to either incorrect zoning, or having old fiber card drivers (qlogic/emulex) on the server. If you update them to the latest, especially for ESX/ESXi, it resolves the issue in most cases.
  • SystemAdmin
    SystemAdmin
    4779 Posts
    ACCEPTED ANSWER

    Re: DS3512 prefered path changing intermittently issue .

    ‏2011-05-09T17:35:31Z  in response to SystemAdmin
    issue resolved.
    thanks for your support
    • itguy4hospice
      itguy4hospice
      1 Post
      ACCEPTED ANSWER

      Re: DS3512 prefered path changing intermittently issue .

      ‏2011-06-29T22:11:42Z  in response to SystemAdmin
      What part of the previous post (if indeed that's the answer) corrected your issue. I am seeing same on our DS3512
    • kashif.adeel
      kashif.adeel
      3 Posts
      ACCEPTED ANSWER

      Re: DS3512 prefered path changing intermittently issue .

      ‏2012-10-18T16:50:25Z  in response to SystemAdmin
      Dear Nouman
      what was the root cause of this issue and how'ld you solved this issue please share.
      • Berniebgbb
        Berniebgbb
        75 Posts
        ACCEPTED ANSWER

        Re: DS3512 prefered path changing intermittently issue .

        ‏2012-10-19T03:27:12Z  in response to kashif.adeel
        Hi all,

        To possibly help a few people I though I would not some core configuration steps, as most people have said the majority of the time "path thrashing" is due to incorrect cabling, zoning or host type / multi-pathing setup issues. Each setup has different specific requirements so my comments below are broad and just a quick summary....

        1. Storage Host Port configuration
        Unless you are using SAS connections, you must have a switch fabric (redundant preferred) between the hosts and the storage.

        Cabling for Fibre channel and iSCSI from storage(generally):
        Controller A Port 1 -- Switch 1
        Controller A Port 2 -- Switch 2
        Controller B Port 1 -- Switch 1
        Controller B Port 2 -- Switch 2

        Cabling from Host:
        Host HBA 1 -- Switch 1
        Host HBA 2 -- Switch 2

        Note: To rule out issues; if you are configuring multipathing to dual controller, ALL hosts should be configured for multipathing with visibility to both controllers. Especially if they are accessing the same LUN's. Generally each HBA in the host should have visibility to BOTH controllers in the storage.

        Zoning (Fibre)
        Create Alia's for your Storage Ports and your Host ports.
        For DS3500 setup "single target, single initiator" zoning. So, create a zone for each host port to each storage port, then put all the zones in a config and enable it.(NOTE: This is different for other arrays E.G: V7000)
        The main thing is that each host can see 4 paths. once you have setup the zoning and the storage is running

        Register hosts in storage (depending on connection type and card type)
        Register 2 x HBA's for fibre to the host
        Register 2 x Hardware iSCSI HBA's to the host
        Register 1 x Sofware iSCSI Initiator to the host (you only have 1 x software initiator however pathing can still be through multiple NIC's for redundancy).

        NOTE: If you IQN or WWN for your cards are not showing up under "Auto Discovered" then there is something wrong, don't bother trying to enter them manually.
        Re-Scan from your hosts to help the HBA's register to the storage....see if they are now register under "Auto Discovered", if not you still have problems.

        Host type:
        Make sure you use the correct host type when registering the host.
        For VMware - select VMWARE (or for old firmware LNXCLVMWARE)
        For Microsoft Servers running IBM DSM Software with Microsoft MPIO select Microsoft XXXXXX
        For Microsoft Servers running Microsoft MPIO with Veritas Dynamic Multi-pathing select Microsoft XXXXX (Supporting DMP)
        When Clustering select the cluster host type option.
        Once you have the hosts register you can MAP a LUN, once the LUN is mapped you can check via the HBA BIOS to see if both HBA's can see the target device and LUN.
        You want to make sure all HBA's can see the Array.
        For Software iSCSI, you want to make sure that you can ping ALL host ports of the Array.

        Host Multipathing
        Once you have your mappings right and your LUN/s mapped you need to setup the multipathing on the Host
        For VMware, multipathing is enabled by default, however you should check your pathing policy and the QTy of paths to be sure the zoning is correct

        For Microsoft, firstly make sure you have the latest HBA firmware and StorPort drives, then you will need to install Microsoft MPIO framework (windows 2008) and then the IBM DSM modules. The IBM DSM modules are now packaged separately to Storage Manager software so make sure you install it. Just installing IBM storage manager will NOT install the multipathing software.

        Once the multipathing is install and the LUN is visible you can run command lines with the IBM DSM to confirm pathing visibility utilising the "SMcli" commands...Google SMcli...

        You should see multiple paths for all luns.

        Another item to check is pathing issues due to fabric issues, so check your port error levels on your FC switches....

        Bernie
  • SystemAdmin
    SystemAdmin
    4779 Posts
    ACCEPTED ANSWER

    Re: DS3512 prefered path changing intermittently issue .

    ‏2011-11-24T17:06:40Z  in response to SystemAdmin
    Hi Sorry for digging up an old post, but I'm having the same issue. DS3500 with Windows 2008 Cluster. Logical Drive not on preferred path due to ADT/RDAC failover errors and the drives would go offline.

    What was the fix for it?? thanks.
  • SystemAdmin
    SystemAdmin
    4779 Posts
    ACCEPTED ANSWER

    Re: DS3512 prefered path changing intermittently issue .

    ‏2012-10-20T02:25:52Z  in response to SystemAdmin
    im also experiencing this behavior after firmware upgrade 7.83. message like "Controller cache not enabled or was internally disabled" or "Logical Drive not on preferred path due to ADT/RDAC failover".
    • Berniebgbb
      Berniebgbb
      75 Posts
      ACCEPTED ANSWER

      Re: DS3512 prefered path changing intermittently issue .

      ‏2012-10-21T10:34:58Z  in response to SystemAdmin
      Rebelinux, can you provide configuration information?

      or even better Support logs for your storage and switches?

      Bernie.
      • SystemAdmin
        SystemAdmin
        4779 Posts
        ACCEPTED ANSWER

        Re: DS3512 prefered path changing intermittently issue .

        ‏2012-10-22T13:18:21Z  in response to Berniebgbb
        We have one ibm bladecenter h with 8 blade server (4 hs22v & 4 hx5). the chassis have 2 brocade fibre switch connected to one ibm ds3512 with 2 controller and 2 exp3512 enclosures. We use this hardware to support a vmware infrastructure cluster version 5.0.U1

        Recently we upgrade the ds3512 FW from version 7.77.34 to 7.83.22 and the brocade fabric os from 6.3.1 to 6.4.2a. Since then we are experiencing this kind of behavior also i suspect that this kind of errors are produced when there are a lot of IOps on the luns

        here is one of the support files.

        thanks.
        • Novikov_Alexander
          Novikov_Alexander
          1392 Posts
          ACCEPTED ANSWER

          Re: DS3512 prefered path changing intermittently issue .

          ‏2012-10-22T17:56:55Z  in response to SystemAdmin
          Dear Jonathan,

          read section Appendix B. Using a storage subsystem with a VMware ESX Server configuration and fix your configuration:

          Installation and Host Support Guide v10.8 - IBM System Storage DS Storage Manager
          https://www-947.ibm.com/support/entry/myportal/docdisplay?lndocid=migr-5090826

          And lets check your blades and switches. Post output of command supportshow from both FC switches, AMM Service Data, and DSA logs from all blades (use DSA Preboot Edition and post resulted files *.xml.gz).

          Regards,
          Alexander Novikov
          Russia, Moscow
          • jfrench
            jfrench
            9 Posts
            ACCEPTED ANSWER

            Re: DS3512 prefered path changing intermittently issue .

            ‏2012-10-23T01:11:48Z  in response to Novikov_Alexander
            Hello,
            I have been having this issue with a DS3524 since May. I have had several calls with IBM open and my last call had 11 techs on it without any resolution. My setup is using iSCSI with a BladeCenter H and 9 blades.

            Can anyone post a specify fix that worked.

            In our case replacing the Switches/Cables in the blade center did not resolve issue and no IBM tech seems to have any idea on cause or fix.

            Thanks,
            Jeff
            • Berniebgbb
              Berniebgbb
              75 Posts
              ACCEPTED ANSWER

              Re: DS3512 prefered path changing intermittently issue .

              ‏2012-10-23T12:34:17Z  in response to jfrench
              jfrench,

              There are so many specifics with respect to a iSCSI solution with Blades with redundant pathing setup there is no simple answer.

              Can you tell us more about your environment and how it is setup?

              1. Storage Setup? Logs
              2. Switch setup? logs
              3. Host setup? Logs
              4. Solution setup? what are you trying to achieve?

              I have found that IBM tech's typically look at what they are responsible for (E.G: Server, Storage, fabric....) and not across the complete environment, and when / if they do it is through many diffident people. However my experience has been that most of the time I get a quick resolution for "known issues", but when it comes to user configurations issues that are nothing to do with hardware or firmware it is a different story....

              Hopefully the forum can help if you provide more info.

              Bernie.
        • Berniebgbb
          Berniebgbb
          75 Posts
          ACCEPTED ANSWER

          Re: DS3512 prefered path changing intermittently issue .

          ‏2012-10-23T12:26:59Z  in response to SystemAdmin
          Rebelinux,

          In the past when I have had "not on preferred path" issues specifically under load, it has been due to a partially faulty Fibre cable. It may not be relevant in your case but still worth checking.

          First capture the support logs from your FC switches as they are, then clear (reset) the port error counts on the external FC ports. In theory there should be no problems with the internal ports as they are through the back-plane of the chassis....

          Keep any eye on the error counts, don't expect zero as it is not un-usually to see errors. However compare the error counts across ports and see if any specific ports stands out for high errors. Example: If you are seeing tens of thousands of errors on a specific port then there are issues ;-).

          Don't divert all our energy to this, just worth reviewing.

          Have you had a chance to note down some details on the issue?

          1. What LUN's are effected?
          2. What date's and time's?
          3. Under what load?
          4. What overlaps with issue? (Batch processing period, Backup period, Login Storm period?, "VMware Event", "Host Event" (E.G: Reboot)).

          If there is a pattern, have you had a chance to isolate any infrastructure components? (E.G: Hosts...)

          Can you let us know when you did the firmware update on the storage? and if you have made any other changes to the storage or VMware environment at this time?
          As mentioned, send through the data on the chassis and the switches...

          Bernie.
          • jfrench
            jfrench
            9 Posts
            ACCEPTED ANSWER

            Re: DS3512 prefered path changing intermittently issue .

            ‏2012-10-23T14:36:19Z  in response to Berniebgbb
            Bernie,
            Thanks for the response but at this point I was just looking to see other people's resolutions for this issue. I am currently dealing with IBM but no resolution has been found yet.
            Jeff
          • SystemAdmin
            SystemAdmin
            4779 Posts
            ACCEPTED ANSWER

            Re: DS3512 prefered path changing intermittently issue .

            ‏2012-11-07T20:23:22Z  in response to Berniebgbb
            I can confirm that disabling VAAI DataMover.HardwareAcceleratedMove and DataMover.HardwareAcceleratedInit resolve my issues of intermittent controller path changes. No more errors seen since 2 weeks. Maybe firmware bug?

            http://pubs.vmware.com/vsphere-51/topic/com.vmware.vsphere.storage.doc/GUID-88969C21-CB0D-499F-8902-8269E31057A3.html
            • SystemAdmin
              SystemAdmin
              4779 Posts
              ACCEPTED ANSWER

              Re: DS3512 prefered path changing intermittently issue .

              ‏2012-11-07T20:30:11Z  in response to SystemAdmin
              also forgot to mention possible data losses. Also we fix this disabling VAAI DataMover.HardwareAcceleratedMove and DataMover.HardwareAcceleratedInit

              http://www.ibm.com/developerworks/forums/thread.jspa?threadID=460553&tstart=0