Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
13 replies Latest Post - ‏2012-11-07T15:06:20Z by DmitriySDM
DmitriySDM
DmitriySDM
9 Posts
ACCEPTED ANSWER

Pinned topic paths failed to Ds3400

‏2012-10-11T11:54:24Z |
Hello all,

We have the following issue with our cluster:
After startup of OS several paths to storage disappeared in about 4 minutes. From four paths to storage remained only two: 1 path by each controller.
If we make switch off and switch on of controller of storage then the path recovered.

In Windows Event Log we have 3 events:
Warning 10.10.2012 17:31:05 ds4dsm 7
DS3400:0:0 Path Failed.

Error 10.10.2012 17:31:50 mpio 16
A fail-over on \Device\MPIODisk3 occurred.

Warning 10.10.2012 17:31:50 mpio 17
\Device\MPIODisk3 is currently in a degraded state. One or more paths have failed, though the process is now complete.

The storage log is empty.

Equipment:
Storage DS3400(f/w version:07.35.66.00)+2Exp3000.
Blades: 3x IBM HS22v,
Installed software on all servers:
OS: Windows server 2008 R2 SP1 Datacenter (all updates), with role Hyper-V.
DSM version 1.3.305.608
Qlogic FCA 9.1.8.38
All servers in failover clusters.

Could you please help us to resolve this issue.

Many thanks in advance.
Updated on 2012-11-07T15:06:20Z at 2012-11-07T15:06:20Z by DmitriySDM
  • Novikov_Alexander
    Novikov_Alexander
    1404 Posts
    ACCEPTED ANSWER

    Re: paths failed to Ds3400

    ‏2012-10-11T18:31:58Z  in response to DmitriySDM
    Dear Dmytro,

    looks like wrong configuration created. Lets check it.
    Could you provide the next data:
    • AMM Service Data;
    • DSA logs from problem servers HS22V;
    • DS Support Data from DS3400;
    • supportshow output from both FC switches (if Brocade FC used).

    Regards,
    Alexander Novikov
    Russia, Moscow
    • DmitriySDM
      DmitriySDM
      9 Posts
      ACCEPTED ANSWER

      Re: paths failed to Ds3400

      ‏2012-10-12T14:07:47Z  in response to Novikov_Alexander
      Thank you for your quick answer.
      I put log files in Skydrive
      https://skydrive.live.com/redir?resid=F22ADD69F7EBCDEC!207&authkey=!AEOZx9kSFcwxOy8

      We have had this issue for a short period of time. Before this issue we had another problem with paths too, but not with OS startup.
      To solve this problem we installed the last versions of dsm driver and http://support.microsoft.com/kb/2406705
      Now we are having an issue, described in the first post.
      • Novikov_Alexander
        Novikov_Alexander
        1404 Posts
        ACCEPTED ANSWER

        Re: paths failed to Ds3400

        ‏2012-10-13T12:43:10Z  in response to DmitriySDM
        Hi Dmytro,

        yes, unfortunately wrong configuration used, as expected. No any zoning, wrong switch settings, wrong fc cables connection, old software, ...
        Read the documentation and fix settings:

        Supported configurations for 8 Gigabit Fibre Channel - IBM BladeCenter H (7989, 8852)
        https://www-947.ibm.com/support/entry/myportal/docdisplay?lndocid=MIGR-5087103

        Installation and Host Support Guide v10.8 - IBM System Storage DS Storage Manager
        http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=migr-5090826

        DS3000 Recommended Firmware Levels
        http://www-304.ibm.com/support/docview.wss?uid=ssg1S1003713

        Installation, User's, and Maintenance Guide - IBM System Storage DS3400
        http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5069916

        IBM System Storage DS3000: Introduction and Implementation Guide
        http://www.redbooks.ibm.com/abstracts/sg247065.html

        Best regards,
        Alexander Novikov
        Russia, Moscow
        • DmitriySDM
          DmitriySDM
          9 Posts
          ACCEPTED ANSWER

          Re: paths failed to Ds3400

          ‏2012-10-16T09:48:45Z  in response to Novikov_Alexander
          Tell me please, if the connections on the picture are correct? or i need to make connections as shown in the IBM guide (without connections highlighted in red on the picture)

          Attachments

          • DmitriySDM
            DmitriySDM
            9 Posts
            ACCEPTED ANSWER

            Re: paths failed to Ds3400

            ‏2012-10-16T09:51:15Z  in response to DmitriySDM
            Here is the correct picture.
            • Novikov_Alexander
              Novikov_Alexander
              1404 Posts
              ACCEPTED ANSWER

              Re: paths failed to Ds3400

              ‏2012-10-16T15:09:00Z  in response to DmitriySDM
              Hi Dmytro,

              personally for you it is wrong picture, because Windows used on your server (migrate to VMware ESX and this picture would be correct).

              Regards,
              Alexander Novikov
              Russia, Moscow
          • Novikov_Alexander
            Novikov_Alexander
            1404 Posts
            ACCEPTED ANSWER

            Re: paths failed to Ds3400

            ‏2012-10-16T15:11:42Z  in response to DmitriySDM
            Dmytro,

            yes, you need to make connections as shown in the IBM guide (without connections highlighted in red on the picture).

            Regards,
            Alexander Novikov
            Russia, Moscow
            • DmitriySDM
              DmitriySDM
              9 Posts
              ACCEPTED ANSWER

              Re: paths failed to Ds3400

              ‏2012-10-22T08:49:29Z  in response to Novikov_Alexander
              Hello, Alexander
              We updated firmware of the storage.
              We made connections to storage as shown in the IBM guide.
              We made and applied zones on the switches.

              Since then we have not had the issue with th path failure on the system startup.

              Can your check the new configuration for mistakes ?
              The new configuration logs are here.
              https://skydrive.live.com/?cid=F22ADD69F7EBCDEC&id=F22ADD69F7EBCDEC%21209
        • DmitriySDM
          DmitriySDM
          9 Posts
          ACCEPTED ANSWER

          Re: paths failed to Ds3400

          ‏2012-10-31T09:49:15Z  in response to Novikov_Alexander
          HI Alexander,
          We have the following issue with our cluster today:
          We have path failed to Storage DS3400 today at 3:00AM
          In Windows Event Log we have 7 events:

          Warning 31.10.2012 3:02:00 ql2300 129
          Reset to device, \Device\RaidPort2, was issued.

          Warning 31.10.2012 3:02:35 ds4dsm 7
          DS3400:1:0 Path Failed.

          Warning 31.10.2012 3:04:40 mpio 17 None
          \Device\MPIODisk1 is currently in a degraded state. One or more paths have failed, though the process is now complete.

          Error 31.10.2012 3:04:40 mpio 16 None
          A fail-over on \Device\MPIODisk2 occurred.

          Warning 31.10.2012 3:04:40 mpio 17 None
          \Device\MPIODisk2 is currently in a degraded state. One or more paths have failed, though the process is now complete.

          Warning 31.10.2012 3:04:44 ds4dsm 164 None
          Lun failover command issued to path DS3400:0:0. (DumpData contains the lun bitmap). (failover event 1).

          Warning 31.10.2012 3:04:48 ds4dsm 801 None
          Failover succeeded to DS3400:0.

          I put log files in Skydrive
          https://skydrive.live.com/?cid=F22ADD69F7EBCDEC&id=F22ADD69F7EBCDEC%21209

          Do we have a problem with Storage DS3400 ?
          We have another storage ds3524+exp3512 and dont have any problems with it.
          • Novikov_Alexander
            Novikov_Alexander
            1404 Posts
            ACCEPTED ANSWER

            Re: paths failed to Ds3400

            ‏2012-11-05T10:25:02Z  in response to DmitriySDM
            Dear Dmytro,

            could you please post logs directly in this Forum. We need DS Support Data of DS3400, DSA log from server and output of command supportshow from both FC switches (Brocade).

            Regards,
            Alexander Novikov
            Russia, Moscow
            • DmitriySDM
              DmitriySDM
              9 Posts
              ACCEPTED ANSWER

              Re: paths failed to Ds3400

              ‏2012-11-07T12:52:23Z  in response to Novikov_Alexander
              We updated firmware of the FC switch.
              We replaced FC cables to storage.
              We are planned to updated the latest firmware for EXP3000 and disk.

              Attachments

            • DmitriySDM
              DmitriySDM
              9 Posts
              ACCEPTED ANSWER

              Re: paths failed to Ds3400

              ‏2012-11-07T14:48:26Z  in response to Novikov_Alexander
              We have found the Rx Errors on the FC switch ports with connected DS3400.

              We updated the FC switch firmware, replaced FC cables from switch to storage and configured fixed speed of 4gb on the port but the situation has not changed. Can the problems with path failure of DS controllers to blades be connected with the described errors?
          • DmitriySDM
            DmitriySDM
            9 Posts
            ACCEPTED ANSWER

            Re: paths failed to Ds3400

            ‏2012-11-07T15:06:20Z  in response to DmitriySDM
            We had a new issue on 31/10/2012 at 3:00am
            The path to controller B disappeared on 31/10/2012 at 3:00am.
            We decided to restart controller B (Offline/Online)
            After these actions the access to Volume 3 from servers disappeared. We determined that the connection between storage DS3400 and one of EXP3000 disappeared, because all the resources stored on this EXP3000 were unavailable. We saw following message In the Storage manager :
            1) Array is missing
            2) Missing HotSpare Drives
            The issue was solved only after we turned off DS3400 and EXP3000 and turned them on again.
            Also after the appearance of the following issue during the startup of the Storage manager appeared the window below