Topic
  • 29 replies
  • Latest Post - ‏2014-07-25T00:07:51Z by ParisWells
Oliver_Wegener
Oliver_Wegener
18 Posts

Pinned topic Problem with replaced drive

‏2008-09-09T07:39:39Z |
Hello,

we have a DS4700 with a SATA expansion configured as RAID5 with two spares. Now following happend:

  • one drive had an "impending drive failure"
  • I've manually failed the drive and a hotspare drive started automaticly to reconstruct the RAID
  • after the successful rebuild I've pulled the drive out of the array
  • we contacted the support and got a new drive
  • then we entered the new drive into the array
  • right after the array recognices the new drive the copyback operation started
  • everything looked good (I saw the time symbol at the volume description)
  • after some time the time symbol disapeared and everything in the properties says optimal

BUT:

  • the new drive is marked with the "yellow replaced symbol"
  • the hotspare is still in use (the drive is gry with the red hotspare symbol on the bottom)
  • the Recovey Guru reports reports that there is a hot spare in use
  • if I click on the Recovery Guru an then on the hot spare in use point the window loads and loads and loads but no recovery steps are shown

Now I'm pretty unsure what to do. I've waited two days but nothing happens.

I've added a screenshot to this post. Could anyone give me an advice? thx
Updated on 2011-06-24T06:47:08Z at 2011-06-24T06:47:08Z by SystemAdmin
  • Novikov_Alexander
    Novikov_Alexander
    1404 Posts

    Re: Problem with replaced drive

    ‏2008-09-09T08:00:19Z  
    Dear Oliver,

    lets check DS4700 profile and logs. Post DS4700 Support Archive to Forum.

    Regards,
    Alexander Novikov
    Russia, Moscow
  • Oliver_Wegener
    Oliver_Wegener
    18 Posts

    Re: Problem with replaced drive

    ‏2008-09-09T08:12:36Z  
    Dear Oliver,

    lets check DS4700 profile and logs. Post DS4700 Support Archive to Forum.

    Regards,
    Alexander Novikov
    Russia, Moscow
    thx

    now first the log

    Attachments

  • Oliver_Wegener
    Oliver_Wegener
    18 Posts

    Re: Problem with replaced drive

    ‏2008-09-09T08:30:42Z  
    Dear Oliver,

    lets check DS4700 profile and logs. Post DS4700 Support Archive to Forum.

    Regards,
    Alexander Novikov
    Russia, Moscow
    now the complete Support Data

    I hope this helps to fix my Problem

    thx
  • Oliver_Wegener
    Oliver_Wegener
    18 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T05:34:08Z  
    Alexander, did you had time to look into the logs? Should I call the support and make an incident?
  • Novikov_Alexander
    Novikov_Alexander
    1404 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T06:56:59Z  
    Alexander, did you had time to look into the logs? Should I call the support and make an incident?
    Oliver,

    sorry for delay.
    First of all update next software:

    • DS4000 Firmware version from 07.10.23.00 to 07.15.07.00;
    • EXP810 firmware from versions 98A3, 9898 to 98C1;
    • SATA HDDs firmware to package versions EP58 and ES58.

    Software and device drivers DS4700 Express
    http://www-304.ibm.com/systems/support/supportsite.wss/selectproduct?taskind=2&brandind=5000028&familyind=5329827&typeind=0&modelind=0&osind=0&psid=sr&continue.x=1&matrix=Y

    Don't forget update DS4000 Storage Manager (and RDAC) to version 10.30.
    I am sure that your problem solved in this fresh software (see readme files).

    Regards,
    Alexander Novikov
    Russia, Moscow
  • Oliver_Wegener
    Oliver_Wegener
    18 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T08:10:18Z  
    Oliver,

    sorry for delay.
    First of all update next software:

    • DS4000 Firmware version from 07.10.23.00 to 07.15.07.00;
    • EXP810 firmware from versions 98A3, 9898 to 98C1;
    • SATA HDDs firmware to package versions EP58 and ES58.

    Software and device drivers DS4700 Express
    http://www-304.ibm.com/systems/support/supportsite.wss/selectproduct?taskind=2&brandind=5000028&familyind=5329827&typeind=0&modelind=0&osind=0&psid=sr&continue.x=1&matrix=Y

    Don't forget update DS4000 Storage Manager (and RDAC) to version 10.30.
    I am sure that your problem solved in this fresh software (see readme files).

    Regards,
    Alexander Novikov
    Russia, Moscow
    thx for the reply.

    Could this be done "online" or does the updating interrupt the connections?
  • Oliver_Wegener
    Oliver_Wegener
    18 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T08:21:03Z  
    Oliver,

    sorry for delay.
    First of all update next software:

    • DS4000 Firmware version from 07.10.23.00 to 07.15.07.00;
    • EXP810 firmware from versions 98A3, 9898 to 98C1;
    • SATA HDDs firmware to package versions EP58 and ES58.

    Software and device drivers DS4700 Express
    http://www-304.ibm.com/systems/support/supportsite.wss/selectproduct?taskind=2&brandind=5000028&familyind=5329827&typeind=0&modelind=0&osind=0&psid=sr&continue.x=1&matrix=Y

    Don't forget update DS4000 Storage Manager (and RDAC) to version 10.30.
    I am sure that your problem solved in this fresh software (see readme files).

    Regards,
    Alexander Novikov
    Russia, Moscow
    Is the SATA HDDs firmware to package versions EP58 and ES58 included in the Hard Disk Drive and ESM Firmware Update Package v1.59?
  • Oliver_Wegener
    Oliver_Wegener
    18 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T08:25:22Z  
    Oliver,

    sorry for delay.
    First of all update next software:

    • DS4000 Firmware version from 07.10.23.00 to 07.15.07.00;
    • EXP810 firmware from versions 98A3, 9898 to 98C1;
    • SATA HDDs firmware to package versions EP58 and ES58.

    Software and device drivers DS4700 Express
    http://www-304.ibm.com/systems/support/supportsite.wss/selectproduct?taskind=2&brandind=5000028&familyind=5329827&typeind=0&modelind=0&osind=0&psid=sr&continue.x=1&matrix=Y

    Don't forget update DS4000 Storage Manager (and RDAC) to version 10.30.
    I am sure that your problem solved in this fresh software (see readme files).

    Regards,
    Alexander Novikov
    Russia, Moscow
    Is the installation in this order correct?

    1. Update Storage Manager
    2. DS4000 Firmware
    3. EX810 Firmwares
    4. SATA Firmwares

    The RDAC is the multipathing, correct? So I should update the connected Windows Servers? Whats with the VMWare ESX Servers?
  • Novikov_Alexander
    Novikov_Alexander
    1404 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T08:31:15Z  
    Is the SATA HDDs firmware to package versions EP58 and ES58 included in the Hard Disk Drive and ESM Firmware Update Package v1.59?
    Oliver,

    try check readme file for Hard Disk Drive and ESM Firmware Update Package v1.59.

    Regards,
    Alexander Novikov
    Russia, Moscow
  • Novikov_Alexander
    Novikov_Alexander
    1404 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T08:32:30Z  
    thx for the reply.

    Could this be done "online" or does the updating interrupt the connections?
    Oliver,

    check readme files regarding any online/offline recomendations.

    Regards,
    Alexander Novikov
    Russia, Moscow
  • Novikov_Alexander
    Novikov_Alexander
    1404 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T08:43:40Z  
    Is the installation in this order correct?

    1. Update Storage Manager
    2. DS4000 Firmware
    3. EX810 Firmwares
    4. SATA Firmwares

    The RDAC is the multipathing, correct? So I should update the connected Windows Servers? Whats with the VMWare ESX Servers?
    Oliver,

    looks good, but don't forget read readme files for updates before updates for checking installation instructions.

    > The RDAC is the multipathing, correct?

    Yes.

    > So I should update the connected Windows Servers?

    IMHO points 2 and 3 must be swapped (see section 1.4 Dependencies for DS4000 Firmware).

    > Whats with the VMWare ESX Servers?

    What do you meant?

    Regards,
    Alexander Novikov
    Russia, Moscow
  • Oliver_Wegener
    Oliver_Wegener
    18 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T08:46:20Z  
    Oliver,

    looks good, but don't forget read readme files for updates before updates for checking installation instructions.

    > The RDAC is the multipathing, correct?

    Yes.

    > So I should update the connected Windows Servers?

    IMHO points 2 and 3 must be swapped (see section 1.4 Dependencies for DS4000 Firmware).

    > Whats with the VMWare ESX Servers?

    What do you meant?

    Regards,
    Alexander Novikov
    Russia, Moscow
    > > Whats with the VMWare ESX Servers?
    >
    > What do you meant?

    Do I have to install anything on the connected ESX 3.5 hosts like an agent or something?
  • Novikov_Alexander
    Novikov_Alexander
    1404 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T08:58:22Z  
    > > Whats with the VMWare ESX Servers?
    >
    > What do you meant?

    Do I have to install anything on the connected ESX 3.5 hosts like an agent or something?
    Oliver,

    read readme file for VMware which available on the same page:

    Software and device drivers DS4700 Express
    http://www-304.ibm.com/systems/support/supportsite.wss/selectproduct?taskind=2&brandind=5000028&familyind=5329827&typeind=0&modelind=0&osind=0&psid=sr&continue.x=1&matrix=Y

    Regards,
    Alexander Novikov
    Russia, Moscow
  • Novikov_Alexander
    Novikov_Alexander
    1404 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T09:02:10Z  
    > > Whats with the VMWare ESX Servers?
    >
    > What do you meant?

    Do I have to install anything on the connected ESX 3.5 hosts like an agent or something?
    Oliver,

    there is good doc (see "Chapter 7. Using DS4000 with a VMware ESX Server configuration"):

    Installation and Support Guide for Windows 2000, Windows Server 2003, NetWare, ESX Server, and Linux - IBM TotalStorage DS4000 Storage Manager V10.10
    http://www-304.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-57808&brandind=5000028

    Regards,
    Alexander Novikov
    Russia, Moscow
  • Oliver_Wegener
    Oliver_Wegener
    18 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T09:18:30Z  
    Oliver,

    looks good, but don't forget read readme files for updates before updates for checking installation instructions.

    > The RDAC is the multipathing, correct?

    Yes.

    > So I should update the connected Windows Servers?

    IMHO points 2 and 3 must be swapped (see section 1.4 Dependencies for DS4000 Firmware).

    > Whats with the VMWare ESX Servers?

    What do you meant?

    Regards,
    Alexander Novikov
    Russia, Moscow
    > > So I should update the connected Windows Servers?
    >
    > IMHO points 2 and 3 must be swapped (see section 1.4 Dependencies for DS4000 Firmware).

    Taken from the Storage Manager readme:

    4. Upgrade the DS4000/DS5000 storage subsystem firmware in the following order -
    1. Controller firmware and NVSRAM
    2. ESM firmware
    3. Drive firmware
    Refer to the IBM System Storage DS Storage Manager Version 10
    Installation and Host Support Guide or the online help in the Subsystem
    Management window for detailed instructions.
  • Oliver_Wegener
    Oliver_Wegener
    18 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T10:04:00Z  
    Now I have a new problem:

    If I want to update the controller firmware with the utilitiy:

    Sep 11, 2008 12:01:30 PM DAMGroup-SAN pre-upgrade tests check hot spares
    In-use hot spare drive: One or more hot spares are in-use. This condition must be corrected before the firmware can be upgraded. Launch the storage manager software and use the recovery guru to fix the problem.

    What now?
  • Novikov_Alexander
    Novikov_Alexander
    1404 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T10:24:05Z  
    Now I have a new problem:

    If I want to update the controller firmware with the utilitiy:

    Sep 11, 2008 12:01:30 PM DAMGroup-SAN pre-upgrade tests check hot spares
    In-use hot spare drive: One or more hot spares are in-use. This condition must be corrected before the firmware can be upgraded. Launch the storage manager software and use the recovery guru to fix the problem.

    What now?
    Oliver,

    did you use DS4000 Storage Manager or Controller Firmware Upgrade Tool? In any case try another one of this tools.

    Regards,
    Alexander Novikov
    Russia, Moscow
  • Oliver_Wegener
    Oliver_Wegener
    18 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T10:47:39Z  
    Oliver,

    did you use DS4000 Storage Manager or Controller Firmware Upgrade Tool? In any case try another one of this tools.

    Regards,
    Alexander Novikov
    Russia, Moscow
    The message was from the Upgrade Tool.

    So i should use the Storage Manager? Isn't that unsafe?
  • Oliver_Wegener
    Oliver_Wegener
    18 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T11:02:34Z  
    The message was from the Upgrade Tool.

    So i should use the Storage Manager? Isn't that unsafe?
    Here a text from the Firmware redame:

    2. Download controller firmware and NVSRAM.

    IMPORTANT:
    The storage subsystem must be optimal and the upgrade must be done non-
    concurrent to any IO or configuration activity when upgrading from
    06.xx.xx.xx to 07.xx.xx.xx.

    Should I really do that with the problem I have?
  • Novikov_Alexander
    Novikov_Alexander
    1404 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T11:34:35Z  
    Here a text from the Firmware redame:

    2. Download controller firmware and NVSRAM.

    IMPORTANT:
    The storage subsystem must be optimal and the upgrade must be done non-
    concurrent to any IO or configuration activity when upgrading from
    06.xx.xx.xx to 07.xx.xx.xx.

    Should I really do that with the problem I have?
    Oliver,

    are you sure that current version of your DS4700 is 06.xx.xx.xx ?
    In provided profile I see version 07.10.23.00. Which DS4700 do you tried update?

    Regards,
    Alexander Novikov
    Russia, Moscow
  • Oliver_Wegener
    Oliver_Wegener
    18 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T11:37:09Z  
    Oliver,

    are you sure that current version of your DS4700 is 06.xx.xx.xx ?
    In provided profile I see version 07.10.23.00. Which DS4700 do you tried update?

    Regards,
    Alexander Novikov
    Russia, Moscow
    sorry I didn't read my own post properly I am at version 07.10.23.00

    So the upgrade shuould be no problem. Multipathing and two controllers are installed in the DS. So there should be no interruption to the hosts, right?
  • Novikov_Alexander
    Novikov_Alexander
    1404 Posts

    Re: Problem with replaced drive

    ‏2008-09-11T11:59:27Z  
    sorry I didn't read my own post properly I am at version 07.10.23.00

    So the upgrade shuould be no problem. Multipathing and two controllers are installed in the DS. So there should be no interruption to the hosts, right?
    Oliver,

    > Multipathing and two controllers are installed in the DS. So there should be no interruption to the hosts, right?

    If multipathing correctly configured (both HW and SW parts) IMHO upgrade DS4700 firmware would be transparent for hosts.

    Regards,
    Alexander Novikov
    Russia, Moscow
  • Oliver_Wegener
    Oliver_Wegener
    18 Posts

    Re: Problem with replaced drive

    ‏2008-09-12T06:54:04Z  
    Oliver,

    > Multipathing and two controllers are installed in the DS. So there should be no interruption to the hosts, right?

    If multipathing correctly configured (both HW and SW parts) IMHO upgrade DS4700 firmware would be transparent for hosts.

    Regards,
    Alexander Novikov
    Russia, Moscow
    Now my first problem is fixed!
    BUT
    Now I have another problem:
    Individual Drive - Degraded Path

    But to start from the beginning what I've done:

    I tried to update the firmware via Upgrad Tool and Storage Manager. Both had reported that they could not update because of a problem with the DS4700.
    Then I removed the replaced drive and put it back into the slot. The copyback operation didn't start but:
    The Storage Manager reported that the system is in optimal state (Recovery Guru removed the HotSpare message). The management interface showed still the same (HotSpare in use)
    Due to the "optimal" state of the array I decided to update the firmware. The Update Tool reports the "Hotpare in use problem" but the Storage Manager does not (really weird). So I started the firmware update from the Storage Manager. Everything went fine and both controllers were updated.
    Right after the update the copyback process started (great).

    After some hours everything was back in optimal state (now the real optimal state). I decided to make the ESM update for the EXP 810 -> no problems.

    Then I made the mistake:

    I wanted to update the drives. I had 6 unused drives so I selected them all for firmware update (I forgot the readme that I must choose max 4 drives). The update failed with "failed-partial". After that the six drives were marked with a red cross. I "revived" the drives with the "troubleshoothing menu". The drives went online.
    Then I started the firmware update again with less drives and eveythin went ok.

    Now only one problem remains:

    I have 6 times following entry in the Recovery Guru:
    Individual Drive - Degraded Path

    What do I have to do now? I've attached a actual support package

    thx
  • Oliver_Wegener
    Oliver_Wegener
    18 Posts

    Re: Problem with replaced drive

    ‏2008-09-12T06:58:59Z  
    Now my first problem is fixed!
    BUT
    Now I have another problem:
    Individual Drive - Degraded Path

    But to start from the beginning what I've done:

    I tried to update the firmware via Upgrad Tool and Storage Manager. Both had reported that they could not update because of a problem with the DS4700.
    Then I removed the replaced drive and put it back into the slot. The copyback operation didn't start but:
    The Storage Manager reported that the system is in optimal state (Recovery Guru removed the HotSpare message). The management interface showed still the same (HotSpare in use)
    Due to the "optimal" state of the array I decided to update the firmware. The Update Tool reports the "Hotpare in use problem" but the Storage Manager does not (really weird). So I started the firmware update from the Storage Manager. Everything went fine and both controllers were updated.
    Right after the update the copyback process started (great).

    After some hours everything was back in optimal state (now the real optimal state). I decided to make the ESM update for the EXP 810 -> no problems.

    Then I made the mistake:

    I wanted to update the drives. I had 6 unused drives so I selected them all for firmware update (I forgot the readme that I must choose max 4 drives). The update failed with "failed-partial". After that the six drives were marked with a red cross. I "revived" the drives with the "troubleshoothing menu". The drives went online.
    Then I started the firmware update again with less drives and eveythin went ok.

    Now only one problem remains:

    I have 6 times following entry in the Recovery Guru:
    Individual Drive - Degraded Path

    What do I have to do now? I've attached a actual support package

    thx
    this is making my wired:

    right now I checked the Recovery Guru again:

    The errors ar gone, weired!

    Does the DS automaticly fix the problem after the revive?