SCSI ServeRAID incorrectly handles hard disk drive medium errors - Servers

Resolving The Problem

RETAIN tip: H186518

With RAID-1 or RAID-10, the system may stall for approximately a minute before recovering and continuing normal operations. In ServeRAID cluster configurations, this can cause an unexpected failover to the partner node.

With RAID-5 or RAID-50 under heavy stress, sluggish performance may occur, and in rare circumstances, may result in data loss.

The system may be any of the following IBM servers:

• xSeries 460, Type 8874, any model
• xSeries 460, Type 8872, any model
• xSeries 445, Type 8870, any model
• xSeries 440, Type 8687, any model
• xSeries 382, Type 8834, any model
• xSeries 380, Type 8683, any model
• xSeries 370, Type 8681, any model
• xSeries 366, Type 8863, any model
• xSeries 365, Type 8862, any model
• xSeries 365, Type 8861, any model
• xSeries 360, Type 8686, any model
• xSeries 350, Type 8682, any model
• xSeries 346, Type 1880, any model
• xSeries 346, Type 8840, any model
• xSeries 345, Type 8670, any model
• xSeries 343, Type 8827, any model
• xSeries 342, Type 8669, any model
• xSeries 340, Type 8656, any model
• xSeries 336, Type 8838, any model
• xSeries 336, Type 8837, any model
• xSeries 335, Type 8830, any model
• xSeries 335, Type 8676, any model
• xSeries 330, Type 8675, any model
• xSeries 330, Type 8674, any model
• xSeries 330, Type 8654, any model
• eServer 326, Type 8848, any model
• eServer 326m, Type 7969, any model
• eServer 325, Type 8835, any model
• xSeries 306, Type 8836, any model
• xSeries 306m, Type 8891, SAS model
• xSeries 306m, Type 8849, SAS model
• xSeries 305, Type 8673, any model
• xSeries 300, Type 8672, any model
• xSeries 260, Type 8865, any model
• xSeries 255, Type 8685, any model
• xSeries 250, Type 8665, any model
• xSeries 240, Type 8664, any model
• xSeries 236, Type 8841, any model
• xSeries 235, Type 8671, any model
• xSeries 232, Type 8668, any model
• xSeries 230, Type 8658, any model
• xSeries 226, Type 8488, any model
• xSeries 226, Type 8648, any model
• xSeries 225, Type 8649, any model
• xSeries 225, Type 8647, any model
• xSeries 220, Type 8646, any model
• xSeries 220, Type 8645, any model
• xSeries 206, Type 8482, any model
• xSeries 206m, Type 8485, SAS model
• xSeries 206m, Type 8490, SAS model

The system is configured with the following ServeRAID option(s):

• a Netfinity ServeRAID-4H Ultra160 SCSI Controller, Option part number 37L6889, replacement part number 37L6892 (controller), replacement part number 37L6902 (cache daughter card), replacement part number 37L6903 (cache battery)
• a Netfinity ServeRAID-4L Ultra160 SCSI Controller, Option part number 37L6091, replacement part number 09N9540
• a Netfinity ServeRAID-4M Ultra160 SCSI Controller, Option part number 37L6080, replacement part number 37L7258
• a Netfinity ServeRAID-4M Ultra160 SCSI Controller (Japan), Option part number 19K0565, replacement part number 00N9543 (Japan)
• a ServeRAID-4Lx Ultra160 SCSI Controller, Option part number 06P5740, replacement part number 06P5741
• a ServeRAID-4Mx Ultra160 SCSI Controller, Option part number 06P5736, replacement part number 06P5737
• a ServeRAID-5i Controller, Option part number 25P3492, replacement part number 02R0970 replaces 32P0016
• a ServeRAID-6M Controller (128MB Cache), Option part number 32P0033, replacement part number 02R0985
• a ServeRAID-6M Controller (256MB Cache), Option part number 13N2197, replacement part number 02R0998
• a ServeRAID-6i Controller, Option part number 71P8595, replacement part number 71P8627
• a ServeRAID-6i+ Controller, Option part number 13N2190, replacement part number 13N2195
• a ServeRAID-7k Controller, Option part number 39R8800 replaces 71P8642, field replacement unit (FRU) 71P8644
  

Apply firmware version 7.12.07 or higher to the SCSI ServeRAID controller.

This is currently available at the following URL:

MIGR-62540

If the controller firmware level is 7.12.04 or earlier, Synchronize each logical drive on a regular basis. Synchronizations correct all medium errors detected on all physical drives in the array. When there are constant changes are made to the data, consider synchronizing the data weekly or biweekly. If the data is static with minimal changes, consider synchronizing the data monthly.

Foreground synchronizations can be initiated two ways, by using the ServeRAID Manager GUI or using the IPSSEND command. The IPSSEND command can be used in a BAT or CMD file and then automated using almost any scheduling utility. This workaround is not necessary after applying the firmware 7.12.07 or higher.

Some SCSI ServeRAID controllers operating with 7.12.04 or earlier versions of firmware may incorrectly handle hard disk drive medium error recovery. This can lead to various symptoms depending on the RAID level configured. In mirrored logical drives e.g. RAID 1 or 10, it was determined that a variable used for linking rebuild commands during handling of medium errors was not initialized thus resulting in the controller getting stuck in looping error recovery for about a minute. When ServeRAID recovers, typically the medium error persists on the drive and a subsequent read may result in the same symptoms. In testing, no data loss was ever detected.

In parity logical drives e.g. RAID 5 or 50, medium error recovery is typically handled correctly until the cumulative latencies of the drives and firmware processing peak. As stress increases to the subsystem, it becomes more likely to incorrectly handle a hard disk drive medium error. Under heavy stress it is possible for an improperly handled medium error to trigger deep error recovery resulting in sluggish performance and in rare circumstances, data loss may occur.