Planning for array configurations

When you plan your network, consideration must be given to the type of RAID configuration that you use. The system supports either a non-distributed array or a distributed array configuration.

Array

A nondistributed array can contain 2 - 16 drives; several arrays create the capacity for a pool. For redundancy, hot-spare drives are allocated to assume read or write operations if any of the other drives fail. The rest of the time, the spare drives are idle and do not process requests for the system. When a member drive fails in the array, the data can only be recovered onto the spare as fast as that drive can write the data. Because of this bottleneck, rebuilding the data can take many hours as the system tries to balance hosts and rebuild workload. As a result, the load on the remaining member drives can significantly increase. The latency of I/O to the rebuilding array is affected for this entire time. Because volume data is striped across MDisks, all volumes are affected during the time it takes to rebuild the drive.

Distributed array

Distributed array configurations create large-scale internal MDisks. There are different types of distributed arrays. Distributed RAID 5 can contain as few as 4 drives initially, while distributed RAID 6 can contain as few as 6 drives initially. They both can be expanded up to and contain a maximum of 128 drives. Distributed RAID 1, on the other hand, can only contain between 2 - 6 drives initially, and can be expanded up to 16 drives of the same capacity.

The concept of distributed RAID 5 and 6 is to distribute an array with width W across a set of X drives. For example, you might have a 2+P RAID 5 array that is distributed across a set of 40 drives. The array type and width define the level of redundancy. In the previous example, a 33% capacity overhead for parity exists. If an array stride needs to be rebuilt, two component strips must be read to rebuild the data for the third component. The set size defines how many drives are used by the distributed array. Performance and usable capacity scale according to the number of drives in the set. The other key feature of a distributed array is that instead of having a hot spare, the set includes rebuild area strips that are also distributed across the set of drives. The data and rebuild areas are distributed such that if one drive in the set fails redundancy can be restored by rebuilding data on to the spare rebuild areas at a rate much greater than the rate of a single component. Within the FlashSystem 5000 family, this rebuild in place process is only supported in IBM® FlashSystem 5015 and IBM FlashSystem 5035 models.
Note: When you determine the array configuration for your system, plan to create arrays of 256 KiB strip size only.

Distributed arrays are used to create large-scale internal managed disks. They can manage 2 - 128 drives and contain their own rebuild areas, when available, to accomplish error recovery when drives fail. As a result, rebuild times are dramatically reduced, which decreases the volumes' exposure to the extra load of recovering redundancy. Because the capacity of these managed disks is potentially so great, when they are configured in the system the overall limits change to allow them to be virtualized. For every distributed array, the space for 16 MDisk extent allocations is reserved. Therefore, 15 other MDisk identities are removed from the overall pool of 4096. Distributed arrays also aim to provide a uniform performance level. A distributed array can contain multiple drive classes if the drives are similar (for example, the drives have the same attributes, but the capacities are larger) to achieve this performance. All the drives in a distributed array must come from the same I/O group to maintain a simple configuration model.

A distributed array has the following key benefits:
  • Quicker rebuild times with less impact to host I/O.
  • More user flexibility in defining how many drives are used by an array (for example, a user can create 9+P arrays with 24 drives without having four drives left unused).
  • Rebuild areas means that no idle drives exist in the system, and thus performance improves slightly.
  • Degraded storage arrays with only two member drives use the rebuild-in-place process, copying the data directly back into a replaced member drive.
    Note: Within the FlashSystem 5000 family, only IBM FlashSystem 5015 and IBM FlashSystem 5035 support distributed RAID 1.

One disadvantage of a distributed array is that the array redundancy is covering a greater number of components. Therefore, mean time between failure (MTBF) is reduced. Quicker rebuild times improve MTBF; however, limits to how widely distributed an array can be before the MTBF becomes unacceptable remain.

Distributed array expansion

Distributed array expansion allows the conversion of a small, not-very-distributed array into a larger distributed array while it preserves volume configuration and restriping for optimal performance. Expansion offers the option of getting better rebuild performance with an existing configuration without the migration steps that might require excess capacity. Expanding a distributed array is preferable over creating a new small array.

Expansion can increase the capacity of an array, but it cannot change the basic parameter of stripe width. When you plan a distributed array configuration, it is necessary to plan for future array requirements. Additionally, a distributed array that might fit within the extent limit (16*128 K extents) at a particular extent size might not fit if you expand it over time. Planning your extent size for the future is also important.

Expansion also benefits NVMe arrays for the same reasons. However, thin provisioned (compressing) NVMe drives add an extra layer of complexity when you calculate the available capacity in the array during an expansion. When you plan for the possible expansion of thin provisioned NVMe arrays, the drives must be the same physical and logical size. When you expand a thin provisioned NVMe array, the usable capacity is not immediately available, and the availability of new usable capacity does not track with logical expansion progress. The expansion process monitors usable capacity usage and analyzes the changes caused by the actions it takes during data restriping. This information is used to release the correct amount of usable capacity as it becomes available.