• 2 replies
  • Latest Post - ‏2013-10-03T23:39:40Z by yuri
87 Posts

Pinned topic NSD ordering after mmrpldisk

‏2013-10-03T18:01:03Z | gpfs mmrlpdisk

What happens to the order of NSDs in the file system after a mmrpldisk?  None of the examples in the manuals show the output of "mmlsnsd" before and after the mmrpldisk command.


original NSD order for GPFS file system "fs1".  Assume the "A" NSDs are on storage array A, and the "B" NSDs are on storage array B

nsd_A1, nsd_B1, nsd_A2, nsd_B2

This configuration is well balanced, toggling between storage array A and storage array B for each sequential NSD.

Replace nsd_B1 with new nsd_B3

mmrpldisk fs1 nsd_B1 -F stanza_file_for_nsd_B3

mmfslsd fs1

What is the NSD ordering after the replace?  Is it:

nsd_A1, nsd_B3, nsd_A2, nsd_B2                  (the new NSD replaces the original NSD in the ordered list)


nsd_A1, nsd_A2, nsd_B2, nsd_B3                (equivalent to order obtained by mmdeldisk followed by mmadddisk)

The documentation indicates that mmrpldisk is logically similar to a mmdeldisk followed by mmadddisk, but more efficient because original NSD's contents are copied to the new NSD and file system balance is maintained (assuming the new NSD is the same size, data/metadata  Usage type, and storage pool assignment).  As such, a file system rebalance is not needed.

However, the documentation is unclear if the ORDERING of the NSDs in the file system is maintained, which can effect IO balance across storage arrays.  In the example above, the first case, the new NSD replaces the original NSD, while keeping the NSD ordering. The file system is still well balanced across storage arrays, toggling between storage array A and storage array B (as the original configuration)

In the second case, the original NSD was effectively deleted from the NSD list, and the new NSD was appended to the tail of the list.  In the example, now the NSD ordering is NOT well balanced across two storage arrays, with two adjacent IOs likely being sent to the different LUNs, but on the same storage array, resulting in higher congestion on the array, and a micro-clumping IO behavior.  Yes, this is a fine point, but it can and does make a difference.

So, in summary, does the mmrpldisk command maintain the NSD ordering?  If not, the administrator may want to make additional adjustments.

Thank you for your help.

Dave B


Updated on 2013-10-03T18:01:49Z at 2013-10-03T18:01:49Z by db808
  • esj
    104 Posts

    Re: NSD ordering after mmrpldisk


    mmrpldisk will preserve the original disk ordering.


  • yuri
    305 Posts

    Re: NSD ordering after mmrpldisk


    GPFS tries to maintain dense packing of disks, and always reuses empty slots in the disk list before adding disks at the end.  So if a disk in slot 1 is deleted, the next mmadddisk will put a new disk in slot 1.  The same applies to mmrpldisk.