Suboptimal performance due to improper mapping of the file system NSDs to the NSD servers

The NSDs in a file system need to be optimally assigned to the NSD servers so that the client I/O is equally distributed across all the NSD servers. For example, consider a file system with 10 NSDs and 2 NSD servers. The NSD-to-server mapping must be done in such a way that each server acts as the primary server for 5 of the NSD in the file system. If the NSD-to-server mapping is unbalanced, it can result in hot spots in one or more of the NSD servers. Presence of hot spots within a system can cause performance degradation.

Problem identification

Issue the mmlsnsd command, and verify that the primary NSD server allocated to a file system is evenly distributed.

Note: The primary server is the first server listed under the NSD server column for a particular file system.

On the NSD client, issue the mmlsdisk <fs> -m command to ensure that the NSD client I/O is distributed evenly across all the NSD servers.

In the following sample output, all the NSDs are assigned to the same primary server c80f1m5n03ib0.

# mmlsnsd

File system   Disk name    NSD servers
---------------------------------------------------------------------------
 gpfs2         Perf2a_NSD01 c80f1m5n03ib0,c80f1m5n02ib0
 gpfs2         Perf2a_NSD02 c80f1m5n03ib0,c80f1m5n02ib0
 gpfs2         Perf2a_NSD03 c80f1m5n03ib0,c80f1m5n02ib0
 gpfs2         Perf2a_NSD04 c80f1m5n03ib0,c80f1m5n02ib0
 gpfs2         Perf2a_NSD05 c80f1m5n03ib0,c80f1m5n02ib0
 gpfs2         Perf2a_NSD06 c80f1m5n03ib0,c80f1m5n02ib0
 gpfs2         Perf2a_NSD07 c80f1m5n03ib0,c80f1m5n02ib0
 gpfs2         Perf2a_NSD08 c80f1m5n03ib0,c80f1m5n02ib0
 gpfs2         Perf2a_NSD09 c80f1m5n03ib0,c80f1m5n02ib0
 gpfs2         Perf2a_NSD10 c80f1m5n03ib0,c80f1m5n02ib0

In this case, all the NSD client I/O for the gpfs2 file system are processed by the single NSD server c80f1m5n03ib0, instead of being equally distributed across both the NSD servers c80f1m5n02ib0 and c80f1m5n03ib0. This can be verified by issuing the mmlsdisk <fs> -m command on the NSD client, as shown in the following sample output:

# mmlsdisk gpfs2 -m

Disk name     IO performed on node     Device             Availability
------------  -----------------------  -----------------  ------------
Perf2a_NSD01  c80f1m5n03ib0            -                  up
Perf2a_NSD02  c80f1m5n03ib0            -                  up
Perf2a_NSD03  c80f1m5n03ib0            -                  up
Perf2a_NSD04  c80f1m5n03ib0            -                  up
Perf2a_NSD05  c80f1m5n03ib0            -                  up
Perf2a_NSD06  c80f1m5n03ib0            -                  up
Perf2a_NSD07  c80f1m5n03ib0            -                  up
Perf2a_NSD08  c80f1m5n03ib0            -                  up
Perf2a_NSD09  c80f1m5n03ib0            -                  up
Perf2a_NSD10  c80f1m5n03ib0            -                  up

Problem resolution and verification

If the NSD-to-primary mapping is unbalanced, issue the mmchnsd command to balance the NSD distribution across the NSD servers. Issue the mmlsnsd command or the mmlsdisk <fs> -m command on the NSD client to ensure that the NSD distribution across the servers is balanced.

In the following sample output, there are 10 NSDs in the gpfs2 file system. The NSDs are evenly distributed between the two servers, such that both servers, c80f1m5n03ib0 and c80f1m5n02ib0 act as primary servers for 5NSDs each.

# mmlsnsd

File system   Disk name    NSD servers
---------------------------------------------------------------------------
 gpfs2         Perf2a_NSD01 c80f1m5n03ib0,c80f1m5n02ib0
 gpfs2         Perf2a_NSD02 c80f1m5n02ib0,c80f1m5n03ib0
 gpfs2         Perf2a_NSD03 c80f1m5n03ib0,c80f1m5n02ib0
 gpfs2         Perf2a_NSD04 c80f1m5n02ib0,c80f1m5n03ib0
 gpfs2         Perf2a_NSD05 c80f1m5n03ib0,c80f1m5n02ib0
 gpfs2         Perf2a_NSD06 c80f1m5n02ib0,c80f1m5n03ib0
 gpfs2         Perf2a_NSD07 c80f1m5n03ib0,c80f1m5n02ib0
 gpfs2         Perf2a_NSD08 c80f1m5n02ib0,c80f1m5n03ib0
 gpfs2         Perf2a_NSD09 c80f1m5n03ib0,c80f1m5n02ib0
 gpfs2         Perf2a_NSD10 c80f1m5n02ib0,c80f1m5n03ib0

The NSD client I/O is also evenly distributed across the two NSD servers, as seen in the following sample output:

# mmlsdisk gpfs2 -m

Disk name     IO performed on node     Device             Availability
------------  -----------------------  -----------------  ------------
Perf2a_NSD01  c80f1m5n03ib0            -                  up
Perf2a_NSD02  c80f1m5n02ib0            -                  up
Perf2a_NSD03  c80f1m5n03ib0            -                  up
Perf2a_NSD04  c80f1m5n02ib0            -                  up
Perf2a_NSD05  c80f1m5n03ib0            -                  up
Perf2a_NSD06  c80f1m5n02ib0            -                  up
Perf2a_NSD07  c80f1m5n03ib0            -                  up
Perf2a_NSD08  c80f1m5n02ib0            -                  up
Perf2a_NSD09  c80f1m5n03ib0            -                  up
Perf2a_NSD10  c80f1m5n02ib0            -                  up