Create IBM Storage Scale Network Shared Disks (NSD)

• Storage Pools
  • System pool contains all of the metadata disks and does not have FPO behavior enabled. The system pool should have a smaller block size than the data pool for performance reasons. If you choose to use dataAndMetadata disks in the system pool, you must set the system pool block size to be the same as the data pool block size as both the pools can have data. For the dataAndMetadata system pool, the block size 1M is recommended.
  • Data pool contains all of the data disks and has FPO behavior enabled by setting allowWriteAffinity=yes, writeAffinityDepth=1, and blockGroupFactor=128.

    The chunk size can be calculated as blockSize * blockGroupFactor. Similar to the HDFS recommendation, the IBM Storage Scale FPO recommendation is blockSize=2M * blockGroupFactor=64 for a chunk size of 128 MB

• NSD
  • Every local disk to be used by IBM Storage Scale must have a matching entry in the disk stanza file
  • The device must match the device path of the disk.
    Note: In the example, /dev/sda is not included because this is the OS disk.

    If MapReduce intermediate and temporary data is stored on etx3/ext4 disks instead of IBM Storage Scale, make sure those disks are not included in the disk file or IBM Storage Scale will format them and include them in the IBM Storage Scale cluster.

  • System pool disks:
    • Should have usage=metadataOnly. It is possible to use usage=dataAndMetadata if there is a reason to have data on the system pool disks. The block size of the dataAndMetadata system pool must be the same as the block size of a data pool in the file system.
    • failureGroup must be a single number if allowWriteAffinity is not enabled (specify allowWriteAffinity=no for system pool definition when doing mmcrnsd or mmcrfs) and it should be the same for all disks on the same node. If allowWriteAffinity is enabled for system pool, the failure group can be of format rack,position,node, for example, 2,0,1; or, it can take the traditional single-number failure group format also.
    • Even when allowWriteAffinity is enabled for system pool, the metadata does not follow data locality rules; these rules apply only to data placement
  • Data pool disks:
    • Must have usage=dataOnly.
    • failureGroup must be of the format [rack,position,node], where position is either 0 or 1 to represent top or bottom half of the rack. The sample environment does not have half racks, so the same position is used for all nodes. Especially, when position and node fields are ignored in the cluster, the failure group can be defined as a single number, [rack, -,-].

Example of NSD disk file created by using the mmcrnsd command:

%pool: pool=system  blockSize=256K layoutMap=cluster allowWriteAffinity=no
%pool: pool=datapool blockSize=2M layoutMap=cluster allowWriteAffinity=yes writeAffinityDepth=1
blockGroupFactor=256

# gpfstest9
%nsd: nsd=node9_meta_sdb  device=/dev/sdb servers=gpfstest9 usage=metadataOnly failureGroup=1 pool=system

%nsd: nsd=node9_data_sdf2  device=/dev/sdf servers=gpfstest9 usage=dataOnly failureGroup=1,0,1 pool=datapool
%nsd: nsd=node9_data_sdg2  device=/dev/sdg servers=gpfstest9 usage=dataOnly failureGroup=1,0,1 pool=datapool

#gpfstest10
%nsd: nsd=node10_meta_sda  device=/dev/sda servers=gpfstest10 usage=metadataOnly failureGroup=2 pool=system

%nsd: nsd=node10_data_sde2  device=/dev/sde servers=gpfstest10 usage=dataOnly failureGroup=2,0,1 pool=datapool
%nsd: nsd=node10_data_sdg2  device=/dev/sdg servers=gpfstest10 usage=dataOnly failureGroup=2,0,1 pool=datapool

#gpfstest11

%nsd: nsd=node11_meta_sdb  device=/dev/sdb servers=gpfstest11 usage=metadataOnly failureGroup=3 pool=system

%nsd: nsd=node11_data_sdf2  device=/dev/sdf servers=gpfstest11 usage=dataOnly failureGroup=3,0,1 pool=datapool
%nsd: nsd=node11_data_sdg2  device=/dev/sdg servers=gpfstest11 usage=dataOnly failureGroup=3,0,1 pool=datapool

If any disks are previously used by IBM Storage Scale, you must use the -v no flag to force IBM Storage Scale to use them again. Use the # mmcrnsd -F diskfile [-v no] command to create NSDs and use the mmlsnsd –m command to display the NSDs.