Installing a typical IBM Storage Scale Erasure Code Edition cluster of synchronous mirroring by using GPFS replication

Use the following steps to install a typical IBM Storage Scale Erasure Code Edition cluster of synchronous mirroring by using GPFS replication.

In the following example, there are three geographically separated sites: Site A, Site B, and Site C.

Site A has a set of storage nodes (nodeA01, nodeA02, nodeA03, and nodeA04) for creating IBM Storage Scale Erasure Code Edition recovery group A.
Site B has another set of storage nodes (nodeB01, nodeB02, nodeB03, and nodeB04) for creating IBM Storage Scale Erasure Code Edition recovery group B.
Site C has one tiebreaker node (nodeC) as the quorum node and a general NSD disk to hold file system quorum descriptor.

Download IBM Storage Scale Erasure Code Edition self-extracting package from the IBM Storage Scale page on the Fix Central.
Extract the installation package.
```
./Storage_Scale_Erasure_Code-<release_ver>-<os_arch>-Linux-install --textonly
```
The installation toolkit gets extracted to the /usr/lpp/mmfs/<release_ver>/ansible-toolkit/ directory.
Change the working directory to where the installation toolkit is extracted.
```
cd /usr/lpp/mmfs/<release_ver>/ansible-toolkit/
```
Specify the installer node and the setup type in the cluster definition file.
The setup type must be "ece" for IBM Storage Scale Erasure Code Edition.
```
./spectrumscale setup -s InstallerNodeIP -st ece
```
Add nodeC as the quorum node for IBM Storage Scale Erasure Code Edition in the cluster definition file.
```
./spectrumscale node add nodeC -q -g -a
```
Add nodeA01, nodeA02, nodeB01, and nodeB02 as the scale-out and quorum nodes for IBM Storage Scale Erasure Code Edition in the cluster definition file.
```
./spectrumscale node add NodeName -so -q
```
Add nodeA03, nodeA04, nodeB03, and nodeB04 as the scale-out node for IBM Storage Scale Erasure Code Edition in the cluster definition file.
```
./spectrumscale node add NodeName -so
```

Check the list of nodes that are specified in the cluster definition file and the respective node designations.

./spectrumscale node list

A sample output is as follows:

[ INFO  ] List of nodes in current configuration:
[ INFO  ] [Installer Node]
[ INFO  ] 192.168.0.1
[ INFO  ]  
[ INFO  ] [Cluster Details]
[ INFO  ] No cluster name configured
[ INFO  ] Setup Type: Erasure Code Edition
[ INFO  ]  
[ INFO  ] [Extended Features]
[ INFO  ] File Audit Logging : Disabled
[ INFO  ] Watch Folder    : Disabled
[ INFO  ] Management GUI    : Enabled
[ INFO  ] Performance Monitoring    : Enabled
[ INFO  ] Callhome    : Disabled
[ INFO  ]  
[ INFO  ] [Extended Features]
[ INFO  ] File Audit logging     : Enabled
[ INFO  ] Watch folder           : Disabled
[ INFO  ] Management GUI         : Disabled
[ INFO  ] Performance Monitoring : Disabled
[ INFO  ] Callhome               : Enabled
[ INFO  ]  
[ INFO  ] GPFS                                 Admin  Quorum  Manager   NSD   Protocol   GUI   Callhome Scale-out   OS   Arch
[ INFO  ] Node                                  Node   Node     Node   Server   Node    Server  Server    Node
[ INFO  ] nodeC                                  X       X                                X                      rhel7  x86_64
[ INFO  ] nodeA01                                        X                                                 X     rhel7  x86_64
[ INFO  ] nodeA01                                        X                                                 X     rhel7  x86_64
[ INFO  ] nodeA03                                                                                          X     rhel7  x86_64
[ INFO  ] nodeA04                                                                                          X     rhel7  x86_64
[ INFO  ] nodeB01                                        X                                                 X     rhel7  x86_64
[ INFO  ] nodeB02                                        X                                                 X     rhel7  x86_64
[ INFO  ] nodeB03                                                                                          X     rhel7  x86_64
[ INFO  ] nodeB04                                                                                          X     rhel7  x86_64
[ INFO  ]  
[ INFO  ] [Export IP address]
[ INFO  ] No export IP addresses configured

Define the recovery group A of site A for IBM Storage Scale Erasure Code Edition in the cluster definition file.
```
./spectrumscale recoverygroup define -rg rgA -nc ncA -N nodeA01,nodeA02,nodeA03,nodeA04
```
Define the recovery group B of site B for IBM Storage Scale Erasure Code Edition in the cluster definition file.
```
./spectrumscale recoverygroup define -rg rgB -nc ncB -N nodeB01,nodeB02,nodeB03,nodeB04
```
Perform environment prechecks before you issue the installation toolkit installation command.
```
./spectrumscale install -pr
```
Perform the installation toolkit installation procedure.
```
./spectrumscale install
```
Check the state of the cluster and of the recovery group. All nodes, rgA, and rgB must be in the active state.
```
mmgetstate -a
mmvdisk rg list
```

Create vdisksets for the file system.

Two declustered arrays of SSD and HDD disks are used in this example.

mmvdisk rg list --da

A sample output is as follows:

 declustered       needs                  capacity              pdisks
recovery group     array     service  type  total raw  free raw  free%  total  spare  background task
--------------  -----------  -------  ----  ---------  --------  -----  -----  -----  ---------------
rgA             DA1          no       HDD      24 TiB    24 TiB   100%     48      3  inactive 0%
rgA             DA2          no       SSD    5074 GiB  5074 GiB   100%     16      2  scrub 4%
rgB             DA1          no       HDD      24 TiB    24 TiB   100%     48      3  inactive 0%
rgB             DA2          no       SSD    5074 GiB  5074 GiB   100%     16      2  scrub 4%

Define the vdiskset for declustered array, SSD for file system metadata, and HDD for file system data, by using the following commands:

mmvdisk vdiskset define --vs SSD01A --rg rgA --code 8+2p --bs 2M --da DA2 --nsd-usage metadataonly --sp system --set-size 90%
mmvdisk vdiskset define --vs SSD01B --copy SSD01A --rg rgB
mmvdisk vdiskset define --vs HDD01A --rg rgA --code 8+2p --bs 8M --da DA1 --nsd-usage dataonly --sp data --set-size 90%
mmvdisk vdiskset define --vs HDD01B --copy HDD01A --rg rgB

Create the vdisks.
```
mmvdisk vdiskset create --vs all
```

Create a file system with two replicas of metadata and data block.
vdisk in rgA of node class ncA has failure group number 1, vdisk in rgB of node class ncB has failure group number 2.
```
mmvdisk filesystem create --fs gpfs1 --vs SSD01A,SSD01B,HDD01A,HDD01B --fg ncA=1,ncB=2 --mmcrfs -T /gpfs1 -r 2 -m 2
```
Add disk of nodeC to the file system as the descOnly disk.
1. Create an NSD stanza file for the disk, it has the failure group number 3.
```
# cat diskC.stanza
%nsd:
 nsd=diskDescNsd
 device=/dev/sdo
 servers=NodeC
 usage=descOnly
 failureGroup=3
 pool=system
```
2. Create the NSD.
```
mmcrnsd -F diskC.stanza
```
3. Add this NSD into the file system.
```
mmadddisk gpfs1 -F diskC.stanza
```
To avoid unexpected mounts on nodeC, create the following empty file on nodeC.
```
touch /var/mmfs/etc/ignoreAnyMount.gpfs1
```
Mount the file system.
```
mmmount all -a
```

Change the configurations as follows:

# mmchconfig nsdRAIDMaxRecoveryRetries=4 -i
# mmchconfig nsdRAIDReadRGDescriptorTimeout=60 -i