IBM Spectrum Scale Erasure Code Edition installation overview

The installation of IBM Spectrum Scale Erasure Code Edition by using the installation toolkit occurs in these phases.

Phase 1: Cluster definition
  1. Installer node is defined by the user.
  2. Setup type is specified as ece by the user.
  3. Scale-out nodes and other node designations are done by the user.
    Other types of nodes that can be designated include protocol, GUI, call home, and file audit logging. If you are planning to use GUI, call home, performance monitoring, or file audit logging, you must add a client node for each of these functions.
    Note: When adding a node in an existing cluster, the installation toolkit only adds the node in the existing cluster with the client or the server license. You must use the mmvdisk command to manually add the node into the existing node class.
  4. Recovery group is defined by the user.
  5. Vdisk set is defined by the user. [Vdisk set definition can be done after the installation phase]
    Note:
    • The installation toolkit cannot accept multiple recovery groups as an argument while defining a vdisk set. If you want to specify more than one recovery group with the vdisk set, use the mmvdisk command after the installation phase is completed.
    • The installation toolkit does not support declustered array as an argument while defining the vdisk set. If you want to specify one or more declustered arrays with the vdisk set, use the mmvdisk command after the installation phase is completed.
  6. File system is defined by the user. [File system definition can be done after the installation phase]
    Note: The installation toolkit cannot accept multiple vdisk sets as an argument while defining the file system. If you want to specify multiple vdisk sets with the file system, use the mmvdisk command after the installation phase is completed.
Phase 2: Installation
This phase starts upon issuing the ./spectrumscale install command.
  1. IBM Spectrum Scale Erasure Code Edition packages including the IBM Spectrum Scale Erasure Code Edition license package are installed.
  2. IBM Spectrum Scale Erasure Code Edition cluster is created.
  3. Quorum and manager nodes are configured.
  4. Server and client licenses are applied.
  5. Node class is created.
  6. Recovery group is created.
Note: During the installation, support packages are also installed. These support packages include supported disk topologies and starting udev rules for each node. There is a rule file placed here: /etc/udev/rules.d/99-ibm-scaleout.rules. These rules have these settings and they are meant to be a good starting point for a typical hardware configuration. You might need to adjust these for your hardware configuration:
#
# IBM Spectrum Scale RAID (GNR) block device attributes for
# Erasure Code Edition (ECE) storage-rich servers.
#
# These are least common denominator settings. It is likely
# that specific installations can increase especially the
# max_sectors_kb for GNR pdisks.
#
# After initial ECE installation and after any change to the
# contents of these rules, run
#       udevadm trigger --subsystem-match=block
# and inspect /var/log/messages for unexpected udev entries.
# Subsequent reboots and block device replacement will
# automatically invoke these rules as "add|change" events.
#
# -------------------------------------------------------------------------
#
# Identify the boot SCSI disk by the presence of a SWAP partition.
# Set boot disk nr_requests and queue_depth to reasonable values.
#
ACTION=="add|change", SUBSYSTEM=="block", 
KERNEL=="sd*[^0-9]", PROGRAM="/usr/bin/lsblk -rno 
FSTYPE,MOUNTPOINT,NAME /dev/%k", RESULT=="*SWAP*", 
ATTR{queue/nr_requests}="128", ATTR{device/queue_depth}="64"
#
# Identify eligible GNR SCSI pdisks by the absence of a SWAP partition.
# Set preferred GNR attributes. The only attribute that should possibly
# be changed is max_sectors_kb, up to a value of 8192, depending on
# what the SCSI driver and disks support.
#
ACTION=="add|change", SUBSYSTEM=="block", 
KERNEL=="sd*[^0-9]", PROGRAM="/usr/bin/lsblk -rno 
FSTYPE,MOUNTPOINT,NAME /dev/%k", 
RESULT!="*SWAP*", ATTR{queue/scheduler}="deadline", 
ATTR{queue/nr_requests}="256", ATTR{device/queue_depth}="31", 
ATTR{queue/max_sectors_kb}="1024", ATTR{queue/read_ahead_kb}="0", 
ATTR{queue/rq_affinity}="2"
#
# Identify eligible GNR NVMe pdisks by the absence of a MOUNTPOINT.
# Set preferred GNR attributes. The only attribute that should possibly
# be changed is max_sectors_kb, up to a value of 8192, depending on
# what the NVMe driver and devices support.
#
ACTION=="add|change", SUBSYSTEM=="block", 
KERNEL=="nvme*", KERNEL!="nvme*p[0-9]", 
PROGRAM="/usr/bin/lsblk -rno 
FSTYPE,MOUNTPOINT,NAME /dev/%k", RESULT!="*/*", 
ATTR{queue/scheduler}="none", ATTR{queue/nr_requests}="256", 
ATTR{queue/max_sectors_kb}="128", 
ATTR{queue/read_ahead_kb}="0", 
ATTR{queue/rq_affinity}="2"
Note: If you are planning to deploy protocols in the IBM Spectrum Scale Erasure Code Edition cluster, you must define a CES shared root file system before initiating the installation toolkit deployment phase by using the following command.
./spectrumscale config protocols -f FileSystem -m MountPoint
Phase 3: Deployment
This phase starts upon issuing the ./spectrumscale deploy command.
  1. Vdisk sets are created.
  2. File systems are created.
  3. Protocols are deployed, if applicable.

Additional IBM Spectrum Scale configuration items

It is recommended to add the following configuration settings for improved performance:
  1. Set node class.
    NC=Erasure Code Edition node class
  2. Update tuning parameters for nodes in the IBM Spectrum Scale Erasure Code Edition node class.
    mmchconfig nsdMaxWorkerThreads=3842     -N $NC        
mmchconfig nsdMinWorkerThreads=3842     -N $NC         
mmchconfig nsdRAIDThreadsPerQueue=16   -N $NC     
mmchconfig nsdSmallThreadRatio=1 -N $NC

Quorum or manager node rules in IBM Spectrum Scale Erasure Code Edition

  • In case of a single recovery group, the following quorum node rules apply.
    • When the number of scale-out nodes is 4, the number of quorum nodes is set to 3.
    • When the number of scale-out nodes is 5 or 6, the number of quorum nodes is set to 5.
    • When the number of scale-out nodes is 7 or more, the number of quorum nodes is set to 7.
  • If the number of recovery groups is more than 1 and less than or equal to 7, 7 quorum nodes are distributed across recovery groups in a round robin manner.
  • If the number of recovery groups is more than 7, 7 recovery groups are selected as quorum holders.
  • If there is no recovery group or quorum node defined in the cluster configuration, the installation toolkit displays the following message.
    "You have not defined any recovery group in the cluster configuration. 
Installer will automatically define the quorum configuration. Do you want to continue"

    If you specify yes then quorum nodes are distributed according to the single recovery group rule.

  • If you are adding a new recovery group in an existing cluster or if you want to add a new node into the existing node class, the existing quorum configuration is not modified by the installation toolkit.
  • For an existing cluster, if you want to have quorum on a different node or a different recovery group then you must use an IBM Spectrum Scale command such as mmchnode to change this configuration.
  • Every scale-out node has the manager mode designation. Scale-out nodes in a recovery group are equivalent so any of them can pick up the cluster manager or the file system manager role.
Parent topic: Installing IBM Spectrum Scale Erasure Code Edition