IBM Storage Scale Erasure Code Edition installation overview

The installation of IBM Storage Scale Erasure Code Edition by using the installation toolkit occurs in these phases.

Phase 1: Network and hardware precheck
  1. Download or clone the following ece_os_readiness, ece_network_readiness, and ece_storage_readiness precheck tools on one of the nodes that are planned for your IBM Storage Scale Erasure Code Edition storage configuration.
  2. Run the ece_os_readiness precheck tool on at least one of your IBM Storage Scale Erasure Code Edition storage nodes for each recovery group. Review the README.md file carefully for prerequisites and execution procedures.
  3. Run the ece_network_readiness and ece_storage_readiness precheck tools that include each IBM Storage Scale Erasure Code Edition storage node. Review the README.md file carefully for prerequisites and execution procedures.
Phase 2: Cluster definition
By using the ./spectrumscale command, the following steps are done:
  1. Installer node is defined by the user.
  2. Setup type is specified as ece by the user.
  3. Scale-out nodes and other node designations are done by the user.
    Other types of nodes that can be designated include protocol, GUI, call home, and file audit logging. If you are planning to use GUI, call home, performance monitoring, or file audit logging, you must add a client node for each of these functions.
    Note: When you are adding a node in an existing cluster, the installation toolkit adds only the node in the existing cluster with the client or the server license. You must use the mmvdisk command to manually add the node into the existing node class.
  4. Recovery group is defined by the user.
    Note: Recovery group definition can be done after the first installation run is done in which the package is installed and the cluster is created. With the package installed, the user can use the IBM Storage Scale Erasure Code Edition slot-mapping tool to create disk slot location configurations on IBM Storage Scale Erasure Code Edition storage servers. Then, the user can define the recovery group and run installation phase again to create the recovery group.
  5. Vdisk set is defined by the user. [Vdisk set definition can be done after the installation phase]
    Note:
    • The installation toolkit can be used to add new nodes to an existing IBM Storage Scale Erasure Code Edition cluster. But, the installation toolkit cannot be used for adding the new node into the existing recovery group. Use the mmvdisk command instead for adding new nodes into the existing recovery group.
    • The installation toolkit cannot accept multiple recovery groups as an argument while you define a vdisk set. If you want to specify more than one recovery group with the vdisk set, use the mmvdisk command after the installation phase is completed.
    • Depending on the cluster configuration, there might be cases where the settings for pagepool, vdisk space, or map memory of the recovery group servers need to be changed to fulfill the requirements for the vdisk set definition. The installation toolkit cannot be used to configure the recovery group servers. Instead, use the mmvdisk command to change the recovery group server configuration after the recovery groups are created.
  6. File system is defined by the user. [File system definition can be done after the installation phase]
Phase 3: Installation
This phase starts upon issuing the ./spectrumscale install command.
  1. IBM Storage Scale Erasure Code Edition packages that include IBM Storage Scale Erasure Code Edition license package are installed.
  2. IBM Storage Scale Erasure Code Edition cluster is created.
  3. Quorum and manager nodes are configured.
  4. Server and client licenses are applied.
  5. Node class is created.
  6. Recovery group is created.
  7. Vdisk sets are created.
  8. File systems are created.
Note: During the installation, support packages are also installed. These support packages include supported disk topologies and starting udev rules for each node. There is a rule file that is placed here: /etc/udev/rules.d/99-ibm-scaleout.rules. These rules have these settings and they are meant to be a good starting point for a typical hardware configuration. You might need to adjust these settings for your hardware configuration:

#
# IBM Storage Scale RAID (GNR) block device attributes for
# Erasure Code Edition (ECE) storage-rich servers.
#
# These are least common denominator settings. It is likely
# that specific installations can increase especially the
# max_sectors_kb for GNR pdisks.
#
# After initial ECE installation and after any change to the
# contents of these rules, run
#       udevadm trigger --subsystem-match=block
# and inspect /var/log/messages for unexpected udev entries.
# Subsequent reboots and block device replacement will
# automatically invoke these rules as "add|change" events.
#
# -------------------------------------------------------------------------
#
# Identify the boot SCSI disk by the presence of a SWAP partition.
# Set boot disk nr_requests and queue_depth to reasonable values.
#
ACTION=="add|change", SUBSYSTEM=="block", 
KERNEL=="sd*[^0-9]", PROGRAM="/usr/bin/lsblk -rno 
FSTYPE,MOUNTPOINT,NAME /dev/%k", RESULT=="*SWAP*", 
ATTR{queue/nr_requests}="128", ATTR{device/queue_depth}="64"
#
# Identify eligible GNR SCSI pdisks by the absence of a SWAP partition.
# Set preferred GNR attributes. The only attribute that should possibly
# be changed is max_sectors_kb, up to a value of 8192, depending on
# what the SCSI driver and disks support.
#
ACTION=="add|change", SUBSYSTEM=="block", 
KERNEL=="sd*[^0-9]", PROGRAM="/usr/bin/lsblk -rno 
FSTYPE,MOUNTPOINT,NAME /dev/%k", 
RESULT!="*SWAP*", ATTR{queue/scheduler}="deadline", 
ATTR{queue/nr_requests}="256", ATTR{device/queue_depth}="31", 
ATTR{queue/max_sectors_kb}="1024", ATTR{queue/read_ahead_kb}="0", 
ATTR{queue/rq_affinity}="2"
#
# Identify eligible GNR NVMe pdisks by the absence of a MOUNTPOINT.
# Set preferred GNR attributes. The only attribute that should possibly
# be changed is max_sectors_kb, up to a value of 8192, depending on
# what the NVMe driver and devices support.
#
ACTION=="add|change", SUBSYSTEM=="block", 
KERNEL=="nvme*", KERNEL!="nvme*p[0-9]", 
PROGRAM="/usr/bin/lsblk -rno 
FSTYPE,MOUNTPOINT,NAME /dev/%k", RESULT!="*/*", 
ATTR{queue/scheduler}="none", ATTR{queue/nr_requests}="256", 
ATTR{queue/max_sectors_kb}="128", 
ATTR{queue/read_ahead_kb}="0", 
ATTR{queue/rq_affinity}="2"
Note: If you are planning to deploy protocols in the IBM Storage Scale Erasure Code Edition cluster, you must define a CES shared root file system before you initiate the installation toolkit deployment phase by using the following command.
./spectrumscale config protocols -f FileSystem -m MountPoint

Quorum or manager node rules in IBM Storage Scale Erasure Code Edition

  • In case of a single recovery group, the following quorum node rules apply.
    • When the number of scale-out nodes is 4, the number of quorum nodes is set to 3.
    • When the number of scale-out nodes is 5 or 6, the number of quorum nodes is set to 5.
    • When the number of scale-out nodes is 7 or more, the number of quorum nodes is set to 7.
  • If the number of recovery groups is more than 1 and less than or equal to 7, 7 quorum nodes are distributed across recovery groups in a round robin manner.
  • If the number of recovery groups is more than 7, 7 recovery groups are selected as quorum holders.
  • If there is no recovery group or quorum node that is defined in the cluster configuration, the installation toolkit displays the following message.
    "You have not defined any recovery group in the cluster configuration. 
Installer will automatically define the quorum configuration. Do you want to continue"

    If you specify yes then the quorum nodes are distributed according to the single recovery group rule.

  • If you are adding a new recovery group in an existing cluster or if you want to add a new node into the existing node class, the existing quorum configuration is not modified by the installation toolkit.
  • For an existing cluster, if you want to have quorum on a different node or a different recovery group then you must use an IBM Storage Scale command such as mmchnode to change this configuration.