ESS Quick Deployment Guide

This guide is for system administrators, installers, and programmers of IBM Spectrum Scale clusters who are experienced with the operating systems on which each IBM Spectrum Scale cluster is based.

ESS quick deployment sheet

This quick sheet lists concise sets of steps for the ESS deployment procedures. For more information, see respective procedures.
Quick deployment steps

Support matrix

Release OS Runs on Can upgrade or deploy
Start of changeESS 3500 6.1.8.3End of change Start of changeRed Hat® Enterprise Linux® 8.6 (x86_64)End of change Start of change
  • POWER9™ EMS
  • x86 EMS (BYOE)1 from 6.1.8.3
End of change		 Start of change
  • ESS 3500 nodes
  • POWER9 EMS
  • POWER9 protocol nodes
End of change
ESS 3200 Start of change6.1.8.3End of change Red Hat Enterprise Linux 8.6 (x86_64)
  • POWER9 EMS
  • ESS 3200 nodes
  • POWER9 EMS
  • POWER9 protocol nodes
ESS 3000 6.1.8.3 Red Hat Enterprise Linux 8.6 (x86_64)
  • POWER8® EMS
  • POWER9 EMS
  • ESS 3000 nodes
  • POWER8 EMS
  • POWER9 EMS
  • POWER8 protocol nodes
  • POWER9 protocol nodes
ESS 5000 Start of change6.1.8.3End of change Red Hat Enterprise Linux 8.6 (PPC64LE)
  • POWER9 EMS
  • ESS 5000 nodes
  • POWER9 EMS
  • POWER9 protocol nodes
ESS Legacy Start of change6.1.8.3End of change
  • Red Hat Enterprise Linux 8.6 (PPC64LE)
  • Red Hat Enterprise Linux 7.9 (PPC64LE)
  • POWER8 EMS
  • POWER9 EMS
  • ESS POWER8 I/O nodes (PPC64LE)
  • ESS POWER8 protocol nodes (PPC64LE)
  • ESS POWER9 protocol nodes (PPC64LE)*
  • POWER8 EMS
  • POWER9 EMS
1 x86 EMS (BYOE) can only upgrade or deploy the ESS 3500 node(s) and the VM image itself.

Network requirements

Figure 1. 1-Gb network switch
1-Gb network switch
Figure 2. Logical view of two switches
Logical view of the two switches
The orange cable that is shown in this figure must be connected between port 22 of the upper switch and 21 of the lower switch as part of the configuration. This cable works as inter-switch link (ISL) between the two switches.

Code version

ESS Legacy, ESS 3000, ESS 3200, ESS 5000, and ESS 3500 releases are included in ESS 6.1.6.x with two editions: Data Management Edition and Data Access Edition. An example of package names is as follows:Start of change
ess_6.1.8.3_1009-02_dme_ppc64le.tar.xz
ess_6.1.8.3_1009-02_dae_ppc64le.tar.xz
ess_6.1.8.3_1009-02_dme_x86_64.tar.xz
ess_6.1.8.3_1009-02_dae_x86_64.tar.xz
End of change
Note:
  • The versions shown here might not be the GA version available on IBM® Fix Central. It is recommended to go to IBM Fix Central and download the latest code.
  • ppc64le in the package name implies that each container runs on a POWER®-based EMS. For details about functions supported by respective containers, see Support matrix.
You can download the latest 6.1.x.x code (6.1.8.3 is the latest) from IBM Fix Central by using the following link.
A unified container is offered with two versions (Data management + Data access). Example package names for each container are as follows:
// Unified Container (Data Access and Data Management versions)
ESS_DAE_BASEIMAGE-6.1.8.3-ppc64LE-Linux.tgz
ESS_DME_BASEIMAGE-6.1.8.3-ppc64LE-Linux.tgz
ESS_DAE_BASEIMAGE-6.1.8.3-x86_64-Linux.tgz
ESS_DME_BASEIMAGE-6.1.8.3-x86_64-Linux.tgz
Note: The container installs and runs on the EMS only. The EMS supported is Power-based only. Running container on a x86-based node is not supported as of now.
Start of change

Upgrade guidance

ESS upgrade chart
Note: Start of change
  • Upgrades to ESS 6.1.2.x follow the N-2 rule. You can upgrade from ESS 6.1.2.x, 6.1.1.x (that is, 6.1.1.2) or 6.1.0.x.
  • Upgrades to ESS 6.1.5.x follow N-3 rule. You can upgrade from 6.1.2.x, 6.1.3.x, and 6.1.4.x.
  • Start of changeStarting with ESS 6.1.5.x, further jumps adhere to the N-3 ruleEnd of change
End of change
End of change
Start of change

Further legacy container migration guidance

You must migrate first to ESS 5.3.7.x before you upgrade to ESS 6.1.x.x (container version).

ESS 5.3.x.x upgrade guidance
  • You can upgrade to 5.3.7.x from 5.3.5.x (online) or 5.3.6.x (online).
  • For online upgrade you can jump one OS version and for offline upgrade you can jump two OS versions.
    • Only exception is RHEL 7.7 to RHEL 7.9 upgrade. Because there is no RHEL 7.8.
    • Online upgrade to RHEL 7.7 from RHEL 7.6 can be done.
    • Upgrade to RHEL 7.7 from RHEL 7.5 must be done online.
ESS 6.1.x.x upgrade guidance
  • It is recommended to convert from ESS 5.3.7.x to ESS 6.1.2.x and follow the normal N-X rules. To convert to ESS 6.1.2.x, use the following table (based on the RHEL 7.9 kernel):
    Table 1. RHEL kernels
    ESS Kernel
    6.1.2.7 3.10.0-1160.92.1.el7
    6.1.2.6 3.10.0-1160.83.1.el7

    4.18.0-372.41.1.el8_6
    6.1.2.5 3.10.0-1160.76.1.el7
    6.1.2.4 3.10.0-1160.71.1.el7
    6.1.2.3 3.10.0-1160.62.1.el7
    6.1.2.2 3.10.0-1160.49.1.el7
    5.3.7.6 3.10.0-1160.62.1.el7
    5.3.7.5 3.10.0-1160.59.1.el
    5.3.7.4 3.10.0-1160.49.1.el7
    5.3.7.3 3.10.0-1160.45.1.el7
    5.3.7.2 3.10.0-1160.31.1
    5.3.7.1 3.10.0-1160.24.1
    5.3.7.0 3.10.0-1160.11.1.el7
    An example of upgrade jump is as follows:
    • To upgrade to ESS 6.1.2.2, you can only upgrade from 5.3.7.4 or lower versions (that is, less than equal to 5.3.7.4).
    • To upgrade to ESS 6.1.2.3, you can only upgrade from 5.3.7.6 or lower versions.
  • It is not recommended to upgrade from ESS 5.3.7.x to ESS 6.1.1.2 anymore. Upgrade directly to ESS 6.1.2.3 or ESS 6.1.2.4. If you are updating from ESS 6.1.1.2, upgrade to 6.1.2.3 or higher (do not upgrade to 6.1.2.2).
  • For ESS 5.3.7.3, consider downgrading MOFED to MLNX_OFED_LINUX-4.9-3.1.5.3, and then convert to 6.1.2.3 or 6.1.2.4. This is to obtain full support for online upgrade when converting to RDMA core libs.
  • When upgrading to 5.3.x.x, first upgrade to ESS 5.3.7.2 or ESS 5.3.7.3, and then upgrade to 6.1.2.3 or 6.1.2.4. This upgrade is to obtain full support for online upgrade when converting to RDMA core libs.
  • You may need to modify the container unblock jumps from a specific 5.3.7.x level. Issue to the following command to upgrade the ESS level in the container:
    vim /opt/ibm/ess/deploy/ansible/vars.yml
  • Change (an example if you want to convert from ESS 5.3.7.1 or higher) LEGACY_SUPPORTED_VERSION: "5.3.7.3" to LEGACY_SUPPORTED_VERSION: "5.3.7.1".
For more information about the ESS 6.1.x.x upgrade, see IBM Storage Scale Alert: Mellanox OFED 5.x considerations in IBM ESS V6.1.2.x+.
End of change

Example of the /etc/hosts file

127.0.0.1 localhost localhost.localdomain.local localhost4 localhost4.localdomain4

## Management IPs 192.168.45.0/24
192.168.45.20 ems1.localdomain.local ems1
192.168.45.21 essio1.localdomain.local essio1
192.168.45.22 essio2.localdomain.local essio2
192.168.45.23 prt1.localdomain.local prt1
192.168.45.24 prt2.localdomain.local prt2

## High-speed IPs 10.0.11.0/24
10.0.11.1 ems1-hs.localdomain.local ems1-hs
10.0.11.2 essio1-hs.localdomain.local essio1-hs
10.0.11.3 essio2-hs.localdomain.local essio2-hs
10.0.11.4 pr1-hs.localdomain.local prt1-hs
10.0.11.5 pr2-hs.localdomain.local prt2-hs


## Protocol CES IPs
10.0.11.100 prt_ces1.localdomain.local prt_ces1
10.0.11.101 prt_ces1.localdomain.local prt_ces1
10.0.11.102 prt_ces2.localdomain.local prt_ces2
10.0.11.103 prt_ces2.localdomain.local prt_ces2

Node classes

All building blocks have node classes except the EMS node. Instead of a node list, you can use these node classes in the essrun command when the inventory file contains these nodes. (The essrun config load command adds the nodes to the inventory file.)

Node classes are as follows:
  • legacy: gss_ppc64le
  • 5000: ess_ppc64le
  • 3000: ess_x86_64
  • 3200: ess3200_x86_64
  • 3500: ess3500_x86_64
  • Protocol Power9: ces_ppc64le

Deployment/upgrade instructions

  1. Log in to the EMS (over management), set root password, set campus connection.
    1. Campus connection interface is called ‘campus’. Use nmtui to set an address.
    2. Set the EMS hostname.
  2. Log out and log in over the campus connection.
  3. Set up the /etc/hosts file.
  4. Copy the GA code into the /etc/deploy directory.
  5. Extract outer tarball.
  6. Decompress the inner xz file.
    xz --decompress ess_6.1.8.3_1009-02_dae_ppc64le.tar.xz
  7. Extract tar file.
    tar xvf ess_6.1.8.3_1009-02_dae_ppc64le.tar
  8. Run the installer.
    1. Start the container.
      sh ess_6.1.8.3_1009-02_dae_ppc64le --start-container
    2. Answer any user prompts.
Configuration in the container
  1. Run Config load.
    essrun -N ems1,essio1,essio2 config load -p
  2. Run update --precheck.
    essrun -N ems1,essio1,essio2 update --precheck
  3. Update EMS.
    When you upgrade ESS 5000 nodes, you cannot upgrade the firmware because of a restriction. Add the --no-fw-update option to prevent firmware upgrades. For example,
    essrun -N ess5kio1,ess5kio2 update --no-fw-update
    Add the --offline option for an offline upgrade and the --online option an online only upgrade.
    1. Update EMS offline.
      essrun -N ems1 update --offline
    2. If you are promted to reboot then run the command again.
    3. If you need to restart the container after reboot run the following command: 
      ./essmkyml
    4. Update the EMS again.
      essrun -N ems1 update --offline
  4. Update I/O nodes.
    • Update I/O nodes online.
      essrun -N <Node List/Node Name> update --no-fw-update
    • Update I/O nodes offline.
      essrun -N <Node List/Node Name> update --offline --no-fw-update
  5. Update POWER firmware. For more information see, Upgrading POWER9 firmware appendix in ESS Deployment Guide.
  6. Create network bonds.
    essrun -N ems1,essio1,essio2 network --suffix=-hs
  7. Run network test.
    ssh essio1
    ESSENV=TEST essnettest -N essio1,essio2 --suffix=-hs
  8. Create a cluster.
    essrun -N <Node List/Node Name> cluster --suffix=-hs
  9. Add EMS.
    essrun -N essio1 cluster --add-ems ems1 --suffix=-hs
  10. Create a file system.
    essrun -N ess3500_x86_64 filesystem --name fs1 --suffix=-hs
    Note: For ESS 3500, you must keep 1.5 TB or more space free if future capacity MES is planned (performance to hybrid). Thus, it is recommended to not use all available space when you create a file system for the performance model. The default allocation is 80% of available space when you use the essrun filesystem command (for x86 nodes).
  11. Final installation check per node.
    essinstallcheck -N localhost
  12. Check whether all nodes performance.
    gnrhealthcheck
    mmhealth node show -a
  13. Set up Chrony/NTP.
  14. Set up call home.
  15. Enable added security (SELinux/firewall/sudo/admin central).
  16. Configure the GUI/collectors.
    essrun -N ems1,essio1,essio2 gui --configure
  17. Set up conserver.
  18. Set up protocol services (3500 or P8/P9 Protocol nodes).
  19. Set up client nodes.
Note: If you want to add a building block, see the Adding additional nodes or building block(s) appendix in the ESS Deployment Guide.