CES and protocol nodes support in ESS
Overview of CES and protocol nodes
Cluster Export Services (CES) provides highly available file and object services to an IBM Spectrum Scale™ cluster by using Network File System (NFS), Object, or Server Message Block (SMB) protocols. Because CES has specific hardware and software requirements, the code must be installed on nodes designated to run the CES software stack. These nodes are called protocol nodes.
Protocol nodes can be added to an IBM Spectrum Scale cluster containing an ESS building block. They can also exist in non-ESS IBM Spectrum Scale clusters. SMB and NFS functions on protocol nodes can exist in clusters in which their storage is remotely mounted.
- IBM Spectrum Scale FAQ
- Covers all IBM Spectrum Scale levels and is frequently updated
- Contains minimum requirements for hardware and software
- Contains code level support statements
- Protocols Quick Overview for IBM Spectrum
Scale
- General flow of the IBM Spectrum Scale installation toolkit along with examples
- ESS specific examples are on page 2
- IBM Spectrum Scale CES protocols documentation:
- Protocols support overview
- Planning for protocols
- Installation and deployment of protocols
- Upgrade of IBM Spectrum Scale components including protocols
- Configuration of CES and protocols
- Managing protocol services
- Managing protocol user authentication
- Managing SMB and NFS data exports
- Managing object storage
- Monitoring IBM Spectrum Scale components including protocols
- Troubleshooting
Configuration changes required on protocol nodes to collect core dump data
- mmsmb command
- mmnfs command
- mmobj command
Supported protocol node configurations
5148-22L protocol node hardware
- 8247-22L Power8 model
- 2 x 10core 3.34 Ghz Power8 Processors
- 128 GB or greater memory
- Two 600 GB 10k RPM SAS HDDs in RAID10 mirror using IPRaid adapter
- 1 GbE 4port network adapter in slot C12
- 1 Three x 16 or x8 network adapters in slots C5, C6, C7
- 1 Four x 8 network adapters in slots C2, C3, C10, C11 available by additional card orders (through MES)
1 It is recommended to plan for the GPFS admin or daemon network to use separate network adapter(s) from the Cluster Export Services.
For more information, see 5148-22L protocol node diagrams.5148-22L protocol node software
5148-22L protocol node components | Supported level | Managed by |
Power8 FW | 01SV860_138 (FW860.42) (in OPAL mode) | User (using update_flash) |
OS | RHEL7.4 PPC64LE | EMS node (using gssdeploy) |
kernel | 3.10.0-693.35.1.el7 | |
systemd | 219-42.el7_4.11 | EMS node (using gssdeploy) |
network manager | 1.8.0-12.el7_4 | EMS node (using gssdeploy) |
Mellanox OFED (Firmware and driver) Mellanox OFED2 (Packaged to support Mellanox CX-2 adapter)
Note: For the CX-2 adapter, only the driver is
updated. Firmware is not updated.
|
Driver: MLNX_OFED_LINUX-4.3-1.0.1.1 OFED2 driver: MLNX_OFED_LINUX-4.1-4.1.6.0 |
EMS node (using gssdeploy) |
Power8 IPR (Firmware and driver) |
18518200 | EMS node (using gssdeploy) |
sysctl, tuned, udev rules | OS tuning for RHEL 7.4 PPC64LE protocol nodes | EMS node (using gssdeploy) |
gpfs.gss.tools |
5.3.1.1 |
EMS node (using gssdeploy) |
Red Hat Enterprise Linux OS syslog | --- | User managed for debug (contained on each Protocol node) |
SSH key setup | --- | EMS node (using gssdeploy) or User |
/etc/hosts | --- | EMS node (using gssdeploy) or User |
Repositories | --- | EMS node (using gssdeploy) (persistent) IBM Spectrum Scale installation toolkit (only while running) |
/etc/resolv.conf | --- | User |
GPFS network configuration | --- | User (using gssgennetwork) |
CES base network configuration | --- | User |
CES shared root creation | --- | EMS node (using gssgenvdisks --crcesfs) |
IBM Spectrum Scale code (RPM install, node add, license) |
5.0.1.2 or later Note: It is recommended to keep IBM Spectrum Scale at the same
level across all nodes, but a higher level of IBM Spectrum Scale can be used for protocol nodes, if
desired. For more information, see IBM Spectrum Scale supported upgrade paths.
|
IBM Spectrum Scale installation toolkit (installation phase) |
IBM Spectrum Scale CES code (rpm install, CES IPs, protocol enablement, license) |
IBM Spectrum Scale installation toolkit (deployment phase) |
|
CES protocol performance monitoring sensors | IBM Spectrum Scale installation toolkit (deployment phase) |
|
CES authentication configuration | User (using mmuserauth) or IBM Spectrum Scale installation toolkit |
|
Call home config | EMS (via gsscallhomeconf) | |
GUI integration of protocol nodes | EMS node (automatically detects CES nodes) User (using GUI rack location adjustment) |
|
GPFS configuration parameters | GPFS configuration parameters for protocol nodes | User (using mmchconfig) User (using GUI) |
Health checks | --- | EMS (using gssinstallcheck) EMS (using gssstoragequickcheck) IBM Spectrum Scale installation toolkit (prechecks and postchecks) User (using mmhealth and GUI) |
Customer supplied protocol node hardware recommendations
Non-5148-22L protocol nodes can be based upon PPC64LE, PPC64BE, or x86_64 architectures and they are suggested to adhere to the guidance in IBM Spectrum Scale FAQ. All hardware, cabling or connections, and power sequencing is owned by the customer.
Support for IBM Spectrum Scale on the chosen hardware is handled by the general IBM Spectrum Scale support and not by ESS Solution Support.
Customer supplied protocol node software management recommendations
Only 5148-22L protocol nodes are supported for management by the EMS node. If a protocol node is not ordered with this exact model and type from IBM manufacturing, it cannot be added to the EMS xCAT server. While non-5148-22L protocol nodes of type PPC64LE, PPC64BE, or x86_64, might be joined to an ESS cluster using the IBM Spectrum Scale installation toolkit or the mmaddnode command, they must be owned, installed, deployed, upgraded, and managed by the customer. The ESS toolsets cannot be used for the management of OS, kernel, network manager, systemd, OFED, or firmware on non-5148-22L nodes. Using the ESS toolsets, including the EMS xCAT server, to manage customer supplied non-5148-22L protocol nodes is not supported.
The IBM Spectrum Scale code on customer supplied non-5148-22L protocol nodes is also managed separately from the ESS toolsets. The IBM Spectrum Scale installation toolkit can be used in this configuration to install or deploy and upgrade the IBM Spectrum Scale code on any customer supplied non-5148-22L protocol nodes. For more information about the installation toolkit, see IBM Spectrum Scale installation toolkit.
For latest information on supported levels, see IBM Spectrum Scale FAQ.