Configuration 2: 5148-22L protocol nodes ordered standalone and added to an existing 5148 ESS (PPC64LE)
Overall flow
- A) Starting point and what to expect
- B) Protocol node OS deployment
- C) Decide which adapter(s) to use for the GPFS network(s) vs CES protocol network(s)
- D) Configure network adapters to be used for GPFS
- E) Configure network adapters to be used for CES protocols
- F) Create a CES shared root file system for use with protocol nodes
- G) Download the IBM Spectrum Scale protocols package (version 5.0.1.2 or later) on one of the protocol nodes
- H) Extract the IBM Spectrum Scale protocols package
- I) Configure the IBM Spectrum Scale installation toolkit
- J) Installation phase of IBM Spectrum Scale installation toolkit
- K) Deployment phase of IBM Spectrum Scale installation toolkit
- L) Tune the protocol nodes as desired
- M) GUI configuration
- N) Call home setup
- O) Movement of quorum or management function to protocol nodes and off EMS or I/O nodes
A) Starting point and what to expect
- An ESS is already installed and running (EMS and 2 I/O server nodes + possibly additional nodes)
- A cluster has already been created.
Run mmlscluster to check.
- A GUI is active on the EMS node and it has been logged into.
Run systemctl status gpfsgui to check.
- The Performance Monitoring collector is configured and running on the EMS node.
Run systemctl status pmcollector and mmperfmon config show to check.
- Protocol nodes may or may not already exist on this cluster.
- The ESS is at code level 5.3.1.1 or later.
Run /opt/ibm/gss/install/rhel7/ppc64le/installer/gssinstall -V to check.
- An xCAT OS image specific for CES exists
(rhels7.4-ppc64le-install-ces).Run the following command to verify.
# lsdef -t osimage
- Newly ordered protocol nodes come with authentication prerequisites pre-installed upon them (includes sssd, ypbind, openldap-clients, krb5-workstation).
- A default deploy template exists for the protocol nodes.
Check for /var/tmp/gssdeployces.cfg.default on the EMS node
- New standalone protocol node orders arrives with OS, kernel, OFED, iprraid, and firmware pre-loaded. This is verified in step B7.
- New standalone protocol node orders arrive with an IBM Spectrum Scale Protocols code package in /root. This is verified in steps G and H.
- New standalone protocol nodes do have not have any GPFS RPMs installed on them
- Hardware and software call home may already be configured on the existing ESS system. Call home is reconfigured after deployment of the protocol nodes.
- Protocol nodes must be cabled up to the ESS switch for use with the xCAT network and the FSP network. For more information, see Figure 2.
- Protocol nodes can be in the powered off state at this point.
B) Protocol node OS deployment
- Verify that the protocol nodes are cabled up to the ESS xCAT network and FSP network.
- On the EMS node, find the /var/tmp/gssdeployces.cfg.default file and copy
to /var/tmp/gssdeployces.cfg.The default CES template is pre-filled so that the only fields needing customization to match the current cluster are:
- DEPLOYMENT_TYPE: If these are your first protocol nodes, the type must be CES. If you are adding more protocol nodes to an ESS system that already has protocol nodes, use the type ADD_CES. Read the tips in the gssdeployces.cfg file carefully because using the incorrect deployment type and filling out the configuration file incorrectly could result in rebooting or reloading of any existing protocol nodes that might be a part of the cluster. Read all on-screen warnings.
- EMS_HOSTNAME
- EMS_MGTNETINTERFACE
- SERVERS_UID
- SERVERS_PASSWD
- SERVERS_SERIAL: Change the serial numbers to match each protocol node being added.
- SERVERS_NODES: Separate each desired protocol node name with a space.
- Configure /etc/hosts on the EMS node to list the protocol nodes.
Here is an example of the IP, FQDN, and hostname configured for EMS, IO, and 3 protocol nodes.
- The EMS node is 192.168.45.20.
- The I/O server nodes are 192.168.45.21 → 192.168.45.30.
- The protocol nodes are 192.168.45.31 → X.
192.168.45.20 ems1.gpfs.net ems1 192.168.45.21 gssio1.gpfs.net gssio1 192.168.45.22 gssio2.gpfs.net gssio2 192.168.45.31 prt01.gpfs.net prt01 192.168.45.32 prt02.gpfs.net prt02 192.168.45.33 prt03.gpfs.net prt03 172.31.250.3 ems1-hs.gpfs.net ems1-hs 172.31.250.1 gssio1-hs.gpfs.net gssio1-hs 172.31.250.2 gssio2-hs.gpfs.net gssio2-hs 172.31.250.11 prt01-hs.gpfs.net prt01-hs 172.31.250.12 prt02-hs.gpfs.net prt02-hs 172.31.250.13 prt03-hs.gpfs.net prt03-hs
Note: If the /etc/hosts file is already set up on the EMS node, copy it to the protocol node(s) first and then modify it. Each protocol node must have the same /etc/hosts file. - Detect and add the protocol node objects to xCAT as
follows.
/var/tmp/gssdeploy -o /var/tmp/gssdeployces.cfg
Proceed through all steps. Protocol nodes should be listed in xCAT afterwards.# lsdef ems1 (node) gssio1 (node) gssio2 (node) prt01 (node) prt02 (node) prt03 (node)
- Deploy the OS, kernel, systemd, netmgr, OFED, and IPR as follows.This is a decision point with two options depending upon the requirement.
- Option 1: All standalone protocol nodes come preinstalled with OS, kernel, systemd,
netmgr, OFED, and IPR. Now that the protocol nodes are discovered by xCAT, they can be set to boot
from their hard drives, without reinstalling anything. If these preloaded levels are sufficient,
then proceed with these steps. This option is quicker than option 2.
- Power off all protocol
nodes.
rpower ProtocolNodesList off
- Set the protocol node(s) to HD boot from the EMS
node.
rsetboot ProtocolNodesList hd
- Power on the protocol
nodes.
rpower ProtocolNodesList on
- Power off all protocol
nodes.
- Option 2: It is also possible to completely wipe and reload the protocol nodes if
desired. Remember: If you already have existing and active protocol nodes in the cluster, you must be very careful about which xCAT group is used and whether your gssdeployces.cfg file has a DEPLOYMENT_TYPE of CES or ADD_CES. Read the tips in the configuration file carefully, and read all on-screen warnings.Run the following commands to proceed.
- Reload the protocol nodes with the same levels of OS, kernel, systemd, netmgr, OFED, and IPR
existing on the EMS / IO
nodes.
/var/tmp/gssdeploy -d /var/tmp/gssdeployces.cfg
- Proceed through all steps of the gssdeploy command. Protocol node
installation progress can be watched using rcons
Node.Note: Unlike on I/O server nodes, this step does not install any GPFS RPMs on protocol nodes except gpfs.gss.tools.
- Reload the protocol nodes with the same levels of OS, kernel, systemd, netmgr, OFED, and IPR
existing on the EMS / IO
nodes.
- Option 1: All standalone protocol nodes come preinstalled with OS, kernel, systemd,
netmgr, OFED, and IPR. Now that the protocol nodes are discovered by xCAT, they can be set to boot
from their hard drives, without reinstalling anything. If these preloaded levels are sufficient,
then proceed with these steps. This option is quicker than option 2.
- If ADD_CES was used to add protocol nodes to a cluster that had existing or
active protocol nodes, they would have been added to an xCAT group other than ces_ppc64. These
protocol nodes must be moved to the ces_ppc64 group using these steps, run from the EMS node.
- Check to see which xCAT group was used for adding protocol nodes. Replace the configuration file
name with the one used for gssdeploy -o and
-d.
# cat /var/tmp/gssdeployces.cfg | grep GSS_GROUP
- Move the added protocol nodes to the ces_ppc64
group.
# chdef GroupNameUsed groups=all,ces_ppc64
- Check to see which xCAT group was used for adding protocol nodes. Replace the configuration file
name with the one used for gssdeploy -o and
-d.
- Once deployed, run gssinstallcheck or gssstoragequickcheck
to verify the nodes are in a healthy state.
- Run gssstoragequickcheck -G ces_ppc64 to verify network adapter types, slots, and machine type model of the protocol nodes.
- Run gssinstallcheck -G ces_ppc64 to verify code and firmware levels on the protocol nodes.
C) Decide which adapter(s) to use for the GPFS network(s) vs CES protocol network(s)
It is recommended to plan for separation of the GPFS and CES networks, both by subnet and by card.
If adding protocol nodes to an existing or active protocol setup, in most cases it is recommended to match configurations of both the GPFS network and CES protocol networks to the existing protocol nodes. If planning a stretch cluster, or configurations in which not all protocol nodes see the same CES network, refer to IBM Spectrum Scale Knowledge Center.
D) Configure network adapters to be used for GPFS
Customer networking requirements are site-specific. The use of bonding to increase fault-tolerance and performance is advised but guidelines for doing this have not been provided in this document. Consult with your local network administrator before proceeding further. Before creating network bonds, carefully read ESS networking considerations.
Make sure that the protocol nodes high speed network IPs and host names are present in /etc/hosts on all nodes.
172.31.250.3 ems1-hs.gpfs.net ems1-hs
172.31.250.1 gssio1-hs.gpfs.net gssio1-hs
172.31.250.2 gssio2-hs.gpfs.net gssio2-hs
172.31.250.11 prt01-hs.gpfs.net prt01-hs
172.31.250.12 prt02-hs.gpfs.net prt02-hs
172.31.250.13 prt03-hs.gpfs.net prt03-hs
- All nodes must be able to resolve all IPs, FQDNs, and host names, and ssh-keys must work.
- If the /etc/hosts file is already set up on the EMS node, copy it to the protocol node(s) first and then modify it. Each protocol node must have the same /etc/hosts file.
gssgennetworks -G ces_ppc64 --create --ipoib --suffix=-hs --mtu 4092
In this example, MTU is set to 4092. The default MTU is 2048 (2K) and the gssgennetworks command supports 2048 (2K) and 4092 (4K) MTU. Consult your network administrator for the proper MTU setting.
gssgennetworks -N ems1,gss_ppc64 --suffix=-hs --create-bond
For information on Infiniband issue with multiple fabrics, see Infiniband with multiple fabric in Customer networking considerations.
E) Configure network adapters to be used for CES protocols
- Bandwidth requirements per protocol node (how many ports per bond, bonding mode, and adapter speed)
- Redundancy of each protocol node, if needed. This determines the bonding mode used.
- Authentication domain and DNS. This determines the subnet(s) required for each protocol node.
- Are VLAN tags needed?
- Set aside 1 IP per protocol node, per desired CES subnet. You will be using these when configuring the CES base adapter(s) on each protocol node. These IPs must be setup for forward and reverse DNS lookup.
- Set aside a pool of CES IPs for later use. These IPs must be in DNS and be setup for both forward and reverse DNS lookup. You will not be assigning these IPs to network adapters on protocol nodes.
- Prepare to configure each protocol node to point to the authentication domain or DNS. You need to do this manually using ifcfg or nmcli commands and by verifying /etc/resolv.conf after the settings have taken effect. When deployed from an EMS node, each protocol node might already have a default domain of gpfs.net present in /etc/resolv.conf and the ifcfg files. This default domain can be removed so that it does not interfere with the authentication setup and DNS for protocols.
Proceed with either configuring the CES protocol adapters manually using ifcfg or nmcli commands or by using gssgennetworks. The gssgennetworks command cannot be used if your CES protocol network requires VLAN tags nor does it set up additional domain or DNS servers.
- Make sure all protocol nodes can ping each other's base CES network by IP, host name, and FQDN.
- Make sure all protocol nodes can ping the authentication server by IP, host name, and FQDN.
- Make sure the authentication server can ping each protocol node base CES network by IP, host name, and FQDN
- Spot check the desired NFS, SMB, or OBJ clients, external to the GPFS cluster and verify that they can ping each protocol node base CES network by IP, hostname, and FQDN
- Even though the CES IP pool is not yet set up, because protocols are not deployed, double check that each protocol node can resolve each CES IP or host name using nslookup.
For an example showing how CES IP aliasing relies upon an established base adapter with proper subnets or routing, see CES IP aliasing to network adapters on protocol nodes.
For an example of CES-IP configuration that can be performed after deployment of protocols, see Configuring CES protocol service IP addresses.
F) Create a CES shared root file system for use with protocol nodes
If you already have an existing or active protocol setup and cesSharedRoot should already exist. Skip this step.
- Run the following command from the EMS
node.
# gssgenvdisks --create-vdisk --create-nsds --create-filesystem --crcesfs # mmmount cesSharedRoot -N ems1-hs
A file system named cesSharedRoot with a mount path of /gpfs/cesSharedRoot is created and mounted. Later in these steps, the IBM Spectrum Scale installation toolkit is pointed to this file system to use when deploying protocols.
G) Download the IBM Spectrum Scale protocols package (version 5.0.1.2 or later) on one of the protocol nodes
An example package name is: Spectrum_Scale_Protocols_Data_Management-5.0.1.2-ppc64LE-Linuxinstall
Each protocol node ships with an IBM Spectrum Scale protocols package in /root. The version and the license of this package matches with the ESS version that the protocol node was ordered with.
- If the package is of the desired version and license, proceed with extraction.
- If a different level is desired, proceed to IBM FixCentral to download and replace this version.If replacing this version, the following rules apply:
- The IBM Spectrum Scale version must be 5.0.1.2 or later.
- The CPU architecture must be PPC64LE.
- The package must be a protocols package (The title and the file name must specifically contain Protocols).
H) Extract the IBM Spectrum Scale protocols package
- Enter the following at the command prompt: /root/Spectrum_Scale_Protocols_Data_Management-5.0.1.2-ppc64LE-Linux-install
- By default, the package is extracted to /usr/lpp/mmfs/5.0.1.2/.
I) Configure the IBM Spectrum Scale installation toolkit
- Change directory to the installation toolkit
directory:
cd /usr/lpp/mmfs/5.0.1.2/installer
View the installation toolkit usage help as follows./usr/lpp/mmfs/5.0.1.2/installer/spectrumscale -h
- Set up the installation toolkit by specifying which local IP to use for communicating with the
rest of the nodes. Preferably, this IP must be the same IP used for the GPFS network. Set the
toolkit to the ESS
mode.
/usr/lpp/mmfs/5.0.1.2/installer/spectrumscale setup -s IP_Address -st ess
- Populate the installation toolkit configuration file with the current cluster configuration by
pointing it to the EMS
node.
/usr/lpp/mmfs/5.0.1.2/installer/spectrumscale config populate -N EMSNode
There are limits to the config populate functionality. If it does not work, simply add the EMS node to the installation toolkit and continue.
View the current cluster configuration as follows./usr/lpp/mmfs/5.0.1.2/installer/spectrumscale node list /usr/lpp/mmfs/5.0.1.2/installer/spectrumscale config gpfs
Note: ESS I/O nodes do not get listed in the installation toolkit node list. - Configure the details of the protocol nodes to be added to the cluster. If adding protocol nodes
to an existing or active protocol setup, make sure to add all existing protocol nodes and
configuration details. Note that a successful config populate operation from step 3 would have
already performed this action.
For more information, see IBM Spectrum Scale installation toolkit../spectrumscale node add ems1-hs.gpfs.net -e -a -g ## No need to perform this step ## if the config populate ran without error ./spectrumscale node add prt01-hs.gpfs.net -p ## Add a protocol node ./spectrumscale node add prt02-hs.gpfs.net -p ## Add a protocol node ./spectrumscale node add client01-hs.gpfs.net ## Example of a non-protocol client node, if desired ./spectrumscale node add nsd01-hs.gpfs.net ## Example of a non-ESS nsd node, if desired ./spectrumscale enable smb ## If you'd like to enable and use the SMB protocol ## (it will be installed regardless) ./spectrumscale enable nfs ## If you'd like to enable and use the NFS protocol ## (it will be installed regardless) ./spectrumscale enable object ## If you'd like to enable and use the Object protocol ## (it will be installed regardless) ./spectrumscale config protocols -e CESIP1,CESIP2,CESIP3 ## Input the CES IPs set aside from step (G) of ## this procedure. Toolkit assigns IPs listed. ./spectrumscale config protocols -f cesSharedRoot -m /gpfs/cesSharedRoot ## FS name and mount point for ## CES shared root, previously ## setup during step (D) ./spectrumscale config object -e <endpoint IP or hostname> ## This address should be an RRDNS or similar address ## that resolves to the pool of CES IP addresses. ./spectrumscale config object -o Object_Fileset ## This fileset will be created during deploy ./spectrumscale config object -f ObjectFS -m /gpfs/ObjectFS ## This must point to an existing FS ## create the FS on EMS if it doesn't already exist ./spectrumscale config object -au admin -ap -dp ## Usernames and passwords for Object ./spectrumscale config perfmon –r on ## Turn on performance sensors for the protocol nodes. ## EMS GUI picks up sensor data once protocols are deployed ./spectrumscale node list ## Lists out the node config (ESS IO nodes never show up here) ./spectrumscale config protocols ## Shows the protocol config
J) Installation phase of IBM Spectrum Scale installation toolkit
- Run the installation toolkit installation
precheck.
./spectrumscale install --precheck
- Run the installation toolkit installation
procedure.
./spectrumscale install
- Install GPFS, call home, performance monitoring, license RPMs on each node specified to the installation toolkit. The EMS and I/O server nodes are not acted upon by the installation toolkit.
- Add nodes to the cluster (protocol, client, NSD).
- Add non-ESS NSDs, if desired.
- Start GPFS and mount all file systems on the newly added nodes.
- Configure performance monitoring sensors.
- Set client or server licenses.
GPFS configuration parameters such as pagepool, maxFilesToCache, verbsPorts need to be set up manually. You can do this after completing the installation phase or after completing the deployment phase. For more information about these parameters, see GPFS configuration parameters for protocol nodes.
K) Deployment phase of IBM Spectrum Scale installation toolkit
- Run the installation toolkit deployment
precheck.
./spectrumscale deploy --precheck
- Run the installation toolkit installation
procedure.
./spectrumscale deploy
- Install SMB, NFS, and object RPMs on each protocol node specified to the installation toolkit.
- Enable one or more protocols.
- Assign CES IPs. IBM Spectrum Scale code aliases these IPs to the CES base network adapter configured during step E.
- Enable authentication for file or object.
- Create additional file systems using non-ESS NSD nodes. You must run installation first to add more non-ESS NSDs.
- Add additional protocol nodes. You must run installation first to add more nodes and then run deployment for the protocol specific piece.
L) Tune the protocol nodes as desired
Protocol nodes should already be tuned with the same tuned and sysctl settings, and udev rules as the I/O server nodes. For more information, see OS tuning for RHEL 7.4 PPC64LE protocol nodes.
- RDMA. If IB RDMA is in use (check using mmlsconfig verbsRDMA), issue mmlsconfig and verify that the verbPorts parameter refers to the correct ports on each protocol node.
- pagepool. Use mmlsconfig for the pagepool
settings of the EMS and I/O server nodes. The protocol nodes do not have
pagepool defined at this point. Define pagepool using
mmchconfig -N cesNodes pagepool=XX command.
Where XX is typically 25% to 50% of the system memory. For more information, see GPFS configuration parameters for protocol nodes.
- maxFilesToCache. Use mmlsconfig to view the
maxFilesToCache settings of the EMS and I/O server nodes. The protocol nodes do
not have maxFilesToCache defined at this point. Define
maxFilesToCache using mmchconfig -N cesNodes
maxFilesToCache=XX command.
Where XX is typically 2M for protocol nodes in a cluster containing ESS. For more information, see GPFS configuration parameters for protocol nodes.
M) GUI configuration
- Open the existing EMS GUI in a web browser using the URL https://EssGuiNode where ESSGuiNode is the host name or IP address of the management server node. If the GUI is not yet set up, perform step 8 in this procedure.
- Software monitoring of protocol nodes occurs automatically by the GUI and all newly added
protocol nodes should now exist in the HOME page Nodes section. Hardware
monitoring of protocol nodes must be configured within the GUI panels as follows.
- Click
The Edit Rack Components wizard is displayed.
. - On the Welcome screen, select Yes, discover new servers and
enclosures first and then click Next.
This might take a few minutes. After the detection is complete, click OK.
- Click Next through the screens allowing edits of various rack components.
The Other Servers screen shows the protocol nodes.
- On the Other Servers screen, select a rack and location for each protocol node and then click Next.
- Click Finish on the Summary screen.
- Click
N) Call home setup
Now that protocol nodes are deployed, call home needs to be configured.
- Check if call home has already been configured and if so, record the settings. Reconfiguring
call home might require entering the settings again.
- Check the hardware call home settings on the EMS
node.
gsscallhomeconf --show -E ems1
- Check the software call home setup.
mmcallhome info list
- Check the hardware call home settings on the EMS
node.
- Set up call home from the EMS
node.
gsscallhomeconf -N ems1,gss_ppc64,ces_ppc64 --suffix=-hs -E ems1 --register all --crvpd
O) Movement of quorum or management function to protocol nodes and off EMS or I/O nodes
Quorum and management functions can be resource intensive. In an ESS cluster that also has extra nodes, such as protocols, within the same cluster, it is recommended to move these functions to the protocol nodes. For information on changing node designation, see mmchnode command.