Configuration 1: 5148-22L protocol nodes ordered and racked with a new 5148 ESS (PPC64LE)
Overall flow
- A) Starting point and what to expect upon initial power-on
- B) Power on the protocol nodes and perform health checks
- C) Decide which adapter(s) to use for the GPFS network(s) vs CES protocol network(s)
- D) Set up the high-speed network for GPFS
- E) Configure network adapters to be used for CES protocols
- F) Create a CES shared root file system for use with protocol nodesG) Locate the IBM Spectrum Scale protocols package on a protocol node in /root
- H) Extract the IBM Spectrum Scale protocols package
- I) Configure the IBM Spectrum Scale installation toolkit
- J) Installation phase of IBM Spectrum Scale installation toolkit
- K) Deployment phase of IBM Spectrum Scale installation toolkit
- L) Tune the protocol nodes as desired
- M) GUI configuration
- N) Call home setup
- O) Movement of quorum or management function to protocol nodes and off EMS or I/O nodes
A) Starting point and what to expect upon initial power-on
- This configuration assumes Fusion mode has been run, thus retaining the shipped xCAT configuration of EMS, I/O server nodes, and protocol nodes. If Fusion mode has not been run and the ESS system has been reinstalled, then start with Configuration 2.
- The EMS and I/O server nodes are installed, high speed network has been created, and GPFS is active and healthy.
- The GUI has not yet been started nor configured. This is done following the protocol node configuration.
- Hardware and software call home is not yet configured. This is done after protocols are deployed.
- Protocol nodes exist in XCAT as prt01, prt0N objects.Run the following command to verify.
# lsdef
- Protocol nodes are part of the ces_ppc64 xCAT group.Run the following command to verify.
# lsdef -t group ces_ppc64
- An xCAT OS image specific for CES exists
(rhels7.4-ppc64le-install-ces).Run the following command to verify.
# lsdef -t osimage
- Protocol nodes come with authentication prerequisites pre-installed upon them (includes sssd, ypbind, openldap-clients, krb5-workstation).
- A deploy template exists, already customized for the protocol nodes.
Check for /var/tmp/gssdeployces.cfg on the EMS node.
- Protocol nodes already have RHEL 7.4 OS, kernel, network manager, systemd, OFED, iprraid, and firmware loaded at the same levels as the EMS and I/O server nodes. This is verified in this step.
- Protocol nodes do not have any GPFS RPMS installed except gpfs.gss.tools.
- Protocol nodes are already cabled to the internal ESS switch for use with the xCAT network and the FSP network. For more information, see Figure 2.
B) Power on the protocol nodes and perform health checks
- Run gssstoragequickcheck -G ces_ppc64 to verify network adapter types, slots, and machine type model of the protocol nodes.
- Run gssinstallcheck -G ces_ppc64 to verify code and firmware levels on the protocol nodes.
C) Decide which adapter(s) to use for the GPFS network(s) vs CES protocol network(s)
It is recommended to plan for separation of the GPFS and CES networks, both by subnet and by card.
D) Set up the high-speed network for GPFS
Customer networking requirements are site-specific. The use of bonding to increase fault-tolerance and performance is advised but guidelines for doing this have not been provided in this document. Consult with your local network administrator before proceeding further. Before creating network bonds, carefully read ESS networking considerations.
Make sure that the protocol nodes high speed network IPs and host names are present in /etc/hosts on all nodes.
172.31.250.3 ems1-hs.gpfs.net ems1-hs
172.31.250.1 gssio1-hs.gpfs.net gssio1-hs
172.31.250.2 gssio2-hs.gpfs.net gssio2-hs
172.31.250.11 prt01-hs.gpfs.net prt01-hs
172.31.250.12 prt02-hs.gpfs.net prt02-hs
172.31.250.13 prt03-hs.gpfs.net prt03-hs
- All nodes must be able to resolve all IPs, FQDNs, and host names, and ssh-keys must work.
- If the /etc/hosts file is already set up on the EMS node, copy it to the protocol node(s) first and then modify it. Each protocol node must have the same /etc/hosts file.
gssgennetworks -G ces_ppc64 --create --ipoib --suffix=-hs --mtu 4092
In this example, MTU is set to 4092. The default MTU is 2048 (2K) and the gssgennetworks command supports 2048 (2K) and 4092 (4K) MTU. Consult your network administrator for the proper MTU setting.
gssgennetworks -N ems1,gss_ppc64 --suffix=-hs --create-bond
For information on Infiniband issue with multiple fabrics, see Infiniband with multiple fabric in Customer networking considerations.
E) Configure network adapters to be used for CES protocols
- Bandwidth requirements per protocol node (how many ports per bond, bonding mode, and adapter speed)
- Redundancy of each protocol node, if needed. This determines the bonding mode used.
- Authentication domain and DNS. This determines the subnet(s) required for each protocol node.
- Are VLAN tags needed?
- Set aside 1 IP per protocol node, per desired CES subnet. You will be using these when configuring the CES base adapter(s) on each protocol node. These IPs must be setup for forward and reverse DNS lookup.
- Set aside a pool of CES IPs for later use. These IPs must be in DNS and be setup for both forward and reverse DNS lookup. You will not be assigning these IPs to network adapters on protocol nodes.
- Prepare to configure each protocol node to point to the authentication domain or DNS. You need to do this manually using ifcfg or nmcli commands and by verifying /etc/resolv.conf after the settings have taken effect. When deployed from an EMS node, each protocol node might already have a default domain of gpfs.net present in /etc/resolv.conf and the ifcfg files. This default domain can be removed so that it does not interfere with the authentication setup and DNS for protocols.
Proceed with either configuring the CES protocol adapters manually using ifcfg or nmcli commands or by using gssgennetworks. The gssgennetworks command cannot be used if your CES protocol network requires VLAN tags nor does it set up additional domain or DNS servers.
- Make sure all protocol nodes can ping each other's base CES network by IP, host name, and FQDN.
- Make sure all protocol nodes can ping the authentication server by IP, host name, and FQDN.
- Make sure the authentication server can ping each protocol node base CES network by IP, host name, and FQDN
- Spot check the desired NFS, SMB, or OBJ clients, external to the GPFS cluster and verify that they can ping each protocol node base CES network by IP, hostname, and FQDN
- Even though the CES IP pool is not yet set up, because protocols are not deployed, double check that each protocol node can resolve each CES IP or host name using nslookup.
For an example showing how CES IP aliasing relies upon an established base adapter with proper subnets or routing, see CES IP aliasing to network adapters on protocol nodes.
For an example of CES-IP configuration that can be performed after deployment of protocols, see Configuring CES protocol service IP addresses.
F) Create a CES shared root file system for use with protocol nodes
- Run the following command from the EMS
node.
# gssgenvdisks --create-vdisk --create-nsds --create-filesystem --crcesfs # mmmount cesSharedRoot -N ems1-hs
A file system named cesSharedRoot with a mount path of /gpfs/cesSharedRoot is created and mounted. Later in these steps, the IBM Spectrum Scale installation toolkit is pointed to this file system to use when deploying protocols.
G) Locate the IBM Spectrum Scale protocols package on a protocol node in /root
An example package name is: Spectrum_Scale_Protocols_Data_Management-5.0.1.2-ppc64LE-Linuxinstall
Each protocol node ships with an IBM Spectrum Scale protocols package in /root. The version and the license of this package matches with the ESS version that the protocol node was ordered with.
- If the package is of the desired version and license, proceed with extraction.
- If a different level is desired, proceed to IBM FixCentral to download and replace this version.If replacing this version, the following rules apply:
- The IBM Spectrum Scale version must be 5.0.1.2 or later.
- The CPU architecture must be PPC64LE.
- The package must be a protocols package (The title and the file name must specifically contain Protocols).
H) Extract the IBM Spectrum Scale protocols package
- Enter the following at the command prompt: /root/Spectrum_Scale_Protocols_Data_Management-5.0.1.2-ppc64LE-Linux-install
- By default, the package is extracted to /usr/lpp/mmfs/5.0.1.2/.
I) Configure the IBM Spectrum Scale installation toolkit
- Change directory to the installation toolkit
directory:
cd /usr/lpp/mmfs/5.0.1.2/installer
View the installation toolkit usage help as follows./usr/lpp/mmfs/5.0.1.2/installer/spectrumscale -h
- Set up the installation toolkit by specifying which local IP to use for communicating with the
rest of the nodes. Preferably, this IP should be the same IP used for the GPFS network. Set the
toolkit to the ESS
mode.
/usr/lpp/mmfs/5.0.1.2/installer/spectrumscale setup -s IP_Address -st ess
- Populate the installation toolkit configuration file with the current cluster configuration by
pointing it to the EMS
node.
/usr/lpp/mmfs/5.0.1.2/installer/spectrumscale config populate -N EMSNode
There are limits to the config populate functionality. If it does not work, simply add the EMS node to the installation toolkit and continue.
View the current cluster configuration as follows./usr/lpp/mmfs/5.0.1.2/installer/spectrumscale node list /usr/lpp/mmfs/5.0.1.2/installer/spectrumscale config gpfs
Note: ESS I/O nodes do not get listed in the installation toolkit node list. - Configure the details of the protocol nodes to be added to the
cluster.
For more information, see IBM Spectrum Scale installation toolkit../spectrumscale node add ems1-hs.gpfs.net -e -a -g ## No need to perform this step ## if the config populate ran without error ./spectrumscale node add prt01-hs.gpfs.net -p ## Designate a protocol node ./spectrumscale node add prt02-hs.gpfs.net -p ## Designate a protocol node ./spectrumscale node add prt03-hs.gpfs.net –p ## Designate a protocol node ./spectrumscale node add client01-hs.gpfs.net ## Example of a non-protocol client node, if desired ./spectrumscale node add nsd01-hs.gpfs.net ## Example of a non-ESS nsd node, if desired ./spectrumscale enable smb ## If you'd like to enable and use the SMB protocol ## (it will be installed regardless) ./spectrumscale enable nfs ## If you'd like to enable and use the NFS protocol ## (it will be installed regardless) ./spectrumscale enable object ## If you'd like to enable and use the Object protocol ## (it will be installed regardless) ./spectrumscale config protocols -e CESIP1,CESIP2,CESIP3 ## Input the CES IPs set aside from step (G) of ## this procedure. Toolkit assigns IPs listed. ./spectrumscale config protocols -f cesSharedRoot -m /gpfs/cesSharedRoot ## FS name and mount point for ## CES shared root, previously ## setup during step (D) ./spectrumscale config object -e <endpoint IP or hostname> ## This address should be an RRDNS or similar address ## that resolves to the pool of CES IP addresses. ./spectrumscale config object -o Object_Fileset ## This fileset will be created during deploy ./spectrumscale config object -f ObjectFS -m /gpfs/ObjectFS ## This must point to an existing FS ## create the FS on EMS if it doesn't already exist ./spectrumscale config object -au admin -ap -dp ## Usernames and passwords for Object ./spectrumscale config perfmon –r on ## Turn on performance sensors for the protocol nodes. ## EMS GUI picks up sensor data once protocols are deployed ./spectrumscale node list ## Lists out the node config (ESS IO nodes never show up here) ./spectrumscale config protocols ## Shows the protocol config
J) Installation phase of IBM Spectrum Scale installation toolkit
- Run the installation toolkit installation
precheck.
./spectrumscale install --precheck
- Run the installation toolkit installation
procedure.
./spectrumscale install
- Install GPFS, call home, performance monitoring, license RPMs on each node specified to the installation toolkit. The EMS and I/O server nodes are not acted upon by the installation toolkit.
- Add nodes to the cluster (protocol, client, NSD).
- Add non-ESS NSDs, if desired.
- Start GPFS and mount all file systems on the newly added nodes.
- Configure performance monitoring sensors.
- Set client or server licenses.
GPFS configuration parameters such as pagepool, maxFilesToCache, verbsPorts need to be set up manually. You can do this after completing the installation phase or after completing the deployment phase. For more information about these parameters, see GPFS configuration parameters for protocol nodes.
K) Deployment phase of IBM Spectrum Scale installation toolkit
- Run the installation toolkit deployment
precheck.
./spectrumscale deploy --precheck
- Run the installation toolkit installation
procedure.
./spectrumscale deploy
- Install SMB, NFS, and object RPMs on each protocol node specified to the installation toolkit.
- Enable one or more protocols.
- Assign CES IPs. IBM Spectrum Scale code aliases these IPs to the CES base network adapter configured during step G.
- Enable authentication for file or object.
- Create additional file systems using non-ESS NSD nodes. You must run installation first to add more non-ESS NSDs.
- Add additional protocol nodes. You must run installation first to add more nodes and then run deployment for the protocol specific piece.
L) Tune the protocol nodes as desired
Protocol nodes should already be tuned with the same tuned and sysctl settings, and udev rules as the I/O server nodes. For more information, see OS tuning for RHEL 7.4 PPC64LE protocol nodes.
- RDMA. If IB RDMA is in use (check using mmlsconfig verbsRDMA), issue mmlsconfig and verify that the verbPorts parameter refers to the correct ports on each protocol node.
- pagepool. Use mmlsconfig to view the pagepool
settings of the EMS and I/O server nodes. The protocol nodes do not have
pagepool defined at this point. Define pagepool using
mmchconfig -N cesNodes pagepool=XX command.
Where XX is typically 25% the system memory for protocol nodes in a cluster containing ESS. For more information, see GPFS configuration parameters for protocol nodes.
- maxFilesToCache. Use mmlsconfig to view the
maxFilesToCache settings of the EMS and I/O server nodes. The protocol nodes do
not have maxFilesToCache defined at this point. Define
maxFilesToCache using mmchconfig -N cesNodes
maxFilesToCache=XX command.
Where XX is typically 2M for protocol nodes in a cluster containing ESS. For more information, see GPFS configuration parameters for protocol nodes.
M) GUI configuration
- Enable and start
gpfsgui:
systemctl enable gpfsgui.service systemctl start gpfsgui
Note: If the GUI was started before protocol nodes were installed, then restart the GUI with systemctl restart gpfsgui command. - To launch the ESS GUI in a browser, go to: https://EssGuiNode where ESSGuiNode is the host name or IP address of the management server node for GUI access. To log in, type admin in the User Name field and your password in the Password field on the login page. The default password for admin is admin001. Walk through each panel and complete the GUI Setup Wizard.
N) Call home setup
Now that protocol nodes are deployed, call home needs to be configured.
- Check if call home has already been configured and if so, record the settings. Reconfiguring
call home might require entering the settings again.
- Check the hardware call home settings on the EMS
node.
gsscallhomeconf --show -E ems1
- Check the software call home setup.
mmcallhome info list
- Check the hardware call home settings on the EMS
node.
- Set up call home from the EMS
node.
gsscallhomeconf -N ems1,gss_ppc64,ces_ppc64 --suffix=-hs -E ems1 --register all --crvpd
O) Movement of quorum or management function to protocol nodes and off EMS or I/O nodes
Quorum and management functions can be resource intensive. In an ESS cluster that also has extra nodes, such as protocols, within the same cluster, it is recommended to move these functions to the protocol nodes. For information on changing node designation, see mmchnode command.