Installation and upgrade related information and checklists

The respective versions for the core components in this release of ESS are as follows:

• Supported architectures: PPC64BE and PPC64LE
• IBM Spectrum Scale™: 5.0.1.2
• xCAT: 2.13.14
• HMC: 860 SP2
• System firmware: SV860_138 (FW860.42)
• Red Hat Enterprise Linux: 7.4 (PPC64BE and PPC64LE)
• Kernel: 3.10.0-693.35.1.el7
• Systemd: 219-42.el7_4.11
• Network Manager: 1.8.0-12.el7_4
• OFED: MLNX_OFED_LINUX-4.3-1.0.1.1
• OFED2 (Packaged to support Mellanox CX-2 adapter): MLNX_OFED_LINUX-4.1-4.1.6.0
• IPR: 18518200
• ESA: 4.3.0-4

PPC64BE

• Standard Edition
• Advanced Edition
• Data Management Edition

PPC64LE

• Standard Edition
• Data Management Edition

• It is advised that when performing normal maintenance operations (or upgrades) that you disable autoload first.

  mmchconfig autoload=no

  Once the maintenance operation (or upgrade) is complete, re-enable autoload.

  mmchconfig autoload=yes

• By default, file systems must only be mounted on the management server node (EMS). Do not mount the file system on any other ESS nodes besides the EMS (where the primary GUI runs) which is mandatory for the GUI to function correctly.
• It is advised that you disable automount for file systems when performing an upgrade to ESS 5.3.1 or later.

  mmchfs Device -A no

  Device is the device name of the file system.

• Do not configure more than 5 failure groups in a single file system.
• Consider moving all supported Infiniband devices to the Datagram mode (CONNECTED_MODE=no) and enhanced IPoIB during upgrade to ESS 5.3.1.x. For more information, see ESS networking considerations.
• If you have 40Gb adapters, enable flow control on your switch. Consider doing the same for 100Gb adapters.
• RDMA over Ethernet (RoCE) is not supported.
• Sudo on the ESS nodes is not supported.
• Enabling the firewall on any ESS node is not supported.
• Enabling SELinux on any ESS node is not supported.
• Running any additional service or protocols on any ESS node is not supported.
• Consider moving quorum, cluster, and file system management responsibilities from the ESS nodes to other server license nodes within the cluster.
• It is not required that the code levels match during a building block addition. Be mindful of changing the release and file system format in mixed IBM Spectrum Scale environments.
• You must take down the GPFS cluster to run firmware updates in parallel.
• Do not independently update IBM Spectrum Scale (or any component) on any ESS node unless specifically advised from the L2 service. Normally this is only needed to resolve an issue. Under normal scenarios it is advised to only upgrade in our tested bundles.
• It is acceptable for LBS or customers to update any security errata available from Red Hat Network (RHN). Only components checked and protected by ESS (kernel, network manager, systemd) must not be modified unless advised by the IBM service.
• Client node deployment is not supported from the ESS management node.
• You must deploy or add building blocks from an EMS with the same architecture. There must be a dedicated EMS for each architecture (PPC64BE or PPC64LE).
• If running in a mixed architecture environment, the GUI and collector are recommended to run on the PPC64LE EMS node.
• Modifying any ESS nodes as a proxy server is not supported.
• Offline upgrades from any prior ESS version are supported.
• File audit logging is not supported on protocol nodes.
• Hardware call home is not supported on 5148-22L protocol nodes
• GPFS configuration parameters' values, other than defaults, are not automatically set on protocol nodes.

The required Red Hat components and md5sum are:

• Red Hat Enterprise Linux 7.4 ISO

  0cfb07d327e94c40fceb2c3da09d46e1  rhel-server-7.4-ppc64-dvd.iso
  6ae2077d4e223e29ed820ea0ff68aded  rhel-server-7.4-ppc64le-dvd.iso

  
• Network manager version : 1.8.0-12.el7_4

  56007 8751 netmanager-5311-2018-1755-LE.tar.gz
  4435 12576 netmanager-5311-2018-1755-BE.tar.gz

  
• Systemd version: 219-42.el7_4.11

  14589 7151 systemd-5311-RHBA-2018-1151-LE.tar.gz
  39566 8172 systemd-5311-RHBA-2018-1151-BE.tar.gz

  
• Kernel version: 3.10.0-693.35.1.e17

  31385 68353 kernel-5311-RHBA-2018-2158-LE.tar.gz
  1721 69271 kernel-5311-RHBA-2018-2158-BE.tar.gz

  

Customers or business partners can download the required Red Hat components from Red Hat Network using the customer license. For more information, see:

• Obtaining kernel for system upgrades
• Obtaining systemd update for system upgrades
• Obtaining Network Manager updates for system upgrades

Available PPC64BE packages:

ESS_STD_BASEIMAGE-5.3.1.1-ppc64-Linux.tgz
ESS_ADV_BASEIMAGE-5.3.1.1-ppc64-Linux.tgz
ESS_DM_BASEIMAGE-5.3.1.1-ppc64-Linux.tgz

Available PPC64LE packages:

ESS_STD_BASEIMAGE-5.3.1.1-ppc64le-Linux.tgz
ESS_DM_BASEIMAGE-5.3.1.1-ppc64le-Linux.tgz

Once downloaded and placed in /home/deploy, untar and uncompress the package to view the contents. For example, for the standard edition PPC64LE package, use the following command:

tar -xvf ESS_STD_BASEIMAGE-5.3.1.1-ppc64le-Linux.tgz

The BASEIMAGE tar file contains the following files that get extracted with the preceding command:

• ESS_5.3.1.1_ppc64le_Release_note_Standard.txt: This file contains the release notes for the latest code.
• gss_install-5.3.1.1_ppc64le_standard_20180814T204615Z.tgz: This .tgz file contains the ESS code.
• gss_install-5.3.1.1_ppc64le_standard_20180814T204615Z.md5: This .md5 file to check the integrity of the tgz file.

The following upgrade paths are supported:

• ESS version 5.1.x, 5.2.x, and 5.3.0.x to version 5.3.x.y on PPC64BE.
• ESS version 5.1.x, 5.2.x, and 5.3.0.x to version 5.3.x.y on PPC64LE.

Before you arrive at a customer site, it is advised that you perform the following tasks:

	Obtain the kernel, systemd, networkmanager, RHEL ISO (Provided by ESS development or L2 | Service), and ESS tarball (FixCentral). Verify that the checksum match with what is listed in this document. Also ensure that you have the correct architecture packages (PPC64LE or PPC64BE).
	Ensure that you read all the information in the ESS Quick Deployment Guide. Make sure that you have the latest copy from the IBM Knowledge Center and the version matches accordingly. You should also refer to the related ESS 5.3.1 documentation in IBM Knowledge Center.
	Obtain the customer RHEL license.
	Contact the local SSR and ensure that all hardware checks have been completed. Make sure all hardware found to have any issues has been replaced.
	If the 1Gb switch is not included in the order, contact the local network administrator to ensure isolated xCAT and FSP VLANs are in place.
	Develop an inventory and plan for how to upgrade, install, or tune the client nodes.
	Upgrade the HMC to SP2 if doing a PPC64BE installation. This can be done concurrently. The SSR or the customer might be able to do this ahead of time.
	Consider talking to the local network administrator regarding ESS switch best practices, especially the prospect of upgrading the high-speed switch firmware at some point prior to moving the system into production, or before an upgrade is complete. For more information, see Customer networking considerations.
	Review Elastic Storage Server: Command Reference.
	Review ESS FAQ and ESS best practices.
	Review the ESS 5.3.1 known issues.
	Ensure that all client node levels are compatible with the ESS version. If needed, prepare to update the client node software on site and possibly other items such as the kernel and the network firmware or driver.
	Power down the storage enclosures, or remove the SAS cables, until the gssdeploy -x operation is complete. For new installations, it is recommended to use the Fusion mode. For more information, see Elastic Storage Server 5.2 or later: Fusion Mode.
	If adding an PPC64LE building block to an existing PPC64BE building block, carefully review Considerations for adding PPC64LE building blocks to ESS PPC64BE building blocks.
	If installing protocol nodes, carefully review Protocol nodes deployment and upgrade.
	Find out if the customer has legacy network adapters (CX-2, ConnectX-EN). If so, be prepared to use the alt-mofed flow (4.1) supplied within this document.
	Carefully study the network diagram for the architecture used. For more information, see ESS networking considerations and 5148-22L protocol node diagrams.
	It is recommended to use a larger block size with IBM Spectrum Scale 5.0.0 or later, even for small I/O tasks. Consult the documentation carefully.

After the installation is completed, it is advised that you verify the following:

	Hardware and software call home have been set up and tested. If applicable, consider postponing the call home setup until the protocol nodes are deployed. For more information, see Drive call home in 5146 and 5148 systems. For information about HMC call home (Server PPC64BE Only), see Configuring HMC Version 8.8.3 and Later for Call Home. For call home support information, see Support for hardware call home. For software call home information, see Software call home.
	GUI has been set up and demonstrated to the customer. If applicable, consider postponing the GUI setup until the protocol nodes are deployed.
	GUI SNMP and SMTP alerts have been set up, if desired.
	The customer RHEL license is registered and active.
	No issues have been found with mmhealth, GUI, gnrhealthcheck, gssinstallcheck, serviceable events.
	No SAS width or speed issues have been found.
	Client nodes are properly tuned. For more information, see Adding IBM Spectrum Scale nodes to an ESS cluster.
	It is advised that you turn on autoload to enable GPFS to recover automatically in case of a daemon problem. mmchconfig autoload=yes
	Connect all nodes to Red Hat Network (RHN).
	Update any security related erratas from RHN if the customer desires (yum –y security).
	Ensure that you have saved a copy of the xCAT database off to a secure location.
	Install or upgrade the protocols. For more information, see Upgrading a cluster containing ESS and protocol nodes.
	Ensure (if possible) that all network switches have had the firmware updated.
	IBM Spectrum Scale release level and file system format have been updated, if applicable.

For help with the following topics, and many others that are unlisted, contact L2 Service.

• Adding a building block (same architecture or LE<->BE)
• Restoring a management server
• Part upgrades or replacements
• VLAN reconfiguration on the 1Gb switch

• SSR checkout complete
• LBS arrival on site
• Plug-n-Play mode demonstrated
• Decisions made on block size, host names, IP addresses (/etc/hosts generated)
• Check high speed switch settings or firmware
• Firmware updated on ESS nodes
• Fusion mode used to bring the system to network bond creation
• Network bonds created
• Cluster created
• Recovery groups, NSDs, file system created
• Stress test performed
• Final checks performed
• GUI setup (w/SNMP alerts if desired)
• Call home setup
• Nodes attached to RHN and security updates applied

Proceed to install the protocol nodes, if applicable.

• SSR checkout is complete
• Check high speed switch settings or firmware
• Ensure that there are no hardware issues
• Ensure client / protocol node compatibility
• Ensure no heavy IO operations are being performed
• Upgrade ESS (rolling upgrade or with cluster down)
  • Always ensure you have quorum (if rolling upgrade)
  • Always carefully balance the recovery groups and scale management functions as you upgrade each node (if rolling upgrade)
• Move the release level and the file system format, if applicable.
• Final checks are performed
• Determine if any mmperfmon changes are required
• Ensure that call home is still working
• Use yum to upgrade any security related errata (yum -y security).