Guidance for upgrading the operating system on IBM Storage Scale nodes

This is a high-level guidance for upgrading the operating system (OS) on your IBM Storage Scale nodes with focus on steps to be performed from the IBM Storage Scale side.

For OS specific upgrade steps, refer to the respective OS documentation.

Warning: If upgrading from an older version of the OS to a newer version requires a reinstallation of the OS, the devices that are used for the install disk must be reviewed carefully before accepting the disk presented by the installer. This is required to avoid situations in which the installer chooses the IBM Storage Scale NSDs and causes file system corruption. Ensure that IBM Storage Scale NSDs are not altered during OS re-installation or upgrade.

Order of upgrade

Important: Before upgrading the OS on a node, ensure that the new OS version that you are planning to upgrade to is supported on the current IBM Storage Scale version. For information on the supported OS versions, see Functional Support Matrices in IBM Storage Scale FAQ.

If the new OS version that you are planning to upgrade to is supported on the current IBM Storage Scale version, upgrading IBM Storage Scale is not mandatory. Proceed with upgrading the OS.
If the new OS version that you are planning to upgrade to is not supported on the current IBM Storage Scale version, do the following:
1. Determine the IBM Storage Scale version on which the new OS version that you are planning to upgrade to is supported by referring to Functional Support Matrices in IBM Storage Scale FAQ.
2. Upgrade IBM Storage Scale.
3. After the IBM Storage Scale upgrade is completed, upgrade to the new OS version.

Prerequisites for upgrading Linux distributions

Download and mount the operating system installation image on each node that is being upgraded.
Create an OS base repository that points to the installation image on each node that is being upgraded.
- If the repository name is the same as an old OS repository, clean the repository cache.
- Verify that the repository works.
If required, download OFED drivers on each node that is being upgraded.
Note the current OS and OFED settings. You might need to restore them after the OS upgrade.
It is assumed that mixed level of OS and drivers across nodes are supported by vendors.
Note the current SELinux and firewall settings. Although an OS upgrade typically does not change these settings, it is recommended to note these settings.
Check the file system space and ensure that /boot has adequate free space.
If you are upgrading protocol nodes to Red Hat Enterprise Linux® 8.x, you must perform a cleanup of object services and the related packages before attempting the upgrade. For more information, see Instructions for removing object protocol packages when upgrading protocol nodes to Red Hat Enterprise Linux 8.x.

OS upgrade flow

Do the following steps on each node on which the OS is being upgraded.

If a protocol node is being upgraded, do the following:

Suspend the protocol node and stop the protocol services running on the node by issuing the following command.
```
mmces node suspend --stop
```

Wait for the protocol service monitor to change to the suspended state.

# mmhealth node show ces
Component             Status         Status Change        Reasons
--------------------------------------------------------------------------------------
CES                   SUSPENDED      30 min. ago           -
AUTH                  SUSPENDED      30 min. ago           -
AUTH_OBJ              SUSPENDED      30 min. ago           -
BLOCK                 SUSPENDED      30 min. ago           -
CESNETWORK            SUSPENDED      30 min. ago           -
NFS                   SUSPENDED      30 min. ago           -
OBJECT                SUSPENDED      31 min. ago           -
SMB                   SUSPENDED      30 min. ago           -

Disable the GPFS autoload setting that causes GPFS to be started whenever the operating system reboots by issuing the following command.
```
mmchconfig autoload=no -N NodeName
```
Unmount GPFS file systems on the node by issuing the following command.
```
mmumount all
```
Stop GPFS on the node by issuing the following command.
```
mmshutdown
```
Check if there are any residual GPFS processes running and kill them, if needed.
If required, uninstall the OFED driver using the bundled uninstallation script in the OFED package.
Perform the OS upgrade steps according to the respective documentation.
Reboot the node when the OS upgrade steps are completed.
If required, install the new OFED driver and reboot the node after OFED driver installation.
Restore any OS and OFED settings (if applicable) that have been overwritten during the upgrade.
If a leapp upgrade from Red Hat Enterprise Linux 7.x to 8.x is performed, make sure to restore some of the removed packages and follow the details in Guidance for Red Hat Enterprise Linux 8.x on IBM Storage Scale nodes.
Rebuild the GPFS portability layer. For more information, see Building the GPFS portability layer on Linux nodes.
Verify that no conflicting services have been started or enabled during the OS upgrade. For example, some versions of RHEL autostart a non-IBM® version of NFS after upgrade and it conflicts with the CES NFS service of IBM Storage Scale.
Verify SELinux and firewall settings.
Enable the GPFS autoload setting, if it was disabled earlier in this procedure, by issuing the following command.
```
mmchconfig autoload=yes -N NodeName
```
Restart GPFS on the node by issuing the following command.
```
mmstartup
```
Verify that all file systems are mounted on the node by issuing the following command.
```
mmlsmount all
```
If a protocol node is being upgraded, do the following:
1. Resume the protocol node and start the protocol services on the node by issuing the following command.
```
mmces node resume --start
```
2. Ensure that CES IPs are hosted by this node and that all services are healthy by issuing the following commands.
```
mmces address list
mmces service list -a
mmces node list
mmhealth node show CES
mmhealth cluster show
```

Repeat these steps on all nodes on which the OS needs to be upgraded.