Upgrading IBM Storage Scale protocol nodes

Use these steps to upgrade protocol nodes in an IBM Storage Scale cluster.

Attention:

S3 users must run I/O workloads with an increased value for the variables AWS_MAX_ATTEMPTS and AWS_MAX_RETRY to sustain the high availability. Because CES IP movement might take time based on the active I/O from multiple protocols, if enabled in the cluster. Refer AWS CLI documentation for how to set these.
Before the cluster is ready for upgrade, back up the S3 configuration (if the S3 protocol is enabled in the cluster). For more information, see Manual backup of the S3 configuration data.

Perform the following steps for each protocol node in the cluster:

Stop all user activity in the file systems on the node that you are upgrading.
Stop any protocol services that are running on the node:
1. Issue the following command to suspend Cluster Export Services (CES) and stop the protocol services on the node:
```
mmces node suspend --stop
```
  If you are upgrading from IBM Storage Scale version 5.0.2.0 or earlier, issue the following commands to suspend the protocol node and stop the protocol services:
```
mmces node suspend
mmces service stop Protocol
```
2. If you are upgrading from IBM Storage Scale version 5.0.1.2 or earlier to version 5.0.2 or later, issue the following command to ensure that the mmunmount of the CES file system completes successfully:
```
mmcesmonitor stop
```
If the call home service is enabled on the cluster, issue the following command to disable it:
```
mmcallhome capability disable
```

Important: Back up your file systems to ensure that your data is protected.

If file systems are mounted on the node, issue the following command to unmount them:
```
mmunmount all
```
Do not use the -f option of the mmunmount command.
Shut down IBM Storage Scale on the node by issuing the following command:
```
mmshutdown
```
Note: You do not need to close mmccrmonitor and other mm processes because they are handled by IBM Storage Scale upgrade post-install scripts.
Verify that the GPFS daemon has terminated and that the kernel extensions are unloaded by using mmfsenv -u. If the mmfsenv -u command reports that it cannot unload the kernel extensions because they are busy, then the installation can proceed, but the node must be rebooted after the installation. Kernel extensions are busy means that a process has a current directory in some GPFS file system directory or it has an open file descriptor. The Linux utility lsof can identify the process and that process can then be killed. Retry mmfsenv -u after killing the process and if the command succeeds, then a reboot of the node can be avoided.
Note:
- The mmfsenv -u command is normally used with assistance from support or for cases where its usage is specifically documented such as in this scenario.
- The mmfsenv -u command only checks the local node from which it is run, not all the nodes in the cluster.
Extract the IBM Storage Scale software packages as described in the topic Extracting the IBM Storage Scale software on Linux nodes. For more information about the location of extracted packages, see Location of extracted packages.
Change to the directory that contains the IBM Storage Scale package files.
- If Red Hat® Enterprise Linux® or SUSE Linux Enterprise Server (SLES) is installed on the node, issue the following command:
```
cd /usr/lpp/mmfs/<x.x.x.x>/gpfs_rpms
```
- If Ubuntu Linux is installed on the node, issue the following command:
```
cd /usr/lpp/mmfs/<x.x.x.x>/gpfs_debs
```
Where <x.x.x.x> is the version level of IBM Storage Scale that you are upgrading to.
Upgrade the protocol node by using one of the following set of steps for depending on the operating system and the product edition that is installed on the node.
Note: The following commands are specific to the current release. If you are upgrading to an earlier release, see the documentation for that release.
Tip: To determine the currently installed edition, issue the following command:
```
mmlslicense -L
```
Note:
- Starting with IBM Storage Scale 5.1.0 or later, there are no separate packages for call home, since call home is integrated into the gpfs.base package.
- If you are upgrading to IBM Storage Scale 5.1.0 or later, you do not need to upgrade the gpfs.kafka package.
- Red Hat Enterprise Linux 7.x, 8.x, and 9.x
  Note: For rpm packages specific to operating systems such as rhel7, rhel8, and rhel9, you need to install them from specific directories such as rhel7/, rhel8/, and rhel9/.
  - If IBM Storage Scale Standard Edition or IBM Storage Scale Data Access Edition is installed on the node, run the following command. The entire command must be on one line.
```
rpm -Fvh ./gpfs.gskit*.rpm ./gpfs.gpl*.rpm ./gpfs.docs*.rpm ./gpfs.compression*.rpm 
./gpfs.base*.rpm ./gpfs.msg.en_US*.rpm 
./gpfs.java*.rpm ./rhel/gpfs.librdkafka*.rpm
```
  - If IBM Storage Scale Advanced Edition or IBM Storage Scale Data Management Edition or IBM Storage Scale Developer Edition is installed on the node, run the following command. The entire command must be on one line:
```
rpm -Fvh ./gpfs.msg.en_US*.rpm ./gpfs.java*.rpm ./gpfs.gskit*.rpm 
./gpfs.gpl*.rpm ./gpfs.docs*.rpm ./gpfs.crypto*.rpm ./gpfs.compression*.rpm 
./gpfs.base*.rpm ./gpfs.adv*.rpm ./rhel/gpfs.librdkafka*.rpm
```
- SUSE Linux Enterprise Server (SLES)
  - If IBM Storage Scale Standard Edition or IBM Storage Scale Data Access Edition is installed on the node, run the following command. The entire command must be on one line:
```
rpm -Fvh ./gpfs.gskit*.rpm ./gpfs.gpl*.rpm ./gpfs.docs*.rpm ./gpfs.compression*.rpm 
./gpfs.base*.rpm ./gpfs.msg.en_US*.rpm 
./gpfs.java*.rpm
```
  - If IBM Storage Scale Advanced Edition or IBM Storage Scale Data Management Edition is installed on the node, run the following command. The entire command must be on one line:
```
rpm -Fvh ./gpfs.msg.en_US*.rpm ./gpfs.java*.rpm ./gpfs.gskit*.rpm 
./gpfs.gpl*.rpm ./gpfs.docs*.rpm ./gpfs.crypto*.rpm ./gpfs.compression*.rpm 
./gpfs.base*.rpm ./gpfs.adv*.rpm 
```
- Ubuntu Linux
  - If IBM Storage Scale Standard Edition or IBM Storage Scale Data Access Edition is installed on the node, run the following command. The entire command must be on one line:
```
apt-get install --only-upgrade ./gpfs.gskit*.deb ./gpfs.gpl*.deb 
./gpfs.docs.deb ./gpfs.compression.deb 
./gpfs.base*.deb ./gpfs.msg.en-us*.deb 
./gpfs.java*.deb ./ubuntu/gpfs.librdkafka*.deb
```
  - If IBM Storage Scale Advanced Edition or IBM Storage Scale Data Management Edition is installed on the node, run the following command. The entire command must be on one line:
```
apt-get install --only-upgrade ./gpfs.gskit*.deb ./gpfs.gpl*.deb 
./gpfs.docs*.deb ./gpfs.crypto*.deb ./gpfs.compression*.deb 
./gpfs.base*.deb ./gpfs.adv*.deb 
./gpfs.msg.en-us*.deb ./gpfs.java*.deb 
./ubuntu/gpfs.librdkafka*.deb
```
If needed, install or upgrade the following packages.
1. For all editions, if you are upgrading from IBM Storage Scale 4.2.3.x or earlier, install the gpfs.compression package:
  - If Red Hat Enterprise Linux or SUSE Linux Enterprise Server Linux is installed on the node, issue the following command from the gpfs_rpms directory:
```
rpm -ivh gpfs.compression*.rpm
```
  - If Ubuntu Linux is installed on the node, issue the following command from the gpfs_debs directory:
```
apt-get install gpfs.compression*.deb
```
2. For all editions, for the gpfs.ext package:
  - If you are upgrading to IBM Storage Scale 5.0.1 or earlier, ensure that you upgrade the gpfs.ext package.
  - In versions 5.0.2 and later, you do not need to upgrade the gpfs.ext package. The package is not present because its functionality is merged into other packages.
    Attention: On Ubuntu Linux nodes, in version 5.0.2 and later, you might see an older version of the gpfs.ext package. You can safely remove this package by issuing the following command:
    dpkg -P gpfs.ext
3. For IBM Storage Scale Advanced Edition and IBM Storage Scale Data Management Edition, if you plan to use the AFM-based Asynchronous Disaster Recovery feature, the file encryption feature, or the transparent cloud tiering feature, you must install the following packages:
  - gpfs.adv
  - gpfs.crypto
After a successful upgrade, rebuild the GPFS portability layer (GPL). For more information, see Building the GPFS portability layer on Linux nodes.
Upgrade management GUI packages. For more information, see Manually upgrading the IBM Storage Scale management GUI.
Start IBM Storage Scale on the node by issuing the following command:
```
mmstartup
```
Mount any file systems that are not mounted automatically when the IBM Storage Scale daemon starts.
Upgrade NFS packages on the protocol node. For more information, see Upgrading NFS packages.
Upgrade S3 packages on the protocol node. For more information, see Upgrading S3 packages.
If you disabled the call home service in a preceding step, issue the following command to enable it:
```
mmcallhome capability enable accept
```

Repeat the preceding steps on all protocol nodes in the cluster. After these steps are done for all protocol nodes in the cluster, proceed with the following steps.

If Swift Object is enabled on cluster then do object version sync and start Swift object on all protocol nodes. For more information, see Upgrading object packages.
Upgrade SMB packages on all protocol nodes. For more information, see Upgrading SMB packages.
If you are upgrading from IBM Storage Scale 5.1.0.x or earlier to IBM Storage Scale 5.1.1.x or later, remove the gpfs.protocols-support package, if it is present.
- For Red Hat Enterprise Linux and SLES:
```
mmdsh -N ProtocolNodeList rpm -ev --nodeps gpfs.protocols-support
```
- For Ubuntu (The entire command must be on one line):
```
mmdsh -N ProtocolNodeList apt-get remove gpfs.protocols-support
```
ProtocolNodeList is the list of all protocol nodes in the cluster.
Do the protocol authentication configuration depending on your setup. For more information, see Protocol authentication configuration changes during upgrade.
Upgrade the license package corresponding to the installed IBM Storage Scale product edition on all nodes in the cluster.
- For Red Hat Enterprise Linux and SLES:
```
mmdsh -N NodeList rpm -Fvh /usr/lpp/mmfs/N.N.N.N/gpfs_rpms/gpfs.license-xxx*.rpm
```
- For Ubuntu:
```
mmdsh -N NodeList apt-get install --only-upgrade /usr/lpp/mmfs/N.N.N.N/gpfs_debs/gpfs.license-xxx*.deb
```
  - NodeList is the list of all nodes in the cluster.
  - N.N.N.N is the product version. For example, 5.0.2.0.
  - xxx is the indicator of the product edition:
    - da: Data Access Edition
    - dm: Data Management Edition
    - dev: Developer Edition
    - std: Standard Edition
    - adv: Advanced Edition
If the product edition is Advanced, Data Management, or Developer and file audit logging or clustered watch folder is enabled in the cluster, see Upgrade paths and commands for file audit logging and clustered watch folder.
- If you are upgrading to IBM Storage Scale 5.1.2 or later and you have previously run the mmmsgqueue config --remove-msgqueue command, no further action is required.
- If you are upgrading to IBM Storage Scale 5.1.2 or later and you have previously not run the mmmsgqueue config --remove-msgqueue command, do one of the following steps.
  - If you have never enabled the message queue, manually remove the gpfs.kafka package.
  - In all other cases, run the mmmsgqueue config --remove-msgqueue command. This command removes the gpfs.kafka package at the end of the cleanup.

When all the nodes in the cluster are successfully upgraded to the new level, the next step is to complete the upgrade. For more information, see Completing the upgrade to a new level of IBM Storage Scale.