Upgrading S3 packages

Use these steps on each protocol node one by one to upgrade IBM Storage Scale S3 packages.

Attention:

If one or more CES nodes cannot be upgraded, then issue the command mmces node suspend -N Node[,Node...] --stop to suspend the nodes and stop all the protocol services that are running on these nodes. This suspension ensures that the S3 service does not automatically start when the nodes are available again.
All protocol nodes must also be upgraded to the newer code level along with base GPFS.
S3 users must run I/O workloads with an increased value for the variables AWS_MAX_ATTEMPTS and AWS_MAX_RETRY to sustain the high availability. Because CES IP movement might take time based on the active I/O from multiple protocols, if enabled in the cluster. Refer AWS CLI documentation for how to set these.
Back up S3 protocol configuration before an offline or online upgrade. For more information, see Manual backup of the S3 configuration data.

Complete the following steps on all nodes:

Suspend the node by issuing the following command.
```
mmces node suspend --stop
```
Note: Suspending nodes triggers IP address reassignment and client failover.
From IBM Storage Scale 5.2.2 onward, the S3 protocol stack logging mechanism has some improvements. If you are using IBM Storage Scale 5.2.1.x, copy the /etc/rsyslog.d/noobaa_syslog.conf to /etc/rsyslog.d/noobaa_syslog_521.conf.
Upgrade S3 packages on the node.
- For Red Hat Enterprise Linux®, issue the following command.
```
rpm -Uvh gpfs.mms3 noobaa-core 
```
When you upgrade from IBM Storage Scale 5.2.1.x, the following files are listed in the /etc/logrotate.d directory:
```
-rw-r--r-- 1 root root 971 Oct 22 23:21 noobaa-logrotate
-rw-r--r-- 1 root root 791 Oct 22 23:56 logrotate_noobaa.conf
```
where, the noobaa-logrotate is the new file created, update this file with any modifications performed in the env previously (that is, in logrotate_noobaa.conf).
Rename the logrotate_noobaa.conf as logrotate_noobaa_521.conf.
Copy the noobaa_syslog.conf.rpmnew file, which NooBaa creates after the upgrade, to the noobaa_syslog.conf file in the /etc/rsyslog.d directory.
Resume the node by issuing the following command.
```
mmces node resume --start
```
Attention: If a cluster is upgrading from an earlier release of IBM Storage Scale 5.2.3.0, issue the following command to disable the NooBaa service:
```
systemctl disable noobaa
```
The NooBaa service does not start automatically after a reboot.
After upgrading S3 on all nodes, issue the following command only one time from any one node to finish the S3 protocol upgrade.
```
mms3 upgrade start [--upgrade_version]
```
Attention: After issuing the mms3 upgrade start command, if the mms3 command is used on nodes where S3 service is not upgraded, then it might lead to unexpected behavior.
Note:
If the S3 service on some CES nodes was not upgraded, whenever these nodes are available for upgrade complete:
1. For each non-upgraded CES node, complete steps 1 through 6 in this procedure.
2. After upgrading S3 on non-upgraded nodes, run the following command from one of the nodes:
  mms3 upgrade start [--upgrade_version]