On the worker nodes

As a first step, upgrade IBM Storage Scale on the worker nodes. Perform the following steps to upgrade IBM Storage Scale on worker nodes:

Cordon the worker node so that scheduling is disabled.
```
kubectl cordon <node>
```
Move any workload off the worker node.
Remove the scale label from the node.
```
kubectl label node <node> scale-
```
Prepare the worker node to shut down IBM Storage Scale.
1. Check for open files.
```
lsof <filesystem>
```
2. Check for mounted kernel modules.
```
lsmod | grep mm
```
3. If there is any mm* present, then unmount and shut down file systems on the worker node. You can use the following options to view the details of the mounted file systems:
  - Check the GPFS status:
```
/usr/lpp/mmfs/bin/mmgetstate
```
    Note: The GPFS state should be active.
  - List the mounted file systems across all nodes by issuing the following command:
```
/usr/lpp/mmfs/bin/mmlsmount all
```
  - List disk space usage to see what file systems are mounted by issuing the df command:
```
df
```
4. Unmount all file systems for the current node.
```
/usr/lpp/mmfs/bin/mmunmount all
```
5. Shut down GPFS on the current node.
```
/usr/lpp/mmfs/bin/mmshutdown
```
6. Check for open files.
```
lsof <file system>
```
7. Check for mounted kernel modules.
```
lsmod | grep mm
```
  Note:
  - All file systems must be unmounted, and GPFS must be shut down. Continue to step 5 to proceed with IBM Storage Scale upgrade.
  - If there are any file systems or mounted kernel modules (mm*) present, then you require a reboot of the worker node to clean up the state. Ensure autoload is set to off for the node before rebooting.
8. Set autoload to off.
```
/usr/lpp/mmfs/bin/mmchconfig autoload=no -N <node>
```
9. Reboot the worker node.
```
reboot
```
Upgrade IBM Storage Scale by using the toolkit, set the worker node as an offline node, and exclude the other nodes.
After the upgrade is completed, do the following steps:
1. Log on to the worker node and ensure that autoload is set back to on.
```
/usr/lpp/mmfs/bin/mmchconfig autoload=yes -N <node>
```
2. Log on to the worker node and start GPFS.
```
/usr/lpp/mmfs/bin/mmstartup
```
3. Uncordon the worker node.
```
kubectl uncordon <node>
```
4. Relabel the node scale.
```
kubectl label node <node> scale=true
```