Upgrading FPO

When the application that runs over the cluster can be stopped, you can shut down the entire GPFS cluster and upgrade FPO. However, if the application over the cluster cannot be stopped, you need to take the rolling-upgrade procedure to upgrade nodes.

During this kind of upgrade, the service is interrupted. In production cluster, service interrupt is not accepted by the customers. If such cases, you need to take the rolling upgrade to upgrade node by node (or failure group by failure group) while keeping GPFS service up in the other nodes.

The guide for rolling upgrade is as follows:

Only upgrade nodes from the same failure group at the same time slot; not operate nodes from two or more failure groups because bringing nodes from more than 1 failure groups will make your data exposed in data lost risk.
Not break the quorum relationship when bringing down the nodes from one failure group. Before you bring down the nodes in one failure group, you need to check the quorum node. If bringing the quorum node in the to-be-operated failure group will break the quorum relationship in the cluster, you need to exclude that node for the rolling upgrade of the failure group.

Prerequisites

Ensure that all disks are in a ready status and up availability. You can check by issuing the mmlsdisk fs-name -L command.
Verify whether the upgraded-to GPFS version is compatible with the running version from IBM Spectrum Scale FAQ in IBM® Knowledge Center (www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html). For example, you cannot upgrade GPFS from 3.4.0.x directly into 3.5.0.24. You need to upgrade to 3.5.0.0 first and then upgrade to the latest PTF. You also need to verify whether the operating system kernel version and the Linux® distro version are compatible with GPFS from IBM Spectrum Scale FAQ in IBM Knowledge Center (www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html).
Find a time period when the whole system work load is low or reserve a maintenance time window to do the upgrade. When cluster manager or file system manager is down intentionally or accidentally, another node is elected to take the management role. But it takes time to keep the cluster configuration and the file system data consistent.
When a file system manager is elected by cluster manager, it does not change even if the file system is unmounted in this node. If the file system is mounted in other nodes, it is also ‘internal’ mounted in the file system manager. This does not affect your ability to unload the kernel modules and upgrade GPFS without a reboot.

Upgrade FPO as follows:

Disable auto recovery for disk failure
To do a rolling upgrade of GPFS, you must shut down GPFS in some nodes. Disks in those nodes are unreachable during that time. It is better to handle this disk down manually with the following step.

Run mmchconfig restripeOnDiskFailure=no -i in any node in cluster to disable auto recovery for disk failure. It is necessary to include -i option for this change to take effect immediately and permanently. GPFS restripeOnDiskFailure is a cluster-wide configuration. So you need to run it only once in any one node in your cluster.
Get the list of nodes for this upgrade cycle
Each upgrade cycle, you can only upgrade GPFS in nodes whose disks in it have the same first two numbers in failure group vector. Save node list in file nodeList, one node name per line in it. For general information on how to specify node names, see Specifying nodes as input to GPFS commands.
Get disk list in nodes for this upgrade cycle
Get all disks attached in nodes for this upgrade cycle. Save it in file diskList. Each line in file diskList saves a disk name. mmlsdisk –L command can show which disks belong to the nodes you want to upgrade in this cycle.
Stop applications by using GPFS file system
Confirming that there is no application which still opens file in GPFS file system. You can use lsof or fuse command to check whether there is still an open instance active for file in GPFS file system.
Unmount GPFS file system
Unmount GPFS file system in all nodes for this upgrade cycle through command:
```
mmumount <fsName> -N <nodeList>
```
Confirming file system has already unmounted in all related nodes through command:
```
mmlsmount <fsName> -L
```
Suspend disks
Suspend all attached disks in nodes for this upgrade cycle to make sure that GPFS does not try to allocate new data block from these disks. GPFS can still read valid data block from suspended disk
```
mmchdisk <fsName> suspend -d <diskList>
```
Confirming disks are suspended properly through command:
```
mmlsdisk <fsName>
```
Shut down GPFS
Shut down GPFS in all nodes for this upgrade cycle through command:
```
mmshutdown -N <nodeList>
```
Confirming GPFS is in down status in these nodes through command:
```
mmgetstate -a
```
Upgrade GPFS
You can upgrade GPFS packages in each node of this upgrade cycle.
For general information on how to install GPFS packages on nodes, see :
Start GPFS
After all packages are ready, you can start up GPFS:
```
mmstartup -N <nodeList>
```
Confirming GPFS are in "Active" status in these nodes through command:
```
mmgetstate -a
```
Resume and start disks
When GPFS is in "Active" status in all nodes, you can resume disks which were suspended intentionally in Step 7.
```
mmchdisk <fsName> resume -a or
```
```
mmchdisk <fsName> resume -d <diskList>
```
When these disks are in "ready" status and if some of these disks are in “down” availability, you can start these disks through the following command:
```
mmchdisk <fsName> start -a or
```
```
mmchdisk <fsName> start -d <diskList>
```
This might take a while since GPFS must do incremental data sync up to keep all data in these suspended disks are up to date. The time it needs depends on how much data has changed when the disks were kept in suspended status. You have to wait for mmchdisk start command to finish to do next step.
Confirming all disks are in ready status and up state through command:
```
mmlsdisk <fsName>
```
Mounts GPFS file system
When all disks in the file system are in up status you can mount file system:
```
mmmount <fsName> -N <nodeList>
```
Confirming GPFS file system is mounted properly through command:
```
mmlsmount <fsName> -L
```
Repeat Step 3 to Step 12 to upgrade GPFS in all nodes in your cluster.
Enable auto recovery for disk failure
Now you can enable auto recovery for disk failure through command:
```
mmchconfig restripeOnDiskFailure=no -i
```
It's necessary to includes -i option for this change to take effect immediately and permanently.
Upgrade GPFS cluster version and file system version
Issue mmchconfig release=LATEST and mmchfs -V compat to ensure the upgrade is successful and the cluster would never revert to the old build for minor GPFS upgrade. It is recommended to use mmchfs –V compat to enable backward-compatible format changes.

For major GPFS upgrade, consult IBM Support Center to verify compatibility between the different GPFS major versions, before issuing mmchfs –V full. For information about specific file system format and function changes when you upgrade to the current release, see File system format changes between versions of IBM Spectrum Scale.