Performing a rolling upgrade

This topic lists the steps to perform a rolling upgrade.

1. Disable auto recovery for disk failure.

   To upgrade a node, shut down IBM Storage Scale running on the node. When IBM Storage Scale is shut down, disks in the node cannot be reached. Instead of letting the disks fail and the automatic recovery initiate, temporarily disable auto recovery.

   Run the mmchconfig restripeOnDiskFailure=no -i command to disable auto recovery for disk failure. With the -i option, the parameter takes effect immediately and permanently. For example, in small clusters, the node number is less than 30 nodes. Therefore, it takes a shorter time for IBM Storage Scale to synchronize the configuration. For large clusters, the node number is in hundreds. Therefore, the time taken to synchronize the configuration is longer. The restripeOnDiskFailure parameter is a cluster-wide configuration.

   After disabling auto recovery, check for auto recovery in the file system manager by running the following commands:

   • If there are multiple file systems in the cluster, run mmlsmgr command to check the fs manager of a single file system.
   • Log in to the fs manager of the file system and run ps -elf | grep -e tschdisk -e tsrestripefs command. If there are processes running, wait for them to complete.
2. Select the nodes that must be upgraded and schedule the time of each upgrade.
   In each upgrade cycle, you can only upgrade IBM Storage Scale on nodes where the disks have the same first two numbers in the failure group. Save the list of nodes in the nodeList file with one node name on each line. Save a list of the disks on the nodes that will be upgraded in this cycle in the diskList file, with each line containing an NSD name. Run the mmlsdisk Device –M command to check which disks belongs to which node.
3. Stop all applications that are using the IBM Storage Scale file system before stopping IBM Storage Scale. To check for open files in the file system, run the lsof or the fuse command.
4. Unmount the IBM Storage Scale file system on all nodes by running the following command:
   mmumount <fsName> -N <nodeList>

   To confirm that the file system has been unmounted on all related nodes, run the following command: mmlsmount <fsName> -L

5. Suspend all disks in the nodes so that IBM Storage Scale does not allocate new data blocks from these disks. IBM Storage Scale can still read data block from suspended disks by running the following command:

   mmchdisk <fsName> suspend -d <diskList>

   To confirm that all disks are suspended properly, run the following command: mmlsdisk <fsName>

6. Shut down IBM Storage Scale on the nodes by running the following command:

   mmshutdown -N <nodeList>

   To confirm IBM Storage Scale has stopped functioning on these nodes, run the following command: mmgetstate -a

   Upgrade IBM Storage Scale packages on each node. For information on how to install IBM Storage Scale packages on node, see the following topics:

   • Installing IBM Storage Scale on Linux nodes and deploying protocols
   • Installing IBM Storage Scale on AIX nodes
   • Installing IBM Storage Scale on Windows nodes

   After everything has been installed and the portability layer has been built, start IBM Storage Scale by running the following command: mmstartup -N <nodeList>

   To confirm that IBM Storage Scale is active on the upgraded nodes, run the following command: mmgetstate -a.

   Resume all the suspended disks by running the following commands: mmchdisk <fsName> resume -a or mmchdisk <fsName> resume -d <diskList>.

   If some of the suspended disks are in the Down availability, start these disks by running the following command: mmchdisk <fsName> start -a or mmchdisk <fsName> start -d <diskList>.

   This may take a while because IBM Storage Scale is performing an incremental data sync up to keep the data in these suspended disks up-to-date. The time taken depends on the data that has been changed while the disks were kept in the Suspended status. Wait for the mmchdisk <fsName> start [. . .] command to finish before moving on to the next step.

   To confirm that all disks are in the ready state, run the following command: mmlsdisk <fsName>.

7. When all the disks in the file system are functioning, mount the file system by running the following command: mmmount <fsName> -N <nodeList>

   Confirm that the IBM Storage Scale file system has mounted by running the following command: mmlsmount <fsName> -L

8. Perform Step through Step to upgrade IBM Storage Scale on all nodes in the cluster.
9. To enable auto recovery for disk failure, run the following command: mmchconfig restripeOnDiskFailure=yes -i

   Ensure that you use the -i option so that this change takes effect immediately and permanently.

10. Upgrade the IBM Storage Scale cluster version and file system version

    If all applications run without any issues, run the mmchconfig release=LATEST command to upgrade the cluster version to the latest. Then, run the mmchfs -V compat command to ensure that the upgrade is successful. To enable backward-compatible format changes, run mmchfs -V compat.

    Note: After running the mmchconfig release=LATEST command, you cannot revert the cluster release version to an older version. After running the mmchfs -V compat command, you cannot revert the file system version to an older version.

    For major IBM Storage Scale upgrade, check IBM Storage Scale FAQ in IBM Documentation or contact scale@us.ibm.com before running the mmchfs –V full command to verify the compatibility between the different IBM Storage Scale major versions. For information about specific file system format and function changes, see File system format changes between versions of IBM Storage Scale.