Online upgrade of IBM Storage Scale Erasure Code Edition by using the installation toolkit

mmvdisk rg list --rg rg_name --fault-tolerance

1. Download IBM Storage Scale Erasure Code Edition self-extracting package from the IBM Storage Scale page on Fix Central.

   The name of IBM Storage Scale Erasure Code Edition self-extracting installation package is similar to Spectrum_Scale_Erasure_Code-5.x.y.z-x86_64-Linux-install.

2. Extract the installation package.

   # ./Spectrum_Scale_Erasure_Code-5.x.y.z-x86_64-Linux-install --textonly

   The installation toolkit gets extracted to the /usr/lpp/mmfs/5.x.y.z/ansible-toolkit/ directory.

   To verify that the extracted package is of IBM Storage Scale Erasure Code Edition, go to the /usr/lpp/mmfs/5.x.y.z/gpfs_rpms directory and check for gpfs.gnr* packages.

3. Change the directory to where the installation toolkit is extracted.

   cd /usr/lpp/mmfs/5.x.y.z/ansible-toolkit/

4. Specify the installer node and the setup type in the cluster definition file.
   The setup type must be ece for IBM Storage Scale Erasure Code Edition.

   ./spectrumscale setup -s InstallerNodeIP -st ece

5. Run the config populate command to populate the cluster definition file with the current cluster configuration.

   ./spectrumscale config populate -N ScaleOutNodeIP

   Note: If the config populate command does not work, you can still use the installation toolkit to populate the cluster configuration by using manual commands such as ./spectrumscale node add.

   Verify the populated configuration of the cluster that is to be upgraded.

   ./spectrumscale node list

   You can exclude nodes from the current upgrade process by using the following command.

    ./spectrumscale upgrade config exclude -N NodeName

   You can list the excluded nodes by using the following command.

   ./spectrumscale upgrade config list

6. Perform the installation toolkit upgrade precheck before the online upgrade.

   ./spectrumscale upgrade precheck

   [ INFO  ] Logging to file: /usr/lpp/mmfs/5.1.y.z/ansible-toolkit/logs/UPGRADE-PRECHECK-dd-mm-yyyy_hh:mm:ss.log
   After a node is upgraded in an ECE setup, certain rebuild and rebalance tasks need to be performed. 
   The default timeout for these tasks is infinite. During the upgrade precheck, you can specify a timeout 
   for these rebuild and rebalance tasks depending on your environment.
   Do you want to continue with default timeout [Y/n]: n
   Please specify a timeout value in minutes : 30
   

   Note: It is recommended to specify a timeout value for scenarios in which an error occurs when the cluster is upgrading.

If you are running applications on scale-out nodes that directly access the GPFS file system in IBM Storage Scale Erasure Code Edition cluster, you need to stop the applications or migrate the applications to other nodes before you upgrade these nodes. Alternatively, you must turn on the interactive mode. Then, the installation toolkit prompts you to stop the applications before you upgrade a node.

7. Change the installation toolkit to interactive mode when scale-out nodes are being used as local file system access nodes.

   ./spectrumscale upgrade config workloadprompt -N all

   Note:

   • Without turning on the interactive mode, the installation toolkit shuts down GPFS even if applications are running on the node that is being upgraded.
   • When the installation toolkit upgrades a protocol node, even if the interactive mode is enabled, no prompts are displayed. The installation toolkit suspends protocol services before you upgrade and after upgrading it resumes protocol services, and exports the file systems again on the node that is being upgraded. To check the consistency of the exported file systems, you can use the showmount -e command before and after the upgrade.
8. Perform the installation toolkit upgrade.

   ./spectrumscale upgrade run

   The upgrade process is divided into several phases. If the nodes being upgraded include both protocol (CES)
    and non-protocol nodes, then protocol nodes and non-protocol nodes are upgraded in separate phases. 
   If the nodes being upgraded are all of the same type, either protocol or non-protocol, then only the 
   upgrade phases applicable to that node type will be performed. 
   The upgrade process may cause a brief outage of Object, SMB, NFS, HDFS and Performance Monitoring components. 
   Do you really want to begin upgrading? [y/N]: y
   [ INFO  ] Logging to file: /usr/lpp/mmfs/5.1.y.z/ansible-toolkit/logs/UPGRADE-dd-mm-yyyy_hh:mm:ss.log
   After a node is upgraded in an ECE setup, certain rebuild and rebalance tasks need to be performed. 
   The default timeout for these tasks is infinite. During the upgrade precheck, you can specify a timeout 
   for these rebuild and rebalance tasks depending on your environment. 
   Do you want to continue with default timeout [Y/n]: n
   Please specify a timeout value in minutes : 30
   

   You can access the installation toolkit upgrade logs from the /usr/lpp/mmfs/5.x.y.z/ansible-toolkit/logs directory.

9. If you are using customized udev rules on your storage nodes, you need to reapply those changes to the new udev rules. The previous rules are saved in the /etc/udev/rules.d/ directory as part of the upgrade. After you apply the changes, activate the changes by using the udevadm command.
10. After the upgrade process is done, complete the upgrade to the new code level to take advantage of the new functions. For more information, see Completing the upgrade to a new level of IBM Spectrum® Scale in IBM Storage Scale: Concepts, Planning, and Installation Guide.
11. After successfully upgrading all ECE nodes to the new release, you must verify and update the configurations on the ECE server with the new release settings. For the detailed steps, see Verifying and updating the configurations.