Upgrading IBM Spectrum Scale components with the installation toolkit

You can use the installation toolkit to upgrade core GPFS™, NFS, SMB, object, management GUI, and performance monitoring from the current version of IBM Spectrum Scale™ to the next higher version. For example, you can use the installation toolkit to upgrade from IBM Spectrum Scale release 4.2.x to release 5.0.x.

For information on the upgrade flow, see IBM Spectrum Scale installation toolkit - upgrade process flow. End of change

Important:

Protocol upgrades are non-concurrent and they cause brief outages to user access. Schedule an outage window for the duration of the upgrade, which is approximately 20 minutes per protocol node.
- For NFS, it is recommended to quiesce all I/O for the duration of the upgrade. NFS experiences I/O pauses, and depending upon the client, mounts and disconnects during the upgrade.
- SMB requires quiescing all I/O for the duration of the upgrade. Due to the SMB clustering functionality, differing SMB levels cannot co-exist within a cluster at the same time. This requires a full outage of SMB during the upgrade.
- For object, it is recommended to quiesce all I/O for the duration of the upgrade. Object service will be down or interrupted at multiple times during the upgrade process. Clients might experience errors or they might be unable to connect during this time. They must retry as appropriate.
The IBM Spectrum Scale cluster that you are upgrading must be healthy. A healthy cluster in the context of this upgrade has the following characteristics:
- GPFS must be up and running on all nodes defined in the cluster definition file.
- All protocols defined on each protocol node must be up and running, if the protocol has been initially enabled.
- The whole cluster is operational.
Before proceeding with the IBM Spectrum Scale upgrade, ensure that Transparent cloud tiering is not active on the cluster. If it is active, shut down Transparent cloud tiering services on all nodes by issuing the mmcloudgateway service stop command. If Transparent cloud tiering is active, the IBM Spectrum Scale upgrade might fail.
The installation toolkit does not support upgrading a cluster containing manually installed protocols, if the entire protocols stack was not originally installed in the cluster. To upgrade these clusters, use the manual procedures. For more information, see Upgrading.
Starting with the IBM Spectrum Scale 4.2.3 release, the installation toolkit supports the installation, deployment, and upgrade of Advanced (gpfs.adv) and Crypto (gpfs.crypto) packages. The upgrade of these packages is handled by the installation toolkit according to your IBM Spectrum Scale product edition.
- For IBM Spectrum Scale Advanced Edition or IBM Spectrum Scale Data Management Edition, if the gpfs.adv and gpfs.crypto packages are installed on the system that you are upgrading, these packages are also upgraded.
- For IBM Spectrum Scale Advanced Edition or IBM Spectrum Scale Data Management Edition, if the gpfs.adv and gpfs.crypto packages are not installed on the system that you are upgrading, these packages are installed.
- These packages are not available with IBM Spectrum Scale Standard Edition.
If you have IBM Spectrum Scale Standard Edition version 4.2.2.x installed, you can use the installation toolkit to upgrade to IBM Spectrum Scale Advanced Edition or IBM Spectrum Scale Data Management Edition version 4.2.3 or later. In this scenario, you must apply the license for the applicable edition.

Before proceeding with upgrade using the installation toolkit, ensure the following:

The prerequisite steps for using the installation toolkit are done. This is particularly important if you are using the installation toolkit to upgrade a cluster that was created manually. For more information, see Preparing to use the installation toolkit.
Review limitations of the installation toolkit related to upgrade. For more information, see Limitations of the installation toolkit.

For performing an upgrade by using the installation toolkit, a key requirement is the cluster definition file that you can obtain by using one of the following ways:

Use the ./spectrumscale config populate command to retrieve the cluster configuration and populate it in the cluster definition file.
Use an existing cluster definition file if it exists in the current version installer configuration directory.

For more information, see Step 4.

Do the following steps to upgrade IBM Spectrum Scale using the installation toolkit:

Download the new IBM Spectrum Scale self-extracting package using the sub-steps and then place it on the installer node.
1. Go to the IBM Spectrum Scale page on Fix Central, select the new spectrumscale package and then click Continue.
2. Choose the download option Download using Download Director to download the new spectrumscale package and place it in the wanted location on the install node.
  Note: Although it is not recommended but if you must use the download option Download using your browser (HTTPS), instead of clicking the down arrow to the left of the package name, you must right-click on the package name and select the Save Link As.. option. If you click the download arrow, the browser might hang.
Extract the new IBM Spectrum Scale self-extracting package by using the package name (for example, /tmp/Spectrum_Scale_Protocols_Standard-5.0.0.0_x86_64-Linux_install).
This creates a new directory structure (/usr/lpp/mmfs/5.0.0.0/).
Accept the license agreement. After the self-extracting package extraction is done several next steps and instructions are displayed on the screen including:
- cluster installation and protocol deployment
- upgrading an existing cluster using the installation toolkit
- adding nodes to an existing cluster using the installation toolkit
- adding NSDs or file systems to an existing cluster using the installation toolkit
Depending on your setup, do one of the following:
- If you are upgrading an IBM Spectrum Scale cluster that was created by using the installation toolkit, do these steps to upgrade by using the installation toolkit.
  Note: If you have used the installation toolkit for the initial installation and deployment, it is assumed that the prerequisites are already in place.
  1. If you want to change the installer node before the upgrade, issue the spectrumscale setup -s InstallNodeIP command to set up the new installer node.
  2. Create a backup of the cluster definition file present in the installer configuration directory of the target version. For example: /usr/lpp/mmfs/5.0.0.0/installer/configuration
    This backup can be used in case the config populate command fails.
  3. From the installer directory of the target version that you are upgrading to, populate the cluster definition file with the current cluster state by issuing the ./spectrumscale config populate command. For more information, see Populating cluster definition file with current cluster state using the installation toolkit and Limitations of config populate option of the installation toolkit.
    If the config populate command fails, before proceeding with the upgrade, do the following.
    1. Copy the cluster definition file from the /usr/lpp/mmfs/4.2.x.x/installer/configuration directory to the /usr/lpp/mmfs/5.0.0.0/installer/configuration directory.
    2. Update the cluster definition file with the cluster state change operations that were done after the initial installation and deployment by using the ./spectrumscale commands. These cluster state change operations include:
      Node-related changes such as adding nodes
      
      NSD-related changes such as adding or modifying NSDs
      
      File system changes such as creating or modifying file systems
      
      Protocols-related changes such as enabling or disabling protocols
- If you are upgrading a manually created IBM Spectrum Scale cluster, do these steps to upgrade by using the installation toolkit.
  1. Ensure that the prerequisite steps for using the installation toolkit are completed. For more information, see Preparing to use the installation toolkit.
  2. Configure your cluster definition file as explained in the following topics:
    - Setting up the installer node
    - Defining the cluster topology for the installation toolkit
  3. From the installer directory of the target version that you are upgrading to, populate the cluster definition file with the current cluster state by issuing the ./spectrumscale config populate command. For more information, see Populating cluster definition file with current cluster state using the installation toolkit and Limitations of config populate option of the installation toolkit.
    If the config populate command fails, before proceeding with the upgrade, update the cluster definition file with the cluster state change operations that were done after the initial installation and deployment by using the ./spectrumscale commands. These cluster state change operations include:
    - Node-related changes such as adding nodes
    - NSD-related changes such as adding or modifying NSDs
    - File system changes such as creating or modifying file systems
    - Protocols-related changes such as enabling or disabling protocols
Ensure that the cluster definition file is accurate and reflects the current state of your IBM Spectrum Scale cluster.
Note: If you are upgrading from IBM Spectrum Scale release 4.2.1 or earlier, ensure that the operating system and architecture fields in the output of the ./spectrumscale node list command are updated. These fields are automatically updated when you perform the upgrade precheck with ./spectrumscale upgrade --precheck or you can update these fields by issuing the ./spectrumscale config update command.

If you want to upgrade IBM Spectrum Scale on IBM Spectrum Archive™ Enterprise Edition (EE) nodes using the installation toolkit, you must perform some additional steps to manage IBM Spectrum Archive components. For more information, see Upgrading IBM Spectrum Scale on IBM Spectrum Archive Enterprise Edition (EE) nodes using the installation toolkit.
Run the upgrade precheck using the ./spectrumscale upgrade --precheck command.
For any cluster with existing performance monitoring collectors, the upgrade prechecks might find that a reconfiguration is required. To enable the installation toolkit to perform the reconfiguration, issue the following command:
```
./spectrumscale config perfmon -r on
```
To keep your existing algorithms, issue the following command:
```
./spectrumscale config perfmon -r off
```
If you are upgrading to IBM Spectrum Scale version 4.2.2 or later, the operating system and architecture fields in the cluster definition file must be updated. The upgrade precheck updates the operating system and CPU architecture fields in the cluster definition file. Alternatively, you can issue the spectrumscale config update command to update the operating system and CPU architecture fields in the cluster definition file.
Run the upgrade process using the ./spectrumscale upgrade command.
Note:
- Accept the warnings for system impact and interruption during the upgrade.
- If you are upgrading to IBM Spectrum Scale release 4.2.2.1 or later on SLES 12 nodes, ignore the following warning generated by the local zypper repository created by the installation toolkit:
```
WARN: Chef::Config[:zypper_check_gpg] was not set. 
All packages will be installed without gpg signature checks. This is a security hazard.
```
After the upgrade is completed, the installation toolkit automatically runs the upgrade post checks. You can manually run the upgrade post checks by issuing the ./spectrumscale upgrade --postchecks command.
After the upgrade process is done, complete the upgrade to the new code level to take advantage of the new functionality. For more information, see Completing the upgrade to a new level of IBM Spectrum Scale.

Installing new functions after upgrade

You can use the installation toolkit to install new functions that are available in the new version that you have upgraded to and that are supported by the installation toolkit. Use the following steps to do so:

Enable the functions in the cluster definition file using the installation toolkit.
Issue the ./spectrumscale install or the ./spectrumsacle deploy command.

Failure recovery

If a failure occurs during an upgrade, examine the log at the location provided in the output from the installation toolkit to determine the cause. More information on each error is provided in the output from the installation toolkit.

Certain failures will cause the upgrade process to stop. In this case, the cause of the failure must be addressed on the node on which it occurred. Once the problem has been addressed, the upgrade command can be run again. This does not affect any nodes that were upgraded successfully and the upgrade process continues from the point of failure.

An example of a failure during upgrade is:

A protocol is enabled in the configuration file, but is not running on a node.: Using the mmces service list command on the node that failed highlights which process is not running (the output from the installation toolkit also reports which component failed). Make sure the component is started on all nodes with mmces service start nfs | smb | obj, or alternatively disable the protocol in the configuration using spectrumscale disable nfs|smb|object if the component has been intentionally stopped.
CAUTION:
When a protocol is disabled, the protocol is stopped on all protocol nodes in the cluster and all protocol-specific configuration data is removed.

For information on troubleshooting installation, deployment, and upgrade related issues, see Resolving most frequent problems related to installation, deployment, and upgrade.