Performing offline upgrade or excluding nodes from upgrade by using installation toolkit

The installation toolkit can upgrade and tolerate nodes that are in an offline state, and it can exclude some nodes from the upgrade.

Note: Online upgrades can be done to the current release (N) from the previous release (N-1). For example, to release 5.1.y (N) from release 5.0.z (N-1). However, you can use the manual offline upgrade procedure to upgrade directly to the current release (N) from a previous release (N-2) or earlier. For example, upgrading to release 5.1.x (N) from release 4.2.x (N-2) or earlier. For more information, see Offline upgrade with complete cluster shutdown. You can also use the installation toolkit to do an offline upgrade from release 5.0.2 or later.
Tip: Before you start with any of the procedures included in the previous list, complete the procedure detailed in Preparing to use the installation toolkit.

Preparing to use the installation toolkit

  1. Download the new IBM Storage Scale self-extracting package by using the substeps and then place it on the installer node.
    1. Go to the IBM Storage Scale page on Fix Central, select the new package, and then click Continue.
    2. Choose the download option Download using Download Director to download the new spectrumscale package and place it in the wanted location on the install node.
      Note: If you use Download using your browser (HTTPS), do not click the down arrow to the left of the package name. Instead, right-click on the package name and select the Save Link As option. If you click the download arrow, the browser might hang.
  2. Extract the new IBM Storage Scale self-extracting package by using the package name. /tmp/Spectrum_Scale_Standard-5.2.1.x_x86_64-Linux_install

    This step creates a new directory structure (/usr/lpp/mmfs/5.2.1.x/).

  3. Accept the license agreement.

    After the self-extracting package extraction is done, several next steps and instructions are displayed on the screen.

    • Cluster installation and protocol deployment
    • Upgrading an existing cluster by using the installation toolkit
    • Adding nodes to an existing cluster by using the installation toolkit
    • Adding NSDs or file systems to an existing cluster by using the installation toolkit
  4. Depending on your setup, do one of the following sets of steps to obtain the current cluster configuration through the cluster definition file.
    If you are upgrading an IBM Storage Scale cluster that is created by using the installation toolkit, do these steps to upgrade by using the installation toolkit.
    Note: If you used the installation toolkit for the initial installation and deployment, it is assumed that the prerequisites are already in place.
    1. If you want to change the installer node before the upgrade, issue the spectrumscale setup -s InstallNodeIP command to set up the new installer node.
    2. From the installer directory of the target version that you are upgrading to, populate the cluster definition file with the current cluster state by issuing the ./spectrumscale config populate command. For more information, see Populating cluster definition file with current cluster state using the installation toolkit and Limitations of config populate option of the installation toolkit.
      If the command fails, update the cluster definition file with the cluster state change operations that were done after the initial installation and deployment by using the ./spectrumscale commands. These cluster state change operations include:
      • Node-related changes such as adding nodes
      • NSD-related changes such as adding or modifying NSDs
      • File system changes such as creating or modifying file systems
      • Protocols-related changes such as enabling or disabling protocols
    If you are upgrading a manually created IBM Storage Scale cluster, do these steps to upgrade by using the installation toolkit.
    1. Ensure that the prerequisite steps for using the installation toolkit are completed. For more information, see Preparing to use the installation toolkit.
    2. Configure your cluster definition file as explained in the following topics:
    3. From the installer directory of the target version that you are upgrading to, populate the cluster definition file with the current cluster state by issuing the ./spectrumscale config populate command. For more information, see Populating cluster definition file with current cluster state using the installation toolkit and Limitations of config populate option of the installation toolkit.
      If the config populate command fails, update the cluster definition file with the cluster state change operations that were done after the initial installation and deployment by using the ./spectrumscale commands. These cluster state change operations include:
      • Node-related changes such as adding nodes
      • NSD-related changes such as adding or modifying NSDs
      • File system changes such as creating or modifying file systems
      • Protocols-related changes such as enabling or disabling protocols
  5. Ensure that the cluster definition file is accurate and reflects the current state of your IBM Storage Scale cluster.

    If you want to upgrade IBM Storage Scale on IBM Storage Archive Enterprise Edition (EE) nodes by using the installation toolkit, you must do some additional steps to manage IBM Spectrum Archive components. For more information, see Upgrading IBM Storage Scale on IBM Storage Archive Enterprise Edition (EE) nodes by using the installation toolkit.

Upgrade when nodes are unhealthy

By using the IBM Storage Scale offline upgrade, you can upgrade your cluster even if one or more nodes are unhealthy. A node is called unhealthy when the services are down but it is reachable through ping commands.

When you designate a node as offline in the cluster configuration, during the upgrade run, the installation toolkit upgrades all installed packages. However, no attempt is made to stop or restart the respective services. Manually restart the previously offline services by using these commands: mmces service start for protocol components and mmstartup for GPFS daemon.

For example, you try to upgrade a 5-node cluster whose nodes are node1, node2, node3, node4, and node5 (protocol node).

On upgrade precheck, you notice the following issues:
  • node3 is reachable but NFS is down.
  • node5 is reachable but SMB is down.
  • node2 is reachable but all services, including GPFS, are down.
Here, you can do the following steps:
  • Designate node3 as offline. During or after the upgrade, the installation toolkit does not restart any services, including NFS, on this node, but all installed packages, including NFS, are upgraded on node3.
  • Designate node5 as offline. During or after the upgrade, the installation toolkit does not restart any services, including SMB, on this node, but all installed packages, including SMB, are upgraded on node5.
  • Designate node2 as offline. All installed packages are upgraded, but none of the services are restarted.
Note:
  • If you designate all nodes in a cluster as offline, then a full offline upgrade is done on all nodes. An upgrade of all installed packages is done without starting or stopping any services.
  • If you try to designate a node that is already excluded as offline, then the exclude designation of the node is cleared, and the offline designation is added. For example,
    ./spectrumscale upgrade config offline -N vm1
    [ INFO ] The node vm1.ibm.com was added in excluded list previously. Clearing this from excluded list. [ INFO ] Adding vm1.ibm.com as smb offline
  • After an offline upgrade, you must ensure that all unhealthy services are manually started by using mmces service start for protocol components and mmstartup for GPFS.

Designating nodes as offline in the upgrade configuration

  • To designate a node as offline, issue this command:
    ./spectrumscale upgrade config offline -N nodename
    An offline upgrade is done on this node, which means that all installed packages are upgraded without restarting any services.
    Important: Before you designate a node as offline, you must ensure that none of the components are active and if the node is a protocol node, then it must be suspended.
    • To check the status of the GPFS daemon, issue the mmgetstate command.
    • To stop the GPFS daemon, issue the mmshutdown command.
    • To check the status of protocol components, issue the mmces service list command.
    • To suspend the protocol node and stop the protocol services, issue the mmces node suspend --stop command.
      If you are upgrading from IBM Storage Scale version 5.0.2.0 or earlier, issue the following commands to suspend the protocol node and stop the protocol services:
      mmces node suspend
mmces service stop Protocol
  • To designate all nodes as offline and do a full offline upgrade across the cluster, issue this command:
    ./spectrumscale upgrade config offline -N node1,node2,node3...,noden

    All installed packages are upgraded on all the nodes in the cluster, but no services are restarted on any of the nodes.

  • Clearing the offline designation.
    • To clear all offline designations from a specific node, issue this command:
      ./spectrumscale upgrade config offline -N nodename --clear
    • To clear all the offline designations from all the nodes, issue this command:
      ./spectrumscale upgrade config offline --clear
  • To clear both the offline and the exclude configuration, issue this command:
    ./spectrumscale upgrade config clear
  • To view all configuration that is done for offline upgrade, issue this command:
    ./spectrumscale upgrade config list

    The output of this command includes the nodes that are excluded and the nodes where the components are designated as offline. An offline upgrade is initiated based on this configuration.

    For example, 
    ./spectrumscale upgrade config list
    [ INFO ] GPFS Node SMB NFS OBJ GPFS 
[ INFO ] 
[ INFO ] Phase1: Non Protocol Nodes Upgrade 
[ INFO ] nsd001st001 - - - 
[ INFO ] nsd002st001 - - - 
[ INFO ] nsd003st001 - - - 
[ INFO ] nsd004st001 - - - 
[ INFO ] 
[ INFO ] Phase2: Protocol Nodes Upgrade 
[ INFO ] prt002st001 
[ INFO ] prt003st001 
[ INFO ] prt004st001 
[ INFO ] prt006st001 
[ INFO ] prt008st001 
[ INFO ] prt009st001 
[ INFO ] prt011st001 
[ INFO ] 
[ INFO ] Excluded Nodes : prt007st001,prt001st001,prt010st001,prt005st001 
[ INFO ]

Upgrade when nodes are not reachable

You can exclude one or more nodes from the current upgrade run, if the nodes are unreachable or if you want to upgrade them later. When you exclude a node from the upgrade configuration, no action is done on this node during the upgrade.
Note:
  • It is not recommended to exclude a subset of protocol nodes. For example, if you have three protocol nodes, then you must exclude all three nodes together. It is not recommended to exclude only a subset (1 or 2) of nodes. For example,
    ./spectrumscale upgrade config exclude -N vm1
    [ INFO ] Adding node vm1.ibm.com in excluded list. 
[ WARN ] Protocol nodes should all be upgraded together if possible, 
since mixed versions of the code are not allowed in CES components (SMB/OBJ). 
You may add the remaining protocol node(s) : vm2.ibm.com in the excluded list or clear node(s): 
vm1.ibm.com with the ./spectrumscale config exclude --clear option so that no protocol nodes are excluded.
  • Ensure that not all admin nodes are excluded and that at least one admin node is available in the nonexcluded list. For example, if you have three admin nodes in a cluster that you want to upgrade, then you can exclude a maximum of two admin nodes only. If you have only one admin node, then it must not be excluded.

Excluding nodes from the upgrade configuration

  • To exclude one or more nodes from the upgrade configuration, issue this command:
    ./spectrumscale upgrade config exclude -N node1,node2
    This command ensures that the installation toolkit does not do any action on node1 and node2 during upgrade.
  • Clearing the exclude designations.
    • To clear the exclude configuration from specific nodes, issue this command:
      ./spectrumscale upgrade config exclude -N node1,node2 --clear
      Note: It is not recommended to clear only a subset of the protocol nodes that are designated as offline.
    • To clear the exclude configuration from all nodes, issue this command:
      ./spectrumscale upgrade config exclude --clear

Upgrading the excluded nodes or offline designated nodes

  1. Ensure that the nodes on which you want to do an offline upgrade are reachable through ping commands.
  2. For nodes that are designated as excluded, clear the exclude designation of the nodes in the cluster definition file by using this command:
    ./spectrumscale upgrade config exclude -N node1,node2 --clear
  3. For nodes that are designated as excluded, designate them as offline if all services are not running by using this command:
    ./spectrumscale upgrade config offline -N node1,node2
  4. Run the upgrade procedure on the offline designated nodes by using this command:
    ./spectrumscale upgrade run
    The installation toolkit upgrades the packages and restarts the services only for an online upgrade. For an offline upgrade, the installation toolkit only upgrades the packages that are currently installed on the offline designated nodes.
  5. After the upgrade procedure is completed, do the following:
    • Restart the GPFS daemon by using the mmstartup command on each offline designated node.
    • If the object protocol is configured, do the post-upgrade object configuration by using the following command from one of the protocol nodes.
      mmobj config manage --version-sync
    • Resume the protocol node and restart the protocol services by using the mmces node resume --start command for every offline designated node that is a protocol node.
      If you are upgrading from IBM Storage Scale version 5.0.2.0 or earlier, issue the following commands to resume the protocol node and start the protocol services:
      mmces node resume
mmces service start Protocol
    • Configure the protocols authentication that is required after you upgrade protocol nodes. For more information, see Protocol authentication configuration changes during upgrade.

Parallel offline upgrade with the installation toolkit

Use the parallel offline upgrade option to upgrade all nodes in the cluster in parallel. By using parallel offline upgrade, you can reduce the overall upgrade duration compared to doing the upgrade in a serial manner.
Note:
  • You can run parallel offline upgrade only:
    • If you can shut down IBM Storage Scale on all nodes in the cluster.
    • If you can suspend protocol services on all protocol nodes, if applicable, in the cluster.
    • If one of the nodes in the upgrade configuration is defined as an admin node.
  • To run a parallel offline upgrade, all nodes in the cluster must be designated offline in the upgrade configuration. Otherwise, all the nodes are upgraded serially. If you want to use parallel offline upgrade on a subset of nodes in the cluster, you must exclude the other nodes from the upgrade configuration by using the ./spectrumscale upgrade config exclude -N Node1, Node2,... command. Only the nodes that are not designated as excluded in the upgrade configuration are upgraded.
Do a parallel offline upgrade as follows.
  1. Shut down IBM Storage Scale on all nodes in the cluster.
    # mmshutdown -a
  2. If your cluster comprises protocol nodes, suspend protocol services on those nodes.
    # mmces node suspend --stop –a
  3. Designate all nodes in the cluster as offline in the upgrade configuration of the installation toolkit.
    # ./spectrumscale upgrade config offline -N all
  4. Run the upgrade procedure.
    # ./spectrumscale upgrade run
    In the 1st phase, the installation toolkit upgrades core components such as GPFS, file audit logging, performance monitoring, and GUI on all nodes in parallel. In the 2nd phase, the installation toolkit upgrades all protocol components on all protocol nodes in parallel.

Upgrading mixed operating system cluster with Ubuntu nodes

On Ubuntu nodes in a mixed operating system cluster, upgrades by using the installation toolkit are not supported. You can upgrade a mixed operating system cluster with Ubuntu nodes with the installation toolkit in two hops. You can do these hops in any order.
  1. Upgrade the non-Ubuntu nodes as follows.
    1. Extract the installation image on the non-Ubuntu installer node.
    2. Run the config populate command.
    3. Exclude all Ubuntu nodes from the upgrade configuration.
    4. Run the upgrade command.
  2. Upgrade the Ubuntu nodes as follows.
    1. Extract the installation image on the Ubuntu installer node.
    2. Run the config populate command.
    3. Exclude all non-Ubuntu nodes from the upgrade configuration.
    4. Run the upgrade command.

Upgrading mixed CPU architecture cluster

You can upgrade a mixed CPU architecture cluster with the installation toolkit in two hops. You can do these hops in any order.
  1. Upgrade the first set of nodes that have the same CPU architecture as follows.
    1. Extract the installation image on the installer node of the same CPU architecture.
    2. Run the config populate command.

      The cluster configuration is populated for the nodes that have the same CPU architecture as the installer node.

    3. Run the upgrade command.
  2. Upgrade the second set of nodes that have the other type of CPU architecture as follows.
    1. Extract the installation image on the installer node for the other type of CPU architecture.
    2. Run the config populate command.

      The cluster configuration is populated for the nodes that have the same CPU architecture as the installer node.

    3. Run the upgrade command.

Offline upgrade with the installation toolkit when OS upgrade is required

If you need to upgrade the OS during an offline upgrade of IBM Storage Scale with the installation toolkit, use the following high-level procedure as guidance.
  1. Run the config populate command.
  2. Stop the services and suspend the protocol nodes.
  3. Do the OS upgrade.
  4. Download and extract the IBM Storage Scale installation package of the version that you want to upgrade to.
  5. Copy the newly generated cluster definition file to the extracted installation package location.
  6. Designate the nodes that you want to upgrade as offline in the upgrade configuration.
  7. Run the upgrade pre-check procedure.
  8. Run the upgrade procedure.
  9. Start the services and resume the protocol nodes.

Populating cluster configuration when nodes are designated as offline in the upgrade configuration

You cannot use the config populate function if one or more nodes in the cluster are designated as offline in the upgrade configuration. Use the following steps to populate the cluster configuration in scenarios in which you plan to designate one or more nodes as offline in the upgrade configuration.
  1. Extract the installation image that you want to use for doing the upgrade.
  2. Use the ./spectrumscale config populate command or create a cluster definition file by using the ./spectrumscale command.
  3. Shut down one or more nodes.
  4. Use the ./spectrumscale upgrade config command to designate the nodes as offline in the upgrade configuration.

Limitations of excluding nodes in the upgrade configuration

You cannot exclude a node if SMB service is running on it. This restriction is not applicable for NFS. For example, when you try to exclude from the upgrade configuration the vm1 node that is running SMB, an error occurs.
./spectrumscale upgrade config exclude -N vm1
[ FATAL ] In order to exclude a protocol node running SMB from the current upgrade, the SMB service must first be stopped on that node. 
Please stop SMB using the mmces command and retry.