Upgrading IBM Spectrum Scale protocol nodes

Use these steps to upgrade protocol nodes in an IBM Spectrum Scale cluster.

Perform the following steps for each protocol node in the cluster.

Stop all user activity in the file systems on the node that you are upgrading.
Stop any protocol services that are running on the node:
1. Issue the following command to suspend Cluster Export Services (CES) and stop the protocol services on the node:
```
mmces node suspend --stop
```
  If you are upgrading from IBM Spectrum Scale version 5.0.2.0 or earlier, issue the following commands to suspend the protocol node and stop protocol services:
```
mmces node suspend
mmces service stop Protocol
```
2. If you are upgrading from IBM Spectrum Scale version 5.0.1.2 or earlier to version 5.0.2 or later, issue the following command to ensure that the mmumount of the CES file system completes successfully:
```
mmcesmonitor stop
```
If the call home service is enabled on the cluster, issue the following command to disable it:
```
mmcallhome capability disable
```

Important: Back up your file systems to ensure that your data is protected.

If file systems are mounted on the node, issue the following command to unmount them:
```
mmumount all
```
Do not use the -f option of the mmumount command.
Shut down IBM Spectrum Scale on the node by issuing the following command:
```
mmshutdown
```
Note: You do not need to close mmccrmonitor and other mm processes because they are handled by IBM Spectrum Scale upgrade post-install scripts.

Attention: Starting with release 4.2.3, IBM Spectrum Scale Express Edition is no longer available. For information on upgrading from IBM Spectrum Scale Express Edition 4.2.2.x or earlier to IBM Spectrum Scale Standard Edition or IBM Spectrum Scale Data Access Edition 4.2.3.x or later, see Changing Express Edition to Standard Edition.

Extract the IBM Spectrum Scale software packages as described in the topic Extracting the IBM Spectrum Scale software on Linux nodes. For information about the location of extracted packages, see Location of extracted packages.
Change to the directory that contains the IBM Spectrum Scale package files.
- If Red Hat Enterprise Linux or SUSE Linux Enterprise Server (SLES) is installed on the node, issue the following command:
```
cd /usr/lpp/mmfs/<x.x.x.x>/gpfs_rpms
```
- If Ubuntu Linux is installed on the node, issue the following command:
```
cd /usr/lpp/mmfs/<x.x.x.x>/gpfs_debs
```
Where <x.x.x.x> is the version level of IBM Spectrum Scale that you are upgrading to.

Upgrade the protocol node by using one of the following set of steps for depending on the operating system and the product edition that is installed on the node.

Tip: To determine the currently installed edition, issue the following command:

mmlslicense -L

Red Hat Enterprise Linux:

If IBM Spectrum Scale Standard Edition or IBM Spectrum Scale Data Access Edition is installed on the node, issue the following command. The entire command must be on one line:

rpm -Fvh ./gpfs.gskit*.rpm ./gpfs.gpl*.rpm ./gpfs.docs*.rpm ./gpfs.compression*.rpm 
./gpfs.callhome-ecc-client*.rpm ./gpfs.base*.rpm ./gpfs.msg.en_US*.rpm 
./gpfs.java*.rpm ./rhel7/python-subprocess32*.rpm 
./rhel7/gpfs.librdkafka*.rpm ./rhel7/gpfs.kafka*.rpm ./rhel7/gpfs.callhome*.rpm

For Red Hat Enterprise Linux PPC64 nodes, issue the following command.

rpm -Fvh ./gpfs.gskit*.rpm ./gpfs.gpl*.rpm ./gpfs.docs*.rpm ./gpfs.compression*.rpm 
./gpfs.callhome-ecc-client*.rpm ./gpfs.base*.rpm ./gpfs.msg.en_US*.rpm 
./gpfs.java*.rpm ./rhel7/gpfs.callhome*.rpm

If IBM Spectrum Scale Advanced Edition or IBM Spectrum Scale Data Management Edition is installed on the node, issue the following command. The entire command must be on one line:

rpm -Fvh ./gpfs.msg.en_US*.rpm ./gpfs.java*.rpm ./gpfs.gskit*.rpm 
./gpfs.gpl*.rpm ./gpfs.docs*.rpm ./gpfs.crypto*.rpm ./gpfs.compression*.rpm 
./gpfs.callhome-ecc-client*.rpm ./gpfs.base*.rpm ./gpfs.adv*.rpm 
./rhel7/python-subprocess32*.rpm ./rhel7/gpfs.librdkafka*.rpm ./rhel7/gpfs.kafka*.rpm 
./rhel7/gpfs.callhome*.rpm

For Red Hat Enterprise Linux PPC64 nodes, issue the following command.

rpm -Fvh ./gpfs.msg.en_US*.rpm ./gpfs.java*.rpm ./gpfs.gskit*.rpm 
./gpfs.gpl*.rpm ./gpfs.docs*.rpm ./gpfs.crypto*.rpm ./gpfs.compression*.rpm 
./gpfs.callhome-ecc-client*.rpm ./gpfs.base*.rpm ./gpfs.adv*.rpm 
./rhel7/gpfs.callhome*.rpm

SUSE Linux Enterprise Server (SLES):

If IBM Spectrum Scale Standard Edition or IBM Spectrum Scale Data Access Edition is installed on the node, issue the following command. The entire command must be on one line:

rpm -Fvh ./gpfs.gskit*.rpm ./gpfs.gpl*.rpm ./gpfs.docs*.rpm ./gpfs.compression*.rpm 
./gpfs.callhome-ecc-client*.rpm ./gpfs.base*.rpm ./gpfs.msg.en_US*.rpm 
./gpfs.java*.rpm ./sles12/python-subprocess32*.rpm ./sles12/gpfs.callhome*.rpm

If IBM Spectrum Scale Advanced Edition or IBM Spectrum Scale Data Management Edition is installed on the node, issue the following command. The entire command must be on one line:

rpm -Fvh ./gpfs.msg.en_US*.rpm ./gpfs.java*.rpm ./gpfs.gskit*.rpm 
./gpfs.gpl*.rpm ./gpfs.docs*.rpm ./gpfs.crypto*.rpm ./gpfs.compression*.rpm 
./gpfs.callhome-ecc-client*.rpm ./gpfs.base*.rpm ./gpfs.adv*.rpm 
./sles12/python-subprocess32*.rpm ./sles12/gpfs.callhome*.rpm

Ubuntu Linux:

If IBM Spectrum Scale Standard Edition or IBM Spectrum Scale Data Access Edition is installed on the node, issue the following command. The entire command must be on one line:

apt-get install --only-upgrade ./gpfs.gskit_8.0.50*.deb ./gpfs.gpl_5.0.*.deb 
./gpfs.docs_5.0*.deb ./gpfs.compression_5.0*.deb ./gpfs.callhome-ecc-client_5.0*.deb 
./gpfs.base_5.0*.deb ./gpfs.msg.en-us_5.0*.deb 
./gpfs.java_5.0*.deb ./ubuntu16/gpfs.librdkafka_5.0*.deb ./ubuntu16/gpfs.kafka_5.0*.deb 
./ubuntu16/gpfs.callhome_5.0*.deb

If IBM Spectrum Scale Advanced Edition or IBM Spectrum Scale Data Management Edition is installed on the node, issue the following command. The entire command must be on one line:

apt-get install --only-upgrade ./gpfs.gskit_8.0.50*.deb ./gpfs.gpl_5.0*.deb 
./gpfs.docs_5.0*.deb ./gpfs.crypto_5.0*.deb ./gpfs.compression_5.0*.deb 
./gpfs.callhome-ecc-client_5.0*.deb ./gpfs.base_5.0*.deb ./gpfs.adv_5.0*.deb 
./gpfs.msg.en-us_5.0*.deb ./gpfs.java_5.0*.deb 
./ubuntu16/gpfs.librdkafka_5.0*.deb ./ubuntu16/gpfs.kafka_5.0*.deb 
./ubuntu16/gpfs.callhome_5.0*.deb

If needed, install the following packages.
1. For all editions, if you are upgrading from IBM Spectrum Scale 4.2.3.x or earlier, install the gpfs.compression package:
  - If Red Hat Enterprise Linux or SUSE Linux Enterprise Server Linux is installed on the node, issue the following command from the gpfs_rpms directory:
```
rpm -ivh gpfs.compression*.rpm
```
  - If Ubuntu Linux is installed on the node, issue the following command from the gpfs_debs directory:
```
apt-get install gpfs.compression*.deb
```
2. For all editions, for the gpfs.ext package:
  - If you are upgrading to IBM Spectrum Scale 5.0.1 or earlier, ensure that you upgrade the gpfs.ext package.
  - In versions 5.0.2 and later, you do not need to upgrade the gpfs.ext package. The package is not present because its functionality is merged into other packages.
    Attention: On Ubuntu Linux nodes, in version 5.0.2 and later, you might see an older version of the gpfs.ext package. You can safely remove this package by issuing the following command:
```
dpkg -P gpfs.ext
```
3. For IBM Spectrum Scale Advanced Edition and IBM Spectrum Scale Data Management Edition, if you plan to use the AFM-based Asynchronous Disaster Recovery feature, the file encryption feature, or the Transparent cloud tiering feature, you must install the following packages:
  - gpfs.adv
  - gpfs.crypto
After a successful upgrade, rebuild the GPFS portability layer (GPL). For more information, see Building the GPFS portability layer on Linux nodes.
Upgrade object packages on the protocol node. For more information, see Upgrading object packages.
Upgrade performance monitoring packages. For more information, see Manually upgrading the performance monitoring tool and Manually upgrading pmswift.
Note: After you upgrade from release 4.2.y to 5.0.x, you might see the error message pmcollector service critical error on GUI nodes. In this problem occurs, restart the pmcollector service by issuing the following command on all GUI nodes:
```
systemctl restart pmcollector
```
Upgrade management GUI packages. For more information, see Manually upgrading the IBM Spectrum Scale management GUI.
Start IBM Spectrum Scale on the node by issuing the following command:
```
mmstartup
```
Mount any file systems that are not mounted automatically when the IBM Spectrum Scale daemon starts.
Upgrade NFS packages on the protocol node. For more information, see Upgrading NFS packages.
If you disabled the call home service in a preceding step, issue the following command to enable it:
```
mmcallhome capability enable accept
```
If you suspended CES in a preceding step, issue the following command to resume CES and start protocol services on the node.
```
mmces node resume --start
```
If you are upgrading from IBM Spectrum Scale version 5.0.2.0 or earlier, issue the following commands to resume the protocol node and start the protocol services:
```
mmces node resume
mmces service start Protocol
```

Repeat the preceding steps on all protocol nodes in the cluster. After these steps are done for all protocol nodes in the cluster, proceed with the following steps.

Do object version sync and start object on all protocol nodes. For more information, see Upgrading object packages.
Upgrade SMB packages on all protocol nodes. For more information, see Upgrading SMB packages.
Upgrade the CES package on all protocol nodes in the cluster.
- For Red Hat Enterprise Linux and SLES:
```
mmdsh -N ProtocolNodeList rpm -Fvh /usr/lpp/mmfs/N.N.N.N/gpfs_rpms/gpfs.protocols-support*.rpm
```
- For Ubuntu (The entire command must be on one line):
```
mmdsh -N ProtocolNodeList apt-get install --only-upgrade 
/usr/lpp/mmfs/N.N.N.N/gpfs_debs/gpfs.protocols-support*.deb
```
  - ProtocolNodeList is the list of all protocol nodes in the cluster.
  - N.N.N.N is the product version. For example: 5.0.2.0.
If the AD authentication is configured, set LDAPMAP_DOMAINS to none in the cluster configuration repository (CCR) by issuing the following commands from one of the protocol nodes.
```
usr/lpp/mmfs/bin/mmccr fget authccr /tmp/authccr
echo LDAPMAP_DOMAINS=none >> /tmp/authccr
/usr/lpp/mmfs/bin/mmccr fput authccr /tmp/authccr
```
Warning: The mmccr command is an IBM Spectrum Scale internal component and it must be used only under the guidance of IBM® Support. If the mmccr command is used incorrectly, the cluster can become nonoperational.
Upgrade the license package corresponding to the installed IBM Spectrum Scale product edition on all nodes in the cluster.
- For Red Hat Enterprise Linux and SLES:
```
mmdsh -N NodeList rpm -Fvh /usr/lpp/mmfs/N.N.N.N/gpfs_rpms/gpfs.license-xxx*.rpm
```
- For Ubuntu:
```
mmdsh -N NodeList apt-get install --only-upgrade /usr/lpp/mmfs/N.N.N.N/gpfs_debs/gpfs.license-xxx*.deb
```
  - NodeList is the list of all nodes in the cluster.
  - N.N.N.N is the product version. For example: 5.0.2.0.
  - xxx is the indicator of the product edition:
    - da: Data Access Edition
    - dm: Data Management Edition
    - std: Standard Edition
    - adv: Advanced Edition
If the product edition is Advanced or Data Management and if watch folder or file audit logging is enabled in the cluster, update the message queue configuration at the cluster level by issuing this command from one of the nodes.
```
mmmsgqueue config --wf-upgrade --skip-wf-enabled-check
```

When all the nodes in the cluster are successfully upgraded to the new level, the next step is to complete the upgrade. For more information, see Completing the upgrade to a new level of IBM Spectrum Scale.