Completing the upgrade to a new level of IBM Spectrum Scale

It is a good idea to use the cluster for a while with the new level of IBM Spectrum Scale™ installed, until you are sure that you are ready to permanently upgrade the cluster to the new level.

When you are ready to permanently upgrade the cluster, follow the steps in this topic to complete the upgrade. If you decide not to complete the upgrade, you can revert to the previous level of IBM Spectrum Scale. For more information, see Reverting to the previous level of IBM Spectrum Scale.

Before you begin this task, verify that you have upgraded all the nodes in the cluster to the latest licensed version of IBM Spectrum Scale.

When you run mmchconfig release=LATEST in Step 2 of these directions, you can add other parameters and their values to the command line:

mmchconfig release=LATEST <parameterN=value> <parameterN+1=value>...

The following table describes the parameters that are referred to in this topic.

Table 1. Other parameters with mmchconfig release=LATEST
Parameter	Purpose	Comment
--accept-empty-cipherlist-security	You must specify this parameter if you want to continue running the cluster with the lowest level of security for communications between nodes or with other clusters. That is, you want the cipherList attribute to remain set to EMPTY or undefined. For more information, see Security mode.	It is a good idea to have some level of security for cluster communications: Set cipherList to AUTHONLY or to a supported cipher: `mmchconfig cipherList=AUTHONLY` Do not include the parameter --accept-empty-cipherlist-security when you run `mmchconfig release=LATEST.` For more information, see Security mode.
mmfsLogTimeStampISO8601={yes \| no}	Setting this parameter to no allows the cluster to continue running with the earlier log time stamp format. For more information, see Security mode.	Set mmfsLogTimeStampISO8061 to no if you save log information and you are not yet ready to switch to the new log time stamp format. After you complete the upgrade, you can change the log time stamp format at any time with the mmchconfig command. Omit this parameter if you are ready to switch to the new format. The default value is yes

Verify that the SHA message digest and the cipherList configuration variable are set to valid values.
Note:
- The SHA message digest is a hash result that is generated by a cryptographic hash function.
- The cipherList variable specifies the security mode for communications among nodes in the cluster and with nodes in other clusters. For more information, see mmchconfig command.
Follow these steps:
1. Display the current values by entering the following command. The listing shows both the command and example output:
```
# mmauth show .
Cluster name:        zounds.cluster (this cluster)
Cipher list:         (none specified)
SHA digest:          (undefined)
File system access:  (all rw)
```
2. If the value for the SHA digest is (undefined), follow these steps:
  1. Enter the following command to generate a public/private key pair and an SHA message digest:
```
mmauth genkey new
```
  2. Enter mmauth show . again and verify that the value for SHA digest is no longer (undefined).
3. If the value for cipherList is (none specified) or EMPTY, do one of the following actions:
  - If you want a level of security in communications between nodes and with other clusters, follow these steps:
    1. Set cipherList to AUTHONLY or to a supported cipher:
```
mmauth update . -l AUTHONLY
```
    2. Enter mmauth show . again and verify that the value for cipherList is no longer (none specified) or EMPTY.
  - If you do not want a level of security cluster communications, let cipherList remain set to (none specified) or EMPTY.
Enter the following command to upgrade the cluster configuration data, upgrade the file systems, and enable new functionality. You can add the parameters that are described in Table 1 to the command line:
```
mmchconfig release=LATEST 
```
Note: Until you run the mmchconfig release=LATEST command, the management GUI might not be fully operational at the new code level.

Important: If the mmchconfig command detects any nodes that are not available or cannot be reached, it lists the names of those nodes. If any such nodes are listed, correct the problem and run the command again until it verifies all the nodes and completes successfully.
Note: If the mmchconfig command fails with an error message that indicates that cipherlist is set to EMPTY, do one of the following actions:
- If you want the cluster to run with a higher security mode than EMPTY, set cipherList to AUTHONLY or to a supported cipher:
```
mmauth update . -l AUTHONLY
```
  Return to the first part of Step 2 and run the mmchconfig command as before.
- If you want the cluster to continue with the security mode set to EMPTY, return to the first part of Step 2 and run the mmchconfig command with the additional parameter --accept-empty-cipherlist-security.
If you have not already done so, assign an appropriate IBM Spectrum Scale license to each of the nodes in the cluster. See IBM Spectrum Scale license designation for a detailed description of the IBM Spectrum Scale license types. To see what the minimum required IBM Spectrum Scale license is for each of the nodes in the cluster, enter the following command:
```
mmlslicense -L
```
To assign an IBM Spectrum Scale server license to the nodes that require it, enter the following command:
```
mmchlicense server -N NodeList
```
To assign an IBM Spectrum Scale client license to the nodes that require it, enter:
```
mmchlicense client -N NodeList
```
Enable backward-compatible format changes or upgrade all file systems to the latest metadata format changes.
Attention: Before you continue with this step, it is important to understand the differences between mmchfs -V compat and mmchfs -V full:
- If you enter mmchfs -V compat, only changes that are backward compatible with the previous file system version are enabled. Nodes in remote clusters that are running at the previous IBM Spectrum Scale version or later will still be able to mount the file system. Nodes running IBM Spectrum Scale version earlier than the previous version will no longer be able to mount the file system. For example, if you are upgrading from IBM Spectrum Scale 4.2.3.x to 5.0.2.x, a file system of version corresponding to IBM Spectrum Scale 4.2.3.x will remain mountable from a remote cluster running IBM Spectrum Scale 4.2.3.x or later. However, a file system of version corresponding to IBM Spectrum Scale 4.2.0.0 will remain mountable from a remote cluster running IBM Spectrum Scale 4.2.0.0 or later.
  Running in the compatibility mode might prevent enablement of new functionality that relies on disk structures available only with the latest version.
- If you enter mmchfs -V full, all new functions that require different on-disk data structures are enabled. Nodes in remote clusters that run an older IBM Spectrum Scale version will no longer be able to mount the file system. If any nodes that run an older IBM Spectrum Scale version have mounted the file system at the time this command is entered, the mmchfs command fails. This consideration might also apply to minor releases within the same major release. For example, release 4.1.0.x and release 4.1.1.x might have different metadata formats.
To enable backward-compatible format changes, enter the following command:
```
mmchfs FileSystem -V compat
```
To upgrade all file systems to the latest metadata format changes, enter the following command:
```
mmchfs FileSystem -V full
```
Certain new file system features might require more processing that cannot be handled by the mmchfs -V command alone. To fully activate such features, in addition to mmchfs -V, you must also run the mmmigratefs command. An example of such a feature is enabling fast extended attributes for file systems older than GPFS 3.4.

Note: The first mount of a file system after you run mmchfs -V might fail with a no-disk-space error. This situation might occur if the file system is relatively low on metadata disk space (10% or less free). If so, enter the command again. Typically the file system is mounted without a problem after the initial failed mount.

After completing the upgrade, you might need to enable cluster configuration repository (CCR) and fast extended attributes (fastea).

Enable cluster configuration repository (CCR) and fast extended attributes (fastea) as follows, if required.
1. Enable the CCR in the cluster as follows.
```
mmchcluster --ccr-enable
```
2. Enable fastea in the cluster as follows.
  1. Check if fastea is enabled.
```
mmlsfs FileSystemName --fastea
```
    A sample output is as follows.
```
flag                value                    description
------------------- ------------------------ -----------------------------------
 --fastea           Yes                      Fast external attributes enabled?
```
  2. If it is not enabled, issue the following command to enable fastea.
```
mmmigratefs FileSystemName --fastea
```
    For more information, see mmmigratefs command.
  Note: The following features are a few among several IBM Spectrum Scale features that require fast extended attributes (fastea) to be enabled to work.
  - Clones
  - Independent filesets
  - Encryption
  - Active file management (AFM)
If you use the mmbackup command to back up your file system and you have not done a full backup since GPFS 3.3 or later, a full backup might be necessary. For more information, see File systems backed up using GPFS 3.2 or earlier versions of mmbackup.
If you have file audit logging enabled on any file systems, follow the steps in Manually upgrading file audit logging authentication.
Note: Ensure that you include the gpfs.kafka and gpfs.librdkafka packages as part of the upgrade process if those packages are currently installed.