IBM has identified a problem in IBM Spectrum Scale (GPFS) V4.1.0 thru V5.0.2 levels where the use of Local Read Only Cache (LROC) may result in directory corruption or undetected data corruption in regular files

Content

Problem Summary:

After cached data is moved from memory to the LROC device, any changes to that data should trigger invalidation of  the data stored in LROC.    Due to a problem with the invalidation logic, it is possible for invalidation of this LROC data to be skipped.   This may lead to stale or incorrect data being recalled from LROC and data in memory becoming corrupted, with potential undetected data corruption on disk.

Users Affected:

This issue affects customers running any level of IBM Spectrum Scale (GPFS) V4.1.0 thru V5.0.2 with LROC enabled.

Problem Determination:

Directory corruption can be detected via offline fsck.   In case of directory corruption, there may be FSSTRUCT errors in the system log when trying to access the corrupted directory.

Corruption of user data cannot be detected by fsck or any tools provided with IBM Spectrum Scale (GPFS).

Recommendations:

1. Users running IBM Spectrum Scale V5.0.0.0 thru V5.0.2.0 on any Linux servers should apply IBM Spectrum Scale V5.0.2.1 or later, available from Fix Central at:  https://www.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%20defined%20storage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=5.0.2&platform=All&function=all, or contact IBM Service to obtain and apply the efix for your level of code, reference APAR IJ10565.

2. Users running IBM Spectrum Scale V4.2.0.0 thru V4.2.3.11 on any Linux servers should apply IBM Spectrum Scale V4.2.3.12 or later, available from Fix Central at:  https://www.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%20defined%20storage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.2.3&platform=All&function=all, or contact IBM Service to obtain and apply the efix for your level of code, reference APAR IJ10573.

3. Users running GPFS V4.1.0.0 thru V4.1.0.8 or IBM Spectrum Scale V4.1.1.0 thru V4.1.1.21 on any Linux servers should apply IBM Spectrum Scale V4.1.1.22 or later, available from Fix Central at:  https://www.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.1.1&platform=All&function=all , or contact IBM Service to obtain and apply the efix for your level of code, reference APAR IJ11409.

4. Until the fix is applied, you should temporarily disable the LROC function to avoid any possibility of directory or user data corruption.