IBM Support

IBM Spectrum Scale(GPFS) Alert for V4.2 and V5.0 levels: data corruption may happen on compressed files.

Flashes (Alerts)


Abstract

IBM has identified an issue in IBM Spectrum Scale (GPFS) V4.2.0.0 through 4.2.3.20 (ESS 4.0 through 5.2.9), and V5.0.0.0 through 5.0.4.2 (ESS 5.3 through 5.3.5.1) levels, in which the concurrent use of mmap write and compression operations may cause the data in data blocks being compressed to be corrupted.

Content

 Problem Summary :
 

Generally, the compression operation will be skipped or fail if a regular file is already memory-mapped. Conversely, if the file has already been compressed before a memory-map operation happens on the same file, the data area being mmap written will first be decompressed. However, when a regular file is being compressed and is then memory-mapped, the mmap read operation could page the data into the Operating System (OS) page-cache before a data block is compressed. If an mmap write operation on the just paged-in page happens at the same time as that data block is being compressed, the newly written data in the page-cache may be written to that compressed data block directly without a decompression being attempted. Such a compressed data block containing some uncompressed data would result in decompression failure with an EIO error being returned in most cases to users or applications. Although possible but less likely, corrupted data undetected by users may be returned, as damaged compression data could be mistakenly interpreted by the decompression process. In addition, an FSErrBadCompressBlock file system structure error may be logged into the system log file.

Users Affected:
This issue may affect customers running any level of IBM Spectrum Scale(GPFS) V4.2.0.0 through 4.2.3.20 (ESS 4.0 through 5.2.9), or V5.0.0.0 through 5.0.4.2 (ESS 5.3 through 5.3.5.1), when they perform file compression and mmap write operations on the same file concurrently.
Users may be affected only if both of the following conditions are met:
1)  The Spectrum Scale(GPFS) file compression functionality is being used.
2)  File compression and mmap write operations on the same file may be occurring concurrently.
Problem Determination:
If the problem happens, users may see read failures (EIO) on compressed files with FSSTRUCT error(s) logged in the system log file. After formatting these log records using the tool  at /usr/lpp/mmfs/samples/debugtools/fsstruct.awk, the FSErrBadCompressBlock structure error (as below) should be visible.
MMFS_GENERIC 30D9195E rc=0 reason=0 lvl=5.0.5.0 line 365 Logger.C
MMFS_FSSTRUCT fs 133 FSErrBadCompressBlock     inodeNum=000000000003FA33
snapId=00000000 blockNum=0000000000000000 winattr=0003 illCompressed=0001s
errCode=00000075
Recommendation:
1. Users running IBM Spectrum Scale V5.0.0.0 through V5.0.4.2 (ESS 5.3 through 5.3.5.1) levels should apply IBM Spectrum Scale V5.0.4.3 (ESS 5.3.5.2) or later.
2. Users running IBM Spectrum Scale V4.2.0.0 through V4.2.3.20 (ESS 4.0 through 5.2.9) levels should apply IBM Spectrum Scale V4.2.3.21 or later.
If you cannot apply the above PTF level, contact IBM service to obtain and apply efix for your code level(s):
  • For IBM Spectrum Scale V5.0.0.0 through 5.0.4.2 (ESS 5.3 through 5.3.5.1),  reference APAR IJ22718
  • For IBM Spectrum Scale V4.2.0.0 through 4.2.3.20 (ESS 4.0 through 5.2.9), reference APAR IJ22922
3. Until the fix is applied, users should temporarily stop using the Spectrum Scale file compression functionality to compress more files, unless the memory-map function is not being used on files. That is, stop applying the policy MIGRATE COMPRESS rule, and stop running the mmchattr, mmrestripefs and mmrestripefile commands to [re]compress files when memory-map on files in the same file system is used.

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"STXKQY","label":"IBM Spectrum Scale"},"Component":"","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}}]

Document Information

Modified date:
21 April 2020

UID

ibm15736753