IBM Support

IBM Spectrum Scale (GPFS): Releases 4.2.3.13 or later and 5.0.2.2 or later have issues where kernel crashes on RHEL7.6

Flashes (Alerts)


Abstract

IBM has identified an issue in IBM Spectrum Scale (GPFS) version that support RHEL7.6 (4.2.3.13 or later and 5.0.2.2 or later), in which a RHEL7.6 node running kernel versions 3.10.0-957.19.1 or higher, including 3.10.0-957.21.2, may encounter a kernel crash while running an IO operations.

Content

Problem Summary:
This issue affects IBM Spectrum Scale (GPFS) versions 4.2.3.13 or later and 5.0.2.2 or later running on RHEL7.6, where upgrading RHEL7.6 kernel version to 3.10.0-957.19.1 or higher, including 3.10.0-957.21.2,  may encounter a kernel crash while running an IO operation on the node.This issue may happen "only when SELinux is configured either in 'enforcing' or 'permissive' mode (but not in the 'disabled' mode)". Use the Linux getenforce command to determine the SELinux mode.
How to determine if system is affected:

Systems running RHEL7.6 kernel 3.10.0-957.19.1 or higher, including 3.10.0-957.21.2, may encounter a kernel crash with the following stack backtrace from dmesg:

[ 2044.103058] kernel BUG at mm/slub.c:3686!
[ 2044.103114] illegal operation: 0001 [#1] SMP
[ 2044.103164] CPU: 0 PID: 47319 Comm: mkdir Kdump: loaded Tainted: G           OE  ------------   3.10.0-957.21.2.el7 #1
[ 2044.103167] task: 00000001e4d13980 ti: 00000001e2d94000 task.ti: 00000001e2d94000
[ 2044.103169] Krnl PSW : 0704c00180000000 00000000002dbafe (kfree+0x1be/0x1c0)
[ 2044.103178]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 EA:3
Krnl GPRS: 00000000002db958 0000000000000000 0000000000000000 00000001e6429e40
[ 2044.103182]            001fffff80e5e3b0 0000000000000140 0000000000000000 000000007fff9b80
[ 2044.103184]            00000001e2d97c18 0000400000023d00 001fffff8078b218 001ffc0103e9dab8
[ 2044.103186]            00000000008f45f4 00000001e6429e40 001fffff8078b218 00000001e2d97900
[ 2044.103199] Krnl Code: 00000000002dbaf2: b9e27091            locgr   %r9,%r1,7
          00000000002dbaf6: a7f4ff5d           brc     15,2db9b0
         #00000000002dbafa: a7f40001           brc     15,2dbafc
         >00000000002dbafe: 0707               bcr     0,%r7
          00000000002dbb00: c0000000000c       larl    %r0,2dbb18
          00000000002dbb06: c0f400000009       brcl    15,2dbb18
          00000000002dbb0c: c60000000005       exrl    %r0,2dbb16
          00000000002dbb12: a7f40000           brc     15,2dbb12
[ 2044.103215] Call Trace:
[ 2044.103216] ([<0000000000009400>] 0x9400)
[ 2044.103219]  [<001fffff8078b218>] cxiInitInodeSecurityCleanup+0x28/0x38 [mmfslinux]
[ 2044.103253]  [<001fffff80e5e3c4>] _Z17newInodeInitLinuxP15KernelOperationP13gpfsVfsData_tPP8OpenFilePPvPP10gpfsNode_tP7FileUIDS6_N5LkObj12LockModeEnumE+0x250/0x294
[mmfs26]
[ 2044.103418]  [<001fffff80e5fc68>] _Z9gpfsMkdirP13gpfsVfsData_tP15KernelOperationP9cxiNode_tPPvPS4_PyS5_PcjjjP10ext_cred_t+0x410/0x7dc [mmfs26]
[ 2044.103451]  [<001fffff807ac17c>] gpfs_i_mkdir+0x1bc/0x470 [mmfslinux]
[ 2044.103462]  [<000000000030af60>] vfs_mkdir+0x130/0x1d0
[ 2044.103468]  [<000000000031168c>] SyS_mkdirat+0x124/0x130
[ 2044.103471]  [<000000000074f59c>] sysc_tracego+0x18/0x1e
[ 2044.103478]  [<000003ffa797c20a>] 0x3ffa797c20a
[ 2044.103480] Last Breaking-Event-Address:
[ 2044.103482]  [<00000000002dbafa>] kfree+0x1ba/0x1c0
Cause:
A retrofit of the following change deployed in newer kernels has caused an inconsistency between the GPFS kernel portability layer and the kernel proper.
Fri Apr 26 2019 [3.10.0-957.19.1.el7]
...
- [security] xattr: Constify ->name member of "struct xattr" [1702286 1607307]
(https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=9548906b
)
 

Problem Determination:
Updating a Spectrum Scale v4.2.3.13 or later and v5.0.2.2 or later code levels running RHEL 7.6 to kernel version 3.10.0-957.19.1 or higher, including 3.10.0-957.21.2, may result in the crash with the stack trace above.
Recommendations:
It is strongly recommended not to upgrade the RHEL7.6 kernel to 3.10.0-957.19.1 or higher, including 3.10.0-957.21.2, i.e do not apply  Red Hat patch RHBA-2019:1337.  A subsequent flash will be issued when a fix is available.
To contact IBM Service, see http://www.ibm.com/planetwide/

[{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"STXKQY","label":"IBM Spectrum Scale"},"Component":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"4.2, 5.0","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}}]

Document Information

Modified date:
12 June 2019

UID

ibm10887213