IJ22152: KERNEL CRASHES AT CENTOS 7.7 W/SPECTRUM SCALE 5.0.4.1

APAR status

• Closed as program error.

Error description

• Kernel crashes at mm/slub.c:3772 on CentOS 7.7 w/Spectrum
  Scale 5.0.4.1.
  
  A customer did manual build of gpl rpm image without
  doing step "make Autoload" before instead of using
  manually created env.mcr file.
  
  In the step of "Create contents of config/env.mcr" it
  brought the wrong macro value
  (may be copied from old kernel release env.mcr, and then
  changed to the new value, but missing one digit).
  
  This caused kernel crashed.
  
  [ 156.733477] kernel BUG at mm/slub.c:3772!
  [ 156.734212] invalid opcode: 0000 [#1] SMP
  [ 156.735017] Modules linked in: ebtable_nat
  ebtable_filter ebtable_broute bridge stp llc ebtables
  mmfs26(OE) mmfslinux(OE) tracedev(OE) rdma_ucm(OE)
  ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE)
  ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE)
  mlx4_core(OE) ip6table_nat nf_nat_ipv6 ip6table_mangle
  ip6table_raw nf_conntrack_ipv6 nf_defrag_ipv6
  ip6table_filter ip6_tables iptable_nat nf_nat_ipv4 nf_nat
  iptable_mangle iptable_raw nf_conntrack_ipv4
  nf_defrag_ipv4 xt_comment xt_multiport xt_conntrack
  nf_conntrack iptable_filter iptable_security nfit
  libnvdimm ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel
  aesni_intel lrw gf128mul glue_helper ablk_helper sg
  joydev pcspkr cryptd parport_pc parport i2c_piix4
  virtio_balloon knem(OE) binfmt_misc ip_tables xfs
  libcrc32c mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) sr_mod
  cdrom ata_generic pata_acpi virtio_console virtio_net
  virtio_blk crct10dif_pclmul crct10dif_common
  mlx5_core(OE) mlxfw(OE) crc32c_intel ptp pps_core devlink
  ata_piix serio_raw mlx_compat(OE) libata virtio_pci
  floppy virtio_ring virtio dm_mirror dm_region_hash dm_log
  dm_mod
  [ 156.754814] CPU: 3 PID: 11826 Comm: request_handle*
  Tainted: G OE ------------ 3.10.0-1062.9.1.el7.x86_64 #1
  [ 156.756782] Hardware name: Red Hat KVM, BIOS
  1.11.0-2.el7 04/01/2014
  [ 156.757978] task: ffff8aeca5bf8000 ti: ffff8ae9f7a24000
  task.ti: ffff8ae9f7a24000
  [ 156.759326] RIP: 0010:[<ffffffffbbe23dec>]
  [<ffffffffbbe23dec>] kfree+0x13c/0x140
  [ 156.760749] RSP: 0018:ffff8ae9f7a27278 EFLAGS: 00010246
  [ 156.761717] RAX: 001fffff00000400 RBX: ffffffffbc6974bf
  RCX: ffffa74dc1bcfb60
  [ 156.763030] RDX: 001fffff00000000 RSI: ffff8aed90fc6500
  RDI: ffffffffbc6974bf
  [ 156.764321] RBP: ffff8ae9f7a27290 R08: 0000000000000014
  R09: 0000000000000003
  [ 156.765612] R10: 0000000000000048 R11: ffffdb5a82d125c0
  R12: ffffa74dc4fd36c0
  [ 156.766938] R13: ffffffffc0a1c562 R14: ffff8ae9f7a272f8
  R15: ffff8ae9f7a27938
  [ 156.768229] FS: 00007f8ffff05700(0000)
  GS:ffff8aedbfd80000(0000) knlGS:0000000000000000
  [ 156.769708] CS: 0010 DS: 0000 ES: 0000 CR0:
  0000000080050033
  [ 156.770754] CR2: 000055963330e2b0 CR3: 0000000325ad2000
  CR4: 00000000003606e0
  [ 156.772076] DR0: 0000000000000000 DR1: 0000000000000000
  DR2: 0000000000000000
  [ 156.773367] DR3: 0000000000000000 DR6: 00000000fffe0ff0
  DR7: 0000000000000400
  [ 156.774663] Call Trace:
  [ 156.775154] [<ffffffffc0a1c562>]
  cxiInitInodeSecurityCleanup+0x12/0x20 [mmfslinux]
  [ 156.776568] [<ffffffffc0b50562>]
  _Z17newInodeInitLinuxP15KernelOperationP13gpfsVfsData_tPP
  8OpenFilePPvPP10gpfsNode_tP7FileUIDS6_N5LkObj12LockModeEn
  umE+0x152/0x290 [mmfs26]
  [ 156.779378] [<ffffffffc0b5cdfa>]
  _Z9gpfsMkdirP13gpfsVfsData_tP15KernelOperationP9cxiNode_t
  PPvPS4_PyS5_PcjjjP10ext_cred_t+0x46a/0x7e0 [mmfs26]
  [ 156.781689] [<ffffffffc0bdb928>] ?
  _ZN14BaseMutexClass15releaseLockHeldEP16KernelSynchState+
  0x18/0x130 [mmfs26]
  [ 156.783565] [<ffffffffc0c3db2d>]
  _ZL21pcacheHandleCacheMissP13gpfsVfsData_tP15KernelOperat
  ionP10gpfsNode_tPvPcPyP12pCacheResp_tPS5_PS4_PjSA_j+0x4bd
  /0x760 [mmfs26]
  [ 156.786228] [<ffffffffc0c40675>]
  _Z12pcacheLookupP13gpfsVfsData_tP15KernelOperationP10gpfs
  Node_tPvPcP7FilesetjjjPS5_PS4_PyPjS9_+0x1ff5/0x21a0
  [mmfs26]
  [ 156.788681] [<ffffffffc0c023ef>] ?
  _Z15findFilesetByIdP15KernelOperationjjPP7Filesetj+0x4f/0
  xa0 [mmfs26]
  [ 156.790448] [<ffffffffc0b6d59c>]
  _Z10gpfsLookupP13gpfsVfsData_tPvP9cxiNode_tS1_S1_PcjPS1_P
  S3_PyP10cxiVattr_tPjP10ext_cred_tjS5_PiS4_SD_+0x65c/0xad0
   [mmfs26]
  [ 156.793032] [<ffffffffc0b8b022>] ?
  _Z33gpfsIsCifsBypassTraversalCheckingv+0xe2/0x130
  [mmfs26]
  [ 156.794588] [<ffffffffc0a36d96>]
  gpfs_i_lookup+0x2e6/0x5a0 [mmfslinux]
  [ 156.795838] [<ffffffffc0b6cf40>] ?
  _Z8gpfsLinkP13gpfsVfsData_tP9cxiNode_tS2_PvPcjjP10ext_cre
  d_t+0x6c0/0x6c0 [mmfs26]
  [ 156.797753] [<ffffffffbbe65d52>] ?
  __d_alloc+0x122/0x180
  [ 156.798763] [<ffffffffbbe65e10>] ? d_alloc+0x60/0x70
  [ 156.799700] [<ffffffffbbe556d3>] lookup_real+0x23/0x60
  [ 156.800651] [<ffffffffbbe560f2>]
  __lookup_hash+0x42/0x60
  [ 156.801675] [<ffffffffbc377874>] lookup_slow+0x42/0xa7
  [ 156.802634] [<ffffffffbbe5ac3f>]
  link_path_walk+0x80f/0x8b0
  [ 156.803666] [<ffffffffbbe5ae4a>]
  path_lookupat+0x7a/0x8b0
  [ 156.804690] [<ffffffffbbdcd2fe>] ?
  lru_cache_add+0xe/0x10
  [ 156.805690] [<ffffffffbbe24ef5>] ?
  kmem_cache_alloc+0x35/0x1f0
  [ 156.806766] [<ffffffffbbe5c45f>] ?
  getname_flags+0x4f/0x1a0
  [ 156.807817] [<ffffffffbbe5b6ab>]
  filename_lookup+0x2b/0xc0
  [ 156.808834] [<ffffffffbbe5d5f7>]
  user_path_at_empty+0x67/0xc0
  [ 156.809923] [<ffffffffbbdf3ecd>] ?
  handle_mm_fault+0x39d/0x9b0
  [ 156.811017] [<ffffffffbbe5d661>] user_path_at+0x11/0x20
  [ 156.811983] [<ffffffffbbe50343>] vfs_fstatat+0x63/0xc0
  [ 156.812951] [<ffffffffbbe506fe>] SYSC_newstat+0x2e/0x60
  [ 156.813931] [<ffffffffbc388a26>] ?
  trace_do_page_fault+0x56/0x150
  [ 156.815050] [<ffffffffbbe50bbe>] SyS_newstat+0xe/0x10
  [ 156.816010] [<ffffffffbc38dede>]
  system_call_fastpath+0x25/0x2a
  [ 156.817104] Code: 49 8b 03 31 f6 f6 c4 40 74 04 41 8b
  73 68 4c 89 df e8 89 2f fa ff eb 84 4c 8b 58 30 48 8b 10
  80 e6 80 4c 0f 44 d8 e9 28 ff ff ff <0f> 0b 66 90 0f 1f
  44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54
  [ 156.822192] RIP [<ffffffffbbe23dec>] kfree+0x13c/0x140
  [ 156.823180] RSP <ffff8ae9f7a27278>
  [ 156.823872] ---[ end trace 142960be4a4feed8 ]---
  [ 156.824806] Kernel panic - not syncing: Fatal exception
  [ 156.826475] Kernel Offset: 0x3ac00000 from
  0xffffffff81000000 (relocation range:
  0xffffffff80000000-0xffffffffbfffffff)
  

Local fix

• Use mmbuildgpl command to build the gpl module on the
  installed node.
  

Problem summary

• There is no automatic method to
  generate the GPL installable package
  for customized RHEL release.
  

Problem conclusion

• Benefits of the solution:
  Provide the capability to generate the
  GPL installable package for customized
  RHEL release.
  
  Work around:
  Follow /usr/lpp/mmfs/src/README steps
  
  Problem trigger:
  N/A
  
  Symptom:
  N/A
  
  Platforms affected:
  All RHEL releases.
  
  Functional Area affected:
  All Scale Users
  
  Customer Impact:
  Suggested
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SPEC SCALE STD

• Fixed component ID

  5737F33AP

Applicable component levels