Troubleshooting
Problem
After OS upgrade/patch of the PDA hosts the GPFS module stops working and is not possible to mount the gpfs filesystem
Symptom
When you try to mount the module is not ready.
[root@host01 ~]# /usr/lpp/mmfs/bin/mmmount all
Mon Apr 10 12:01:52 BST 2017: mmmount: Mounting file systems ...
mmremote: GPFS is not ready to handle commands yet.
mmmount: Command failed. Examine previous error messages to determine cause.
We try to start it manually but it stays DOWN.
[root@host01 ~]# /usr/lpp/mmfs/bin/mmstartup
Mon Apr 10 12:03:54 BST 2017: mmstartup: Starting GPFS ...
[root@host01 ~]# /usr/lpp/mmfs/bin/mmgetstate
Node number Node name GPFS state
------------------------------------------
4 PDA-01 down
Cause
One of the possible causes is that the module is not correctly loaded because is missing .ko files (kernel files)
Environment
PDA N1001-001 with redhat 5.11
Diagnosing The Problem
Check the GPFS logs to see the problem
[root@host01 ~]# tailf /var/adm/ras/mmfs.log.latest
Removing old /var/adm/ras/mmfs.log.* files:
./mmfs.log.2015.12.15.13.34.40.PDA-01
Unloading modules from /lib/modules/2.6.18-417.el5/extra
runmmfs: The /lib/modules/2.6.18-417.el5/extra/mmfslinux.ko kernel extension does not exist.
runmmfs: Unable to verify kernel/module configuration.
Loading modules from /lib/modules/2.6.18-417.el5/extra
runmmfs: The /lib/modules/2.6.18-417.el5/extra/mmfslinux.ko kernel extension does not exist.
runmmfs: Unable to verify kernel/module configuration.
Mon Apr 10 12:03:54 BST 2017 runmmfs: error in loading or unloading the mmfs kernel extension
Mon Apr 10 12:03:54 BST 2017 runmmfs: stopping GPFS
Check for the .ko file. It is available but not in the folder that is expected.
[root@host01 ~]# locate mmfslinux.ko
/lib/modules/2.6.18-402.el5/extra/mmfslinux.ko
/lib/modules/2.6.18-408.el5/weak-updates/mmfslinux.ko
/lib/modules/2.6.18-417.el5/weak-updates/mmfslinux.ko
/usr/lpp/mmfs/src/gpl-linux/.mmfslinux.ko.cmd
/usr/lpp/mmfs/src/gpl-linux/mmfslinux.ko
Resolving The Problem
Copy the files mmfs26.ko , mmfslinux.ko and tracedev.ko
FROM
/lib/modules/2.6.18-417.el5/weak-updates/
TO
/lib/modules/2.6.18-417.el5/extra/
Start GPFS
[root@host01 ~]# /usr/lpp/mmfs/bin/mmstartup
[root@host01 ~]# tailf /var/adm/ras/mmfs.log.latest
mount: Stale NFS file handle
Mon Apr 10 12:12:13 BST 2017: finished mounting /dev/gpfs2
Mon Apr 10 12:12:13 BST 2017: mounting /dev/gpfs1
Mon Apr 10 12:12:13.492 2017: Command: mount gpfs1
Mon Apr 10 12:12:13.493 2017: Connecting to 192.168.1.131 host23 <c0n1>
Mon Apr 10 12:12:13.495 2017: Connected to 192.168.1.131 host23 <c0n1>
Mon Apr 10 12:12:13 BST 2017: mmcommon preunmount invoked. File system: gpfs2 Reason: SGPanic
Mon Apr 10 12:12:13.638 2017: Command: err 0: mount gpfs1
Mon Apr 10 12:12:13 BST 2017: finished mounting /dev/gpfs1
Mon Apr 10 12:12:10 BST 2017: mmstartup: Starting GPFS ...
Check State
[root@cs-prd-netz01 ~]# /usr/lpp/mmfs/bin/mmgetstate
Node number Node name GPFS state
------------------------------------------
4 PDA-01 active
Mount the filesystem
[root@host01 ~]# /usr/lpp/mmfs/bin/mmmount all
Mon Apr 10 12:13:44 BST 2017: mmmount: Mounting file systems ...
mount: Stale NFS file handle
Check df-ha
[root@host01 ~]# df -ha
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 35G 1.2G 32G 4% /
proc 0 0 0 - /proc
sysfs 0 0 0 - /sys
devpts 0 0 0 - /dev/pts
/dev/sda10 7.8G 1.3G 6.1G 18% /usr
/dev/sda12 3.9G 3.7G 0 100% /usr/local
/dev/sda9 7.8G 559M 6.9G 8% /var
/dev/sda8 7.8G 3.5G 3.9G 48% /opt
/dev/sda7 16G 1.4G 14G 9% /tmp
/dev/sda5 419G 80G 318G 21% /nzscratch
/dev/sda1 996M 55M 890M 6% /boot
none 77G 158M 77G 1% /dev/shm
none 0 0 0 - /proc/sys/fs/binfmt_misc
sunrpc 0 0 0 - /var/lib/nfs/rpc_pipefs
nfsd 0 0 0 - /proc/fs/nfsd
/dev/drbd0 16G 2.8G 13G 19% /export/home
/dev/drbd1 296G 63G 218G 23% /nz
debugfs 0 0 0 - /sys/kernel/debug
/dev/gpfs1 10T 1T 9.1T 88% /gpfs
Repeat the steps in ha2
Was this topic helpful?
Document Information
Modified date:
17 October 2019
UID
swg22001775