Topic
  • 4 replies
  • Latest Post - ‏2013-05-22T20:50:42Z by pgiorda
SystemAdmin
SystemAdmin
2402 Posts

Pinned topic nmon segv fault in NFS statistics on RHEL6 x86_64

‏2013-03-13T21:59:23Z |
I'm encountering an NMON bug that appears very similar to the one documented here: http://www.ibm.com/developerworks/forums/thread.jspa?threadID=158460.

I'm running NMON 14g on Red Hat Enterprise Linux 6.2 (2.6.32-220.17.1.el6.x86_64).

If I simply run:
nmon -fN -s1 -c10

NMON will crash with the segmentation fault. This is reproducible if I run NMON interactively and select option "N".

The strace -rvf output captures this:

0.000056 write(10, "ZZZZ,T0001,21:53:07,13-MAR-2013\n", 32) = 32
0.000133 lseek(9, 0, SEEK_SET) = 0
0.000033 read(9, "MemTotal: 74075560 kB\nMemF"..., 3072) = 1198
0.000322 read(9, "", 2048) = 0
0.000098 lseek(8, 0, SEEK_SET) = 0
0.000032 read(8, "nr_free_pages 15820886\nnr_inacti"..., 3072) = 1852
0.000066 read(8, "", 2048) = 0
0.000064 lseek(4, 0, SEEK_SET) = 0
0.000033 read(4, "29256.43 218437.59\n", 3072) = 19
0.000039 read(4, "", 3072) = 0
0.000035 lseek(5, 0, SEEK_SET) = 0
0.000032 read(5, "0.06 0.36 0.49 1/6373 36896\n", 3072) = 28
0.000039 read(5, "", 3072) = 0
0.000061 open("/proc/net/rpc/nfs", O_RDONLY) = -1 ENOENT (No such file or directory)
0.000096 open("/proc/net/rpc/nfsd", O_RDONLY) = 1
0.000059 fstat(1, {st_dev=makedev(0, 3), st_ino=4026532807, st_mode=S_IFREG|0444, st_nlink=1, st_uid=0, st_gid=0, st_blksize=1024, st_blocks=0, st_size=0, st_atime=2013/03/13-21:53:07, st_mtime=2013/03/13-21:53:07, st_ctime=2013/03/13-21:53:07}) = 0
0.000071 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1f88a59000
0.000058 lseek(1, 0, SEEK_SET) = 0
0.000045 read(1, "rc 0 4110866 96386306\nfh 1 0 0 0"..., 3072) = 535
0.000060 read(1, "", 3072) = 0
0.000040 --- SIGSEGV (Segmentation fault) @ 0 (0) ---
Similarly to the other bug report, the last line in the resulting nmon file is the "ZZZ" timestamp line, and it looks like the segfault happens when NMON is processing /proc/net/rpc/nfsd just like in the other post.

$ cat /proc/net/rpc/nfsd
rc 0 4156462 97243738
fh 1 0 0 0 0
io 986186421 2123234858
th 156 0 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
ra 320 2467762 0 0 0 0 0 0 0 0 0 0
net 101372843 0 101382568 5874
rpc 101407849 0 0 0 0
proc2 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc3 22 0 55724257 317284 25825298 9240672 34 2544459 1473038 1740614 11250 569 0 559055 5248 32956 16413 610387 162326 3099996 0 0 48569
proc4ops 59 76772 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

This works fine on other RHEL6u2 hosts, but not this one. This host is exporting NFS (while the other hosts are clients only), so perhaps that is a factor. Thanks in advance.
Updated on 2013-04-05T21:50:47Z at 2013-04-05T21:50:47Z by nagger
  • nagger
    nagger
    1640 Posts

    Re: nmon segv fault in NFS statistics on RHEL6 x86_64

    ‏2013-03-25T21:25:26Z  
    Hi,
    Which downloaded nmon binary file are you running or did you recompile?

    Can I have the nmon file it created?

    Good observation on the NFS server/client - I will try to boot a suitable RHEL to investigate.

    Thanks, Nigel
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: nmon segv fault in NFS statistics on RHEL6 x86_64

    ‏2013-03-26T13:48:29Z  
    • nagger
    • ‏2013-03-25T21:25:26Z
    Hi,
    Which downloaded nmon binary file are you running or did you recompile?

    Can I have the nmon file it created?

    Good observation on the NFS server/client - I will try to boot a suitable RHEL to investigate.

    Thanks, Nigel
    Hi Nigel,

    I'm using the NMON 14g pre-compiled binary as downloaded from Sourceforge. I can send a sanitized version of the nmon data, how would you like me to get it to you?

    The last few lines of the NMON file look like this:

    BBBP,7258,uptime
    BBBP,7259,uptime," 21:00:05 up 6:20, 2 users, load average: 0.89, 0.45, 0.46"
    BBBP,7260,getconf PAGESIZE
    BBBP,7261,getconf PAGESIZE,"4096"
    ZZZZ,T0001,21:00:06,12-MAR-2013

    Thanks in advance!

    -Chip
  • nagger
    nagger
    1640 Posts

    Re: nmon segv fault in NFS statistics on RHEL6 x86_64

    ‏2013-04-05T21:50:47Z  
    Hi Nigel,

    I'm using the NMON 14g pre-compiled binary as downloaded from Sourceforge. I can send a sanitized version of the nmon data, how would you like me to get it to you?

    The last few lines of the NMON file look like this:

    BBBP,7258,uptime
    BBBP,7259,uptime," 21:00:05 up 6:20, 2 users, load average: 0.89, 0.45, 0.46"
    BBBP,7260,getconf PAGESIZE
    BBBP,7261,getconf PAGESIZE,"4096"
    ZZZZ,T0001,21:00:06,12-MAR-2013

    Thanks in advance!

    -Chip
    Yes, I would like to see the data file - nag at UK dot ibm.com
    I would be looking for the machine running out of resources of some type like memory.
    Or wacko unusual disk types.

    cheers, Nigel
  • pgiorda
    pgiorda
    1 Post

    Re: nmon segv fault in NFS statistics on RHEL6 x86_64

    ‏2013-05-22T20:50:42Z  
    • nagger
    • ‏2013-04-05T21:50:47Z
    Yes, I would like to see the data file - nag at UK dot ibm.com
    I would be looking for the machine running out of resources of some type like memory.
    Or wacko unusual disk types.

    cheers, Nigel

    On line 1512 change P_NFS to P_NFSD, and the same on line 1521. Recompile.