Topic
20 replies Latest Post - ‏2013-12-18T08:44:09Z by nagger
nagger
nagger
1620 Posts
ACCEPTED ANSWER

Pinned topic nmon2rrd version 16a released

‏2013-01-02T12:48:41Z |
Hi,
I have just released nmon2rrd version 16a and you can find it at
https://www.ibm.com/developerworks/wikis/display/WikiPtype/nmon
  • This version 16 has drastic changes which extracts column header names from the nmon file which makes it less sensitive to new columns and removed loads of code working around the variations in file output over the years.
  • It has also been beautified with the AIX cb -s command to fix indentation mistakes
  • Change comments (peoples names and company names) removed including mine - this is a sample code
  • This needs to be done to remove ownership issues in the future and gets messy.

I really appreciate the fixes supplied and have tried to include them but this version needs a good regression test.
  • Note the MEM section columns now match the nmon output file. Does this cause a problem with older .rrd files that you have been adding to for a longer period?
  • please let me know either way - as we could add a work around.

Can anyone testing the new version and finding a failure please report the issue and the problem file so I can reproduce the issue?
Can we have a new Forum Entry would be good with say "nmon2rrd v16 problem" in the title.

Thanks Nigel Griffiths
Updated on 2013-02-10T23:25:55Z at 2013-02-10T23:25:55Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    2404 Posts
    ACCEPTED ANSWER

    Re: nmon2rrd version 16a released

    ‏2013-01-03T07:37:31Z  in response to nagger
    Whoa
    Nice work. I will test it against my nmon data sets asap, lets see how it works.
    I will "time" them as well to see any changes in the efficiency of the compilation
    optimizations. I have noted that gcc compilation of mine binds to single processor only.

    Vesa
  • SystemAdmin
    SystemAdmin
    2404 Posts
    ACCEPTED ANSWER

    Re: nmon2rrd version 16a released

    ‏2013-01-03T09:49:43Z  in response to nagger
    Hi,
    nmon2rrd_16 Binary core dumped on my 5.3.8.3 for obvious reasons. I recompiled it with gcc.
    I did not do timed tests, since most of the ticks are consumed actually by the rrdtool, which is bastard who do not thread.

    Test results:

    12e nmondata.5.3.0.56 lpar section graphs empty
    12e nmondata.5.3.8.3 lpar section graphs empty
    topasnmon nmondata.5.3.11.3 lpar section graphs empty
    topasnmon nmondata.6.1.4.7 lpar section graphs empty
    topasnmon nmondata.6.1.6.3 lpar section graphs empty
    topasnmon nmondata.6.1.6.15 ok
    topasnmon nmondata.6.1.7.2 ok
    topasnmon nmondata.6.1.7.16 ok
    topasnmon nmondata.6.1.7.16+ame ok
    topasnmon nmondata.7.1.1.16+ame lpar graphs not created, "Invalid DS name" in rrd creation.
    This is most likely due too long DS names. You cannot reference them
    directly from nmondata to rrd. I remember hitting the same issue in November.
    So you need an algoritm to shorten them if they are too long, like taking
    vowels off. See attached file.

    I just checked if the graphs were populated, nothing else.

    PS. Lparstat output is a good addition. Could be in table though, for readability.
    Like processes are.

    Noted that you have changed to the fat line for most of the graphs.
    Hmmm..."Mar's Law"

    Vesa
    • SystemAdmin
      SystemAdmin
      2404 Posts
      ACCEPTED ANSWER

      Re: nmon2rrd version 16a released

      ‏2013-01-03T10:06:42Z  in response to SystemAdmin
      Reason for empty graphs:

      ERROR: expected 21 data source readings (got 20).

      create lpar.rrd --start 1352066453 --step 85
      DS:PhysicalCPU:GAUGE:170:U:U
      DS:virtualCPUs:GAUGE:170:U:U
      DS:logicalCPUs:GAUGE:170:U:U
      DS:poolCPUs:GAUGE:170:U:U
      DS:entitled:GAUGE:170:U:U
      DS:weight:GAUGE:170:U:U
      DS:PoolIdle:GAUGE:170:U:U
      DS:usedAllCPU:GAUGE:170:U:U
      DS:usedPoolCPU:GAUGE:170:U:U
      DS:SharedCPU:GAUGE:170:U:U
      DS:Capped:GAUGE:170:U:U
      DS:EC_User:GAUGE:170:U:U
      DS:EC_Sys:GAUGE:170:U:U
      DS:EC_Wait:GAUGE:170:U:U
      DS:EC_Idle:GAUGE:170:U:U
      DS:VP_User:GAUGE:170:U:U
      DS:VP_Sys:GAUGE:170:U:U
      DS:VP_Wait:GAUGE:170:U:U
      DS:VP_Idle:GAUGE:170:U:U
      DS:Folded:GAUGE:170:U:U
      DS:Pool_id:GAUGE:170:U:U
      RRA:AVERAGE:0.5:1:1024

      But what is actually missing :-/
      update lpar.rrd 1352151168:0.606:2:4:35:0.20:128:0.00:1.73:1.73:1:0:158.20:139.13:0.78:4.97:15.82:13.91:0.08:0.50:0
      update lpar.rrd 1352151253:0.022:2:4:35:0.20:128:0.00:0.06:0.06:1:0:2.14:7.70:0.01:1.22:0.21:0.77:0.00:0.12:1
      update lpar.rrd 1352151338:0.019:2:4:35:0.20:128:0.00:0.05:0.05:1:0:2.25:6.02:0.00:1.19:0.22:0.60:0.00:0.12:1
      update lpar.rrd 1352151423:0.035:2:4:35:0.20:128:0.00:0.10:0.10:1:0:4.04:12.16:0.21:1.26:0.40:1.22:0.02:0.13:1
      update lpar.rrd 1352151509:0.204:2:4:35:0.20:128:0.00:0.58:0.58:1:0:43.08:54.81:0.46:3.59:4.31:5.48:0.05:0.36:0
  • SystemAdmin
    SystemAdmin
    2404 Posts
    ACCEPTED ANSWER

    Re: nmon2rrd version 16a released

    ‏2013-01-03T10:49:03Z  in response to nagger
    6.1.6.3 data with lpar columns mismatch
    • nagger
      nagger
      1620 Posts
      ACCEPTED ANSWER

      Re: nmon2rrd version 16a Test Data

      ‏2013-01-08T09:29:25Z  in response to SystemAdmin
      Thanks for the test data - I will look into these problems.

      thanks Nigel Griffiths
      • armink
        armink
        30 Posts
        ACCEPTED ANSWER

        Re: nmon2rrd version 16a Test Data

        ‏2013-01-11T13:47:44Z  in response to nagger
        I checked the output for the following AIX versions (version strings taken from nmon):

        5.3.0.73 LPAR empty, but most of the values are 0.. and I really don't care about this version :-)
        5.3.12.5 ok
        6.1.7.1 with AME ok
        6.1.7.2 without AME ok
        6.1.7.3 with/without AME ok
        6.1.7.15 with/without AME ok
        6.1.7.16 without AME ok
        6.1.7.17 without AME ok
        6.1.8.1 without AME ok
        7.1.1.15 with/without AME ok
        7.1.2.1 with/without AME ok (also ok in dedicated mode)

        Thanks a bunch to the both of you!!!
        Armin
  • SystemAdmin
    SystemAdmin
    2404 Posts
    ACCEPTED ANSWER

    Re: nmon2rrd version 16a released

    ‏2013-01-22T09:36:44Z  in response to nagger
    Hi Nagger,

    I got an issue when testing the v16. the old verison 12e works fine but when running 16a, it shows Segmentation fault(coredump):

    ramon@weresn:dsfr> nmon2rrd
    Error: nmon filename missing
    nmon2rrd -f nmonfile -d directory -x -w width -h height Version:12e
    -f nmonfile the regular CSV nmon output file
    -d directory dirname for the output
    -w width graph width (default 800)
    -h height graph height (default 300)
    -x execute the output files
    Example:
    nmon2rrd -f m1_030811_1534.nmon -d /webpages/docs/m1/030811 -x
    ramon@weresn:dsfr> nmon2rrd_16
    Segmentation fault(coredump)

    Have you ever met this issue?

    Br
    Ramon
    • SystemAdmin
      SystemAdmin
      2404 Posts
      ACCEPTED ANSWER

      Re: nmon2rrd version 16a released

      ‏2013-01-24T08:15:59Z  in response to SystemAdmin
      Hi,

      After days of test, I just found when version16 works well on AIX6 but can not running on AIX5.3, (especially on 5300-07-07-0846). is there any limititation for the new version? or is there any solution for this?

      cause our nmon2web server is running on AIX5 and it is an very old server that can not be upgraded to AIX6...

      Thank you!

      Br
      Ramon
      • SystemAdmin
        SystemAdmin
        2404 Posts
        ACCEPTED ANSWER

        Re: nmon2rrd version 16a released

        ‏2013-01-24T08:29:59Z  in response to SystemAdmin
        True,
        If I understood correctly, author compiled it under AIX 6.1 using heavy optimizations so his binary will not work on AIX 5.3 which is BTW out of support.
        What you can do is to compile it by yourself from c source, use xlC if you have one or get the latest version of gcc from www.perzl.org and then just run command
        gcc -O3 nmon2rrd.c -o your_binary
        • SystemAdmin
          SystemAdmin
          2404 Posts
          ACCEPTED ANSWER

          Re: nmon2rrd version 16a released

          ‏2013-01-25T06:30:00Z  in response to SystemAdmin
          Wow, that's amazing, I did follow your advise and it works fine now!!

          Much appreciate for your kindly help!

          Br
          Ramon
          • nagger
            nagger
            1620 Posts
            ACCEPTED ANSWER

            Re: nmon2rrd version 17a released

            ‏2013-02-02T18:44:55Z  in response to SystemAdmin
            NEW RELEASE 17a is available
            This was tested against the sample files above and another 50 large samples I have.
            Download website - https://www.ibm.com/developerworks/mydeveloperworks/wikis/home?lang=en#/wiki/Power%20Systems/page/nmon2rrd

            Feedback welcome.
            Feedback on specific extra graphs your would like - check the website for a list

            By the way,
            Vendors promise backward compatibility = New OS will run programs compiled on previous versions of the OS and its compilers.
            They don't promise forward compatibility = Old OS runs programs compiled on later OS versions and their compilers.

            Running nmon2rrd compiled on AIX 6 should never be expected to run on AIX 5 - it would be a miracle or you are very very luck. Sorry, I don't have a AIX 5 machine with a compiler to generate binaries for AIX5. Fortunately, the nmon2rrd source code is released as sample code with dealing with the nmon format.
            • SystemAdmin
              SystemAdmin
              2404 Posts
              ACCEPTED ANSWER

              Re: nmon2rrd version 17a released

              ‏2013-02-10T23:25:55Z  in response to nagger
              Hi Nigel and thank you for this wonderful tool.

              I have been trying to compile and use nmon2rrd v12, v16 and v17 on Linux CentOS 6 64bit , but with a few issues.

              My first build failed with the following error :

              gcc nmon2rrd_17.c -o nmon2rrd
              nmon2rrd_17.c: In function ‘search_for_tstring’:
              nmon2rrd_17.c:57:13: warning: incompatible implicit declaration of built-in function ‘strlen’ enabled by default

              After adding the following include in "nmon2rrd_17.c" the build was fine (gcc nmon2rrd_17.c -o nmon2rrd) :
              #include <string.h>

              I then had 2 issues :

              • nmon2rrd failed after a few seconds with ->

              str2array skip failure <DGBUSY DiskGroupBusyip1033175170> skip=2
              Segmentation fault

              It seams that nmon sends the DGBUSY lines even when there is no Disk Group, but that nmon2rrd doesn't like it ...

              A quick hack (assuming that i have no Disk Group) was to shunt it in "nmon2rrd_17.c":

              n = findfirst("DGBUSY,Disk");
              if (n != -1)
              dg_found = 0;

              • most of the time nmon2rrd doesn't fail but keep running endlessly while creating the "rrd_create" file and appendng thousands lines of "create cpuXXXX.rrd ..."

              nmon2rrd does not detect the appropriate number of CPU when there is only one core in the server (AWS m1.small) ->

              ./nmon2rrd -f ./ip-10-33-175-170_130210_2130.nmon -d .
              nmon2rrd version 17a
              progname=nmon_x86_64_centos6
              nmonversion line=AAA,version,14g
              nmonversion=14g
              host=ip-10-33-175-170
              user=root
              runname=ip-10-33-175-170
              rundate=10-FEB-2013
              aix=unknown
              aixtl=unknown
              hardware=unknown
              kernel=unknown
              interval=3
              snapshots=10
              cpus=1790098328 online out of a maximum of 1
              hour=21 minutes=30 seconds=58
              day=10 month=2 year=13
              Warning: no network error line found
              Found 1 DISKBUSY Section(s)
              str2array skip failure <DGBUSY DiskGroupBusyip1033175170> skip=2
              Erreur de segmentation

              if I edit the nmon output file manually and change the "AAA,cpus" line from this "AAA,cpus,1" to that "AAA,cpus,1,1"
              it works

              It would be great if v18 could take care of this :)
              PS : I am not using nmon2rrd for the GIF creation part, but for the RRD rrd_update
              Did you ever think about a new tool that could be a nmon2graphite (http://graphite.wikidot.com/) ?

              Xavier
            • VesaSaarenpää
              VesaSaarenpää
              4 Posts
              ACCEPTED ANSWER

              Re: nmon2rrd version 17a released

              ‏2013-09-13T11:41:41Z  in response to nagger

              Hi,

              Finally got the AIX 7.1 on Power7 to test it. Works like a charm and consumes 0.6x less cpu cycles than my Gnu compilation on power6. Probably has something to do with the rrdtool update as well. It went from 1.2.30 to 1.4.8 and I'm using Perzl's binary.

              Nice to have the AME graphs now :D

              Vesa

  • dzodzo
    dzodzo
    6 Posts
    ACCEPTED ANSWER

    Re: nmon2rrd version 16a released

    ‏2013-10-02T11:27:15Z  in response to nagger

    Greetings, does this tool work with topas_nmon files? And does it work in nmon_grapher fashion with keeping track of history or it will generate rrd just for the given nmon file and then overwrite?

    • nagger
      nagger
      1620 Posts
      ACCEPTED ANSWER

      Re: nmon2rrd version 16a released

      ‏2013-10-02T12:14:51Z  in response to dzodzo

      Just to let you know the nmon2rrd website is now https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power%20Systems/page/nmon2rrd

      and the current version is 17a

      topas_nmon is the nmon that comes with AIX for the last few years and you can just run it with the nmon command, so yes it works with the nmon files this produces.

      nmon2rrd is a one shot filter program to generate graphs for that one .nmon file - then throw the .rrd file away. With some work you can use the output files to collect longer term data and graph it.  I am working on a few changes to make this much easier for nmon2rrd v18.  This idea is for have  directory full of nmon files from one LPAR/VM and it will graph the lot for long term trending which can help for capacity planning and new machine sizing.

      nmon_grapher - not heard of this one. Nor has google.com !!

      • dzodzo
        dzodzo
        6 Posts
        ACCEPTED ANSWER

        Re: nmon2rrd version 16a released

        ‏2013-10-02T12:58:58Z  in response to nagger

        Greetings, nmon_grapher should be IBM script as well, at least the commentary in the script says so:

        # Licensed Materials - Property of IBM
        #
        # (C) COPYRIGHT International Business Machines Corp. 2000
        # All Rights Reserved
        #
        # US Government Users Restricted Rights - Use, duplication or
        # disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
        #
        # Program       : nmon_grapher
        # Version       : 1.2
        # Author        : Maarten Kreuger
        #
        # Gather nmon output remotely and store locally in a rrd database.

        It seems it's no longer developed, this script worked with original nmon that supported the (undocumented) -R flag to produce rrd files, it would set up an at job to run hourly samples of data:

        root@r4289 [/tmp] ps -ef|grep nmon
            root 15007766        1   0 14:00:02      -  0:00 /var/adm/nmon/nmon12e_aix61 -R -F /tmp/nmon_grapher.data.1014182 -c 12 -s 300

        The file contains just a sequence of commands to update the databases for specific node:

        root@r4289 [/tmp] head nmon_grapher.data.1014182
        rrdtool update cpus.rrd 1380715202:54.5:81.8:45.5:63.6:54.5:63.6:45.5:60.0
        rrdtool update cpu.rrd 1380715202:33.3:26.0:2.1:38.5
        rrdtool update mem.rrd 1380715202:1.1:90.3:371.8:14789.1:33024.0:16384.0
        rrdtool update memnew.rrd 1380715202:68.0:6.5:24.4:1.1:26.2:70.0
        ...

        This is then collected on a central machine and updates data for nodes. It stores history as well so you can view data a year back or so. Topas nmon doesn't know this option so we can't use it this way:

        root@r4289 [/tmp] /usr/bin/topas_nmon -R
        /usr/bin/topas_nmon: illegal option -- R

        If there was a way to extract these data so that history will be kept? E.g. design the nmon2rrd in a way that will allow you to specify archives you want to use. Or perhaps have an option not to overwrite the files with data from that one nmon file but just to add these data to the archives.

        • nagger
          nagger
          1620 Posts
          ACCEPTED ANSWER

          Re: nmon2rrd version 16a released

          ‏2013-10-02T15:32:26Z  in response to dzodzo

          Well I never use nmon_grapher so I can't comment on its support.  The nmon -R output was silently dropped when nmon was taken in to AIX and the user based was practically undetectable. But sorry that you were a keen user of that feature.  The nmon code had lots of statements when it cam to outputting data and the prim difference was the delimiters "," and ":".  You are unlikely to get the -R option back but if you are ready and able to do some "hacking" of file formats you may get to continue. Converting nmon files to rrd updates is very tricky due to the time stamps.  The output from nmon2rrd and its rrd_update file might be compatible with your rrd database files - you would have to try and work around problems.  This would probably need some rrdtool format experience and scripting to automate.  I have no idea how close the format would be but the numbers should be exactly the same.   Also compare the rrd_create file which makes the rrd files to what you have to check compatibility. Good luck.

          An alternative is changing to nmon2web. You could extracting your current rrdtool database and loading them into the nmon2web ones then using nmon2web to load .nmon files in the future. Again this is advanced rrdtool use.  Good luck.

  • DavidStaskiewicz
    DavidStaskiewicz
    1 Post
    ACCEPTED ANSWER

    Re: nmon2rrd version 16a released

    ‏2013-12-18T02:52:23Z  in response to nagger

    Nigel,

    Do you have any plans to make a binary available for Linux? Can you also tell me if any additional work is being done to nmon2web?

    Thank you,

    David

    • nagger
      nagger
      1620 Posts
      ACCEPTED ANSWER

      Re: nmon2rrd version 16a released

      ‏2013-12-18T08:44:09Z  in response to DavidStaskiewicz

      Dave,

      The problem with Linux is there is so many version available and it runs on different hardware. The nmon2rrd.c source code is available for you to compile on you favourite version. It is very simple C code and should compile.

      nmon2web is supported by Bruce Spencer and is actively supported and new features added. This is perl and a .cgi script - again you can fix it up and support yourself if needed.

       

      I hope this helps, Cheers, Nigel Griffiths