Topic
47 replies Latest Post - ‏2013-03-19T20:14:06Z by SystemAdmin
BruceSpencer
BruceSpencer
297 Posts
ACCEPTED ANSWER

Pinned topic Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

‏2006-02-25T16:02:32Z |
FYI, I've created a couple of scripts that illustrate how the [b]nmon2rrd[/b] tool could be automated to convert nmon data files into web pages. All you need to do is to get the nmon file(s) into a directory on a web server, and the script automates everything from there.

Part 1: http://www.aixtips.com/AIXtip/nmon2rrdv1.htm
Part 2: http://www.aixtips.com/AIXtip/nmon2rrdv2.htm

Part 1 shows how to automate the collection, conversion, organization and display of daily nmon data from multiple servers into web pages using the nmon2rrd tool.

Part 2 extends Part 1 to include long term trend charts.
Updated on 2013-03-19T20:14:06Z at 2013-03-19T20:14:06Z by SystemAdmin
  • KentPerrier
    KentPerrier
    28 Posts
    ACCEPTED ANSWER

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2006-02-27T16:45:08Z  in response to BruceSpencer
    Thank you! Thank you! Thank you!

    I have been fitzing around with this, in what little free time that I have, for a while now. Thank you for putting this together! Perhaps I will have something that works now!
  • Scott_Carroll
    Scott_Carroll
    2 Posts
    ACCEPTED ANSWER

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2006-03-10T00:49:14Z  in response to BruceSpencer
    I'm seeing a java script error, when attempting to use nmon2rrdv2:
    Error: missing ) after argument list
    Source File: http://xx.xx.xx.xx/dumps/nmon/results/server_list.js
    Line: 4, Column: 22
    Source Code:
    "augusta|20060309",n"denver|20060209",n"frankfort|20060209",n"salem|20060209",n"albany|20060209",n"annapolis|20060209"

    Specifically it's complaining about the ,"n"denver <--pointing at the d

    Any ideas?

    Scott
  • KentPerrier
    KentPerrier
    28 Posts
    ACCEPTED ANSWER

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2006-03-10T19:24:28Z  in response to BruceSpencer
    I like what you have done here, but I would like to enhance what you have done. Unfortunately, my knowledge of rrdtool sucks and my attempts to figure it out have been less than successful. I am hoping you will be able to point me in the right direction :)

    Besides the daily charts (for us every 5 minutes) and the year chart of daily averages, I would like 4 (or perhaps 8) weeks of hourly averages. Can you point me in the direction of what to do so I can extend what you have done to do this?

    Thanks!
    • BruceSpencer
      BruceSpencer
      297 Posts
      ACCEPTED ANSWER

      Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

      ‏2006-03-11T14:16:54Z  in response to KentPerrier
      Kent:

      I'm embarrassed to say how long it took me to gain a working knowledge of the rrdtool. It wasn't intuitive for me.

      Here's what you need to know to add charts with different sample intervals. You need to work with the "rrd_create" and "rrd_create" files (nmon2rrd output files)

      The "rrd_create" file has the following format. The "RRA" field is where you'll want to adjust.

      [b]
      create cpu_all.rrd --start 1140272299 --step 900
      DS:User:GAUGE:1800:U:U
      DS:Sys:GAUGE:1800:U:U
      DS:Wait:GAUGE:1800:U:U
      DS:Idle:GAUGE:1800:U:U
      RRA:AVERAGE:0.5:1:192
      RRA:AVERAGE:0.5:96:365
      RRA:AVERAGE:0.5:2920:48
      [/b]

      cpu_all.rrd - the name of the rrd database file.
      --start: the UTC start time for the data collection (UTC = seconds since 1970)

      --step: The nmon sampling interval (seconds). From the "AAA,interval,900" line in the nmon csv file.

      DS: "data set" defines a variable in the rrd database. The "1800" is the maximum amount of time between samples. If the time between samples exceeds 1800, rrd will assume missing data, and store NA for the interval. My script sets this field = 2 * step = 2 * 900 = 1800

      RRA:AVERAGE:0.5:1:192 - This statement creates a moving average of 900 seconds. The interval size is 1 step * 900 seconds/step = 900 seconds. The "192" specifies a maximun of 192 readings (192 * 900 sec = 54 hours or about 2 days). When you exceed 192 entries, rrdtool will wrap to the beginning of the database and overwrite starting at record #1 (round robin database).

      RRA:AVERAGE:0.5:96:365 - This statement creates a second moving average 24 hours (96 steps * 900 seconds/step = 24 hours). It will maintain a years worth of data (365 samples (or days) after which it starts rewriting at the beginning of the database.

      RRA:AVERAGE:0.5:2920:48 - This is the monthly moving average (2920 steps * 900 sec/step = 30.4 days). It will store up to 48 months of data, before it overwrites itself.

      You can add/delete the RRA definitions. You'll need at least 1 RRA definition. I chose three.

      When rrdtool creates the graph, it chooses which moving average to chart based on the "start" and "end" times (see the rrd_graph file). There's no way to directly specify which moving average (daily, monthly, ...). You'll have to experiment by changing the

      1)the RRA fields in the rrd_create file (database size)
      2)Specifying a suitable start and end time in the rrd_graph file
      • BruceSpencer
        BruceSpencer
        297 Posts
        ACCEPTED ANSWER

        Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

        ‏2006-03-13T15:56:20Z  in response to BruceSpencer
        Update: there is a way to specify the time interval in rrdtool graphs. It's the "--step" option. Be sure the rrd database is setup to store the number of samples to meet your needs. I believe my current setting would be insufficient for your requirements (~2 days for the short interval).
      • KentPerrier
        KentPerrier
        28 Posts
        ACCEPTED ANSWER

        Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

        ‏2006-03-14T15:47:58Z  in response to BruceSpencer
        Thanks for the (better) description!

        A quick question, where is the --start time stamp taken from? Is that in the nmon file somewhere?

        For a sanity check, here is my config. We are gathering data every 300 seconds. The RRA line for 5 weeks worth of hourly averages would look like:

        RRA:AVERAGE:0.5:12:840

        12 steps is 1 hour, 840 hours in 5 weeks? Do I have that right?

        Assuming I do have that right, and I want to keep the daily average data that you already create, I would need to change the line:
        cat $OUTPUT_DIR/rrd_create |sed -e "s/:0-9*.$/:$days RRA:AVERAGE:0.5:$spd:365 RRA:AVERAGE:0.5:$spm:48/" > $TREND_DIR/rrd_create

        in the nmon2rrd.v2 script to include the output I stated above? Of course, I will have to modify your html and javascript to provide the UI to look at the new charts.

        I want to verify that I am on the right track.

        Thanks again for providing this starting point!

        Kent
  • BruceSpencer
    BruceSpencer
    297 Posts
    ACCEPTED ANSWER

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2006-03-11T13:16:17Z  in response to BruceSpencer
    Scott:

    As a sort term mitigation to your javascript problem, manually edit the server_list.js file. Remove the n's that are outside the parentheses. The js file should have the following format.

    // List of servers for index.htm file
    // Format: server|first archive date (yyyymmdd)
    var servers= new Array(
    "prdibm1b|20051201",
    "p520-sr2|20060219",
    "p520-sr4|20060220",
    "dev-aix-03|20060201"
    );
    I think the problem might be the following line of the nmon2rrd.v2 script. If the "\n" were changed to "n", the output would look like your js file. If so, change it back.

    cat $BACKUP | sed -e "s/\"$/\",\n\"$SERVER\|$YEAR$MM$DAY\"/" > server_list.js

    • Scott_Carroll
      Scott_Carroll
      2 Posts
      ACCEPTED ANSWER

      Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

      ‏2006-03-14T20:48:49Z  in response to BruceSpencer
      Here's the kludge that I used. I'm looking for all of the directories in $WEB_DIR and building the list from that...
      function update_server_list {
      cd $WEB_DIR

      a='"'`ls -1p|grep /|sed "s#\/#\|$YEAR$MM$DAY\", \"#g"`
      b=`echo $a|awk '{ gsub(/, " /, ",\042"); print }'`
      c=`echo $b|awk '{ gsub(/, "/, ""); print }'`
      echo "// List of servers for index.htm file">server_list.js
      echo "// Format: server|first archive date (yyyymmdd)">>server_list.js
      echo "var servers= new Array(">>server_list.js
      echo $c|awk '{ gsub(/","/, "\042,\012\042"); print }'>>server_list.js
      echo ");">>server_list.js
      } # end update_server_list
      • SystemAdmin
        SystemAdmin
        2402 Posts
        ACCEPTED ANSWER

        Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool/ Error

        ‏2006-03-17T12:27:05Z  in response to Scott_Carroll
        does this work on Aix 433.
        On my server I have more than 128 disk. When i run it I get the following error.

        "Found 1 DISKBUSY Section(s)
        Warning: More than 128 disks in this DISK section.
        This can not shown on a single graph, so above 128 disks ignored.
        rm -f *.rrd
        rm -f *.gif
        rrdtool - < rrd_create
        sh: rrdtool: not found.
        rrdtool - < rrd_update >rrd_update.log
        sh: rrdtool: not found.
        rrdtool - < rrd_top >rrd_top.log
        sh: rrdtool: not found.
        rrdtool - < rrd_graph
        sh: rrdtool: not found."

        It there any thing i can change to accomodate the disk I have?

        Jubel
        • BruceSpencer
          BruceSpencer
          297 Posts
          ACCEPTED ANSWER

          Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool/ Error

          ‏2006-03-21T16:16:29Z  in response to SystemAdmin
          The message you see regarding disks is a nmon or nmor2rrd error message. It's not coming from the shell scripts.

          PS do you have the rrdtool installed? The error message
          "sh: rrdtool: not found." implies it needs to be installed. If so you can download it from the Linux Toolbox for AIX or from the UCLA web page (see documentation in the shell script).
      • BruceSpencer
        BruceSpencer
        297 Posts
        ACCEPTED ANSWER

        Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

        ‏2006-03-21T16:37:31Z  in response to Scott_Carroll
        Have you got this working? I tried your code on my server and I see an extra apostrophe. The server_list.js format needs to be as in my last post. If your output looks like this, you should be good to go.
    • SystemAdmin
      SystemAdmin
      2402 Posts
      ACCEPTED ANSWER

      Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

      ‏2006-04-28T22:36:47Z  in response to BruceSpencer
      Scott/Bruce,

      The following modification resolved the problem for me:

      Original Line:
      cat $BACKUP | sed -e "s/\"$/\",\n\"$SERVER\|$YEAR$MM$DAY\"/" > server_list.js

      Modified Line:
      cat $BACKUP | sed -e "s/\"$/\",\\
      \"$SERVER\|$YEAR$MM$DAY\"/" > server_list.js

      Apparently the "\n" newline value only works with gnu sed. Bourne/KSH you have to use one backslash "\" then press "Enter" if the script is using single quotes, or in this case two backslashes "\\" since we are using double quotes.
  • DanCreed
    DanCreed
    3 Posts
    ACCEPTED ANSWER

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2008-10-09T22:00:53Z  in response to BruceSpencer
    Any idea why this is happening... I get blank LPAR graphs...
    Thanks,
    Dan
    • SystemAdmin
      SystemAdmin
      2402 Posts
      ACCEPTED ANSWER

      Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

      ‏2008-10-16T15:12:14Z  in response to DanCreed
      I've the same problem.

      My nmon files are :
      -rwxr-xr-x 210 200 515179 Mar 30 23:27:57 2008 nmon12e_aix534
      -rwxr-xr-x 210 200 515368 Mar 30 23:27:57 2008 nmon12e_aix535
      -rwxr-xr-x 210 200 539536 Mar 30 23:27:57 2008 nmon12e_aix536
      -rwxr-xr-x 210 200 542388 Mar 30 23:27:57 2008 nmon12e_aix537

      I work with nmon2rrdv12f

      Thanks for your help.
      • SystemAdmin
        SystemAdmin
        2402 Posts
        ACCEPTED ANSWER

        Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

        ‏2009-03-03T08:08:39Z  in response to SystemAdmin
        Me too,

        after migrating to 12e the following graphs are missing :

        Absolute CPU Utilization vs. Entitlement
        Shared Pool Utlization
        Asynchronous I/O

        I've checked the raw nmon files with Stephen Atkins' nmon analyser macro and the data is still there, but somehow the nmon2web.pl script drops this information during graph processing.

        It doesn't matter if I use the nmon 12e download from Nigel's wiki or the built-in nmon in AIX5.3 TL9.

        Any ideas ?? I would really appreciate it.

        Thanks
        Marc
        • BruceSpencer
          BruceSpencer
          297 Posts
          ACCEPTED ANSWER

          Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

          ‏2009-03-04T02:36:52Z  in response to SystemAdmin
          There's a new nmon2web.pl file for nmon12. Nigel rearranged some fields in nmon12e output, and it doesn't parse the same.

          You can download the new nmon2web version at link:http://www.aixtips.com/AIXtip/nmon2web.htm

          The only file you need out of the tar file is nmon2web.pl. Simply replace the old with the new. (Be sure to change the directories in the customization section to reflect the location of your files.)

          Along with nmon12 support, I added some enhancements. In the past you were limited to 1 nmon file per day. I've removed that restriction, and you can now update at any frequency (within reason). I suggest a minimum of 1-2 hour sample size at 15-20 min intervals). See the above web site for more information.
          • SystemAdmin
            SystemAdmin
            2402 Posts
            ACCEPTED ANSWER

            Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

            ‏2009-03-04T16:41:17Z  in response to BruceSpencer
            Hello Bruce,

            I have noticed the missing graph data too. I am using your latest version of nmon2web.pl and have just attempted to process daily nmon files created over the last 3 months for an LPAR. I can see it breaking in Feb when I ugraded to nmon 12e.

            For info I am running AIX 5.3 TL8 and therefore nmon12e_aix537.

            Any thoughts appreciated.
            • BruceSpencer
              BruceSpencer
              297 Posts
              ACCEPTED ANSWER

              Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

              ‏2009-03-05T02:32:20Z  in response to SystemAdmin
              Can you tar the nmon files to this site? I'd like to see the 3 days before and 3 days after the break in February. I'll test it on my server. It's possible you have a combination I haven't tested.

              How is it working otherwise?
              • SystemAdmin
                SystemAdmin
                2402 Posts
                ACCEPTED ANSWER

                Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                ‏2009-03-05T11:13:18Z  in response to BruceSpencer
                Hello Bruce,

                Thank you very much for your help in this! I find your nmon2web script a very useful tool.

                I have attached a few more files to help you compare. For me the graphs stop appearing from 2nd Feb when, as I have said, I upgraded nmon to 12e. Also included in the attachment is an example trend graph which shows when it breaks. Worryingly the "Over all CPU Utilization" graph suddenly jumps up at the same time so I suspect that this is not a true representation. I have checked this on a few other trend graphs and they all share this pattern.

                Regards,
                Steve
                • SystemAdmin
                  SystemAdmin
                  2402 Posts
                  ACCEPTED ANSWER

                  Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                  ‏2009-03-05T14:15:59Z  in response to SystemAdmin
                  Hello Steve,

                  I see exactly the same behaviour of the Overall CPU graph.

                  Regards
                  Marc
          • SystemAdmin
            SystemAdmin
            2402 Posts
            ACCEPTED ANSWER

            Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

            ‏2009-03-05T08:22:23Z  in response to BruceSpencer
            Hello Bruce,

            I'm already using the new nmon2web.pl script. I'm using AIX 5.3 TL07 SP3 and AIX 5.3 TL09 SP2 and when the nmon data file is processed then the 3 graphs mentioned above are missing. All other graphs are working fine as far as I can tell.

            I've attached some nmon data files. The files in february (TL07) show the lpar graphs, the march files (TL09) don't.

            Regards
            Marc
            • BruceSpencer
              BruceSpencer
              297 Posts
              ACCEPTED ANSWER

              Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

              ‏2009-03-05T14:22:50Z  in response to SystemAdmin
              I have some preliminary results. I ran your files on my server and have attached the Trend output as a pdf file.

              On one hand I don't see a problem with missing charts. I see both "CPU Utilization" and "Shared Pool" (1st and second graph. The Aysnch I/O chart is missing because that data wasn't in your nmon file). Can you verify these are the charts you are missing?

              On the other hand, I see a problem that could cause missing charts. You are using two versions of nmon (surprise!). This could be a problem.

              The problem started with the AIX upgrade from TL7 to TL9. At TL7 you were running nmon 12e. At TL9 you are running nmon "nmon-topas". As you may know, nmon is part of AIX starting with TL9. (Congratulations Nigel!). However, I recall Nigel saying they incorporated nmon11 into TL9. Apparently your PATH statement is such that you are now using "nmon-topas" after the upgrade.

              Try changing your PATH statement to use nmon12, and see if that helps.

              I haven't completed my tests, and I won't be able to finish till tomorrow as I'm traveling today.
              • BruceSpencer
                BruceSpencer
                297 Posts
                ACCEPTED ANSWER

                Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                ‏2009-03-05T14:37:20Z  in response to BruceSpencer
                There is a different problem on shammer's data. I see where the graph stops after changing from nmon 11e to 12e. I haven't identified the problem yet, and I have to travel today. I'll look at it when I return.

                (PS the PDF file in my last post shows 3 missing days of data in the middle of the chart. This is not a problem because there were no nmon files for those days.)
              • SystemAdmin
                SystemAdmin
                2402 Posts
                ACCEPTED ANSWER

                Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                ‏2009-03-05T16:21:12Z  in response to BruceSpencer
                Yes, in your PDF all the graphs that I'm missing are there. Strange...

                I will try to use nmon12e with TL09. I didn't know that they used nmon11 in TL09. I thought it would be nmon12e as well.

                Thank you very much for your fast help !!!

                Kind regards
                Marc
                • BruceSpencer
                  BruceSpencer
                  297 Posts
                  ACCEPTED ANSWER

                  Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                  ‏2009-03-07T19:32:15Z  in response to SystemAdmin
                  I found a backward compatibility issue in the February update of nmon2web.pl. The problem is missing data in some charts after upgrading from nmon 11e to 12e. I've uploaded a revised version "nmon2web_7mar09.tar.gz" which fixes the problem. For those already using nmon2web, just replace the nmon2web.pl file (after making the necessary customizations).
                  • SystemAdmin
                    SystemAdmin
                    2402 Posts
                    ACCEPTED ANSWER

                    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                    ‏2009-03-10T09:36:57Z  in response to BruceSpencer
                    Hello Bruce,

                    Thank you very much for that. I have tested with a few nmon files from different LPARs and the data is now included.

                    However, I am still seeing the anomaly with the "Over all CPU Utilization" graphs. The utilization seems to jump up heavily when processing 12e files compared to 11e (I have attached a few example graphs). It is of course possible that the readings are true and that this was something fixed in nmon 12e but I think this unlikely as the examples I've attached show high values even when all other graphs would suggest the server is quiet. Your thoughts would be very much appreciated?

                    Regards,

                    Steve
                    • SystemAdmin
                      SystemAdmin
                      2402 Posts
                      ACCEPTED ANSWER

                      Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                      ‏2009-03-10T10:01:37Z  in response to SystemAdmin
                      No sure why the attachment didn't add so I have split it down into 2.
                      • SystemAdmin
                        SystemAdmin
                        2402 Posts
                        ACCEPTED ANSWER

                        Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                        ‏2009-03-10T10:03:57Z  in response to SystemAdmin
                        And some more graphs...
                        • BruceSpencer
                          BruceSpencer
                          297 Posts
                          ACCEPTED ANSWER

                          Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                          ‏2009-03-10T15:23:32Z  in response to SystemAdmin
                          I noticed the change in CPU utilization in the nmon csv files. This is not a problem with nmon2web. It's just displaying the data in the nmon csv file. This would be a good question for Nigel. I'll look at your data tonight.
                          • BruceSpencer
                            BruceSpencer
                            297 Posts
                            ACCEPTED ANSWER

                            Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                            ‏2009-03-11T17:46:22Z  in response to BruceSpencer
                            FYI, I emailed Nigel asking why %CPU increased going from 11e to 12e. I got an "out of office" response. So it may be a few days before we hear back. Just wanted update whats going on.
                            • SystemAdmin
                              SystemAdmin
                              2402 Posts
                              ACCEPTED ANSWER

                              Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                              ‏2009-04-07T14:42:36Z  in response to BruceSpencer
                              Hi,

                              I am attempting to set up the nmon2web.

                              The cgi script errors with:
                              Month '-1' out of range 0..11 at nmon2web.cgi line 327
                              Tue Apr 07 10:36:15 2009 error http://client 10.36.7.9 Premature end of script headers: nmon2web.cgi

                              Can you help?
                              • BruceSpencer
                                BruceSpencer
                                297 Posts
                                ACCEPTED ANSWER

                                Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                                ‏2009-04-07T17:11:34Z  in response to SystemAdmin
                                The error was caused when the web browser passing an invalid month "-1" to the nmon2web.cgi script. In this case, the error (Month -1 out of range) suggests you have not selected a date on the browser. The browser passes "-1" value if no selection has been made on a dropdown menu.
                                • SystemAdmin
                                  SystemAdmin
                                  2402 Posts
                                  ACCEPTED ANSWER

                                  Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                                  ‏2009-04-07T18:03:01Z  in response to BruceSpencer
                                  Hi Bruce,

                                  Thanks.

                                  As I am a beginner - can you explain the process?
                                  I installed the nmon2web.cgi script in the /usr/IBMIHS/cgi-bin dir and changed the variables
                                  I installed the nmon2web.pl in /usr/local/bin and changed the variables.

                                  How do I get the web server to use these?
                                  If I manually navigate to the index.html I do get the main index page (top gif is missing)
                                  I can select a server (out of 2 for now) and a date (1 - Apr 5) and it brings up the graphs.

                                  Thanks again!!
                                  • BruceSpencer
                                    BruceSpencer
                                    297 Posts
                                    ACCEPTED ANSWER

                                    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                                    ‏2009-04-07T19:02:07Z  in response to SystemAdmin
                                    Sounds like you have it working (or are almost there). The index.html file has to be on a web server, not AIX server. My design point was to have a central web server for processing the nmon.csv files and displaying the charts.

                                    Each server that you want to monitor sends the completed nmon.csv file to the central web server (Some people automate it using ssh, ftp or NFS.) The web server has a cron job to run nmon2web.pl at regular intervals. The nmon2web.pl has everything it needs to create the web pages and update the index.html (and idx.html) files.

                                    Again the requirement is that the web server has access to the index.html file as well as the charts created by nmon2web.pl
                                    • SystemAdmin
                                      SystemAdmin
                                      2402 Posts
                                      ACCEPTED ANSWER

                                      Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                                      ‏2009-04-07T20:40:43Z  in response to BruceSpencer
                                      I am using the IBM web server that has access to the raw nmon files and for now I am manually running the nmon2web.pl:
                                      root@pinfra1/usr/local/bin# nmon2web.pl

                                      • Ignore messages like "ls: ../rrddata/diskserv*.rrd: No such file or directory". This informational message occurs when nmon2web.pl doesn't find an optional file. ***

                                      16:28:9: Begin processing file = /celerra2/SA_Info/nmon_pdbcrm1/nmon_web/pdbcrm1_090405_0700.nmon
                                      Elasped Time (sec): Subroutine name
                                      4: Finished get_nmon_data
                                      5: Finished setup_new_server
                                      31: Finished update_rrddata_top
                                      102: Finished update_rrddata_disk
                                      102: Finished update_rrddata
                                      119: Finished mk_rrdgraph
                                      136: Finished mk_rrdgraph
                                      cat: 0652-050 Cannot open /celerra2/SA_Info/nmon_pdbcrm1/nmon_web/02021A13F/Dedicated/pdbcrm1/config/current.cfg.
                                      136: Finished mk_index pdbcrm1 Long Term Performance
                                      cat: 0652-050 Cannot open /celerra2/SA_Info/nmon_pdbcrm1/nmon_web/02021A13F/Dedicated/pdbcrm1/config/current.cfg.
                                      136: Finished mk_index pdbcrm1 Performance 05-APR-2009
                                      16:30:26: End processing /celerra2/SA_Info/nmon_pdbcrm1/nmon_web/pdbcrm1_090405_0700.nmon

                                      16:30:26: Begin processing file = /celerra2/SA_Info/nmon_pdbcrm1/nmon_web/pdbebs1_090405_0700.nmon
                                      Elasped Time (sec): Subroutine name
                                      2: Finished get_nmon_data
                                      3: Finished setup_new_server
                                      44: Finished update_rrddata_top
                                      125: Finished update_rrddata_disk
                                      125: Finished update_rrddata
                                      141: Finished mk_rrdgraph
                                      156: Finished mk_rrdgraph
                                      cat: 0652-050 Cannot open /celerra2/SA_Info/nmon_pdbcrm1/nmon_web/0202832F0/Dedicated/pdbebs1/config/current.cfg.
                                      156: Finished mk_index pdbebs1 Long Term Performance
                                      cat: 0652-050 Cannot open /celerra2/SA_Info/nmon_pdbcrm1/nmon_web/0202832F0/Dedicated/pdbebs1/config/current.cfg.
                                      157: Finished mk_index pdbebs1 Performance 05-APR-2009
                                      16:33:3: End processing /celerra2/SA_Info/nmon_pdbcrm1/nmon_web/pdbebs1_090405_0700.nmon

                                      ----
                                      Neither the index or idx files get updated??

                                      If I navigate my browser to the index.html - as I said I get graphs

                                      Thanks again for all your help!! and time!!
                                      • BruceSpencer
                                        BruceSpencer
                                        297 Posts
                                        ACCEPTED ANSWER

                                        Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                                        ‏2009-04-07T22:16:09Z  in response to SystemAdmin
                                        This looks ok. The error "cat: 0652-050 Cannot open /celerra2/SA_Info/nmon_pdbcrm1/nmon_web/02021A13F/Dedicated/pdbcrm1/config/current.cfg" is not a problem. It's an initialization issue. I will go away after processing 2 days of nmon data.

                                        Strictly speaking, nmon2web.pl does not update index.html. It updates the server_list.js file, which is read by index.html.
                                        • SystemAdmin
                                          SystemAdmin
                                          2402 Posts
                                          ACCEPTED ANSWER

                                          Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                                          ‏2009-04-08T12:46:30Z  in response to BruceSpencer
                                          Awesome - thanks again!!
                            • edgodd1
                              edgodd1
                              2 Posts
                              ACCEPTED ANSWER

                              Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                              ‏2009-05-05T14:27:46Z  in response to BruceSpencer
                              Bruce,

                              Have you gotten any updateds from Nigel regarding the CPU spike in nmon 12e?

                              I have a bunch of LPARS running on power 6 and saw this on some, but not others when upgrading to 12e.

                              Here's one graph showing the high CPU. I have confirmed that it is a bug and this is not the actual CPU usage.

                              oh and by the way.. go celtics!
                              • BruceSpencer
                                BruceSpencer
                                297 Posts
                                ACCEPTED ANSWER

                                Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                                ‏2009-05-06T13:37:13Z  in response to edgodd1
                                Haven't heard from Nigel. I sent him an email, but recall he was out of the office at the time. My email probably got lost.
                                When I looked at this, the nmon2web.pl was displaying the data correctly, so the problem is upstream somewhere.

                                I'll ping Nigel again.

                                Go Lakers!
                                • BruceSpencer
                                  BruceSpencer
                                  297 Posts
                                  ACCEPTED ANSWER

                                  Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                                  ‏2009-05-07T12:53:14Z  in response to BruceSpencer
                                  I heard back from Nigel. To "net it out", use Physical CPU (number of cores used). There can be problems with "%CPU" utilization.
                                  It's a random problem involving libperfstat. No ETA for the fix as it outside of the nmon code.
                                  • BruceSpencer
                                    BruceSpencer
                                    297 Posts
                                    ACCEPTED ANSWER

                                    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                                    ‏2009-05-07T13:01:25Z  in response to BruceSpencer
                                    I've updated the nmon2web code to work with partition mobility. The original version assumed a unique hostname. That is no longer a valid assumption as hosts can move between physical servers.

                                    The new version is available at
                                    link:http://www.aixtips.com/AIXtip/nmon2web.htm

                                    For current nmon2web users, the update is involves 2 files. Replace the "index.html" file, and change one line in nmon2web.pl.
                                    • edgodd1
                                      edgodd1
                                      2 Posts
                                      ACCEPTED ANSWER

                                      Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                                      ‏2009-05-07T13:50:39Z  in response to BruceSpencer
                                      Hi Bruce,

                                      I installed the 2 May, 2009 version a couple of days ago to support partition mobility. Now, it seems that after doing the processing every night, a duplicate host entry is generated for every data file. After three days, all my hosts are listed in triplicate.

                                      When selecting a host from the Daily Chart list, It lists the host with a date range of when it first started recording nmon to a few days ago, then another entry for the same host with the data range May 5 to May 5, then another entry for May 6 to May 6. I assume it will continue add entries tonight and going forward.

                                      The same goes for the long term list selection, although it doesn't seem to matter which entry I pick, I get the same page with the same data.

                                      I haven't done any partition mobility recently.

                                      Attached is an image to show what I mean.
                                      Let me know if there's any data I can post to help.

                                      Thanks again for this wonderful tool and the time you put in to support it!
                                      • BruceSpencer
                                        BruceSpencer
                                        297 Posts
                                        ACCEPTED ANSWER

                                        Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                                        ‏2009-05-08T16:38:13Z  in response to edgodd1
                                        I suspect the problem is in the nmon2web.pl.

                                        Can you do a "diff" on the original and the new version of nmon2web.pl in the tar file?
                                        This is what I get:

                                        
                                        diff nmon2web.pl.new nmon2web.pl.old 769,771c769 <       # Changed 4/28/2009 to check 
                                        
                                        for unique SN-Hostname. <       # Previously assumed unique Hostname. Not 
                                        
                                        true w/partion mobility <       
                                        
                                        if ($js[$i] =~ /\$SN\|$TYPE\|$HOSTNAME\|/ ) 
                                        {  # Update --- >       
                                        
                                        if ($js[$i] =~ /\|$HOSTNAME\|/ ) 
                                        {  # Update
                                        
                                    • SystemAdmin
                                      SystemAdmin
                                      2402 Posts
                                      ACCEPTED ANSWER

                                      Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                                      ‏2009-05-28T12:57:19Z  in response to BruceSpencer
                                      Hello Bruce,

                                      new version of nmon2web (nmon2web_25may09.tar.gz) with partition mobility update is not available on your site (link:http://www.aixtips.com/AIXtip/nmon2web.htm). Maybe you could provide it as attachment with forum post of fix the link on mentioned site?

                                      Error I receive:

                                      Sorry, the Server failed to retrieve the requested data.

                                      file is not found

                                      Thanks in advance!
                                      • BruceSpencer
                                        BruceSpencer
                                        297 Posts
                                        ACCEPTED ANSWER

                                        Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                                        ‏2009-05-28T13:58:08Z  in response to SystemAdmin
                                        I fixed it. You can download now.
                                        • SystemAdmin
                                          SystemAdmin
                                          2402 Posts
                                          ACCEPTED ANSWER

                                          Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

                                          ‏2011-01-05T05:05:28Z  in response to BruceSpencer
                                          Hi Bruce,

                                          This is Excellent Tool, Currently we are using for daily trend analysis. In any chance we can segregate and generate for weekly and monthly web report's on different location in same way generating for daily from the source nmon files.

                                          So that we can prepare the monthly report's that makes more easy to compare the trend's of server's between months.

                                          Appreciate for you support

                                          Regards
                                          Sankar
  • SystemAdmin
    SystemAdmin
    2402 Posts
    ACCEPTED ANSWER

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2013-03-19T20:14:06Z  in response to BruceSpencer
    I am trying to Open the link you have provided however i am not able to go to that link....Did that link changed by any chance?