Topic
  • 47 replies
  • Latest Post - ‏2013-03-19T20:14:06Z by SystemAdmin
BruceSpencer
BruceSpencer
297 Posts

Pinned topic Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

‏2006-02-25T16:02:32Z |
FYI, I've created a couple of scripts that illustrate how the [b]nmon2rrd[/b] tool could be automated to convert nmon data files into web pages. All you need to do is to get the nmon file(s) into a directory on a web server, and the script automates everything from there.

Part 1: http://www.aixtips.com/AIXtip/nmon2rrdv1.htm
Part 2: http://www.aixtips.com/AIXtip/nmon2rrdv2.htm

Part 1 shows how to automate the collection, conversion, organization and display of daily nmon data from multiple servers into web pages using the nmon2rrd tool.

Part 2 extends Part 1 to include long term trend charts.
Updated on 2013-03-19T20:14:06Z at 2013-03-19T20:14:06Z by SystemAdmin
  • KentPerrier
    KentPerrier
    28 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2006-02-27T16:45:08Z  
    Thank you! Thank you! Thank you!

    I have been fitzing around with this, in what little free time that I have, for a while now. Thank you for putting this together! Perhaps I will have something that works now!
  • Scott_Carroll
    Scott_Carroll
    2 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2006-03-10T00:49:14Z  
    I'm seeing a java script error, when attempting to use nmon2rrdv2:
    Error: missing ) after argument list
    Source File: http://xx.xx.xx.xx/dumps/nmon/results/server_list.js
    Line: 4, Column: 22
    Source Code:
    "augusta|20060309",n"denver|20060209",n"frankfort|20060209",n"salem|20060209",n"albany|20060209",n"annapolis|20060209"

    Specifically it's complaining about the ,"n"denver <--pointing at the d

    Any ideas?

    Scott
  • KentPerrier
    KentPerrier
    28 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2006-03-10T19:24:28Z  
    I like what you have done here, but I would like to enhance what you have done. Unfortunately, my knowledge of rrdtool sucks and my attempts to figure it out have been less than successful. I am hoping you will be able to point me in the right direction :)

    Besides the daily charts (for us every 5 minutes) and the year chart of daily averages, I would like 4 (or perhaps 8) weeks of hourly averages. Can you point me in the direction of what to do so I can extend what you have done to do this?

    Thanks!
  • BruceSpencer
    BruceSpencer
    297 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2006-03-11T13:16:17Z  
    Scott:

    As a sort term mitigation to your javascript problem, manually edit the server_list.js file. Remove the n's that are outside the parentheses. The js file should have the following format.

    // List of servers for index.htm file
    // Format: server|first archive date (yyyymmdd)
    var servers= new Array(
    "prdibm1b|20051201",
    "p520-sr2|20060219",
    "p520-sr4|20060220",
    "dev-aix-03|20060201"
    );
    I think the problem might be the following line of the nmon2rrd.v2 script. If the "\n" were changed to "n", the output would look like your js file. If so, change it back.

    cat $BACKUP | sed -e "s/\"$/\",\n\"$SERVER\|$YEAR$MM$DAY\"/" > server_list.js

  • BruceSpencer
    BruceSpencer
    297 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2006-03-11T14:16:54Z  
    I like what you have done here, but I would like to enhance what you have done. Unfortunately, my knowledge of rrdtool sucks and my attempts to figure it out have been less than successful. I am hoping you will be able to point me in the right direction :)

    Besides the daily charts (for us every 5 minutes) and the year chart of daily averages, I would like 4 (or perhaps 8) weeks of hourly averages. Can you point me in the direction of what to do so I can extend what you have done to do this?

    Thanks!
    Kent:

    I'm embarrassed to say how long it took me to gain a working knowledge of the rrdtool. It wasn't intuitive for me.

    Here's what you need to know to add charts with different sample intervals. You need to work with the "rrd_create" and "rrd_create" files (nmon2rrd output files)

    The "rrd_create" file has the following format. The "RRA" field is where you'll want to adjust.

    [b]
    create cpu_all.rrd --start 1140272299 --step 900
    DS:User:GAUGE:1800:U:U
    DS:Sys:GAUGE:1800:U:U
    DS:Wait:GAUGE:1800:U:U
    DS:Idle:GAUGE:1800:U:U
    RRA:AVERAGE:0.5:1:192
    RRA:AVERAGE:0.5:96:365
    RRA:AVERAGE:0.5:2920:48
    [/b]

    cpu_all.rrd - the name of the rrd database file.
    --start: the UTC start time for the data collection (UTC = seconds since 1970)

    --step: The nmon sampling interval (seconds). From the "AAA,interval,900" line in the nmon csv file.

    DS: "data set" defines a variable in the rrd database. The "1800" is the maximum amount of time between samples. If the time between samples exceeds 1800, rrd will assume missing data, and store NA for the interval. My script sets this field = 2 * step = 2 * 900 = 1800

    RRA:AVERAGE:0.5:1:192 - This statement creates a moving average of 900 seconds. The interval size is 1 step * 900 seconds/step = 900 seconds. The "192" specifies a maximun of 192 readings (192 * 900 sec = 54 hours or about 2 days). When you exceed 192 entries, rrdtool will wrap to the beginning of the database and overwrite starting at record #1 (round robin database).

    RRA:AVERAGE:0.5:96:365 - This statement creates a second moving average 24 hours (96 steps * 900 seconds/step = 24 hours). It will maintain a years worth of data (365 samples (or days) after which it starts rewriting at the beginning of the database.

    RRA:AVERAGE:0.5:2920:48 - This is the monthly moving average (2920 steps * 900 sec/step = 30.4 days). It will store up to 48 months of data, before it overwrites itself.

    You can add/delete the RRA definitions. You'll need at least 1 RRA definition. I chose three.

    When rrdtool creates the graph, it chooses which moving average to chart based on the "start" and "end" times (see the rrd_graph file). There's no way to directly specify which moving average (daily, monthly, ...). You'll have to experiment by changing the

    1)the RRA fields in the rrd_create file (database size)
    2)Specifying a suitable start and end time in the rrd_graph file
  • BruceSpencer
    BruceSpencer
    297 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2006-03-13T15:56:20Z  
    Kent:

    I'm embarrassed to say how long it took me to gain a working knowledge of the rrdtool. It wasn't intuitive for me.

    Here's what you need to know to add charts with different sample intervals. You need to work with the "rrd_create" and "rrd_create" files (nmon2rrd output files)

    The "rrd_create" file has the following format. The "RRA" field is where you'll want to adjust.

    [b]
    create cpu_all.rrd --start 1140272299 --step 900
    DS:User:GAUGE:1800:U:U
    DS:Sys:GAUGE:1800:U:U
    DS:Wait:GAUGE:1800:U:U
    DS:Idle:GAUGE:1800:U:U
    RRA:AVERAGE:0.5:1:192
    RRA:AVERAGE:0.5:96:365
    RRA:AVERAGE:0.5:2920:48
    [/b]

    cpu_all.rrd - the name of the rrd database file.
    --start: the UTC start time for the data collection (UTC = seconds since 1970)

    --step: The nmon sampling interval (seconds). From the "AAA,interval,900" line in the nmon csv file.

    DS: "data set" defines a variable in the rrd database. The "1800" is the maximum amount of time between samples. If the time between samples exceeds 1800, rrd will assume missing data, and store NA for the interval. My script sets this field = 2 * step = 2 * 900 = 1800

    RRA:AVERAGE:0.5:1:192 - This statement creates a moving average of 900 seconds. The interval size is 1 step * 900 seconds/step = 900 seconds. The "192" specifies a maximun of 192 readings (192 * 900 sec = 54 hours or about 2 days). When you exceed 192 entries, rrdtool will wrap to the beginning of the database and overwrite starting at record #1 (round robin database).

    RRA:AVERAGE:0.5:96:365 - This statement creates a second moving average 24 hours (96 steps * 900 seconds/step = 24 hours). It will maintain a years worth of data (365 samples (or days) after which it starts rewriting at the beginning of the database.

    RRA:AVERAGE:0.5:2920:48 - This is the monthly moving average (2920 steps * 900 sec/step = 30.4 days). It will store up to 48 months of data, before it overwrites itself.

    You can add/delete the RRA definitions. You'll need at least 1 RRA definition. I chose three.

    When rrdtool creates the graph, it chooses which moving average to chart based on the "start" and "end" times (see the rrd_graph file). There's no way to directly specify which moving average (daily, monthly, ...). You'll have to experiment by changing the

    1)the RRA fields in the rrd_create file (database size)
    2)Specifying a suitable start and end time in the rrd_graph file
    Update: there is a way to specify the time interval in rrdtool graphs. It's the "--step" option. Be sure the rrd database is setup to store the number of samples to meet your needs. I believe my current setting would be insufficient for your requirements (~2 days for the short interval).
  • KentPerrier
    KentPerrier
    28 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2006-03-14T15:47:58Z  
    Kent:

    I'm embarrassed to say how long it took me to gain a working knowledge of the rrdtool. It wasn't intuitive for me.

    Here's what you need to know to add charts with different sample intervals. You need to work with the "rrd_create" and "rrd_create" files (nmon2rrd output files)

    The "rrd_create" file has the following format. The "RRA" field is where you'll want to adjust.

    [b]
    create cpu_all.rrd --start 1140272299 --step 900
    DS:User:GAUGE:1800:U:U
    DS:Sys:GAUGE:1800:U:U
    DS:Wait:GAUGE:1800:U:U
    DS:Idle:GAUGE:1800:U:U
    RRA:AVERAGE:0.5:1:192
    RRA:AVERAGE:0.5:96:365
    RRA:AVERAGE:0.5:2920:48
    [/b]

    cpu_all.rrd - the name of the rrd database file.
    --start: the UTC start time for the data collection (UTC = seconds since 1970)

    --step: The nmon sampling interval (seconds). From the "AAA,interval,900" line in the nmon csv file.

    DS: "data set" defines a variable in the rrd database. The "1800" is the maximum amount of time between samples. If the time between samples exceeds 1800, rrd will assume missing data, and store NA for the interval. My script sets this field = 2 * step = 2 * 900 = 1800

    RRA:AVERAGE:0.5:1:192 - This statement creates a moving average of 900 seconds. The interval size is 1 step * 900 seconds/step = 900 seconds. The "192" specifies a maximun of 192 readings (192 * 900 sec = 54 hours or about 2 days). When you exceed 192 entries, rrdtool will wrap to the beginning of the database and overwrite starting at record #1 (round robin database).

    RRA:AVERAGE:0.5:96:365 - This statement creates a second moving average 24 hours (96 steps * 900 seconds/step = 24 hours). It will maintain a years worth of data (365 samples (or days) after which it starts rewriting at the beginning of the database.

    RRA:AVERAGE:0.5:2920:48 - This is the monthly moving average (2920 steps * 900 sec/step = 30.4 days). It will store up to 48 months of data, before it overwrites itself.

    You can add/delete the RRA definitions. You'll need at least 1 RRA definition. I chose three.

    When rrdtool creates the graph, it chooses which moving average to chart based on the "start" and "end" times (see the rrd_graph file). There's no way to directly specify which moving average (daily, monthly, ...). You'll have to experiment by changing the

    1)the RRA fields in the rrd_create file (database size)
    2)Specifying a suitable start and end time in the rrd_graph file
    Thanks for the (better) description!

    A quick question, where is the --start time stamp taken from? Is that in the nmon file somewhere?

    For a sanity check, here is my config. We are gathering data every 300 seconds. The RRA line for 5 weeks worth of hourly averages would look like:

    RRA:AVERAGE:0.5:12:840

    12 steps is 1 hour, 840 hours in 5 weeks? Do I have that right?

    Assuming I do have that right, and I want to keep the daily average data that you already create, I would need to change the line:
    cat $OUTPUT_DIR/rrd_create |sed -e "s/:0-9*.$/:$days RRA:AVERAGE:0.5:$spd:365 RRA:AVERAGE:0.5:$spm:48/" > $TREND_DIR/rrd_create

    in the nmon2rrd.v2 script to include the output I stated above? Of course, I will have to modify your html and javascript to provide the UI to look at the new charts.

    I want to verify that I am on the right track.

    Thanks again for providing this starting point!

    Kent
  • Scott_Carroll
    Scott_Carroll
    2 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2006-03-14T20:48:49Z  
    Scott:

    As a sort term mitigation to your javascript problem, manually edit the server_list.js file. Remove the n's that are outside the parentheses. The js file should have the following format.

    // List of servers for index.htm file
    // Format: server|first archive date (yyyymmdd)
    var servers= new Array(
    "prdibm1b|20051201",
    "p520-sr2|20060219",
    "p520-sr4|20060220",
    "dev-aix-03|20060201"
    );
    I think the problem might be the following line of the nmon2rrd.v2 script. If the "\n" were changed to "n", the output would look like your js file. If so, change it back.

    cat $BACKUP | sed -e "s/\"$/\",\n\"$SERVER\|$YEAR$MM$DAY\"/" > server_list.js

    Here's the kludge that I used. I'm looking for all of the directories in $WEB_DIR and building the list from that...
    function update_server_list {
    cd $WEB_DIR

    a='"'`ls -1p|grep /|sed "s#\/#\|$YEAR$MM$DAY\", \"#g"`
    b=`echo $a|awk '{ gsub(/, " /, ",\042"); print }'`
    c=`echo $b|awk '{ gsub(/, "/, ""); print }'`
    echo "// List of servers for index.htm file">server_list.js
    echo "// Format: server|first archive date (yyyymmdd)">>server_list.js
    echo "var servers= new Array(">>server_list.js
    echo $c|awk '{ gsub(/","/, "\042,\012\042"); print }'>>server_list.js
    echo ");">>server_list.js
    } # end update_server_list
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool/ Error

    ‏2006-03-17T12:27:05Z  
    Here's the kludge that I used. I'm looking for all of the directories in $WEB_DIR and building the list from that...
    function update_server_list {
    cd $WEB_DIR

    a='"'`ls -1p|grep /|sed "s#\/#\|$YEAR$MM$DAY\", \"#g"`
    b=`echo $a|awk '{ gsub(/, " /, ",\042"); print }'`
    c=`echo $b|awk '{ gsub(/, "/, ""); print }'`
    echo "// List of servers for index.htm file">server_list.js
    echo "// Format: server|first archive date (yyyymmdd)">>server_list.js
    echo "var servers= new Array(">>server_list.js
    echo $c|awk '{ gsub(/","/, "\042,\012\042"); print }'>>server_list.js
    echo ");">>server_list.js
    } # end update_server_list
    does this work on Aix 433.
    On my server I have more than 128 disk. When i run it I get the following error.

    "Found 1 DISKBUSY Section(s)
    Warning: More than 128 disks in this DISK section.
    This can not shown on a single graph, so above 128 disks ignored.
    rm -f *.rrd
    rm -f *.gif
    rrdtool - < rrd_create
    sh: rrdtool: not found.
    rrdtool - < rrd_update >rrd_update.log
    sh: rrdtool: not found.
    rrdtool - < rrd_top >rrd_top.log
    sh: rrdtool: not found.
    rrdtool - < rrd_graph
    sh: rrdtool: not found."

    It there any thing i can change to accomodate the disk I have?

    Jubel
  • BruceSpencer
    BruceSpencer
    297 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool/ Error

    ‏2006-03-21T16:16:29Z  
    does this work on Aix 433.
    On my server I have more than 128 disk. When i run it I get the following error.

    "Found 1 DISKBUSY Section(s)
    Warning: More than 128 disks in this DISK section.
    This can not shown on a single graph, so above 128 disks ignored.
    rm -f *.rrd
    rm -f *.gif
    rrdtool - < rrd_create
    sh: rrdtool: not found.
    rrdtool - < rrd_update >rrd_update.log
    sh: rrdtool: not found.
    rrdtool - < rrd_top >rrd_top.log
    sh: rrdtool: not found.
    rrdtool - < rrd_graph
    sh: rrdtool: not found."

    It there any thing i can change to accomodate the disk I have?

    Jubel
    The message you see regarding disks is a nmon or nmor2rrd error message. It's not coming from the shell scripts.

    PS do you have the rrdtool installed? The error message
    "sh: rrdtool: not found." implies it needs to be installed. If so you can download it from the Linux Toolbox for AIX or from the UCLA web page (see documentation in the shell script).
  • BruceSpencer
    BruceSpencer
    297 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2006-03-21T16:37:31Z  
    Here's the kludge that I used. I'm looking for all of the directories in $WEB_DIR and building the list from that...
    function update_server_list {
    cd $WEB_DIR

    a='"'`ls -1p|grep /|sed "s#\/#\|$YEAR$MM$DAY\", \"#g"`
    b=`echo $a|awk '{ gsub(/, " /, ",\042"); print }'`
    c=`echo $b|awk '{ gsub(/, "/, ""); print }'`
    echo "// List of servers for index.htm file">server_list.js
    echo "// Format: server|first archive date (yyyymmdd)">>server_list.js
    echo "var servers= new Array(">>server_list.js
    echo $c|awk '{ gsub(/","/, "\042,\012\042"); print }'>>server_list.js
    echo ");">>server_list.js
    } # end update_server_list
    Have you got this working? I tried your code on my server and I see an extra apostrophe. The server_list.js format needs to be as in my last post. If your output looks like this, you should be good to go.
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2006-04-28T22:36:47Z  
    Scott:

    As a sort term mitigation to your javascript problem, manually edit the server_list.js file. Remove the n's that are outside the parentheses. The js file should have the following format.

    // List of servers for index.htm file
    // Format: server|first archive date (yyyymmdd)
    var servers= new Array(
    "prdibm1b|20051201",
    "p520-sr2|20060219",
    "p520-sr4|20060220",
    "dev-aix-03|20060201"
    );
    I think the problem might be the following line of the nmon2rrd.v2 script. If the "\n" were changed to "n", the output would look like your js file. If so, change it back.

    cat $BACKUP | sed -e "s/\"$/\",\n\"$SERVER\|$YEAR$MM$DAY\"/" > server_list.js

    Scott/Bruce,

    The following modification resolved the problem for me:

    Original Line:
    cat $BACKUP | sed -e "s/\"$/\",\n\"$SERVER\|$YEAR$MM$DAY\"/" > server_list.js

    Modified Line:
    cat $BACKUP | sed -e "s/\"$/\",\\
    \"$SERVER\|$YEAR$MM$DAY\"/" > server_list.js

    Apparently the "\n" newline value only works with gnu sed. Bourne/KSH you have to use one backslash "\" then press "Enter" if the script is using single quotes, or in this case two backslashes "\\" since we are using double quotes.
  • DanCreed
    DanCreed
    3 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2008-10-09T22:00:53Z  
    Any idea why this is happening... I get blank LPAR graphs...
    Thanks,
    Dan
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2008-10-16T15:12:14Z  
    • DanCreed
    • ‏2008-10-09T22:00:53Z
    Any idea why this is happening... I get blank LPAR graphs...
    Thanks,
    Dan
    I've the same problem.

    My nmon files are :
    -rwxr-xr-x 210 200 515179 Mar 30 23:27:57 2008 nmon12e_aix534
    -rwxr-xr-x 210 200 515368 Mar 30 23:27:57 2008 nmon12e_aix535
    -rwxr-xr-x 210 200 539536 Mar 30 23:27:57 2008 nmon12e_aix536
    -rwxr-xr-x 210 200 542388 Mar 30 23:27:57 2008 nmon12e_aix537

    I work with nmon2rrdv12f

    Thanks for your help.
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2009-03-03T08:08:39Z  
    I've the same problem.

    My nmon files are :
    -rwxr-xr-x 210 200 515179 Mar 30 23:27:57 2008 nmon12e_aix534
    -rwxr-xr-x 210 200 515368 Mar 30 23:27:57 2008 nmon12e_aix535
    -rwxr-xr-x 210 200 539536 Mar 30 23:27:57 2008 nmon12e_aix536
    -rwxr-xr-x 210 200 542388 Mar 30 23:27:57 2008 nmon12e_aix537

    I work with nmon2rrdv12f

    Thanks for your help.
    Me too,

    after migrating to 12e the following graphs are missing :

    Absolute CPU Utilization vs. Entitlement
    Shared Pool Utlization
    Asynchronous I/O

    I've checked the raw nmon files with Stephen Atkins' nmon analyser macro and the data is still there, but somehow the nmon2web.pl script drops this information during graph processing.

    It doesn't matter if I use the nmon 12e download from Nigel's wiki or the built-in nmon in AIX5.3 TL9.

    Any ideas ?? I would really appreciate it.

    Thanks
    Marc
  • BruceSpencer
    BruceSpencer
    297 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2009-03-04T02:36:52Z  
    Me too,

    after migrating to 12e the following graphs are missing :

    Absolute CPU Utilization vs. Entitlement
    Shared Pool Utlization
    Asynchronous I/O

    I've checked the raw nmon files with Stephen Atkins' nmon analyser macro and the data is still there, but somehow the nmon2web.pl script drops this information during graph processing.

    It doesn't matter if I use the nmon 12e download from Nigel's wiki or the built-in nmon in AIX5.3 TL9.

    Any ideas ?? I would really appreciate it.

    Thanks
    Marc
    There's a new nmon2web.pl file for nmon12. Nigel rearranged some fields in nmon12e output, and it doesn't parse the same.

    You can download the new nmon2web version at link:http://www.aixtips.com/AIXtip/nmon2web.htm

    The only file you need out of the tar file is nmon2web.pl. Simply replace the old with the new. (Be sure to change the directories in the customization section to reflect the location of your files.)

    Along with nmon12 support, I added some enhancements. In the past you were limited to 1 nmon file per day. I've removed that restriction, and you can now update at any frequency (within reason). I suggest a minimum of 1-2 hour sample size at 15-20 min intervals). See the above web site for more information.
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2009-03-04T16:41:17Z  
    There's a new nmon2web.pl file for nmon12. Nigel rearranged some fields in nmon12e output, and it doesn't parse the same.

    You can download the new nmon2web version at link:http://www.aixtips.com/AIXtip/nmon2web.htm

    The only file you need out of the tar file is nmon2web.pl. Simply replace the old with the new. (Be sure to change the directories in the customization section to reflect the location of your files.)

    Along with nmon12 support, I added some enhancements. In the past you were limited to 1 nmon file per day. I've removed that restriction, and you can now update at any frequency (within reason). I suggest a minimum of 1-2 hour sample size at 15-20 min intervals). See the above web site for more information.
    Hello Bruce,

    I have noticed the missing graph data too. I am using your latest version of nmon2web.pl and have just attempted to process daily nmon files created over the last 3 months for an LPAR. I can see it breaking in Feb when I ugraded to nmon 12e.

    For info I am running AIX 5.3 TL8 and therefore nmon12e_aix537.

    Any thoughts appreciated.
  • BruceSpencer
    BruceSpencer
    297 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2009-03-05T02:32:20Z  
    Hello Bruce,

    I have noticed the missing graph data too. I am using your latest version of nmon2web.pl and have just attempted to process daily nmon files created over the last 3 months for an LPAR. I can see it breaking in Feb when I ugraded to nmon 12e.

    For info I am running AIX 5.3 TL8 and therefore nmon12e_aix537.

    Any thoughts appreciated.
    Can you tar the nmon files to this site? I'd like to see the 3 days before and 3 days after the break in February. I'll test it on my server. It's possible you have a combination I haven't tested.

    How is it working otherwise?
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2009-03-05T08:22:23Z  
    There's a new nmon2web.pl file for nmon12. Nigel rearranged some fields in nmon12e output, and it doesn't parse the same.

    You can download the new nmon2web version at link:http://www.aixtips.com/AIXtip/nmon2web.htm

    The only file you need out of the tar file is nmon2web.pl. Simply replace the old with the new. (Be sure to change the directories in the customization section to reflect the location of your files.)

    Along with nmon12 support, I added some enhancements. In the past you were limited to 1 nmon file per day. I've removed that restriction, and you can now update at any frequency (within reason). I suggest a minimum of 1-2 hour sample size at 15-20 min intervals). See the above web site for more information.
    Hello Bruce,

    I'm already using the new nmon2web.pl script. I'm using AIX 5.3 TL07 SP3 and AIX 5.3 TL09 SP2 and when the nmon data file is processed then the 3 graphs mentioned above are missing. All other graphs are working fine as far as I can tell.

    I've attached some nmon data files. The files in february (TL07) show the lpar graphs, the march files (TL09) don't.

    Regards
    Marc
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2009-03-05T11:13:18Z  
    Can you tar the nmon files to this site? I'd like to see the 3 days before and 3 days after the break in February. I'll test it on my server. It's possible you have a combination I haven't tested.

    How is it working otherwise?
    Hello Bruce,

    Thank you very much for your help in this! I find your nmon2web script a very useful tool.

    I have attached a few more files to help you compare. For me the graphs stop appearing from 2nd Feb when, as I have said, I upgraded nmon to 12e. Also included in the attachment is an example trend graph which shows when it breaks. Worryingly the "Over all CPU Utilization" graph suddenly jumps up at the same time so I suspect that this is not a true representation. I have checked this on a few other trend graphs and they all share this pattern.

    Regards,
    Steve
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2009-03-05T14:15:59Z  
    Hello Bruce,

    Thank you very much for your help in this! I find your nmon2web script a very useful tool.

    I have attached a few more files to help you compare. For me the graphs stop appearing from 2nd Feb when, as I have said, I upgraded nmon to 12e. Also included in the attachment is an example trend graph which shows when it breaks. Worryingly the "Over all CPU Utilization" graph suddenly jumps up at the same time so I suspect that this is not a true representation. I have checked this on a few other trend graphs and they all share this pattern.

    Regards,
    Steve
    Hello Steve,

    I see exactly the same behaviour of the Overall CPU graph.

    Regards
    Marc
  • BruceSpencer
    BruceSpencer
    297 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2009-03-05T14:22:50Z  
    Hello Bruce,

    I'm already using the new nmon2web.pl script. I'm using AIX 5.3 TL07 SP3 and AIX 5.3 TL09 SP2 and when the nmon data file is processed then the 3 graphs mentioned above are missing. All other graphs are working fine as far as I can tell.

    I've attached some nmon data files. The files in february (TL07) show the lpar graphs, the march files (TL09) don't.

    Regards
    Marc
    I have some preliminary results. I ran your files on my server and have attached the Trend output as a pdf file.

    On one hand I don't see a problem with missing charts. I see both "CPU Utilization" and "Shared Pool" (1st and second graph. The Aysnch I/O chart is missing because that data wasn't in your nmon file). Can you verify these are the charts you are missing?

    On the other hand, I see a problem that could cause missing charts. You are using two versions of nmon (surprise!). This could be a problem.

    The problem started with the AIX upgrade from TL7 to TL9. At TL7 you were running nmon 12e. At TL9 you are running nmon "nmon-topas". As you may know, nmon is part of AIX starting with TL9. (Congratulations Nigel!). However, I recall Nigel saying they incorporated nmon11 into TL9. Apparently your PATH statement is such that you are now using "nmon-topas" after the upgrade.

    Try changing your PATH statement to use nmon12, and see if that helps.

    I haven't completed my tests, and I won't be able to finish till tomorrow as I'm traveling today.
  • BruceSpencer
    BruceSpencer
    297 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2009-03-05T14:37:20Z  
    I have some preliminary results. I ran your files on my server and have attached the Trend output as a pdf file.

    On one hand I don't see a problem with missing charts. I see both "CPU Utilization" and "Shared Pool" (1st and second graph. The Aysnch I/O chart is missing because that data wasn't in your nmon file). Can you verify these are the charts you are missing?

    On the other hand, I see a problem that could cause missing charts. You are using two versions of nmon (surprise!). This could be a problem.

    The problem started with the AIX upgrade from TL7 to TL9. At TL7 you were running nmon 12e. At TL9 you are running nmon "nmon-topas". As you may know, nmon is part of AIX starting with TL9. (Congratulations Nigel!). However, I recall Nigel saying they incorporated nmon11 into TL9. Apparently your PATH statement is such that you are now using "nmon-topas" after the upgrade.

    Try changing your PATH statement to use nmon12, and see if that helps.

    I haven't completed my tests, and I won't be able to finish till tomorrow as I'm traveling today.
    There is a different problem on shammer's data. I see where the graph stops after changing from nmon 11e to 12e. I haven't identified the problem yet, and I have to travel today. I'll look at it when I return.

    (PS the PDF file in my last post shows 3 missing days of data in the middle of the chart. This is not a problem because there were no nmon files for those days.)
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: Automating the Conversion of nmon csv Files into Web Pages Using nmon2rrd tool

    ‏2009-03-05T16:21:12Z  
    I have some preliminary results. I ran your files on my server and have attached the Trend output as a pdf file.

    On one hand I don't see a problem with missing charts. I see both "CPU Utilization" and "Shared Pool" (1st and second graph. The Aysnch I/O chart is missing because that data wasn't in your nmon file). Can you verify these are the charts you are missing?

    On the other hand, I see a problem that could cause missing charts. You are using two versions of nmon (surprise!). This could be a problem.

    The problem started with the AIX upgrade from TL7 to TL9. At TL7 you were running nmon 12e. At TL9 you are running nmon "nmon-topas". As you may know, nmon is part of AIX starting with TL9. (Congratulations Nigel!). However, I recall Nigel saying they incorporated nmon11 into TL9. Apparently your PATH statement is such that you are now using "nmon-topas" after the upgrade.

    Try changing your PATH statement to use nmon12, and see if that helps.

    I haven't completed my tests, and I won't be able to finish till tomorrow as I'm traveling today.
    Yes, in your PDF all the graphs that I'm missing are there. Strange...

    I will try to use nmon12e with TL09. I didn't know that they used nmon11 in TL09. I thought it would be nmon12e as well.

    Thank you very much for your fast help !!!

    Kind regards
    Marc