Topic
  • 23 replies
  • Latest Post - ‏2013-01-02T12:46:31Z by nagger
SystemAdmin
SystemAdmin
2402 Posts

Pinned topic nmon2rrd vworking binary with AIX 7.1 and topas_nmon

‏2012-10-02T12:43:17Z |
Attached binary should work with topas_nmon and aix 7.1.
Will attach source to next thread

Vesa
Updated on 2013-01-02T12:46:31Z at 2013-01-02T12:46:31Z by nagger
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-10-02T12:44:54Z  
    And here is the ugly source code.

    Sorry, I'm not a C coder - just desperate :P

    Vesa

    Attachments

  • armink
    armink
    30 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-10-19T13:58:11Z  
    Many thanks for making things work with AIX 7.1!

    Can you describe the differences between the earlier versions (e.g. in AIX 6.1 and 5.3)?
    Your binary works fine with AIX 7.1 but some graphs (pools, entitlement, memory, system parameters) are missing with nmon in AIX 6.1 and 5.3.
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-10-24T10:51:07Z  
    • armink
    • ‏2012-10-19T13:58:11Z
    Many thanks for making things work with AIX 7.1!

    Can you describe the differences between the earlier versions (e.g. in AIX 6.1 and 5.3)?
    Your binary works fine with AIX 7.1 but some graphs (pools, entitlement, memory, system parameters) are missing with nmon in AIX 6.1 and 5.3.
    Sure,
    Nmon2rrd requires nmon data to be specific and previously it detected it by using nmon version information (12h, 12a,...). So, using topas_nmon this no longer works, since version
    is always "topas_nmon". To overcome this, I added detection of the AIX version. For example, if aix version starts with 7 then use setup x for data, if aix version is 6.1.6.15 then use another
    set.

    Since I do not have all possible topas_nmon and Aix version combinations, I have not added them all and obviously it does not work with all AIX versions. There are a lot of holes
    concerning AIX 6.1 versions. I try to guess them and add them when I have more time to do it.

    If your AIX is not supporting topas_nmon, use old nagger nmon and nmon2rrd. I try to avoid using nmon (12) version to record data, when there is topas_nmon available.

    In order to add support for different data sets, I have to peek in to data what is causing the graphs to become empty. Usually its that there is a new column in the
    data. But in 7.1 there has been other issues as well, like unnecessary whitespaces in the data causing C to fail when its expecting a number instead of a character.

    Armink, could you specify AIX 6.1 version which do not work with this one and I can try to make an another compilation for you?
  • armink
    armink
    30 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-10-31T17:54:59Z  
    Sure,
    Nmon2rrd requires nmon data to be specific and previously it detected it by using nmon version information (12h, 12a,...). So, using topas_nmon this no longer works, since version
    is always "topas_nmon". To overcome this, I added detection of the AIX version. For example, if aix version starts with 7 then use setup x for data, if aix version is 6.1.6.15 then use another
    set.

    Since I do not have all possible topas_nmon and Aix version combinations, I have not added them all and obviously it does not work with all AIX versions. There are a lot of holes
    concerning AIX 6.1 versions. I try to guess them and add them when I have more time to do it.

    If your AIX is not supporting topas_nmon, use old nagger nmon and nmon2rrd. I try to avoid using nmon (12) version to record data, when there is topas_nmon available.

    In order to add support for different data sets, I have to peek in to data what is causing the graphs to become empty. Usually its that there is a new column in the
    data. But in 7.1 there has been other issues as well, like unnecessary whitespaces in the data causing C to fail when its expecting a number instead of a character.

    Armink, could you specify AIX 6.1 version which do not work with this one and I can try to make an another compilation for you?
    I have tested things with AIX 5.3 TL12, AIX 6.1 TL6 and TL7... all with "native" topas_nmon.
    If you don't run this version I can supply you with the nmon data.
    Please tell me what data (e.g. what rows) you need and I'll post it here.

    Thanks in advance!
    Armin
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-11-05T10:43:59Z  
    • armink
    • ‏2012-10-31T17:54:59Z
    I have tested things with AIX 5.3 TL12, AIX 6.1 TL6 and TL7... all with "native" topas_nmon.
    If you don't run this version I can supply you with the nmon data.
    Please tell me what data (e.g. what rows) you need and I'll post it here.

    Thanks in advance!
    Armin
    Send me first the 4th line of the nmon report of the both machines. That
    can be found from data as well.

    For example:
    4. AIX: 6.1.6.15
    Vesa
  • armink
    armink
    30 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-11-06T12:50:50Z  
    Send me first the 4th line of the nmon report of the both machines. That
    can be found from data as well.

    For example:
    4. AIX: 6.1.6.15
    Vesa
    What forth line do you mean? In the nmon dump file or in the resulting Excel sheet or anywhere else?
    In the nmon dump it's the 8th line and is like "AAA,AIX," and then
    "5.3.12.5" 5.3 TL12 SP5
    "6.1.6.15" 6.1 TL6 SP5
    "6.1.7.2" 6.1 TL07 SP2
    "6.1.7.3" 6.1 TL07 SP3
    "6.1.7.15" 6.1 TL07 SP4
    "6.1.7.16" 6.1 TL07 SP5

    AAA,version is always TOPAS-NMON
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-11-06T12:56:02Z  
    • armink
    • ‏2012-11-06T12:50:50Z
    What forth line do you mean? In the nmon dump file or in the resulting Excel sheet or anywhere else?
    In the nmon dump it's the 8th line and is like "AAA,AIX," and then
    "5.3.12.5" 5.3 TL12 SP5
    "6.1.6.15" 6.1 TL6 SP5
    "6.1.7.2" 6.1 TL07 SP2
    "6.1.7.3" 6.1 TL07 SP3
    "6.1.7.15" 6.1 TL07 SP4
    "6.1.7.16" 6.1 TL07 SP5

    AAA,version is always TOPAS-NMON
    Yes!
    That is exactly what I was looking for...

    Now i can proceed. BTW, which SP of 6.1 TL7 is equivalent for 6.1 TL6 SP5???
  • armink
    armink
    30 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-11-06T13:50:15Z  
    Yes!
    That is exactly what I was looking for...

    Now i can proceed. BTW, which SP of 6.1 TL7 is equivalent for 6.1 TL6 SP5???
    6.1 TL6 SP5 is 6100-06-05-1115
    6.1 TL7 SP0 is 6100-07-00-1140
    6.1 TL7 SP1 is 6100-07-01-1141

    so for me it looks like there is no equivalent.
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-11-06T21:32:42Z  
    • armink
    • ‏2012-11-06T13:50:15Z
    6.1 TL6 SP5 is 6100-06-05-1115
    6.1 TL7 SP0 is 6100-07-00-1140
    6.1 TL7 SP1 is 6100-07-01-1141

    so for me it looks like there is no equivalent.
    Yep, they seem to be about 25 weeks newer - which makes this a bit easier I hope.
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-11-06T22:09:07Z  
    • armink
    • ‏2012-11-06T12:50:50Z
    What forth line do you mean? In the nmon dump file or in the resulting Excel sheet or anywhere else?
    In the nmon dump it's the 8th line and is like "AAA,AIX," and then
    "5.3.12.5" 5.3 TL12 SP5
    "6.1.6.15" 6.1 TL6 SP5
    "6.1.7.2" 6.1 TL07 SP2
    "6.1.7.3" 6.1 TL07 SP3
    "6.1.7.15" 6.1 TL07 SP4
    "6.1.7.16" 6.1 TL07 SP5

    AAA,version is always TOPAS-NMON
    Attached binary worked with my 6.1.7.2 and 6.1.7.16 when i tested it. Hope this helps?
    (Luckily there were installed while I was abscent for a month :D )

    There is another change as well, binary supports now up to 1500 snapshots instead of 1024.
    So you can gather daily data with exatch 60 second interval which brings 1440 samples
    on a normal day and 1500 when daylight saving times occur.

    Armink, could you send 5.3.12.5 data for inspection. You can alter sensitive data
    manually if you like. 10 snaphots is enough. Or just paste it here?
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-11-06T23:40:49Z  
    Attached binary worked with my 6.1.7.2 and 6.1.7.16 when i tested it. Hope this helps?
    (Luckily there were installed while I was abscent for a month :D )

    There is another change as well, binary supports now up to 1500 snapshots instead of 1024.
    So you can gather daily data with exatch 60 second interval which brings 1440 samples
    on a normal day and 1500 when daylight saving times occur.

    Armink, could you send 5.3.12.5 data for inspection. You can alter sensitive data
    manually if you like. 10 snaphots is enough. Or just paste it here?
    Armin, how does the 5.3.12.5 first line of PROC look like in the data?

    PROC,Processes lpar,Runnable,Swap-in,pswitch,syscall,read,write,fork,exec,sem,msg,asleep_bufio,asleep_rawio,asleep_diocio

    or

    PROC,Processes lpar,Runnable,Swap-in,pswitch,syscall,read,write,fork,exec,sem,msg
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-11-06T23:50:28Z  
    Armin, how does the 5.3.12.5 first line of PROC look like in the data?

    PROC,Processes lpar,Runnable,Swap-in,pswitch,syscall,read,write,fork,exec,sem,msg,asleep_bufio,asleep_rawio,asleep_diocio

    or

    PROC,Processes lpar,Runnable,Swap-in,pswitch,syscall,read,write,fork,exec,sem,msg
    And Armin, 5.3.12.5 MEMUSE, is it
    MEMUSE,Memory Use lparx,%numperm,%minperm,%maxperm,minfree,maxfree,%numclient,%maxclient

    or

    MEMUSE,Memory Use lpary,%numperm,%minperm,%maxperm,minfree,maxfree,%numclient,%maxclient, lruable pages

    PS. Here you can see a little mistake in the data by the coders,
    the whitespace before lruable pages, which is in the actual data too

    MEMUSE,T1023,18.3,3.0,90.0,960,1088,18.3,90.0, 4052736.0

    PS2. I have a new version waiting for your responses, Armin ;-)
  • armink
    armink
    30 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-11-08T15:17:08Z  
    And Armin, 5.3.12.5 MEMUSE, is it
    MEMUSE,Memory Use lparx,%numperm,%minperm,%maxperm,minfree,maxfree,%numclient,%maxclient

    or

    MEMUSE,Memory Use lpary,%numperm,%minperm,%maxperm,minfree,maxfree,%numclient,%maxclient, lruable pages

    PS. Here you can see a little mistake in the data by the coders,
    the whitespace before lruable pages, which is in the actual data too

    MEMUSE,T1023,18.3,3.0,90.0,960,1088,18.3,90.0, 4052736.0

    PS2. I have a new version waiting for your responses, Armin ;-)
    In AIX 5.3 TL12 SP5 (topas_nmon) it's
    PROC,Processes lparx,Runnable,Swap-in,pswitch,syscall,read,write,fork,exec,sem,msg,asleep_bufio,asleep_rawio,asleep_diocio
    and
    MEMUSE,Memory Use lparx,%numperm,%minperm,%maxperm,minfree,maxfree,%numclient,%maxclient

    Since the value "lruable pages" is missing, there is no white space problem either.

    I'll check the new version and post my feedback.

    Thanks a lot for your assistance!
  • armink
    armink
    30 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-11-08T17:52:47Z  
    • armink
    • ‏2012-11-08T15:17:08Z
    In AIX 5.3 TL12 SP5 (topas_nmon) it's
    PROC,Processes lparx,Runnable,Swap-in,pswitch,syscall,read,write,fork,exec,sem,msg,asleep_bufio,asleep_rawio,asleep_diocio
    and
    MEMUSE,Memory Use lparx,%numperm,%minperm,%maxperm,minfree,maxfree,%numclient,%maxclient

    Since the value "lruable pages" is missing, there is no white space problem either.

    I'll check the new version and post my feedback.

    Thanks a lot for your assistance!
    There seems to be a difference in the nmon output (more columns!) if nmon runs with AME enabled.
    This is probably the reason why we don't get certain graphs.

    Here is an example for AIX 7.1 (AME on 6.1 is the same):
    MEM,Memory lparx,Real Free %,Virtual free %,Real free(MB),Virtual free(MB),Real total(MB),Virtual total(MB),Size of the Compressed pool(MB), Size of true memory(MB), Expanded memory size(MB), Size of the Uncompressed pool(MB)
    MEMNEW,Memory New lparx,Process%,FScache%,System%,Free%,Pinned%,User%,Compressed Pool%
    MEMUSE,Memory Use lparx,%numperm,%minperm,%maxperm,minfree,maxfree,%numclient,%maxclient, lruable pages
    PAGE,Paging lparx,faults,pgin,pgout,pgsin,pgsout,reclaims,scans,cycles,Compressed pool pgins, Compressed pool pgouts
    PROC,Processes lparx,Runnable,Swap-in,pswitch,syscall,read,write,fork,exec,sem,msg,asleep_bufio,asleep_rawio,asleep_diocio

    Notice that there are many leading whitespaces... this sucks :-)

    There are still (other) problems with certain TLs of 5.3 and 6.1 TL6 on lpars without AME. I'll post more if I find a pattern.

    Regards,
    Armin
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-11-09T07:05:14Z  
    • armink
    • ‏2012-11-08T15:17:08Z
    In AIX 5.3 TL12 SP5 (topas_nmon) it's
    PROC,Processes lparx,Runnable,Swap-in,pswitch,syscall,read,write,fork,exec,sem,msg,asleep_bufio,asleep_rawio,asleep_diocio
    and
    MEMUSE,Memory Use lparx,%numperm,%minperm,%maxperm,minfree,maxfree,%numclient,%maxclient

    Since the value "lruable pages" is missing, there is no white space problem either.

    I'll check the new version and post my feedback.

    Thanks a lot for your assistance!
    OK, I will add detection for the 5.3.12.5.

    Extra spaces in the data have been resolved by translating the spaces to zero's
    ...0,1,2, 123 --> ...0,1,2,0123
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-11-09T07:42:58Z  
    • armink
    • ‏2012-11-08T17:52:47Z
    There seems to be a difference in the nmon output (more columns!) if nmon runs with AME enabled.
    This is probably the reason why we don't get certain graphs.

    Here is an example for AIX 7.1 (AME on 6.1 is the same):
    MEM,Memory lparx,Real Free %,Virtual free %,Real free(MB),Virtual free(MB),Real total(MB),Virtual total(MB),Size of the Compressed pool(MB), Size of true memory(MB), Expanded memory size(MB), Size of the Uncompressed pool(MB)
    MEMNEW,Memory New lparx,Process%,FScache%,System%,Free%,Pinned%,User%,Compressed Pool%
    MEMUSE,Memory Use lparx,%numperm,%minperm,%maxperm,minfree,maxfree,%numclient,%maxclient, lruable pages
    PAGE,Paging lparx,faults,pgin,pgout,pgsin,pgsout,reclaims,scans,cycles,Compressed pool pgins, Compressed pool pgouts
    PROC,Processes lparx,Runnable,Swap-in,pswitch,syscall,read,write,fork,exec,sem,msg,asleep_bufio,asleep_rawio,asleep_diocio

    Notice that there are many leading whitespaces... this sucks :-)

    There are still (other) problems with certain TLs of 5.3 and 6.1 TL6 on lpars without AME. I'll post more if I find a pattern.

    Regards,
    Armin
    This AME will cause lots of head scratching :P

    To MEM, MEMNEW and PAGE AME has an influence. We need a extra detection for AME to the nmon2rrd...
    MEMUSE and PROC are as usual.

    I would be kind of easy thing to do, if AME had been mentioned in the AAA fields.
    I checked our P7 aix 7.1 data where AME is off and there nothing that refers to it.
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-11-09T07:55:18Z  
    • armink
    • ‏2012-11-08T15:17:08Z
    In AIX 5.3 TL12 SP5 (topas_nmon) it's
    PROC,Processes lparx,Runnable,Swap-in,pswitch,syscall,read,write,fork,exec,sem,msg,asleep_bufio,asleep_rawio,asleep_diocio
    and
    MEMUSE,Memory Use lparx,%numperm,%minperm,%maxperm,minfree,maxfree,%numclient,%maxclient

    Since the value "lruable pages" is missing, there is no white space problem either.

    I'll check the new version and post my feedback.

    Thanks a lot for your assistance!
    One additional question regarding 5.3.12.5, is there "Folded" column in the LPAR field?

    LPAR,Logical Partition lparx,PhysicalCPU,virtualCPUs,logicalCPUs,poolCPUs,entitled,weight,PoolIdle,usedAllCPU%,usedPoolCPU%,Sh
    aredCPU,Capped,EC_User%,EC_Sys%,EC_Wait%,EC_Idle%,VP_User%,VP_Sys%,VP_Wait%,VP_Idle%,Folded,Pool_id
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-11-09T08:41:09Z  
    • armink
    • ‏2012-11-08T15:17:08Z
    In AIX 5.3 TL12 SP5 (topas_nmon) it's
    PROC,Processes lparx,Runnable,Swap-in,pswitch,syscall,read,write,fork,exec,sem,msg,asleep_bufio,asleep_rawio,asleep_diocio
    and
    MEMUSE,Memory Use lparx,%numperm,%minperm,%maxperm,minfree,maxfree,%numclient,%maxclient

    Since the value "lruable pages" is missing, there is no white space problem either.

    I'll check the new version and post my feedback.

    Thanks a lot for your assistance!
    Attached version 12o where is 5.3.12.5 detection and it expects that there is the Folded field in the LPAR data.
    Please, test it Armin.
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-11-09T13:47:34Z  
    This AME will cause lots of head scratching :P

    To MEM, MEMNEW and PAGE AME has an influence. We need a extra detection for AME to the nmon2rrd...
    MEMUSE and PROC are as usual.

    I would be kind of easy thing to do, if AME had been mentioned in the AAA fields.
    I checked our P7 aix 7.1 data where AME is off and there nothing that refers to it.
    AME with MEM and MEMNEW resolved, PAGE left to the next week to be coded.
    Have a nice weekend...
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-11-12T07:20:13Z  
    AME with MEM and MEMNEW resolved, PAGE left to the next week to be coded.
    Have a nice weekend...
    Good morning
    Here is attached latest AME aware 12p version of the nmon2rrd. Now Paging and Memory graphs shouldn't be
    empty when AME is on.

    Armin could you test it?

    I will test it on these AIX versions where AME is off:
    5.3.0.56
    5.3.11.3
    5.3.8.3
    6.1.4.7
    6.1.6.15
    6.1.6.3
    6.1.7.16
    6.1.7.2

    And AME on:
    7.1.1.16
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-11-15T21:29:54Z  
    Good morning
    Here is attached latest AME aware 12p version of the nmon2rrd. Now Paging and Memory graphs shouldn't be
    empty when AME is on.

    Armin could you test it?

    I will test it on these AIX versions where AME is off:
    5.3.0.56
    5.3.11.3
    5.3.8.3
    6.1.4.7
    6.1.6.15
    6.1.6.3
    6.1.7.16
    6.1.7.2

    And AME on:
    7.1.1.16
    Now tested and works like charm...

    Armin, when you are back, please test it and report if its any better now?

    R Vesa
  • SystemAdmin
    SystemAdmin
    2402 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2012-12-31T10:16:39Z  
    And here is the ugly source code.

    Sorry, I'm not a C coder - just desperate :P

    Vesa
    Oops,
    Here is the C source of the 12p version of the nmon2rrd, which I have forgotten to attach. Sorry for the delay :}

    Happy New Year,
    Vesa

    Attachments

  • nagger
    nagger
    1638 Posts

    Re: nmon2rrd vworking binary with AIX 7.1 and topas_nmon

    ‏2013-01-02T12:46:31Z  
    Attached version 12o where is 5.3.12.5 detection and it expects that there is the Folded field in the LPAR data.
    Please, test it Armin.
    Hi,
    I have just released nmon2rrd version 16a and you can find it at
    https://www.ibm.com/developerworks/wikis/display/WikiPtype/nmon
    • This version 16 has drastic changes which extracts column header names from the nmon file which makes it less sensitive to new columns and removed loads of code working around the variations in file output over the years.
    • It has also been beautified with the AIX cb -s command to fix indentation mistakes
    • Change comments (peoples names and company names) removed including mine - this is a sample code
    • This needs to be done to remove ownership issues in the future and gets messy.

    I really appreciate the fixes supplied and have tried to include them but this version needs a good regression test.
    • Note the MEM section columns now match the nmon output file. Does this cause a problem with older .rrd files that you have been adding to for a longer period?
    • please let me know either way - as we could add a work around.

    Can anyone testing the new version and finding a failure please report the issue and the problem file so I can reproduce the issue? but NOT reported here 22 replies down the list is hard to find - Can we have a new Forum Entry would be good with say "nmon2rrd v16 problem" in the title.

    Thanks Nigel Griffiths