Topic
  • 11 replies
  • Latest Post - ‏2014-06-20T09:08:50Z by fbenfredj
fbenfredj
fbenfredj
40 Posts

Pinned topic Nmon2web : multiple nmon2web.pl processes

‏2014-06-03T10:35:57Z |

Hi Bruce,

I have problem with nmon2web : many nmon2web.pl processes are blocked which causes low performance and time out in nmon2web.cgi.

    nmon 11796578 14221420   8   Jun 02      - 14:11 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 14483510 14221420   0   Jun 02      - 15:12 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 21233700 14221420  10 00:06:00      -  2:58 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon  9240610 14221420  12 02:06:00      -  2:39 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 14745658 14221420   8 03:06:00      -  2:14 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 12058668 14221420  11 04:06:00      -  1:58 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 23396394 14221420   0 05:06:00      -  1:43 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 21102624 14221420  13 06:06:00      -  1:27 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 19267812 14221420   0 07:06:00      -  1:21 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 16056366 14221420  12 08:06:00      -  1:10 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 15728802 14221420  12 09:06:00      -  0:56 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 10027232 14221420  11 10:06:00      -  0:42 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon  6553844 14221420  21 11:06:00      -  0:23 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 10092730 14221420   0 12:06:00      -  0:02 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 22151358 14221420   8 13:06:00      - 13:53 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 21626972 14221420   0 14:06:00      - 13:52 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 24379516 14221420  15 18:06:00      -  4:30 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 15597700 14221420   8 19:06:00      -  4:05 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 12976158 14221420  10 20:06:00      -  4:05 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 17236116 14221420   9 22:06:00      -  3:32 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
    nmon 16318608 14221420  12 23:06:00      -  3:15 /usr/bin/perl /usr2/produits/nmon2web/nmon2web.pl
 

Thanks for your help

 

 

  • BruceSpencer
    BruceSpencer
    297 Posts

    Re: Nmon2web : multiple nmon2web.pl processes

    ‏2014-06-09T15:40:09Z  

    The nmon2web.pl is not completing the ETL processing correctly.  The process should die when done, and at most you should see one nmon2web.pl process.  The number of nmon2web.pl processes on your system isn't "normal".

    It appears you kick off nmon2web.pl hourly.  A problem could occur if the previous job is still running.  The two processes could compete for processing the nmon files.  So, how long does processing take?  How many servers are you processing?

    If this isn't the problem, post a couple of your nmon files.  There's always the possibility of something new in the data that I haven't seen before.

     

     

     

  • fbenfredj
    fbenfredj
    40 Posts

    Re: Nmon2web : multiple nmon2web.pl processes

    ‏2014-06-10T07:50:21Z  

    The nmon2web.pl is not completing the ETL processing correctly.  The process should die when done, and at most you should see one nmon2web.pl process.  The number of nmon2web.pl processes on your system isn't "normal".

    It appears you kick off nmon2web.pl hourly.  A problem could occur if the previous job is still running.  The two processes could compete for processing the nmon files.  So, how long does processing take?  How many servers are you processing?

    If this isn't the problem, post a couple of your nmon files.  There's always the possibility of something new in the data that I haven't seen before.

     

     

     

    Yes, that was the problem. 

    So, I decreased the number of servers from 95 to 87 and  increased the interval of nmon crontab on some servers  from 1 mn to 2 mn. 

    It a problem now to add other servers. 

  • BruceSpencer
    BruceSpencer
    297 Posts

    Re: Nmon2web : multiple nmon2web.pl processes

    ‏2014-06-10T20:39:43Z  
    • fbenfredj
    • ‏2014-06-10T07:50:21Z

    Yes, that was the problem. 

    So, I decreased the number of servers from 95 to 87 and  increased the interval of nmon crontab on some servers  from 1 mn to 2 mn. 

    It a problem now to add other servers. 

    Here are a couple ideas to reduce the ETL time. 

    1.  Reduce the number of charts.

    2.  Run parallel nmon2web.pl jobs. 

     

    I modeled nmon2web output after nmon2rrd.  Personally I think there are too many charts.  If you agree, you can comment out unneeded charts in the nmon2web.pl script at about line  203.  Just put a "#" in front of the print statement as I've illustrated below in red.

    #######################
    #### Static charts ####
    #######################

    # print ("rrdgraph: static charts\n"); # Debug

    print RRDGRAPH (qq|graph cpu_all.gif -r -l 0 -u 100 -v "Percent Stacked" --start $START --end $UTC_END --width $WIDTH --height $HEIGHT --title "Over all CPU Utilization $HOSTNAME ($DATE)" DEF:User=$RDIR/cpu_all.rrd:UserPct:AVERAGE AREA:User#0000FF:"User%" DEF:Sys=$RDIR/cpu_all.rrd:SysPct:AVERAGE STACK:Sys#00FF00:"Sys%" DEF:Wait=$RDIR/cpu_all.rrd:WaitPct:AVERAGE STACK:Wait#FF0000:"Wait%" \n|);

    # print RRDGRAPH (qq|graph system_file.gif -r -l 0 -v "Per Second" --start $START --end $UTC_END --width $WIDTH --height $HEIGHT --title "File System functions $HOSTNAME ($DATE)" DEF:iget=$RDIR/file.rrd:iget:AVERAGE LINE3:iget#FF0000:"iget" DEF:namei=$RDIR/file.rrd:namei:AVERAGE LINE3:namei#00FF00:"namei" DEF:dirblk=$RDIR/file.rrd:dirblk:AVERAGE LINE3:dirblk#0000FF:"dirblk" \n|);
     

    Another possibility would be to run multiple nmon2web.pl jobs in parallel.  Each job would process a different set of servers.  For example, you could create 2 copies of nmon2web.pl  ( eg  nmon2web_a-m.pl,  nmon2web_n-z.pl). Then modify line 64 in each file as follows:

    nmon2web_a-m.pl:   @nmon_files=`ls $NMON_DIR/[a-m]*.nmon $NMON_DIR/[a-m]*.csv 2>/dev/null`;
    nmon2web_n-z.pl:   @nmon_files=`ls $NMON_DIR/[n-z]*.nmon $NMON_DIR/[n-z]*.csv 2>/dev/null`;

    Run these scripts in parallel.  (My file name ranges were just an illustration.  You can run as many copies as makes sense for your operation.  The only requirement is that the ranges don't overlap, and spread the load evenly between the scripts.)

     

     

  • BruceSpencer
    BruceSpencer
    297 Posts

    Re: Nmon2web : multiple nmon2web.pl processes

    ‏2014-06-11T14:42:42Z  

    Here are a couple ideas to reduce the ETL time. 

    1.  Reduce the number of charts.

    2.  Run parallel nmon2web.pl jobs. 

     

    I modeled nmon2web output after nmon2rrd.  Personally I think there are too many charts.  If you agree, you can comment out unneeded charts in the nmon2web.pl script at about line  203.  Just put a "#" in front of the print statement as I've illustrated below in red.

    #######################
    #### Static charts ####
    #######################

    # print ("rrdgraph: static charts\n"); # Debug

    print RRDGRAPH (qq|graph cpu_all.gif -r -l 0 -u 100 -v "Percent Stacked" --start $START --end $UTC_END --width $WIDTH --height $HEIGHT --title "Over all CPU Utilization $HOSTNAME ($DATE)" DEF:User=$RDIR/cpu_all.rrd:UserPct:AVERAGE AREA:User#0000FF:"User%" DEF:Sys=$RDIR/cpu_all.rrd:SysPct:AVERAGE STACK:Sys#00FF00:"Sys%" DEF:Wait=$RDIR/cpu_all.rrd:WaitPct:AVERAGE STACK:Wait#FF0000:"Wait%" \n|);

    # print RRDGRAPH (qq|graph system_file.gif -r -l 0 -v "Per Second" --start $START --end $UTC_END --width $WIDTH --height $HEIGHT --title "File System functions $HOSTNAME ($DATE)" DEF:iget=$RDIR/file.rrd:iget:AVERAGE LINE3:iget#FF0000:"iget" DEF:namei=$RDIR/file.rrd:namei:AVERAGE LINE3:namei#00FF00:"namei" DEF:dirblk=$RDIR/file.rrd:dirblk:AVERAGE LINE3:dirblk#0000FF:"dirblk" \n|);
     

    Another possibility would be to run multiple nmon2web.pl jobs in parallel.  Each job would process a different set of servers.  For example, you could create 2 copies of nmon2web.pl  ( eg  nmon2web_a-m.pl,  nmon2web_n-z.pl). Then modify line 64 in each file as follows:

    nmon2web_a-m.pl:   @nmon_files=`ls $NMON_DIR/[a-m]*.nmon $NMON_DIR/[a-m]*.csv 2>/dev/null`;
    nmon2web_n-z.pl:   @nmon_files=`ls $NMON_DIR/[n-z]*.nmon $NMON_DIR/[n-z]*.csv 2>/dev/null`;

    Run these scripts in parallel.  (My file name ranges were just an illustration.  You can run as many copies as makes sense for your operation.  The only requirement is that the ranges don't overlap, and spread the load evenly between the scripts.)

     

     

    Upon further reflection, if you choose to run nmon2web in parallel, add four file locks (flock) to nmon2web.pl.  The locks would protect the file server_list.js from concurrent writes.  Add the "flock" statements to the subroutine "server_list_js".

     

    sub server_list_js {

    my @js;  # Javascript file for index.html
    my $i;

    # Month has to be in 2 digit format, 01, 02, ....
    if ( length($MON) eq 1 ) { $MON="0" . $MON;}

    if ( -f "$WEB_FULLPATH/server_list.js" ){

        ### File server_list.js exists ###
        open(OUTFILE, "$WEB_FULLPATH/server_list.js");
        flock(OUTFILE,1);
        @js=<OUTFILE>;
        close(OUTFILE);

        # Add or update timestamp for server update
        for($i=0;$i<@js;$i++){

            # Changed 4/28/2009 to check for unique SN-Hostname.
            # Previously assumed unique Hostname. Not true w/partion mobility
            if ($js[$i] =~ /$SN\|$TYPE\|$HOSTNAME\|/ ) {  # Update

               # Host found, change last timestamp field (col4)
               @cols=split(/\|/,$js[$i]);
               $cols[4]="$YR$MON$DAY";
               $cols[5]="$MODEL\"\n";
               $js[$i]=join('|',@cols);
               unless (open(OUTFILE, ">$WEB_FULLPATH/server_list.js")) {
                    die ("Unable to open $WEB_FULLPATH/server_list.js "); }
               flock(OUTFILE,2);
               print OUTFILE (@js);
               close (OUTFILE);
               return(0);

            } elsif ( $js[$i] =~ /\)\;/ ) { # Add host

               $js[$i]=",\"$SN|$TYPE|$HOSTNAME|$YR$MON$DAY|$YR$MON$DAY|$MODEL\"\n";
               $js[$i+1]=');';
               unless (open(OUTFILE, ">$WEB_FULLPATH/server_list.js")) {
                    die ("Unable to open $WEB_FULLPATH/server_list.js "); }
               flock(OUTFILE,2);
               print OUTFILE (@js);
               close (OUTFILE);
               return(0);

            } # end if

        }# end for

    } else {
        ## Create js file (one time only)
        unless (open(OUTFILE, ">$WEB_FULLPATH/server_list.js")) { die ("Unable to open $WEB_FULLPATH/server_list.js "); }
        flock(OUTFILE,2);

    print OUTFILE <<END_JS;
    // List of servers for index.htm file
    // Format:  server|type|hostname|first archive date|last date (yyyymmdd)
    var servers= new Array(
    "$SN|$TYPE|$HOSTNAME|$YR$MON$DAY|$YR$MON$DAY|$MODEL"
    );
    END_JS
    close(OUTFILE);

    } # end if

    } # end
     

     

     

    Updated on 2014-06-11T14:45:11Z at 2014-06-11T14:45:11Z by BruceSpencer
  • fbenfredj
    fbenfredj
    40 Posts

    Re: Nmon2web : multiple nmon2web.pl processes

    ‏2014-06-12T07:55:54Z  

    Upon further reflection, if you choose to run nmon2web in parallel, add four file locks (flock) to nmon2web.pl.  The locks would protect the file server_list.js from concurrent writes.  Add the "flock" statements to the subroutine "server_list_js".

     

    sub server_list_js {

    my @js;  # Javascript file for index.html
    my $i;

    # Month has to be in 2 digit format, 01, 02, ....
    if ( length($MON) eq 1 ) { $MON="0" . $MON;}

    if ( -f "$WEB_FULLPATH/server_list.js" ){

        ### File server_list.js exists ###
        open(OUTFILE, "$WEB_FULLPATH/server_list.js");
        flock(OUTFILE,1);
        @js=<OUTFILE>;
        close(OUTFILE);

        # Add or update timestamp for server update
        for($i=0;$i<@js;$i++){

            # Changed 4/28/2009 to check for unique SN-Hostname.
            # Previously assumed unique Hostname. Not true w/partion mobility
            if ($js[$i] =~ /$SN\|$TYPE\|$HOSTNAME\|/ ) {  # Update

               # Host found, change last timestamp field (col4)
               @cols=split(/\|/,$js[$i]);
               $cols[4]="$YR$MON$DAY";
               $cols[5]="$MODEL\"\n";
               $js[$i]=join('|',@cols);
               unless (open(OUTFILE, ">$WEB_FULLPATH/server_list.js")) {
                    die ("Unable to open $WEB_FULLPATH/server_list.js "); }
               flock(OUTFILE,2);
               print OUTFILE (@js);
               close (OUTFILE);
               return(0);

            } elsif ( $js[$i] =~ /\)\;/ ) { # Add host

               $js[$i]=",\"$SN|$TYPE|$HOSTNAME|$YR$MON$DAY|$YR$MON$DAY|$MODEL\"\n";
               $js[$i+1]=');';
               unless (open(OUTFILE, ">$WEB_FULLPATH/server_list.js")) {
                    die ("Unable to open $WEB_FULLPATH/server_list.js "); }
               flock(OUTFILE,2);
               print OUTFILE (@js);
               close (OUTFILE);
               return(0);

            } # end if

        }# end for

    } else {
        ## Create js file (one time only)
        unless (open(OUTFILE, ">$WEB_FULLPATH/server_list.js")) { die ("Unable to open $WEB_FULLPATH/server_list.js "); }
        flock(OUTFILE,2);

    print OUTFILE <<END_JS;
    // List of servers for index.htm file
    // Format:  server|type|hostname|first archive date|last date (yyyymmdd)
    var servers= new Array(
    "$SN|$TYPE|$HOSTNAME|$YR$MON$DAY|$YR$MON$DAY|$MODEL"
    );
    END_JS
    close(OUTFILE);

    } # end if

    } # end
     

     

     

    It is a good idea to run 2 nmon2web.pl in parallel. 

    But I can not do it. In fact, nmon2web consume 1 P7 processor permanently. 

    And I do not have enough processors for my perf lpar. 

     

     

  • BruceSpencer
    BruceSpencer
    297 Posts

    Re: Nmon2web : multiple nmon2web.pl processes

    ‏2014-06-12T15:01:04Z  
    • fbenfredj
    • ‏2014-06-12T07:55:54Z

    It is a good idea to run 2 nmon2web.pl in parallel. 

    But I can not do it. In fact, nmon2web consume 1 P7 processor permanently. 

    And I do not have enough processors for my perf lpar. 

     

     

    Running in parallel still might help.  The nmon2web process is single threaded and the P7 processor is multithreaded.  In P7, 100% utilization doesn't mean 100% capacity.  There may be room for more throughput.  If your P7 processor is in a shared pool, you can give the partition a lower weight, or even cap it.

    What nmon2web version are you running?  The latest version* (Feb 2014) creates half the charts compared to previous versions, which translates to about a 25% improvement. (Previous versions created daily and trend charts. The current version creates just the daily charts.  The trend charts are created dynamically by the web user).  On my old P5 520, it takes roughly 24 seconds per nmon file, or about 120 files per hour.   This should handle your 95 servers on a P7.   (* the latest version requires rrdtool version 1.4)

    Finally, here are some other ways to reduce the CPU load when running nmon2web.  (Some you've already tried, but I include here for others).  In no particular order:

    1.  Process less data:  Increase sample time (5 minutes). Some people use a sample time of 1 second (overkill in my opinion). Increasing the sample time from 1s to 5min reduces the data by 300x.

    2.  Erase the rrd update and/or nmon files.  Currently these files are gzip'd after processing.   I'll have to code this if you want to do this. 

    3.  Process the nmon files in parallel (see above)

    4.  Reduce the number of charts (see above)

    5.  Comment out  "&change_control"  (approx line 117) if not used.  It looks for changes in AIX settings, and is displayed at the bottom of the charts.   It takes a little processing overhead, but everything helps.

     

     

  • fbenfredj
    fbenfredj
    40 Posts

    Re: Nmon2web : multiple nmon2web.pl processes

    ‏2014-06-16T16:40:32Z  

    Running in parallel still might help.  The nmon2web process is single threaded and the P7 processor is multithreaded.  In P7, 100% utilization doesn't mean 100% capacity.  There may be room for more throughput.  If your P7 processor is in a shared pool, you can give the partition a lower weight, or even cap it.

    What nmon2web version are you running?  The latest version* (Feb 2014) creates half the charts compared to previous versions, which translates to about a 25% improvement. (Previous versions created daily and trend charts. The current version creates just the daily charts.  The trend charts are created dynamically by the web user).  On my old P5 520, it takes roughly 24 seconds per nmon file, or about 120 files per hour.   This should handle your 95 servers on a P7.   (* the latest version requires rrdtool version 1.4)

    Finally, here are some other ways to reduce the CPU load when running nmon2web.  (Some you've already tried, but I include here for others).  In no particular order:

    1.  Process less data:  Increase sample time (5 minutes). Some people use a sample time of 1 second (overkill in my opinion). Increasing the sample time from 1s to 5min reduces the data by 300x.

    2.  Erase the rrd update and/or nmon files.  Currently these files are gzip'd after processing.   I'll have to code this if you want to do this. 

    3.  Process the nmon files in parallel (see above)

    4.  Reduce the number of charts (see above)

    5.  Comment out  "&change_control"  (approx line 117) if not used.  It looks for changes in AIX settings, and is displayed at the bottom of the charts.   It takes a little processing overhead, but everything helps.

     

     

    Hi Bruce,

    Thanks for reply and ideas.

    But, I discovered the root cause of my problem. In fact, I have 2 lpars which nmon files causes the problems.

    When I deactivted the nmon2web processing of the nmon files of theses 2 lpars, the processing time of the others lpar's files has been divided by 3 or 4.

    Now, for 96 lpars the nmon2web ETL time is 30 mn. Before it was 1H30mn.

     

  • BruceSpencer
    BruceSpencer
    297 Posts

    Re: Nmon2web : multiple nmon2web.pl processes

    ‏2014-06-17T00:05:14Z  
    • fbenfredj
    • ‏2014-06-16T16:40:32Z

    Hi Bruce,

    Thanks for reply and ideas.

    But, I discovered the root cause of my problem. In fact, I have 2 lpars which nmon files causes the problems.

    When I deactivted the nmon2web processing of the nmon files of theses 2 lpars, the processing time of the others lpar's files has been divided by 3 or 4.

    Now, for 96 lpars the nmon2web ETL time is 30 mn. Before it was 1H30mn.

     

    Can you determine the difference with the nmon files that run slow?

    If you'd like, I can test the files on my server.  Just post them.

     

  • fbenfredj
    fbenfredj
    40 Posts

    Re: Nmon2web : multiple nmon2web.pl processes

    ‏2014-06-19T15:12:36Z  

    Can you determine the difference with the nmon files that run slow?

    If you'd like, I can test the files on my server.  Just post them.

     

    On theses 2 lpars topas_nmon coredumps every day.

    We do not know the reason.

    You will find attached the nmon files.

    Attachments

  • BruceSpencer
    BruceSpencer
    297 Posts

    Re: Nmon2web : multiple nmon2web.pl processes

    ‏2014-06-19T21:09:56Z  
    • fbenfredj
    • ‏2014-06-19T15:12:36Z

    On theses 2 lpars topas_nmon coredumps every day.

    We do not know the reason.

    You will find attached the nmon files.

    I ran your files on my server without any problems.  It took around 30 seconds per file on a P5 520.  How did they run on your server?

  • fbenfredj
    fbenfredj
    40 Posts

    Re: Nmon2web : multiple nmon2web.pl processes

    ‏2014-06-20T09:08:50Z  

    I ran your files on my server without any problems.  It took around 30 seconds per file on a P5 520.  How did they run on your server?

    The problem began before May 20th. But, I can not send you all the nmon files because the size of gzip file will be huge.

    Here are processing times when I have the problem :

    nmon2web.pl.20_11h.log:1305: End processing /usr2/datas/nmon2web/adm-unx-tinsn1_140520_1000.nmon
    nmon2web.pl.20_12h.log:1273: End processing /usr2/datas/nmon2web/adm-unx-tinsn1_140520_1100.nmon
    nmon2web.pl.20_13h.log:1245: End processing /usr2/datas/nmon2web/adm-unx-tinsn1_140520_1200.nmon
    nmon2web.pl.20_14h.log:1204: End processing /usr2/datas/nmon2web/adm-unx-tinsn1_140520_1300.nmon
    nmon2web.pl.20_15h.log:1190: End processing /usr2/datas/nmon2web/adm-unx-tinsn1_140520_1400.nmon
    nmon2web.pl.20_16h.log:1173: End processing /usr2/datas/nmon2web/adm-unx-tinsn1_140520_1500.nmon
    nmon2web.pl.20_17h.log:1231: End processing /usr2/datas/nmon2web/adm-unx-tinsn1_140520_1600.nmon
    nmon2web.pl.20_18h.log:1144: End processing /usr2/datas/nmon2web/adm-unx-tinsn1_140520_1700.nmon
    nmon2web.pl.20_19h.log:1251: End processing /usr2/datas/nmon2web/adm-unx-tinsn1_140520_1800.nmon
    nmon2web.pl.20_20h.log:1329: End processing /usr2/datas/nmon2web/adm-unx-tinsn1_140520_1900.nmon
    nmon2web.pl.20_21h.log:1408: End processing /usr2/datas/nmon2web/adm-unx-tinsn1_140520_2000.nmon
    nmon2web.pl.20_22h.log:1419: End processing /usr2/datas/nmon2web/adm-unx-tinsn1_140520_2100.nmon
    nmon2web.pl.20_23h.log:1614: End processing /usr2/datas/nmon2web/adm-unx-tinsn1_140520_2200.nmon
     

     

    nmon2web.pl.20_11h.log:677: End processing /usr2/datas/nmon2web/adm-unx-tina01_140520_1000.nmon
    nmon2web.pl.20_12h.log:681: End processing /usr2/datas/nmon2web/adm-unx-tina01_140520_1100.nmon
    nmon2web.pl.20_13h.log:694: End processing /usr2/datas/nmon2web/adm-unx-tina01_140520_1200.nmon
    nmon2web.pl.20_14h.log:663: End processing /usr2/datas/nmon2web/adm-unx-tina01_140520_1300.nmon
    nmon2web.pl.20_15h.log:650: End processing /usr2/datas/nmon2web/adm-unx-tina01_140520_1400.nmon
    nmon2web.pl.20_16h.log:653: End processing /usr2/datas/nmon2web/adm-unx-tina01_140520_1500.nmon
    nmon2web.pl.20_17h.log:716: End processing /usr2/datas/nmon2web/adm-unx-tina01_140520_1600.nmon
    nmon2web.pl.20_18h.log:544: End processing /usr2/datas/nmon2web/adm-unx-tina01_140520_1700.nmon
    nmon2web.pl.20_19h.log:530: End processing /usr2/datas/nmon2web/adm-unx-tina01_140520_1800.nmon
    nmon2web.pl.20_20h.log:740: End processing /usr2/datas/nmon2web/adm-unx-tina01_140520_1900.nmon
    nmon2web.pl.20_21h.log:634: End processing /usr2/datas/nmon2web/adm-unx-tina01_140520_2000.nmon
    nmon2web.pl.20_22h.log:700: End processing /usr2/datas/nmon2web/adm-unx-tina01_140520_2100.nmon
    nmon2web.pl.20_23h.log:785: End processing /usr2/datas/nmon2web/adm-unx-tina01_140520_2200.nmon
    nmon2web.pl.21.log:35: End processing /usr2/datas/nmon2web/adm-unx-tina01_140309_2000.nmon