Topic
4 replies Latest Post - ‏2013-11-12T05:55:40Z by puvichakravarthy
nagger
nagger
1594 Posts
ACCEPTED ANSWER

Pinned topic What is the "Time Total Dispatch" nmon p=partition panel online

‏2010-10-12T19:28:54Z |
I am posting this question that arrived by email and will copy my answer in the reply.

What does the "Time Total Dispatch" field conveying?
The manual at
ttp://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds4/nmon.htm

Does not help much?

thanks - "a large well known banking group"
Updated on 2010-10-12T19:47:34Z at 2010-10-12T19:47:34Z by nagger
  • nagger
    nagger
    1594 Posts
    ACCEPTED ANSWER

    Re: What is the "Time Total Dispatch" nmon p=partition panel online

    ‏2010-10-12T19:47:34Z  in response to nagger
    Below is an example of an idle LPAR showing the "Time Total Dispatch" (lower right corner) and the CPU text graphs including the UsedCPU (top right corner).

    
    ┌─topas_nmon──1=Top-Basics───────Host=orange4────────Refresh=2 secs───02:27.35─┐ │ CPU-Utilisation-Small-View ───────────EntitledCPU=  0.50 UsedCPU=  0.009─────│ │Logical  CPUs              0----------25-----------50----------75----------100│ │CPU User%  Sys% Wait% Idle%|           |            |           |            |│ │  0   8.6  60.2   0.0  31.1|UUUUssssssssssssssssssssssssssssss               >│ │  1  23.1  50.3   0.0  26.6|UUUUUUUUUUUsssssssssssssssssssssssss            >|│ │  2   0.0  36.2   0.0  63.8|ssssssssssssssssss      >                        |│ │  3   0.0  46.1   0.0  53.9|sssssssssssssssssssssss   >                      |│ │EntitleCapacity/VirtualCPU +-----------|------------|-----------|------------+│ │ EC   0.3   1.1   0.0   0.6|-------------------------------------------------|│ │ VP   0.1   0.3   0.0   0.1|-------------------------------------------------|│ │EC=   1.9%  VP=   0.5%     +--No Cap---|--Folded=1--|-----------100% VP=2 CPU+│ │ Shared-CPU-Logical-Partition ────────────────────────────────────────────────│ │Partition:Number=4 
    "orange4 WPAR"                                             │ │Flags:LPARed DRable SMT-bound Shared UnCapped Migratable.                     │ │Summary: Entitled= 0.50  Used     0.01 (  1.9%)  0.2% of CPUs in System       │ │ - No Shared CPU Pool Util. Authority/Allow Perf. Info. Collect               │ │CPU-Stats---------   Capacity----------------  ID-Memory--------------------  │ │max Phys in sys   4  Cap. Processor Min  0.20  LPAR ID Group:Pool  32772:0    │ │Phys CPU in sys   4  Cap. Processor Max  2.00  Memory(MB) Min:Max  1024:4096  │ │Virtual Online    2  Cap. Increment      0.01  Memory(MB) Online   2048       │ │Logical Online    4  Cap. Unallocated    0.00  Memory Region LMB   64MB min   │ │Physical pool     4  Cap. Entitled       0.50  Time------------------Seconds  │ │SMT threads/CPU   2   -MinReqVirtualCPU  0.10  Time Dispatch Wheel    0.0100  │ │CPU---------Min-Max  Weight------------------  MaxDispatch Latency    0.0150  │ │Virtual       1   4  Weight Variable      128  Time Pool Idle         0.0000  │ │Logical       1   8  Weight Unallocated     0  Time Total Dispatch    0.0094  │ │---------------------------------------------------------------------------   │
    


    The dispatch time is the time the Hypervisor gives this LPAR.
    In the above, you can see its is called the Used CPU else where in nmon - the dispatch time has more decimal digits.
    Its good to see that, at idle the CPU is only being for 9 thousandth of a second on the CPU each second.
    Also note that the topas_nmon shows the CPU graphs about what the CPU is doing while its actually running the LPAR.
    Thus we have CPU 00 running 8% User and 60% system during that 10 thousandth of a second.
    Actually, this LPAR is running 4 also idle WPARs.
    This tiny fraction of a tiny faction of a CPU time is running clock interrupts, and device drivers.
    Oh yes and its running nmon and pushing the characters down the network to me 55 miles away!

    Of course, this 0.009 CPU cycles might not all be in one chunk hence its the Total.
    From the AIX kernel, we get the ever increasing counter of the nano seconds on the processor for the LPAR.
    nmon takes the delta (difference of this captured value and the previous one) and works out the number of seconds on the processor in the last period and divides by the elapsed time.
    It gives up the seconds spent on the CPU per second. a value of 2.0 means we are occupying two CPU seconds per second - i.e. keeping two CPUs busy.

    "dispatch" is from an old UNIX Kernel internals term, where you dispatch a process to go and run on a particular CPU, of course, with PowerVM we are a further level above that and dispatch whole LPARs to run on virtual CPUs.

    Below we see an busier time.

    
    ┌─topas_nmon──r=Resources────────Host=orange4────────Refresh=2 secs───02:26.45─┐ │ CPU-Utilisation-Small-View ───────────EntitledCPU=  0.50 UsedCPU=  1.294─────│ │Logical  CPUs              0----------25-----------50----------75----------100│ │CPU User%  Sys% Wait% Idle%|           |            |           |            |│ │  0  97.4   2.1   0.0   0.5|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUs>│ │  1  96.1   2.4   0.0   1.5|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUs>│ │  2  99.3   0.7   0.0   0.1|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU>│ │  3  99.2   0.7   0.0   0.0|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU>│ │EntitleCapacity/VirtualCPU +-----------|------------|-----------|------------+│ │EC+  98.8   1.0   0.0   0.2|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU|│ │ VP  63.9   0.7   0.0   0.1|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU------------------|│ │EC= 258.8%  VP=  64.7%     +--No Cap---|------------|-----------100% VP=2 CPU+│ │ Shared-CPU-Logical-Partition ────────────────────────────────────────────────│ │Partition:Number=4 
    "orange4 WPAR"                                             │ │Flags:LPARed DRable SMT-bound Shared UnCapped Migratable.                     │ │Summary: Entitled= 0.50  Used     1.29 (258.8%) 32.4% of CPUs in System       │ │ - No Shared CPU Pool Util. Authority/Allow Perf. Info. Collect               │ │CPU-Stats---------   Capacity----------------  ID-Memory--------------------  │ │max Phys in sys   4 Cap. Processor Min  0.20  LPAR ID Group:Pool  32772:0    │ │Phys CPU in sys   4  Cap. Processor Max  2.00  Memory(MB) Min:Max  1024:4096  │ │Virtual Online    2  Cap. Increment      0.01  Memory(MB) Online   2048       │ │Logical Online    4  Cap. Unallocated    0.00  Memory Region LMB   64MB min   │ │Physical pool     4  Cap. Entitled       0.50  Time------------------Seconds  │ │SMT threads/CPU   2   -MinReqVirtualCPU  0.10  Time Dispatch Wheel    0.0100  │ │CPU---------Min-Max  Weight------------------  MaxDispatch Latency    0.0150  │ │Virtual       1   4  Weight Variable      128  Time Pool Idle         0.0000  │ │Logical       1   8  Weight Unallocated     0  Time Total Dispatch    1.2942  │
    


    Now we are keeping the 2 SMT threads on 2 POWER6 CPUS busy all the time and running user programs. We are over the entitlement so the utilisations on the CPUs are reports 100% - of course this 100% does not highlight that we are 2.58 times Entitlement but that's the way it gets reported.

    A comment on yielding the CPU and the LPAR holding on to it:
    An LPAR does not have a choice about yielding (giving) the CPU - its forced off by the Hypervisor. It gets a chance every 100th of a second during the clock interrupt as a minimum but AIX processes also makes tens of thousands of system calls - which generate interrupts, and hardware disks and network also interrupt, all of which the Hypervisor can use to gain control.
    It is better to think of ALL the shared CPU LPARs CPU cycles are given to the pool and it has to fight for them.
    The Entitlement is guaranteed so it wins that fight easily but from then on its down to weights, VP limits and priorities (LPARs that failed to use their entitlement at first but them suddenly get work to do can jump in high priority).

    Hope this helps, Nigel
    • Shak
      Shak
      1 Post
      ACCEPTED ANSWER

      Re: What is the "Time Total Dispatch" nmon p=partition panel online

      ‏2013-11-05T13:36:57Z  in response to nagger

      Hi Nigel,

      Thanks for the details, I have been monitoring couple of POWER 6, 9119-FHA's, On one system nmon reports poolidle as always zero, implies the app filed in lparstat is always zero, but I highly doubt that all the 64 CPUS are being utilised all the time, details below, when I check the shared CPU utilization using nmon (p) the last two lines are incorrectly displayed where as on an another POWER6 p595, nmon (p) shows the shared CPU details peoperly and I can also see app field in lparstat o/p which shows available physical processors, please see the screen shot below

      Problematic system

      ┌─topas_nmon──p=Partitions───────Host=bad_host────Refresh=2 secs───13:24.36────────────────────────────────────────────────────────────────────────────────────────────────┐
      │ Shared-CPU-Logical-Partition ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────│
      │Partition:Number=9 "LP22_bad_host"                                                                                                                                    │
      │Flags:LPARed DRable SMT Shared UnCapped PoolAuth Migratable Not-Donating AMSable.                                                                                            │
      │Summary: Entitled= 6.00  Used     8.76 (146.1%) 13.7% of CPUs in System                                                                                                      │
      │         PoolCPUs=64     Unused   0.00          13.7% of CPUs in Pool                                                                                                        │
      │CPU-Stats---------   Capacity----------------  ID-Memory--------------------                                                                                                 │
      │max Phys in sys  64  Cap. Processor Min  0.10  SPLPAR Group:Pool  32777:0                                                                                                    │
      │Phys CPU in sys  64  Cap. Processor Max 64.00  Memory(GB) Min:Max  1:300                                                                                                     │
      │Virtual Online   25  Cap. Increment      0.01  Memory(GB) Online    175                                                                                                      │
      │Logical Online   50  Cap. Unallocated    0.00  Memory Region LMB   256MB min                                                                                                 │
      │Physical pool    64  Cap. Entitled       6.00  Time------------------Seconds                                                                                                 │
      │SMT threads/CPU   2   -MinReqVirtualCPU  0.10  Time Dispatch Wheel    0.0100                                                                                                 │
      │CPU---------Min-Max  Weight------------------  MaxDispatch Latency    0.0152                                                                                                 │
      │Virtual       1  64  Weight Variable      196  Time Pool Idle         0.0000                                                                                                 │
      │Logical       1 128  Weight Unallocated     0  Time Total Dispatch    8.7640                                                                                                 │
      │---------------------------------------------------------------------------                                                                                                  │
      │Event= 0 ---    ---     SerialNo Old=---    Current=C607F2 When=---                                                                                                          │
      │---------------------------------------------------------------------------                                                                                                  │
      │Shared_Pools   MaxPoolCapacity=64.00 MyPoolMax = 0.00 SharedCPU-Total= 0.00                                                                                                  │
      │SharedCPU=64   EntPoolCapacity=30.10 MyPoolBusy= 0.00 SharedCPU-Busy = 0.00 

       

      On another system where it reports the available physical processor (lparstat app)

       

      ─topas_nmon──d=Disk-Graph───────Host=good_host────Refresh=2 secs───13:25.53────────────────────────────────────────────────────────────────────────────────────────────────┐
      │ Shared-CPU-Logical-Partition ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────│
      │Partition:Number=16 "LP16_good_host"                                                                                                                                   │
      │Flags:LPARed DRable SMT Shared UnCapped PoolAuth Migratable Not-Donating AMSable.                                                                                            │
      │Summary: Entitled= 6.00  Used    10.88 (181.4%) 17.0% of CPUs in System                                                                                                      │
      │         PoolCPUs=64     Unused  37.14          17.0% of CPUs in Pool                                                                                                        │
      │CPU-Stats---------   Capacity----------------  ID-Memory--------------------                                                                                                 │
      │max Phys in sys  64  Cap. Processor Min  0.10  SPLPAR Group:Pool  32784:0                                                                                                    │
      │Phys CPU in sys  64  Cap. Processor Max 64.00  Memory(GB) Min:Max  1:200                                                                                                     │
      │Virtual Online   25  Cap. Increment      0.01  Memory(GB) Online    176                                                                                                      │
      │Logical Online   50  Cap. Unallocated    0.00  Memory Region LMB   256MB min                                                                                                 │
      │Physical pool    64  Cap. Entitled       6.00  Time------------------Seconds                                                                                                 │
      │SMT threads/CPU   2   -MinReqVirtualCPU  0.10  Time Dispatch Wheel    0.0100                                                                                                 │
      │CPU---------Min-Max  Weight------------------  MaxDispatch Latency    0.0152                                                                                                 │
      │Virtual       1  64  Weight Variable      196  Time Pool Idle        37.1401                                                                                                 │
      │Logical       1 128  Weight Unallocated     0  Time Total Dispatch   10.8823                                                                                                 │
      │---------------------------------------------------------------------------                                                                                                  │
      │Event= 0 ---    ---     SerialNo Old=---    Current=CACF62 When=---                                                                                                          │
      │---------------------------------------------------------------------------                                                                                                  │
      │Shared_Pools   MaxPoolCapacity=64.00 MyPoolMax =64.00 SharedCPU-Total=64.00                                                                                                  │
      │SharedCPU=64   EntPoolCapacity=27.20 MyPoolBusy=25.35 SharedCPU-Busy =25.35                                                                                                  │
      │───────────────────────────────────────────────────────────────────

      • nagger
        nagger
        1594 Posts
        ACCEPTED ANSWER

        Re: What is the "Time Total Dispatch" nmon p=partition panel online

        ‏2013-11-11T14:17:29Z  in response to Shak

        If your Pool Idle is always zero then you have forgotten to switch on stats collection. On the HMC and LPAR Properties click on Allow performance information collection

    • puvichakravarthy
      puvichakravarthy
      52 Posts
      ACCEPTED ANSWER

      Re: What is the "Time Total Dispatch" nmon p=partition panel online

      ‏2013-11-12T05:55:40Z  in response to nagger

      'Dispatch time' should be the same like physc in lparstat or mpstat.

      Updated on 2013-11-12T05:56:04Z at 2013-11-12T05:56:04Z by puvichakravarthy