Over the 2016 Xmas break, I have been busy improving nmon and this is a summary with pictures:

1 New Performance Statistics

GPU

These are the stats on-screen

1. GPU MHz
2. CPU and Memory Utilisation
3. Temperature in C
4. Electrical power in Watts

This is also saved to the nmon output file.

MHz

Note this machine is very busy and most CPU core threads are at 3491 MHz (roughly 3.5 GHz) but threads 145 to 152 are idling (this makes up one SMT=8 CPU core) at 2061 MHz (2 GHz).

Below there are just the CPU cores and now there are more cores at the slower MHz 2061.

Then a final "M" and we get the same CPU core numbers but a graph format - so it is easy to spot fast and slow CPU cores.
 

Above obvious only 2 CPU cores are at the lower MHz

OK that does look a little confusing as its a stacked chart of the MHz for all CPU core threads. The main problem is that it reported for all 80 threads and that it only ever has two values 2061 MHz and 3491 MHz. As this is the way it is reported but Linux in the /proc/cpuinfo file.  An unstacked graph would not work as all the lines would be on top of each other. I may decide to only save the CPU core values as currently, all the threads on a particular core have the frequency. In this case, the 80 threads would drop to just the 10 POWER8 cores which is more manageable. The problem then is that on Intel / AMD CPUs the hyperthreading is very difficult to calculate and many of the statistics are reported clearly wrong.  

Linux Utilisation categories

Classic UNIX has three categories plus Idle for CPU Utilisation:

1. User = running programs, applications, or commands
2. System = the UNIX Kernel running system calls or Interrupts
3. Wait for I/O = This is the same as Idle but there is disk I/O outstanding (it shows a disk bottleneck but the CPUs are actually Idle waiting for data)
4. Idle = the CPU has nothing to do

Linux over the years has added some more and currently has ten categories for CPU Utilisation as follows
 

1. User = running programs, applications, or  commands (like UNIX above)
2. User Nice = As User but the processes have had their priority dropped using the nice command or system call
3. System = the Linux Kernel running system calls but not Interrupts (see below Irq and Softirq)
4. Idle  = the CPU has nothing to do
5. IO Wait = This is the same as Idle but there is disk I/O outstanding (it shows a disk bottleneck but the CPUs are actually Idle waiting for data)
6. Irq = Kernel system time Handing Interrupt. I have seen embedded CPU Linux stats where the CPU is very largely interrupt driven so this is useful to highlight that.
7. Softirq = Software Interrupt - these are like Interrupts but not driven from hardware event but by special CPU instruction to get CPU interrupted for the Kernel to take actions.
8. Steal = Only applies to a Virtual Machine and highlights that the VM has runnable processes that can't run because the CPU is currently running a different VM. This VM effectively thinks someone else stole the CPU cycles. Over-committed CPUs in a highly virtualised environment can get a lot of this. These Steal stats highlight the issue.
9. Guest - Only applies to a VM Hosting operating system like KVM or PowerKVM hypervisor environments - this CPU time is used running one or more of the hosted guest virtual machines.  To the hosting OS, this is like User time - but within the VM it could be User, System or the other categories.
10. Guest Nice = The same as the about Guest category above but the hosts are low priority.

1. a 100% tells you a CPU core thread is busy - if we user 100 / 80 = 1.2% it looks like noise
2. having a number like 3400% informs you there is roughly 34 CPU cores worth of CPU time in used. If reported as 3400 /80 = 42% then you have to do the backward maths to get to 34 CPU core threads.

The Initial nmonchart graphs of this look like this below:

I hope to capture other machines with busy steal and guest numbers.

 2 On-screen Face Lift

The Help panel looks like this in green:

Memory

Kernel stats:

There are loads more - see the bottom link to all the current screen shots.

NEW Wide CPU View

1. "lscpu" output is captured to the nmon file and also displays on screen
2. internal code change to organise the command line option and key to change screen content
3. nmon -h output rewritten - still rather cryptic but much better and assumes less.
4. Kernel stats on screen new format
5. Still working on formatting the boot time from the second since the epoch to the date time.  The Uptime value overflows far too quickly now.
6. Less time drift by allowing for nmon compute time for long-running data capture to file session.
7. New Top Process functions to handle ordered by I/O of the processes (type: 4) (you have to be root for this to work due to a /proc quirk)
8. Reduced screen artefacts with better cursors pad use.
9. Automatic generation of User Defined Disk Groups using the lsblk command) to remove reporting of duplicated disk stats (/proc has both disk and disk partition stats mixed together). Use the option "-g auto" to enable for on-screen and capture to file.  For the on-screen panel type: g
10. Capture to file: If you use -g auto and the -D you also get more disk stats: disk wait time & disk service times, merges, in-flight I/O count & backlog
11. JFS stats are more accurate and they know about the hidden block saved for root user - if your disks are large this can waste a lot of space!

More information:

1. nmon for Linus download http://nmon.sourceforge.net/pmwiki.php?n=Site.Download
2. nmonchart download http://nmon.sourceforge.net/pmwiki.php?n=Site.Nmonchart