WHAT A PERFORMANCE!

Many performance commands such as vmstat and iostat can report accumulated statistics since the last reboot. So it's important to know when that reboot happened. The  uptime command will give you that information.

Here is the result of running uptime on an LPAR I'm using via the IBM Virtual Loaner Program:

lpar05p682e_pub[/] > uptime

  04:45AM   up 2 days,   5:08,  1 user,  load average: 1.00, 0.75, 0.66

The LPAR has been up for 2 days (and 5 hours, 8 mins) and there is just one user logged in. (I like my personal space.)

The load average from the uptime command shows the total number of runnable processes in the last 1 minute, 5 minutes and 15 minutes. So uptime gives you a quick glance of the system, without running more detailed commands such as topas or nmon. (Those last two commands in AIX 6.1 are actually the same executable, and you can toggle between topas and nmon screens using the tilde ~  key).

Watch out for the RAM page!

Although uptime is helpful, it really is only a quick snapshot. When poor performance is being reported, you really need to drill down some more. One of the commands you can use is vmstat -s which, as the documentation explains, shows you "an absolute count of paging events since system initialization". (A "page" is a page of virtual memory, not just paging space.)

A clock doing overtime

vmstat -s can really highlight some serious issues. Take a look at this Oracle LPAR, for example:

The uptime command shows it's been up for just 10 days:

 09:14PM   up 10 days,  11:42,  3 users,  load average: 1.86, 1.61, 1.58

The vmstat -s command shows (among other things) that in that time there were

2259 revolutions of the clock hand

This is

less than optimal

In fact, it's what you might even call bad.

A revolution of the clock hand means that in the 10 days of uptime, the system had to scan the whole of the system's memory for something free 2259 times. That's about 225 times per day, or between 9 and 10 times per hour.

---

Earl Jew's Performance Tuning

That's not the whole story about memory, of course. If you want to know more about vmstat and other key performance tuning commands, you could refer to the AIX 5L Performance Tuning Redbook. Still relevant, even though AIX 7 is about to be released.

I also  recommend you watch Earl Jew's excellent presentation on AIX Tuning. It's amazing how much Earl packs into 90 minutes. He outlines some really important performance tuning principles and then explains - with examples - some of the bottlenecks you're likely to see.

You can download a replay of that presentation from the AIX Virtual User Group - USA (originally known as AIX Virtual User Group - Central Region - USA)..

Go to the session marked:

July 29, 2010 - AIX Tuning with Earl Jew

There is also a script which you can run and a very detailed explanation of all of those puzzling numbers from vmstat, iostat, lparstat and a few other key performance commands. Earl does amazing comparisons between values and explains just how concerned you should be when you see a statistic like

70593409 paging space page outs

(Hint: have an ambulance on standby).
 

Tags:  virtual_loaner_program topas vlp aix lpar busy performance vmstat tuning statistics uptime free_pages virtual_memory iostat earl_jew paging oracle memory nmon