AIXpert Blog is about the AIX operating system from IBM running on POWER based machines called Power Systems and software related to it like IBM Systems Director, PowerVM for virtualisation and PowerSC for security plus performance monitoring and nmon
Update: The developers and the manuals call this Load Sharing but most people think it is called Load Balancing. Perhaps, balancing gives the wrong impression of fine grain packet by packet balancing where we actually have higher level, cruder splitting of the work with Sharing. Below I use the word Balancing but mean Sharing. I have got a few questions recently on how to set this up as there are announcement with near zero information on setup, the configuration needed and a worked example. So here goes. For a long time now we have had SEA... [More]
Before we look further in to memory affinity we need to recap on the scheduling of processes and process threads of a multi-threaded process to simultaneous multi-threading (SMT) processors like POWER7. POWER5 and POWER6 had two modes of SMT off (one thread) and on (two threads) - with SMT=on two processes run at the same time (in the same clock cycle) on the CPU-core but using the different logical units inside of the CPU-core (units like the integer maths (there is more than one), floating point maths (there is more than one), compare and... [More]
I am preparing sessions for a POWER/AIX Technical Conference in Norway - unfortunately the only place they could book for the event was a Ski Resort up in the mountains and there is still good snow. It is a tough gig but someone has to do it :-) As part of that conference, I am updating them on Systems Director and demonstrating it. I had a slide with these hints and tips. A slide with zero percent marks for style (far to many words and just a long list) and I thought ... I should share these with everyone, so here they are. Most of these are... [More]
This mysterious AIX CPU Folding area is often misunderstood, so below is what I know from osmosis from talking to various guru level developers over the last 10 years. Shared Processor virtual machines (LPARs for the old fashioned) have a setting called Virtual Processors (or VP for short). This is the number of physical CPUs that the virtual machines can spread out across - in fact, I prefer to call it the "spreading factor" as it is much more obvious what it means. This can be the upper threshold for the number of CPUs that can be... [More]
The title should read "Local, Near & Far ..." - I will not correct it or links might fail. With a shared processor virtual machine (I am calling this "VM" but was called LPAR!) there are various suggestions of setting Entitlement ("Desired processing units" on the LPAR profile on the HMC, I am calling this "E") and Virtual Processor numbers (I am calling this "VP"). For Capped, the Entitlement is the maximum guaranteed CPU time that you can't go over and you round up the Entitlement to the... [More]
I have had a couple of Power systems administrators make assumptions about the virtual Ethernet speed improvements when they install a 10 Gb IVE/HEA in a VIOS which are simply not true. I guess that if three teams have made this mistake then others are about too. So I intend here to put the record straight. The expectation is that (deliberately fully spelt out long-hand to make it very clear): When they upgrade the Integrated Virtual Ethernet (also called a Host Ethernet Adapter) from 1 Gigabit per second to 10 Gigabit per second that the... [More]
So you know about Power7 Local, Near and Far memory for your actual machine but what is your Virtual Machine (LPAR) actually using? There are three key commands to show you (lssrad, mpstat and topas) and we will look at some example output. First, we need to define an SRAD or a Scheduler Resource Affinity Domain . If you have used Resource Sets with AIX WLM or WPAR then you have a good idea what these are like. An SRAD is a group of resources but in our case CPU/cores and the associated memory that is directly attached to it. As an example... [More]
I was asked how to get the inlet and exhaust temperature from a POWER8 machine and the amount of electricity it is using. This was once done with Systems Director and the Active Energy Manager (AEM) but I a told this does not work with POWER8 or if it does its just luck!
At first I thought it impossible but I asked friends in development and they pointed me to two documents that point you in the right direction
Here they are
IBM EnergyScale for POWER8 Systems
Assuming you have dozens to hundreds of AIX VM's now running on your PowerVC controlled machines at some point you will have to upgrade the AIX version to keep current and ensure security.
PowerVC does not have OS upgrade functions but it does have features that can help make upgrades safe and simpler. The upgrading of AIX is already handled by tried and tested methods and to be blunt I am not convinced I want PowerVC to do it as I want to visually check the install worked and immediately address any issues to ensure... [More]
Please provide comments on this blog entry - I want to know where I have it wrong?
With IBM's massive investment in Linux on Power my brain hurts trying to express the advantages. This is because the technology is excellent but the bits you want depend on where you are coming from. So I have tried to break out potential users in to various types.
For the record I am an Ab Linux on POWER User type!
A - Current AIX and POWER user
Already understands the POWER infrastructure, AIX and/or IBM i and are adding... [More]
I got asked these questions recently and had to go look the subject up ... again! I seem to have forgotten some of the details and then I thought I would use some new features of AIX for the second part. In the distant past there was various way to stop core files being dumped in to the current working directory of the program that failed. In AIX 5.3, AIX 6 and 7, the "chcore" command does all the hard work for us by letting us
Choose a specific directory for core files - which is best in it own filesystem so it... [More]
Just completed the move of the POWER and AIX Hands-on Movies to YouTube videos - after pruning we have 97 of them. This means you can watch them on your mobile Smartphone and Tablets in addition to your workstation/personal computer. You can find them all here:
YouTube look for my "Nigel Griffiths" Channel http://www.youtube.com/user/nigelargriffiths This should mean they are simpler to find, search for and view. During the transfer I pruned out a few of the older ones particular Systems 6.2 as we now have a set of Systems... [More]
I thought I should summarise the long eleven part Local, Near & Far POWER7 Affinity series. 1) Placement: Find out the layout of your boxes CPU and RAM and if the RAM is evenly distributed across available DIMMs Find out the placement of your Virtual Machines (LPARs) with lssrad -av - or - topas -M 2) SMT4 : Expect POWER7 SMT4 CPU use to “look” different POWER5 & 6 have two equal threads POWER7 shuts down threads 3 & 4 and even thread 2 - when there is not enough processes running. 3) Entitlement : Only set minimum Entitlements,... [More]
I just realised that I got help from the AIX developers to explain the output of the mpstat -d command for my POWER7 and Affinity Technical University sessions in Miami and Copenhagen that I never passed on to every one. In an earlier blog we examined some mpstat columns but skipped others. The nearly undocumented mpstat stats are: S0rd, S1rd, S2rd, S3rd, S4rdand S5rd. In the AIX manuals it states: S0rd = The percentage of thread re-dispatches
within the scheduling affinity domain 0.
And likewise for the other numbers - but what does... [More]
I have been wondering why the lower end POWER7 machines have local and far memory and not local and near. Perhaps you wondered too! Well at the Miami Power Technical University, I got to talk to Dr Joel Tendler (IBMer) and a POWER7 processor guru and put the question to him. He covered this sort of architectural topic in his presentation at the event and I learnt a lot in this area by listening to the "master". Below is some background and the explanation too. The POWER7 chips has two memory controllers for maximum performance but... [More]
N-Port Id Virtualisation (NPIV) and virtual Fibre Channel adapters - like high speed physical adapters (see an earlier blog) do require memory to operator at full speed. NPIV effectively turns the Virtual I/O Server (VIOS) into a kind of virtual SAN switch where packets are passed through with no changes from the physical adapter to the client Virtual Machine (LPAR). I found it very hard to find a recommendation in the documentation but was given a rule of thumb. As before it should be noted that starving the VIOS and the Hypervisor of... [More]
I was just asked this question by customer and it got me thinking, Items I would include: WPAR takes seconds to create and LPARs minutes LPARs need setting VIOS LV or LUNs but WPARs add simpler NFS mount points, or can just use Global AIX diskspace options LPAR needs 512 to 1GB to boot AIX and a WPAR takes just ~60 MB (yes sixty megabytes) You can share application code say 1 GB in each and every LPAR (40 LPAR = 40 GB) or just one shared read-only copy for all WPARs 40 WPAR = 1 GB).This saves man-power in maintenance, disk space AND memory (if... [More]
I need to make it clear that these pictures are from an Early Ship Program machine which might vary a little from the final Generally Available (GA) models in November 2014. For example, missing labels or stickers plus colours of handles and parts etc. The engineering will be much the same.
Please see my 8 minute video of the machine on YouTube here:
POWER8 Enterprise - Power E870 First Look http://youtu.be/jOzPTopt7HE
These pictures were take by Gareth Coates and the... [More]
If while installing SUSE SLES 11.3 on your POWER8 machine from the regular DVD media, you get this on the console after 5 seconds after booting:
Welcome to SuSE Linux Enterprise 11!
Type "install" to start the YaST installer on this CD/DVD
Type "slp" to start the YaST install via network
Type "rescue" to start the rescue system on this CD/DVD
Welcome to yaboot version r22.8-r1190.SuSE... [More]
Odd. I thought I have put this on this blog before but I can find it now so just in case ... here it is or here it is again.
The SSP4 can have mirrored failure group FC disks to handle adapter, FC cable, FC switch, entire FC disk sub-system, site or VIOS failure.
But there is still a single SSP Repository disk. Isn't this a single point of failure?
The answer is " No because you can quickly rebuild the contents of the Repository Disk after it has failed ."
You are not meant to know this... [More]
I was told that Fedora's latest build for the Power ppc64 processor would support SMT=8 , if you "err ... um" have machine with a processor that support that SMT=8 !!! This is not an announcement.
So I gave it ago - here is my report:
And I can't help thinking PowerVC would remove all this work, once the new version arrives that support my ... umm err! new machine!!
I can tell you know that this was a POWER8 S824... [More]
Inspired by this TED Talks Video on Leadership
TED Talk by Simon Sinek
Topic: How Great Leaders Inspire Action
18 minutes long
It tells us to start with Why? To win hearts and minds.
And not explain What? or How? until the Why? is very clear.
I thought I would make an attempt:
We believe server sprawl means a future in which we drown in small computers taking 80% of the... [More]
IBM announced lots of Power Systems software yesterday but the official announcements are pretty thin on the details. I have many Twitter tweets asking "What the heck does that all really mean?" and the official statements are marketing messages and not technical information like:
How do you get that working?
What are the pre-reqs?
What are the actual commands?
What does the Graphical User Interface look like?
How does that help me as a System Admin person?
A very nice chart from Richard Milton of IBM on the minimum levels for AIX and VIOS of Power7 and Power7+ hardware.
Download as a PDF: AIX_Levels_on_POWER7_Hardware_2013_06_20.pdf
We will try to update this as new machines arrive.
Having just done this and getting questions on it from customers, I thought I should share a few notes:
1) The process is nicely documented and you need to follow the instructions. I recommend using the Release Notes as a check list as missing out a stop causes the next bit for complain or fail. Fortunately, the recovery is just doing the bit I missed.
2) Read the Release notes - even better print them out and cross them off as you go:
http://www.ibm.com/support/docview.wss?uid=hpc1vios117f5701 - upgrade... [More]
POWER Relative Performance (rPerf) is often used as a way to approximate the expected difference in performance between two Power Systems servers. Although rPerf is a useful tool, it is important to understand the limitations of using rPerf to provide an estimate the performance of your specific workloads in your particular environment with a new server, First, rPerf numbers, like any published benchmark, represent the best case result when the application, configuration and system resources are all optimized -- factors that are likely not... [More]
Common question: I see XX% of Used Memory but I don't know who is using it and how?
A simple question but there is no simple answer as memory use if complex.
nmon then hit "m" will quickly show you a few big uses of memory as below:
The shared memory used by lots of applications like DB2 and Oracle - check the SEGSZ for the size.
The Owner column usually tells you what it is used for like the oracle user for the SGA or db2inst1 for the DB2 buffer... [More]
It has bee a long few weeks traveling to Dublin, Las Vegas, Slovenia, Germany and Sweden to give lots of presentations. Six Weeks and five countries and the expenses are piling up on my desk. I was presenting a core set of topics like POWER7+ announcement summary, VIOS Shared Storage Pools (SSP3), Workload Partitions (WPAR) and Versioned WPARs to run AIX 5.2 and AIX 5.3, quick introduction to PowerSC (security tools), a Techie "what you need to know about" Systems Director for AIX, and Quick Dips in to Active Memory Deduplication... [More]
So I got asked, just as an example configuration which forces lots of workload per CPU: Given a 16 CPU POWER machine and a need to run 100 workloads, would I recommend 100 LPARs or 100 WPARs? In case you are not familiar with POWER technology: LPAR = Logical Partition used the PowerVM to split CPU, memory and I/O across multiple virtual machines (also called virtual servers). The I/O is handles by a special purpose LPAR called the Virtual I/O Server, which provides virtual network, virtual disks, virtual optical and virtual tape. Each LPAR has... [More]