AIXpert Blog is about the AIX operating system from IBM running on POWER based machines called Power Systems and software related to it like IBM Systems Director, PowerVM for virtualisation and PowerSC for security plus performance monitoring and nmon
The SDMC Redbook is available now - actually on the announcement date - so some one was planning ahead! IBM Systems Director Management Console: Introduction and Overview , SG24-7860-00 http://www.redbooks.ibm.com/abstracts/sg247860.html 6.9 MB and 430 pages
So a long download of about 7 hours for the 1.9 GB but a reasonably quick install. Fresh AIX 6.1 TL6 and service pack 6 installed. See my Update below for a problem with this AIX level. Prepared a /database filesystem for the DB2 database. The Pre-Install Utility told me to fix some file system sizes, more paging space and ulimit - good stuff. The Installer worked fine. The CIM Server install seems to take for ages at the start! It then refreshes the Common Agent, installed the default "managed" DB2 database invisibly behind the... [More]
N-Port Id Virtualisation (NPIV) and virtual Fibre Channel adapters - like high speed physical adapters (see an earlier blog) do require memory to operator at full speed. NPIV effectively turns the Virtual I/O Server (VIOS) into a kind of virtual SAN switch where packets are passed through with no changes from the physical adapter to the client Virtual Machine (LPAR). I found it very hard to find a recommendation in the documentation but was given a rule of thumb. As before it should be noted that starving the VIOS and the Hypervisor of... [More]
Here is a recovery procedure for replacing a failed client disk on a Virtual IO server. It assumes the client partitions have mirrored (virtual) disks. The recovery involves both the VIO server and its client partitions. However, it is non disruptive for the client partitions (no downtime), and may be non disruptive on the VIO server (depending on disk configuration). This procedure does not apply to Raid5 or SAN disk failures. The test system had two VIO servers and an AIX client. The AIX client had two virtual disks (one disk from each... [More]
On Power6 the largest machine was the Power 595 with 64 Physical CPUs (cores) across eight CPU books in the machine - each CPU book having 4 Power6 chips and so 8 CPUs (Power6 is a dual CPU chip design). However, with Power7 that has stepped up to 256 CPUs across the same eight CPU books with four chips but with 8 CPUs each so that is 32 CPUs per book. I might be stating the obvious but memory access to memory directly attached to the Power chip on which your process is running is slightly faster than memory access via a Power chip near by... [More]
Federico Vagnini has written pGraph, an excellent "freeware" tool for viewing AIX and Linux. performance graphs. The viewer works with multiple input formats, including * nmon (either on AIX and on Linux) * vmstat -t on AIX * topasout (AIX's continuous performance data collector) * topasout on AIX * iostat -alDT on AIX * HMC's data collection extracted using lslparutil command * Workload Partition data collected on AIX6 (currently tested only on Open Beta) You can download pGraph from... [More]
I just noted two great new things to help those running Java on AIX and it seems hard to avoid Java these days! I was just monitoring a customer machine with 1500 WebSphere Application Servers (WAS) each with 100 to 150 threads so roughly 200,000 threads!!! More impressing was that just 20 Power7 physical CPUs was supporting this workload. So here are two things that can help: 1. Java Performance Advisor (JPA) Available for download to anyone on IBM DeveloperWorks website. This is a powerful new tool to help people identify opportunities to... [More]
While I was installing a re-purposed POWER6 machine over the weekend, got round to installing my Dual Virtual I/O Servers, from DVD and then upgrading from there and I noticed a new service pack 2 for the latest VIOS version. It must have sneaked out without me noticing and there is a Interim fix too. The VIOS 18.104.22.168-FP24 SP2 Readme highlights lots of fixes including some for the Shared Storage Pool feature. A feature that I have been using and talking about (see this blog for more information). This got me thinking! I talk to lots... [More]
Ganglia is a open source performance and configuration tool that collects data from a tiny daemon on each OS and then creates a Web server based graphical tools to draw performance data and show the configuration. With a few add-ons, you can dynamically decide the historical data you want graphed and with POWER add-ons you get the PowerVM stats for shared processors, Entitlement, physical CPU use etc. Under the covers it is using the amazing rrdtool so you can extract the data for other uses and it automatically space manages the data. By... [More]
Warning: touchy feely comment coming up! I have always found it odd that IBM markets the Hardware = POWER7 a lot and very little marketing on the Operating System = AIX7. Personally,
I have more good feelings and positive vibes for AIX operating system
because I log in to every day and use it for hours than I do for the
POWER hardware running it. I very rarely go to the machine room and hug the machine! Does anyone else feel this way or am I sadly alone? Put
it another way would you spend two Dollar Euro Pounds (all increasingly
You may have seen comments or even presentations with a few slides on VIOS Next Generation or VIOS NextGen in the past year. I did at the Power Technical Universities and was very interested. Well, the first phase has been release but you may well have missed it as it appears as VIOS 22.214.171.124 FixPack 24 Service Pack 01 - not a catchy title and it slipped out below my radar! It is a regular service pack for the current VIOS and include some other features and fixes too. The first thing you have to note is that only a few features are available... [More]
Configuring MPIO for the Virtual Client , describes how to set up Multi-Path I/O boot disks for AIX clients of the virtual I/O server. There some considerations for booting off the SAN. The "pro's" include: IBM supported configuration Flexibility The "con's" include: If the SAN fails Can not capture an AIX dump file (Consider putting dump space on an internal disk) Lose the ability to diagnose the failure from the server side. Updating MPIO on the AIX client may be disruptive (depends on MPIO vendor/version) "Momentary SAN stalls" can cause... [More]
I just realised that I got help from the AIX developers to explain the output of the mpstat -d command for my POWER7 and Affinity Technical University sessions in Miami and Copenhagen that I never passed on to every one. In an earlier blog we examined some mpstat columns but skipped others. The nearly undocumented mpstat stats are: S0rd, S1rd, S2rd, S3rd, S4rdand S5rd. In the AIX manuals it states: S0rd = The percentage of thread re-dispatches
within the scheduling affinity domain 0.
And likewise for the other numbers - but what does... [More]
I am preparing sessions for a POWER/AIX Technical Conference in Norway - unfortunately the only place they could book for the event was a Ski Resort up in the mountains and there is still good snow. It is a tough gig but someone has to do it :-) As part of that conference, I am updating them on Systems Director and demonstrating it. I had a slide with these hints and tips. A slide with zero percent marks for style (far to many words and just a long list) and I thought ... I should share these with everyone, so here they are. Most of these are... [More]
Starting with AIX 5.3, you can get the process tree showing all the children of a particular process in a tree format. This is a part of the Linux affinity push: Example: $ ps -T 119016 PID TTY TIME CMD 119016 - 0:00 sshd 1015808 - 0:00 \--sshd 418006 - 0:00 \--ksh 913436 pts/0 0:00 \--ksh 639072 pts/0 0:00 \--ps
I have just upgraded my EMEA Power Systems Advanced Technical Support internal Wiki Apache Web-server to the very latest AIX level, which is AIX 6.1 Technology Level 6 Service Pack 5 or AIX6.1TL6sp5 for short! Then I noticed that the nmon at this AIX level has been updated - I worked the code for the internal prototype of this new feature some time ago and it has arrived in the official topas/nmon code at this release - C O O L . The latest release AIX 7.1 service pack has this new options too. The problem: As the numbers of CPU goes up with... [More]
The title should read "Local, Near & Far ..." - I will not correct it or links might fail. In this entry we carry on from part 5 but we are going to look at setting the virtual processor number for the virtual machine . There is a side effect that is not obvious and after 6 years of using them, it never occurred to me so perhaps it is news to others too. The problem of virtual processors is that they are ephemeral - i.e. they don't actually exist and costs nothing. So I find most systems administrators feel they can be generous... [More]
The official rPerf rating (relative performance rating of Power based machines) are available in the pSeries, System p and now Power Systems "Facts and Features" documents that are on the IBM website. It is best to use Google for them (searching for "pseries Facts and Features" works for me). You may want to know the rPerf rating for your LPAR and an LPAR that is not one of the officially rated numbers of CPUs/Cores. There may be ratings for 8 and 16 CPUs but your LPAR has 13 CPUs! Well, you can run the rperf script on the... [More]
Your first reaction might be "Why would you do that, Nigel?" and "Isn't this a AIX blog: Why are we talking about Linux?"
Well AIX is great and all that but when it comes to Open Source it is not Linux. Like it or not most Open Source stuff is developed on Linux and there are readily available binary compiled for Linux and 100's of Open Source packages actually come with the the distribution on their DVD media and arch repositories on the Internet for rapid and straight forward install and updates.
*** STOP PRESS *** - See the update in red at the bottom.
I was asked this lots of times at the recent POWER Technical Universities in Orlando and Athens.
Compared to a low numbers of local disks
The LUNs on a SAN disk subsystem with caching are very fast - even a simple test like AIX install times have halved. In addition, you don't have to fiddle about with stripping AIX Logical Volumes across disks or retro fit this to rootvg Volume Group filesystems after the install.
Compared to a... [More]
I am using the VIOS Shared Storage Pools more these days and it reduces my system admin time and I really like the Thin Provisioning feature as I have limited SAN disks. Any way, I dusted off an machine that I did my initial investigation on and decided to rebuilt the cluster (of one VIOS) now that I know what I am doing :-) The command : cluster -create -clustername galaxy -repopvs hdisk2 -spname atlantic -sppvs hdisk3 hdisk4 hdisk5 -hostname diamondvios1 fails with "PV IS IN USE hdisk4". PV meaning Physical Volume. Yes,... [More]
Installing Open Source on AIX systems can be a little .... um .... frustrating and I have just been round that loop with installing Ganglia the excellent Performance Monitoring tool to demonstrate it and actually use it too on my own servers. The hard bit is that for every Open Source program, that you want to install, there is 10 to 20 prerequisite packages that you need to install first and you go round this the loop four times as the prerequisites have prerequisites .... This has got known as the "RPM dependencies form... [More]
LPAR2RRD Briefly, Based on the same HMC data that Systems Director gathers is LPAR2RRD but it saves the data in the famous and excellent rrdtool database and graphing tools. This is a supported by the developer which is Pavel Hampl, IBM Czech republic, as a personal project tool and does a very good job. For more details and a sample website to look round see - AIX Wiki page for LPAR2RRD I note this is a sourceforge.net project now - See the above for a link to the project and code Cut from that AIX wiki page for the project here (all... [More]
I thought I should summarise the long eleven part Local, Near & Far POWER7 Affinity series. 1) Placement: Find out the layout of your boxes CPU and RAM and if the RAM is evenly distributed across available DIMMs Find out the placement of your Virtual Machines (LPARs) with lssrad -av - or - topas -M 2) SMT4 : Expect POWER7 SMT4 CPU use to “look” different POWER5 & 6 have two equal threads POWER7 shuts down threads 3 & 4 and even thread 2 - when there is not enough processes running. 3) Entitlement : Only set minimum Entitlements,... [More]
This week I spent 4 hours with a customer covering many advanced WPAR topics and took way a bunch of questions that I had to check the answers and ask the WPAR developers themselves to be sure I had the right answers. If the questions were not clear to my customers and I did know initially know the answers then there may be others with similar issues so I thought I would share the answers with everyone. 1) Workload Partitions - What is the road map? Well, I am not going to make announcements on a blog for sure. If you are new to WPAR, I can... [More]
I have been looking at some nmon data from an IBMer looking into a customers machine. The virtual machine (LPAR) is running the Oracle RDBMS with 75 dedicated CPUs - on a POWER7 Power 795 at 4 GHz.
The Excel spreadsheet is quite scary at 144 MB thank goodness my Thinkpad has a SSD but not the record by a long way. Too many data save points can ready add u and so too can lots of processes if you are saving process data. Also with 75 CPU and SMT=4 that is 300 Tabs for Logical CPU, 300 for PCPU and 300 for SCPU and that... [More]
IBM announced lots of Power Systems software yesterday but the official announcements are pretty thin on the details. I have many Twitter tweets asking "What the heck does that all really mean?" and the official statements are marketing messages and not technical information like:
How do you get that working?
What are the pre-reqs?
What are the actual commands?
What does the Graphical User Interface look like?
How does that help me as a System Admin person?
Finally, I found the time to try this new exciting feature and it was really simple to create and operate. I captured movie footage of my first attempt and it all worked (once I removed my spelling mistakes in commands). It only took two commands. So the technology has advanced concepts but it really easy to use, COOL! Some of the commands come with long option names like "-custername". A quick misquote from Star-Trek, never goes wrong: "It is UNIX Jim, but not as we know it!" One command to create my cluster,... [More]
AIX or VIOS Start-up Memory Problems I have been working with a couple of customers that have a newly setup AIX Logical Partition (LPAR) that fails to start due to not having enough memory. The common factor is that the LPAR for AIX (or VIOS) includes a good number of what I call "high speed" adapters like 10 Gbps Ethernet, or 4 Gbps or 8 Gbps Fibre Channel adapters. They tend to have multiple adapters of each type for redundancy and throughput. For example, trying to start the LPAR with 1 or 2 GB of memory but it turned out they need... [More]
The below URL takes you to the top of the Systems Director 6.3 online documentation http://publib.boulder.ibm.com/infocenter/director/pubs/topic/com.ibm.director.main.helps.doc/fqm0_main.html As a rule I use Firefox 3.6.24 all the time for my browsing as it works well for Systems Director and Hardware Management Console (HMC). Later Firefox versions have "variable" results which re frustrating and that just slow me down. The Firefox developer when mad and got to version 8 suddenly over what appeared to be a few weeks but have realised... [More]