AIXpert Blog is about the AIX operating system from IBM running on POWER based machines called Power Systems and software related to it like IBM Systems Director, PowerVM for virtualisation and PowerSC for security plus performance monitoring and nmon
Yes, it does happen. When you want to use all the main unit (CEC) adapters slots for high speed adapters like Fibre Channel and the ROCE network and want to avoid adding the extra POWER8 I/O Drawers for just 1 or 2 adapters - but did not realises the ROCE adapters can't do a NIM install.
I think it is actually due to the basic boot code "PROM" not having the "smarts" to run the advanced adapters in a basic limp along mode to run the network for the bootp and tftpboot download before you... [More]
I was recently sent pictures and got questions about the handle colours on the Power System E870 / E880 CEC and System Control Unit.
On my early ship beta test machine, the adapter carriers have terracotta (red-ish) handles and on GA customer shipped machine blue .
Terracotta handles means customer hot-plug capable - but don't just grab and yank. You must to go through the HMC maintenance procedure at the very least for the adapter slot to be powered off to make it safe to be removed.
Blue handles means... [More]
I was asked how to get the inlet and exhaust temperature from a POWER8 machine and the amount of electricity it is using. This was once done with Systems Director and the Active Energy Manager (AEM) but I a told this does not work with POWER8 or if it does its just luck!
At first I thought it impossible but I asked friends in development and they pointed me to two documents that point you in the right direction
Here they are
IBM EnergyScale for POWER8 Systems
I had this combination recently and it drove me nuts.
Plug the Fibre Channel cable from the Storwize V7000 in to the Brocade FC Switch (IBM 2498-B40 = Brocade 5100) and the ports would not go green but flash orange on and off and end up in Port Status=Disabled and Healthy=Faulty on the switch and the V7000 in waiting for a cable state. Both the the V7000 and FC Switch are the latest firmware.
I assumed faulty GBIC at one or other end and/or a faulty FC cables at first but after trying many combinations and my two new... [More]
This Blog is brought to you by Clive Benjamin (Principal Engineer) who worked out how to do it.
If your HMC has a quality high resolution local screen or KVM screen then the Restricted Shell terminal window font size is far too small to read and the font used very narrow lines - in our humble opinion . This blog shows you how to fix this and "no", the obvious window Font Size option does not work.
1) Below: On the HMC start the Restricted Shell window
2) Below: With the cursor on the terminal window used... [More]
This is a story board about the mystical POWER8 Lift Tool and decommissioning our Power Systems Enterprise E870. The EMEA Advanced Technology Support machine was a temporary loan from the Power Development team in the USA as part of the Early Ship Program for beta testing, skills development, usability testing and feedback plus social media like the YouTube videos, Tweets and this blog. We have certain enjoyed learning about and using the machine although I must say it was hard work to use up all the 4 TB's of memory!!
IBM Systems Director and PowerVC make use of a private API to use Shared Storage Pool (SSP) clones of LU (like a LUN allocated from the pool). But there is no command to let you do that. LU Clones are quick and good for duplicating a master copy for small virtual machines to boot the same OS level and it means greatly reduced disk use as blocks are shared until written too.
As there is not VIOS command to clone there is no command to let you see what is cloned and what is not.
I tripped over this when trying to clean up my SSP... [More]
If you go looking then you can find IBM websites covering AIX that tell you about AIX 7 Express Edition being limited to LPARs (virtual machines) with a maximum of 4 CPUs and a maximum 32 GB or memory. The idea being a reduces price for small machines hosting small virtual machines.
You can also find the Announcement of the removal of these two restrictions for certain POWER7 models like Power 710, 720 and 730.
UPDATED: 30 JANUARY 2015
The IBM Official Technote can be found here http://www-01.ibm.com/support/docview.wss?uid=nas8N1020533
It clarifies the mean of the ASMI settings i.e. it is the number of adapter slots AND the specific slots AND a further bug only some settings work until the next firmware release.
Even with the newer firmware my recommendation stands:
Completely Switch OFF the I/O Adapter Enhanced Capacity feature as you install the machine as it needs a power off
- Unless you are running Linux on... [More]
If you follow this blog you will already be aware that we (Gareth Coates, Clive Benjamin, Mike Pearson, Jyoti Dodhia, Rachel Meager and the guys from Lab Servers that are running Oracle, WAS etc performance tests to hammer the machine) have had a Power Systems Enterprise E870 since early September 2014 as part of the Early Ship Program. This program gives access to new machines before they are releases and their matching software for testing, learning and skills development and feedback. It has been very interesting and we have shared what we... [More]
Most of the Power Systems range of servers since POWER7 allow for the HMC organised online hot replacement of:
and the online hot adding, removing or replacement of:
Internal hard disks
From POWER8 onwards we are not going to have the online add or remove CEC nor adding or removing Processors modules nor Memory cards. Of course online changes using Capacity Upgrade on Demand is still very much an option. On the POWER8, I/O Drawers we... [More]
People share nmon captured data files with me all the time because they would like me to review the data and make suggestions about the hot spot and tuning.
I guess they trust me :-)
Some user get a little paranoid about the data (mostly due to having to follow many procedures to get the nmon file through firewalls so it can be emailed to me) and ask questions about if it contains information they don't want to accidentally share. Thankfully the nmon data files is just a text file that you can edit with "vi" so there... [More]
I need to make it clear that these pictures are from an Early Ship Program machine which might vary a little from the final Generally Available (GA) models from November 2014. For example, missing labels or stickers plus colours of handles and parts etc. The engineering will be much the same.
Our machine is part of the Early Ship Program and we get to run hardware tests before machines are shipped to customers from GA. These test include things we would never expect customer to do -... [More]
I need to make it clear that these pictures are from an Early Ship Program machine which might vary a little from the final Generally Available (GA) models in November 2014. For example, missing labels or stickers plus colours of handles and parts etc. The engineering will be much the same.
Please see my 8 minute video of the machine on YouTube here:
POWER8 Enterprise - Power E870 First Look http://youtu.be/jOzPTopt7HE
These pictures were take by Gareth Coates and the... [More]
Yes this is still the AIX pert Blog but my beautiful POWER8 machines can also run that "odd ball" operating system now called IBM i. That should generate some comments :-) We have a project to install IBM i 7.2 and then adopt it into PowerVC for Capture and Deploy. This feature is in beta test and selected customers allowed to give it ago. It is planned for full release later this year (that is not an IBM promise).
If you are interested in testing PowerVC with IBM i visit:... [More]
Assuming you have dozens to hundreds of AIX VM's now running on your PowerVC controlled machines at some point you will have to upgrade the AIX version to keep current and ensure security.
PowerVC does not have OS upgrade functions but it does have features that can help make upgrades safe and simpler. The upgrading of AIX is already handled by tried and tested methods and to be blunt I am not convinced I want PowerVC to do it as I want to visually check the install worked and immediately address any issues to ensure... [More]
If while installing SUSE SLES 11.3 on your POWER8 machine from the regular DVD media, you get this on the console after 5 seconds after booting:
Welcome to SuSE Linux Enterprise 11!
Type "install" to start the YaST installer on this CD/DVD
Type "slp" to start the YaST install via network
Type "rescue" to start the rescue system on this CD/DVD
Welcome to yaboot version r22.8-r1190.SuSE... [More]
We have the new HMC 8.1 code stream to manage the POWER8 machines. It also manages the POWER6 and POWER7 machines but not POWER5.
I find the HMC Virtual Storage Management panel very useful and saves me time for assigning Shared Storage Pool (SSP) space to virtual machines (VM's) and automatically doing this for Dual VIO Servers without having to log-on to the VIOS servers at all. It is also good for assigning virtual DVDs to VMs to boot and install the various operating systems, AIX, RHEL, SLES, Fedora, Debian, etc.
RHEL (6.4 and 6.5) Servers show their PC heritage when you assign server style static IP addresses instead of home network DHCP controlled IP networks. [[That will generate a few comments!!]]
I find the /etc/resolv.conf file gets modified after a reboot and so DNS fails - this is a broken as designed feature. I would enjoy someone trying to defend this feature and why it does this for my benefit! [[That will generate a even more comments!!]]
I search the network and the comments are the... [More]
Many people in computing, we seem fixated on "percentages" when talking about CPU use - We need to fix that.
When a computer has an absolutely fixed number of CPUs on the time scale of a few years the percentages of the box makes a lot of sense.
But in a virtualised world we will have to change.
For 10 years, we can do Dynamic CPU changes to a LPAR (Virtual Machine).
Yesterdays 40% used is today's 80% because the admin guy added a couple of CPUs over night.
If we look at the percentages and see a... [More]
In the past with the POWER7 and POWER7+ based machines at the lower end like the Power710 to Power740 Models, users had to shut down the whole machine to add, remove or replace an adapter safely. In fact, some machines would power off as you removed the lid.
This is no longer the case with the whole POWER8 range including the Scale-Out Models like the S822 and S824. These machines do not have the blind swap cassettes which allow the adapters to be removed from the rear of the machine. So you must have the cable... [More]
Odd. I thought I have put this on this blog before but I can find it now so just in case ... here it is or here it is again.
The SSP4 can have mirrored failure group FC disks to handle adapter, FC cable, FC switch, entire FC disk sub-system, site or VIOS failure.
But there is still a single SSP Repository disk. Isn't this a single point of failure?
The answer is " No because you can quickly rebuild the contents of the Repository Disk after it has failed ."
You are not meant to know this... [More]
I was told that Fedora's latest build for the Power ppc64 processor would support SMT=8 , if you "err ... um" have machine with a processor that support that SMT=8 !!! This is not an announcement.
So I gave it ago - here is my report:
And I can't help thinking PowerVC would remove all this work, once the new version arrives that support my ... umm err! new machine!!
I can tell you know that this was a POWER8 S824... [More]
With Power Systems there are many ways to run your Operating System(s) and each has advantages and each has a different install process.
1 Native Headless Whole Machine Mode
Number of operating systems = 1
Operating Systems: AIX, IBM i, Linux
Console = RS232 Dumb terminal
Virtual Machines = none
Machine types = Blades, Power Systems or PowerLinux (only runs Linux, of course)
Can't make use of the Power hardware based virtualisation advantage.
This blog also applies to Power Servers running Linux on Power native on the machine.
While you can perform a text based Linux install on the dumb console terminal emulator (see previous AIXpert Blog about the serial cable that you need and the software) as a challenge to survive the Tab navigation, space selection, sometimes the cursor works and occasional Function Key show stopper (SUSE SLES 11) - I would recommend a VNC install.
First, you can get back to your desk to run VNC viewer client to complete the install and not... [More]
Normally I recommended to go virtual:
I would recommend anyone trying a Power System or PowerLinux system to run it with a Hardware Management Console (HMC) -now called a Power Management Console - or using Integrated Virtualisation Manager (IVM = like HMC and VIOS combination as the first LPAR / VM).
This way you are using the full power of the Power based machines to run LPARs / Virtual Machines and push higher utilisation and extra flexibility.
But if you must go commando!
But some what to ignore... [More]
Inspired by this TED Talks Video on Leadership
TED Talk by Simon Sinek
Topic: How Great Leaders Inspire Action
18 minutes long
It tells us to start with Why? To win hearts and minds.
And not explain What? or How? until the Why? is very clear.
I thought I would make an attempt:
We believe server sprawl means a future in which we drown in small computers taking 80% of the... [More]
If you have:
High-end Power 770, Power780 or Power 795 machines
With Capacity Upgrade on Demand (CUoD)
More than 4 CPU cores and 32 GB RAM which are not Activated (dark)
... then you are ready for Power Integrated Facility for Linux (IFL) - which is a offers a package of 4 cores and 32 GB of memory at a special low price (given the resources, you can order more than one IFL)
includes the PowerVM license and PowerVM SWMA for the newly activated CPU cores
excludes a Linux license... [More]
Please provide comments on this blog entry - I want to know where I have it wrong?
With IBM's massive investment in Linux on Power my brain hurts trying to express the advantages. This is because the technology is excellent but the bits you want depend on where you are coming from. So I have tried to break out potential users in to various types.
For the record I am an Ab Linux on POWER User type!
A - Current AIX and POWER user
Already understands the POWER infrastructure, AIX and/or IBM i and are adding... [More]
I have been installing beta copies of PowerVC, the new IBM product which is available tomorrow 6th Dec 2013. It is quick to install and allows rapid rolling out of new virtual machines to a group of physical Power machines from your list of pre-packaged software images in just a few seconds of your time and roughly 2 to 3 minutes elapse time till you can login and use it. PowerVC is based on OpenStack which has lots of modules to install and configure. It is installed on Red Hat Enterprise Linux RHEL 6.4 on Power or x86 - So it is a... [More]
*** STOP PRESS *** - See the update in red at the bottom.
I was asked this lots of times at the recent POWER Technical Universities in Orlando and Athens.
Compared to a low numbers of local disks
The LUNs on a SAN disk subsystem with caching are very fast - even a simple test like AIX install times have halved. In addition, you don't have to fiddle about with stripping AIX Logical Volumes across disks or retro fit this to rootvg Volume Group filesystems after the install.
Compared to a... [More]
IBM announced lots of Power Systems software yesterday but the official announcements are pretty thin on the details. I have many Twitter tweets asking "What the heck does that all really mean?" and the official statements are marketing messages and not technical information like:
How do you get that working?
What are the pre-reqs?
What are the actual commands?
What does the Graphical User Interface look like?
How does that help me as a System Admin person?
I regularly get asked to run a demonstration for IBM Systems Director but that is a time consuming business for both me and the audience. Especially, when you can find out every thing that I know from the comfort of you own office or even home office plus Stop, Replay and Pause via YouTube videos. So here is a reminder and hopefully a page I can point to when I next gets asked.
ISD63 Workshop 1 Techie Intro... [More]
The latest Versioned Workload Partition (VWPAR) Version 1.1.2 is available but largely secret!
With this update I get on my AIX 7 virtual machine, in which I have installed both AIX5.2 and AIX 5.3 support:
# lslpp -L | grep -i vwpar
vwpar.52.rte 18.104.22.168 C F AIX 5.2 Versioned WPAR Runtime
vwpar.images.52 22.214.171.124 C F Versioned WPAR 5.2 Support
vwpar.images.53 126.96.36.199 C F Versioned WPAR 5.3 Support
vwpar.images.base 188.8.131.52 C F ... [More]
Most AIX system administrators use: smitty tcpip to add a hostname, IP address, network mask, gateway and DNS server setting to get the virtual machines (LPAR) and AIX on a network. And as a side benefit so we don't need to use the ghastly VTERM console any more as ssh or the dreaded unsecure telnet (just don't do it) can now be used - as the root user:
# smitty tcpip
Minimum Configuration & Startup
Use DHCP for TCPIP Configuration & Startup
When I log on to my home machine being AIX,of course, and type hostname I get:
nag@blue:/home/nag$ hostname -s
I get this because when I set the hostname in smitty it looks like this:
* HOSTNAME [ blue.aixncc.uk.ibm.com ]
* Internet ADDRESS (dotted decimal) [184.108.40.206]
Network MASK (dotted decimal) ... [More]
I had better point out that any performance tool of AXI command in a POWER Virtual Machine (LPAR) can only see its use of the Physical CPU, the size of its own CPU pool and (if Monitoring is switched on at the HMC), it has a number of the CPU use by "Other LPARs" use of its own CPU Pool. nmon gets these numbers by default and the nmon analyser graphs this data in the LPAR tab - this assumes its a Shared CPU LPAR. If it is a dedicated CPU LPAR then it can't get any of this Pool data and there is no LPAR tab in the nmon... [More]
I regularly get asked a question like: I have 4 months of data from 25 machines and have to develop a Capacity Planning model to size these LPARs on to new machines but I am having problems with having so much data. What can you recommend? We need graphs of
CPU compared to Entitlement
CPU Physical CPU use
Maximum real memory use
Sometimes this data is needed as input in to the Workload Estimator tool or Server Consolidation tools.
The answer is "it depends" and there is no Rule Of Thumb - IMHO.
The 2 times ROT is very old school and for small servers where peaks could really run out of RAM fast and hurt.
The worst I saw was 6 times but that was OS guys and RDBMS guys and app guys all said 2 times ... each!! and then the System Admin guy just added them up not realising it was the same 2 times! Basically, bonkers and they used 2% in the end.
With machines with large memory, say starting at 64 GB, the memory demand... [More]
When you install Linux on an IBM POWER processor machine (Red Hat, SUSE or now includes Fedora), you then need to add the IBM supplied additional RPM packages. These give you a whole bunch of useful tools like HMC comms so you can Dynamic LPAR change and LPM, diagnostic tools, and configuration listing commands. Plus more importantly nmon :-) The repository allows you to update your packages as new versions become available and to benefit from fixes and additional functions. Without these you don't get the full Power... [More]
The fifth POWER Ask the Experts is a one day customer technical event in the UK which ran in Manchester and London on 15th/16th July 2013. It proved to be a very popular free event. We had well over a 100 people attend which was a mixture of customers, a few Business partners and some IBMers.
10:00 - Power Systems Update - Pat O'Rourke: Austin Briefing Centre, USA Download 22MB .pdf
11:15 - Performance Best Practices with POWER7 - Nigel Griffiths, ATS Europe ... [More]
I have tried to get permission to pass on these numbers publically but got no answer at all. As these are facts rather than opinions, I see no harm in them. So I will work on the old theory: "it is easier to ask for forgiveness than permission" ... here goes.
The AIX Virtual User Group often asks a few questions as part of the registration process for the next webinar session and it generates fascinating statistics from real POWER and AIX users.
The people that join the AIX Virtual User Group are... [More]
Following on from my previous AIXpert blog on 1st July 2013 - "Core files filling important filesystems? Want email alerts about each core dump?" it was pointed out to me by AIX guru Mathew Accapadi that we can for many years use the ODM and AIX error reporting framework to send emails on the generation of core files. See the previous blog on how to force the core files to a particular directory - and not allow them to fill up important system directories. On the alternative reporting method see below:
Check out the... [More]
I got asked these questions recently and had to go look the subject up ... again! I seem to have forgotten some of the details and then I thought I would use some new features of AIX for the second part. In the distant past there was various way to stop core files being dumped in to the current working directory of the program that failed. In AIX 5.3, AIX 6 and 7, the "chcore" command does all the hard work for us by letting us
Choose a specific directory for core files - which is best in it own filesystem so it... [More]
A very nice chart from Richard Milton of IBM on the minimum levels for AIX and VIOS of Power7 and Power7+ hardware.
Download as a PDF: AIX_Levels_on_POWER7_Hardware_2013_06_20.pdf
We will try to update this as new machines arrive.
I have been looking at some nmon data from an IBMer looking into a customers machine. The virtual machine (LPAR) is running the Oracle RDBMS with 75 dedicated CPUs - on a POWER7 Power 795 at 4 GHz.
The Excel spreadsheet is quite scary at 144 MB thank goodness my Thinkpad has a SSD but not the record by a long way. Too many data save points can ready add u and so too can lots of processes if you are saving process data. Also with 75 CPU and SMT=4 that is 300 Tabs for Logical CPU, 300 for PCPU and 300 for SCPU and that... [More]
In Advanced Technology Support (ATS), we like to think we know what we are doing. Then something catches us out and we think that we had better let every other Power Systems administrators know about this or there are going to be many horrified users out there and many complaints. Note this problem hit our combination of HMC, VIOS and machines - you might get lucky but don't risk it!
Gareth Coates (my ATS partner) volunteered to perform the regular 6 month update of our systems but hit a BIG problem. He writes:... [More]
This is a typical request from IBMers in services or direct from customers wanting to confirm a good disk setup or confirm disks deliver as promised. This time "... a test of SAN disk performance where LUN's are carved from a IBM Flash System 820 with SVC. Primary objective is to show the IBM Flash System 820 can do hundreds of thousands IOPS with response time <1ms with good throughput. This is for an Oracle RDBMS using ASM"
Wow!! those new Flash Systems are really the next generation of disks. Well disk testing is not... [More]