AIXpert Blog is about the AIX operating system from IBM running on POWER based machines called Power Systems and software related to it like IBM Systems Director, PowerVM for virtualisation and PowerSC for security plus performance monitoring and nmon
With Power Systems there are many ways to run your Operating System(s) and each has advantages and each has a different install process.
1 Native Headless Whole Machine Mode
Number of operating systems = 1
Operating Systems: AIX, IBM i, Linux
Console = RS232 Dumb terminal
Virtual Machines = none
Machine types = Blades, Power Systems or PowerLinux (only runs Linux, of course)
Can't make use of the Power hardware based virtualisation advantage.
This blog also applies to Power Servers running Linux on Power native on the machine.
While you can perform a text based Linux install on the dumb console terminal emulator (see previous AIXpert Blog about the serial cable that you need and the software) as a challenge to survive the Tab navigation, space selection, sometimes the cursor works and occasional Function Key show stopper (SUSE SLES 11) - I would recommend a VNC install.
First, you can get back to your desk to run VNC viewer client to complete the install and not... [More]
Normally I recommended to go virtual:
I would recommend anyone trying a Power System or PowerLinux system to run it with a Hardware Management Console (HMC) -now called a Power Management Console - or using Integrated Virtualisation Manager (IVM = like HMC and VIOS combination as the first LPAR / VM).
This way you are using the full power of the Power based machines to run LPARs / Virtual Machines and push higher utilisation and extra flexibility.
But if you must go commando!
But some what to ignore... [More]
Inspired by this TED Talks Video on Leadership
TED Talk by Simon Sinek
Topic: How Great Leaders Inspire Action
18 minutes long
It tells us to start with Why? To win hearts and minds.
And not explain What? or How? until the Why? is very clear.
I thought I would make an attempt:
We believe server sprawl means a future in which we drown in small computers taking 80% of the... [More]
If you have:
High-end Power 770, Power780 or Power 795 machines
With Capacity Upgrade on Demand (CUoD)
More than 4 CPU cores and 32 GB RAM which are not Activated (dark)
... then you are ready for Power Integrated Facility for Linux (IFL) - which is a offers a package of 4 cores and 32 GB of memory at a special low price (given the resources, you can order more than one IFL)
includes the PowerVM license and PowerVM SWMA for the newly activated CPU cores
excludes a Linux license... [More]
Please provide comments on this blog entry - I want to know where I have it wrong?
With IBM's massive investment in Linux on Power my brain hurts trying to express the advantages. This is because the technology is excellent but the bits you want depend on where you are coming from. So I have tried to break out potential users in to various types.
For the record I am an Ab Linux on POWER User type!
A - Current AIX and POWER user
Already understands the POWER infrastructure, AIX and/or IBM i and are adding... [More]
I have been installing beta copies of PowerVC, the new IBM product which is available tomorrow 6th Dec 2013. It is quick to install and allows rapid rolling out of new virtual machines to a group of physical Power machines from your list of pre-packaged software images in just a few seconds of your time and roughly 2 to 3 minutes elapse time till you can login and use it. PowerVC is based on OpenStack which has lots of modules to install and configure. It is installed on Red Hat Enterprise Linux RHEL 6.4 on Power or x86 - So it is a... [More]
*** STOP PRESS *** - See the update in red at the bottom.
I was asked this lots of times at the recent POWER Technical Universities in Orlando and Athens.
Compared to a low numbers of local disks
The LUNs on a SAN disk subsystem with caching are very fast - even a simple test like AIX install times have halved. In addition, you don't have to fiddle about with stripping AIX Logical Volumes across disks or retro fit this to rootvg Volume Group filesystems after the install.
Compared to a... [More]
IBM announced lots of Power Systems software yesterday but the official announcements are pretty thin on the details. I have many Twitter tweets asking "What the heck does that all really mean?" and the official statements are marketing messages and not technical information like:
How do you get that working?
What are the pre-reqs?
What are the actual commands?
What does the Graphical User Interface look like?
How does that help me as a System Admin person?
I regularly get asked to run a demonstration for IBM Systems Director but that is a time consuming business for both me and the audience. Especially, when you can find out every thing that I know from the comfort of you own office or even home office plus Stop, Replay and Pause via YouTube videos. So here is a reminder and hopefully a page I can point to when I next gets asked.
ISD63 Workshop 1 Techie Intro... [More]
The latest Versioned Workload Partition (VWPAR) Version 1.1.2 is available but largely secret!
With this update I get on my AIX 7 virtual machine, in which I have installed both AIX5.2 and AIX 5.3 support:
# lslpp -L | grep -i vwpar
vwpar.52.rte 18.104.22.168 C F AIX 5.2 Versioned WPAR Runtime
vwpar.images.52 22.214.171.124 C F Versioned WPAR 5.2 Support
vwpar.images.53 126.96.36.199 C F Versioned WPAR 5.3 Support
vwpar.images.base 188.8.131.52 C F ... [More]
Most AIX system administrators use: smitty tcpip to add a hostname, IP address, network mask, gateway and DNS server setting to get the virtual machines (LPAR) and AIX on a network. And as a side benefit so we don't need to use the ghastly VTERM console any more as ssh or the dreaded unsecure telnet (just don't do it) can now be used - as the root user:
# smitty tcpip
Minimum Configuration & Startup
Use DHCP for TCPIP Configuration & Startup
When I log on to my home machine being AIX,of course, and type hostname I get:
nag@blue:/home/nag$ hostname -s
I get this because when I set the hostname in smitty it looks like this:
* HOSTNAME [ blue.aixncc.uk.ibm.com ]
* Internet ADDRESS (dotted decimal) [184.108.40.206]
Network MASK (dotted decimal) ... [More]
I had better point out that any performance tool of AXI command in a POWER Virtual Machine (LPAR) can only see its use of the Physical CPU, the size of its own CPU pool and (if Monitoring is switched on at the HMC), it has a number of the CPU use by "Other LPARs" use of its own CPU Pool. nmon gets these numbers by default and the nmon analyser graphs this data in the LPAR tab - this assumes its a Shared CPU LPAR. If it is a dedicated CPU LPAR then it can't get any of this Pool data and there is no LPAR tab in the nmon... [More]
I regularly get asked a question like: I have 4 months of data from 25 machines and have to develop a Capacity Planning model to size these LPARs on to new machines but I am having problems with having so much data. What can you recommend? We need graphs of
CPU compared to Entitlement
CPU Physical CPU use
Maximum real memory use
Sometimes this data is needed as input in to the Workload Estimator tool or Server Consolidation tools.
The answer is "it depends" and there is no Rule Of Thumb - IMHO.
The 2 times ROT is very old school and for small servers where peaks could really run out of RAM fast and hurt.
The worst I saw was 6 times but that was OS guys and RDBMS guys and app guys all said 2 times ... each!! and then the System Admin guy just added them up not realising it was the same 2 times! Basically, bonkers and they used 2% in the end.
With machines with large memory, say starting at 64 GB, the memory demand... [More]
When you install Linux on an IBM POWER processor machine (Red Hat, SUSE or now includes Fedora), you then need to add the IBM supplied additional RPM packages. These give you a whole bunch of useful tools like HMC comms so you can Dynamic LPAR change and LPM, diagnostic tools, and configuration listing commands. Plus more importantly nmon :-) The repository allows you to update your packages as new versions become available and to benefit from fixes and additional functions. Without these you don't get the full Power... [More]
The fifth POWER Ask the Experts is a one day customer technical event in the UK which ran in Manchester and London on 15th/16th July 2013. It proved to be a very popular free event. We had well over a 100 people attend which was a mixture of customers, a few Business partners and some IBMers.
10:00 - Power Systems Update - Pat O'Rourke: Austin Briefing Centre, USA Download 22MB .pdf
11:15 - Performance Best Practices with POWER7 - Nigel Griffiths, ATS Europe ... [More]
I have tried to get permission to pass on these numbers publically but got no answer at all. As these are facts rather than opinions, I see no harm in them. So I will work on the old theory: "it is easier to ask for forgiveness than permission" ... here goes.
The AIX Virtual User Group often asks a few questions as part of the registration process for the next webinar session and it generates fascinating statistics from real POWER and AIX users.
The people that join the AIX Virtual User Group are... [More]
Following on from my previous AIXpert blog on 1st July 2013 - "Core files filling important filesystems? Want email alerts about each core dump?" it was pointed out to me by AIX guru Mathew Accapadi that we can for many years use the ODM and AIX error reporting framework to send emails on the generation of core files. See the previous blog on how to force the core files to a particular directory - and not allow them to fill up important system directories. On the alternative reporting method see below:
Check out the... [More]
I got asked these questions recently and had to go look the subject up ... again! I seem to have forgotten some of the details and then I thought I would use some new features of AIX for the second part. In the distant past there was various way to stop core files being dumped in to the current working directory of the program that failed. In AIX 5.3, AIX 6 and 7, the "chcore" command does all the hard work for us by letting us
Choose a specific directory for core files - which is best in it own filesystem so it... [More]
A very nice chart from Richard Milton of IBM on the minimum levels for AIX and VIOS of Power7 and Power7+ hardware.
Download as a PDF: AIX_Levels_on_POWER7_Hardware_2013_06_20.pdf
We will try to update this as new machines arrive.
I have been looking at some nmon data from an IBMer looking into a customers machine. The virtual machine (LPAR) is running the Oracle RDBMS with 75 dedicated CPUs - on a POWER7 Power 795 at 4 GHz.
The Excel spreadsheet is quite scary at 144 MB thank goodness my Thinkpad has a SSD but not the record by a long way. Too many data save points can ready add u and so too can lots of processes if you are saving process data. Also with 75 CPU and SMT=4 that is 300 Tabs for Logical CPU, 300 for PCPU and 300 for SCPU and that... [More]
In Advanced Technology Support (ATS), we like to think we know what we are doing. Then something catches us out and we think that we had better let every other Power Systems administrators know about this or there are going to be many horrified users out there and many complaints. Note this problem hit our combination of HMC, VIOS and machines - you might get lucky but don't risk it!
Gareth Coates (my ATS partner) volunteered to perform the regular 6 month update of our systems but hit a BIG problem. He writes:... [More]
This is a typical request from IBMers in services or direct from customers wanting to confirm a good disk setup or confirm disks deliver as promised. This time "... a test of SAN disk performance where LUN's are carved from a IBM Flash System 820 with SVC. Primary objective is to show the IBM Flash System 820 can do hundreds of thousands IOPS with response time <1ms with good throughput. This is for an Oracle RDBMS using ASM"
Wow!! those new Flash Systems are really the next generation of disks. Well disk testing is not... [More]
I am getting asked more about security these days than in the previous decade - perhaps there is renewed focus on this area as we hear about hacker groups taking what they want from organisations and companies in the News every day. Below is an answer I gave to a recent email question on what AIX offers on the security side, spotting vital files being changed and what parts of PowerSC to use for quick wins.
For AIX security the three big hitters for "out of the box" AIX hardening are:
Ditch telnet/ftp in the... [More]
Your first reaction might be "Why would you do that, Nigel?" and "Isn't this a AIX blog: Why are we talking about Linux?"
Well AIX is great and all that but when it comes to Open Source it is not Linux. Like it or not most Open Source stuff is developed on Linux and there are readily available binary compiled for Linux and 100's of Open Source packages actually come with the the distribution on their DVD media and arch repositories on the Internet for rapid and straight forward install and updates.
Having just done this and getting questions on it from customers, I thought I should share a few notes:
1) The process is nicely documented and you need to follow the instructions. I recommend using the Release Notes as a check list as missing out a stop causes the next bit for complain or fail. Fortunately, the recovery is just doing the bit I missed.
2) Read the Release notes - even better print them out and cross them off as you go:
http://www.ibm.com/support/docview.wss?uid=hpc1vios117f5701 - upgrade... [More]
If you read this blog regularly then you will know I am a fan of the VIOS Shared Storage Pool technology .
It is a major peice of the "Making PowerVM and VIOS - easier to live with and reduce the man-power" story and it is hard to believe that we started withPOWER4 virtualisation in 2000 and VIOS in 2005. This does mean that the tools we use (HMC and VIOS padmin) need a bit of an update and I know these are in plans for the future. Even now we can start using SSP to save lots of system admin time and be more flexibly... [More]
Recently I was running the "lsmap -l" command and it output error messages that were less then useless at the end of the output:
Command did not complete.
"Determine if backing device is a PV or LV or optical." was last subcommand run.
Command did not complete.
Determining device type failed. Command did not complete.
"Determine if backing device is a PV or LV or optical." was last subcommand run.
Command did not complete.
Determining device type failed. Command did not complete.
"Determine if... [More]
Installing Open Source on AIX systems can be a little .... um .... frustrating and I have just been round that loop with installing Ganglia the excellent Performance Monitoring tool to demonstrate it and actually use it too on my own servers. The hard bit is that for every Open Source program, that you want to install, there is 10 to 20 prerequisite packages that you need to install first and you go round this the loop four times as the prerequisites have prerequisites .... This has got known as the "RPM dependencies form... [More]
POWER Relative Performance (rPerf) is often used as a way to approximate the expected difference in performance between two Power Systems servers. Although rPerf is a useful tool, it is important to understand the limitations of using rPerf to provide an estimate the performance of your specific workloads in your particular environment with a new server, First, rPerf numbers, like any published benchmark, represent the best case result when the application, configuration and system resources are all optimized -- factors that are likely not... [More]
Questions I have a number of customers asking me these types question:
Are there are good practices on how to size disks for the storage pools. If I want a 5 TB pool, how many disks which size, does it matter ? Also are there any document limits regarding, pool disk, LUN and so on. Answers Not sure about best practice but there is common sense - will that do? Restrictions are included with the VIOS announcement "Read Me" notes, which you can find here for the current VIOS 220.127.116.11... [More]
Shared Storage Pools - just got even more interesting ! SSP Clusters with disks pools and super fast disk allocation VIOS Shared Storage Pools (phase 3) allows 16 VIOS on different machines to operate as a VIOS cluster with a set of SAN LUNs in the pool - think in terms of a number of TBs of disk space in the pool. The VIOS systems administrator can then allocate disk space to a new or existing Virtual Machine (LPAR) in around a second which can be thin or thick provisioning regardless of the underlying disks. This drastically reduces the... [More]
I HAVE UPDATED THIS BLOG WITH BETTER, UP TO DATE AND CORRECTED INFORMATION FROM THE DEVELOPERS Read the comments for lots of useful information - Thanks Guys for the corrections WARNING: Advance Technology Support (ATS) Europe is getting reports today from users and IBMers that the POWER7 firmware 770 level as a pre-req check missing! could be much better . This results in the HMC agreeing to upgrade your POWER7 or POWER7+ machine to firmware level 770, when it is not supported and does not work. This causes very bad problems: On smaller... [More]
Just completed the move of the POWER and AIX Hands-on Movies to YouTube videos - after pruning we have 97 of them. This means you can watch them on your mobile Smartphone and Tablets in addition to your workstation/personal computer. You can find them all here:
YouTube look for my "Nigel Griffiths" Channel http://www.youtube.com/user/nigelargriffiths This should mean they are simpler to find, search for and view. During the transfer I pruned out a few of the older ones particular Systems 6.2 as we now have a set of Systems... [More]
Fourteen new movies/Video's on System Director on YouTube I have just last week uploaded 14 new System Director movies that were previously a whole day Technical University session and I thought it a good idea to share these with every body. It comes to about 4 hours in total and it a mixture of sizes from 7 minutes to 29 minutes. The focus is on core Systems Director function rather than advanced function plug-ins. It worries me a little that we often try to talk about and sell Systems Director for the advanced stuff but you really need to get... [More]
Our Advanced Technology Support, Europe team have had a few months secretly testing the Power 750/760 internally to IBM as part of the normal Early Ship Program which lets a few machine out of the development labs to be tested by others to gain feedback on the product, documentation, features and usability. I thought others would like to see a few pictures of the team and the equipment involved. Please note: This is an Early Ship Machine, so there are a few details like stickers and handle colours that may differ slightly from the final product... [More]
Hot news this morning is IBM announcing the updating of the Lower End Power 710/730 and Power 720/740 models to include Power7+ processor with 2.5 larger L3 cache, Active Memory Expansion accelerator, and GHz boost Double the memory size This gives a large performance boost and drastically higher "bangs per buck" as the Americans say or "Value for money" as we Brits say. Almost everything else is the same so if you know the older models it is easy to understand Announcement letters for the new Power7+ models: Power 710 ,... [More]
Another good questions from a customer and my answer was simple: The PowerVM internal network on POWER machines operates just like a physical network. If two LPAR are on the same internal network and the root user puts the virtual adapter in to promiscuous mode then it can see all packets. Just like a physical network. If the two LPARs are on different internal networks then they can never see the other LPARs packets. Just like two physical networks. If two LPAR are on the same internal network but on different VLANs then they can never see... [More]
Hi, Rob McNelly, the esteemed POWER7 Champion, sent me a really good questions and I thought would make a good blog as I get asked this about once a month. I have cut down the questions a but but it is a two parts: His customer is thinking about putting Systems Director on to their NIM server. Rob remembered me commenting on this but wants the details. They are planning to give it 1 dedicated Power7 core and 12G memory. What do I think about that? Answer 1: 2 years ago this combination (NIM & ISD) was not allowed ( I think it was just not... [More]
The new VIOS Shared Storage Pools 3 (SSP3) arrived a month ago and I have all my VIOS updated now to VIOS 18.104.22.168 plus fixes. This brings with it the regular VIOS improvements but also new features for SSP3 including: Better pool use stats for example see the below lssp command and the Used percentage and Unused(mb) columns and the over view output too has new data $ lssp -clustername stars -sp atlantic -bd Lu Name Size(mb) ProvisionType %Used Unused(mb) Lu Udid orange7a 32768 THIN 9% 29615 7d58538152 …... [More]
Time to correct a few assumptions and statements that I made a while back. Previously I said the POWER Hypervisor decides where to put a Virtual Machine (VM/LPAR) based on the Virtual Processor number (spreading factor). Well, apart from it nearly being right ... I was actually wrong! I got talking to one of these very impressive Hypervisor developers in Germany and he put me right. In most of my larger machines, I do what I think is fairly normal for creating an VM - like working with a CPU core to RAM ratio that my customers use, like 1... [More]
I have indirectly been getting copies of this 4 page, .PDF, once a quarter, Power Systems Newsletter written by Richard Milton (an IBMer in the USA) and every time found a few "Wow! That's Cool!" items in it. I just checked with Richard and he says anyone is free to sign up to the email list and you will get it emailed to you once a quarter. I have signed up. Needless to say, it is for Techies and not marketing fluff but has a couple of one page quick details of particular products and experience and comments on them. Then there... [More]
A large UK customer is running a series of tests on AIX and POWER7 machines and wanted to display the results on a webserver using rrdtool and needed help getting a working example running. So I stepped up to the challenge one night rather than it becoming a search for rrdtool skills inside IBM - when I could sort something out in a few hours. No one in IBM would claim to be a rrdtool expert until they knew the details of what was required. Many people know it well enough to get what they want done but would not put rrdtool guru on their CV. ... [More]
I am currently updating my AIX systems and AIX 7 TL2 with its first service pack is now available for download from ESS or IBM Entitlem Software Support to spell it out in full (http://www.ibm.com/servers/eserver/ess) for install media and FixCentral for updates (http://www-933.ibm.com/support/fixcentral/). While downloading the install media files I noticed there is a new Firefox for AIX available Firefox V3.6.25. This is a jump up from the previous one and I have had (in the past) a few annoying problems with Firefox on AIX Some weird issue... [More]
Four things I think you should know before you try this: The documentation for the update can be found here: http://www-01.ibm.com/support/docview.wss?uid=hpc1vios117f5701 I updated three of my Virtual I/O Servers to 22.214.171.124 from 126.96.36.199 over the weekend. I started these late on Friday night with the "no reboot" options and they finished about the same time on Sunday afternoon i.e. 48 hours . The Release Manager downloads the update directly to the VIOS /opt/VIOS/update/VIOS and sub directory for the release name and this takes over... [More]
Common question: I see XX% of Used Memory but I don't know who is using it and how?
A simple question but there is no simple answer as memory use if complex.
nmon then hit "m" will quickly show you a few big uses of memory as below:
The shared memory used by lots of applications like DB2 and Oracle - check the SEGSZ for the size.
The Owner column usually tells you what it is used for like the oracle user for the SGA or db2inst1 for the DB2 buffer... [More]
At a recent POWER & AIX Workshop, I was challenged when I stated that you can download IBM Systems Director (ISD) and run it at no costs. Granted: If you want IBM Support for ISD functions then you do need to purchase the license and SWMA for every endpoint (think HMC and every core of any machine that ISD is controlling). Compared to other systems management tools (I have worked with in the past) the ISD prices are very low so its worth asking for the local currency price. I also think most people (including myself) will get stuck on... [More]