AIXpert Blog is about the AIX operating system from IBM running on POWER based machines called Power Systems and software related to it like IBM Systems Director, PowerVM for virtualisation and PowerSC for security plus performance monitoring and nmon
This blog also applies to Power Servers running Linux on Power native on the machine.
While you can perform a text based Linux install on the dumb console terminal emulator (see previous AIXpert Blog about the serial cable that you need and the software) as a challenge to survive the Tab navigation, space selection, sometimes the cursor works and occasional Function Key show stopper (SUSE SLES 11) - I would recommend a VNC install.
First, you can get back to your desk to run VNC viewer client to complete the install and not... [More]
Normally I recommended to go virtual:
I would recommend anyone trying a Power System or PowerLinux system to run it with a Hardware Management Console (HMC) -now called a Power Management Console - or using Integrated Virtualisation Manager (IVM = like HMC and VIOS combination as the first LPAR / VM).
This way you are using the full power of the Power based machines to run LPARs / Virtual Machines and push higher utilisation and extra flexibility.
But if you must go commando!
But some what to ignore... [More]
Inspired by this TED Talks Video on Leadership
TED Talk by Simon Sinek
Topic: How Great Leaders Inspire Action
18 minutes long
It tells us to start with Why? To win hearts and minds.
And not explain What? or How? until the Why? is very clear.
I thought I would make an attempt:
We believe server sprawl means a future in which we drown in small computers taking 80% of the... [More]
If you have:
High-end Power 770, Power780 or Power 795 machines
With Capacity Upgrade on Demand (CUoD)
More than 4 CPU cores and 32 GB RAM which are not Activated (dark)
... then you are ready for Power Integrated Facility for Linux (IFL) - which is a offers a package of 4 cores and 32 GB of memory at a special low price (given the resources, you can order more than one IFL)
includes the PowerVM license and PowerVM SWMA for the newly activated CPU cores
excludes a Linux license... [More]
Please provide comments on this blog entry - I want to know where I have it wrong?
With IBM's massive investment in Linux on Power my brain hurts trying to express the advantages. This is because the technology is excellent but the bits you want depend on where you are coming from. So I have tried to break out potential users in to various types.
For the record I am an Ab Linux on POWER User type!
A - Current AIX and POWER user
Already understands the POWER infrastructure, AIX and/or IBM i and are adding... [More]
I have been installing beta copies of PowerVC, the new IBM product which is available tomorrow 6th Dec 2013. It is quick to install and allows rapid rolling out of new virtual machines to a group of physical Power machines from your list of pre-packaged software images in just a few seconds of your time and roughly 2 to 3 minutes elapse time till you can login and use it. PowerVC is based on OpenStack which has lots of modules to install and configure. It is installed on Red Hat Enterprise Linux RHEL 6.4 on Power or x86 - So it is a... [More]
*** STOP PRESS *** - See the update in red at the bottom.
I was asked this lots of times at the recent POWER Technical Universities in Orlando and Athens.
Compared to a low numbers of local disks
The LUNs on a SAN disk subsystem with caching are very fast - even a simple test like AIX install times have halved. In addition, you don't have to fiddle about with stripping AIX Logical Volumes across disks or retro fit this to rootvg Volume Group filesystems after the install.
Compared to a... [More]
IBM announced lots of Power Systems software yesterday but the official announcements are pretty thin on the details. I have many Twitter tweets asking "What the heck does that all really mean?" and the official statements are marketing messages and not technical information like:
How do you get that working?
What are the pre-reqs?
What are the actual commands?
What does the Graphical User Interface look like?
How does that help me as a System Admin person?
I regularly get asked to run a demonstration for IBM Systems Director but that is a time consuming business for both me and the audience. Especially, when you can find out every thing that I know from the comfort of you own office or even home office plus Stop, Replay and Pause via YouTube videos. So here is a reminder and hopefully a page I can point to when I next gets asked.
ISD63 Workshop 1 Techie Intro... [More]
The latest Versioned Workload Partition (VWPAR) Version 1.1.2 is available but largely secret!
With this update I get on my AIX 7 virtual machine, in which I have installed both AIX5.2 and AIX 5.3 support:
# lslpp -L | grep -i vwpar
vwpar.52.rte 18.104.22.168 C F AIX 5.2 Versioned WPAR Runtime
vwpar.images.52 22.214.171.124 C F Versioned WPAR 5.2 Support
vwpar.images.53 126.96.36.199 C F Versioned WPAR 5.3 Support
vwpar.images.base 188.8.131.52 C F ... [More]
Most AIX system administrators use: smitty tcpip to add a hostname, IP address, network mask, gateway and DNS server setting to get the virtual machines (LPAR) and AIX on a network. And as a side benefit so we don't need to use the ghastly VTERM console any more as ssh or the dreaded unsecure telnet (just don't do it) can now be used - as the root user:
# smitty tcpip
Minimum Configuration & Startup
Use DHCP for TCPIP Configuration & Startup
When I log on to my home machine being AIX,of course, and type hostname I get:
nag@blue:/home/nag$ hostname -s
I get this because when I set the hostname in smitty it looks like this:
* HOSTNAME [ blue.aixncc.uk.ibm.com ]
* Internet ADDRESS (dotted decimal) [184.108.40.206]
Network MASK (dotted decimal) ... [More]
I had better point out that any performance tool of AXI command in a POWER Virtual Machine (LPAR) can only see its use of the Physical CPU, the size of its own CPU pool and (if Monitoring is switched on at the HMC), it has a number of the CPU use by "Other LPARs" use of its own CPU Pool. nmon gets these numbers by default and the nmon analyser graphs this data in the LPAR tab - this assumes its a Shared CPU LPAR. If it is a dedicated CPU LPAR then it can't get any of this Pool data and there is no LPAR tab in the nmon... [More]
I regularly get asked a question like: I have 4 months of data from 25 machines and have to develop a Capacity Planning model to size these LPARs on to new machines but I am having problems with having so much data. What can you recommend? We need graphs of
CPU compared to Entitlement
CPU Physical CPU use
Maximum real memory use
Sometimes this data is needed as input in to the Workload Estimator tool or Server Consolidation tools.
The answer is "it depends" and there is no Rule Of Thumb - IMHO.
The 2 times ROT is very old school and for small servers where peaks could really run out of RAM fast and hurt.
The worst I saw was 6 times but that was OS guys and RDBMS guys and app guys all said 2 times ... each!! and then the System Admin guy just added them up not realising it was the same 2 times! Basically, bonkers and they used 2% in the end.
With machines with large memory, say starting at 64 GB, the memory demand... [More]
When you install Linux on an IBM POWER processor machine (Red Hat, SUSE or now includes Fedora), you then need to add the IBM supplied additional RPM packages. These give you a whole bunch of useful tools like HMC comms so you can Dynamic LPAR change and LPM, diagnostic tools, and configuration listing commands. Plus more importantly nmon :-) The repository allows you to update your packages as new versions become available and to benefit from fixes and additional functions. Without these you don't get the full Power... [More]
The fifth POWER Ask the Experts is a one day customer technical event in the UK which ran in Manchester and London on 15th/16th July 2013. It proved to be a very popular free event. We had well over a 100 people attend which was a mixture of customers, a few Business partners and some IBMers.
10:00 - Power Systems Update - Pat O'Rourke: Austin Briefing Centre, USA Download 22MB .pdf
11:15 - Performance Best Practices with POWER7 - Nigel Griffiths, ATS Europe ... [More]
I have tried to get permission to pass on these numbers publically but got no answer at all. As these are facts rather than opinions, I see no harm in them. So I will work on the old theory: "it is easier to ask for forgiveness than permission" ... here goes.
The AIX Virtual User Group often asks a few questions as part of the registration process for the next webinar session and it generates fascinating statistics from real POWER and AIX users.
The people that join the AIX Virtual User Group are... [More]
Following on from my previous AIXpert blog on 1st July 2013 - "Core files filling important filesystems? Want email alerts about each core dump?" it was pointed out to me by AIX guru Mathew Accapadi that we can for many years use the ODM and AIX error reporting framework to send emails on the generation of core files. See the previous blog on how to force the core files to a particular directory - and not allow them to fill up important system directories. On the alternative reporting method see below:
Check out the... [More]
I got asked these questions recently and had to go look the subject up ... again! I seem to have forgotten some of the details and then I thought I would use some new features of AIX for the second part. In the distant past there was various way to stop core files being dumped in to the current working directory of the program that failed. In AIX 5.3, AIX 6 and 7, the "chcore" command does all the hard work for us by letting us
Choose a specific directory for core files - which is best in it own filesystem so it... [More]
A very nice chart from Richard Milton of IBM on the minimum levels for AIX and VIOS of Power7 and Power7+ hardware.
Download as a PDF: AIX_Levels_on_POWER7_Hardware_2013_06_20.pdf
We will try to update this as new machines arrive.
I have been looking at some nmon data from an IBMer looking into a customers machine. The virtual machine (LPAR) is running the Oracle RDBMS with 75 dedicated CPUs - on a POWER7 Power 795 at 4 GHz.
The Excel spreadsheet is quite scary at 144 MB thank goodness my Thinkpad has a SSD but not the record by a long way. Too many data save points can ready add u and so too can lots of processes if you are saving process data. Also with 75 CPU and SMT=4 that is 300 Tabs for Logical CPU, 300 for PCPU and 300 for SCPU and that... [More]
In Advanced Technology Support (ATS), we like to think we know what we are doing. Then something catches us out and we think that we had better let every other Power Systems administrators know about this or there are going to be many horrified users out there and many complaints. Note this problem hit our combination of HMC, VIOS and machines - you might get lucky but don't risk it!
Gareth Coates (my ATS partner) volunteered to perform the regular 6 month update of our systems but hit a BIG problem. He writes:... [More]
This is a typical request from IBMers in services or direct from customers wanting to confirm a good disk setup or confirm disks deliver as promised. This time "... a test of SAN disk performance where LUN's are carved from a IBM Flash System 820 with SVC. Primary objective is to show the IBM Flash System 820 can do hundreds of thousands IOPS with response time <1ms with good throughput. This is for an Oracle RDBMS using ASM"
Wow!! those new Flash Systems are really the next generation of disks. Well disk testing is not... [More]
I am getting asked more about security these days than in the previous decade - perhaps there is renewed focus on this area as we hear about hacker groups taking what they want from organisations and companies in the News every day. Below is an answer I gave to a recent email question on what AIX offers on the security side, spotting vital files being changed and what parts of PowerSC to use for quick wins.
For AIX security the three big hitters for "out of the box" AIX hardening are:
Ditch telnet/ftp in the... [More]
Your first reaction might be "Why would you do that, Nigel?" and "Isn't this a AIX blog: Why are we talking about Linux?"
Well AIX is great and all that but when it comes to Open Source it is not Linux. Like it or not most Open Source stuff is developed on Linux and there are readily available binary compiled for Linux and 100's of Open Source packages actually come with the the distribution on their DVD media and arch repositories on the Internet for rapid and straight forward install and updates.
Having just done this and getting questions on it from customers, I thought I should share a few notes:
1) The process is nicely documented and you need to follow the instructions. I recommend using the Release Notes as a check list as missing out a stop causes the next bit for complain or fail. Fortunately, the recovery is just doing the bit I missed.
2) Read the Release notes - even better print them out and cross them off as you go:
http://www.ibm.com/support/docview.wss?uid=hpc1vios117f5701 - upgrade... [More]
If you read this blog regularly then you will know I am a fan of the VIOS Shared Storage Pool technology .
It is a major peice of the "Making PowerVM and VIOS - easier to live with and reduce the man-power" story and it is hard to believe that we started withPOWER4 virtualisation in 2000 and VIOS in 2005. This does mean that the tools we use (HMC and VIOS padmin) need a bit of an update and I know these are in plans for the future. Even now we can start using SSP to save lots of system admin time and be more flexibly... [More]
Recently I was running the "lsmap -l" command and it output error messages that were less then useless at the end of the output:
Command did not complete.
"Determine if backing device is a PV or LV or optical." was last subcommand run.
Command did not complete.
Determining device type failed. Command did not complete.
"Determine if backing device is a PV or LV or optical." was last subcommand run.
Command did not complete.
Determining device type failed. Command did not complete.
"Determine if... [More]
Installing Open Source on AIX systems can be a little .... um .... frustrating and I have just been round that loop with installing Ganglia the excellent Performance Monitoring tool to demonstrate it and actually use it too on my own servers. The hard bit is that for every Open Source program, that you want to install, there is 10 to 20 prerequisite packages that you need to install first and you go round this the loop four times as the prerequisites have prerequisites .... This has got known as the "RPM dependencies form... [More]
POWER Relative Performance (rPerf) is often used as a way to approximate the expected difference in performance between two Power Systems servers. Although rPerf is a useful tool, it is important to understand the limitations of using rPerf to provide an estimate the performance of your specific workloads in your particular environment with a new server, First, rPerf numbers, like any published benchmark, represent the best case result when the application, configuration and system resources are all optimized -- factors that are likely not... [More]
Questions I have a number of customers asking me these types question:
Are there are good practices on how to size disks for the storage pools. If I want a 5 TB pool, how many disks which size, does it matter ? Also are there any document limits regarding, pool disk, LUN and so on. Answers Not sure about best practice but there is common sense - will that do? Restrictions are included with the VIOS announcement "Read Me" notes, which you can find here for the current VIOS 220.127.116.11... [More]
Shared Storage Pools - just got even more interesting ! SSP Clusters with disks pools and super fast disk allocation VIOS Shared Storage Pools (phase 3) allows 16 VIOS on different machines to operate as a VIOS cluster with a set of SAN LUNs in the pool - think in terms of a number of TBs of disk space in the pool. The VIOS systems administrator can then allocate disk space to a new or existing Virtual Machine (LPAR) in around a second which can be thin or thick provisioning regardless of the underlying disks. This drastically reduces the... [More]
I HAVE UPDATED THIS BLOG WITH BETTER, UP TO DATE AND CORRECTED INFORMATION FROM THE DEVELOPERS Read the comments for lots of useful information - Thanks Guys for the corrections WARNING: Advance Technology Support (ATS) Europe is getting reports today from users and IBMers that the POWER7 firmware 770 level as a pre-req check missing! could be much better . This results in the HMC agreeing to upgrade your POWER7 or POWER7+ machine to firmware level 770, when it is not supported and does not work. This causes very bad problems: On smaller... [More]
Just completed the move of the POWER and AIX Hands-on Movies to YouTube videos - after pruning we have 97 of them. This means you can watch them on your mobile Smartphone and Tablets in addition to your workstation/personal computer. You can find them all here:
YouTube look for my "Nigel Griffiths" Channel http://www.youtube.com/user/nigelargriffiths This should mean they are simpler to find, search for and view. During the transfer I pruned out a few of the older ones particular Systems 6.2 as we now have a set of Systems... [More]
Fourteen new movies/Video's on System Director on YouTube I have just last week uploaded 14 new System Director movies that were previously a whole day Technical University session and I thought it a good idea to share these with every body. It comes to about 4 hours in total and it a mixture of sizes from 7 minutes to 29 minutes. The focus is on core Systems Director function rather than advanced function plug-ins. It worries me a little that we often try to talk about and sell Systems Director for the advanced stuff but you really need to get... [More]
Our Advanced Technology Support, Europe team have had a few months secretly testing the Power 750/760 internally to IBM as part of the normal Early Ship Program which lets a few machine out of the development labs to be tested by others to gain feedback on the product, documentation, features and usability. I thought others would like to see a few pictures of the team and the equipment involved. Please note: This is an Early Ship Machine, so there are a few details like stickers and handle colours that may differ slightly from the final product... [More]
Hot news this morning is IBM announcing the updating of the Lower End Power 710/730 and Power 720/740 models to include Power7+ processor with 2.5 larger L3 cache, Active Memory Expansion accelerator, and GHz boost Double the memory size This gives a large performance boost and drastically higher "bangs per buck" as the Americans say or "Value for money" as we Brits say. Almost everything else is the same so if you know the older models it is easy to understand Announcement letters for the new Power7+ models: Power 710 ,... [More]
Another good questions from a customer and my answer was simple: The PowerVM internal network on POWER machines operates just like a physical network. If two LPAR are on the same internal network and the root user puts the virtual adapter in to promiscuous mode then it can see all packets. Just like a physical network. If the two LPARs are on different internal networks then they can never see the other LPARs packets. Just like two physical networks. If two LPAR are on the same internal network but on different VLANs then they can never see... [More]
Hi, Rob McNelly, the esteemed POWER7 Champion, sent me a really good questions and I thought would make a good blog as I get asked this about once a month. I have cut down the questions a but but it is a two parts: His customer is thinking about putting Systems Director on to their NIM server. Rob remembered me commenting on this but wants the details. They are planning to give it 1 dedicated Power7 core and 12G memory. What do I think about that? Answer 1: 2 years ago this combination (NIM & ISD) was not allowed ( I think it was just not... [More]
The new VIOS Shared Storage Pools 3 (SSP3) arrived a month ago and I have all my VIOS updated now to VIOS 18.104.22.168 plus fixes. This brings with it the regular VIOS improvements but also new features for SSP3 including: Better pool use stats for example see the below lssp command and the Used percentage and Unused(mb) columns and the over view output too has new data $ lssp -clustername stars -sp atlantic -bd Lu Name Size(mb) ProvisionType %Used Unused(mb) Lu Udid orange7a 32768 THIN 9% 29615 7d58538152 …... [More]
Time to correct a few assumptions and statements that I made a while back. Previously I said the POWER Hypervisor decides where to put a Virtual Machine (VM/LPAR) based on the Virtual Processor number (spreading factor). Well, apart from it nearly being right ... I was actually wrong! I got talking to one of these very impressive Hypervisor developers in Germany and he put me right. In most of my larger machines, I do what I think is fairly normal for creating an VM - like working with a CPU core to RAM ratio that my customers use, like 1... [More]
I have indirectly been getting copies of this 4 page, .PDF, once a quarter, Power Systems Newsletter written by Richard Milton (an IBMer in the USA) and every time found a few "Wow! That's Cool!" items in it. I just checked with Richard and he says anyone is free to sign up to the email list and you will get it emailed to you once a quarter. I have signed up. Needless to say, it is for Techies and not marketing fluff but has a couple of one page quick details of particular products and experience and comments on them. Then there... [More]
A large UK customer is running a series of tests on AIX and POWER7 machines and wanted to display the results on a webserver using rrdtool and needed help getting a working example running. So I stepped up to the challenge one night rather than it becoming a search for rrdtool skills inside IBM - when I could sort something out in a few hours. No one in IBM would claim to be a rrdtool expert until they knew the details of what was required. Many people know it well enough to get what they want done but would not put rrdtool guru on their CV. ... [More]
I am currently updating my AIX systems and AIX 7 TL2 with its first service pack is now available for download from ESS or IBM Entitlem Software Support to spell it out in full (http://www.ibm.com/servers/eserver/ess) for install media and FixCentral for updates (http://www-933.ibm.com/support/fixcentral/). While downloading the install media files I noticed there is a new Firefox for AIX available Firefox V3.6.25. This is a jump up from the previous one and I have had (in the past) a few annoying problems with Firefox on AIX Some weird issue... [More]
Four things I think you should know before you try this: The documentation for the update can be found here: http://www-01.ibm.com/support/docview.wss?uid=hpc1vios117f5701 I updated three of my Virtual I/O Servers to 22.214.171.124 from 126.96.36.199 over the weekend. I started these late on Friday night with the "no reboot" options and they finished about the same time on Sunday afternoon i.e. 48 hours . The Release Manager downloads the update directly to the VIOS /opt/VIOS/update/VIOS and sub directory for the release name and this takes over... [More]
Common question: I see XX% of Used Memory but I don't know who is using it and how?
A simple question but there is no simple answer as memory use if complex.
nmon then hit "m" will quickly show you a few big uses of memory as below:
The shared memory used by lots of applications like DB2 and Oracle - check the SEGSZ for the size.
The Owner column usually tells you what it is used for like the oracle user for the SGA or db2inst1 for the DB2 buffer... [More]
At a recent POWER & AIX Workshop, I was challenged when I stated that you can download IBM Systems Director (ISD) and run it at no costs. Granted: If you want IBM Support for ISD functions then you do need to purchase the license and SWMA for every endpoint (think HMC and every core of any machine that ISD is controlling). Compared to other systems management tools (I have worked with in the past) the ISD prices are very low so its worth asking for the local currency price. I also think most people (including myself) will get stuck on... [More]
It has bee a long few weeks traveling to Dublin, Las Vegas, Slovenia, Germany and Sweden to give lots of presentations. Six Weeks and five countries and the expenses are piling up on my desk. I was presenting a core set of topics like POWER7+ announcement summary, VIOS Shared Storage Pools (SSP3), Workload Partitions (WPAR) and Versioned WPARs to run AIX 5.2 and AIX 5.3, quick introduction to PowerSC (security tools), a Techie "what you need to know about" Systems Director for AIX, and Quick Dips in to Active Memory Deduplication... [More]
I suddenly could not get AIX updates to work via my NIM server. The problem number DNZPAX075E just means the NIM command failed for unknown reasons. Not very helpful when you have a ISD Server, NIM Server and a Target AIX Server and the problem could be anywhere. I looked around and found a few other people were stuck on this one. So now I worked it out - with the help excellent help of systems Director support. 1) Restart the Systems Director Server and the NIM Server - just good house keeping and check you have free disk space in the... [More]
This is part 2 of Systems Director licensing or at least working out what you are doing with System Director that requires a license. In part 1, we looked at listing (via the command line) the POWER machines that Systems Director was connected too. This time we what to find out the machines that Active Energy Manager (AEM) is controlling. Once you have installed AEM it is tempting to start monitoring and controlling all your Power Systems electrical use and temperatures but you should only use AEM if you have a license for AEM for each... [More]
I have long thought it is rather silly that IBM Systems Director can't give you a list the licenses that you should have! It does, after all, know what machines it is controlling and the data is in the database. Perhaps, that will come in the future. What can we do in the short term? Generally, I like to do everything from the graphical user interface (GUI) . I have not got time to learn and use both GUI and CLI. The system configuration data can be extracted using the command line interface (CLI) on the Systems Director server, so I spent... [More]
Stephen Atkins (the Guru behind the nmon Analyser) has released a new version - two days ago. This includes loads of improvements and some new features. Best of all - less problems running on newer Excel releases (it works around inconsistencies with the Microsoft API). I have always said that at least 50% of the popularity of my nmon is down to the excellent Analyser graphs. Very few people regularly see the online screen view. Thanks again Steve, for all your many hours of hard work in your personal time on the behalf of the tens of... [More]
Yes, I admit it I am a "hacker" in the old sense of the word i.e. Definition 4 = A person who is good at programming quickly.See Hackers Dictionary below. I was think that there are certain books every one in the computer industry should have read before they can claim any level of computer expertise. If you have not read them don't tell any one and quickly get yourself a copy and read it quickly. Thumbnails are borrowed from Amazon - thanks guys and good to see all these important books are still available. My five top books all... [More]
I have just updated my article on DeveloperWorks that cover the nweb Webserver. You can find it here: nweb: a tiny, safe Web server (static pages only) The what? I wrote the nweb Webserver as example C code to show the concepts and details of how they work. It was under 100 lines but I thought it actually was a pretty useful tool and used it, for real, in a number of projects and others have used it too. Then I though I had better put in some safety checks - otherwise you could give away read access to ALL the files on your AIX, UNIX or... [More]
I have never really understood why I get peaks and troughs in nmon questions and urgent situations but July seems to be a peak and fortunately I was not at work for a large part of it. Performance and nmon Questions trend to come in three flavours: really dumb, genuine and mega-urgent critical lets blame the hardware. Let me give you a flavour of each type of question from the last month. Perhaps, reading these will let you avoid the same problem or at least let you learn "you are not alone". Really dumb "nmon has stopped... [More]
Well, I'm back from a 2.5 week holiday and there is a heap of topics that have built up in that time. My email is very full of things I need to share so expect a blog per day for a while. My friend for the last couple of decades Michael Perzl in Germany has been very busy in the last month or two. Lots of people that are using AIX every day also want to make use of the popular open source packages. Some of these come as part of AIX (or on extension packs for legal reasons) like Perl, ssh and the basic IBM HTTP server (Apache) part of WAS.... [More]
I highly recommend the AIX Virtual User Group run by Joe Armstrong that roughly has a hot topic Webinar session once a month. Typically, there is 300 to 700 online to the session, many include a demo and there are Slides and Replay downloadable if you miss one or look back over the previous 70 or more sessions and want an update. On the larger hot topics there are lots of questions and answers handled during the session. Officially, it is called the USA AIX Virtual User group but they take registrations from around the world but schedule for... [More]
So I got asked, just as an example configuration which forces lots of workload per CPU: Given a 16 CPU POWER machine and a need to run 100 workloads, would I recommend 100 LPARs or 100 WPARs? In case you are not familiar with POWER technology: LPAR = Logical Partition used the PowerVM to split CPU, memory and I/O across multiple virtual machines (also called virtual servers). The I/O is handles by a special purpose LPAR called the Virtual I/O Server, which provides virtual network, virtual disks, virtual optical and virtual tape. Each LPAR has... [More]
Earlier today I received email from a customer reporting their large POWER7 based machines where on firmware 720_64 to 720_90 and their reluctance to take the outage to upgrade it. They were asking for fine details of newer firmware levels and what advantages this would bring to " justify the outage to their user departments ". To be blunt this is a horror story: I lay awake at night in a cold sweat about stories like this. The customer has the whole "running computers plan upside down". The question should be " can... [More]
I got another WPAR question today: From the Global AIX, I can add a filesystem to /wpars/WPARname/directory so the WPAR has access but what if I don't want to have the filesystem mounted there in the Global AIX? For example: I have a Global AIX filesystem called /tools that I want to mount in to one or many WPARs and appear and /temp in the WPAR(s)? My first answer is well you can use NFS because I do that all the time and I am a heavy user of NFS - no jokes please. I use NFS heavily! If the NFS server exports the mount point for every... [More]
This week I spent 4 hours with a customer covering many advanced WPAR topics and took way a bunch of questions that I had to check the answers and ask the WPAR developers themselves to be sure I had the right answers. If the questions were not clear to my customers and I did know initially know the answers then there may be others with similar issues so I thought I would share the answers with everyone. 1) Workload Partitions - What is the road map? Well, I am not going to make announcements on a blog for sure. If you are new to WPAR, I can... [More]
Recently, I delivered a 7.5 hour Systems Director 6.3 Instructor lead demo at IBM Zurich - that gives us 25 more people ready to dive into using it. It is a very polished building and the IBM Forum is amazing and they even have IBM food :-) I had to take a second mortgage out on my house to pay for the taxi to the airport (fortunately they take a credit card) and I thought Norway was expensive until now! Anyway, when I got back to the UK, I decided to perform the Systems Director 6.3 to 6.3.1 update. You get Systems Director to update... [More]
It was my birthday on Saturday and by pure luck my Raspberry Pi was delivered the day before - what a great day "hacking" (in the old sense of the word). I have been waiting for a few months for this delivery - What a treat!! If you have not heard of this credit size computer before here is the back ground. The Problem In the UK, school information technology education has been pitifully reduced to Microsoft Word, Excel and Powerpoint. My own kids came home badly depressed as they started the classes as they knew far more than the... [More]
Systems Director 6.3.1 is available This includes a fix I needed. On a completely fresh install the Update Manager will not work with the NIM server for AIX updates due to a bug that forgets to create /export/um_lpp_source on the NIM server - oops! Fixed with: /usr/sbin/crfs -v jfs2 -g rootvg -m /export/um_lpp_source -a size=1G But make sure it mounts by default on AIX reboot. Find the What's New for 6.3.1 Install the new ISD 6.3.1 update and lots of fixes - you get Systems Director to do upgrade as normal
Warning: touchy feely comment coming up! I have always found it odd that IBM markets the Hardware = POWER7 a lot and very little marketing on the Operating System = AIX7. Personally,
I have more good feelings and positive vibes for AIX operating system
because I log in to every day and use it for hours than I do for the
POWER hardware running it. I very rarely go to the machine room and hug the machine! Does anyone else feel this way or am I sadly alone? Put
it another way would you spend two Dollar Euro Pounds (all increasingly
I have been touring cities in Europe: Birmingham for the 4 day Technical University, Manchester, central London and Dublin for the Ask the Experts 1 day event and last week Copenhagen for a POWER & AIX User group Technical Event. I have had about 250 to 300 people in my sessions and asked every time - How knows about the AIX Virtual User Group and only had about 30 people put their hands up. I am amazed how few people have found this invaluable resource. The organiser, Joe Armstrong an IBMer, set-ups up roughly monthly world wide... [More]
It has been a busy fortnight and I just realised I have micro-blogged (Twitter) comments and link but neglected this AIXpert Blog - sorry about that. Here is a summary and a lot more thoughts that Twitter allows. I actually like the discipline of the Twitter 140 characters as if forces you to get to the point!
Firefox First, I have upgraded to Firefox 10 Enterprise from 3.6 - I was having multiple problems with the older release that had previously seemed to be the only safe one for using both HMC and IBM Systems Director. In particular,... [More]
I may have written about these in previous blogs but these have been updated and added too them. They are in my regular Systems Director presentation but I thought they should be shared to a wider audience - no sense is sitting on good information. These are very much my thoughts and opinions for POWER and mostly AIX systems. These are not official IBM statements. 1) Quick Introduction . If you want a 12 minute look at how Systems Director can benefit Power and AIX system administrators in eight ways then watch my 12 minute movie. After... [More]
Many countries switch to Daylight Saving Time (DST) next weekend March 25th - unfortunately you may need to apply a fix to AIX 6 and AIX 7 before the change or programs can get inconsistent time from various time function and systems calls. This could have a major impact on many tools, for example, make, logs and databases ... well just about anything that uses a time value. There was an earlier temporary fix available but now there is a fully details final fix. Please got to the bellow website for all the details :... [More]
As part of my moving computer room project, I had to change the IP address of my Systems Director server. I recall a long time ago there were issues in this area for (I think) ISD 6.1, so I was not looking forward to this change. Just for a change, I thought I would check the documentation first :-) Once I found the 6.3 part of the documentation there was a whole page on how to do this IP address change. It turns out to be very simple on ISD 6.3. Stop Systems Director with: smstop Change the IP address and DNS server, of course, to match.... [More]
We have all heard the old story of a computer company that was so successful that is had to move to smaller premises! The point being that they upgraded the computers to much smaller but more powerful ones and did not need the floor space. Well, it is happening to me! My Advanced Technical Support group are moving from IBM Bedfont Lakes (near Heathrow), UK from building 2 to building 3. That might not sound like much and it is only a stones throw away but we are going from 10 racks down to 3 racks. The old POWER5 machines that were not... [More]
Hi, I just release a fifth hands-on movie this month and on this interesting topic. You can find the movie here: Shared Storage Pools 2 - Thin provisioning, monitoring free space + Alerts Shared Storage Pool Thin provisioning is pretty cool and saves a lot of disk space . Effectively the 100's of GBs of unused disks space in 100's of Virtual Machines (LPARs) are bought together in one place in the Shared Storage Pool and then can be used for real. All this without the use of clever disk subsystems. The risk is that you use all the disk... [More]
In case you missed the announcements, I have released four new movies this week: Shared Storage Pools phase 2 - Getting Started - movie 101 Active Memory Deduplication Theory - movie 102 Active System Optimizer - movie 103 System Director 6.3 First Use - movie 104 Catch them all at the home of POWER6, POWER7, PowerVM, Systems Director and AIX Movies - or save this link for 21 hours of intensive techie learning - http://tinyurl.com/AIXmovies ps: The previous movie last month about installing Systems Director 6.3 on AIX is a good movie... [More]
At last I have got round to rebuilding my Shared Storage Pool cluster of machines. We have the four VIOS limit. So I was thinking: For machines with Dual Virtual I/O Servers - the options are limited so I would pair up machines for mutual Live Partition Mobility. For machines with single VIOSs - I would create a cluster of four machines for some extra flexibility and LPM across any of the machines. The "cluster -create" command to make the first node of the cluster too about 3 minutes and the "cluster -addnode" command... [More]
As we all know, the advanced PowerVM virtualisation functions of POWER7 based machines are implemented in the hardware (the POWER7 processor itself and the System Firmware, which includes PowerVM Hypervisor) for increased security, high performance and reliability. After more than a year of POWER7 field experience and fixes to increase RAS, further development of still more virtualisation functions and improved algorithms for higher performance - it is very important for all of us to install the newer levels of System firmware to avoid already... [More]
I have been running the nmon ( n igel's mon itor) for AIX project for 15 years now but I am still learning. I initially wrote it to help me run AIX 3.2 then AIX 4 benchmarks on RS/6000 machines as running the older "down the screen" and "line by line" tools like vmstat and iostat was driving me mad (literally making my eye hurt) and so using screen curses enabled the number to stop moving on the screen. nmon's second most important feature was to make graphing the results simpler as a graph includes the time dimension, it... [More]
I spent two days with a "well known brand" UK retailer trying out Versioned Workload Partitions (vWPAR). Actually, it was with an IBM Business Partner that runs the machines for them- two knowledgeable and fun guys to work with - we were asked by the European manager in the next room to keep the noise down! This Versioned WPAR technology lets you run AIX 5.2 or AIX 5.3 inside an AIX 7 on Power7 Workload Partition (WPAR). You simply get the source AIX up to the required level (which you should already by running), then take a mksysb... [More]
Update: The developers and the manuals call this Load Sharing but most people think it is called Load Balancing. Perhaps, balancing gives the wrong impression of fine grain packet by packet balancing where we actually have higher level, cruder splitting of the work with Sharing. Below I use the word Balancing but mean Sharing. I have got a few questions recently on how to set this up as there are announcement with near zero information on setup, the configuration needed and a worked example. So here goes. For a long time now we have had SEA... [More]
Seasonal Greeting. My family decided to give me a Kindle e-book - Wow! So after the usual experiments with reading a book, playing with the web browser via WiFi, I then thought; Could this help me at work too! So reading up I noted it fully supports PDF format files and the IBM Redbooks come in PDF. So a quick download over USB cable and I can read them easily enough in landscape mode. As part of Advanced Technical Support, I am expected to "know everything" and over the past 10 years I have read all the POWER4, 5, 6, 7 machine... [More]
Shared Storage Pools is the next big thing to owed systems admin time and boost flexibility for Power Systems, PowerVM and controlling SAN disks from the VIOS and HMC. I will not go on about SSP here but I particular like the fact Live Partition Mobility even with SAN disks is ready to go with SSP and there is no need to reZone the SAN disks. To start using the new and improved SSP, you need the latest Virtual I/O Server (VIOS) - the version name is a nightmare! Who ever thought VIOS 188.8.131.52 FP25 SP01 with iFix was a sensible name for a... [More]
The below URL takes you to the top of the Systems Director 6.3 online documentation http://publib.boulder.ibm.com/infocenter/director/pubs/topic/com.ibm.director.main.helps.doc/fqm0_main.html As a rule I use Firefox 3.6.24 all the time for my browsing as it works well for Systems Director and Hardware Management Console (HMC). Later Firefox versions have "variable" results which re frustrating and that just slow me down. The Firefox developer when mad and got to version 8 suddenly over what appeared to be a few weeks but have realised... [More]
So a long download of about 7 hours for the 1.9 GB but a reasonably quick install. Fresh AIX 6.1 TL6 and service pack 6 installed. See my Update below for a problem with this AIX level. Prepared a /database filesystem for the DB2 database. The Pre-Install Utility told me to fix some file system sizes, more paging space and ulimit - good stuff. The Installer worked fine. The CIM Server install seems to take for ages at the start! It then refreshes the Common Agent, installed the default "managed" DB2 database invisibly behind the... [More]
IBM Systems Director 6.3 Arrives Early Yes, you read that right - it arrived earlier than the date in the IBM Announcement - a free to use unexpected Xmas present from IBM. I have never heard of an early delivery before - perhaps I am getting to old and cynical :-) The Base Server download for AIX is a larger 1.9 GB file (was 1.3 GB for 6.2) but that includes includes AEM,VMControl (and Service and support manager but we probably don't need that as we have the function in Power Systems via the HMC). They all get installed in one go but you... [More]
I thought I should summarise the long eleven part Local, Near & Far POWER7 Affinity series. 1) Placement: Find out the layout of your boxes CPU and RAM and if the RAM is evenly distributed across available DIMMs Find out the placement of your Virtual Machines (LPARs) with lssrad -av - or - topas -M 2) SMT4 : Expect POWER7 SMT4 CPU use to “look” different POWER5 & 6 have two equal threads POWER7 shuts down threads 3 & 4 and even thread 2 - when there is not enough processes running. 3) Entitlement : Only set minimum Entitlements,... [More]
At the Miami and Copenhagen Power Systems Technical Universities, I covered what we knew about the next release of Systems Director (6.3) planned for 16th December 2011. One of the hot topics was High Availability using disk replication for the embedded DB2 database for active:passive fail-over. It turns out this feature is not included with Systems Director 6.3 release - sorry about that. The good news, at least, is that this feature is in development and working for the next release of SDMC, which is based on Systems Director 6.3... [More]
Below is from your Power Systems, Advanced Technical Support team in Europe (EMEA) with lots of input from many people.
We find many people are over optimistic and making assumptions - which can catch them out - we learnt the hard way too.
Don't Assume 1 - The same boost as last time
It may be that you migrated from 2Mit to 10Mbit and saw approximately a 5 fold increase in performance.
Similarly when you went from 10Mbit to 100Mbit, and from 100Mbit to 1Gbit; you saw... [More]
I just realised that I got help from the AIX developers to explain the output of the mpstat -d command for my POWER7 and Affinity Technical University sessions in Miami and Copenhagen that I never passed on to every one. In an earlier blog we examined some mpstat columns but skipped others. The nearly undocumented mpstat stats are: S0rd, S1rd, S2rd, S3rd, S4rdand S5rd. In the AIX manuals it states: S0rd = The percentage of thread re-dispatches
within the scheduling affinity domain 0.
And likewise for the other numbers - but what does... [More]
I have been using Shared Storage Pools phase 2 (SSP2) on the beta test a lot recently and it works well - I am very impressed. One key side effect of SSP2 is that it really makes Live Partition Mobility (LPM) very simple and safe. But then I hit a problem that stumped me - nothing to do with the SSP2 technology - it was just that I in the Advanced Technical Support group have got rusty in setting up and using LPM. The first problem that most people hit is that the source and target Power machines must have the same LMB size - the what??... [More]
Active Memory Deduplication was announcement on 12th October and available on 14th October but there has been a little confusion on: what you need, how you install it and run it and how it works? This blog will attempt to demystify this interesting new Power Systems feature. This is one of many features to reduce the memory foot print and so reduce costs for POWER and AIX logical partitions. So far we have: Workload Partitions - reduce the memory used by many copies of AIX to just one Global copy and applications installed at the Global... [More]
The officials words I was sent: Starting on November 13, 2011, each IBM client accessing Fix Central (whether through their employees or other authorized representatives) will require an individual IBM ID to download fixes for a limited number of additional product areas (including AIX and VIOS). This requirement will be expanded to additional products in the future.This requirement will be expanded to additional products in the future. The registration process will be quick and simple and will provide users with a customized experience to... [More]
I have been wondering why the lower end POWER7 machines have local and far memory and not local and near. Perhaps you wondered too! Well at the Miami Power Technical University, I got to talk to Dr Joel Tendler (IBMer) and a POWER7 processor guru and put the question to him. He covered this sort of architectural topic in his presentation at the event and I learnt a lot in this area by listening to the "master". Below is some background and the explanation too. The POWER7 chips has two memory controllers for maximum performance but... [More]
Finally, I have the types of memory by POWER7 model double checked. I refreshed the table in part 1 but thought I should make sure every one sees the final version so I have not mislead people. Only the Power 770/780 and Power 797 have Near memory. All the smaller machine operate a Local memory meaning on the same same POWER7 chip and Far memory for any access to another POWER7 chip's memory.
The Hypervisor makes the virtual machine (LPAR) placement decisions and changes during DLPAR. The Hypervisor is part of the Power Systems Firmware. The POWER7 machines first arrived in the early months of 2010. Since then we have had a lot of field experience with the larger machines with large configurations with large applications and tens of thousands of processes running and in particularly the machines with all three layers of memory - which are the Power 770, Power 780 and Power 795. From this experience, the POWER developers have... [More]
After you have started and used your virtual machine (VM) for a while, you may decide to change its size using a Dynamic LPAR (DLPAR) change from the HMC (or SDMC or IVM, of course). This has virtual machine placement implications. When shrinking your VM, the hypervisor will decide which CPU or memory (LBM) to release and it might not select what you think is the obvious choice. When enlarging your VM, we might thing there is an obvious way to grow in a balanced way but we can't see: 1) where physically our virtual machine is placed in the... [More]
We all tend to concentrate on the CPU first and the memory second. CPUs, as the "brains" of the machine, does get a high focus and have a lot of extreme technology within it but the RAM is the "guts" of the machine to "feed" the CPU with nutrient data. OK, let us stop the analogy there :-) Along with reducing the number of CPUs via a lower virtual processor count, we also need to have the CPUs matching the memory - so AIX has a fighting change to localise a running process to its home SRAD and thus have it's data... [More]
The title should read "Local, Near & Far ..." - I will not correct it or links might fail. In this entry we carry on from part 5 but we are going to look at setting the virtual processor number for the virtual machine . There is a side effect that is not obvious and after 6 years of using them, it never occurred to me so perhaps it is news to others too. The problem of virtual processors is that they are ephemeral - i.e. they don't actually exist and costs nothing. So I find most systems administrators feel they can be generous... [More]
The title should read "Local, Near & Far ..." - I will not correct it or links might fail. With a shared processor virtual machine (I am calling this "VM" but was called LPAR!) there are various suggestions of setting Entitlement ("Desired processing units" on the LPAR profile on the HMC, I am calling this "E") and Virtual Processor numbers (I am calling this "VP"). For Capped, the Entitlement is the maximum guaranteed CPU time that you can't go over and you round up the Entitlement to the... [More]
This is a follow on to a previous blog on " PowerVM Virtual Ethernet Speed is often confused with VIOS, SEA IVE/HEA speed "- here is a Direct Link I regularly get asked : "I have just heard about this [PowerVM virtual switch | vSwitch | Hypervisor Ethernet Switch] is available but I can't find out any information at all, help!" They are wanting to run two (or more) virtual switches within the one Power machine to completely separate the network packets of groups of virtual machines (LPARs) so they know for sure there can... [More]