AIXpert Blog is about the AIX operating system from IBM running on POWER based machines called Power Systems and software related to it like IBM Systems Director, PowerVM for virtualisation and PowerSC for security plus performance monitoring and nmon
Yes, it does happen. When you want to use all the main unit (CEC) adapters slots for high speed adapters like Fibre Channel and the ROCE network and want to avoid adding the extra POWER8 I/O Drawers for just 1 or 2 adapters - but did not realises the ROCE adapters can't do a NIM install.
I think it is actually due to the basic boot code "PROM" not having the "smarts" to run the advanced adapters in a basic limp along mode to run the network for the bootp and tftpboot download before you... [More]
I was recently sent pictures and got questions about the handle colours on the Power System E870 / E880 CEC and System Control Unit.
On my early ship beta test machine, the adapter carriers have terracotta (red-ish) handles and on GA customer shipped machine blue .
Terracotta handles means customer hot-plug capable - but don't just grab and yank. You must to go through the HMC maintenance procedure at the very least for the adapter slot to be powered off to make it safe to be removed.
Blue handles means... [More]
I was asked how to get the inlet and exhaust temperature from a POWER8 machine and the amount of electricity it is using. This was once done with Systems Director and the Active Energy Manager (AEM) but I a told this does not work with POWER8 or if it does its just luck!
At first I thought it impossible but I asked friends in development and they pointed me to two documents that point you in the right direction
Here they are
IBM EnergyScale for POWER8 Systems
I had this combination recently and it drove me nuts.
Plug the Fibre Channel cable from the Storwize V7000 in to the Brocade FC Switch (IBM 2498-B40 = Brocade 5100) and the ports would not go green but flash orange on and off and end up in Port Status=Disabled and Healthy=Faulty on the switch and the V7000 in waiting for a cable state. Both the the V7000 and FC Switch are the latest firmware.
I assumed faulty GBIC at one or other end and/or a faulty FC cables at first but after trying many combinations and my two new... [More]
This Blog is brought to you by Clive Benjamin (Principal Engineer) who worked out how to do it.
If your HMC has a quality high resolution local screen or KVM screen then the Restricted Shell terminal window font size is far too small to read and the font used very narrow lines - in our humble opinion . This blog shows you how to fix this and "no", the obvious window Font Size option does not work.
1) Below: On the HMC start the Restricted Shell window
2) Below: With the cursor on the terminal window used... [More]
This is a story board about the mystical POWER8 Lift Tool and decommissioning our Power Systems Enterprise E870. The EMEA Advanced Technology Support machine was a temporary loan from the Power Development team in the USA as part of the Early Ship Program for beta testing, skills development, usability testing and feedback plus social media like the YouTube videos, Tweets and this blog. We have certain enjoyed learning about and using the machine although I must say it was hard work to use up all the 4 TB's of memory!!
IBM Systems Director and PowerVC make use of a private API to use Shared Storage Pool (SSP) clones of LU (like a LUN allocated from the pool). But there is no command to let you do that. LU Clones are quick and good for duplicating a master copy for small virtual machines to boot the same OS level and it means greatly reduced disk use as blocks are shared until written too.
As there is not VIOS command to clone there is no command to let you see what is cloned and what is not.
I tripped over this when trying to clean up my SSP... [More]
If you go looking then you can find IBM websites covering AIX that tell you about AIX 7 Express Edition being limited to LPARs (virtual machines) with a maximum of 4 CPUs and a maximum 32 GB or memory. The idea being a reduces price for small machines hosting small virtual machines.
You can also find the Announcement of the removal of these two restrictions for certain POWER7 models like Power 710, 720 and 730.
UPDATED: 30 JANUARY 2015
The IBM Official Technote can be found here http://www-01.ibm.com/support/docview.wss?uid=nas8N1020533
It clarifies the mean of the ASMI settings i.e. it is the number of adapter slots AND the specific slots AND a further bug only some settings work until the next firmware release.
Even with the newer firmware my recommendation stands:
Completely Switch OFF the I/O Adapter Enhanced Capacity feature as you install the machine as it needs a power off
- Unless you are running Linux on... [More]
If you follow this blog you will already be aware that we (Gareth Coates, Clive Benjamin, Mike Pearson, Jyoti Dodhia, Rachel Meager and the guys from Lab Servers that are running Oracle, WAS etc performance tests to hammer the machine) have had a Power Systems Enterprise E870 since early September 2014 as part of the Early Ship Program. This program gives access to new machines before they are releases and their matching software for testing, learning and skills development and feedback. It has been very interesting and we have shared what we... [More]
Most of the Power Systems range of servers since POWER7 allow for the HMC organised online hot replacement of:
and the online hot adding, removing or replacement of:
Internal hard disks
From POWER8 onwards we are not going to have the online add or remove CEC nor adding or removing Processors modules nor Memory cards. Of course online changes using Capacity Upgrade on Demand is still very much an option. On the POWER8, I/O Drawers we... [More]
People share nmon captured data files with me all the time because they would like me to review the data and make suggestions about the hot spot and tuning.
I guess they trust me :-)
Some user get a little paranoid about the data (mostly due to having to follow many procedures to get the nmon file through firewalls so it can be emailed to me) and ask questions about if it contains information they don't want to accidentally share. Thankfully the nmon data files is just a text file that you can edit with "vi" so there... [More]
I need to make it clear that these pictures are from an Early Ship Program machine which might vary a little from the final Generally Available (GA) models from November 2014. For example, missing labels or stickers plus colours of handles and parts etc. The engineering will be much the same.
Our machine is part of the Early Ship Program and we get to run hardware tests before machines are shipped to customers from GA. These test include things we would never expect customer to do -... [More]
I need to make it clear that these pictures are from an Early Ship Program machine which might vary a little from the final Generally Available (GA) models in November 2014. For example, missing labels or stickers plus colours of handles and parts etc. The engineering will be much the same.
Please see my 8 minute video of the machine on YouTube here:
POWER8 Enterprise - Power E870 First Look http://youtu.be/jOzPTopt7HE
These pictures were take by Gareth Coates and the... [More]
Yes this is still the AIX pert Blog but my beautiful POWER8 machines can also run that "odd ball" operating system now called IBM i. That should generate some comments :-) We have a project to install IBM i 7.2 and then adopt it into PowerVC for Capture and Deploy. This feature is in beta test and selected customers allowed to give it ago. It is planned for full release later this year (that is not an IBM promise).
If you are interested in testing PowerVC with IBM i visit:... [More]
Assuming you have dozens to hundreds of AIX VM's now running on your PowerVC controlled machines at some point you will have to upgrade the AIX version to keep current and ensure security.
PowerVC does not have OS upgrade functions but it does have features that can help make upgrades safe and simpler. The upgrading of AIX is already handled by tried and tested methods and to be blunt I am not convinced I want PowerVC to do it as I want to visually check the install worked and immediately address any issues to ensure... [More]
If while installing SUSE SLES 11.3 on your POWER8 machine from the regular DVD media, you get this on the console after 5 seconds after booting:
Welcome to SuSE Linux Enterprise 11!
Type "install" to start the YaST installer on this CD/DVD
Type "slp" to start the YaST install via network
Type "rescue" to start the rescue system on this CD/DVD
Welcome to yaboot version r22.8-r1190.SuSE... [More]
We have the new HMC 8.1 code stream to manage the POWER8 machines. It also manages the POWER6 and POWER7 machines but not POWER5.
I find the HMC Virtual Storage Management panel very useful and saves me time for assigning Shared Storage Pool (SSP) space to virtual machines (VM's) and automatically doing this for Dual VIO Servers without having to log-on to the VIOS servers at all. It is also good for assigning virtual DVDs to VMs to boot and install the various operating systems, AIX, RHEL, SLES, Fedora, Debian, etc.
RHEL (6.4 and 6.5) Servers show their PC heritage when you assign server style static IP addresses instead of home network DHCP controlled IP networks. [[That will generate a few comments!!]]
I find the /etc/resolv.conf file gets modified after a reboot and so DNS fails - this is a broken as designed feature. I would enjoy someone trying to defend this feature and why it does this for my benefit! [[That will generate a even more comments!!]]
I search the network and the comments are the... [More]
Many people in computing, we seem fixated on "percentages" when talking about CPU use - We need to fix that.
When a computer has an absolutely fixed number of CPUs on the time scale of a few years the percentages of the box makes a lot of sense.
But in a virtualised world we will have to change.
For 10 years, we can do Dynamic CPU changes to a LPAR (Virtual Machine).
Yesterdays 40% used is today's 80% because the admin guy added a couple of CPUs over night.
If we look at the percentages and see a... [More]
In the past with the POWER7 and POWER7+ based machines at the lower end like the Power710 to Power740 Models, users had to shut down the whole machine to add, remove or replace an adapter safely. In fact, some machines would power off as you removed the lid.
This is no longer the case with the whole POWER8 range including the Scale-Out Models like the S822 and S824. These machines do not have the blind swap cassettes which allow the adapters to be removed from the rear of the machine. So you must have the cable... [More]
Odd. I thought I have put this on this blog before but I can find it now so just in case ... here it is or here it is again.
The SSP4 can have mirrored failure group FC disks to handle adapter, FC cable, FC switch, entire FC disk sub-system, site or VIOS failure.
But there is still a single SSP Repository disk. Isn't this a single point of failure?
The answer is " No because you can quickly rebuild the contents of the Repository Disk after it has failed ."
You are not meant to know this... [More]
I was told that Fedora's latest build for the Power ppc64 processor would support SMT=8 , if you "err ... um" have machine with a processor that support that SMT=8 !!! This is not an announcement.
So I gave it ago - here is my report:
And I can't help thinking PowerVC would remove all this work, once the new version arrives that support my ... umm err! new machine!!
I can tell you know that this was a POWER8 S824... [More]
With Power Systems there are many ways to run your Operating System(s) and each has advantages and each has a different install process.
1 Native Headless Whole Machine Mode
Number of operating systems = 1
Operating Systems: AIX, IBM i, Linux
Console = RS232 Dumb terminal
Virtual Machines = none
Machine types = Blades, Power Systems or PowerLinux (only runs Linux, of course)
Can't make use of the Power hardware based virtualisation advantage.
This blog also applies to Power Servers running Linux on Power native on the machine.
While you can perform a text based Linux install on the dumb console terminal emulator (see previous AIXpert Blog about the serial cable that you need and the software) as a challenge to survive the Tab navigation, space selection, sometimes the cursor works and occasional Function Key show stopper (SUSE SLES 11) - I would recommend a VNC install.
First, you can get back to your desk to run VNC viewer client to complete the install and not... [More]
Normally I recommended to go virtual:
I would recommend anyone trying a Power System or PowerLinux system to run it with a Hardware Management Console (HMC) -now called a Power Management Console - or using Integrated Virtualisation Manager (IVM = like HMC and VIOS combination as the first LPAR / VM).
This way you are using the full power of the Power based machines to run LPARs / Virtual Machines and push higher utilisation and extra flexibility.
But if you must go commando!
But some what to ignore... [More]
Inspired by this TED Talks Video on Leadership
TED Talk by Simon Sinek
Topic: How Great Leaders Inspire Action
18 minutes long
It tells us to start with Why? To win hearts and minds.
And not explain What? or How? until the Why? is very clear.
I thought I would make an attempt:
We believe server sprawl means a future in which we drown in small computers taking 80% of the... [More]
If you have:
High-end Power 770, Power780 or Power 795 machines
With Capacity Upgrade on Demand (CUoD)
More than 4 CPU cores and 32 GB RAM which are not Activated (dark)
... then you are ready for Power Integrated Facility for Linux (IFL) - which is a offers a package of 4 cores and 32 GB of memory at a special low price (given the resources, you can order more than one IFL)
includes the PowerVM license and PowerVM SWMA for the newly activated CPU cores
excludes a Linux license... [More]
Please provide comments on this blog entry - I want to know where I have it wrong?
With IBM's massive investment in Linux on Power my brain hurts trying to express the advantages. This is because the technology is excellent but the bits you want depend on where you are coming from. So I have tried to break out potential users in to various types.
For the record I am an Ab Linux on POWER User type!
A - Current AIX and POWER user
Already understands the POWER infrastructure, AIX and/or IBM i and are adding... [More]
I have been installing beta copies of PowerVC, the new IBM product which is available tomorrow 6th Dec 2013. It is quick to install and allows rapid rolling out of new virtual machines to a group of physical Power machines from your list of pre-packaged software images in just a few seconds of your time and roughly 2 to 3 minutes elapse time till you can login and use it. PowerVC is based on OpenStack which has lots of modules to install and configure. It is installed on Red Hat Enterprise Linux RHEL 6.4 on Power or x86 - So it is a... [More]
*** STOP PRESS *** - See the update in red at the bottom.
I was asked this lots of times at the recent POWER Technical Universities in Orlando and Athens.
Compared to a low numbers of local disks
The LUNs on a SAN disk subsystem with caching are very fast - even a simple test like AIX install times have halved. In addition, you don't have to fiddle about with stripping AIX Logical Volumes across disks or retro fit this to rootvg Volume Group filesystems after the install.
Compared to a... [More]
IBM announced lots of Power Systems software yesterday but the official announcements are pretty thin on the details. I have many Twitter tweets asking "What the heck does that all really mean?" and the official statements are marketing messages and not technical information like:
How do you get that working?
What are the pre-reqs?
What are the actual commands?
What does the Graphical User Interface look like?
How does that help me as a System Admin person?
I regularly get asked to run a demonstration for IBM Systems Director but that is a time consuming business for both me and the audience. Especially, when you can find out every thing that I know from the comfort of you own office or even home office plus Stop, Replay and Pause via YouTube videos. So here is a reminder and hopefully a page I can point to when I next gets asked.
ISD63 Workshop 1 Techie Intro... [More]
The latest Versioned Workload Partition (VWPAR) Version 1.1.2 is available but largely secret!
With this update I get on my AIX 7 virtual machine, in which I have installed both AIX5.2 and AIX 5.3 support:
# lslpp -L | grep -i vwpar
vwpar.52.rte 188.8.131.52 C F AIX 5.2 Versioned WPAR Runtime
vwpar.images.52 184.108.40.206 C F Versioned WPAR 5.2 Support
vwpar.images.53 220.127.116.11 C F Versioned WPAR 5.3 Support
vwpar.images.base 18.104.22.168 C F ... [More]
Most AIX system administrators use: smitty tcpip to add a hostname, IP address, network mask, gateway and DNS server setting to get the virtual machines (LPAR) and AIX on a network. And as a side benefit so we don't need to use the ghastly VTERM console any more as ssh or the dreaded unsecure telnet (just don't do it) can now be used - as the root user:
# smitty tcpip
Minimum Configuration & Startup
Use DHCP for TCPIP Configuration & Startup
When I log on to my home machine being AIX,of course, and type hostname I get:
nag@blue:/home/nag$ hostname -s
I get this because when I set the hostname in smitty it looks like this:
* HOSTNAME [ blue.aixncc.uk.ibm.com ]
* Internet ADDRESS (dotted decimal) [22.214.171.124]
Network MASK (dotted decimal) ... [More]
I had better point out that any performance tool of AXI command in a POWER Virtual Machine (LPAR) can only see its use of the Physical CPU, the size of its own CPU pool and (if Monitoring is switched on at the HMC), it has a number of the CPU use by "Other LPARs" use of its own CPU Pool. nmon gets these numbers by default and the nmon analyser graphs this data in the LPAR tab - this assumes its a Shared CPU LPAR. If it is a dedicated CPU LPAR then it can't get any of this Pool data and there is no LPAR tab in the nmon... [More]
I regularly get asked a question like: I have 4 months of data from 25 machines and have to develop a Capacity Planning model to size these LPARs on to new machines but I am having problems with having so much data. What can you recommend? We need graphs of
CPU compared to Entitlement
CPU Physical CPU use
Maximum real memory use
Sometimes this data is needed as input in to the Workload Estimator tool or Server Consolidation tools.
The answer is "it depends" and there is no Rule Of Thumb - IMHO.
The 2 times ROT is very old school and for small servers where peaks could really run out of RAM fast and hurt.
The worst I saw was 6 times but that was OS guys and RDBMS guys and app guys all said 2 times ... each!! and then the System Admin guy just added them up not realising it was the same 2 times! Basically, bonkers and they used 2% in the end.
With machines with large memory, say starting at 64 GB, the memory demand... [More]
When you install Linux on an IBM POWER processor machine (Red Hat, SUSE or now includes Fedora), you then need to add the IBM supplied additional RPM packages. These give you a whole bunch of useful tools like HMC comms so you can Dynamic LPAR change and LPM, diagnostic tools, and configuration listing commands. Plus more importantly nmon :-) The repository allows you to update your packages as new versions become available and to benefit from fixes and additional functions. Without these you don't get the full Power... [More]
The fifth POWER Ask the Experts is a one day customer technical event in the UK which ran in Manchester and London on 15th/16th July 2013. It proved to be a very popular free event. We had well over a 100 people attend which was a mixture of customers, a few Business partners and some IBMers.
10:00 - Power Systems Update - Pat O'Rourke: Austin Briefing Centre, USA Download 22MB .pdf
11:15 - Performance Best Practices with POWER7 - Nigel Griffiths, ATS Europe ... [More]
I have tried to get permission to pass on these numbers publically but got no answer at all. As these are facts rather than opinions, I see no harm in them. So I will work on the old theory: "it is easier to ask for forgiveness than permission" ... here goes.
The AIX Virtual User Group often asks a few questions as part of the registration process for the next webinar session and it generates fascinating statistics from real POWER and AIX users.
The people that join the AIX Virtual User Group are... [More]
Following on from my previous AIXpert blog on 1st July 2013 - "Core files filling important filesystems? Want email alerts about each core dump?" it was pointed out to me by AIX guru Mathew Accapadi that we can for many years use the ODM and AIX error reporting framework to send emails on the generation of core files. See the previous blog on how to force the core files to a particular directory - and not allow them to fill up important system directories. On the alternative reporting method see below:
Check out the... [More]
I got asked these questions recently and had to go look the subject up ... again! I seem to have forgotten some of the details and then I thought I would use some new features of AIX for the second part. In the distant past there was various way to stop core files being dumped in to the current working directory of the program that failed. In AIX 5.3, AIX 6 and 7, the "chcore" command does all the hard work for us by letting us
Choose a specific directory for core files - which is best in it own filesystem so it... [More]
A very nice chart from Richard Milton of IBM on the minimum levels for AIX and VIOS of Power7 and Power7+ hardware.
Download as a PDF: AIX_Levels_on_POWER7_Hardware_2013_06_20.pdf
We will try to update this as new machines arrive.
I have been looking at some nmon data from an IBMer looking into a customers machine. The virtual machine (LPAR) is running the Oracle RDBMS with 75 dedicated CPUs - on a POWER7 Power 795 at 4 GHz.
The Excel spreadsheet is quite scary at 144 MB thank goodness my Thinkpad has a SSD but not the record by a long way. Too many data save points can ready add u and so too can lots of processes if you are saving process data. Also with 75 CPU and SMT=4 that is 300 Tabs for Logical CPU, 300 for PCPU and 300 for SCPU and that... [More]
In Advanced Technology Support (ATS), we like to think we know what we are doing. Then something catches us out and we think that we had better let every other Power Systems administrators know about this or there are going to be many horrified users out there and many complaints. Note this problem hit our combination of HMC, VIOS and machines - you might get lucky but don't risk it!
Gareth Coates (my ATS partner) volunteered to perform the regular 6 month update of our systems but hit a BIG problem. He writes:... [More]
This is a typical request from IBMers in services or direct from customers wanting to confirm a good disk setup or confirm disks deliver as promised. This time "... a test of SAN disk performance where LUN's are carved from a IBM Flash System 820 with SVC. Primary objective is to show the IBM Flash System 820 can do hundreds of thousands IOPS with response time <1ms with good throughput. This is for an Oracle RDBMS using ASM"
Wow!! those new Flash Systems are really the next generation of disks. Well disk testing is not... [More]
I am getting asked more about security these days than in the previous decade - perhaps there is renewed focus on this area as we hear about hacker groups taking what they want from organisations and companies in the News every day. Below is an answer I gave to a recent email question on what AIX offers on the security side, spotting vital files being changed and what parts of PowerSC to use for quick wins.
For AIX security the three big hitters for "out of the box" AIX hardening are:
Ditch telnet/ftp in the... [More]
Your first reaction might be "Why would you do that, Nigel?" and "Isn't this a AIX blog: Why are we talking about Linux?"
Well AIX is great and all that but when it comes to Open Source it is not Linux. Like it or not most Open Source stuff is developed on Linux and there are readily available binary compiled for Linux and 100's of Open Source packages actually come with the the distribution on their DVD media and arch repositories on the Internet for rapid and straight forward install and updates.
Having just done this and getting questions on it from customers, I thought I should share a few notes:
1) The process is nicely documented and you need to follow the instructions. I recommend using the Release Notes as a check list as missing out a stop causes the next bit for complain or fail. Fortunately, the recovery is just doing the bit I missed.
2) Read the Release notes - even better print them out and cross them off as you go:
http://www.ibm.com/support/docview.wss?uid=hpc1vios117f5701 - upgrade... [More]
If you read this blog regularly then you will know I am a fan of the VIOS Shared Storage Pool technology .
It is a major peice of the "Making PowerVM and VIOS - easier to live with and reduce the man-power" story and it is hard to believe that we started withPOWER4 virtualisation in 2000 and VIOS in 2005. This does mean that the tools we use (HMC and VIOS padmin) need a bit of an update and I know these are in plans for the future. Even now we can start using SSP to save lots of system admin time and be more flexibly... [More]
Recently I was running the "lsmap -l" command and it output error messages that were less then useless at the end of the output:
Command did not complete.
"Determine if backing device is a PV or LV or optical." was last subcommand run.
Command did not complete.
Determining device type failed. Command did not complete.
"Determine if backing device is a PV or LV or optical." was last subcommand run.
Command did not complete.
Determining device type failed. Command did not complete.
"Determine if... [More]
Installing Open Source on AIX systems can be a little .... um .... frustrating and I have just been round that loop with installing Ganglia the excellent Performance Monitoring tool to demonstrate it and actually use it too on my own servers. The hard bit is that for every Open Source program, that you want to install, there is 10 to 20 prerequisite packages that you need to install first and you go round this the loop four times as the prerequisites have prerequisites .... This has got known as the "RPM dependencies form... [More]
POWER Relative Performance (rPerf) is often used as a way to approximate the expected difference in performance between two Power Systems servers. Although rPerf is a useful tool, it is important to understand the limitations of using rPerf to provide an estimate the performance of your specific workloads in your particular environment with a new server, First, rPerf numbers, like any published benchmark, represent the best case result when the application, configuration and system resources are all optimized -- factors that are likely not... [More]
Questions I have a number of customers asking me these types question:
Are there are good practices on how to size disks for the storage pools. If I want a 5 TB pool, how many disks which size, does it matter ? Also are there any document limits regarding, pool disk, LUN and so on. Answers Not sure about best practice but there is common sense - will that do? Restrictions are included with the VIOS announcement "Read Me" notes, which you can find here for the current VIOS 126.96.36.199... [More]
Shared Storage Pools - just got even more interesting ! SSP Clusters with disks pools and super fast disk allocation VIOS Shared Storage Pools (phase 3) allows 16 VIOS on different machines to operate as a VIOS cluster with a set of SAN LUNs in the pool - think in terms of a number of TBs of disk space in the pool. The VIOS systems administrator can then allocate disk space to a new or existing Virtual Machine (LPAR) in around a second which can be thin or thick provisioning regardless of the underlying disks. This drastically reduces the... [More]
I HAVE UPDATED THIS BLOG WITH BETTER, UP TO DATE AND CORRECTED INFORMATION FROM THE DEVELOPERS Read the comments for lots of useful information - Thanks Guys for the corrections WARNING: Advance Technology Support (ATS) Europe is getting reports today from users and IBMers that the POWER7 firmware 770 level as a pre-req check missing! could be much better . This results in the HMC agreeing to upgrade your POWER7 or POWER7+ machine to firmware level 770, when it is not supported and does not work. This causes very bad problems: On smaller... [More]
Just completed the move of the POWER and AIX Hands-on Movies to YouTube videos - after pruning we have 97 of them. This means you can watch them on your mobile Smartphone and Tablets in addition to your workstation/personal computer. You can find them all here:
YouTube look for my "Nigel Griffiths" Channel http://www.youtube.com/user/nigelargriffiths This should mean they are simpler to find, search for and view. During the transfer I pruned out a few of the older ones particular Systems 6.2 as we now have a set of Systems... [More]
Fourteen new movies/Video's on System Director on YouTube I have just last week uploaded 14 new System Director movies that were previously a whole day Technical University session and I thought it a good idea to share these with every body. It comes to about 4 hours in total and it a mixture of sizes from 7 minutes to 29 minutes. The focus is on core Systems Director function rather than advanced function plug-ins. It worries me a little that we often try to talk about and sell Systems Director for the advanced stuff but you really need to get... [More]
Our Advanced Technology Support, Europe team have had a few months secretly testing the Power 750/760 internally to IBM as part of the normal Early Ship Program which lets a few machine out of the development labs to be tested by others to gain feedback on the product, documentation, features and usability. I thought others would like to see a few pictures of the team and the equipment involved. Please note: This is an Early Ship Machine, so there are a few details like stickers and handle colours that may differ slightly from the final product... [More]
Hot news this morning is IBM announcing the updating of the Lower End Power 710/730 and Power 720/740 models to include Power7+ processor with 2.5 larger L3 cache, Active Memory Expansion accelerator, and GHz boost Double the memory size This gives a large performance boost and drastically higher "bangs per buck" as the Americans say or "Value for money" as we Brits say. Almost everything else is the same so if you know the older models it is easy to understand Announcement letters for the new Power7+ models: Power 710 ,... [More]
Another good questions from a customer and my answer was simple: The PowerVM internal network on POWER machines operates just like a physical network. If two LPAR are on the same internal network and the root user puts the virtual adapter in to promiscuous mode then it can see all packets. Just like a physical network. If the two LPARs are on different internal networks then they can never see the other LPARs packets. Just like two physical networks. If two LPAR are on the same internal network but on different VLANs then they can never see... [More]
Hi, Rob McNelly, the esteemed POWER7 Champion, sent me a really good questions and I thought would make a good blog as I get asked this about once a month. I have cut down the questions a but but it is a two parts: His customer is thinking about putting Systems Director on to their NIM server. Rob remembered me commenting on this but wants the details. They are planning to give it 1 dedicated Power7 core and 12G memory. What do I think about that? Answer 1: 2 years ago this combination (NIM & ISD) was not allowed ( I think it was just not... [More]
The new VIOS Shared Storage Pools 3 (SSP3) arrived a month ago and I have all my VIOS updated now to VIOS 188.8.131.52 plus fixes. This brings with it the regular VIOS improvements but also new features for SSP3 including: Better pool use stats for example see the below lssp command and the Used percentage and Unused(mb) columns and the over view output too has new data $ lssp -clustername stars -sp atlantic -bd Lu Name Size(mb) ProvisionType %Used Unused(mb) Lu Udid orange7a 32768 THIN 9% 29615 7d58538152 …... [More]
Time to correct a few assumptions and statements that I made a while back. Previously I said the POWER Hypervisor decides where to put a Virtual Machine (VM/LPAR) based on the Virtual Processor number (spreading factor). Well, apart from it nearly being right ... I was actually wrong! I got talking to one of these very impressive Hypervisor developers in Germany and he put me right. In most of my larger machines, I do what I think is fairly normal for creating an VM - like working with a CPU core to RAM ratio that my customers use, like 1... [More]
I have indirectly been getting copies of this 4 page, .PDF, once a quarter, Power Systems Newsletter written by Richard Milton (an IBMer in the USA) and every time found a few "Wow! That's Cool!" items in it. I just checked with Richard and he says anyone is free to sign up to the email list and you will get it emailed to you once a quarter. I have signed up. Needless to say, it is for Techies and not marketing fluff but has a couple of one page quick details of particular products and experience and comments on them. Then there... [More]
A large UK customer is running a series of tests on AIX and POWER7 machines and wanted to display the results on a webserver using rrdtool and needed help getting a working example running. So I stepped up to the challenge one night rather than it becoming a search for rrdtool skills inside IBM - when I could sort something out in a few hours. No one in IBM would claim to be a rrdtool expert until they knew the details of what was required. Many people know it well enough to get what they want done but would not put rrdtool guru on their CV. ... [More]
I am currently updating my AIX systems and AIX 7 TL2 with its first service pack is now available for download from ESS or IBM Entitlem Software Support to spell it out in full (http://www.ibm.com/servers/eserver/ess) for install media and FixCentral for updates (http://www-933.ibm.com/support/fixcentral/). While downloading the install media files I noticed there is a new Firefox for AIX available Firefox V3.6.25. This is a jump up from the previous one and I have had (in the past) a few annoying problems with Firefox on AIX Some weird issue... [More]
Four things I think you should know before you try this: The documentation for the update can be found here: http://www-01.ibm.com/support/docview.wss?uid=hpc1vios117f5701 I updated three of my Virtual I/O Servers to 184.108.40.206 from 220.127.116.11 over the weekend. I started these late on Friday night with the "no reboot" options and they finished about the same time on Sunday afternoon i.e. 48 hours . The Release Manager downloads the update directly to the VIOS /opt/VIOS/update/VIOS and sub directory for the release name and this takes over... [More]
Common question: I see XX% of Used Memory but I don't know who is using it and how?
A simple question but there is no simple answer as memory use if complex.
nmon then hit "m" will quickly show you a few big uses of memory as below:
The shared memory used by lots of applications like DB2 and Oracle - check the SEGSZ for the size.
The Owner column usually tells you what it is used for like the oracle user for the SGA or db2inst1 for the DB2 buffer... [More]
At a recent POWER & AIX Workshop, I was challenged when I stated that you can download IBM Systems Director (ISD) and run it at no costs. Granted: If you want IBM Support for ISD functions then you do need to purchase the license and SWMA for every endpoint (think HMC and every core of any machine that ISD is controlling). Compared to other systems management tools (I have worked with in the past) the ISD prices are very low so its worth asking for the local currency price. I also think most people (including myself) will get stuck on... [More]
It has bee a long few weeks traveling to Dublin, Las Vegas, Slovenia, Germany and Sweden to give lots of presentations. Six Weeks and five countries and the expenses are piling up on my desk. I was presenting a core set of topics like POWER7+ announcement summary, VIOS Shared Storage Pools (SSP3), Workload Partitions (WPAR) and Versioned WPARs to run AIX 5.2 and AIX 5.3, quick introduction to PowerSC (security tools), a Techie "what you need to know about" Systems Director for AIX, and Quick Dips in to Active Memory Deduplication... [More]
I suddenly could not get AIX updates to work via my NIM server. The problem number DNZPAX075E just means the NIM command failed for unknown reasons. Not very helpful when you have a ISD Server, NIM Server and a Target AIX Server and the problem could be anywhere. I looked around and found a few other people were stuck on this one. So now I worked it out - with the help excellent help of systems Director support. 1) Restart the Systems Director Server and the NIM Server - just good house keeping and check you have free disk space in the... [More]
This is part 2 of Systems Director licensing or at least working out what you are doing with System Director that requires a license. In part 1, we looked at listing (via the command line) the POWER machines that Systems Director was connected too. This time we what to find out the machines that Active Energy Manager (AEM) is controlling. Once you have installed AEM it is tempting to start monitoring and controlling all your Power Systems electrical use and temperatures but you should only use AEM if you have a license for AEM for each... [More]
I have long thought it is rather silly that IBM Systems Director can't give you a list the licenses that you should have! It does, after all, know what machines it is controlling and the data is in the database. Perhaps, that will come in the future. What can we do in the short term? Generally, I like to do everything from the graphical user interface (GUI) . I have not got time to learn and use both GUI and CLI. The system configuration data can be extracted using the command line interface (CLI) on the Systems Director server, so I spent... [More]
Stephen Atkins (the Guru behind the nmon Analyser) has released a new version - two days ago. This includes loads of improvements and some new features. Best of all - less problems running on newer Excel releases (it works around inconsistencies with the Microsoft API). I have always said that at least 50% of the popularity of my nmon is down to the excellent Analyser graphs. Very few people regularly see the online screen view. Thanks again Steve, for all your many hours of hard work in your personal time on the behalf of the tens of... [More]
Yes, I admit it I am a "hacker" in the old sense of the word i.e. Definition 4 = A person who is good at programming quickly.See Hackers Dictionary below. I was think that there are certain books every one in the computer industry should have read before they can claim any level of computer expertise. If you have not read them don't tell any one and quickly get yourself a copy and read it quickly. Thumbnails are borrowed from Amazon - thanks guys and good to see all these important books are still available. My five top books all... [More]
I have just updated my article on DeveloperWorks that cover the nweb Webserver. You can find it here: nweb: a tiny, safe Web server (static pages only) The what? I wrote the nweb Webserver as example C code to show the concepts and details of how they work. It was under 100 lines but I thought it actually was a pretty useful tool and used it, for real, in a number of projects and others have used it too. Then I though I had better put in some safety checks - otherwise you could give away read access to ALL the files on your AIX, UNIX or... [More]
I have never really understood why I get peaks and troughs in nmon questions and urgent situations but July seems to be a peak and fortunately I was not at work for a large part of it. Performance and nmon Questions trend to come in three flavours: really dumb, genuine and mega-urgent critical lets blame the hardware. Let me give you a flavour of each type of question from the last month. Perhaps, reading these will let you avoid the same problem or at least let you learn "you are not alone". Really dumb "nmon has stopped... [More]
Well, I'm back from a 2.5 week holiday and there is a heap of topics that have built up in that time. My email is very full of things I need to share so expect a blog per day for a while. My friend for the last couple of decades Michael Perzl in Germany has been very busy in the last month or two. Lots of people that are using AIX every day also want to make use of the popular open source packages. Some of these come as part of AIX (or on extension packs for legal reasons) like Perl, ssh and the basic IBM HTTP server (Apache) part of WAS.... [More]
I highly recommend the AIX Virtual User Group run by Joe Armstrong that roughly has a hot topic Webinar session once a month. Typically, there is 300 to 700 online to the session, many include a demo and there are Slides and Replay downloadable if you miss one or look back over the previous 70 or more sessions and want an update. On the larger hot topics there are lots of questions and answers handled during the session. Officially, it is called the USA AIX Virtual User group but they take registrations from around the world but schedule for... [More]
So I got asked, just as an example configuration which forces lots of workload per CPU: Given a 16 CPU POWER machine and a need to run 100 workloads, would I recommend 100 LPARs or 100 WPARs? In case you are not familiar with POWER technology: LPAR = Logical Partition used the PowerVM to split CPU, memory and I/O across multiple virtual machines (also called virtual servers). The I/O is handles by a special purpose LPAR called the Virtual I/O Server, which provides virtual network, virtual disks, virtual optical and virtual tape. Each LPAR has... [More]
Earlier today I received email from a customer reporting their large POWER7 based machines where on firmware 720_64 to 720_90 and their reluctance to take the outage to upgrade it. They were asking for fine details of newer firmware levels and what advantages this would bring to " justify the outage to their user departments ". To be blunt this is a horror story: I lay awake at night in a cold sweat about stories like this. The customer has the whole "running computers plan upside down". The question should be " can... [More]
I got another WPAR question today: From the Global AIX, I can add a filesystem to /wpars/WPARname/directory so the WPAR has access but what if I don't want to have the filesystem mounted there in the Global AIX? For example: I have a Global AIX filesystem called /tools that I want to mount in to one or many WPARs and appear and /temp in the WPAR(s)? My first answer is well you can use NFS because I do that all the time and I am a heavy user of NFS - no jokes please. I use NFS heavily! If the NFS server exports the mount point for every... [More]
This week I spent 4 hours with a customer covering many advanced WPAR topics and took way a bunch of questions that I had to check the answers and ask the WPAR developers themselves to be sure I had the right answers. If the questions were not clear to my customers and I did know initially know the answers then there may be others with similar issues so I thought I would share the answers with everyone. 1) Workload Partitions - What is the road map? Well, I am not going to make announcements on a blog for sure. If you are new to WPAR, I can... [More]
Recently, I delivered a 7.5 hour Systems Director 6.3 Instructor lead demo at IBM Zurich - that gives us 25 more people ready to dive into using it. It is a very polished building and the IBM Forum is amazing and they even have IBM food :-) I had to take a second mortgage out on my house to pay for the taxi to the airport (fortunately they take a credit card) and I thought Norway was expensive until now! Anyway, when I got back to the UK, I decided to perform the Systems Director 6.3 to 6.3.1 update. You get Systems Director to update... [More]
It was my birthday on Saturday and by pure luck my Raspberry Pi was delivered the day before - what a great day "hacking" (in the old sense of the word). I have been waiting for a few months for this delivery - What a treat!! If you have not heard of this credit size computer before here is the back ground. The Problem In the UK, school information technology education has been pitifully reduced to Microsoft Word, Excel and Powerpoint. My own kids came home badly depressed as they started the classes as they knew far more than the... [More]
Systems Director 6.3.1 is available This includes a fix I needed. On a completely fresh install the Update Manager will not work with the NIM server for AIX updates due to a bug that forgets to create /export/um_lpp_source on the NIM server - oops! Fixed with: /usr/sbin/crfs -v jfs2 -g rootvg -m /export/um_lpp_source -a size=1G But make sure it mounts by default on AIX reboot. Find the What's New for 6.3.1 Install the new ISD 6.3.1 update and lots of fixes - you get Systems Director to do upgrade as normal
Warning: touchy feely comment coming up! I have always found it odd that IBM markets the Hardware = POWER7 a lot and very little marketing on the Operating System = AIX7. Personally,
I have more good feelings and positive vibes for AIX operating system
because I log in to every day and use it for hours than I do for the
POWER hardware running it. I very rarely go to the machine room and hug the machine! Does anyone else feel this way or am I sadly alone? Put
it another way would you spend two Dollar Euro Pounds (all increasingly
I have been touring cities in Europe: Birmingham for the 4 day Technical University, Manchester, central London and Dublin for the Ask the Experts 1 day event and last week Copenhagen for a POWER & AIX User group Technical Event. I have had about 250 to 300 people in my sessions and asked every time - How knows about the AIX Virtual User Group and only had about 30 people put their hands up. I am amazed how few people have found this invaluable resource. The organiser, Joe Armstrong an IBMer, set-ups up roughly monthly world wide... [More]
It has been a busy fortnight and I just realised I have micro-blogged (Twitter) comments and link but neglected this AIXpert Blog - sorry about that. Here is a summary and a lot more thoughts that Twitter allows. I actually like the discipline of the Twitter 140 characters as if forces you to get to the point!
Firefox First, I have upgraded to Firefox 10 Enterprise from 3.6 - I was having multiple problems with the older release that had previously seemed to be the only safe one for using both HMC and IBM Systems Director. In particular,... [More]
I may have written about these in previous blogs but these have been updated and added too them. They are in my regular Systems Director presentation but I thought they should be shared to a wider audience - no sense is sitting on good information. These are very much my thoughts and opinions for POWER and mostly AIX systems. These are not official IBM statements. 1) Quick Introduction . If you want a 12 minute look at how Systems Director can benefit Power and AIX system administrators in eight ways then watch my 12 minute movie. After... [More]
Many countries switch to Daylight Saving Time (DST) next weekend March 25th - unfortunately you may need to apply a fix to AIX 6 and AIX 7 before the change or programs can get inconsistent time from various time function and systems calls. This could have a major impact on many tools, for example, make, logs and databases ... well just about anything that uses a time value. There was an earlier temporary fix available but now there is a fully details final fix. Please got to the bellow website for all the details :... [More]
As part of my moving computer room project, I had to change the IP address of my Systems Director server. I recall a long time ago there were issues in this area for (I think) ISD 6.1, so I was not looking forward to this change. Just for a change, I thought I would check the documentation first :-) Once I found the 6.3 part of the documentation there was a whole page on how to do this IP address change. It turns out to be very simple on ISD 6.3. Stop Systems Director with: smstop Change the IP address and DNS server, of course, to match.... [More]
We have all heard the old story of a computer company that was so successful that is had to move to smaller premises! The point being that they upgraded the computers to much smaller but more powerful ones and did not need the floor space. Well, it is happening to me! My Advanced Technical Support group are moving from IBM Bedfont Lakes (near Heathrow), UK from building 2 to building 3. That might not sound like much and it is only a stones throw away but we are going from 10 racks down to 3 racks. The old POWER5 machines that were not... [More]
Hi, I just release a fifth hands-on movie this month and on this interesting topic. You can find the movie here: Shared Storage Pools 2 - Thin provisioning, monitoring free space + Alerts Shared Storage Pool Thin provisioning is pretty cool and saves a lot of disk space . Effectively the 100's of GBs of unused disks space in 100's of Virtual Machines (LPARs) are bought together in one place in the Shared Storage Pool and then can be used for real. All this without the use of clever disk subsystems. The risk is that you use all the disk... [More]
In case you missed the announcements, I have released four new movies this week: Shared Storage Pools phase 2 - Getting Started - movie 101 Active Memory Deduplication Theory - movie 102 Active System Optimizer - movie 103 System Director 6.3 First Use - movie 104 Catch them all at the home of POWER6, POWER7, PowerVM, Systems Director and AIX Movies - or save this link for 21 hours of intensive techie learning - http://tinyurl.com/AIXmovies ps: The previous movie last month about installing Systems Director 6.3 on AIX is a good movie... [More]