AIX Down Under
AnthonyEnglish 270000RKFN Marcações:  interview valerie_skinner author aix aix_down_under anthony_english 2 Comentários 8.243 Visualizações
Interview for developerWorks
Valerie Skinner has kindly interviewed me for IBM developerWorks. If you want to find out more about the author of this blog than you ever cared to know, check it out.
AnthonyEnglish 270000RKFN Marcações:  syncvg san lvm synchronise disk logical_volume_manager varyonvg reducevg mirrrovg storage extendvg lv aix mirror unmirrorvg smit migratepv pavlova lslv mklvcopy lsvg lun smitty vg logical_volume rmlvcopy volume_group 3 Comentários 26.331 Visualizações
SERVING UP LVs, PVs and PAVs
Since we've got redundant arrays on SANs these days, it may seem almost quaint to speak about software mirroring using the AIX Logical Volume Manager. Even so, LVM is very useful when you want to move data around. If you need to move to a new storage subsystem or just to a new LUN, and you're not able to do it on the backend, the LVM may be just the ticket.
For example, supposing you are using a LUN that's a whole lot bigger than you need. There might be a lot of reasons how it came to that but the most common one is you may have slightly overestimated the amount of disk you needed when you first went with your begging bowl to the storage team. Admit it. You asked for thirty times more disk than you needed, just in case. And the reason you did that is because you never listened to your mum when you couldn't finish your pavlova ("pav" in Aussie-land). Don't you remember her telling you:
"Your eyes are bigger than your stomach"
Well now's your chance to
Make IT history!
and hand back some storage you don't need.
LV to new PV in same VG
First you allocate a new, leaner (smaller!) LUN to AIX then add it to the volume group using extendvg. (You may need to change its queue depth, preferred path, health check interval etc). Once it's a member of the VG, you can mirror at the logical volume level using mklvcopy.
You can mirror a whole volume group, (mirrorvg), and that's really the best way to do it with rootvg, because it has boot disks and dump devices which need special treatment. For other volume groups I often use mklvcopy because it allows me to mirror one logical volume at a time. You don't need to synchronise the two copies immediately (using the -k flag), but until you do, the lslv command (and the lsvg command) will show some partitions are stale. You can create the copies and wait for a quieter time to run the synchronise. It's a lot faster if the disks aren't busy, but it's perfectly legitimate to synchronise them while they're in use.
If you want to synchronise the two copies, use syncvg. You can synchronise the whole volume group using
syncvg -v VGNAME
The varyonvg command (which activates a volume group) will do the same thing, and you can run that command - varyonvg - even if the volume group is already active. With both varyonvg and syncvg, if there are no stale partitions to be synchronised, the shell prompt will come back in a jif.
If you want to synchronise a single logical volume, use
syncvg -l LVNAME
Not seven years' bad luck
Once you've synchronised the mirror to the new LUN, you can break the mirror to the old one, by using
rmlvcopy LVNAME 1 hdiskN
And you'll be stopped from running rmlvcopy if there will be only stale partitions left afterwards; you're not allowed to remove the last good copy of a physical partition. That's nice, isn't it? You also get a big warning if you try to remove a pv from a volume group when it still has any partitions on it.
The oft-quoted Chris Gibson has an article showing how he migrated to a new SAN using LVM. The same principle applies for a single LUN. There is also the migratepv command which is a simple way of moving everything off one pv to another. As with mirrorvg and mklvcopy, the target pv has to be a member of the volume group first.
These commands are so much fun that it's a shame to use SMIT, but you can do it that way if you want to.
Shock the SAN team
Once you've run rmlvcopy (or unmirrorvg which will remove your seven years of bad luck), you can remove the PV from the volume group (reducevg) before taking it out of the ODM using
rmdev -dl hdiskN
If you've removed the old giant LUN from the configuration all along the way (VG, ODM, VIO server), you can hand it back to your SAN team for them to recycle. Once they realise you're not playing some sort of practical joke, they'll be grateful. Shocked, but grateful. They may need the disk for someone else who didn't listen to his mum when she served up the pav.
AnthonyEnglish 270000RKFN Marcações:  vio lpar aix install virtual_user_group soe nim initial standard_operating_enviro... virtual_media tuning 8.213 Visualizações
We do have standards!
Having a Standard Operating Environment (SOE) for AIX has a lot of benefits. It's not just the consistency between images and some sort of version control over what you're rolling out. Using a SOE also saves time.
I have discussed building a SOE image for AIX before. And as you know, you could use the VIO server Virtual Media Library to do it (Not that again - doesn't this guy ever let up? - Ed).
This post covers some suggestions from some AIX big wigs on what they put into their SOE image.
Tuning and tools
First we start with Jaqui Lynch, whose articles regularly appear in the IBM Systems Magazine, AIX edition. (If you don't subscribe, DO!) Jaqui uses a short script to set tuning parameters, and then installs third-party software such as gcc, lsof, openssl / ssh and a number of tools from the Linux toolkit. You'll find more details in the article Installing and Upgrading to AIX 6.1. The article's from 2008, when 6.1 was still in nappies (diapers), but still applicable apart from a few small changes (nmon now is part of AIX and doesn't need to be installed separately). The SOE recommendations are in the section "Installing AIX 6.1 from Scratch."
Next comes this gem from Steve Knudson, who gave a couple of presentations on NIM. Steve builds a standard environment on a NIM client and then rolls it out via NIM. Here is a screen shot of some of his standard settings:
While you're at it, have you checked out the webinars of the AIX Virtual User Group - USA? Their webinars - usually one a month - cover all sorts of interesting topics presented in a hands-on way by technical people (real people!). I have a link to their Wiki in the Links on the right of this blog page. The two NIM presentations are listed under:
SOE read the Redbook
Speaking of NIM, as I mentioned in my previous post on the SOE LPAR, the NIM from A to Z Redbook has a valuable section on building a SOE LPAR. This section was written by an Aussie AIX friend, Chris Gibson.
Finally, there are two articles from Rob McNelly (an AIX friend from across the drink) which offer very helpful ideas: Establishing Good Server Build Standards
AnthonyEnglish 270000RKFN Marcações:  image dvd mkdvd iso chvopy vio virtual_media loadopt aix vios vmlibrary mksysb mkcd read_only 30.851 Visualizações
Not a spitting image
If you want to create a DVD image on AIX, you can do it right from the command line, without even touching physical media, using the mkdvd command.
It was a wonder back in ... well, a long time ago ... when these new-fangled CDs came out and you could play music and use them as coffee cup holders (not at the same time). But the novelty of having a button on your computer that spits out a CD or DVD at you has somewhat faded. We want an image, but not a spitting image.
On AIX you can create an ISO format image of a directory without actually having to hit that eject button.
The mkdvd command, as its documentation explains, lets you create a DVD
It doesn't need to be restricted only to mksysb backups. It also lets you:
And you don't need to have a physical writeable drive attached to your AIX system. You can create an ISO-format file, which can then be loaded onto the VIO server virtual media library.
Here's how to create the DVD image:
mkdvd -r /mycd -S
The -r flag lets you specify the directory you want to back up.
S is for slob
What's the -S for? It actually stands for stop. Here's why, The mkdvd command creates a file system before it copies the images onto your physical DVD, and then, being a well-brought-up AIX command, it cleans up after itself. Since we don't want to use the physical DVD, removing the image would somewhat defeat the purpose of the exercise. So no clean up, please, be a slob (maybe that's what -S really stands for).
I SO want to remember your name
The file in the /mkcd/cd_images directory includes the process ID in its name. If it spans over multiple DVDs, you get a .vol1 at the end of the first ISO image and I'll let you guess what the second one has appended to it. If you're going to use this file for longer than three picoseconds, you'll probably want to rename it to something memorable as you copy it to somewhere useful.
Once the DVD image is created, you can scp it to the VIO server and load it into the VM Library using mkvopt (or if you're cheeky, copy it straight into the file system /var/vio/VMLibrary without using mkvopt. You can always change its permissions to read only using the VIO server chvopt command:
chvopt -name memorable_file.iso -access ro
That allows you to share it with several virtual optical devices via the loadopt command. Or if you're too young to remember what CDs look like, use the HMC GUI or IVM.
Who needs the media?
The mkdvd command has some other options, such as specifying an alternate directory for the CD image with the -I flag, and creating the new file system in a different volume group for the file systems with the -V flag. And of course you can use it to create a new mksysb (bootable via the VM Library), or simply burn an existing mksysb image.
Why your boss likes you
There may have been 100 people apply for your job, but you got it. Why do you think that happened? I'll give you two possibilities:
AnthonyEnglish 270000RKFN Marcações:  paging_space active_memory_sharing profile shutdown dedicated_memory memory vios adapter activate required desired hmc active_memory_expansion partition minimum lpar hardware_management_conso... maximum aix 41.868 Visualizações
MEET MIN, MAX and DES
When you create an LPAR (Logical Partition), you have to create a profile to specify the processors, memory and I/O slots you want to allocate. I’m going to focus on memory right now, and specifically what the minimum, desired and maximum memory settings mean. You’ll see that the memory allocation can change while the LPAR is up and running using Dynamic LPAR (DLPAR) and DLPAR's can only go as low as min and as high as max. Makes sense, doesn't it?
This is about setting up LPARs that use dedicated memory. I’m not touching on Active Memory Sharing because sharing is not dedicated. I’m also not going to go into the Power7 magic called Active Memory Expansion. Just the vanilla answer to “how much memory am I assigning to my LPAR and can I shoot anyone who tries to steal it?”
The minimum memory is the amount which is required for the LPAR to start. If you don’t have the minimum, you can’t activate the LPAR. And for as long as that LPAR is up, nothing will take away that memory. You can’t even dynamically remove it using DLPAR. If you really don’t have the minimum available when it comes time to activate your LPAR, there is only one option: change the profile settings and attempt to activate it again. By the way, if you change the settings in a profile once an LPAR is activated, you need to shut it down and activate the new profile for the settings to take effect. A software reboot ( for example, using shutdown –r) won’t activate a new LPAR profile.
Minimum memory and Required adapters
There’s a useful comparison between the minimum requirements of memory and setting adapters as required. Supposing you have a physical adapter that will give you access to disks or the network. You want that adapter, because if you don’t have the disks you don’t have the LPAR. So you’d make that adapter required. If that adapter was also set to required on a second LPAR, the first LPAR to get activated would get the adapter, and the second LPAR to get activated … wouldn’t get activated. Required means business. It’s a demand, not a wish and if the LPAR doesn't get it, it's going to take its bat and go home.
Now just as an adapter can be set to required, an LPAR can have a minimum amount of memory. Once the LPAR is activated with that profile, the minimum memory, like the adapter, can’t be taken away until the LPAR is shut down. If you try it with DLPAR, you’ll get told off very quickly.
The desired memory generally is the amount of memory you actually get when you activate an LPAR. At least, you’ll get that much if it’s available (not used by other activated LPARs or the hypervisor). As you’d expect, the desired memory can’t be less than the minimum but it can be the same. It’s usually higher, because it’s your memory wish list, and because if you play your cards right (and set your values properly), you usually get it. When you activate your LPAR, your memory allocation will be as close to the desired as is available at the time. So you could have a minimum of 1 GB, a desired of 24 GB and the actual allocation of only 10 GB because that's all that was available when the LPAR got activated.
Whatever you end up with when you activate the LPAR (somewhere between min and desired), if someone happens to run a Dynamic LPAR operation to remove some of your allocated memory, then you could go below your desired amount. Your memory could go right down till your LPAR has the minimum set in the activated profile. No one can take it below your minimum without a shut down of your LPAR.
Once again, let’s consider the adapter comparison. Supposing you had an adapter which had a DVD-ROM connected to it. You’d be happy to have it moved around between LPARs without having to shut either one down. So you make the adapter desired on both LPARs. The first one to get activated gets it, but not for keeps. You can still move the adapter (and the DVD-ROM that hangs off it) to the second LPAR, or for that matter, even to another LPAR which doesn’t have it as desired at all.
Next we get to maximum memory. Actually, we don’t get to it next. You really only get to see the max when you assign extra memory using Dynamic LPAR. Maximum memory is the high water mark for the LPAR. You won’t ever exceed this amount unless you shut down, adjust your profile settings and activate the LPAR with the new profile. The maximum memory can't be less than desired but it can be the same. It's usually greater. And guess what? The maximum memory can't be less than the minimum. Just thought I'd mention it for some readers who needed to hear it (not you, of course).
So, supposing your system has 512 GB of physical memory, and you set your min, desired and max to 1 GB, 2 GB and 500 GB respectively. For the life of the LPAR, the OS will have 2 GB of memory. If it needs more, it will go to paging space, unless you dynamically allocate more or do the shutdown / edit profile / activate thing. IBM Power Systems are good, but they’re not mind readers. If you want more memory, you have to say so! Change the Desired amount before you activate your profile, or adjust it using Dynamic LPAR, OK?
Now if you find in the course of running your partition that it needs more memory, you can assign more dynamically up to the maximum memory setting in the profile which you have activated. Sounds like a good reason to set the max for all LPARs to the highest possible value, doesn’t it? Except that there’s a little man inside the hypervisor who’s on alert, just in case you want to grow the memory on demand. He’s a bit expensive, keeping him on call with all that spare memory in his back pocket, so if your max memory is absurdly high, you’re setting aside some memory in the hypervisor for him to be able to grow your LPAR’s memory just in case. He doesn't reserve everything you might possibly need to allocate, but it is an overhead if you're sailing close to the wind with your total system memory.
WATCH OUT FOR MAX: CUSTOMER SERVICE WITH A FROWN
When I was a young system admin and file systems were 2 GB, if someone asked for an extra 2 GB, I told them that my motto was “the customer is always wrong” and that if they thought I was going to throw away good disk after bad just because they were too lazy to clean up their files they had another thing coming. So they cleaned up and I gave them some extra disk, and told them to come back with their begging bowl in a month if they dared.
The purpose of this little lesson in customer service is to show that assigning extra resources isn’t always a smart solution. Expensive, yes, but not always wise. In setting the memory requirements for your LPARs, it’s a bit hit and miss, but you hopefully will, in the end, hit the right amount of min, desired and max without breaking the budget or losing all your friends.
AnthonyEnglish 270000RKFN Marcações:  integrated_virtualization... sea management ethernet lpar virtual ivm id outage shared rob_mcnelly hmc v22.214.171.124 dlpar power server aix adapter ibm hardware_management_conso... migration 12.531 Visualizações
In September 2009 Rob McNelly wrote on his AIXChange blog about Migrating from the IVM to the HMC. I have documented my own experience of this procedure. You can download it from here, at a very affordable price of USD 0.00 (no refunds).
The IVM or Integrated Virtualization Manager, is a browser interface to the VIO server on smaller systems, and it has HMC-like functionality, such as Dynamic LPAR, the ability to configure LPARs, stop and start them and so on.
The HMC (Hardware Management Console, as you know) is able to manage several physical servers and is mandatory for larger systems. It can also be used for smaller systems, and is a worthwhile investment, in my view, once you get beyond a single small server.
Two servers, two IVMs
I had a client who had bought a production Power6 550 and a P6-520 for Dev and Test. After some months of discussion, their Business Partner convinced them of the benefit of investing in an HMC to manage these two systems with their growing number of LPARs. The challenge was migrating each of the servers from being IVM-managed to the HMC. I have put together a document of my own experience of the migration. It doesn't attempt to be a step-by-step guide. More of a diary for my own benefit but you may find it useful.
Forward planning brings us unstuck
We thought we were being safe by getting some work done ahead of the outage time. We racked and cabled the HMC and put it on the network, in preparation for the scheduled outage two weeks hence. Problem was, no one told the HMC the planned go live date. To our surprise, it immediately discovered the two servers. At the same time, the HMC was reporting the two servers were in "Recovery" state, but it wouldn't take further control of the systems or their LPARs until the outage which was scheduled for after a huge month end. The IVM had been effectively disabled, so any IVM-specific commands were out of bounds. No profile backups, no DLPAR, no shutdown and activation of LPARs was permitted, either from the IVM or from the HMC. Nothing would undo it - not even powering off and disconnecting the HMC from the network.
We had a VIO server, but no IVM and no HMC that we could do anything useful with. It was the technological equivalent of a hung parliament.
All's well what ends well
In the end, it all worked, and the customer has been running happily on the HMC for many months now. Still, it was a challenge. You can find my comments about the migration from IVM to HMC Migration - A Customer's Experience
Looking back, it was quite funny, I suppose. As long as you weren't me.
AnthonyEnglish 270000RKFN Marcações:  ctos aix7 6.1 7.1 sar toggle disk tools iostat aix6 performance aix monitoring topas nmon memory 1 Comentário 22.522 Visualizações
topas and nmon - what a performance!
UPDATE: I'm grateful to Chris Gibson for highlighting some great new features available in topas for AIX 7.1 and AIX 6.1 TL 6:
The topas command is a very popular tool for checking performance of an AIX system. Another highly popular one is nmon (Nigel's monitor, so named because it was written by Nigel Griffiths, the man behind most of the excellent Power6 / Power7 and AIX6 Wiki Movies as well as countless other presentations which make AIX look easy).
Both nmon and topas give a general summary screen allowing you to drill down to sub screens. You're probably familiar with at least one of these tools, but there are a couple of features which may make your experience even better.
Tilde toggle nmon to topas to nmon
You may be a topas fan, but want to switch to nmon to view certain screens. Or you might like nmon but your topas-loving colleague is watching over your shoulder. From AIX 6.1 and beyond, the nmon and topas commands run from the same executable. If you're running topas you can press the tilde key:
and you're in the nmon screen. In the same way, nmon returns the complement: toggle to topas.
(In the days of green screens, we used to have a little computer game running on CTOS and we had a magic button like that - if the boss was wandering around, you hit the button and a spreadsheet screen would appear instantly - perfectly useless other than as a screen saver ... and maybe a job saver).Within topas you can press the following keys to see sub-screens:
topas screen sorts
Here's a list of the hot disks using topas and then pressing D, or from the shell prompt topas -D :
In many of the topas subsections where you have a set of metrics you can use the arrow key to jump across to the next column heading. This automatically sorts the values by that field:
There is a lot of documentation on both tools, starting with the command documentation: topas and nmon. Both of these performance monitors provide invaluable views of how your system is tracking, and it's worth getting familiar with at least one of them.
AnthonyEnglish 270000RKFN Marcações:  layout under customize twitter down aix blog settings 6.878 Visualizações
A little bit of housekeeping.
I've played with some of the templates for the blog using the advice in How to Customize your (developerWorks) blog, so things may not appear as they appeared to appear.
AnthonyEnglish 270000RKFN Marcações:  fcstat tuning vmstat iostat memory fibre queue_depth adapter disk sat performance i/o channel aix lvmstat 2 Comentários 21.098 Visualizações
NOW FOR THE I/O
I highlighted some of the excellent AIX performance tuning resources here. They particularly dealt with memory and paging space. Now to I/O tuning.
Fewer larger LUNs and checking queue_depth
Jaqui Lynch has the follow up article in her two-part performance tuning series. This one's on Disk I/O and Network Tuning. She deals especially with queue_depth and explains why one great big LUN with lots of spindles may need to have this tuned:
Jaqui also covers monitoring fibre channel adapters with the fcstat command. The changes will need to be done with some caution, but as I/O is more commonly the bottleneck on a slow system, it's worth looking into the systems you work on, especially the ones that are so slow that people have given up complaining about them and started complaining about you.
I implemented the memory tuning parameters on a SAP production system and it has had a marked improvement on paging space (from 24% to not in use at all now), and the dreaded revolutions of the clock hand - a complete scan of memory looking desperately for something spare - is down to 0. The vmstat -s is looking much healthier, thanks to Jaqui Lynch's first article, Paging, Memory and I/O DelaysLVM Monitoring
There is also Part II of the developerWorks article on Optimizing AIX 7 performance - Monitoring Logical volumes. This is especially helpful for placement of logical volumes if you're using local disk.
There are also good examples of how to monitor a volume group using the lvmstat command. You need to enable statistics first using
You can do this online - no need to varyoff and varyon the volume group.
Here's an example of the LVs in a volume group on a system I've been looking at. Good argument for naming your logical volumes according to the mount points rather than using lv00, lv01 etc.
When you've got a performance problem, sometimes the solution is to throw more iron at, but a simple tweak to a parameter here and there can have a vast improvement and also save money for more important things for the company ... like paying you.
AnthonyEnglish 270000RKFN Marcações:  permissions enhanced_rbac su authorizations sudo privilege privcmds root aix setkst privfiles authorization table command kernel rbac roles role_based_access_control 10.332 Visualizações
Enhanced RBAC at your service
From AIX 6.1 when you install AIX filesets you'll often see mysterious messages advising updates to the Kernel Authorization Table (and some other tables), as in the example below:
installp: APPLYING software for:
So, do you sudo?
These messages are all to do with Enhanced RBAC (Role Based Access Control). Briefly, Enhanced RBAC allows you to give certain privileges to selected users, or groups of users, without granting them the root password or using sudo.
As the IBM RBAC documentation explains:
It then goes to list the databases which Enhanced RBAC uses:
In AIX 6.1 and 7.1, Enhanced RBAC is enabled by default.
ASCII is less pesky
Because the AIX people are nice (just like you), configuring complicated functions can often be done by editing ASCII files or via SMIT, and then a daemon or executable is run to read the files in from human-readable format to human unreadable format..
These are the principal ASCII files used by Enhanced RBAC:
So here's where your mysterious messages come in. After editing any of these files you need to set the entries in the Kernel Security Tables by running setkst. This is the command which gets invoked when you install most AIX software these days (since AIX 6.1).
Man, what is RBAC doing there?
You may have already seen references to RBAC in the man pages for AIX commands. For certain commands, the documentation will have a note indicating that the command can do "privileged operations". That doesn't mean you have to post the root password up on your intranet. You could grant limited access to a group of users for what would have required root or sudo access in the past.
Here's the RBAC note for the chdev command:
Attention RBAC users and Trusted AIX users: This command can perform privileged operations. Only privileged users can run privileged operations. For more information about authorizations and privileges, see Privileged Command Database in Security. For a list of privileges and the authorizations associated with this command, see the lssecattr command or the getcmdattr subcommand.If you want to know more about Enhanced RBAC, have a look at RBAC in simple steps. You could also watch Nigel Griffiths' Wiki movie 14 AIX 6 RBAC. Of course, the AIX 6 Advanced Security Features Redbook is a wealth of information with practical examples.
Why do I care?
Even if you don't use RBAC, you never know when someone's going to watch you doing whatever AIX admins do and ask "what does that Kernel Authorization message mean?" Well, now you know, and you're a better person for it.