A colleague in another country had an AIX logical partition that went down and couldn't boot. He asked how he could boot it without media. It was a development logical partition.
I knew nothing at all of the environment, and had no possibility of access to it myself, so I found out the following pertinent facts:
- there was an HMC
- there was no Virtual I/O Server (VIOS)
- there was a NIM server but it was at a lower level than the AIX host that couldn't boot
- they had a mksysb backup which was in a file system on a different AIX host
So how would they boot it? Maybe you've already come up with your own answer:
- Easy! Just use NIM.
- Have they got a VIOS? Use the Virtual Media Library.
- Boot from media. Just find the AIX DVD. It must be somewhere.
- What's media? What's a DVD? (This answer is for our younger readers).
Before you ask
I had a few other questions swirling around my head.
- What triggered the host going down? Was it a user-initiated reboot? If so, why did they reboot in the first place? If not, do they know what event brought it down? Is that fixed, or fixable?
- Has a call been placed with IBM? If there was a system dump, has that been sent through to IBM to get some support advice?
- What release of AIX was it running?
- Do they have a mksysb backup? Is it recent enough to use? Was it backed up to NIM?
Remember, the NIM server was down level in comparison to the AIX host which was (I expect) a NIM client. That's a big no-no. You're supposed to have the NIM server at the latest level and then use that to update the clients, but obviously they had bypassed NIM in building and/or updating the AIX host.
"So, update NIM already!"
Easier said than done. In this environment, the NIM server is a production host and it can't just be updated without notice. Yes, it should have been up to date all along before they worked on updating their dev host, but these things happen.
What about the VIOS Virtual Media Library?
Obvious solution, isn't it? You can load an ISO image of the AIX installation media and/or the mksysb (produced by the mkdvd command) and boot off that. Hopefully, the problem with booting up is just a bootlist that needed to be updated or something simple. But they didn't have a VIOS and I didn't know if they had the resources to build one - particularly an adapter to connect to a disk for the VIOS rootvg.
Restoring mksysb to another disk
There was another approach they could have used. Since they had a good mksysb backup, they could build an alternate disk with that backup (they'd have to do it from a running AIX system, not from the dead one!), and then reassign the logical partition to boot off that disk. I've never tried this myself, but it would be done with the alt_disk_mksysb
A new NIM
I later heard that they built a new, temporary NIM server and booted the logical partition from that. I didn't get a full briefing but perhaps they restored from the mksysb.
This little account probably has got you thinking of what you would or wouldn't do. There were some things in the environment which could have been done better, particularly having the NIM server up to date.
Sometimes shortcuts are the slowest way to get your destination. No doubt my colleague and his team will revisit this environment, and consider updating NIM, and then keeping it updated before they do any patching or installation of other partitions. They may also virtualise a little more by installing a VIOS. And they could also look at a regular test restoration from a mksysb.
In the end, they were able to recover the dev host that had gone down. I'm sure they'll consider new approaches to the AIX environment which will make the process smoother next time.
Over to you ...
If you've got some ideas about how to address this situation, you can add a comment to this post. Comments are moderated, mainly to stop people trying to put in spurious links that sell handbags. You'll need to log in to IBM developerWorks to add a comment. I'd be interested in your feedback.
mksysb - an oldie but a goodie
Long before the days of HMCs, VIO servers and LPARs, AIX version 3 and 4 ran on standalone systems. A system had its own tape drive. This would be useful for mksysb backups, which backed up the rootvg. Reliability, Availability, Scalability 1990s style
Actually, a rootvg backup was usually a full backup of everything - including all the data - since the idea of separate disks and volume group for data was a luxury most of us hadn't thought of. I saw many sites which survived for 10 years on a single 1 GB SCSI disk. The backup tapes were kept in a cardboard box on top of the server and never taken off site. The server itself was next to the sysadmin's desk in the main office. Aircon was a business hours concept except when the server was out on the factory floor. In those cases the server's main protection from the less than pristine environment were several layers of dust.Done and dusted
Gone are the days of dusty servers and decade-old disks, but some things have survived the test of time. Like the mksysb
. The mksysb command
has been around forever, but it's still a valuable tool. With the development of virtualisation, it's got wider use than recovering your system when your only 1 GB disk dies.
You can use a mksysb backup to:
mkdvd for ISO images
- clone your OS, for example from a Standard Operating Environment (SOE) LPAR: a gold image
- migrate to new hardware
- a disaster recovery test
We have the ability today to create a mksysb backup into an ISO format image using the mkdvd command
. You can then copy that ISO file to a VIO server and use it as as a file-backed device by means of the Virtual Media Library
(also called the Virtual Media Repository). This allows you to present the mksysb backup on a virtual optical device without having to use a NIM server or rely on physical media such as DVDs. For more details on how this all holds together, see Reliable Restores
The mkdvd command creates its own temporary file systems: one each for
- making a new mksysb (unless you use an existing mksysb specified by the -m flag)
- a DVD file system structure from which a single ISO file will be built; and
- a location for the final ISO image (or images if the backup doesn't fit into 4.7 GB).
By default, these file systems are prefixed with /mkcd
because the mkcd command
was originally used to burn mksysbs onto physical CDs. And they are created in rootvg unless you specify an alternate volume group using the mkdvd -V
flag. The mkdvd command
is one of many reasons why it's smart to leave some spare disk in rootvg
If you don't want to use the default file systems when you run mkdvd, you can overwrite them using the flags shown in the table below. For example, you may want to use an NFS mount for all your temporary workspace.
Location of existing mksysb
Create new mksysb
Volume group for new file systems
Location of new mksysb
File system for DVD file system structure
Location of final DVD images
You don't only have to use mkdvd for mksysbs. You can also use the mkdvd command to create an ISO file of an existing directory structure. It's a simple way of sharing software around to several LPARs without copying it to them all. To create an ISO file of some directory and all that sails in her, use
-r /some_directory -S
I use mkdvd all the time, because I use the Virtual Media Library
all the time. It's really worth getting used to some of its most common flags.
For more information on mksysb and mkdvd, have a look at the article in the IBM Systems Magazine Reliable Restores
For ... ahem .. undisclosed reasons, an organisation's AIX administrator has gone Missing in Action
. The empty shoes are to be filled by you. You've never heard of the company "Clean Doubt" and never logged into their systems. You don't know what applications or databases they run, so you're full of questions for the handover meeting. You are introduced to the boss who has an iron handshake and a gravelly voice. His name is Mr. Clean "call me Squeaky" - and he explains that the previous sysadmin "had to be let go". "Accidents do happen", he adds with a steely smile. He hands you a piece of paper with the root passwords on it. He wants to know how his systems are running. You've got two hours to report back.
Somehow you both know that the handover session is over.Incentive
You don't have local knowledge, but after your meeting with Squeaky, you do have motivation to get that knowledge. Quickly.
If you've worked on IBM Power Systems before, you're actually a good deal ahead of the game. After all, you can check a few basic things which will give you half a clue about what the systems are running:
- Check file systems, volume groups
- File system names can give a good indication of the sort of applications that might be running
- Check processes
- the output of ps -ef will soon tell you what applications or databases are running
- Host names
- Most places seem to change their host name policy every 18 months to two years (coincidentally about the same shelf life as the management). Do the host names contain some clues as to what is running? Consider these hostnames. Some shed more light than others.
- oraprod (Oracle production)
- saptst (SAP test)
- jupiter05 (too many LPARs - wish they'd discovered some more planets)
- rs12erp (ERP system migrated from our RS/6000 server)
- brat (Brisbane Disaster Recovery AIX Test)
- Cron jobs and scripts
- Are there regularly scheduled jobs running in the cron or initiated by some other scheduler? Are there some scripts which run frequently and may be central to the healthy flow of data?
- Backups and Disaster Recovery
- Are backups being taken? At all? How often? What's the backup utility? Is there a backup checklist or some documentation?
- Is there a DR plan? Has it been tested recently? Has there ever been a disaster? Was there a recovery?
- Is there documentation which can help you get a feel for the environment? Is it reasonably up to date? Are there project managers, colleagues, anyone who can shed some light on what the system does?
- Do you have access to your predecessor's bookmarks or favourites? This alone can tell you a lot about how organised, up to date and ready to learn the person was. Did your unmentionable forerunner follow Rob McNelly's AIXchange blog? What about Chris Gibson's AIX blog? Or AIXpert? Did the unfortunate sysadmin click on the link to follow AIX Down Under on Twitter? Was that decision the right one? Was it the last one?
- NIM and management servers
- Can you find a NIM server? How about an HMC or IVM? Is there an IBM Systems Director server or maybe some other server which does system monitoring? These can give you a quick snapshot of the lay of the land, so you can get a quick (albeit superficial) grasp of what the systems are all about.
- Is there a DBA screaming for something urgent? See if you can empathise and then discreetly ask what kind of database the system is running. Are there developers who have an urgent release and users demanding to be able to log in? This sort of buzz can give you a little bit of a feel about what's important, what's hot, what's not.
- does anyone else want to cry on your shoulder about the guy who
got the chop disappeared in mysterious circumstances? It's amazing what you learn over a cup of coffee.
Checking the versions which are running will give you an idea of how up to date the system is. It may indicate if your predecessor (whose name is never to be mentioned) was keen to keep things crisp and clean or was more interested in counting down the days and planning an early
retirement. There's nothing to be gained from bad-mouthing the absent, but it may give you an idea of how much work you've got cut out for you. You'll also get a feel for how well peoples' expectations have been managed, which is a very important part of an administrator's job.
Are there 30 LPARs, maybe some WPARs? Are there any VIO servers or do all the hosts still run with dedicated adapters? For that matter, the hardware vintage will help you know whether there's a collection of old standalone servers accrued over the centuries or a Power7 that's on the latest firmware and running PowerVM with all the bells and whistles. The prtconf
command will help you here. What version is the HMC? What does the output of the VIOS command ioslevel
tell you? What about the AIX levels using oslevel -s
Much as you'd like to walk in and pick up the pieces of a system running AIX without any assistance, there is some local knowledge that would save you an immense amount of time. Well, spare a thought for your successor.
This time of year is a chance to tidy up a little, document, and provide some sort of framework for a quick handover, in the unlikely event that you're the one who goes M.I.A, courtesy of Mr. S. Clean.
MAKING THE BEST OF WHAT YOU'VE GOT
A customer wants to clone his production LPAR running AIX 6.1 - it's on a Power6 server - to an LPAR on his DR site which runs an older p5 server. Both sites have HMCs. As the environment is very small, all I/O is handled via dedicated adapters on each LPAR. There is no VIO server (no IVM) or NIM server, and although the p6 production server has a DVD-ROM, the p5 server at DR only has a CD-ROM, not a DVD. That means the mksysb from the production LPAR will need to be in CD format, rather than DVD. We'll use the mkcd command
to do that.Update: the customer discovered that there was a DVD-ROM on the p5 server after all. All went well with the mksysb restore, albeit using CDs rather than a DVD.
Product media required?
The difficulty is that at the end of the mksysb restore at DR, we may be asked to load the AIX 6.1 product media. We have that product media, but only in DVD format which the CD-ROM won't be able to read. There may also be some other AIX 6.1 filesets we'll want to install later from the product media. So the big question is:
Are we going to need the AIX 6.1 CDs?
On some investigation, we have found that there is an entry in the bosinst.data file which can include all the device drivers we want in the mksysb. The field in the bosinst.data is ALL_DEVICE_KERNELS. As the documentation on creating a system backup explains:
All devices and kernels are installed by default when performing a base operating system installation. This allows you to create a system backup that contains all devices and kernel types. Because the system backup contains all the devices and kernel support, the system backup can be used to install another system without the need for the AIX® product media.
Still, I'm not sure his system was originally set up with that ALL_DEVICE_KERNELS option. As the target LPAR at DR has dedicated adapters, it would be nice to have access to AIX 6.1 installation images at the end of the mksysb restore, in case we need them. We may want to install some other filesets anyway after the cloned LPAR is rebooted.
So here are our options:
create a NIM server
buy, beg, borrow or steal a DVD-ROM for the DR site
borrow or steal some AIX 6.1 CDs from someone
none of the above
As IT support for this very small company is a one-man shop (he's a jack of all trades), the client ruled out the first two options. He didn't want to invest his limited time in learning NIM or the VIOS, although I am sure he'd have no difficulty picking up either of them very quickly. The dedicated adapters on each LPAR at Prod and DR are long since paid for, and he's happy to use dedicated devices for the rare occasions where he needs to install or migrate his OS.
As getting hold of a DVD-ROM drive would take some time, we thought of a way of getting hold of AIX 6.1 CDs: download them! Although 6.1 is only available in DVD format (ISO images), older copies (6.1 TL 4) are still downloadable in CD format. The 6.1 ISO and CD format images can be downloaded from IBM Entitled Software Support
The customer downloaded the AIX 6.1 CDs and burned them onto physical media in no time at all.
We will boot off the mksysb, load whatever AIX 6.1 TL 4 CDs may be required at the end of the mksysb restore, and then update to the latest TL from a fix pack downloaded from IBM Support Portal
. We can use those AIX 6.1 CDs for installation of other base filesets if we need them.
The customer will probably still arrange to purchase a DVD-ROM and from that point, he may reconsider the VIO server and / or NIM, but for the time being we've got all we need to do the clone, without NIM or the VIO VM Library, or a DVD.
We do have standards!
Having a Standard Operating Environment
(SOE) for AIX has a lot of benefits. It's not just the consistency between images and some sort of version control over what you're rolling out. Using a SOE also saves time.
- If you build AIX systems all the time, then there are so many steps which you won't need to repeat - just clone an existing image.
- And if you don't build new AIX images too often, then it's a lot of work remembering what you have to do every time.
I have discussed building a SOE image for AIX
before. And as you know, you could use the VIO server Virtual Media Library
to do it (Not that again - doesn't this guy ever let up? - Ed
This post covers some suggestions from some AIX big wigs on what they put into their SOE image.Tuning and tools
First we start with Jaqui Lynch, whose articles regularly appear in the IBM Systems Magazine, AIX edition
. (If you don't subscribe, DO!) Jaqui uses a short script to set tuning parameters, and then installs third-party software such as gcc, lsof, openssl / ssh and a number of tools from the Linux toolkit. You'll find more details in the article Installing and Upgrading to AIX 6.1
. The article's from 2008, when 6.1 was still in nappies (diapers), but still applicable apart from a few small changes (nmon now is part of AIX and doesn't need to be installed separately). The SOE recommendations are in the section "Installing AIX 6.1 from Scratch."MiNIMum NIM
Next comes this gem from Steve Knudson, who gave a couple of presentations on NIM. Steve builds a standard environment on a NIM client and then rolls it out via NIM. Here is a screen shot of some of his standard settings:
While you're at it, have you checked out the webinars of the AIX Virtual User Group - USA
Their webinars - usually one a month - cover all sorts of interesting
topics presented in a hands-on way by technical people (real people!). I
have a link to their Wiki in the Links on the right of this blog page.
The two NIM presentations are listed under:
June 23 & 24, 2010 - NIM Basics and NIM Advanced by Steve Knudson
SOE read the Redbook
Speaking of NIM, as I mentioned in my previous post on the SOE LPAR, the NIM from A to Z Redbook
has a valuable section on building a SOE LPAR. This section was written by an Aussie AIX friend, Chris Gibson. AIXChange
Finally, there are two articles from Rob McNelly (an AIX friend from across the drink) which offer very helpful ideas: Establishing Good Server Build Standards and a follow up article are well worth reading and imbibing.
Now maybe some enterprising soul will plagiarise all those excellent ideas, collate them into a how-to article or book and send me the royalties.
Fun out of the Sun
In the last few days I've had lots of fun with migrations to AIX 6 and 7. And when I say "fun", I don't mean it in its usual technical sense within IT (insurmountable problems, unrelenting stress, obscure workarounds and chronic sleep deprivation).
I mean "fun" according to its (almost obsolete) usage:
noun (enjoyment, amusement, or light-hearted pleasure)
Over the last few days, I haveAll of the upgrades went smoothly
, with no challenges or what IT people mean when they say "fun".Making a fresh start
My last feather in the AIX 7 cap was to do a "New and complete overwrite" installation of AIX 7.1. You might do this instead of migrating your system from an earlier release. If your LPARs are full of inconsistencies and undocumented workarounds, maybe it's better to build a system from scratch.
If you want to (or need to) make a fresh start, so that you can clean out the soul of your system from the sins of the past, it's good to know that AIX 7.1 is fully binary compatible
with AIX 6 and AIX 5. Building a shiny new LPAR is especially useful if you decide to create an AIX Standard Operating Environment (SOE) LPAR
.Half an hour to AIX 7.1
The fresh install of AIX 7.1 took around 25 minutes, using the VIO Server Virtual Media Library
, which is how I did the other AIX migrations. The new and complete overwrite installed 591 filesets as the main course, and then another 4 for dessert.
Here you see the creation of the boot image at the end:
The reboot time took around 3 minutes. I've yet to install the mandatory AIX 7.1 service pack
but I expect it will be all over red rover in about 5 minutes. It can be downloaded from the IBM Fix Central web site
As with the other installations, the VM Library was running off internal SAS disk allocated to the VIO Server. The target LPAR boots from SAN, using vscsi disk which is passing through two VIO servers using MPIO.AIX 7.1 is fun
I'm enjoying these AIX installations and migrations but they were so straightforward that it's getting a little quiet around here.
I wonder when AIX 8 will be ready.
(or not using) NIM
- the AIX Network Installation Manager - allows you to install and
maintain multiple hosts from the one master. Among people newer to
AIX, NIM is often left on the backburner. With the VIO
Virtual Media Library, a good deal of the NIM master's
functionality has alternatives, at least if you're building LPARs
which are VIO clients on the same system as the VIO server
NIM still has its uses - especially for providing a
source for building across to a different server or a different site.
Some AIXers wouldn't be without NIM. Others tend either to live
without NIM entirely or build an initial NIM server and never touch
Part of the hesitation about NIM is confusion about
some of its components, so it may be worth knowing of some helpful
sources for information about NIM.
you're a little NIM numb and in need of being unconfused, an
excellent start is the AIX
Virtual User Group Webinars. There you can download a 90 minute
session on NIM basics and another one on more advanced topics.
June 23 & 24, 2010 - NIM Basics and NIM Advanced by Steve Knudson
those presentations, Steve Knudson explains what on earth is meant by
SPOT and lpp_source. Here's my transcription along with my comments:
The lpp_source is a
directory to hold installp images from the AIX media.
- shared product object tree - is a /usr file system that we create
from the lpp_source. Clients use commands from the spot during the
install process. When the network image first comes down and the
client first begins to install, he [you've got to admire someone who calls LPARs "he"] has to run commands, and the
commands aren't all included in the boot image. He's got to run mkvg
to create a root volume group. He has to run mklv to create logical
volumes, mkfs to create file systems. He finds those commands in an
NFS-mounted SPOT. That's why we have a /usr NFS-mounted file system
as well as an lpp_source.
The Webinars cover lots of other
explanations and provide some very helpful presentation materials.
you're after a little more info on NIM, you could have a look at this
article by Jaqui Lynch from the IBM Systems Magazine called
with NIM. Jaqui points out that
basically provides a central point of management for installing and
maintaining AIX images for both LPARs and individual servers."
She also shows how NIM can take
care of mksysb images for multiple LPARs and servers over several
different sites. Have a look at her more recent article on the same
AIX LPAR and Server Management with NIM.(The link above to the AIX 7
Open Beta forum
was incorrect. It should work now.
The IBM Redbook
from A to Z in AIX 5L is still valuable, even though it was
written for AIX 5.3 and we're now testing the AIX 7 Open
(Incidentally, since NIM must be at the latest level,
you can't install the AIX 7 Open Beta mksysb from a lower level NIM
server. You can follow this issue on the AIX 7 Open Beta user forum).
AIX 7 Open Beta forum:
We have provided NIM filesets to enable network installation. Please see the Download tab on the AIX 7 Open Beta website Link:https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/aix7ob/index.shtml An updated "Getting Started" guide (v1.4) containing instructions for NIM fileset installation has also been provided on the Library tab.
Introduction to PowerVM Architecture
Today I stumbled across this great introduction to IBM PowerVM architecture
. (I say "stumbled" because I wouldn't usually have looked into WebSphere CloudBurst
, so it was a very pleasant surprise.)
It's brief, as any good introduction ought to be, but it covers a lot of the key components which every AIX administrator should know about:
YAA (Yet Another AcroNIM)
- What is an HMC?
- The VIO server
- IBM Systems Director
- VM Control
We techos can be a little fond of acronyms and when you hear a new one you're never quite sure whether to pretend you know all about it or ask what it means. Well, this intro to PowerVM explains them all, even the acronym NIM (which she spells out "N. I. M." - I've never heard that before.)
Here are the links shown in the graphic above:
VIOS Main Page
IBM Systems Director
Step-by-step movie "hands-on" demos
Maybe these should be called "over-the-shoulder" movies, as we watch Nigel Griffiths and co take us through the setup of Power Systems. An excellent and fast way of learning AIX, the HMC, VIO server .. lots of things about IBM Power systems. The "Back to Basics" movies are especially good if you're starting out on Power systems or are not quite up with AIX and its powerful virtualisation features.
These links and simple explanations can all be found in the movie linked to at the top of this post. Thanks to whoever put this together for explaining the PowerVM architecture so well. A great, simple overview of how it the Power system components all hold together.
This is a great way to spend 7 minutes.
In his AIXchange blog
, Rob McNelly speaks about VIO backups and explains how he uses the Virtual Media library to build a new Power system
in a completely new environment. It may be helpful to run through Rob's procedure and provide some background and documentation on the steps which he uses.One VIO, then NIM
The system is straight out of the box, and there is no NIM server, so here are the basic steps which Rob suggests. (I've added a couple of steps of my own along the way).
- Boot first VIO server from the DVD drive
- Create Virtual Media library using mkrep
- Copy other physical media (AIX and VIO CDs) using mkvopt (for examples, see this post from Rob)
- Use VIO VM Library to build first AIX client
- Make it a NIM server
Coffee break (my idea, not Rob's)
Step back and admire your good work so far.
At this point, you should have a VIO server with a VM Library, and an AIX LPAR which is a NIM server. That NIM server can be used for all other builds. Here are the steps:
- Map the AIX .iso image to the NIM server using mkvdev
- Load any other required VIO servers from NIM
- Use NIM to build other AIX client LPARs.
Multiple methods of cat-skinning
Rob's post was really about a new option for backing up the VIO server, but his summary of how he installs a system from bare metal is very helpful.
Of course, there are several alternatives to many of these steps. As Nigel Griffiths explained on the AIXpert blog, you can now download all your licensed IBM software
instead of using physical media. This makes it more attractive to build your AIX LPARs using the VM Library
, although NIM is always an option.
And as we have seen before on AIX Down Under, you could create an AIX SOE LPAR
and clone that using NIM or the VM Library.Just the beginning
Summarising a complete installation strategy in a few paragraphs is only a broad brush outline. There are lots more considerations ... and there's plenty more configuration work (storage, network, redundancy, backups, DR, security, monitoring - not to mention installing applications and migrating data). Still, it's helpful to have a road map for a complete build of a new AIX environment. It can also be useful explaining the system layout to clients and colleagues, or bouncing around some ideas with others before you jump right in.
The aim is to end up with an environment that is as simple as you can make it, takes advantage of the technology and, above all, does the job it was meant to do.
A Standard Operating Environment LPAR
It is so easy to build an AIX LPAR these days, that we can end up with lots of them very quickly. When this happens, or even before it does, it may be worth building one more : a S
nvironment or SOE. This can provide consistency in your organisation's configuration and make it even faster to build new LPARs without starting from scratch.What is an SOE for AIX?
The NIM from A to Z in AIX 5L Redbook
has a valuable section on building an SOE. Although it's
for NIM on AIX 5, it raises some important issues. The Best Practices section explains that an SOE for AIX systems is "a set of
best practices, procedures, documents, standards, conventions, tools,
and utilities with which you install and administer an AIX server." The server doesn't need to be physical - an LPAR is also a server. It's a logical server, so it doesn't need to have its own dedicated hardware.Virtually ready for an SOE
A virtualised environment makes it easy to build new LPARs very quickly. You no longer need dedicated hardware: disk, network, CPU, memory can all be allocated without hardware investments or even stepping into a data centre. As we've seen before on AIX Down Under
, it's easy to load ISO images via the VIO server
. That makes it straightforward to build a new SOE and then to clone it to create other LPARs. If you prefer to use NIM, you can refer to the Redbook mentioned above as well as online documentation for NIM.Detonate, don't renovate
Plenty of green field AIX sites start out with a single
LPAR which is meant to serve as the "golden image" from which all new
LPARs will be built. If it's the first LPAR then it ends up having all sorts of software installed for testing.
Standards are broken quicker than they are created and what was the
golden image soon looks a little more like it's made of lead. It's probably time to start again.Carrying the
sins of the past
In other situations, a previous "legacy"
version of AIX is in place. It has many mysterious hacks, features,
links, installations and settings which no one really understands and
everyone is afraid to touch. Although you could clone this system, with
its workarounds (that word should always send a shiver up your spine), you will probably find that it's worth making a
completely fresh start with a New
and Complete Overwrite of AIX
. You can always review the old
system's configuration, but at least the changes you make to the new
image will be understood and documented, rather than remaining in the secret realm of IT whizzes from generations before us.Laying the foundations
So how do you build a Standard Operating
Environment (SOE) LPAR from scratch? First, as we saw in an earlier
post, you can install a base AIX using
the VIO server virtual media library
. Every organisation is
different and it's impossible to outline a standard which will fit
everyone, but here are some ideas which will help:
as you go. It's important that if someone else has to build the SOE
again, it will be easy to do. When the SOE changes, the SOE document
should also. And it also needs to be easy to find, so that others can rely on your past efforts without interrupting your holidays ... or your retirement.
the latest AIX. Get the latest fix packs from IBM Fix Central and
use the trusty VIO VM Library to patch it up. Check your prerequisites
are in place (HMC, firmware etc.). AIX 6.1 has been around since
November 2007 and it's got lots of features which make it worthwhile and
necessary to move on from earlier versions. A good starting point is
6.1 Installation and Migration page.
- Use the technology. Take advantage of virtualisation, redundancy, eliminating single points of failure and removing unnecessary dependencies. While we're on the subject, use the standard tools. Too many systems are on old operating system versions because of a single unsupported, unlicensed, unrelocatable* piece of software which can easily be replaced by something that comes with a supported version of AIX.
- Tighten security up front. Use
to turn on security. It's far easier to loosen security settings than
tighten them once you're in production.
- Keep version control. Within the SOE LPAR
you can include a file with a version number. That should get copied when the SOE gets cloned. If a new package gets rolled out to all
LPARs, or a config change needs to be made, you can have a ready history on each cloned LPAR of what SOE
version it was built from.
- Keep a backup of
major SOE releases. The mkdvd command allows you to keep the mksysb
backup of the OS in ISO format. A backup allows a quick load of the ISO image if you need to check
anything from an earlier version or even rebuild your SOE.
Global or local?
Some configurations are likely to be universal across all AIX LPARs. The software and configuration settings which ought to apply to every LPAR, ought to be in the SOE. Other changes will only apply to certain LPARs. Is every LPAR going to be a TSM client, or a DB2 or Oracle client? Does Apache get installed? What about monitoring agents, standard scripts, tuning parameters? Every change from a vanilla install is going to be either global (on the SOE) or local - restricted to a limited number of LPARs. Once you understand which changes belong to which category, you're a long way towards building your SOE LPAR. The more localised software might be sitting in a common file system, perhaps on a NIM server or in a management LPAR.Learning to SOE
There are many more factors in building a SOE LPAR - it's impossible to cover them all here. But if you have a good grasp of the steps involved, you could have a new SOE LPAR built by this afternoon (except that I'm writing this late at night, Sydney time, but you get the point).* I'm not sure if "unrelocatable" is a real word, but my surname is English, so I reserve the right to invent my own dictionary.