For ... ahem .. undisclosed reasons, an organisation's AIX administrator has gone Missing in Action. The empty shoes are to be filled by you. You've never heard of the company "Clean Doubt" and never logged into their systems. You don't know what applications or databases they run, so you're full of questions for the handover meeting. You are introduced to the boss who has an iron handshake and a gravelly voice. His name is Mr. Clean "call me Squeaky" - and he explains that the previous sysadmin "had to be let go". "Accidents do happen", he adds with a steely smile. He hands you a piece of paper with the root passwords on it. He wants to know how his systems are running. You've got two hours to report back.
Somehow you both know that the handover session is over.
You don't have local knowledge, but after your meeting with Squeaky, you do have motivation to get that knowledge. Quickly.
If you've worked on IBM Power Systems before, you're actually a good deal ahead of the game. After all, you can check a few basic things which will give you half a clue about what the systems are running:
- Check file systems, volume groups
- File system names can give a good indication of the sort of applications that might be running
- Check processes
- the output of ps -ef will soon tell you what applications or databases are running
- Host names
- Most places seem to change their host name policy every 18 months to two years (coincidentally about the same shelf life as the management). Do the host names contain some clues as to what is running? Consider these hostnames. Some shed more light than others.
- oraprod (Oracle production)
- saptst (SAP test)
- jupiter05 (too many LPARs - wish they'd discovered some more planets)
- rs12erp (ERP system migrated from our RS/6000 server)
- brat (Brisbane Disaster Recovery AIX Test)
- Cron jobs and scripts
- Are there regularly scheduled jobs running in the cron or initiated by some other scheduler? Are there some scripts which run frequently and may be central to the healthy flow of data?
- Backups and Disaster Recovery
- Are backups being taken? At all? How often? What's the backup utility? Is there a backup checklist or some documentation?
- Is there a DR plan? Has it been tested recently? Has there ever been a disaster? Was there a recovery?
- Is there documentation which can help you get a feel for the environment? Is it reasonably up to date? Are there project managers, colleagues, anyone who can shed some light on what the system does?
- Do you have access to your predecessor's bookmarks or favourites? This alone can tell you a lot about how organised, up to date and ready to learn the person was. Did your unmentionable forerunner follow Rob McNelly's AIXchange blog? What about Chris Gibson's AIX blog? Or AIXpert? Did the unfortunate sysadmin click on the link to follow AIX Down Under on Twitter? Was that decision the right one? Was it the last one?
- NIM and management servers
- Can you find a NIM server? How about an HMC or IVM? Is there an IBM Systems Director server or maybe some other server which does system monitoring? These can give you a quick snapshot of the lay of the land, so you can get a quick (albeit superficial) grasp of what the systems are all about.
- Is there a DBA screaming for something urgent? See if you can empathise and then discreetly ask what kind of database the system is running. Are there developers who have an urgent release and users demanding to be able to log in? This sort of buzz can give you a little bit of a feel about what's important, what's hot, what's not.
- does anyone else want to cry on your shoulder about the guy who
got the chopdisappeared in mysterious circumstances? It's amazing what you learn over a cup of coffee.
Checking the versions which are running will give you an idea of how up to date the system is. It may indicate if your predecessor (whose name is never to be mentioned) was keen to keep things crisp and clean or was more interested in counting down the days and planning an early
Are there 30 LPARs, maybe some WPARs? Are there any VIO servers or do all the hosts still run with dedicated adapters? For that matter, the hardware vintage will help you know whether there's a collection of old standalone servers accrued over the centuries or a Power7 that's on the latest firmware and running PowerVM with all the bells and whistles. The prtconf command will help you here. What version is the HMC? What does the output of the VIOS command ioslevel tell you? What about the AIX levels using oslevel -s?
Much as you'd like to walk in and pick up the pieces of a system running AIX without any assistance, there is some local knowledge that would save you an immense amount of time. Well, spare a thought for your successor.
This time of year is a chance to tidy up a little, document, and provide some sort of framework for a quick handover, in the unlikely event that you're the one who goes M.I.A, courtesy of Mr. S. Clean.