AIX Down Under

PREVENTING SPOF ATTACKS

When you think about it, our first job is to keep our systems up and running. Or at least walking, crawling, even staggering. But keep them up we must. The mail must get through, the system must stay up. We build redundancy at all sorts of levels into our systems because we want to avoid that dreaded disease known as SPOF - the Single Point of Failure.

Outages are expensive

Unscheduled outages bring your computer system to its knees. They can have similar effects on your nervous system. There is the post-mortem where you hope to find the root cause and hope to prove it's not you.

But even scheduled outages - the ones we plan weeks ahead - are costly. There is the planning, begging for permission for the outage, negotiating a time which most suits the users (and most disrupts your family), the co-ordinating of key stakeholders to be on standby at an ungodly hour. Maybe there are fallback plans which need to be in place and then there is that intangible user perception that the system is unstable because they keep having to work around another weekend of downtime.

So before you plan to inconvenience maybe thousands of users by shutting down your server, ask yourself one question:

Do you really want an out?

Even if you get the agreement from the business that you can have an outage on your Power server, think of whether having a Hardware Management Console (HMC) as part of your configuration can help you to reclaim your weekends or sleep time. Here are some ways the HMC allows you to skip several of those outages:

• concurrent firmware updates when you're updating within the same release of system firmware
• firmware updates to several servers simultaneously - one HMC can do it all
  
• dual VIO servers - you can upgrade one VIOS at a time without losing your disk or network connectivity
• some hardware installation and replacement
  
• live partition mobility which lets you move a running LPAR to a different server
  
• migration of Workload Partitions (WPARs)
  

There still are circumstances when a Power System has to be brought down, but there are a lot of scheduled system outages which are entirely avoidable.

HMC or IVM?

On smaller systems, you may not need a dedicated HMC for management. Even so, it is worth thinking about having one anyway, especially when you consider the benefits of not having to schedule system outages for many key tasks. If you don't have an HMC and you use the Integrated Virtualization Manager (IVM), many of the tasks mentioned above will still require an outage. For example, with the IVM, every firmware update is disruptive - it requires a reboot of all LPARs on the system. And if you want dual VIO servers for redundancy, you must have an HMC.

Here's how the two VIO servers can work to prevent unnecessary outages.

Dual VIO servers

On an HMC-managed system, you can set up SEA Failover for the network and even have SEA Failover with VLAN tagging. When VIO Server 1 goes down, traffic can still go through VIO 2. Similarly, you can configure multiple paths to storage, as explained in this excellent presentation by Janel Barfield on Advanced Power6 VIO configurations.

Converting from IVM to HMC

Although there is no official procedure to migrate from IVM to HMC, I have done it successfully for a p6-520 and a p6-550. Rob McNelly has a post on IVM to HMC Migrations. If you're scheduling lots of outages for tasks which could be done during business hours if only you had your systems connected to an HMC, you might find the total savings of avoiding those outages more than justifies the purchase of a single HMC to manage multiple Power systems.

Install X-Windows without the pane

However hardline you are about the command line, there may just be a time when you need to use X-Windows. Alright, you might not actually need to, but it might just be good to know how to set it up anyway. The Windows menu utility WebSM comes in handy sometimes. I used it recently for the aixpert tool. It allowed me to have a granular view of security settings in a clickable menu. You might also need X-Windows to install other products, such as Apache or WebSphere.

The more you think about it, X-Windows can be helpful. Here's how I set it up so I could run WebSM:

1. install an X server on your PC
2. enable X11 forwarding in Putty
3. enable X11 forwarding for the SSH daemon
4. restart SSH daemon
5. text X session using xclock
   
6. run WebSM
   

1. Install an X server on your PC.

There are lots of X Servers you can use. I downloaded Xming because it was free and easy to configure.

2. Change Putty config

If you are using Putty, there is an option under Connection > SSH > X11 called Enable X11 forwarding. Of course this has to be checked before you start your Putty session.

3. Change ssh daemon configuration file

Once you've change Putty, you need to do a similar thing for the SSH daemon. On the AIX LPAR itself, I had openssh installed, but this still required a change to a parameter in a config file.

cd /etc/ssh

vi sshd_config

(Not to be confused with ssh_config in the same directory)

Uncomment the entry:

#X11Forwarding no

The “no” indicates the default setting. Change it to “yes”

X11Forwarding yes

Of course this assumes that you have the Common Desktop Environment (CDE) filesets installed on your AIX system. IBM has a technote on installing X-Windows if you need to.

4. Stop and start ssh daemon

Stop and start the ssh daemon (sshd), which can be done even while you're connected via ssh:

stopsrc -s sshd   # No space after the stop

(wait a second)

startsrc -s sshd

6. Test your X session using xclock

If this works (and you don't get an error such as "Can't open DISPLAY"), you should see a tab flashing on your PC. When you open it, you will see a clock. If you're too young to know what a clock is, ask your grandparents.

7. Run WebSM from within your Putty session

The wsm command runs the admin menu WebSM:

/usr/websm/bin/wsm

You may need to enable and/or start wsmserver first.

Have a look around at WebSM. It gives you access to a few important admin menus and can be less daunting than SMIT or the command line, especially for newcomers to AIX.

CAREFUL, THEY MIGHT HEAR YOU

The word “auditor”, of course, comes from the Latin audire 'to hear', since audits were originally presented orally. Hearing is the auditors' profession, so there's no need to shout at them. If you say your system is secure, they're listening.

Security is your business

Many administrators see security as a burden, an obstruction to their work ... until there's a real violation. Your company, and your job depend on keeping the bad guys out (and I don't mean the auditors). It isn't just malicious attacks which can compromise your systems and even wreck your business. Security is also about stopping innocent parties from doing something dangerous without meaning to do damage. By the way, those "innocent" parties might just include you, pal.

Call in the aixpert

There are so many components to system security that it's hard to keep up with them. So the nice people at IBM did us a favour and wrapped it all up into a single command that at the same time makes you feel important: aixpert (not to be confused with the excellent AIXpert Blog). Having over 300 security settings in a single command should make your job easier. Still, you need to be careful. As I explained to a manager recently: "I can secure your operating system and break your business in a single command."

Don't do it yourself

Unless you're only managing a single LPAR, it's simply mad to script or configure all the settings one by one. Use the aixpert command. That's what it was written for. There is some very helpful documentation explaining why and how to run aixpert, including:

• man pages for aixpert command
• IBM Wiki movie 54: System Hardening with aixpert
  
• aixpert - Hints & Tips wiki
• Ken Milberg's developerWorks article Using AIX Security Expert
• AIX V6 Security Features Redbook
• AIX Virtual User Group Webinar: October 29, 2009 - AIX Security presented by Bhargavi Reddy
  

Interface to the expert

You can configure aixpert security settings for an LPAR

• via the command line
  or
  
• IBM Systems Director (pConsole) (see IBM Wiki Movie 13)
  or
• WebSM
  or
  
• using SMIT (known as smitty to his friends), but that is only for a lock-up-the-lot-of-'em no-exceptions approach to security.
  

If you want to see each rule in detail and are not keen on editing XML files, WebSM provides a more granular view than the other interfaces. WebSM allows you to select or deselect a rule in a menu with common rules grouped together (e.g. Password Policy, Login Policy, Tuning Network Settings). To use WebSM you will need to set up an X-Session.  The recommended way, though, is by running aixpert on the command line. You can do this in conjunction with XML files provided for the AIX Security Expert.

XML for aixpert

Available settings
ALL available aixpert settings which you can choose from are in the file /etc/security/aixpert/core/aixpertall.xml This file includes the level of security you have selected, a description of each of the rules and the actual commands which are run for each rule.
 
Applied settings
All the settings you last applied via aixpert are saved into an XML file called /etc/security/aixpert/core/appliedaixpert.xml

Security levels
If you choose a security level (high / medium / low or default, which is lower than low), then only those rules for that level will be applied. (Each rule name starts with a three-character prefix i.e. hls= high level security, mls = medium, lls = low, dls = default). You could then create a set of these rules (e.g. high level), save it to an XML file and then strip out or tailor that list of rules for your system. You can also run a check on your appliedaixpert.xml file (if it exists) to see which security rules fail to comply. These would be the rules you either want to fix or explain to the auditors why they are not applicable for your environment.

XML for smarties
Having all your settings in a single XML file allows you to edit the file, view it in a spreadsheet or XML editor, and distribute the same rules to other LPARs and apply them using aixpert -f filename.xml. This also enables you to retain a history of XML files, provided you save them off to some other file name or back them up. It's a smart way of doing it.

Warning! Warning! Warning!

Although the aixpert tool is undoubtedly helpful, here are some initial findings which I have yet to confirm, but which are worth being aware of as potential problems.

• there is no XML snapshot of the system's current settings if you have made changes without using aixpert. If you did use aixpert, the new settings should be in appliedaixpert.xml
• you may want to apply default level security before applying any higher security levels from your XML file. This provides you with some sort of baseline.
• The undo option doesn't seem to roll back recursively. It looks to save only one previous version of the security settings.
  
• WebSM only displays all available settings (from aixpertall.xml). It doesn't show you which of the settings you have actually changed.
  
• You can't see what your current security level is set to except via WebSM (or try fpm -s)
  
• If you want to get very granular, turning off lots of rules, it's hard work and you need to explain the rules you have disabled to the auditors.
  
• Migration and patching overwrites your security settings according to that Webinar linked to above. After moving to a new version of AIX you would need to apply the previous settings again using the aixpert -f option and the XML file which should have been saved off as part of the AIX upgrade.
  
• Setting SOX-CORBIT level doesn't include settings which are already covered by High security.
  

Is it worth it?

With all those difficulties, is it still worth using aixpert? Yes, I would say so. Each of the warnings has a simple enough workaround , and saving XML files as you go is usually it. The question is not whether aixpert is worth the trouble. It's whether not using it is worth the risk. Don't tell me your system is secure; tell the auditors. They're paid to listen.

A NEW ADDITION

You have a p5 or p6 which has been running fine for 2, 3 or 4 years but new applications or business expansion mean you're getting a brand new server as well. You have an HMC which is fairly up to date. How quickly can you get the server going, ready to install your first LPAR?

E.T.A.: "pretty quickly"

Assuming you've done your homework on sizing the system, cabling and power needs etc, you can get your system online pretty quickly.

1. Rack-mount the server. This should be by far the slowest step. The reason is simple: you can't virtualise a rack.
2. Connect it to power.
3. Connect cable to the HMC. This may be a direct connection to the HMC, or better still to a switch.
4. Log onto HMC and "Add Managed System." You may use DHCP to discover the server.
5. Power on server via the HMC.
6. Update System Firmware if you need to. HMC and System firmware updates can be found at the IBM Support Portal.
   

At this point your server should be listed in "STANDBY" mode. You can Create a Partition running VIO Server or AIX or Linux. If you're running with one or two VIO servers, you can install them first so they can provide I/O to the partitions. Once your first partition is activated, the system will go from STANDBY to Available.

You may need to connect cables to storage, once again probably via a switch rather than directly to the storage subsystem. You will also have to Create Logical Partitions and either build them from scratch or restore via a mksysb.

It's always fun powering on a brand new server. It's not difficult at all and it's a chance to get familiar with the new technology before it has to be installed for its real job - not your playground, but the benefit of the business.

High Level high[-er] availabilty options

IBM HACMP (High Availability Cluster Multiprocessing) has been available since 1991.  It was renamed to IBM PowerHA and, more recently PowerHA System Mirror for AIX. While studying up on this recently, I came across some excellent comparisons between PowerHA, Live Partition Mobility and migration of Worload Partitions (WPARs). This is a high-level comparison of how the three might help with managing outages and make for higher availability, even if they aren't all PowerHA.

LPM and HA

First, Live Partition Mobility (LPM). This is the facility which allows you to migrate a running partition with its applications from one physical server to another without disrupting services. The Redbook for IBM PowerVM Live Partition Mobility explains:

.Live Partition Mobility increases global availability, but it is not a high availability solution. It requires that both source and destination systems be operational and that the partition is not in a failed state. In addition, it does not monitor operating system and application state and it is, by default, a user-initiated action.

Unplanned outages still require specific actions that are normally executed by cluster solutions such as IBM PowerHA.

So LPM is good if the outage is planned.

"Live" for a reason

The difference between HA and LPM came home to me recently when I was watching a presentation by an IBMer on PowerHA. Shawn Bodily has worked on HACMP/PowerHA for over ten years. He was on the team that wrote the excellent PowerHA for AIX Cookbook and he presented webinars (see below) over three days in July 2009 on PowerHA (as it was then called). In the second of those days, Shawn answers a question about LPM as an alternative to PowerHA. Here is my transcript of the relevant section:

"It is called Live Partition Mobility for a reason. You are not moving a dead partition.

[With PowerHA] Partition fails: we start the services automatically on another system.

Live Partition Mobility you perform a move from one frame to another frame. That partition is running for the whole time. If the partition is dead [this doesn't work] - that's why it's not called dead partition mobility."

Unplanning your outages

You may need to rethink how you plan your outages. Shawn made these crucial observations about downtime:

WPARs for AIX update

So what can you use when you want to minimise your (planned) outage when you update AIX? Our oft-quoted Nigel Griffiths answers this in a post on the AIXpert blog about the savings of WPARS over LPARs. Points 5 and 6 are are real gems.

5. Maintenance of one Global AIX plus syncwpar command is much less work than say updating 40 copies of AIX

6. Especially if you run WPAR buckets:This means instead of running say 40 LPARs you run three Global AIX LPARs for different 3 AIX 6 Technology Levels that you need - WPARs can be move between these as they are updated to high TL's.  Eventually, you empty the lowest one and remove it. Then create a new Global AIX for the next new TL as it becomes available.

 

AIX Webinars

If you're interested in seeing Shawn Bodily's PowerHA webinars, go to the AIX Virtual User Group where you can download them. You can also learn lots about:

• PowerHA
• AIX
• LPM
• NIM
• NMon
• VIOS and
• Lots of other acronyms

These are all worth watching, but be careful. If you're getting close to your monthly download limit, make sure you save some morsels of bandwidth for reading AIX Down Under. If you miss one of our gripping posts, you'll need a note from your mum.

What? A dump?

AIX, like many other operating systems, provides the capability of making a system dump. This is nothing to do with the Windows Recycle bin or a Trash folder, a kind of holding place in case you want to undelete something. ("Undelete"? Who invented that word?). This IBM technote on Managing System Devices explains that you need a system dump which "automatically copies selected areas of kernel data to the primary dump device."

Care factor zero?

Now "automatically copying selected areas of kernel data" may not be right up there on the scale of life-changing news, so let's put it in layman's terms: you need to store a system dump in the event of a

catastrophic operating system failure

First the good news ...

You almost certainly already have a system dump in place. Use the command sysdumpdev -l to list the primary dump device. If you prefer to avoid the command line you can use smitty dump.

sysdumpdev -l

primary              /dev/lg_dumplv
secondary            /dev/sysdumpnull
copy directory       /var/adm/ras
forced copy flag     FALSE
always allow dump    FALSE
dump compression     OFF

In AIX 5.x the default dump device was /dev/hd6, except for servers or LPARs which at installation time had more than 4GB of memory. Then the default dump device was /dev/lg_dumplv. That's probably what you see as the primary dump device now.

... and now the bad news

bad news #1: Your dump device may be too small

Your dump device may not be big enough to capture a system dump. You can check the error report or run the dumpcheck command which should be scheduled in the cron to run once a day anyway. You'll probably need to specify the full path, which is /usr/lib/ras/dumpcheck. If nothing appears in the error report (errpt command), then at least one of your dump devices is large enough for that crash you're just waiting for. No news is good news. Aren't you glad you made enough space for rootvg?

bad news #2: dump won't prevent first crash

Having a system dump device is not going to prevent that catastrophic operating system failure. At least it won't for the first time you crash. What the dump does give you is something you can send off to your second level support to find out what caused that failure so that you can hopefully prevent it happening again.

bad news #3 (don't think I'm enjoying this): you may need a secondary dump device

If your primary dump device becomes unavailable, usually because the disk it sits on is lost somehow, you may need to have a secondary dump device. You probably don;t have one. If it's set to /dev/sysdumpnull then you're completely dependent on your primary dump device.

It's easy to set up a dump device: create a logical volume(smitty lv)of type sysdump and then use the command sysdumpdev -s to set it as your secondary device. The smitty dump menus give lots of helpful information.

Clean up this dump!

For a step-by-step explanation of how to manage system dumps, have a look at that IBM technote which was mentioned above.

If you need to configure or increase the size of a dump device, it's easy to do. Hopefully it will never be used, but if that system crash does happen, you'll be glad you have something which will help diagnose the root cause.

"AIX7 OB boarding now"

Are you someone who is always behind the times? Do you buy the last ticket on the last train into the evacuated city? Well, take off your N minus ten leadweight and join the discerning  passengers of AIX Down Under on our first flight on AIX 7 Open Beta.

AIX 7 Open Beta is available from a free download in ISO format, which means you could burn it onto a DVD or (better still) use the VIO server virtual media library. Of course you'll need to boot off the DVD / virtual optical device. It's a mksysb restore.

Use Existing architecture

Don't think you have to buy a Power 770 to install AIX 7. Of course it works on Power7, but It even runs on p4, p5 or p6 hardware. For this trial we've used a p5 520. The installation took about half an hour (techo time - give or take an hour). After the simple install, the oslevel command returned:

7100-00-00

Test flight stopover: WPAR administration

This is a short test flight, and apart from the oslevel, the first thing we noticed which is different from 6.1 is a new SMIT menu for Workload Partition (WPAR) Administration, called Versioned Workload Partitions. If you've heard anything about AIX 7, you'd know that it supports WPARs running AIX 5.2. This is only supported on Power7 hardware but may enable you to decommission that ancient server which is holding you back on that N-10 train. Even if versioned partitions are not going to fulfil a need in your environment, why not take a look at WPARs?

WPARs are software partitions available from AIX 6.1. That means you don't need an HMC or IVM to create or manage them. You don't even need a VIO server. In fact, we were able to create a system WPAR on a standalone p5 server using a serial console.

For a good introduction to WPARs and whether they would suit your environment, have a look at this article on AIX 6.1 Workload Partitions on IBM developerWorks.

"Thanks for flying"

We were just getting started and it's already time to touch down. For more information, my colleague and compatriot, Chris Gibson, provides all the interesting links and details in his blog: AIX 7 Open beta - it's time to test.

There are still plenty of customers on AIX 5.3 who are missing out on many of the features of 6.1 because no one managed to try it out. Don't be left behind this time. Even if you don't end up migrating to AIX 7 when the commercial version is officially released, you'll be glad you have at least been exposed to a beta version. Your feedback may also be influential in the final product. That's got to be a much better way of spending your resources than waiting for that train.

POPULAR PUTTY 

When it comes to connecting to your AIX system, Putty is a very popular option. It allows you to use an ssh connection, with cut-and-paste facilities and the ability to customise your sessions. You download Putty for free from the Putty Download Page and there are lots of sites out there which can step you through a simple setup.

Having watched how others use Putty, I can say there are one or two simple features which many people don't seem to be aware of.

Rectangle block selection : alt and mouse drag

You don't need to cut and paste entire lines if all you want is a column or two: just use alt while you use your mouse to select a rectangular block. As the Putty documentation explains:

(‘Rectangular block’), dragging the mouse between two points defines a rectangle, and everything within that rectangle is copied.

Here's how a rectangular selection looks:

You don't need to limit yourself to a single column or to white space. You can cut and paste any rectangle within your Putty session screen. You can even change this to be the default behaviour, so you don't need to use alt.

Backspace default

The second tip is so obvious, it's hardly worth mentioning, except that so many people overlook it. It's setting the backspace key to what suits your terminal type. If you constantly use ctrl-v or the stty command after logging in, maybe it's time for you to go to Terminal > Keyboard > The Backspace Key and set it to Control-H.

Keep longer history

The default scrollback history for a Putty session is 200 lines. Considering that commands such as lslpp, du and find may display thousands of lines, it may well be worthwhile upping the scrollback history. Session > Window > Lines of scrollback. If your Putty log file is going to blow your disk space on your PC, we at AIX Down Under are too delicate to comment on your desktop support team.

Change Default Settings

As well as having a list of sessions which can connect to various hosts, there is an entry in Putty for Default Settings. It is there to allow you to set your backspace, scroll history, colours, window size etc. once as the default instead of each time you set up a new host. Of course it won't change any existing host configuration settings. They are, by definition, already customised. But it's worth doing for future sessions you will create when you configure a new host to connect to.

Putty Documentation

The Putty documentation page gives a wealth of information about this free and popular client for ssh, telnet and rlogin. Have a read, then have a play.

Using (or not using) NIM

NIM - the AIX Network Installation Manager - allows you to install and maintain multiple hosts from the one master. Among people newer to AIX, NIM is often left on the backburner. With the VIO Virtual Media Library, a good deal of the NIM master's functionality has alternatives, at least if you're building LPARs which are VIO clients on the same system as the VIO server itself.

NIM still has its uses - especially for providing a source for building across to a different server or a different site. Some AIXers wouldn't be without NIM. Others tend either to live without NIM entirely or build an initial NIM server and never touch it again.

Part of the hesitation about NIM is confusion about some of its components, so it may be worth knowing of some helpful sources for information about NIM.

NIM numb?

If you're a little NIM numb and in need of being unconfused, an excellent start is the AIX Virtual User Group Webinars. There you can download a 90 minute session on NIM basics and another one on more advanced topics.
June 23 & 24, 2010 - NIM Basics and NIM Advanced by Steve Knudson

What's the SPOT?

In those presentations, Steve Knudson explains what on earth is meant by SPOT and lpp_source. Here's my transcription along with my comments:

The lpp_source is a directory to hold installp images from the AIX media.

The spot - shared product object tree - is a /usr file system that we create from the lpp_source. Clients use commands from the spot during the install process. When the network image first comes down and the client first begins to install, he [you've got to admire someone who calls LPARs "he"] has to run commands, and the commands aren't all included in the boot image. He's got to run mkvg to create a root volume group. He has to run mklv to create logical volumes, mkfs to create file systems. He finds those commands in an NFS-mounted SPOT. That's why we have a /usr NFS-mounted file system as well as an lpp_source.

The Webinars cover lots of other explanations and provide some very helpful presentation materials.

NIM Resources

If you're after a little more info on NIM, you could have a look at this article by Jaqui Lynch from the IBM Systems Magazine called Simplifying with NIM. Jaqui points out that

"NIM basically provides a central point of management for installing and maintaining AIX images for both LPARs and individual servers."

She also shows how NIM can take care of mksysb images for multiple LPARs and servers over several different sites. Have a look at her more recent article on the same site Centralize AIX LPAR and Server Management with NIM.

The IBM Redbook NIM from A to Z in AIX 5L is still valuable, even though it was written for AIX 5.3 and we're now testing the AIX 7 Open Beta.

(Incidentally, since NIM must be at the latest level, you can't install the AIX 7 Open Beta mksysb from a lower level NIM server. You can follow this issue on the AIX 7 Open Beta user forum).

(The link above to the AIX 7 Open Beta forum was incorrect. It should work now. - Anthony)

Update from AIX 7 Open Beta forum:

We have provided NIM filesets to enable network installation. Please see the Download tab on the AIX 7 Open Beta website Link:https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/aix7ob/index.shtml An updated "Getting Started" guide (v1.4) containing instructions for NIM fileset installation has also been provided on the Library tab.

https://www.ibm.com/developerworks/forums/thread.jspa?threadID=334967&tstart=0