AIX Down Under
AnthonyEnglish 270000RKFN Tags:  backup splitlvcopy multibos alt_disk_copy splitvg 2 Comments 10,503 Views
A couple of years ago I wrote about the splitvg command. This takes a volume group (VG) that has been mirrored using mirrorvg, and effectively breaks off one copy of the mirror, turning it into a point-in-time snapshot. The snapshot can then be used for reports, backups or for building a new database. I'm not sure how much this is used in the real world these days, but if you are still doing your backups this way, perhaps it's time to revisit the strategy.
Split & Resynch
Never heard of it? Here's how it works.
After you've done the split of the VG, you can continue business on the "live" copy of the volume group, and then make use of the frozen snapshot copy, for example for backups, or to run some reports. It's a little like creating an alternate root disk, or running multibos to create a standby instance of the OS. With breaking the volume group mirror, you have one live copy and one snapshot copy of the same VG. As the copy which has been snapshotted (with apologies to any grammarians) starts life as a perfect replica of the primary live copy, after you've finished with the snapshot and want to resynchronise the two copies again, the process is usually faster than when you break a volume group mirror altogether using unmirrorvg. Of course, the longer the database is up on the primary copy without its mirror, and the more updates are done to that live database, the more data there will be to synchronise when you're ready to turn the snapshot back into a real mirrored copy.
I've seen this split/resynch approach used in smaller sites, especially before the days of SAN storage. One manufacturing company I worked with in the 1990s couldn't afford to shut down their database every night, do a cold backup for 3 or more hours, and then start the database again. So they did the shutdown of the database, split the mirror, and then got the database started again. Database downtime was about 10 minutes. The frozen copy was then backed up to tape. After a (hopefully) successful backup, the snapshot copy was then resynchronised, so that the two active copies of the database were back in place for the purposes of redundancy.
Back to the 90s
Now if you remember this approach from a time way back in your IT career, you may be feeling nostalgic for the 1990s when we were suspicious of Storage Area Networks (and all networks, for that matter), and when businesses relied on a nightly cold backup (and a weekly reboot: just to clear out the cobwebs). Who had ever heard the phrase "24 X 7"? Ah yes, those were the days! These days it's far more common to do backups online (without the database being shut down), or by using utilities such as SAN replication. Also, since most sites today use at the very least a small SAN, there is little value in doing software mirroring on the OS, since the SAN handles the redundancy. Nevertheless, there may still be some systems out there using the split-the-mirror approach. Often this is simply because the approach has worked for them, even if it does mean a cold backup (shutting the database down for the duration of the backup).
Splitting a Mirror Can Bring Bad Luck
Now, if you are using the split volume group strategy, and you're doing it for backups as I've just described, keep in mind two issues.
1. You will lose your redundancy from the time of running the splitvg command until you resynch the mirror again.
2. If you lose a disk, you not only lose redundancy. You also lose your backups until the disk is replaced and the mirror reinstated. This is pretty serious.
You may be able to get by for a while with losing a single disk, but that's only because you have the comfort of the data being on other disks, and at the very worst, you could revert to a backup. However, if you lose the disk and the backup, you may have a lot of explaining to do. And that's why a broken (volume group) mirror can bring you bad luck.
Update: Jim Carstensen has kindly offered some good comments. The AIX Logical Volume Manager (LVM) allows you to have three copies of a logical volume, or of an entire volume group. So you could have a three-copy mirror, split off just one copy for your snapshot, and still have redundancy by maintaining two good copies on production. It's a very good point. He also notes that even if the splitvg snapshot for backups isn't optimal, it's still better than nothing.
Look at the mirror
If you're still using this split volume group approach for your nightly backups, it may be time to look at a new strategy. Online backups, even database exports, might be preferable. Otherwise, losing a disk may mean losing a database, and leave you with no recent backup to rely on.
AnthonyEnglish 270000RKFN Tags:  file_systems paging_space logical_volume_manager chfs delivering_baby volume_group obstetrics_down_under multibos dump_devices size rootvg lvm aix 15,737 Views
TRACING YOUR ROOTS
Every AIX system has a
volume group (a logical group of disks) called rootvg, which is created automatically at
the time you install AIX. The rootvg may be made up of one or several physical disks or virtual disks (SAN LUNs, Logical volumes on the VIO Server). Whatever "disks" make up rootvg, it's
important to keep the rootvg file system footprint small, but give the total disk assigned to rootvg enough
space to take giant steps. Let me explain.
After you've done an initial base AIX install, it's usually best to create your own new logical volumes and file systems in a separate volume group from rootvg, provided you've got the disk available. A separate data volume group (datavg) is better, unless your data and application file systems (and LVs) are relatively small.
It's important to keep your rootvg fairly lean.
That will speed up mksysb backups, recovery of the operating system,
migrations to new releases or cloning systems. Even though those upgrade and recovery activities are fairly rare, when they do happen you want to lose as little time as possible.
So the rule I (generally) follow is: rootvg for the OS, data file systems in their own volume group(s).
… large steps
Although it's sensible to keep your non-AIX data away from rootvg, it's even more important to be able to have lots of unused space within rootvg or the ability to increase disk allocation to rootvg quickly. Among other things, it's handy to have some head room in rootvg for:
Digging around the rootvg
To check how much available space you have in rootvg, use the lsvg command and run lsvg rootvg. Look at Free PPs to see how much rootvg disk is available for expansion.
Increase rootvg dynamically
If your rootvg “disk” is actually virtual, such as a SAN LUN or a logical volume on the VIO server, then it usually can be expanded on the SAN (or using extendlv on the VIOS) and then recognised on the AIX LPAR using the -g flag of the chvg command:
That "brief" introduction was a high level view of why rootvg can be hungry for disk. The rest is details, so you can
STOP READING NOW!
Unless you want a little more explanation on those steps above or are curious about something you won't find in a Redbook.
STEP BY STEP
File system growth
This should be obvious enough. If you
keep your apps and data in datavg or other volume groups, rootvg
should remain fairly stable. The /usr file system may even be sitting
at 98% and only get increased automatically when you install software
using AIX utilities such as smitty
or installp. Still, it's good to have some room for log files and temporary files. With disk space, prevention is better than cure.
Shrinking file systems
On recent releases of AIX, reducing a file system can be done using chfs without unmounting it. For example, this command will reduce /usr by 10 GB:
You need to have enough wriggle room to do this, though, so that's where the Free PPs are important.
AIX patching and installing new software
When you install patches such as AIX fix packs from the IBM Support Portal, you can either Apply or Commit the software updates. If you apply them, you can reject them, effectively rolling them back. Applying (rather than committing) software updates is a smart interim measure, but it does take a little more space. You'll obviously also need some room for when you install other software that gets written into the rootvg file systems.
System dump devices
System updates with multibos
If you haven't realised how easily you can have a dual boot on the same disk using multibos, you really are missing out on a very simple way of patching AIX while minimising downtime. My compatriot Chris Gibson has two excellent articles on AIX Updates with Multibos and Working With multibos. Easy to learn, and worth knowing, and in some ways an easier alternative to cloning the entire rootvg using alt_disk_install.
Nothing can bring an AIX system to its
knees as effectively as running out of paging space. Although paging
space (swap space) doesn't have to be in the rootvg, it's always
worth having some room for expansion of paging space. You will need
to reassess your paging space requirements if you increase your
memory allocation using Dynamic
LPAR or Changing
a partition profile's properties.
The mksysb command can do a backup of the rootvg by writing it to a file. Similarly, the mkcd and mkdvd commands back up the rootvg, but they need to create temporary file systems first. You can use flags to indicate an alternate volume group to use, but if you don't, then guess which one needs the extra space?
Learning more about LVM
If you want to learn more about LVM, have a look at a Wiki by the indefatigable Nigel Griffiths, where he deals with LVM theory and practice.
Not in the Redbooks
If you got this far, congratulations. Since you've obviously got nothing better to do with your time, let me explain why I couldn't resist including a graphic of baby feet.
I delivered a baby for the first time on Friday (you won't find a Redbook on that) after standing by over the years for the births of our five other children. Couldn't have done it without my wife. An obstetrician was there. He suggested I add "delivering babies" to the hobbies section of my resume.