NIMADM and alt_disk on DLPAR disks

Answer

For experienced system administrators who are looking for a NIMADM solution to alt_disk and DLPAR operations, the following steps can be used as a workaround.

References:

Here’s a list of useful alt_disk and NIMADM documents you might want to read before continuing with this:

1. Introduction to Alt_Cloning on AIX 6.1 and AIX 7.1

http://www-01.ibm.com/support/docview.wss?uid=isg3T1011055

2. Using NIM Alternate Disk Migration (NIMADM)

http://www-01.ibm.com/support/docview.wss?uid=isg3T1012571

3. Managing multiple instances of altinst_rootvg and applying fixes to them

http://www-01.ibm.com/support/docview.wss?uid=isg3T1024618

5. NIM setup guide

http://www-01.ibm.com/support/docview.wss?uid=isg3T1010383

4. IBM Redbook – NIM A-Z

http://www.redbooks.ibm.com/abstracts/sg247296.html?Open

The DLPAR operation allows the dynamic addition, movement or removal of resources without having to reactivate the partition or power down the operating system. This in turn leads to no interruption in the service.

You can dynamically add physical I/O slots to a running LPAR using the HMC (Hardware Management Console), which allows you to add I/O capabilities to a running LPAR without having to shut it down.

This also includes disks. If a user dynamically adds disks, the # bootinfo –B command will not indicate that the disk is bootable until the system goes through an IPL (reboot). This is one limitation of dynamic partitioning. As a result, this issue impacts the alt_disk and NIMADM commands, because they run the # bootinfo –B command in the background to verify the “bootability” of the disk that is the target of the operation. If the bootability cannot be confirmed, alt_disk and NIMADM will not perform the installation and the commands will return the following error:

0505-212 alt_disk_install: System checks indicate that hdisk0 may not be bootable.

This may be because the disk's parent adapter does not support boot, or it does support boot but was dynamically added (in which case the disk should still be bootable).
If you believe this disk is bootable, re-execute this command with the "-g" (ignore boot check) flag.
0505-187 nimadm: Error cloning altinst_rootvg on client.

· Workarounds applicable for alt_disk

· Workarounds applicable for NIMADM

· DLPAR operations

· All different flags for the alt_disk command, requirements & limitations

· All different flags for the NIMADM command, requirements & limitations

The best way to fix this would be to simply reboot the system with the dynamically added disk. However, this is not always an option, especially when dealing with production servers.
When using the alt_disk method, the –g flag is applicable, which ignores the boot check against the target disk and allows the operation to continue as normal.
However there is no –g flag in the NIMADM command options.

Below are 2 workarounds that can be used with the NIMADM command to bypass the boot check of the target disk.

All the testing has been performed in the following test environment:
Chance (NIM client) – AIX 6.1 TL9 SP9
Casinobso (NIM master*) – AIX 7.2 TL1 SP2

*The NIM master is only used to run the NIMADM operation on the NIM client, "Chance"

1. Mksysb to mksysb Migration

One possible workaround is to take a mksysb of the NIM client LPAR "chance", do a mksysb-to-mksysb migration on the NIM master, move the mksysb back to the NIM client and restore it with alt_disk_mksysb.

For this operation you must have a mksysb resource defined on the NIM master and migrate it using an lpp_source and SPOT.

If you do not have a mksysb backup and resource on the NIM master, you can do it with the following command:

# nim –o define –t mksysb –a server=master –a source=chance -a mk_image=yes –a mksysb_flags=ie
–a location=/export/mksysb/chance_61_mksysb chance_61_mksysb

Using SMIT:

# smitty nim_mkres
> Select mksysb = a mksysb image

On the following screen, give the mksysb resource a name, select the master as server of resource, specify a location to store the mksysb, set to create system backup image and choose the client to backup:

* Resource Name [chance_61_mksysb]
* Resource Type mksysb
* Server of Resource [master]
* Location of Resource [export/mksysb/chance_61_mksysb]
CREATE system backup image? Yes
NIM CLIENT to backup chance

Once you get the following prompt:

0512-038 savevg: Backup Completed Successfully.

You can verify that the mksysb resource is on the system with the following command on the NIM master:

# lsnim | grep chance
chance machines standalone
chance_61_mksysb resources mksysb

I can see the NIM client definition, as well as the 6.1 mksysb I just captured.
You can also verify the integrity of the mksysb with the restore command:
# restore –Tqvf /full/path/to/mksysb | tail -2

You will see the following prompts:
Cluster 51200 bytes (100 blocks).
Volume number 1
Date of backup: Tue Aug 1 08:49:13 2017
Files backed up by name
User root
0 ./tmp/6815962.mnt0
total size: 7649274888

This means that the mksysb is good and ready to be migrated. If you get asked for Volume 2 – this means that the mksysb is corrupt and needs to be re-taken.

When you have the mksysb resource ready for migrating with the appropriate lpp_source and SPOT levels, the following command can be used for mksysb-to-mksysb migration:

# nimadm –T chance_61_mksysb –O /export/mksysb/chance72 –s 7200_01_02_spot –l 7200_01_02_base –j nimvg –Y

(there is no SMIT menu for mksysb-to-mksysb migration)

NOTE 1: When doing a mksysb-to-mksysb migration, you need to have a caching volume group on the NIM master and use the –j flag (-j nimvg in the example above).

Once the NIMADM finishes with the below messages:

+------------------------------------------------------------------+
Executing nimadm phase 11.
+------------------------------------------------------------------+
Defining NIM mksysb resource ...
New NIM mksysb resource name is "chance_72"
+------------------------------------------------------------------+
Executing nimadm phase 12.
+------------------------------------------------------------------+
Cleaning up alt_disk_migration on the NIM master.
Removing nimadm cache file systems.

You need to move the mksysb to the source LPAR (chance) and run alt_disk_mksysb with the –g flag to ignore the bootcheck:

# lspv
hdisk0 00f634865c5fb96e rootvg active
hdisk1 00f63486adf6e755 None

# alt_disk_mksysb –m /mksysb_images/chance72 –g –d hdisk1 –B
Restoring /image.data from mksysb image.
Checking disk sizes.
Creating cloned rootvg volume group and associated logical volumes.
Creating logical volume alt_hd5.
..........
Creating /alt_inst/var/adm/ras/livedump file system.
Restoring mksysb image to alternate disk(s).
Changing logical volume names in volume group descriptor area.
Fixing LV control blocks...
forced unmount of /alt_inst/var/adm/ras/livedump
..........
forced unmount of /alt_inst
Fixing file system superblocks...

Verify that there is an alternate rootvg on the system:
# lspv
hdisk0 00f634865c5fb96e rootvg active
hdisk1 00f63486adf6e755 altinst_rootvg

Now you need to simply change the bootlist and reboot the server:
# bootlist –om normal hdisk1
# shutdown -Fr

2. Do a local alt_disk copy with the –g flag, and then start the NIMADM from P2

Another approach to the problem is to do an alt_disk on the client (chance) and stop it after phase 1. Then once the altinst_rootvg is created, you can migrate it with NIMADM starting from phase 2.

The alt_disk_copy is divided into three phases:
Phase 1 – Creates the altinst_rootvg volume group, the alt_ logical volumes, the /alt_inst filesystems and restores the mksysb or rootvg data.

Phase 2 - Runs any specified customization scripts, installs updates, new filesets, fixes or bundles, copies a resolv.conf file if specified and copies files over to remain a NIM client if specified.

Phase 3 – Unmounts the /alt_inst filesystems, renames the filesystems and logical volumes, removes the alt_ logical volumes, names ODM and varies off the altinst_rootvg. If specified, also sets the bootlist and reboots the system.

The bulk of the work is done in Phase 1 (cloning of the rootvg). NIMADM runs in 12 phases, the first of which is the same as alt_disk_copy – create the altinst_rootvg and clone the rootvg data. The actual migration starts from Phase 2.

You can run the alt_disk copy with the –g flag and only execute phase 1 with the following command:

# alt_disk_copy -g -d hdisk1 -P 1
Calling mkszfile to create new /image.data file.
Checking disk sizes.
Creating cloned rootvg volume group and associated logical volumes.
Creating logical volume alt_hd5.
……….
Backing-up the rootvg files and restoring them to the alternate file system...
Phase 1 complete.

Once phase 1 completes, you can start the NIMADM on the NIM master from Phase 2. Since the default for NIMADM is to execute all phases, you need to manually type out all phases to be executed:

# nimadm -c <client> -s <spot> -l <lpp_source> -d hdiskX -P

2,3,4,5,6,7,8,9,10,11,12 –Y

Example:
# nimadm –c chance –s 7200_01_02_spot– l 7200_01_02_base –d hdisk1 –P 2,3,4,5,6,7,8,9,10,11,12 –Y

Using SMIT:

# smitty nimadm
> Perform NIM Alternate Disk Migration
On the following screen, specify the client to migrate, the resources to use, phases 2-12 and Accept new license agreements:

* Target NIM Client [chance]
* NIM LPP_SOURCE resource [7200_01_02_base]
* NIM SPOT resource [7200_01_02_spot]
* Target Disk(s) to install [hdisk1]
Phase to execute [2,3,4,5,6,7,8,9,10,11,12]
ACCEPT new license agreements? yes

Next the NIMADM will start and you can monitor the progress:

Initializing the NIM master.
Initializing NIM client robotron.austin.ibm.com.
Initializing log: /var/adm/ras/alt_mig/robotron_alt_mig.log
Starting Alternate Disk Migration.
+------------------------------------------------------------------+
Executing nimadm phase 2.
+------------------------------------------------------------------+

……

Changing logical volume names in volume group descriptor area.
Fixing LV control blocks...
Fixing file system superblocks...
Bootlist is set to the boot disk: hdisk1 blv=hd5
+------------------------------------------------------------------+
Executing nimadm phase 12.
+------------------------------------------------------------------+

Cleaning up alt_disk_migration on the NIM master.
Cleaning up alt_disk_migration on client chance.

The NIMADM has finished successfully and back on the NIM client we can see the alternate rootvg:

# lspv
hdisk0 00f634865c5fb96e rootvg active
hdisk1 00f63486adf6e755 altinst_rootvg

Quick check before we reboot the system:
# oslevel –s
6100-09-09-1717
# lppchk –vm3
<no output>

Now simply reboot the server, as the NIMADM has changed the bootlist to the migrated disk:
# bootlist –om normal
hdisk1 blv=hd5 pathid=0

# shutdown -Fr

Once the server is back up, verify it is at the correct level and that there are no broken or partially installed filesets:

# oslevel –s
7200-01-02-1717
# lppchk –vm3
<no output>

Those are two quick ways to work around the DLPAR disks bootability check if you are trying to do a NIMADM.

Thank you for the time to read through this guide. I hope you found the information both useful and helpful. If you feel there are any mistakes or inconsistencies, please email me at ted.todorov@bg.ibm.com. If there are any technical questions regarding this document, please follow support procedures and open a PMR by calling 1-800-426-7378, and select the option for software support.