I was attempting to configure a new installation of lpar2rrd (version 2.65) for a customer today when I encountered the following error during the initial load of data from the HMC:

 

lpar2rrd@nim [/home/lpar2rrd/lpar2rrd]$ ./load.sh

LPAR2RRD version 2.65

HMC            : hmc01

managed system : 750-1 (type_model*serial : 8233-E8B*0646D0P)

HMC version    : 77301

There is no RRD: attempting to do initial load, be patient, it might take some time

sample rate    : 60 seconds

fetching HMC   : hmc01:750-DR CPU lpar config

fetching HMC   : hmc01:750-DR CPU/MEM global config

sys change     : hmc01:750-DR:100

installing WWW : install-html.sh 2.59

An error occured in lpar2rrd, check ./error.log and output of load.sh

 

$ tail -3 ./error.log

 

Argument "" isn't numeric in numeric le (<=) at /home/lpar2rrd/lpar2rrd/lpar2rrd.pl line 532.

Can't use string ("") as a SCALAR ref while "strict refs" in use at /home/lpar2rrd/lpar2rrd/lpar2rrd.pl line 533.

Wed 23 Nov 12:04:09 2011 : hmc01 : An error occured in lpar2rrd.pl, check ./error.log and output of load.sh

 

We were able to get this working by hacking the lpar2rrd.pl script.

 

The following code:

 

    if ( $loadhours <= 0 || $loadmins <= 0) {   # workaround as this sometimes is negative , need to check it out ...

      my $t1 = localtime($$last_rec);

      my $t2 = localtime($t);

      print STDERR "$act_time: time issue 1   : $host:$managedname $loadhours $loadmins \n";

      print STDERR "  Last saved record (HMC lslparutil time) : $$last_rec ; HMC time : $t \n";

      print "$act_time: time issue 1   : $host:$managedname $loadhours $loadmins \n" if $DEBUG ;

      print "  Last saved record (HMC lslparutil time) : $$last_rec ; HMC time : $t \n" if $DEBUG ;

      $loadhours = 3;

      $loadmins  = 140;

    }

 

Was changed to:

 

      $loadmins  = 140;

    if ( $loadhours <= 0 || $loadmins <= 0) {   # workaround as this sometimes is negative , need to check it out ...

      my $t1 = localtime($$last_rec);

      my $t2 = localtime($t);

      print STDERR "$act_time: time issue 1   : $host:$managedname $loadhours $loadmins \n";

      print STDERR "  Last saved record (HMC lslparutil time) : $$last_rec ; HMC time : $t \n";

      print "$act_time: time issue 1   : $host:$managedname $loadhours $loadmins \n" if $DEBUG ;

      print "  Last saved record (HMC lslparutil time) : $$last_rec ; HMC time : $t \n" if $DEBUG ;

      $loadhours = 3;

    }

 

The initial load then ran successfully.

 

Then we changed the code back to its previous configuration:

 

    if ( $loadhours <= 0 || $loadmins <= 0) {   # workaround as this sometimes is negative , need to check it out ...

      my $t1 = localtime($$last_rec);

      my $t2 = localtime($t);

      print STDERR "$act_time: time issue 1   : $host:$managedname $loadhours $loadmins \n";

      print STDERR "  Last saved record (HMC lslparutil time) : $$last_rec ; HMC time : $t \n";

      print "$act_time: time issue 1   : $host:$managedname $loadhours $loadmins \n" if $DEBUG ;

      print "  Last saved record (HMC lslparutil time) : $$last_rec ; HMC time : $t \n" if $DEBUG ;

      $loadhours = 3;

      $loadmins  = 140;

    }

After which, all subsequent loads worked as expected! Fortunately my colleague, Nathan Edwards, had already hit this bug a while ago and was able to help me. Thanks again Nathan!

 

According to Pavel Hampel (lpar2rrd developer), the next version of lpar2rrd (2.66) will fix this issue.

 

I came across this strange LPM issue recently. Thought I’d share it with you.

 

All the customers VIOS were configured with the viosecure level set to high. 

When the VIOS is configured with a security profile set to high, PCI or DoD, a new feature is enabled during LPM. This new feature is called “Secure LPM”. As a result when you initiate an LPM operation, the “Secure LPM” feature automatically enables the VIOS firewall and configures a secure (ipsec) tunnel for all LPM traffic over the network.

Before we started LPM, the VIOS firewall is off:

 

$ viosecure -firewall view

IPv4 Firewall OFF

 

                          ALLOWED   PORTS

           Local   Remote

Interface  Port    Port    Service      IPAddress       Expiration

Time(seconds)

---------  ----    ----    -------      ---------       ---------------

 

And it's activated after we start LPM:

 

$ viosecure -firewall view

IPv4 Firewall ON

 

                          ALLOWED   PORTS

           Local   Remote

Interface  Port    Port    Service      IPAddress       Expiration

Time(seconds)

---------  ----    ----    -------      ---------       ---------------

In the customer’s case, we moved a partition from a 795 running a pair of VIOS at 2.2.1.3 to another 795 running another pair of VIOS at 2.2.2.1. This worked OK. When we attempted to move the partition back, we received an error stating that the source MSP had rejected the request.

HSCLA230 

The mover service partition on the source managed system has rejected the request to stop the migration. Verify that the migration state of the partition is Migration Starting, and try the operation again.

After a while we discovered that when the LPM operation started, the source VIOS would enable its firewall and we would immediately loose connectivity (i.e. our SSH session to the VIOS would hang). We also found that RMC connectivity between the source VIOS and HMC also dropped. Somehow the VIOS firewall was blocking network connectivity to pretty much everything. 

Eventually, after much digging we found that the source VIOS was the victim of an errant firewall rule (configured on the source VIOS itself). 

# smitty ips4_conf_filter

List IP Security Filter Rules

1 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 no udp eq 4001 eq 4001 both both no all packets 0 all 0 none  Default Rule

2 *** Dynamic filter placement rule for IKE tunnels *** no

3 deny 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 y all any 0 any 0 all both inbound no yes 0 no 0 patt_none

0 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 yes all any 0 any 0 both both no all packets 0 all 0 none  Default Rule

We removed this rule (after some discussion with IBM support).

# smitty ips4_conf_filter

Delete IP Security Filter Rules

3 deny 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 y all any 0 any 0 all both inbound no yes 0 no 0 patt_none

# smitty ips4_conf_filter

List IP Security Filter Rules

1 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 no udp eq 4001 eq 4001 both both no all packets 0 all 0 none  Default Rule

2 *** Dynamic filter placement rule for IKE tunnels *** no

0 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 yes all any 0 any 0 both both no all packets 0 all 0 none  Default Rule

We were then able to move the partition back to the original frame. We are still searching for an answer to how this happened. The IBM support team believe the rule must have been created by an administrator at some point. Of course, the administrator claims that is untrue.

If you come across the following LPM issue with IVM and JS22 blades,

IZ63357: LPM HUNG AT 100% ON BOTH SOURCE AND DESTINATION IVMS

Error description

The problem that these customers are seeing is the same

As described in this particular defect.

Migrating mobile partition from one IVM to another:

The destination IVM GUI shows the migration is complete,

but the percent complete is "unknown".

The source IVM GUI shows the "migration status" as

Migration Complete and "percent Complete" as 100%.

Once the recover operation is performed, the status shows

that the LPAR  is now "Running" on the target and removed

from the source, as expected.

There are two PMRs with the same issue and the

Environment is the same on both the cases:

* Firmware level FSP: EA340_075

* IVM VIOS Level: 2.1.1.10-FP21

* Machine Type:   JS22 blade

Local fix

Run the "Recover" operation as the Migration appears to

be "hung". Once the recover is complete, the status shows

that the LPAR is now "Running" on the target and removed

from the source, as expected.

 

Then I recommend you request the following ifix from IBM support:

 

IZ63357F21.091110.epkg.Z

 

From IBM support:

 

“It appears that this problem is due to a timing issue.

The trace data shows that IVM is calling the migmover -m end command on the destination partition. That command in turn calls a system call. When everything is in the correct state, that system call will end the migration and cleanup. The problem is it is not being called because the mover has not yet been notified that VASI is in the END state as indicated by the mover's state being: MIG_STATE_COMPLETED. The trace shows this notification arriving a split second after the migmover -m end command was issued.

On a good migration, the mover is notified by VASI of the state change before IVM calls migmover -m. IVM runs migmover -m end once it gets a migration completed message from phyp. If this message is issued about the same time as the message that tells VASI that
the migration has ended, a timing issue seems a real possibility.

The problem was fixed in the VIOS code. A new call was added when the "end" state change comes in from IVM/HMC. If the state is complete we are good to go, otherwise we issue an error.”

 

Installing this fix on my VIOS resolved the problem. My VIOS are running 2.1.1.10-FP21. If you are running FP22 for VIOS 2.1, then you must request a different ifix from IBM support.

The lsdevinfo utility appears to an undocumented command. However it looks like it could come in handy occasionally. Here are some examples of using it on AIX 7.1:

 

# oslevel -s

7100-00-01-1037

 

# lslpp -w /usr/sbin/lsdevinfo

  File                                        Fileset               Type

  ----------------------------------------------------------------------------

  /usr/sbin/lsdevinfo                         bos.rte.methods       File

 

# lsdevinfo -h

 

Usage:

lsdevinfo [-q Criteria] [-F Format] [-R] [-c] [-h]

 

# lsdevinfo | more

device:

        name = "sys0"

        status = "1"

        chgstatus = "1"

        ddins = "planar_pal_chrp"

        aixloc = ""

        uniquetype = "sys/node/chrp"

        class = "sys"

        subclass = "node"

        type = "chrp"

        prefix = "sys"

        devid = "chrp"

        base = "1"

        hasvpd = "0"

        detectable = "0"

        busext = "0"

        fru = "1"

        led = "2065"

        catalog = "devices.cat"

        setno = "1"

        msgno = "1"

        driver = "planar_pal_chrp"

        define = "/usr/lib/methods/define_rspc -n"

        configure = "/usr/lib/methods/cfgsys_chrp"

        change = "/usr/lib/methods/chggen"

        unconfigure = "/usr/lib/methods/ucfgdevice"

        undefine = "/usr/lib/methods/undefine"

        start = ""

        stop = ""

        inventory = "0"

        syscons = "/dev/vty0"

        sysconsp = ""

        modelcode = "0x0800004c"

        keylock = "normal"

        maxbuf = "20"

        maxmbuf = "0"

        maxuproc = "128"

        autorestart = "true"

        iostat = "false"

        realmem = "6291456"

        conslogin = "enable"

        fwversion = "IBM,AM710_058"

        maxpout = "8193"

        minpout = "4096"

        fullcore = "false"

        pre430core = "false"

        ncargs = "256"

        pre520tune = "disable"

        max_logname = "9"

        platform_desc = "chrp"

        rtasversion = "1"

        modelname = "IBM,9179-MHB"

........

 

 

# lsdevinfo | grep -p hdisk0

device:

        name = "hdisk0"

        status = "1"

        chgstatus = "2"

        ddins = "scsidisk"

        aixloc = ""

        uniquetype = "disk/vscsi/vdisk"

        class = "disk"

        subclass = "vscsi"

        type = "vdisk"

        prefix = "hdisk"

        devid = ""

        base = "1"

        hasvpd = "0"

        detectable = "1"

        busext = "0"

        fru = "0"

        led = "9652"

        catalog = "vscsi_init.cat"

        setno = "2"

        msgno = "0"

        driver = "scsidisk"

        define = "/etc/methods/define"

        configure = "/etc/methods/cfgscsidisk"

        change = "/etc/methods/chgdisk"

        unconfigure = "/etc/methods/ucfgdevice"

        undefine = "/etc/methods/undefine"

        start = ""

        stop = ""

        inventory = "0"

        pv = "no"

        pvid = "00f6050a2cd79ef80000000000000000"

        model_name = "VCS_DEFLT_DISK  "

        queue_depth = "3"

        max_transfer = "0x40000"

        cfgmgr_psafe = ""

        unique_id = "48333321360050768019183002800000000000F1304214503IBMfcp05VDASD03AIXvscsi"

        sn_location = "609F"

        reserve_policy = "no_reserve"

        PCM = "PCM/friend/vscsi"

        safe_relocate = "0"

        kext_errsave = "yes"

        message_no = "1"

 

 

# lsdevinfo | grep -p cluster0

device:

        name = "cluster0"

        status = "1"

        chgstatus = "1"

        ddins = ""

        aixloc = ""

        uniquetype = "pseudo/node/cluster"

        class = "pseudo"

        subclass = "node"

        type = "cluster"

        prefix = "cluster"

        devid = ""

        base = "0"

        hasvpd = "0"

        detectable = "0"

        busext = "0"

        fru = "0"

        led = "8232"

        catalog = "devices.cat"

        setno = "170"

        msgno = "1"

        driver = ""

        define = "/usr/lib/methods/define -n"

        configure = "/usr/lib/methods/cfgcluster"

        change = "/usr/lib/methods/chgcluster"

        unconfigure = "/usr/lib/methods/ucfgdevice"

        undefine = "/usr/lib/methods/undefine"

        start = ""

        stop = ""

        inventory = "0"

        clvdisk = "60050768-0191-8300-2800-000000000f15"

        node_uuid = "60510e58-be33-11e0-8387-66da93d16b18"

 

# lsdevinfo -c

name="sys0",status="1",chgstatus="1",ddins="planar_pal_chrp",aixloc="",uniquetype="sys/node/chrp",class="sys",subclass="node",type="chrp",prefix="sys",devid="chrp",base="1",hasvpd="0",detectable="0",busext="0",fru="1",led="2065",catalog="devices.cat",setno="1",msgno="1",driver="planar_pal_chrp",define="/usr/lib/methods/define_rspc -n",configure="/usr/lib/methods/cfgsys_chrp",change="/usr/lib/methods/chggen",unconfigure="/usr/lib/methods/ucfgdevice",undefine="/usr/lib/methods/undefine",start="",stop="",inventory="0",syscons="/dev/vty0",sysconsp="",modelcode="0x0800004c",keylock="normal",maxbuf="20",maxmbuf="0",maxuproc="128",autorestart="true",iostat="false",realmem="6291456",conslogin="enable",fwversion="IBM,AM710_058",maxpout="8193",minpout="4096",fullcore="false",pre430core="false",ncargs="256",pre520tune="disable",max_logname="9",platform_desc="chrp",rtasversion="1",modelname="IBM,9179-MHB",systemid="IBM,02100488P",boottype="disk",dasd_spin_delay="5",SW_dist_intr="false",pci_full_cfg="1",cpuguard="enable",frequency="6400000000",capacity_inc="0.01",capped="false",dedicated="false",ent_capacity="0.50",min_capacity="0.10",max_capacity="0.50",variable_weight="128",cod_cust_info="",cod_mail="",cod_modelsys="",cod_proc="0",id_to_system="0X800000302B900000",id_to_partition="0X800000302B900043",ngroups_allowed="128",sed_config="select",nfs4_acl_compat="secure",enhanced_RBAC="true",log_pg_dealloc="true",ghostdev="0"

........................

 

# lsdevinfo -q name="cluster0"

device:

        name = "cluster0"

        status = "1"

        chgstatus = "1"

        ddins = ""

        aixloc = ""

        uniquetype = "pseudo/node/cluster"

        class = "pseudo"

        subclass = "node"

        type = "cluster"

        prefix = "cluster"

        devid = ""

        base = "0"

        hasvpd = "0"

        detectable = "0"

        busext = "0"

        fru = "0"

        led = "8232"

        catalog = "devices.cat"

        setno = "170"

        msgno = "1"

        driver = ""

        define = "/usr/lib/methods/define -n"

        configure = "/usr/lib/methods/cfgcluster"

        change = "/usr/lib/methods/chgcluster"

        unconfigure = "/usr/lib/methods/ucfgdevice"

        undefine = "/usr/lib/methods/undefine"

        start = ""

        stop = ""

        inventory = "0"

        clvdisk = "60050768-0191-8300-2800-000000000f15"

        node_uuid = "60510e58-be33-11e0-8387-66da93d16b18"

 

Make sure you are monitoring file system space on your VIOS.

 

Why?

 

If you run out of space in the root file system, odd things can happen when you try to map virtual devices to virtual adapters with mkvdev.

 

For example, a colleague of mine was attempting to map a new hdisk to a vhost adapter on a pair of VIOS. The VIOS was running a recent version of code. He received the following error message (see below). It wasn’t a very helpful message. At first I thought it was due to the fact that he had not set the reserve_policy attribute for the new disk to no_reserve on both VIOS. Changing the value for that attribute did not help.

 

$ ioslevel

2.2.1.3

 

$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1

*******************************************************************************

The command's response was not recognized.  This may or may not indicate a problem.

*******************************************************************************

 

I enabled debug output from the padmin command line and discovered that the mkdev command was failing for some reason.

 

$ export CLI_DEBUG=33

 

$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1

AIX: "lspv -l hdisk1 2>&1 | grep 0516-320"

AIX: "mkdev  -V hdisk1  -p vhost0  -l vhdisk1 "

*******************************************************************************

The command's response was not recognized.  This may or may not indicate a problem.

*******************************************************************************

 

I entered oem_setup_env and ran the mkdev command manually. I was presented with a very clear error message describing the problem. The root file system (/) was full!

 

$ oem_setup_env

 

# mkdev  -V hdisk1  -p vhost0  -l vhdisk1

mkdev: 0514-548 Cannot perform the requested function because

                the / filesystem is full or is out of inodes.

 

# df -g /

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4           0.50      0.00  100%     7728    75% /

 

I found a 45MB core file (generated by cimserver) in the root file system. I moved the file to another file system (with plenty of free space) for further investigation.

 

# mv core /tmp/core.cg

# df -m .

Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4         512.00     45.93   92%     7727    38% /

 

Then I ran the mkvdev command successfully.

 

$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1

AIX: "lspv -l hdisk1 2>&1 | grep 0516-320"

AIX: "mkdev  -V hdisk1  -p vhost0  -l vhdisk1 "

vhdisk1 Available

AIX: "lsattr -El hdisk1 -a max_transfer |cut -d" " -f2"

 

$ lsmap -all

AIX: "lsdev -c adapter -t IBM,v-scsi-host -s vdevice -F "name" | wc -l -c"

AIX: "lsdev -c adapter -t IBM,v-scsi-host -s vdevice -F "name""

AIX: "lsdev -C -l vhost0 -F "physloc""

AIX: "lsdev -p vhost0 -F "name" | wc -l -c"

AIX: "lsdev -p vhost0 -F "name""

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost0          U9119.FHA.87654B1-V6-C21                   0x0000002e

 

VTD                   vhdisk1

Status                Available

LUN                   0x8100000000000000

Backing device        hdisk1

Physloc               U5873.001.9SS005Y-P1-C9-T1-W50060E8006D0206A-L1000000000000

Mirrored              false

 

AIX: "lsdev -C -l vhost1 -F "physloc""

AIX: "lsdev -p vhost1 -F "name" | wc -l -c"

AIX: "lsdev -p vhost1 -F "name""

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost1          U9119.FHA.87654B1-V6-C22                   0x0000002g

 

VTD                   NO VIRTUAL TARGET DEVICE FOUND

 

I found the same issue on the second VIOS i.e. a full root file system due to a core file (from cimserver). I also found no trace of a full file system event in the error report. Perhaps someone had taken it upon themselves to “clean house” at some point and had removed entries from the VIOS error log.

 

Make sure you monitor file system space on your VIOS. Who knows what else might fail if you run out of space in a critical file system.

It’s possible, with PowerVC, to manage existing AIX and Linux partitions in your environment. This may be useful for customers that have recently installed PowerVC in their environment and have existing AIX partitions. The question that is often asked is “Can I manage these existing partitions with PowerVC?”. These existing partitions were deployed outside of PowerVC, often long before PowerVC was even available as product from IBM. Fortunately the answer to this question is most likely yes!

 

Before you add an existing virtual machine to PowerVC, ensure that the following requirements are met:

 

http://www-01.ibm.com/support/knowledgecenter/SSXK2N_1.2.1/com.ibm.powervc.standard.help.doc/powervc_addingpre-existingvms_hmc.html

 

Here’s an example of managing an existing partition with PowerVC. Click on the Hosts icon from the PowerVC home screen then click on the “Manage Existing Virtual Machines” icon.

 

 

You are presented with a drop-down list of all the (Power Systems) Hosts known to PowerVC. Select the appropriate host where the existing partition currently resides.

 

 

Next you’ll be asked if you want to manage any supported virtual machines that are not currently being managed by PowerVC. “Supported” (with PowerVC Standard Edition v1.2.1) means any virtual machine that is either using virtual fibre channel adapters attached to supported SAN and storage devices (typically brocade switches and v7000) and/or virtual SCSI adapters attached to disk in a VIOS shared storage pool, that is currently managed by PowerVC.

 

Or you can select a specific virtual machine (or machines) to manage. I’ve selected one specific partition I wish to manage with PowerVC. It is an AIX partition that is connected to a shared storage pool (SSP). The SSP is already managed by PowerVC but the AIX partition is not.

 

 

 

A list of virtual machines (partitions) will be displayed next. Select the partition you’d like to manage with PowerVC and click on Manage.

 

 

Under the Virtual Machines view you’ll see the partition that you selected. It will appear in an Active and Pending state.

 

 

After a minute or two, the partition will change to an Active and OK state. The virtual machine is now under the management of PowerVC. You can now stop, start, restart, resize, migrate, attach volumes to, capture and/or delete the virtual machine through PowerVC.

 

 

Just as a point of interest, you can also access the Manage Existing option from the Virtual Machines view in PowerVC.

 

 

If you need to remove a partition (virtual machine) from PowerVC management (without deleting the partition) you can do so using the following procedure. From the Hosts view, select the desired Host and under Virtual Machines, select the partition you want to un-manage.

 

 

PowerVC will prompt you to confirm the removal of the virtual machine from PowerVC management. This does not delete the virtual machine; it will continue to run without disruption.

 

 

 

The virtual machine is removed from PowerVC management.

 

 

Starting with AIX 7.1, CSM is no longer supported or available. It has been replaced by Distributed Systems Managment (DSM).   Section 5.2 of the IBM AIX 7.1 Differences Guide Redbook provides details of the new DSM capabilities.

 

Fortunately DSM still provides access to the dsh command.  I’ve written about how I’ve used this utility in the past. The new dsh command (and other tools) are provided in the new DSM filesets named dsm.core and dsm.dsh.

 

root@nim# lslpp -L dsm*

  Fileset                      Level  State  Type  Description (Uninstaller)

  ----------------------------------------------------------------------------

  dsm.core                   7.1.1.2    A     F    Distributed Systems Management

                                                   Core

  dsm.dsh                    7.1.0.0    C     F    Distributed Systems Management

                                                   Dsh

 

root@nim# lslpp -w /usr/bin/dsh

  File                                        Fileset               Type

  ----------------------------------------------------------------------------

  /usr/bin/dsh                                dsm.dsh               Symlink

 

 

These filesets are NOT installed by default. You must manually install them. They can be found on your AIX 7.1 media.

 

If dsh is something you use, then I recommend you read the section on DSM in the Redbook. Also take a look at section 5.2.7 Using DSM and NIM, in which it describes how you can integrate DSM and NIM and completely automate the installation of AIX:

 

“The AIX Network Installation Manager (NIM) has been enhanced to work with the Distributed System Management (DSM) commands. This integration enables the automatic installation of new AIX systems that are either currently powered on or off.”

 

Although I’ve written about the dsh command before, there’s one usage I’ve not covered.  And that is using dsh to manage users across a group of LPARs. In particular, changing a user’s password.

 

Before I go any further, I should state that for the following to work you must first configure ssh keys on your NIM master (or central mgmt AIX system) so that you can communicate with all of your AIX systems via SSH, as root,  without being prompted for a password. Read my article on dsh to find out how to do this if necessary.

 

In the following example, I use dsh from my NIM master. It is my central point of control for my AIX environment.

 

My ssh keys for root on my NIM master have been generated and distributed to all of my LPARs.

 

root@nim# ssh-keygen -d

Generating public/private dsa key pair.

Enter file in which to save the key (/.ssh/id_dsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /.ssh/id_dsa.

Your public key has been saved in /.ssh/id_dsa.pub.

The key fingerprint is:

ed:18:e9:00:37:13:7c:7c:74:6a:a9:e0:ad:c0:09:a9 root@nim

The key's randomart image is:

+--[ DSA 1024]----+

|     ... .. .    |

|   .  ..o .+     |

|  o . =. .+      |

| . o = = =       |

|E   + o S .      |

|     . + +       |

|      . o .      |

|                 |

|                 |

+-----------------+

 

root@nim# ls -ltra

total 40

-rw-------    1 root     system          214 17 Sep 2010  authorized_keys

drwxr-xr-x    7 root     system         4096 16 Nov 11:43 ..

-rw-r--r--    1 root     system         3615 16 Nov 12:04 known_hosts

-rw-r--r--    1 root     system          601 16 Nov 12:06 id_dsa.pub

-rw-------    1 root     system          672 16 Nov 12:06 id_dsa

drwx------    2 root     system          256 16 Nov 12:06 .

 

On my AIX LPARs, the authorized_keys file has been updated with the public ssh key from my NIM master:

 

root@aixlpar1 [/.ssh]# chmod 600 known_hosts authorized_keys

root@aixlpar1 [/.ssh]# ls -ltr

total 0

-rw-------    1 root     system          406 Sep 29 15:30 known_hosts

-rw-------    1 root     system          601 Nov 16 12:07 authorized_keys

 

root@aixlpar1 [/.ssh]# cat authorized_keys

ssh-dss AAAAB3NzaC1kc3MAAACBAPJpekVQgV2CTu5tCOsukwXwMrkfBabLAN7vwjgPk3S2p65COJkFwaDRWwBoK87cnMrrc7/GuOLdk/RiqYw1K8Z0d9RWnsymYrNm0MBALWioGGXwQ+YlLiYHIQsSArnajZOH6Ye36qz81uWNoj6u4xIxi/iGSYoU4x3UE/c/H833AAAAFQDKEtWVFk3G+KAyxUsI6FcpbJZqDQAAAIAVUVRak/s+X0mMElCaEs8Rh96GDfhorXWV1YzI0nbRo8QX2AMix+3LSdEmYCRn8TYpk1Udln3GG4gzy5oI/3Rd1RTWgAo+067LwytR6IJfwYt219Pwh2ktrm5iZu3c+yDBqQj6hsga2YIun5lxqYFy39CZNtslnR1wTx9uCcYkowAAAIEAuVFTsW9ugVM76CgluspNo0faz9xFo4DCOjCgye+k/ZqSDaNq6Xrvxktw+dkHuKHbhvofPgeiMrlqhdbrE0ovLUr8aadxArRhitnu10CFYbRwlrjAxyAaRwjD2pXG8M3a5D1XXAWDCDm6bEs5yxJqcl1wq+8UAhpKiS5yj2pwApk= root@nim

 

 

On the NIM master, the root user was configured for the DSH environment. The following entry was placed in roots .profile:

 

root@nim# cat /.profile

ENV=$HOME/.kshrc

 

The following entry was placed in roots .kshrc file:

 

root@nim# cat /.kshrc

export DSH_NODE_RSH=/usr/bin/ssh

export DSH_NODE_LIST=/usr/local/etc/nodes

 

A /usr/local/etc/nodes file was created on the NIM master. This file contains a list of each of the nodes that dsh can communicate with from NIM:

 

root@nim# cat /usr/local/etc/nodes

aixlpar1

aixlpar2

aixlpar3

aixlpar4

aixlpar5

aixlpar6

aixlpar7

aixlpar8

aixlpar9

aixlpar10

aixlpar11

 

The first time that the dsh command is run against a new host, the following message will be displayed. dsh uses the FQDN, and the FQDN needs to be added to the known_hosts file for ssh. Therefore you must make an ssh connection first with FQDN to the host:

 

root@nim# dsh uptime

aixlpar1.cg.com.au : Host key verification failed.

dsh:  2617-009 aixlpar1.cg.com.au  remote shell had exit code 255

 

It is necessary to ssh directly to each node using its FQDN. This step is only required once for each node. For example:

 

root@nim# ssh aixlpar1.cg.com.au

The authenticity of host 'aixlpar1.cg.com.au  (172.1.6.17)' can't be established.

RSA key fingerprint is 83:13:1d:cb:97:ee:00:7a:04:75:62:98:bd:e8:30:79.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'aixlpar1.cg.com.au ' (RSA) to the list of known hosts.

Last unsuccessful login: Tue Sep 27 03:04:33 GMT+10:00 2011 on ssh from 172.1.6.15

Last login: Wed Nov 16 12:08:08 GMT+11:00 2011 on /dev/pts/3 from nim

 

 

root@aixlpar1 [/root]# exit

Connection to aixlpar1.cg.com.au  closed.

 

Now we can use the dsh command to run a command across all the nodes defined in the DSH_NODE_LIST. For example, the following command will run the uptime command on all nodes:

 

root@nim# dsh uptime

aixlpar1.cg.com.au :   12:14PM   up 10 days,  22:32,  3 users,  load average: 0.47, 0.53, 0.64

aixlpar2.cg.com.au :   12:14PM   up 21 days,  21:13,  1 users,  load average: 0.11, 0.31, 0.51

aixlpar3.cg.com.au :   12:14PM   up 15 days,  20:33,  5 users,  load average: 0.17, 0.43, 0.54

...etc...

 

With dsh now configured, we can use it to perform several admin tasks from a central location. For example, the following commands will do the following:

 

1.       Create a user, named cg, on all nodes.

2.        Set cg’s password to abc123 using the chpasswd command.

3.        Remove the ADMCHG flag for the user cg. This will stop them from being prompted to change their password on their first login attempt.

 

First I check that user does not exist on the AIX systems.

 

root@nim# dsh lsuser cg

aixlpar1: 3004-687 User "cg" does not exist.

aixlpar2: 3004-687 User "cg" does not exist.

aixlpar3: 3004-687 User "cg" does not exist.

aixlpar4: 3004-687 User "cg" does not exist.

aixlpar5: 3004-687 User "cg" does not exist.

...etc...

 

I create the user with the mkuser command.

 

root@nim# dsh mkuser cg

 

I confirm that the user has been created on all the AIX systems, with the lsuser command.

 

root@nim# dsh lsuser cg

aixlpar1: cg id=204 pgrp=staff groups=staff home=/home/cg shell=/usr/bin/ksh login=true su=true rlogin=true daemon=true admin=false sugroups=ALL admgroups= tpath=nosak ttys=ALL expires=0 auth1=SYSTEM auth2=NONE umask=22 registry=files SYSTEM=compat logintimes= loginretries=0 pwdwarntime=0 account_locked=false minage=0 maxage=0 maxexpired=-1 minalpha=0 minother=0 mindiff=0 maxrepeats=8 minlen=0 histexpire=0 histsize=0 pwdchecks= dictionlist= fsize=-1 cpu=-1 data=262144 stack=65536 core=2097151 rss=65536 nofiles=2000 roles=

aixlpar2: cg id=204 pgrp=staff groups=staff home=/home/cg shell=/usr/bin/ksh login=true su=true rlogin=true daemon=true admin=false sugroups=ALL admgroups= tpath=nosak ttys=ALL expires=0 auth1=SYSTEM auth2=NONE umask=22 registry=files SYSTEM=compat logintimes= loginretries=0 pwdwarntime=0 account_locked=false minage=0 maxage=0 maxexpired=-1 minalpha=0 minother=0 mindiff=0 maxrepeats=8 minlen=0 histexpire=0 histsize=0 pwdchecks= dictionlist= fsize=-1 cpu=-1 data=262144 stack=65536 core=2097151 rss=65536 nofiles=2000 roles=

aixlpar3: cg id=204 pgrp=staff groups=staff home=/home/cg shell=/usr/bin/ksh login=true su=true rlogin=true daemon=true admin=false sugroups=ALL admgroups= tpath=nosak ttys=ALL expires=0 auth1=SYSTEM auth2=NONE umask=22 registry=files SYSTEM=compat logintimes= loginretries=0 pwdwarntime=0 account_locked=false minage=0 maxage=0 maxexpired=-1 minalpha=0 minother=0 mindiff=0 maxrepeats=8 minlen=0 histexpire=0 histsize=0 pwdchecks= dictionlist= fsize=-1 cpu=-1 data=262144 stack=65536 core=2097151 rss=65536 nofiles=2000 roles=

 

I set the users password to abc123, using the chpasswd utility. I also remove the ADMCHG flag so that the user is not prompted to change their password on their first logon attempt.

 

root@nim# dsh 'echo cg:abc123 | chpasswd -c'

 

I confirm that I can logon with the new user with the specified password, on one of the AIX LPARs.

 

root@nim# ssh cg@aixlpar1

cg@aixlpar1’s password:

Last login: Thu Mar  1 20:05:01 CST 2012 on /dev/pts/1 from aix71

$ id

uid=204(cg) gid=1(staff)

 

Another nice feature of dsh is the dshbak utility. This utility presents formatted output from the dsh command. For example:

 

root@nim 520 [/.ssh]# dsh errpt | dshbak

HOST: aixlpar1.cg.com.au

------------------------------------

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION

AA8AB241   1116110811 T O OPERATOR       OPERATOR NOTIFICATION

A6DF45AA   1104135011 I O RMCdaemon      The daemon is started.

2BFA76F6   1104134111 T S SYSPROC        SYSTEM SHUTDOWN BY USER

9DBCFDEE   1104134111 T O errdemon       ERROR LOGGING TURNED ON

 

HOST: aixlpar2.cg.com.au

-------------------------------

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION

DE9A52D1   1111012611 I S rmt10          AAA1

4865FA9B   1111012211 P H rmt10          TAPE OPERATION ERROR

DE9A52D1   1110233511 I S rmt0           AAA1

4865FA9B   1110225511 P H rmt0           TAPE OPERATION ERROR

DE9A52D1   1109180311 I S rmt0           AAA1

4865FA9B   1109180011 P H rmt0           TAPE OPERATION ERROR

DE9A52D1   1108180411 I S rmt2           AAA1

4865FA9B   1108180211 P H rmt2           TAPE OPERATION ERROR

DE9A52D1   1108165711 I S rmt6           AAA1

4865FA9B   1108165111 P H rmt6           TAPE OPERATION ERROR

A2205861   1102085311 P S SYSPROC        Excessive interrupt disablement time

F7FA22C9   1031134111 I O SYSJ2          UNABLE TO ALLOCATE SPACE IN FILE SYSTEM

DE9A52D1   1030163411 I S rmt0           AAA1

4865FA9B   1030163411 P H rmt0           TAPE OPERATION ERROR

....etc....

 

WARNING: Please be VERY CAREFUL when using the dsh command. Issuing the wrong command can cause damage to all your AIX LPARS!

 

The dsm.dsh package contains the following utilities:

 

# lslpp -f dsm.dsh | grep /usr/bin

                        /usr/bin/dcp -> /opt/ibm/sysmgt/dsm/bin/dcp

                        /usr/bin/dsh -> /opt/ibm/sysmgt/dsm/bin/dsh

                        /usr/bin/dping -> /opt/ibm/sysmgt/dsm/bin/dping

                        /usr/bin/dshbak -> /opt/ibm/sysmgt/dsm/bin/dshbak

 

If you are a fan of the dping command, you are going to be disappointed. Although the command is currently included in the dsm.dsh fileset, it probably won’t be for much longer.

 

The command works, “sort of”:

 

root@nim# dping aixlpar1

aixlpar1: ping (alive)

 

But if you run ‘dping –a’:

 

root@nim# dping -a

dping: 2651-095 CSM license has expired or has not been accepted. Run csmconfig -L if you have installed a new release.

 

According to the developers, dping is no longer supported and will eventually be removed from the DSM package. The response from the  developers was as follows:

"The reason "dping -a" is failing with the license check is because the command is calling “/usr/bin/runact-api –c IBM.DmsCtrl::::isLicenseValid" and the license is not set. So the command fails.  Since CSM is not supported anymore and went end of life. "

“... please consider the dping command as being "deprecated" code pending removal from the dsm.dsh package.”

When I asked why the command was listed in the AIX 7.1 online documentation if it was no longer available, I was informed: “We are  in the process of working with component owner regarding the DOCs and updating them.”. At this stage I’ve not been able to find an alternative command (in AIX). If I find one, I’ll update this post.

 

If you are planning on migrating to AIX 7.1 please be aware that CSM is no longer supported or available with AIX 7.1. CSM is now ‘end of life’.

 

 

http://www.ibmsystemsmag.com/aix/tipstechniques/systemsmanagement/DSH-Simplifies-the-Task-of-Managing-Multiple-Syste/

 

http://www.redbooks.ibm.com/abstracts/sg247910.html?Open

 

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.cmds/doc/aixcmds1/chpasswd.htm

 

I like using the IBM Microcode Discovery Service (MDS) to perform spot checks on POWER system and adapter firmware levels. This tool gives me snapshot of a POWER systems current firmware levels. I can tell almost immediately if I need to update the firmware on the system or any device installed in the server. This report is especially useful when performing a health check on a customer’s system.

 

The tool is easy to use. First you need to download the latest inventory scout catalog file from the IBM website. The most current microcode catalog can be downloaded here:

 

ftp://ftp.software.ibm.com/software/server/firmware/catalog.mic

 

Once downloaded, copy the catalog file to an AIX (or VIOS) partition on the physical server. In most cases, a Virtual I/O Server (VIOS) is the most logical place to copy this file. The VIOS typically owns all the physical adapter devices on the system.

 

# scp catalog.mic padmin@vio1:

padmin@vio1's password:

catalog.mic                                         100%  263KB 263.2KB/s   00:00

 

Next, login to the VIOS and move the new catalog file into the correct location for use with the inventory scout (invscout) tool.

 

# ssh padmin@vio1

padmin@vio1's password:

$ ls -ltr

total 968

drwxr-xr-x    3 padmin   staff           256 May 17 21:20 tivoli

-rw-r--r--    1 padmin   staff          2389 Aug 19 04:43 smit.transaction

-rw-r--r--    1 padmin   staff          1290 Aug 19 04:43 smit.script

-rw-r--r--    1 padmin   staff         11247 Aug 27 11:57 smit.log

drwxr-xr-x    2 padmin   staff          4096 Sep 15 15:26 SV810_081

-rw-r--r--    1 root     system         5103 Sep 15 15:53 install.log

drwxr-xr-x    2 padmin   staff           256 Sep 15 15:53 fixes

-rw-r--r--    1 padmin   staff        176510 Sep 15 18:34 errlog.out

drwxrwxr--    2 root     staff           256 Sep 16 13:25 config

-rw-r-----    1 padmin   staff        269478 Sep 17 20:18 catalog.mic

-rw-r--r--    1 root     staff          5143 Sep 17 20:18 ioscli.log

$

$ r oem

oem_setup_env

# ls -ltr

total 8

-rw-r--r--    1 root     system           88 May 17 20:26 catalog.mic

 

# cd /var/adm/invscout/microcode

# mv catalog.mic catalog.mic.old

# cp /home/padmin/catalog.mic .

# ls -ltr

total 536

-rw-r--r--    1 root     system           88 May 17 20:26 catalog.mic.old

-rw-r-----    1 root     staff        269478 Sep 17 20:20 catalog.mic

#

 

Now all we need to do is run the invscout command to create a new MDS upload file (MUP file).

 

# invscout

 

******  Command  ----  V2.2.0.20

******  Logic Database V2.2.0.2

 

Initializing ...

Identifying the system ...

Working ...

Getting system microcode level(s) ...

Scanning for device microcode level(s) ...

 

117 devices detected; each dot (.)

represents 10 devices processed:

...........

 

Writing Microcode Survey upload file ...

 

Microcode Survey complete

 

The output files can be found at:

Upload file: /var/adm/invscout/vio1.mup

Report file: /var/adm/invscout/invs.mrp

Report file: /var/adm/invscout/invs.mrrup

 

To transfer the invscout 'Upload file' for microcode

comparison, see your service provider's web page.

#

 

The new MUP file (vio1.mup) can be uploaded to the MDS tool at the following URL:

 

https://www-304.ibm.com/webapp/set2/mds/fetch?page=mdsUpload.html

 

In the following screenshot I’ve selected the file to be uploaded. Then I clicked on ‘Upload File’.

 

 

 

I’m presented with the MDS report screen.

  

 

 

Immediately my eyes are drawn to the items highlighted in red. Both the PCIe2 4-Port (10GbE SFP+ & 1GbE RJ45) and PCIe3 RAID SAS Adapter have new firmware available for them. I should update the firmware on these devices! But what fixes are included in the new firmware? Well, directly below the devices report is the ‘Microcode by Type’ table.

 

 

 

I can simply click on the ‘Readme’ hyperlink in the report and a new browser window opens with the readme file for the adapters firmware.

 

 

 

So, should I install these fixes right away? How critical are they? Again, the MDS report provides you with a quick way to answer these questions. Under the ‘Severity’ column, you will see several terms that describe the severity of the related fix. If you click on the word ‘Severity’ you’ll be presented with an ‘Understanding the MDS Report’ window.

  

 

On this page you’ll find the definition and description of each severity type. This will make it easier for you to ascertain if a fix should be installed ASAP or if it can wait until the next maintenance window.

 

 

I often save the MDS report as a PDF and include it in my health check reports. Alternatively you can save the entire page as a HTML document and send this to the customer. This will ensure the hyperlinks still work and can be referenced by the customer as needed e.g.

 

file:///C:/data/MDS/CustomerName%20Microcode%20Discovery%20Service.htm.

 

By the way, if the microcode catalog file you used to generate the MDS report is not the latest, the MDS tool will warn you. If you see this message, simply download the latest file and send it over to the server where it’s needed and re-run invscout and upload the new MUP file e.g.

 

“WARNING: the microcode catalog used in surveying host vio1 is now out of date. Version 2014.09.10 was used for the survey and version 2014.09.17 is the most current. We recommend that you either download and install the latest microcode catalog, and run this program again, or run the MDS Applet and allow it to automatically update the catalog.”

 

MDS and Inventory Scout

https://www-304.ibm.com/webapp/set2/mds/fetch?page=invreadme.html

 

 

Last year I wrote an article on using nimadm to migrate to AIX 6.1.

The procedures outlined how to migrate a live system with nimadm and a spare disk.

Another method that I have found to be extremely useful is the migration of mksysb images from one version of AIX to another using (you guessed it) nimadm.

In the following example I migrated an AIX 6.1 mksysb image to AIX 7.1 and then used that image to install a new LPAR via NIM.

The source LPAR was running AIX 6.1 TL6 SP3. I took a mksysb of this system to my NIM master.

# oslevel -s

6100-06-03-1048

#

 

The NIM mksysb resource was named cg-aix61

 

root@nim1 / # lsnim -t mksysb

vio1-mksysb     resources       mksysb

cg-aix61        resources       mksysb

 

root@nim1 / # lsnim -l cg-aix61

cg-aix61:

   class         = resources

   type          = mksysb

   arch          = power

   Rstate        = ready for use

   prev_state    = unavailable for use

   location      = /export/mksysb/cg-aix61

   version       = 6

   release       = 1

   mod           = 6

   oslevel_r     = 6100-06

   alloc_count   = 0

   server        = master

   creation_date = Thu Jan 6 10:00:22 2011

   source_image  = cg

root@nim1 / #

 

To migrate the mksysb image from AIX 6.1 to AIX 7.1, I ran the following nimadm command:

 

root@nim1 / # nimadm -T cg-aix61 -O /export/mksysb/cg-aix71 -s spotaix7tl0sp2 -l aix7tl0sp2 -j nimadmvg -Y -N cg-aix71

 

The flags indicate the following:

 

-T specifies the existing AIX 6.1 NIM mksysb resource.

-O specifies the output location for the migrated mksysb resource.

-s indicates the AIX 7.1 NIM SPOT resource for the migration.

-l indicates the AIX 7.1 NIM lpp_source resource for the migration.

-j identifies the volume group that will be used on the NIM master to create file systems.

-Y Agrees to required software license agreements for software to be installed.

-N specifies the name of the new AIX 7.1 NIM mksysb resource to be created after migration.

 

Before running this command I made sure that I had installed the bos.alt_disk_install.rte fileset into my AIX 7.1 SPOT. If this fileset was not installed I would have received the following message immediately after issuing the nimadm command:

 

0042-001 nim: processing error encountered on "master":

   /usr/bin/lslpp: Fileset bos.alt_disk_install.rte not installed.

 

0505-204 nimadm: SPOT spotaix7tl0sp2 does not have bos.alt_disk_install.rte installed.

0505-205 nimadm: The level of bos.alt_disk_install.rte installed in SPOT

spotaix7tl0sp2 (0.0.0.0) does not match the NIM master's level (7.1.0.2).

Cleaning up alt_disk_migration on the NIM master.

 

I also had an AIX 7.1 lpp_source and SPOT ready and waiting on my NIM master.

 

root@nim1 / # lsnim -t lpp_source

aix61tl6sp3     resources       lpp_source

openssh         resources       lpp_source

aix7tl0sp2      resources       lpp_source

 

root@nim1 / # lsnim -t spot

vio1-spot           resources       spot

spotaix61tl6sp3     resources       spot

spotaix7tl0sp2      resources       spot

 

OK, back to my nimadm operation. Upon initiating the migration I saw the following output on my screen (click here for a copy of the complete log file):

 

root@nim1 / # nimadm -T cg-aix61 -O /export/mksysb/cg-aix71 -s spotaix7tl0sp2 -l aix7tl0sp2 -j nimadmvg -Y -N cg-aix71

Initializing the NIM master.

Verifying alt_disk_migration eligibility.

Initializing log: /var/adm/ras/alt_mig/cg-aix61_alt_mig.log

Starting Alternate Disk Migration.

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 1.

+-----------------------------------------------------------------------------+

Processing target mksysb resource cg-aix61.

Restoring /image.data from mksysb image.

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 2.

+-----------------------------------------------------------------------------+

Creating nimadm cache file systems on volume group nimadmvg.

Checking for initial required migration space.

Creating cache file system /cg-aix61_mm_alt/alt_inst

Creating cache file system /cg-aix61_mm_alt/alt_inst/admin

Creating cache file system /cg-aix61_mm_alt/alt_inst/home

Creating cache file system /cg-aix61_mm_alt/alt_inst/opt

Creating cache file system /cg-aix61_mm_alt/alt_inst/tmp

Creating cache file system /cg-aix61_mm_alt/alt_inst/usr

Creating cache file system /cg-aix61_mm_alt/alt_inst/var

Creating cache file system /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 3.

+-----------------------------------------------------------------------------+

Syncing mksysb data to cache ...

 

.....etc......

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 8.

+-----------------------------------------------------------------------------+

Creating client boot image.

bosboot: Boot image is 45709 512 byte blocks.

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 9.

+-----------------------------------------------------------------------------+

Adjusting client file system sizes ...

Adjusting size for /

Adjusting size for /admin

Adjusting size for /home

Adjusting size for /opt

Adjusting size for /tmp

Adjusting size for /usr

Adjusting size for /var

Adjusting size for /var/adm/ras/livedump

Backing up cache data to mksysb file /export/mksysb/cg-aix71 ...

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 10.

+-----------------------------------------------------------------------------+

Unmounting client mounts on the NIM master.

forced unmount of /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

forced unmount of /cg-aix61_mm_alt/alt_inst/var

forced unmount of /cg-aix61_mm_alt/alt_inst/usr

forced unmount of /cg-aix61_mm_alt/alt_inst/tmp

forced unmount of /cg-aix61_mm_alt/alt_inst/opt

forced unmount of /cg-aix61_mm_alt/alt_inst/home

forced unmount of /cg-aix61_mm_alt/alt_inst/admin

forced unmount of /cg-aix61_mm_alt/alt_inst

Removing nimadm cache file systems.

Removing cache file system /cg-aix61_mm_alt/alt_inst

Removing cache file system /cg-aix61_mm_alt/alt_inst/admin

Removing cache file system /cg-aix61_mm_alt/alt_inst/home

Removing cache file system /cg-aix61_mm_alt/alt_inst/opt

Removing cache file system /cg-aix61_mm_alt/alt_inst/tmp

Removing cache file system /cg-aix61_mm_alt/alt_inst/usr

Removing cache file system /cg-aix61_mm_alt/alt_inst/var

Removing cache file system /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 11.

+-----------------------------------------------------------------------------+

Defining NIM mksysb resource ...

New NIMmksysb resource name is cg-aix71

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 12.

+-----------------------------------------------------------------------------+

Cleaning up alt_disk_migration on the NIM master.

root@nim1 / #

 

For a description of all the phases in a nimadm operation, I suggest you refer to my previous nimadm article or the AIX infocenter (links below).

 

Once the migration had completed, I found a new mksysb file in /export/mksysb (as expected). The file was named cg-aix71. And a new NIM mksysb resource was now available.

 

root@nim1 / # ls -ltr /export/mksysb/

total 30793928

drwxr-xr-x    2 root     system          256 Dec 21 10:40 lost+found

-rw-r--r--    1 root     system   4893952000 Dec 21 14:42 vio1-mksysb

-rw-r--r--    1 root     system   2323097600 Jan 06 10:00 cg-aix61

-rw-r--r--    1 root     system   1967411200 Jan 06 11:00 cg-aix71

 

root@nim1 / # lsnim -t mksysb

vio1-mksysb     resources       mksysb

cg-aix61        resources       mksysb

cg1-mksysb      resources       mksysb

cg-aix71        resources       mksysb

 

root@nim1 / # lsnim -l cg-aix71

cg-aix71:

   class         = resources

   type          = mksysb

   comments      = Created by nimadm on Thu Jan 6 11:00:51 EETDT 2011

   Rstate        = ready for use

   prev_state    = unavailable for use

   location      = /export/mksysb/cg-aix71

   version       = 7

   release       = 1

   mod           = 0

   oslevel_r     = 7100-00

   alloc_count   = 0

   server        = master

   creation_date = Thu Jan 6 11:00:54 2011

 

During the migration I noticed the following file systems were mounted on the NIM master.

 

root@nim1 / # df

Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4         1572864   1149640   27%    10471     8% /

/dev/hd2         5242880   1597696   70%    42812    19% /usr

/dev/hd9var      1048576    250456   77%     8260    23% /var

/dev/hd3         8912896   8287752    8%      111     1% /tmp

/dev/hd1          524288    523520    1%        5     1% /home

/dev/hd11admin     524288    523488    1%        5     1% /admin

/proc                  -         -    -         -     -  /proc

/dev/hd10opt     1048576    664176   37%     7262     9% /opt

/dev/livedump     524288    523552    1%        4     1% /var/adm/ras/livedump

/dev/tftpbootlv     524288    419656   20%       10     1% /tftpboot

/dev/exportlv     524288    523408    1%       13     1% /export

/dev/lppsrclv   25690112   4383632   83%     3114     1% /export/lppsource

/dev/spotlv      8912896   4848464   46%    43450     8% /export/spot

/dev/mksysblv   38273024  11315200   71%        9     1% /export/mksysb

/dev/lv00         425984    412432    4%       25     1% /cg-aix61_mm_alt/alt_inst

/dev/lv01          65536     63376    4%       18     1% /cg-aix61_mm_alt/alt_inst/admin

/dev/lv02          65536     63384    4%       17     1% /cg-aix61_mm_alt/alt_inst/home

/dev/lv03         557056    414720   26%      201     1% /cg-aix61_mm_alt/alt_inst/opt

/dev/lv04         294912    285184    4%       21     1% /cg-aix61_mm_alt/alt_inst/tmp

/dev/lv05        4653056   4506904    4%       17     1% /cg-aix61_mm_alt/alt_inst/usr

/dev/lv06         720896    698160    4%       20     1% /cg-aix61_mm_alt/alt_inst/var

/dev/lv07          65536     63384    4%       17     1% /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

 

And in my nimadmvg volume group, there were several new logical volumes.

 

root@nim1 / # lsvg -l nimadmvg

nimadmvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

loglv02             jfslog     1       1       1    open/syncd    N/A

lv00                jfs        13      13      1    open/syncd    /cg-aix61_mm_alt/alt_inst

lv01                jfs        2       2       1    open/syncd    /cg-aix61_mm_alt/alt_inst/admin

lv02                jfs        2       2       1    open/syncd    /cg-aix61_mm_alt/alt_inst/home

lv03                jfs        17      17      1    open/syncd    /cg-aix61_mm_alt/alt_inst/opt

lv04                jfs        9       9       1    open/syncd    /cg-aix61_mm_alt/alt_inst/tmp

lv05                jfs        142     142     1    open/syncd    /cg-aix61_mm_alt/alt_inst/usr

lv06                jfs        22      22      1    open/syncd    /cg-aix61_mm_alt/alt_inst/var

lv07                jfs        2       2       1    open/syncd    /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

root@nim1 / #

 

These file systems were used to extract the mksysb locally on the NIM master and then migrate the image to AIX 7.1. I noticed the restbyname process consuming some CPU during the restore process.

 

Topas Monitor for host:nim1                     EVENTS/QUEUES    FILE/TTY

Thu Jan  6 10:15:38 2011   Interval:2           Cswitch    1155  Readch  2655.6K

                                                Syscall    3143  Writech 2664.7K

CPU     User% Kern% Wait% Idle%   Physc  Entc%  Reads       454  Rawin         0

Total     6.1  22.0   8.2  63.7    0.09  47.04  Writes     1168  Ttyout      578

                                                Forks        12  Igets        85

Network    BPS  I-Pkts  O-Pkts    B-In   B-Out  Execs         0  Namei       521

Total    7.74K    5.50    1.50   5.74K   2.00K  Runqueue      0  Dirblk       97

                                                Waitqueue   1.0

Disk    Busy%      BPS     TPS  B-Read  B-Writ                   MEMORY

Total     0.0    5.21M   383.0   2.02M   3.19M  PAGING           Real,MB    2048

                                                Faults      776  % Comp     67

FileSystem          BPS    TPS  B-Read  B-Writ  Steals     1118  % Noncomp  32

Total             2.03M  85.50   2.03M       0  PgspIn        0  % Client   19

                                                PgspOut       0

Name           PID  CPU%  PgSp Owner            PageIn      517  PAGING SPACE

restbyna   15532232 16.4 1.21M root             PageOut     816  Size,MB     512

vmmd         458766  5.5 1.19M root             Sios        969  % Used      3

lrud         262152  1.5  512K root                              % Free     97

syncd       2097296  1.1  592K root             NFS (calls/sec)

topas      16973864  0.5 1.98M root             SerV2         0  WPAR Activ    0

xmgc         851994  0.5  448K root             CliV2         0  WPAR Total    0

nimesis    12320772  0.3 2.41M root             SerV3         0  Press: "h"-help

getty       7798832  0.1  580K root             CliV3         0         "q"-quit

xmtopas     8847610  0.1  816K root

nfsd        5111964  0.1 4.75M root

xmtopasa    6947052  0.1  916K root

gil         1441836  0.0  960K root

java        4784136  0.0 57.5M pconsole

java        8716308  0.0 70.7M root

rpc.lock    6029528  0.0 1.19M root

clcomd      6160578  0.0 1.69M root

pilegc       786456  0.0  640K root

aixmibd     3997882  0.0 1.24M root

netm        1376298  0.0  448K root

topasrec    8454202  0.0 1.01M root

 

 

root@nim1 / # ps -ef | grep rest

    root 15270132  3342336   1 10:15:55  pts/1  0:00 grep rest

    root 15532232 10747910  37 10:14:02  pts/0  0:02 restbyname -x -q -f /export/mksysb/cg-aix61

root@nim1 / #

 

I didn’t encounter any issues with my nimadm operation but if I had I could have reviewed the nimadm log file located in /var/adm/ras:

 

root@nim1 /var/adm/ras/alt_mig # ls -ltr

total 856

-rw-r--r--    1 root     system       435009 Jan 06 11:00 cg-aix61_alt_mig.log

 

 

If you ever encounter problems with nimadm, I highly recommend that you re-run the operation with the –V and –D flags. This will enable verbose output, as well as debugging information which can assist you greatly in determining the root cause of your issue.

 

I was able to use the new, migrated, AIX 7.1 mksysb image to install a new LPAR.

 

AIX Version 7

Copyright IBM Corporation, 1982, 2010.

login: root

root's Password:

*******************************************************************************

*                                                                             *

*                                                                             *

*  Welcome to AIX Version 7.1!                                                *

*                                                                             *

*                                                                             *

*  Please see the README file in /usr/lpp/bos for information pertinent to    *

*  this release of the AIX Operating System.                                  *

*                                                                             *

*                                                                             *

*******************************************************************************

Last login: Thu Jan  6 13:21:04 EETDT 2011 on /dev/vty0

 

# oslevel -s

7100-00-02-1041

#

 

You’ve gotta love NIM! Take a look at the nimadm documentation for more information:

 

http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds4/nimadm.htm

 

By the way, if you are interested in migrating from AIX 4.3 to AIX 7.1, then you’ll need to consider the mksysb migration method. I’ve not done this myself (i.e. 4.3 to 7.1 migration) but the feedback I received from the AIX development team indicates that it does work as expected.

 

“... you had expressed some interest in mksysb migration.  Since we were not aware of anyone having attempted it with 7.1.  I went ahead and created a 4.3.3.75 system and created a mksysb and then used that mksysb in conjunction with nim and successfully installed another system using the 4.3 mksysb and 7.1 nim resources to perform a mksysb migration to 7.1.  I just followed the steps as noted in the Installation guide for mksysb migration. “

Refer to the following document for further information on mksysb migration with NIM.

http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.install/doc/insgdrf/mksysbmigration.htm

 

NIM from A to Z in AIX 5L Redbook.

http://www.redbooks.ibm.com/redpieces/abstracts/sg247296.html?Open

On more than one occasion I’ve written articles and blogs on how to migrate AIX systems to a new version of the AIX operating system. You can find a list these articles (and others) here. Each of the NIM articles focuses on migrating AIX systems (which are configured as NIM clients). The approach is always the same; use a NIM master to migrate the NIM clients to the next release of AIX. What they don’t discuss is how to migrate your NIM master to a new version of the OS. This is the very first step you should take before you consider migrating your clients. The NIM master should always be at the same or higher AIX level than the level you are migrating to on a client. So, in this latest article, I’ll discuss, step by step, how to migrate your NIM master from AIX 7.1 to AIX 7.2 (the latest release of the AIX OS).

 

https://www.ibm.com/developerworks/community/blogs/cgaix/resource/Migrating_NIMMaster_AIX72_VML.pdf

AIX 7.2 requires 2GB of memory to boot, but this minimum is not enforced in the LPAR profile except by Live Update (to ensure we'll be able to boot the surrogate LPAR). You can check your Minimum Memory setting in your LPARs profile by running the lparstat command (as shown below).

 

# lparstat -i | grep Memory
Online Memory                              : 4096 MB
Maximum Memory                             : 8192 MB
Minimum Memory                             : 2048 MB

 

If your partition does not meet the minimum memory profile requirement, you’ll receive the following error message when you perform a live update preview (with geninstall –p –k).

 

Checking lpar minimal memory size:

------------------------------------------

Required memory size: 2048 MB

Current memory size: 1024 MB

1430-119 FAILED: the lpar minimal memory size is not sufficient

 

You’ll need to change partition profile so that the minimum memory setting is, at least, 2048 (2GB) and then stop & start the partition for the profile update to take effect.

 

The /var/adm/ras/liveupdate/logs/lvupdlog log file will also contain error messages indicating the problem:

 

LVUPD 11/28/2016-18:52:00.716 DEBUG lvupdate_utils32.c - 6713 - lvup_check_minmem_size: Partition minimum memory size (1024 MB) on p8tlc-lvup is lower than the minimum memory size required (2048 MB).

 

LVUPD 11/28/2016-18:52:00.716 ERROR lvupdate_utils32.c - 8647 - lvup_preview: ERROR(s) while checking the current mimimal memory size against the computed required size.

 

 

 

 

A colleague was attempting to recreate a WPAR. The WPAR had previously been removed using the WPAR Manager Web GUI. This worked, or so he tells me!

 

Anyway, when he tried to create the WPAR again from the GUI, using the same WPAR name (wpar20), it failed. The error claimed that /wpars/wpar20 already existed. But it didn’t!

 

Now, before I go any further, looking back at the issue now, it is pretty obvious what the problem might have been. However, the solution was not evident at first. This proves one thing for certain: some problems cannot be resolved without TWO cups of coffee in the morning.

 

Of course, I resorted to the command line to see what was REALLY happening.

 

The mkwpar command was indeed complaining that /wpars/wpar20 already existed.

 

# mkwpar -n wpar20

mkwpar: 0960-288 The /wpars/wpar20 file system already exists.

mkwpar: 0960-417 Specify -p to create this workload partition using the existing file system data.

 

I could not find any directory or file system with that name, anywhere under /wpars or elsewhere.

 

# ls -l /wpars/wpar20

/wpars/wpar20 not found

 

# ls -l /wpars

total 0

#

 

# df –g | grep –i wpar

#

 

# lsvg | lsvg –il | grep –i wpar

#

 

I ran the mkwpar with truss to try to find out where it was finding the reference to /wpars/wpar20. At this point you have probably already determined where the problem might be!

 

The truss output pointed me in the direction of /etc/filesystems.....of course! Another “facepalm“ moment!

 

# truss mkwpar -n wpar20 > /tmp/wpar20 2>&1

# vi /tmp/wpar220

 

statx("/wpars/wpar20", 0x30008AB8, 128, 010)    = -1

kopen("/etc/filesystems", O_RDONLY|O_LARGEFILE) = 4

 

Whatever my learned colleague had done when he “removed” the WPAR with the GUI, it left entries for the WPARs file systems in /etc/filesystems.

 

# grep wpar20 /etc/filesystems | grep -v type

/wpars/wpar20:

/wpars/wpar20/home:

/wpars/wpar20/opt:

/wpars/wpar20/proc:

/wpars/wpar20/tmp:

/wpars/wpar20/usr:

/wpars/wpar20/var:

 

And after removing these entries with the rmfs command, like so:

 

# grep wpar20 /etc/filesystems | grep -v type | sed -e 's/://'

/wpars/wpar20

/wpars/wpar20/home

/wpars/wpar20/opt

/wpars/wpar20/proc

/wpars/wpar20/tmp

/wpars/wpar20/usr

/wpars/wpar20/var

 

# grep wpar20 /etc/filesystems | grep -v type | sed -e 's/://'

# cg="/wpars/wpar20

> /wpars/wpar20/home

> /wpars/wpar20/opt

> /wpars/wpar20/proc

> /wpars/wpar20/tmp

> /wpars/wpar20/usr

> /wpars/wpar20/var

> "

 

# echo $cg

/wpars/wpar20 /wpars/wpar20/home /wpars/wpar20/opt /wpars/wpar20/proc /wpars/wpar20/tmp /wpars/wpar20/usr /wpars/wpar20/var

 

# for i in $cg

> do

> rmfs $i

> done

 

We were able to create the WPAR once more without a problem.

 

# mkwpar -n wpar20

mkwpar: Creating file systems...

/

/home

/opt

/proc

/tmp

/usr

/var

Mounting all workload partition file systems.

x ./usr

x ./lib

x ./admin

x ./admin/tmp

x ./audit

x ./dev

x ./etc

x ./etc/check_config.files

x ./etc/consdef

x ./etc/cronlog.conf

x ./etc/csh.cshrc

x ./etc/csh.login

x ./etc/dlpi.conf

x ./etc/dumpdates

x ./etc/environment

x ./etc/ewlm

x ./etc/ewlm/limits

x ./etc/ewlm/trc

x ./etc/ewlm/trc/config_schema.xsd

x ./etc/ewlm/trc/output_schema.xsd

x ./etc/filesystems

x ./etc/group

x ./etc/inittab

x ./etc/magic

...etc...

 

Lessons I learnt from this were twofold, truss is (still) useful and drink more coffee before tackling a problem!

A quick tip for monitoring Multiple Shared Processor Pools with topas.

Execute topas from the AIX command line. Press the ‘p’ key to toggle on ‘pool metrics’. You will see the default pool (pool id 0) and any other pools defined on your system. In the example below, there are two pools, the default (0) and a user defined pool (1). The default display shows the LPARs in the default pool (0), which in this case is sys06_partition8.

Using the up and down arrow keys, I place the cursor on pool ID number one, in the pool column on the left-hand side and then press the ‘f’ key to toggle the focus. This changes what is displayed in the lower portion of the output. The LPAR associated with pool 1 (sys06_partition9) is shown.

# topas –C

  

 

  

It’s no secret that I like NIM. I’ve written about NIM on several occasions, in Redbooks, technical articles and on my blog. It’s a great tool for installing and maintaining AIX. Occasionally though, it can throw you an unusual error or two when things go wrong. For the uninitiated resolving these errors can be challenging.

 

Here’s a couple of NIM related questions I received recently. Both are troubleshooting queries.

 

“I'm getting the following error when trying to create a NIM lpp_source for AIX 6.1”:

 

                               Define a Resource

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                                                        [Entry Fields]

* Resource Name                               [6100-06_lpp]

* Resource Type                                lpp_source

* Server of Resource                          [master]                +

* Location of Resource                        [/export/eznim]          /

  NFS Client Security Method                  []                      +

  NFS Version Access                          []                      +

  Architecture of Resource                    []                      +

  Source of Install Images                    [/dev/cd0]              +/

  Names of Option Packages                    []

  Show Progress                               [yes]                   +

  Comments                                    []

 

...

Preparing to copy install images (this will take several minutes)...

 

0042-001 nim: processing error encountered on "master":

   0042-081 m_mk_lpp_source: a resource already exists on "master" at

location "/export/eznim/bid_ow"; due to NFS export restrictions, the

        new location "/export/eznim" may not be used

...

 

I could see what could be causing the issue but I first asked to see the contents of the /etc/exports file. Just to make sure that /export/eznim was not explicitly exported. It wasn’t.

 

# grep eznim /etc/exports

/export/eznim/vios/mksysb_image -ro,root=vio1,access=vio1

/export/eznim/bid_ow -ro,root=nim1,access=nim1

 

I suggested they re-try the operation again but this time specify the full path to the lpp_source directory, like so:

 

                               Define a Resource

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                                                        [Entry Fields]

* Resource Name                               [6100-06_lpp]

* Resource Type                                lpp_source

* Server of Resource                          [master]                +

* Location of Resource                        [/export/eznim/6100-06_lpp]          /

  NFS Client Security Method                  []                      +

  NFS Version Access                          []                      +

  Architecture of Resource                    []                      +

  Source of Install Images                    [/dev/cd0]              +/

  Names of Option Packages                    []

  Show Progress                               [yes]                   +

  Comments                                    []

 

This worked fine. And the lpp_source was created successfully.

 

#  lsnim -l 6100-06_lpp

6100-06_lpp:

   class       = resources

   type        = lpp_source

   locked      = 26738726

   Rstate      = unavailable for use

   prev_state  =

   location    = /export/eznim/6100-06_lpp

   alloc_count = 0

   server      = master

 

The second issue related to down-level filesets after an AIX 6.1 migration with nimadm.  There was also an issue with disk space that generated an error during NIM operations.

 

“Hi Chris,

 

Thanks for replying back to my post. I am a newbie to AIX/pSeries world.

I was following your document on upgrading AIX 5.3 to 6.1 using nimadm, it seemed to have worked partially for me, as I got to AIX 6100-00-00. The mksysb and the SPOT I created were for AIX 6100-06-05.

 

Before I re-attempt the upgrade, is there anything I need to be clean up on AIX 5.3 side that could have perhaps caused issues with the upgrade.

 

Thanks”

 

I recommended that the following command be run on the NIM client, to determine which filesets were down-level after the migration.

 

# oslevel -s

# instfix -i | grep AIX

# instfix -i | grep SP

 

If the output contained messages like "Not all filesets for 6100-06_AIX_ML were found", or something similar, then they should run the following command against each line (there could be more than one).

 

# instfix -icqk 6100-06_AIX_ML | grep ":-:"

 

This will tell you which filesets were not updated.

 

The output from ‘oslevel –s’and ‘instfix –i | grep AIX’ looked OK.

 

# oslevel -s

6100-06-04-1112

 

# instfix -i | grep AIX

    All filesets for 6100-00_AIX_ML were found.

    All filesets for 6100-01_AIX_ML were found.

    All filesets for 6100-02_AIX_ML were found.

    All filesets for 6100-03_AIX_ML were found.

    All filesets for 6100-04_AIX_ML were found.

    All filesets for 6100-05_AIX_ML were found.

    All filesets for 6100-06_AIX_ML were found.

 

But the output from ‘instfix –i  | grep SP’ indicated an issue with 61-06-051115_SP.

 

# instfix -i | grep SP

    All filesets for 61-00-010748_SP were found.

    All filesets for 61-00-020750_SP were found.

    All filesets for 61-00-030808_SP were found.

    All filesets for 61-00-040815_SP were found.

    All filesets for 61-01-010823_SP were found.

    All filesets for 61-00-050822_SP were found.

    All filesets for 61-00-060834_SP were found.

    All filesets for 61-01-020834_SP were found.

    All filesets for 61-00-070846_SP were found.

    All filesets for 61-01-030846_SP were found.

    All filesets for 61-02-010847_SP were found.

    All filesets for 61-02-020849_SP were found.

    All filesets for 61-00-080909_SP were found.

    All filesets for 61-01-040909_SP were found.

    All filesets for 61-02-030909_SP were found.

    All filesets for 61-03-010921_SP were found.

    All filesets for 61-00-090920_SP were found.

    All filesets for 61-01-050920_SP were found.

    All filesets for 61-02-040920_SP were found.

    All filesets for 61-00-100939_SP were found.

    All filesets for 61-01-060939_SP were found.

    All filesets for 61-02-050939_SP were found.

    All filesets for 61-03-020939_SP were found.

    All filesets for 61-04-010944_SP were found.

    All filesets for 61-04-021007_SP were found.

    All filesets for 61-04-031009_SP were found.

    All filesets for 61-00-110943_SP were found.

    All filesets for 61-01-081014_SP were found.

    All filesets for 61-02-060943_SP were found.

    All filesets for 61-03-030943_SP were found.

    All filesets for 61-03-041014_SP were found.

    All filesets for 61-04-041014_SP were found.

    All filesets for 61-05-011016_SP were found.

    All filesets for 61-01-091015_SP were found.

    All filesets for 61-02-081015_SP were found.

    All filesets for 61-03-051015_SP were found.

    All filesets for 61-04-051015_SP were found.

    All filesets for 61-01-070943_SP were found.

    All filesets for 61-02-071014_SP were found.

    All filesets for 61-02-091034_SP were found.

    All filesets for 61-03-061034_SP were found.

    All filesets for 61-04-061034_SP were found.

    All filesets for 61-05-021034_SP were found.

    All filesets for 61-06-011043_SP were found.

    All filesets for 61-02-101036_SP were found.

    All filesets for 61-03-071036_SP were found.

    All filesets for 61-04-071036_SP were found.

    All filesets for 61-05-031036_SP were found.

    All filesets for 61-06-021044_SP were found.

    All filesets for 61-06-031048_SP were found.

    All filesets for 61-05-041048_SP were found.

    All filesets for 61-04-081048_SP were found.

    All filesets for 61-03-081048_SP were found.

    All filesets for 61-03-091112_SP were found.

    All filesets for 61-04-091112_SP were found.

    All filesets for 61-05-051112_SP were found.

    All filesets for 61-06-041112_SP were found.

    Not all filesets for 61-06-051115_SP were found.

 

The culprit was the csm.hc_utils fileset from TL6 SP5. It was still at level 1.7.1.0 instead of the expected 1.7.1.1.

 

# instfix -ciqk "61-06-051115_SP" |grep ":-:"

61-06-051115_SP:csm.hc_utils:1.7.1.1:1.7.1.0:-:AIX 6100-06-05 Service Pack

 

I recommended that the csm.hc_utils 1.7.1.1 fileset be added to the AIX 6.1 lpp_source. The migration was attempted again and the client migrated to AIX 6.1 TLP6 SP5 successfully.  I also suggested they determine if this fileset was really used and/or needed (CSM is no longer supported/available starting with AIX 7.1 anyway).

 

And just when I thought all his problems were resolved, he replied with: “however upon trying to install these from the NIM server, I get errors on NIM”.

 

*********************************************************

errors:

0042-001 nim: processing error encountered on "master":                |

 |      0042-001 m_showres: processing error encountered on "master":       |

 |      0042-185 c_ch_nfsexp: Failed to link or copy /etc/.tmp.nim.exports. |

 |   12451894 to /etc/exports                                               |

 |           Check permissions and filesystem space

*********************************************************

 

Fortunately a quick look at the output from the df command highlighted the problem immediately.

He was out of space in the root file system. The very place that /etc/exports, and the NIM database (/etc/objrepos), reside.

 

# df -g

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4           2.50      0.00  100%     3388     1% /

/dev/hd2           4.50      1.74   62%    42150     4% /usr

/dev/hd9var        1.00      0.52   48%     3262     2% /var

/dev/hd3           4.00      3.84    5%       74     1% /tmp

/dev/hd1           0.50      0.37   27%      479     1% /home

/proc                 -         -    -         -     -  /proc

/dev/hd10opt       0.75      0.21   73%     5698     3% /opt

/dev/livedump      0.25      0.24    4%       17     1% /var/adm/ras/livedump

/dev/exportlv    136.25    124.03    9%    20128     1% /export

 

After cleaning up unwanted files in / and making sure there was plenty of free space, the error no longer appeared in future NIM operations.

 

 

A colleague of mine was planning to modify the max_xfer_size attribute on a couple of FC adapters in one of his AIX LPARs. As he was describing his plan to me, I asked him how he intended to back out of the change should the LPAR fail to boot after the modifications. “But, what could possibly go wrong?” he fired back. I advised him to use multibos to create a standby (backup) instance of the AIX OS, just in case. He begrudgingly did so, just to keep me happy.

 

The next day he told me the following tale.

 

He had modified the FC adapters max_xfer_size attribute as planned. First, checking the current values, for the attribute on both adapters.

 

aixlpar1 : / # lsattr -El fcs0 -a max_xfer_size

max_xfer_size 0x100000 Maximum Transfer Size True

 

aixlpar1 : / # lsattr -El fcs1 -a max_xfer_size

max_xfer_size 0x100000 Maximum Transfer Size True

 

He’d created a standby AIX instance before making changes to the adapters. He also prevented multibos from changing the bootlist to the standby boot logical volume (BLV).

 

aixlpar1 : / # multibos -sXt

Initializing multibos methods ...

Initializing log /etc/multibos/logs/op.alog ...

Gathering system information ...

 

+-----------------------------------------------------------------------------+

Setup Operation

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

Creating image.data file ...

 

He modified the FC adapters as planned.

 

aixlpar1 : / # chdev -l fcs0 -a max_xfer_size=0x200000 -P

fcs0 changed

 

aixlpar1 : / # chdev -l fcs1 -a max_xfer_size=0x200000 -P

fcs1 changed

 

aixlpar1 : / # lsattr -El fcs0 -a max_xfer_size

max_xfer_size 0x200000 Maximum Transfer Size True

 

aixlpar1 : / # lsattr -El fcs1 -a max_xfer_size

max_xfer_size 0x200000 Maximum Transfer Size True

 

 

He verified that the standby instance still held the original values for both FC adapters.

 

aixlpar1 : / # multibos -S

Initializing multibos methods ...

Initializing log /etc/multibos/logs/op.alog ...

Gathering system information ...

 

+-----------------------------------------------------------------------------+

Multibos Shell Operation

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

 

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Mounting all standby BOS file systems ...

Mounting /bos_inst

Mounting /bos_inst/usr

Mounting /bos_inst/var

Mounting /bos_inst/opt

 

+-----------------------------------------------------------------------------+

Multibos Root Shell

+-----------------------------------------------------------------------------+

Starting multibos root shell ...

Active boot logical volume is hd5.

Script command is started. The file is /etc/multibos/logs/scriptlog.120713124518.txt.

aixlpar1 : / # lsattr -El fcs0 -a max_xfer_size

max_xfer_size 0x100000 Maximum Transfer Size True

aixlpar1 : / # lsattr -El fcs1 -a max_xfer_size

max_xfer_size 0x100000 Maximum Transfer Size True

aixlpar1 : / # exit

Script command is complete. The file is /etc/multibos/logs/scriptlog.120713124518.txt.

Stopping multibos root shell ...

Compressing script log file ...

Compressed script log file is /etc/multibos/logs/scriptlog.120713124518.txt.Z

 

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Unmounting all standby BOS file systems ...

Unmounting /bos_inst/opt

Unmounting /bos_inst/var

Unmounting /bos_inst/usr

Unmounting /bos_inst

 

Log file is /etc/multibos/logs/op.alog

Return Status = SUCCESS

 

Then he manually changed the LPARs boot list to include the standby BLV.

 

aixlpar1 : / # bootlist -m normal hdisk2 blv=hd5 hdisk2 blv=bos_hd5

aixlpar1 : / # bootlist -m normal -o

hdisk2 blv=hd5 pathid=0

hdisk2 blv=hd5 pathid=1

hdisk2 blv=bos_hd5 pathid=0

hdisk2 blv=bos_hd5 pathid=1

 

 

He carefully recorded the bootlist output, just in case the boot failed with new max_xfer_size values. He could use the vdevice name and location to manually select the standby BLV to start the system in an emergency.

 

aixlpar1 : / # bootlist -m normal -ov

'ibm,max-boot-devices' = 0x5

NVRAM variable: (boot-device=/vdevice/vfc-client@30000014/disk@50060e8006d0206a,1000000000000:2 /vdevice/vfc-client@3000001e/disk@50060e8006d0207a,1000000000000:2 /vdevice/vfc-client@30000014/disk@50060e8006d0206a,1000000000000:4 /vdevice/vfc-client@3000001e/disk@50060e8006d0207a,1000000000000:4)

Path name: (/vdevice/vfc-client@30000014/disk@50060e8006d0206a,1000000000000:2)

match_specific_info: ut=disk/fcp/htcvspmpio

hdisk2 blv=hd5 pathid=0

Path name: (/vdevice/vfc-client@3000001e/disk@50060e8006d0207a,1000000000000:2)

match_specific_info: ut=disk/fcp/htcvspmpio

hdisk2 blv=hd5 pathid=1

Path name: (/vdevice/vfc-client@30000014/disk@50060e8006d0206a,1000000000000:4)

match_specific_info: ut=disk/fcp/htcvspmpio

hdisk2 blv=bos_hd5 pathid=0

Path name: (/vdevice/vfc-client@3000001e/disk@50060e8006d0207a,1000000000000:4)

match_specific_info: ut=disk/fcp/htcvspmpio

hdisk2 blv=bos_hd5 pathid=1

 

 

He restarted the LPAR using the primary BLV, with the modified FC attributes.

 

The system hung at LED 554.

 

hscroot@hmc1:~> lsrefcode -m 795-1 -r lpar --filter "lpar_names=aixlpar1" -F lpar_name:refcode

aixlpar1:0554

 

He restarted the LPAR in Open Firmware mode.

 

 

At the Open Firmware prompt, he entered the following to boot the LPAR from the standby instance BLV:

 

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

 

          1 = SMS Menu                          5 = Default Boot List

          8 = Open Firmware Prompt              6 = Stored Boot List

 

 

     Memory      Keyboard     Network     SCSI     Speaker  ok

0 > boot /vdevice/vfc-client@30000014/disk@50060e8006d0206a,1000000000000:4 |

 

The system booted OK on the standby AIX instance.

 

Elapsed time since release of system processors: 28858 mins 55 secs

 

 

 

 

 

 

 

-------------------------------------------------------------------------------

                                Welcome to AIX.

                   boot image timestamp: 02:39:13 07/13/2012

                 The current time and date: 02:54:02 07/13/2012

        processor count: 1;  memory size: 4096MB;  kernel size: 35062697

boot device: /vdevice/vfc-client@30000014/disk@50060e8006d0206a,1000000000000:4

-------------------------------------------------------------------------------

 

 

The FC adapters were running on the previous values, stored in the standby instance of AIX.

 

aixlpar1 : / # lsattr -El fcs0 -a max_xfer_size

max_xfer_size 0x100000 Maximum Transfer Size True

 

aixlpar1 : / # lsattr -El fcs1 -a max_xfer_size

max_xfer_size 0x100000 Maximum Transfer Size True

 

The original AIX instance still held the modified values for the FC adapters.

 

aixlpar1 : / # multibos -S

Initializing multibos methods ...

Initializing log /etc/multibos/logs/op.alog ...

Gathering system information ...

 

+-----------------------------------------------------------------------------+

Multibos Shell Operation

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

 

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Mounting all standby BOS file systems ...

Mounting /bos_inst

Mounting /bos_inst/usr

Mounting /bos_inst/var

Mounting /bos_inst/opt

 

+-----------------------------------------------------------------------------+

Multibos Root Shell

+-----------------------------------------------------------------------------+

Starting multibos root shell ...

Active boot logical volume is bos_hd5.

Script command is started. The file is /etc/multibos/logs/scriptlog.120713125542.txt.

aixlpar1 : / # lsattr -El fcs0 -a max_xfer_size

max_xfer_size 0x200000 Maximum Transfer Size True

aixlpar1 : / # lsattr -El fcs1 -a max_xfer_size

max_xfer_size 0x200000 Maximum Transfer Size True

aixlpar1 : / # exit

Script command is complete. The file is /etc/multibos/logs/scriptlog.120713125542.txt.

Stopping multibos root shell ...

Compressing script log file ...

Compressed script log file is /etc/multibos/logs/scriptlog.120713125542.txt.Z

 

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Unmounting all standby BOS file systems ...

Unmounting /bos_inst/opt

Unmounting /bos_inst/var

Unmounting /bos_inst/usr

Unmounting /bos_inst

 

Log file is /etc/multibos/logs/op.alog

Return Status = SUCCESS

 

The cause of the 554 hang appeared to be related to the fact that the VIOS physical adapters needed their max_xfer_size value changed to the new value before the client LPAR virtual fibre channel adapters were modified.

 

My colleague was glad he used multibos. It saved his bacon.

If you turn on Multiboot from within the SMS menu (as shown below), your LPAR will always boot into SMS, regardless of your AIX boot list and/or LPAR profile settings.

  

PowerPC Firmware

Version EM310_048

SMS 1.6 (c) Copyright IBM Corp. 2000,2005 All rights reserved.

-------------------------------------------------------------------------------

Multiboot

1. Select Install/Boot Device

2. Configure Boot Device Order

3. Multiboot Startup <ON>

-------------------------------------------------------------------------------

Navigation keys:

M = return to Main Menu

ESC key = return to previous screen X = eXit System Management Services

-------------------------------------------------------------------------------

Type menu item number and press Enter or select Navigation key:

 

If your LPAR always boots to the SMS menu, check the Multiboot Startup setting. If its set to ON, change it to OFF (as shown below). To access this option you must enter “5. Select Boot Options” from the main SMS menu.

 

PowerPC Firmware

Version EM310_048

SMS 1.6 (c) Copyright IBM Corp. 2000,2005 All rights reserved.

-------------------------------------------------------------------------------

Multiboot

1. Select Install/Boot Device

2. Configure Boot Device Order

3. Multiboot Startup <OFF>

-------------------------------------------------------------------------------

Navigation keys:

M = return to Main Menu

ESC key = return to previous screen X = eXit System Management Services

-------------------------------------------------------------------------------

Type menu item number and press Enter or select Navigation key:

 

Enabling Multiboot is only useful if you’re planning on dual-booting AIX and Linux within an LPAR. With Multiboot turned on, each time you reboot the LPAR, you are taken to SMS so you can choose which disk to boot from i.e. the AIX boot disk or the Linux boot disk. I’ve never come across anyone actually doing this in the commercial world. Mathew Davis (from IBM) wrote an article about dual booting AIX and Linux on pSeries back in 2005:

http://www.ibmsystemsmag.com/aix/aprilmay05/tipstechniques/6640p1.aspx

 

There can be other reasons why an LPAR always boots into SMS. The following link has some information on this:

 

http://www.aixmind.com/?tag=keeps-booting-to-sms

 

This IBM document explains how to select a boot disk from within the SMS menu.

 

http://www-01.ibm.com/support/docview.wss?rs=1209&context=SWG10&dc=DB520&q1=multiboot+SMS&uid=isg3T1011805&loc=en_US&cs=UTF-8&lang=en

 

 

Remember, if the disk doesn’t have a valid boot image, it won’t appear in the SMS menu as a boot device!

 

One way to check you have a boot device on your system is to run the ipl_varyon command from the AIX command line (before restarting your LPAR!):

 

# ipl_varyon -i

PVNAME          BOOT DEVICE     PVID                    VOLUME GROUP ID

hdisk0          YES             00f6048868b4e0ef0000    00f6048800004c00

 

If you don’t see a boot device on your system, perhaps someone has accidentally removed the boot record? Or maybe it was never created?

 

For example, in the commands that follow, I have “accidentally” run the ‘chpv –c’ command against my boot disk. This clears the boot record from the disk! The ipl_varyon command no longer shows hdisk0 as a bootable device.

 

Leaving the system in this state would prevent the system from booting!

 

To recreate the boot image, I run the bosboot command against hdisk0. Now ipl_varyon recognises hdisk0 as a bootable device again. Phew!

 

# chpv -c hdisk0

# ipl_varyon -i

PVNAME          BOOT DEVICE     PVID                    VOLUME GROUP ID

hdisk0          NO              00f6048868b4e0ef0000    00f6048800004c00

 

# bosboot -a -d /dev/hdisk0

bosboot: Boot image is 42689 512 byte blocks.

 

# ipl_varyon -i

PVNAME          BOOT DEVICE     PVID                    VOLUME GROUP ID

hdisk0          YES             00f6048868b4e0ef0000    00f6048800004c00

So, your root volume group is called old_rootvg not rootvg?

 

Wait……what!?

 

aixlpar1 : # lspv

hdisk0          00c342c68dfcbdfb             old_rootvg      active  < This is BAD!

hdisk1          00c342c6161c6b47             appvg           active

 

How did you end up in this situation? Did you reinstall AIX with some strange incantation or did some alternate disk activity go horribly wrong? Not sure? Me neither!

 

Fortunately the following tech-note and associated script can help.

 

http://www-01.ibm.com/support/docview.wss?uid=isg3T1000119

 

“Question

Rebuilding a Volume Group's Customized Device Database.

 

Answer

Run the script in this document when the ODM (Object Data Manager) entries for the root volume group are corrupted.”

 

“The script below assumes you are rebuilding the ODM definitions for the rootvg, which cannot be exported and then re-imported on a live system.

For most other volume groups, an export/import is the best way to clean up ODM problems.

If you need to rebuild ODM definitions for volume groups other than the rootvg, substitute appropriate values for PV and VG.”

So here’s the script:

#!/usr/bin/ksh
PV=`lslv -l hd5 | grep hdisk | head -1 | awk '{print $1}'`
VG=rootvg
lqueryvg -Lp $PV | awk '{ print $2 }' | while read LVname; do
odmdelete -q "name = $LVname" -o CuAt
odmdelete -q "name = $LVname" -o CuDv
odmdelete -q "value3 = $LVname" -o CuDvDr
odmdelete -q "dependency = $LVname" -o CuDep
done
odmdelete -q "name = $VG" -o CuAt
odmdelete -q "parent = $VG" -o CuDv
odmdelete -q "name = $VG" -o CuDv
odmdelete -q "name = $VG" -o CuDep
odmdelete -q "dependency = $VG" -o CuDep
if [ "$VG" = rootvg ]
then
odmdelete -q "value1 = 10" -o CuDvDr
else
odmdelete -q "value1 = $VG" -o CuDvDr
fi
odmdelete -q "value3 = $VG" -o CuDvDr
importvg -y $VG $PV # ignore lvaryoffvg errors
varyonvg $VG
synclvodm -Pv $VG # use -P only if running AIX V4.2.1 or later
savebase

 

Let’s execute the script to resolve this problem.

 

Note: Try this on a crash’n’burn system before unleashing it’s fury on a real AIX system (i.e. one that has users that depend on it!). Always take a mksysb backup before performing this type of activity.

 

aixlpar1 : /tmp # ksh -x fixmyrootvg.ksh

+ + lslv -l hd5

+ grep hdisk

+ head -1

+ awk {print $1}

PV=hdisk0

+ VG=rootvg

+ lqueryvg -Lp hdisk0

+ awk { print $2 }

+ read LVname

+ odmdelete -q name = hd5 -o CuAt

0518-307 odmdelete: 5 objects deleted.

+ odmdelete -q name = hd5 -o CuDv

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q value3 = hd5 -o CuDvDr

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q dependency = hd5 -o CuDep

0518-307 odmdelete: 1 objects deleted.

+ read LVname

+ odmdelete -q name = hd6 -o CuAt

0518-307 odmdelete: 4 objects deleted.

+ odmdelete -q name = hd6 -o CuDv

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q value3 = hd6 -o CuDvDr

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q dependency = hd6 -o CuDep

0518-307 odmdelete: 1 objects deleted.

+ read LVname

+ odmdelete -q name = hd8 -o CuAt

0518-307 odmdelete: 3 objects deleted.

+ odmdelete -q name = hd8 -o CuDv

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q value3 = hd8 -o CuDvDr

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q dependency = hd8 -o CuDep

0518-307 odmdelete: 1 objects deleted.

+ read LVname

+ odmdelete -q name = hd4 -o CuAt

0518-307 odmdelete: 5 objects deleted.

+ odmdelete -q name = hd4 -o CuDv

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q value3 = hd4 -o CuDvDr

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q dependency = hd4 -o CuDep

0518-307 odmdelete: 1 objects deleted.

+ read LVname

+ odmdelete -q name = hd2 -o CuAt

0518-307 odmdelete: 5 objects deleted.

+ odmdelete -q name = hd2 -o CuDv

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q value3 = hd2 -o CuDvDr

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q dependency = hd2 -o CuDep

0518-307 odmdelete: 1 objects deleted.

+ read LVname

+ odmdelete -q name = hd9var -o CuAt

0518-307 odmdelete: 5 objects deleted.

+ odmdelete -q name = hd9var -o CuDv

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q value3 = hd9var -o CuDvDr

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q dependency = hd9var -o CuDep

0518-307 odmdelete: 1 objects deleted.

+ read LVname

+ odmdelete -q name = hd3 -o CuAt

0518-307 odmdelete: 5 objects deleted.

+ odmdelete -q name = hd3 -o CuDv

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q value3 = hd3 -o CuDvDr

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q dependency = hd3 -o CuDep

0518-307 odmdelete: 1 objects deleted.

+ read LVname

+ odmdelete -q name = hd1 -o CuAt

0518-307 odmdelete: 5 objects deleted.

+ odmdelete -q name = hd1 -o CuDv

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q value3 = hd1 -o CuDvDr

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q dependency = hd1 -o CuDep

0518-307 odmdelete: 1 objects deleted.

+ read LVname

+ odmdelete -q name = hd10opt -o CuAt

0518-307 odmdelete: 5 objects deleted.

+ odmdelete -q name = hd10opt -o CuDv

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q value3 = hd10opt -o CuDvDr

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q dependency = hd10opt -o CuDep

0518-307 odmdelete: 1 objects deleted.

+ read LVname

+ odmdelete -q name = local -o CuAt

0518-307 odmdelete: 4 objects deleted.

+ odmdelete -q name = local -o CuDv

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q value3 = local -o CuDvDr

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q dependency = local -o CuDep

0518-307 odmdelete: 1 objects deleted.

+ read LVname

+ odmdelete -q name = hd7 -o CuAt

0518-307 odmdelete: 3 objects deleted.

+ odmdelete -q name = hd7 -o CuDv

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q value3 = hd7 -o CuDvDr

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q dependency = hd7 -o CuDep

0518-307 odmdelete: 1 objects deleted.

+ read LVname

+ odmdelete -q name = hd11admin -o CuAt

0518-307 odmdelete: 5 objects deleted.

+ odmdelete -q name = hd11admin -o CuDv

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q value3 = hd11admin -o CuDvDr

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q dependency = hd11admin -o CuDep

0518-307 odmdelete: 1 objects deleted.

+ read LVname

+ odmdelete -q name = rootvg -o CuAt

0518-307 odmdelete: 3 objects deleted.

+ odmdelete -q parent = rootvg -o CuDv

0518-307 odmdelete: 0 objects deleted.

+ odmdelete -q name = rootvg -o CuDv

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q name = rootvg -o CuDep

0518-307 odmdelete: 0 objects deleted.

+ odmdelete -q dependency = rootvg -o CuDep

0518-307 odmdelete: 0 objects deleted.

+ [ rootvg = rootvg ]

+ odmdelete -q value1 = 10 -o CuDvDr

0518-307 odmdelete: 1 objects deleted.

+ odmdelete -q value3 = rootvg -o CuDvDr

0518-307 odmdelete: 0 objects deleted.

+ importvg -y rootvg hdisk0

rootvg

0516-012 lvaryoffvg: Logical volume must be closed.  If the logical

        volume contains a filesystem, the umount command will close

        the LV device.

0516-942 varyoffvg: Unable to vary off volume group rootvg.

+ varyonvg rootvg

+ synclvodm -Pv rootvg

synclvodm: Physical volume data updated.

synclvodm: Logical volume hd5 updated.

synclvodm: Logical volume hd6 updated.

synclvodm: Logical volume hd8 updated.

synclvodm: Logical volume hd4 updated.

synclvodm: Logical volume hd2 updated.

synclvodm: Logical volume hd9var updated.

synclvodm: Logical volume hd3 updated.

synclvodm: Logical volume hd1 updated.

synclvodm: Logical volume hd10opt updated.

synclvodm: Logical volume hd7 updated.

synclvodm: Logical volume hd11admin updated.

+ savebase

 

Hey Presto! The root volume group is now named rootvg just the way we like it!

 

aixlpar1 : # lspv

hdisk0          00c342c68dfcbdfb                    rootvg          active

hdisk1          00c342c6161c6b47                    appvg            active

 

I NEVER thought this would happen! But, “NEVER SAY NEVER!”.  IBM just announced that AIX 5.3 can now run on POWER8.

 

http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&infotype=an&appname=iSource&supplier=872&letternum=ENUSAP16-0152

 

As you’d expect, there are some caveats!

 

A special PTF (U86665.bff) provides POWER8 support on AIX  5.3 with the following limitations:

 

• The LPAR must be at AIX 5.3 TL12 SP9 (latest 5.3 release).
• The POWER8 LPAR will run in POWER6 compatibility mode and is limited to SMT2 mode. SMT2 mode results in some capacity loss compared to SMT4/SMT8 mode. IBM publishes SMT2 rPerf values that can be used to quantify Power8 SMT2 capacity.
• AIX 5.3 POWER8 LPARs only support Virtual I/O configurations: vSCSI, NIPV, VLAN.
• LPM is supported.

 

The only supported AIX 5.3 POWER8 installation methods are:

 

• mksysb – First perform an in place update to a supported (P5/P6/P7) AIX 5.3 TL12 SP9 LPAR with PTF U866665. Standard mksysb command can then be used to capture a P8 capable mksysb image. The mksysb image can then be used to install POEWR8 LPARs
• NIM – An AIX 5.3 TL12 SP9 NIM environment must be updated to support POWER8. An AIX 5.3 TL12 SP9 NIM lpp_source must be updated to include PTF U866665. A NIM SPOT must then be created to utilize the updated lpp_source.

 

All POWER8 systems are supported with the following restrictions:

 

• POWER8 system must be at 840 firmware level.
• POWER8 LPAR must be served by a 2.2.4.10 VIOS.

 

Other restrictions:

 

• Remote restart is not supported with POWER8 5.3 LPARs
• POWER8 AIX 5.3 LPARs are not supported with a VIOS using Shared Storage Pools.

 

 

I tested this in my lab and it all worked as expected. I installed the PTF on an existing AIX 5.3 LPAR on a POWER7 750 and rebooted the partition.

 

[root@53gibbo]/ # lspv

hdisk0          00f603cdcd9ba6de                    rootvg          active

 

[root@53gibbo]/ # oslevel -s

5300-12-09-1341

 

[root@53gibbo]/ # lsconf | more

System Model: IBM,8233-E8B

Machine Serial Number: 1003CDP

Processor Type: PowerPC_POWER7

Processor Implementation Mode: POWER 6

Processor Version: PV_6_Compat

Number Of Processors: 2

Processor Clock Speed: 3550 MHz

CPU Type: 64-bit

Kernel Type: 64-bit

LPAR Info: 3 53gibbo

Memory Size: 2048 MB

Good Memory Size: 2048 MB

Platform Firmware level: AL730_142

Firmware Version: IBM,AL730_142

 

[root@53gibbo]/tmp/cg/aix53p8 # inutoc .

[root@53gibbo]/tmp/cg/aix53p8 # ls -ltr

total 35808

-rw-r-----    1 root     system      8994816 Feb  8 09:35 U866665.bff

-rw-r-----    1 root     system      8994816 Feb  8 09:37 bos.mp64.5.3.12.10.U

-rw-r--r--    1 root     system       328592 Feb  8 09:37 .toc

 

[root@53gibbo]/tmp/cg/aix53p8 # update_all -d .

install_all_updates: Initializing system parameters.

install_all_updates: Log file is /var/adm/ras/install_all_updates.log

install_all_updates: Checking for updated install utilities on media.

install_all_updates: Processing media.

install_all_updates: Generating list of updatable installp filesets.

install_all_updates: The following filesets have been selected as updates

to currently installed software:

 

   bos.mp64 5.3.12.10

 

   << End of Fileset List >>

 

install_all_updates: Performing installp update.

+-----------------------------------------------------------------------------+

                    Pre-installation Verification...

+-----------------------------------------------------------------------------+

Verifying selections...done

Verifying requisites...done

Results...

 

SUCCESSES

---------

  Filesets listed in this section passed pre-installation verification

  and will be installed.

 

  Selected Filesets

  -----------------

  bos.mp64 5.3.12.10                          # Base Operating System 64-bit...

 

  << End of Success Section >>

 

+-----------------------------------------------------------------------------+

                   BUILDDATE Verification ...

+-----------------------------------------------------------------------------+

Verifying build dates...done

FILESET STATISTICS

------------------

    1  Selected to be installed, of which:

        1  Passed pre-installation verification

  ----

    1  Total to be installed

 

installp:  bosboot verification starting...

installp:  bosboot verification completed.

+-----------------------------------------------------------------------------+

                         Installing Software...

+-----------------------------------------------------------------------------+

 

installp: APPLYING software for:

        bos.mp64 5.3.12.10

 

 

. . . . . << Copyright notice for bos.mp64 >> . . . . . . .

 Licensed Materials - Property of IBM

 

 5765G0399

   Copyright International Business Machines Corp. 2000, 2015.

   Copyright William Erik Baxter, 2004.

   Copyright RSA Data Security, Inc. 1991-1992.

 

 All rights reserved.

 US Government Users Restricted Rights - Use, duplication or disclosure

 restricted by GSA ADP Schedule Contract with IBM Corp.

. . . . . << End of copyright notice for bos.mp64 >>. . . .

 

Finished processing all filesets.  (Total time:  3 secs).

 

installp:  bosboot verification starting...

installp:  bosboot verification completed.

installp:  bosboot process starting...

 

bosboot: Boot image is 45802 512 byte blocks.

0503-292 This update will not fully take effect until after a

        system reboot.

 

    * * *  A T T E N T I O N  * * *

    System boot image has been updated. You should reboot the

    system as soon as possible to properly integrate the changes

    and to avoid disruption of current functionality.

 

installp:  bosboot process completed.

+-----------------------------------------------------------------------------+

                                Summaries:

+-----------------------------------------------------------------------------+

 

Installation Summary

--------------------

Name                        Level           Part        Event       Result

-------------------------------------------------------------------------------

bos.mp64                    5.3.12.10       USR         APPLY       SUCCESS

 

    * * *  A T T E N T I O N  * * *

    System boot image has been updated. You must reboot the system

    after install_all_updates completes to properly integrate the updates.

 

install_all_updates: Checking for recommended maintenance level 5300-12.

install_all_updates: Executing /usr/bin/oslevel -rf, Result = 5300-12

install_all_updates: Verification completed.

install_all_updates: Log file is /var/adm/ras/install_all_updates.log

install_all_updates: Result = SUCCESS

 

[root@53gibbo]/tmp/cg/aix53p8 # oslevel -s

5300-12-09-1341

 

[root@53gibbo]/tmp/cg/aix53p8 # lslpp -l bos.mp64

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  bos.mp64                 5.3.12.10  APPLIED    Base Operating System 64-bit

                                                 Multiprocessor Runtime

 

Path: /etc/objrepos

  bos.mp64                  5.3.11.2  COMMITTED  Base Operating System 64-bit

                                                 Multiprocessor Runtime

[root@53gibbo]/tmp/cg/aix53p8 # at now

shutdown –Fr

 

My POWER8 system was an S824 running the latest firmware (840_056), with VIOS running the latest code (2.2.4.10) and the latest HMC code (8.4).

 

p8vio1:/home/padmin# lsconf | head -20

System Model: IBM,8286-42A

Machine Serial Number: 214F55V

Processor Type: PowerPC_POWER8

Processor Implementation Mode: POWER 7

Processor Version: PV_7_Compat

Number Of Processors: 2

Processor Clock Speed: 3525 MHz

CPU Type: 64-bit

Kernel Type: 64-bit

LPAR Info: 101 82861_vio1

Memory Size: 18432 MB

Good Memory Size: 18432 MB

Platform Firmware level: SV840_056

Firmware Version: IBM,FW840.00 (SV840_056)

 

On my NIM master I created a new lppsource and SPOT for AIX 5.3 TL12 SP9 and included the required PTF U86665.

 

# lsnim -t spot | grep 53

spotAIX53TL12SP9     resources       spot

 

# lsnim -t lpp_source | grep 53

AIX53TL12SP9             resources       lpp_source

 

# nim -o showres spotAIX53TL12SP9 | grep bos.mp64

  bos.mp64                 5.3.12.10    C     F    Base Operating System 64-bit

 

# nim -o showres AIX53TL12SP9_P8 | grep bos.mp64

  bos.mp64                    5.3.11.2                   I  b usr,root

  bos.mp64                    5.3.12.9                   S  b usr

  bos.mp64                    5.3.12.10                  S  b usr

 

I took a mksysb on the 5.3 LPAR to my NIM master. I then used the mksysb to perform an NIM mksysb install (restore) to a new partition on the S824. This worked flawlessly. The partition was configured in POWER6 mode with SMT2 only.

 

[root@53gibbo]/ # oslevel -s

5300-12-09-1341

 

[root@53gibbo]/ # lsconf | head -10

System Model: IBM,8286-42A

Machine Serial Number: 214F55V

Processor Type: PowerPC_POWER8

Processor Implementation Mode: POWER 6

Processor Version: PV_6_Compat

Number Of Processors: 2

Processor Clock Speed: 3525 MHz

CPU Type: 64-bit

Kernel Type: 64-bit

LPAR Info: 2 53gibbo

Memory Size: 2048 MB

Good Memory Size: 2048 MB

Platform Firmware level: SV840_056

Firmware Version: IBM,FW840.00 (SV840_056)

Console Login: enable

Auto Restart: true

Full Core: false

 

[root@53gibbo]/ # smtctl

 

This system is SMT capable.

SMT is currently enabled.

SMT boot mode is not set.

SMT threads are bound to the same virtual processor.

 

proc0 has 2 SMT threads.

Bind processor 0 is bound with proc0

Bind processor 1 is bound with proc0

 

 

proc2 has 2 SMT threads.

Bind processor 2 is bound with proc2

Bind processor 3 is bound with proc2

 

[root@53gibbo]/ # lparstat -i

Node Name                                  : 53gibbo

Partition Name                             : 53gibbo

Partition Number                           : 2

Type                                       : Shared-SMT

Mode                                       : Uncapped

Entitled Capacity                          : 0.20

Partition Group-ID                         : 32770

Shared Pool ID                             : 0

Online Virtual CPUs                        : 2

Maximum Virtual CPUs                       : 4

Minimum Virtual CPUs                       : 1

Online Memory                              : 2048 MB

Maximum Memory                             : 4096 MB

Minimum Memory                             : 1024 MB

Variable Capacity Weight                   : 128

Minimum Capacity                           : 0.10

Maximum Capacity                           : 4.00

Capacity Increment                         : 0.01

Maximum Physical CPUs in system            : 24

Active Physical CPUs in system             : 24

Active CPUs in Pool                        : 14

Shared Physical CPUs in system             : 14

Maximum Capacity of Pool                   : 1400

Entitled Capacity of Pool                  : 1160

Unallocated Capacity                       : 0.00

Physical CPU Percentage                    : 10.00%

Unallocated Weight                         : 0

Desired Virtual CPUs                       : 2

Desired Memory                             : 2048 MB

Desired Variable Capacity Weight           : 128

Desired Capacity                           : 0.20

 

The example above shows you how to migrate an AIX 5.3 system using NIM and mksysb. But you don’t have to do this, as LPM is fully supported. The migration effort for many customers is as easy as kicking off an LPM operation to move their (updated) AIX 5.3 partitions from their older Power servers to POWER8. Easy!

 

So I guess customers that still need to run AIX 5.3 on POWER8 now have an alternative to AIX 5.3 Versioned WPARs. But please keep in mind the performance and support considerations when making your decision.

Additional README information available here:

https://www.ibm.com/developerworks/community/blogs/cgaix/resource/U866665README.pdf

 

 

IBM recently announced some significant changes to their ‘AIX Release and Service Delivery’ strategy. Starting February 2011, the following changes will apply for both AIX 6 and AIX 7.

               

•          Three years of support for each Technology Level (TL).

•          A single Technology Level per AIX version, per year.

•          Service Packs will be released approximately 4 times per year per Technology Level.

 

Some of the benefits of the new strategy are:

 

• Longer support for Technology Levels.

–      More closely matches client deployment requirements.

 

·         Single Technology Level per year, per version, and about one Service Pack per quarter.

–      Fewer updates reduce administrative workload.

 

·         New hardware is supported on prior Technology Levels.

–      Easier integration of new hardware into existing environments.

 

You can review the official presentation and details here. I particularly like this FAQ from the presentation.

 

Q: Why doesn’t the new strategy include AIX V5.3?

A: AIX V5.3 is nearing end of life with the End of Marketing for AIX V5.3 set for April 2011

 

So if you are on AIX 5.3, start planning a migration to AIX 7.1 (or at least AIX 6.1), NOW! I’ve provided some further information from the presentation below.

 

 

 

Longer Support for each Technology Level

The 2011 Release Strategy extends the support for each Technology Level for up to three years from the introduction of the Technology Level. This means that clients with a Software Maintenance Agreement for the AIX OS will receive IBM defect support during that two year period without having to move up to the latest Technology Level update.

For example, a Technology Level introduced in the second half of 2010 will be supported though the second half of 2013. Since each Technology Level will be supported for three years and IBM plans to release one Technology Level per year, IBM eventually will support up to six Technology Levels for each AIX release.  Please note that due to minor variations in release dates, some Technology Levels will be supported for slightly more that three years and some will be supported for slightly less than three years. A three year service life for each Technology Level is an objective, not an absolute limit.

 

Fewer Technology Levels and Fewer Service Packs

Historically, new Technology Levels were the only way to deliver new hardware support to clients. Since 2007, we have supported most new hardware on previous Technology Levels via a service pack. Since this new approach to supporting new hardware has been well proven with the introduction of POWER7 in 2010, we decided to eliminate the “Spring” Technology Level which reduces the number of updates clients have to deal with. The number of Service Packs has been about 5 per year and we want to reduce this number to about once per quarter.

 

 

 

Between Technology Level releases, clients maintain their AIX operating systems by installing Service Packs (SP) or Interim Fixes for the entire support life of the Technology Level update. Service Packs are also used to provide support for newly released hardware.

The release dates and frequency of Service Packs are variable due to many factors. including new Technology Level releases, new hardware introduction and the need to deliver software fixes.

 

In the past IBM delivered new Service Packs about every eight to twelve weeks or about five to eight times per year per Technology Level. With the 2011 release strategy, IBM intends to reduce the number of Service Packs released per year, per AIX release to about four. As stated earlier, there are many factors that drive the release of a Service Pack, so releasing four Service Packs per year, per Technology Level represents a goal and objective, but not an absolute limit.

 

Note that new Technology Levels are almost always accompanied very closely by the first Service Pack for that new Technology Level. This first Service Pack includes fixes to problems that are discovered between the time the new Technology Level is released to manufacturing for the media to be replicated and the time the new Technology Level is actually available to clients.

 

Summary

The providers of all software, including platform software such as the AIX operating system, must balance the client’s need for stability against the need to enhance the software to provide new functionality. The changes introduced with the AIX OS Release and Service Strategy of 2011 balances those two conflicting goals to help provide our clients with significant improvements in the manageability of the AIX operating system.

The increased service life, better manageability and support for new hardware on older fix levels provided by the new strategy are in direct response to our client’s requirements.

As stated earlier, this new strategy represents goals and objectives of IBM and is subject to change. Our clients understand that substantial changes such as this new strategy are an indication of the commitment that IBM has to improving the market leading capabilities of the AIX operating system.

 

For more information and best practices please refer to:

 

https://www14.software.ibm.com/webapp/set2/sas/f/best/home.html