Q: Does alt_disk_copy convert JFS filesystems to JFS2?
A: Yes! Starting at 6100-04, a new flag was introduced that would convert JFS filesystems to JFS2. We now have the -T which does the conversion.

http://www-01.ibm.com/support/docview.wss?uid=isg3T1011055

 

Here’s an example of converting rootvg file systems from JFS to JFS2 using alt_disk_copy.

 

My lab system was migrated from AIX 5.3 to 7.1 via nimadm. Unfortunately, nimadm does not convert JFS file systems to JFS2 during the migration. So, in this case, even though I’ve migrated to AIX 7.1 (which is a good thing) I’m still left with legacy JFS file systems in rootvg.

                                                                        

And because the AIX 5.3 version of the alt_disk_copy command does not have the –T option, I can’t convert my JFS file systems to JFS2 before I migrate to AIX 7.1. So my best option is to migrate to AIX 7.1 then convert rootvg to JFS2 file systems. A few hops in the process but it’s good enough.

 

aixlpar1 : / # lsvg -l rootvg

rootvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

hd5                 boot       1       1       1    closed/syncd  N/A

hd6                 paging     32      32      1    open/syncd    N/A

hd8                 jfslog     1       1       1    open/syncd    N/A

hd4                 jfs        4       4       1    open/syncd    /

hd2                 jfs        29      29      1    open/syncd    /usr

hd9var              jfs        20      20      1    open/syncd    /var

hd3                 jfs        164     164     1    open/syncd    /tmp

hd1                 jfs        4       4       1    open/syncd    /home

hd10opt             jfs        4       4       1    open/syncd    /opt

local               jfs        4       4       1    open/syncd    /usr/local

loglv               jfs        4       4       1    open/syncd    /var/log

hd7                 sysdump    9       9       1    open/syncd    N/A

hd71                sysdump    9       9       1    open/syncd    N/A

hd11admin           jfs        2       2       1    open/syncd    /admin

 

aixlpar1 : / # oslevel -s

7100-01-01-1141

 

I clone rootvg to a spare disk using alt_disk_copy and the –T flag (which will convert the file systems to JFS2). The process converts the file systems to JFS2, as shown in the section below (highlighted in green).

 

aixlpar1 : / # alt_disk_copy -d hdisk1 -T

Source boot disk is: hdisk2

jfs2j2: Current data file /image.data moved to /image.data.acct.save.3735802.

jfs2j2: Creating new data file (/image.data).

jfs2j2: Converting filesystems in data file (/image.data) from jfs to jfs2.

jfs2j2: Converted data file (/image.data) moved to /tmp/image.data.conv.3407932.

 

jfs2j2: Cleaning up.

jfs2j2: Moving previous data file /image.data.acct.save.3735802 to IDFILE.

jfs2j2: Successfully completed converting image.data to jfs2.

Checking disk sizes.

Creating cloned rootvg volume group and associated logical volumes.

Creating logical volume alt_hd5

Creating logical volume alt_hd6

Creating logical volume alt_hd8

Creating logical volume alt_hd4

Creating logical volume alt_hd2

Creating logical volume alt_hd9var

Creating logical volume alt_hd3

Creating logical volume alt_hd1

Creating logical volume alt_hd10opt

Creating logical volume alt_local

Creating logical volume alt_loglv

Creating logical volume alt_hd7

Creating logical volume alt_hd71

Creating logical volume alt_hd11admin

Creating /alt_inst/ file system.

/alt_inst filesystem not converted.

        Small inode extents are already enabled.

Creating /alt_inst/admin file system.

/alt_inst/admin filesystem not converted.

        Small inode extents are already enabled.

Creating /alt_inst/home file system.

/alt_inst/home filesystem not converted.

        Small inode extents are already enabled.

Creating /alt_inst/opt file system.

/alt_inst/opt filesystem not converted.

        Small inode extents are already enabled.

Creating /alt_inst/tmp file system.

/alt_inst/tmp filesystem not converted.

        Small inode extents are already enabled.

Creating /alt_inst/usr file system.

/alt_inst/usr filesystem not converted.

        Small inode extents are already enabled.

Creating /alt_inst/usr/local file system.

/alt_inst/usr/local filesystem not converted.

        Small inode extents are already enabled.

Creating /alt_inst/var file system.

/alt_inst/var filesystem not converted.

        Small inode extents are already enabled.

Creating /alt_inst/var/log file system.

/alt_inst/var/log filesystem not converted.

        Small inode extents are already enabled.

Generating a list of files

for backup and restore into the alternate file system...

Backing-up the rootvg files and restoring them to the

alternate file system...

Running user specified customization script /tmp/cust.script.conv.3407932...

Modifying ODM on cloned disk.

Building boot image on cloned disk.

forced unmount of /alt_inst/var/log

forced unmount of /alt_inst/var

forced unmount of /alt_inst/var

forced unmount of /alt_inst/usr/local/chksys

forced unmount of /alt_inst/usr/local

forced unmount of /alt_inst/usr

forced unmount of /alt_inst/tmp

forced unmount of /alt_inst/opt

forced unmount of /alt_inst/home

forced unmount of /alt_inst/admin

forced unmount of /alt_inst

Changing logical volume names in volume group descriptor area.

Fixing LV control blocks...

Fixing file system superblocks...

Bootlist is set to the boot disk: hdisk1 blv=hd5

 

Before I reboot the system on the alternate rootvg I verify that the cloned volume group now contains JFS2 file systems only. I “wake up” the altinst_rootvg and run the lsvg command to confirm the file system is correct. I then put the altinst_rootvg to “sleep”, reboot the system and verify all rootvg file systems are mounted as jfs2.

 

aixlpar1 : / # alt_rootvg_op -W -d hdisk1

Waking up altinst_rootvg volume group ...

 

aixlpar1 : / # lsvg -l altinst_rootvg

altinst_rootvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

alt_hd5             boot       1       1       1    closed/syncd  N/A

alt_hd6             paging     32      32      1    closed/syncd  N/A

alt_hd8             jfs2log    1       1       1    open/syncd    N/A

alt_hd4             jfs2       4       4       1    open/syncd    /alt_inst

alt_hd2             jfs2       29      29      1    open/syncd    /alt_inst/usr

alt_hd9var          jfs2       20      20      1    open/syncd    /alt_inst/var

alt_hd3             jfs2       164     164     1    open/syncd    /alt_inst/tmp

alt_hd1             jfs2       4       4       1    open/syncd    /alt_inst/home

alt_hd10opt         jfs2       4       4       1    open/syncd    /alt_inst/opt

alt_local           jfs2       4       4       1    open/syncd    /alt_inst/usr/local

alt_loglv           jfs2       4       4       1    open/syncd    /alt_inst/var/log

alt_hd7             sysdump    9       9       1    closed/syncd  N/A

alt_hd71            sysdump    9       9       1    closed/syncd  N/A

alt_hd11admin       jfs2       2       2       1    open/syncd    /alt_inst/admin

 

aixlpar1 : / # alt_rootvg_op -S altinst_rootvg

Putting volume group altinst_rootvg to sleep ...

forced unmount of /alt_inst/var/log

forced unmount of /alt_inst/var

forced unmount of /alt_inst/usr/local

forced unmount of /alt_inst/usr

forced unmount of /alt_inst/tmp

forced unmount of /alt_inst/opt

forced unmount of /alt_inst/home

forced unmount of /alt_inst/admin

forced unmount of /alt_inst

Fixing LV control blocks...

Fixing file system superblocks...

aixlpar1 : / #

 

; Reboot on the alternate rootvg hdisk

 

aixlpar1 : / # uptime

  10:18AM   up 1 min,  1 user,  load average: 0.32, 0.09, 0.03

 

aixlpar1 : / # lspv

hdisk1          00c342c637f21a59                    rootvg          active

hdisk2          00c342c6161c6b47                    old_rootvg

 

aixlpar1 : / # df

Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4          524288    411568   22%     3040     7% /

/dev/hd2         3801088    537176   86%    34089    35% /usr

/dev/hd9var      2621440   2413520    8%     3554     2% /var

/dev/hd3        21495808  21476312    1%      110     1% /tmp

/dev/hd1          524288    522456    1%       90     1% /home

/proc                  -         -    -         -     -  /proc

/dev/hd10opt      524288    266560   50%     5133    15% /opt

/dev/local        524288    495320    6%      249     1% /usr/local

/dev/loglv        524288    522040    1%       49     1% /var/log

/dev/hd11admin     262144    261384    1%        7     1% /admin

 

aixlpar1 : / # lsvg -l rootvg

rootvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

hd5                 boot       1       1       1    closed/syncd  N/A

hd6                 paging     32      32      1    open/syncd    N/A

hd8                 jfs2log    1       1       1    open/syncd    N/A

hd4                 jfs2       4       4       1    open/syncd    /

hd2                 jfs2       29      29      1    open/syncd    /usr

hd9var              jfs2       20      20      1    open/syncd    /var

hd3                 jfs2       164     164     1    open/syncd    /tmp

hd1                 jfs2       4       4       1    open/syncd    /home

hd10opt             jfs2       4       4       1    open/syncd    /opt

local               jfs2       4       4       1    open/syncd    /usr/local

loglv               jfs2       4       4       1    open/syncd    /var/log

hd7                 sysdump    9       9       1    open/syncd    N/A

hd71                sysdump    9       9       1    open/syncd    N/A

hd11admin           jfs2       2       2       1    open/syncd    /admin

 

===

The message “filesystem not converted” (below) is not related to the JFS to JFS2 conversion. This messge refers to whether or not the file system needs to be changed to use Variable Inode Extents (VIX). This is the default setting for JFS2 file systems.

 

Creating /alt_inst/ file system.

/alt_inst filesystem not converted.

        Small inode extents are already enabled.

 

aixlpar1 : / # lsfs -q /

Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting

/dev/hd4        --         /                      jfs2  524288  --         yes  no

  (lv size: 524288, fs size: 524288, block size: 4096, sparse files: yes, inline log: no, inline log size: 0, EAformat: v1, Quota: no, DMAPI: no, VIX: yes, EFS: no, ISNAPSHOT: no, MAXEXT: 0, MountGuard: no)

aixlpar1 : / #

 

Please refer to the following link for more information on VIX in AIX.

 

Use of Variable Inode Extents (VIX) in AIX

http://www-01.ibm.com/support/docview.wss?uid=isg3T1010368

===

 

Why would I want to convert rootvg to JFS2 anyway? Well for starters, it’s generally considered best practice to use JFS2 as it offers several performance & scalability enhancements over JFS. For example, you cannot create files greater than 2GB on JFS, unless the file system was created as “large (big) file” enabled; jfs file systems in rootvg were never created as large file enabled.

 

Another reason…..eventually JFS will be retired.

 

Here’s an example of a potential problem with JFS in rootvg. You try to create a file of a size greater than 2GB in /tmp (type jfs). Even though the ulimit settings are not restricting the creation of a file of this size, the JFS file system will not allow it. The file creation process fails. The bf attribute for the /tmp file system is set to false. This indicates the file system is not “large file” enabled.

 

aixlpar1 : /tmp # df -g .

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

/dev/hd3          10.25      6.94   33%      117     1% /tmp

 

aixlpar1 : /tmp # cat 1gbfile 1gbfile 1gbfile > 3gbfile

cat: 0652-054 Cannot write to output.

Error 0

 

aixlpar1 : /tmp # ulimit -a

time(seconds)        unlimited

file(blocks)         unlimited

data(kbytes)         unlimited

stack(kbytes)        unlimited

memory(kbytes)       unlimited

coredump(blocks)     2097151

nofiles(descriptors) unlimited

threads(per process) unlimited

processes(per user)  unlimited

 

aixlpar1 : /tmp # lsfs -q /tmp

Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting

/dev/hd3        --         /tmp                   jfs   21495808 --         yes  no

  (lv size: 21495808, fs size: 21495808, frag size: 4096, nbpi: 4096, compress: no, bf: false, ag: 8)

 

OK, so creating files greater than 2GB in rootvg is probably not a big problem for most AIX shops. But why risk it! My advice is to always use JFS2 (even in rootvg).

 

Keep this in mind when migrating from AIX 5.3 to 6.1 or 7.1 via nimadm.

 

The following link provides an overview and comparison of both JFS and JFS2 file systems.

 

http://pic.dhe.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.baseadmn/doc/baseadmndita/fs_jfs_jfs2.htm

 

 

I’ve shared my tips for resolving DLPAR problems in the past. So this week, when one of my colleagues was experiencing an issue with DLPAR, I referred him to my blog post and suggested he follow the troubleshooting steps. He did so and I went about my business. Later that same day I asked him how he had fared. He told me that DLPAR was still not working on his particular AIX LPAR. It was an AIX 5.3 system and he was attempting to another Virtual Processor to the LPAR. He expressed his frustration with the situation, so I offered to take a look for him.

 

What I found was that the system was missing an important fileset. A fileset that enabled DLPAR operations on AIX 5.3 systems. The fileset in question was named csm.client. Without this fileset installed DLPAR would never work.

 

I advised my colleague of the problem and suggested he follow the steps below to resolve the issue. After he reinstalled the fileset, RMC communication between the HMC and LPAR was restored and his DLPAR processor add operation completed without issue.

 

1.  Mount the NIM masters lpp_source file system:

 

aix53lpar1 : / # mount nim1:/export/lpp_source /mnt

 

2.  Verify CSM filesets are not installed and the IBM.DRM subsystem is either inoperative or missing.

 

aix53lpar1 : /# oslevel -s

5300-12-04-1119

 

aix53lpar1 : / # lslpp -l | grep -i csm

aix53lpar1 : / #

 

aix53lpar1 : / # lssrc -a | grep rsct

 ctrmc            rsct             290870       active

 IBM.ServiceRM    rsct_rm          237818       active

 ctcas            rsct                          inoperative

 IBM.ERRM         rsct_rm                       inoperative

 IBM.AuditRM      rsct_rm                       inoperative

 IBM.LPRM         rsct_rm                       inoperative

 IBM.HostRM       rsct_rm                       inoperative

 IBM.DRM          rsct_rm                       inoperative

 IBM.SensorRM     rsct_rm                       inoperative

 

aix53lpar1 : / # lsrsrc IBM.ManagementServer

/usr/sbin/rsct/bin/lsrsrc-api: 2612-010 Resource class IBM.ManagementServer is not defined.

 

 

3.  Verify that the HMC is unable to establish RMC communication with the LPAR (refer to RMC message below on the HMC).

4.  Install the csm.client fileset (and pre-req’s) from the 5300-12-04-1119 lpp_source.

 

aix53lpar1 : / # cd /mnt/lpp_sourceaix531204

aix53lpar1 : /mnt/lpp_sourceaix531204 # installp -agd . csm.client

+-----------------------------------------------------------------------------+

                    Pre-installation Verification...

+-----------------------------------------------------------------------------+

Verifying selections...done

Verifying requisites...done

Results...

 

…etc…

 

Installation Summary

--------------------

Name                        Level           Part        Event       Result

-------------------------------------------------------------------------------

csm.core                    1.7.1.0         USR         APPLY       SUCCESS

csm.core                    1.7.1.0         ROOT        APPLY       SUCCESS

csm.client                  1.7.1.0         USR         APPLY       SUCCESS

csm.client                  1.7.1.0         ROOT        APPLY       SUCCESS

 

5.  Verify that the IBM.DRM subsystem is now active and the HMC can communicate with the LPAR via RMC.

 

aix53lpar1 : / # lssrc -a | grep rsct

 IBM.ServiceRM    rsct_rm          237818       active

 ctrmc            rsct             290876       active

 IBM.CSMAgentRM   rsct_rm          360522       active

 IBM.AuditRM      rsct_rm          446624       active

 IBM.HostRM       rsct_rm          504016       active

 IBM.LPRM         rsct_rm          401568       active

 IBM.DRM          rsct_rm          348364       active

 ctcas            rsct                          inoperative

 IBM.ERRM         rsct_rm                       inoperative

 IBM.SensorRM     rsct_rm                       inoperative

 IBM.MgmtDomainRM rsct_rm                       inoperative

 

aix53lpar1 : / # lsrsrc IBM.ManagementServer

Resource Persistent Attributes for IBM.ManagementServer

resource 1:

        Name             = "10.10.10.252"

        Hostname         = "10.10.10.252"

        ManagerType      = "HMC"

        LocalHostname    = "10.10.10.65"

        ClusterTM        = "9078-160"

        ClusterSNum      = ""

        ActivePeerDomain = ""

        NodeNameList     = {"aix53lpar1"}

resource 2:

        Name             = "10.10.10.251"

        Hostname         = "10.10.10.251"

        ManagerType      = "HMC"

        LocalHostname    = "10.10.10.65"

        ClusterTM        = "9078-160"

        ClusterSNum      = ""

        ActivePeerDomain = ""

        NodeNameList     = {"aix53lpar1"}

 

aix53lpar1 : / # lsrsrc IBM.MCP

Resource Persistent Attributes for IBM.MCP

resource 1:

        MNName           = "10.10.10.65"

        NodeID           = 12019572219563664974

        KeyToken         = "10.10.10.251"

        IPAddresses      = {"10.10.10.251","172.16.0.1"}

        ActivePeerDomain = ""

        NodeNameList     = {"aix53lpar1"}

resource 2:

        MNName           = "10.10.10.65"

        NodeID           = 6374098991051202638

        KeyToken         = "10.10.10.252"

        IPAddresses      = {"10.10.10.252","172.17.0.2"}

        ActivePeerDomain = ""

        NodeNameList     = {"aix53lpar1"}

 

I’ve received a couple of requests for an example of using a post migration script with nimadm. What follows is a simple example of using such a resource with NIM. If you are not familiar with the nimadm tool then perhaps you’d like to start first by reading my article on using nimadm to migrate to AIX 6.1.

 

The nimadm utility can perform both pre and post migration tasks. This is accomplished by running NIM scripts either before or after a migration. The tool accepts the following flags for pre and post migration script resources:

 

-a PreMigrationScript Specifies the pre-migration NIM script resource.

-z PostMigrationScript Specifies the post-migration NIM script resource.

 

pre-migration

This script resource that is run on the NIM master, but in the environment of the client's alt_inst file system that is mounted on the master (this is done by using the chroot command). This script is run before the migration begins.

 

post-migration

This script resource is similar to the pre-migration script, but it is executed after the migration is complete.

 

We are going to focus on post-migration only, although the configuration is the same for both.

 

In this example I need to uninstall and install a 3^rd party device fileset for a storage device. I need to perform this task as part of the migration process. To protect the innocent, I have not named the storage vendor in this post. But I will say that it was not IBM storage we are dealing with in this case.

 

Before we start, first we collect all the necessary device filesets that provide support for this type of storage on AIX. We place them into a local directory on my NIM master. Along with the software, I also place a copy of my NIM script in the same directory on the NIM master. The script name is XYZpost.ksh.

 

root@nim1 : /usr/local/XYZ # ls -ltr

total 544

-r-xr-xr-x    1 root     system        51200 Mar 11 2011  MPIO_1001I

-r-xr-xr-x    1 root     system        51200 Mar 11 2011  MPIO_1002U

-r-xr-xr-x    1 root     system        51200 Mar 11 2011  MPIO_1003U

-r-xr-xr-x    1 root     system        51200 Mar 11 2011  MPIO_1004U

-r-xr-xr-x    1 root     system        51200 May 18 16:39 MPIO_1005U

-r-xr-xr-x    1 root     system          715 May 24 16:57 XYZpost.ksh

-rw-r--r--    1 root     system         2310 May 25 14:57 .toc

 

The contents of my script are simple. This script will de-install the old device fileset and then immediately install the latest version of the VendorXYZ’s device fileset. The script will then change the attributes for the vendor’s storage to more appropriate default values.

 

 

 

At this point I copy the same directory and all of its contents to the NIM client.

 

root@nim1 : /usr/local # scp –pr XYZ lparaix01:/usr/local/

…etc…

 

lparaix01 : /usr/local/XYZ # ls -ltr

total 0

-r-xr-xr-x    1 root     system        51200 Mar 11 2011  MPIO_1001I

-r-xr-xr-x    1 root     system        51200 Mar 11 2011  MPIO_1002U

-r-xr-xr-x    1 root     system        51200 Mar 11 2011  MPIO_1003U

-r-xr-xr-x    1 root     system        51200 Mar 11 2011  MPIO_1004U

-r-xr-xr-x    1 root     system        51200 May 18 16:39 MPIO_1005U

-r-xr-xr-x    1 root     system          715 May 24 16:57 XYZpost.ksh

-rw-r--r--    1 root     system         2310 May 25 14:57 .toc

 

Make sure that any scripts you write for use with nimadm start with an appropriate ‘hashbang’ to announce it is a shell script and the shell that must be used to execute it e.g. #!/usr/bin/ksh. If you forget to do this nimadm will fail to execute your script and will report an error message similar to the following:

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 7.

+-----------------------------------------------------------------------------+

Executing user chroot script /usr/local/XYZ/XYZpost.ksh.

/lparaix01_alt/alt_inst/tmp/.alt_mig_chroot_script.11731036: Cannot run a file that does not have a valid format.

 

The next step is to define the script as a NIM resource so that nimadm can call the resource during the migration process. I’ve decided to call this new NIM resource, XYZPOST.

 

This is easily achieved using smit nim_mkres:

 

root@nim1 : / # smit nim_mkres

 

|   script          = an executable file which is executed on a client     |

 

  

          Define a Resource

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                                                        [Entry Fields]

* Resource Name                                      [XYZPOST]

* Resource Type                                       script

* Server of Resource                                 [master]         +

* Location of Resource                [/usr/local/XYZ/XYZpost.ksh]    /

 

We can confirm that the NIM script resource is now available using the lsnim command.

 

root@nim1 : / # lsnim -t script

XYZPOST     resources       script

 

root@nim1 : / # lsnim -l XYZPOST

XYZPOST:

   class       = resources

   type        = script

   Rstate      = ready for use

   prev_state  = unavailable for use

   location    = /usr/local/XYZ/XYZpost.ksh

   alloc_count = 0

   server      = master                                                                 

 

Now that the script is in place, and defined to NIM, we are ready to test it. We will migrate the system from AIX 5.3 to AIX 6.1 using nimadm. Once the migration phase is complete (phases 1 to 6), the post-migration script will be executed in the NIM clients nimadm (chroot) environment on the NIM master. Once this is finished the NIM clients data is synced back to the NIM clients alternate disk and the boot image is created. The migration process is then complete.

 

We add the –z flag to our nimadm command line options to specify the post migration resource.

 

In normal operation we would simply let nimadm run all phases in sequence with the following command.

 

root@nim1 : / # nimadm -j nimadmvg -c lparaix01 -s spotaix610605 -l lpp_sourceaix610605 -d hdisk2 -z XYZPOST -Y

 

In our case we would like to test the post-migration script phase as a separate step.

 

We will start by running only the first 6 phases. We will stop at the completion of phase 6 and run phase 7 (post-migration script) manually, before moving on to phases 8 through 12.

 

You’ll observe that we have specified the –z option with the name of the NIM script resource and the –P flag with phases 1 to 6 only.

 

 

root@nim1 : / # nimadm -j nimadmvg -c lparaix01 -s spotaix610605 -l lpp_sourceaix610605 -d hdisk2 -z XYZPOST -P1,2,3,4,5,6 -Y

Initializing the NIM master.

Initializing NIM client lparaix01.

Verifying alt_disk_migration eligibility.

Initializing log: /var/adm/ras/alt_mig/lparaix01_alt_mig.log

Starting Alternate Disk Migration.

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 1.

+-----------------------------------------------------------------------------+

Cloning altinst_rootvg on client, Phase 1.

Client alt_disk_install command: alt_disk_copy -j -M 6.1 -P1 -d "hdisk2"

Calling mkszfile to create new /image.data file.

...etc...

Filesets processed:  1 of 2  (Total time:  3 secs).

 

installp:  COMMITTING software for:

        X11.compat.lib.X11R6_motif 6.1.6.15

 

Finished processing all filesets.  (Total time:  4 secs).

...etc...

install_all_updates: Checking for recommended maintenance level 6100-06.

install_all_updates: Executing /usr/bin/oslevel -rf, Result = 6100-06

install_all_updates: Verification completed.

install_all_updates: Log file is /var/adm/ras/install_all_updates.log

install_all_updates: Result = SUCCESS

Known Recommended Maintenance Levels

------------------------------------

Restoring device ODM database.

 

root@nim1 : / #

 

Phase 6 has completed successfully. The NIM clients rootvg data has been migrated from AIX 5.3 to 6.1 on the NIM master. The data has not yet been synced back to the NIM client.

 

At this stage we can now run phase 7 separately and ensure that it performs the required task. We expect it will de-install the device fileset, install the latest version and change the ODM default attributes for the device type. Again, you’ll notice that we specify the –P flag for phase 7 only.

 

Let’s test it.

 

root@nim1 : / # nimadm -j nimadmvg -c lparaix01 -s spotaix610605 -l lpp_sourceaix610605 -d hdisk2 -z XYZPOST -P7 -Y

Initializing the NIM master.

Initializing NIM client lparaix01.

Initializing log: /var/adm/ras/alt_mig/lparaix01_alt_mig.log

Starting Alternate Disk Migration.

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 7.

+-----------------------------------------------------------------------------+

Executing user chroot script /usr/local/XYZ/XYZpost.ksh.

Uninstalling XYZ ODM fileset: devices.fcp.disk.VendorXYZ.mpio.rte

+-----------------------------------------------------------------------------+

                    Pre-deinstall Verification...

+-----------------------------------------------------------------------------+

Verifying selections...done

Verifying requisites...done

Results...

 

FILESET STATISTICS

------------------

    1  Selected to be deinstalled, of which:

        1  Passed pre-deinstall verification

  ----

    1  Total to be deinstalled

 

SUCCESSES

---------

  Filesets listed in this section passed pre-deinstall verification

  and will be removed.

 

  Selected Filesets

  -----------------

  devices.fcp.disk.VendorXYZ.mpio.rte  1.2.0.0 # AIX MPIO Support for VendorXYZ Disk

 

  << End of Success Section >>

 

+-----------------------------------------------------------------------------+

                           Deinstalling Software...

+-----------------------------------------------------------------------------+

 

installp:  DEINSTALLING software for:

        devices.fcp.disk.VendorXYZ.mpio.rte  1.2.0.0

 

...etc...

Installation Summary

--------------------

Name                        Level           Part        Event       Result

-------------------------------------------------------------------------------

devices.fcp.disk.VendorXYZ.mpio 1.2.0.0         USR         DEINSTALL   SUCCESS

Return code is: 0

 

Installing XYZ ODM fileset: devices.fcp.disk.VendorXYZ.mpio.rte

+-----------------------------------------------------------------------------+

                    Pre-installation Verification...

+-----------------------------------------------------------------------------+

Verifying selections...done

Verifying requisites...done

Results...

 

FILESET STATISTICS

------------------

    1  Selected to be installed, of which:

        1  Passed pre-installation verification

  ----

    1  Total to be installed

 

SUCCESSES

---------

  Filesets listed in this section passed pre-installation verification

  and will be installed.

 

  Selected Filesets

  -----------------

  devices.fcp.disk.VendorXYZ.mpio.rte  1.4.0.0 # AIX MPIO Support for VendorXYZ Disk

 

  << End of Success Section >>

 

+-----------------------------------------------------------------------------+

                         Installing Software...

+-----------------------------------------------------------------------------+

 

installp:  APPLYING software for:

        devices.fcp.disk.VendorXYZ.mpio.rte  1.4.0.0

 

 

. . . . . << Copyright notice for devices.fcp.disk.VendorXYZ.mpio >> . . . . . . .

...etc...

Installation Summary

--------------------

Name                        Level           Part        Event       Result

-------------------------------------------------------------------------------

devices.fcp.disk.VendorXYZ.mpio 1.4.0.0         USR         APPLY       SUCCESS

Return code is: 0

 

Changing default ODM (PdAt) attributes for VendorXYZ storage

 

+ odmget -q uniquetype=disk/fcp/xyzmpio and attribute=queue_depth PdAt

+ sed s/deflt = "2"/deflt = "32"/g

+ 1> /usr/local/XYZ/chg_pdat_qd

+ odmchange -o PdAt -q uniquetype=disk/fcp/xyzmpio and attribute=queue_depth /usr/local/XYZ/chg_pdat_qd

+ odmget -q uniquetype=disk/fcp/xyzmpio and attribute=reserve_policy PdAt

+ sed s/deflt = "single_path"/deflt = "no_reserve"/g

+ 1> /usr/local/XYZ/chg_pdat_rp

+ odmchange -o PdAt -q uniquetype=disk/fcp/xyzmpio and attribute=reserve_policy /usr/local/XYZ/chg_pdat_rp

+ odmget -q uniquetype=PCM/friend/VendorXYZ and attribute=algorithm PdAt

+ sed s/deflt = "fail_over"/deflt = "round_robin"/g

+ 1> /usr/local/XYZ/chg_pdat_al

+ odmchange -o PdAt -q uniquetype=PCM/friend/VendorXYZ and attribute=algorithm /usr/local/XYZ/chg_pdat_al

 

root@nim1 : / #

 

Great news! Our script has worked as expected. The old fileset was de-installed, the new fileset was installed and the PdAt default attributes were changed successfully.

 

Note: You can also review the post migration script output at a later date if you wish. All nimadm activities are logged, on the NIM master, to /var/adm/ras/alt_mig/NIMclientname_alt_mig.log (where NIMclientname is the name of the NIM client being migrated by nimadm).

 

With regard to nimadm log files, please be aware that if you choose to run nimadm in phases (as I’ve shown in this example) that each run will generate a new log file. So in my case, when I ran phases 1 to 6, this created a log file named lparaix01_alt_mig.log. When I ran phase 7, the original log file was moved to lparaix01_alt_mig.log.prev. A new log file was created and used for phase 7. Then when I ran phases 8 to 12, the phase 7 log file was moved to lparaix01_alt_mig.log.prev and a new log file was used for phases 8-12. For this reason you may want to backup each log file to a unique file name as you execute each phase group, so that you do not lose any of the information logged to the .log or .log.prev files.

 

Now we can complete the rest of the migration and execute the remaining phases, 8 through 12.

 

root@nim1 : / # nimadm -j nimadmvg -c lparaix01 -s spotaix610605 -l lpp_sourceaix610605 -d hdisk2 -z XYZPOST -P8,9,10,11,12 -Y

Initializing the NIM master.

Initializing NIM client lparaix01.

Initializing log: /var/adm/ras/alt_mig/lparaix01_alt_mig.log

Starting Alternate Disk Migration.

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 8.

+-----------------------------------------------------------------------------+

Creating client boot image.

bosboot: Boot image is 46989 512 byte blocks.

Writing boot image to client's alternate boot disk hdisk2.

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 9.

+-----------------------------------------------------------------------------+

Adjusting client file system sizes ...

Adjusting size for /

Adjusting size for /admin

Adjusting size for /home

Adjusting size for /opt

Adjusting size for /tmp

Adjusting size for /usr

Expanding /alt_inst/usr client filesystem.

Filesystem size changed to 4325376

Adjusting size for /usr/local

Adjusting size for /usr/local/chksys

Adjusting size for /var

Adjusting size for /var/log

Adjusting size for /var/log/audit

Adjusting size for /var/tsm/log

Syncing cache data to client ...

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 10.

+-----------------------------------------------------------------------------+

Unmounting client mounts on the NIM master.

forced unmount of /lparaix01_alt/alt_inst/var/log/audit

forced unmount of /lparaix01_alt/alt_inst/var/log

forced unmount of /lparaix01_alt/alt_inst/var

forced unmount of /lparaix01_alt/alt_inst/usr/local

forced unmount of /lparaix01_alt/alt_inst/usr

forced unmount of /lparaix01_alt/alt_inst/tmp

forced unmount of /lparaix01_alt/alt_inst/opt

forced unmount of /lparaix01_alt/alt_inst/home

forced unmount of /lparaix01_alt/alt_inst/admin

forced unmount of /lparaix01_alt/alt_inst

Removing nimadm cache file systems.

Removing cache file system /lparaix01_alt/alt_inst

Removing cache file system /lparaix01_alt/alt_inst/admin

Removing cache file system /lparaix01_alt/alt_inst/home

Removing cache file system /lparaix01_alt/alt_inst/opt

Removing cache file system /lparaix01_alt/alt_inst/tmp

Removing cache file system /lparaix01_alt/alt_inst/usr

Removing cache file system /lparaix01_alt/alt_inst/usr/local

Removing cache file system /lparaix01_alt/alt_inst/var

Removing cache file system /lparaix01_alt/alt_inst/var/log

Removing cache file system /lparaix01_alt/alt_inst/var/log/audit

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 11.

+-----------------------------------------------------------------------------+

Cloning altinst_rootvg on client, Phase 3.

Client alt_disk_install command: alt_disk_copy -j -M 6.1 -P3 -d "hdisk2"

## Phase 3 ###################

Verifying altinst_rootvg...

Modifying ODM on cloned disk.

forced unmount of /alt_inst/var/log/audit

forced unmount of /alt_inst/var/log

forced unmount of /alt_inst/var

forced unmount of /alt_inst/usr/local

forced unmount of /alt_inst/usr

forced unmount of /alt_inst/tmp

forced unmount of /alt_inst/opt

forced unmount of /alt_inst/home

forced unmount of /alt_inst/admin

forced unmount of /alt_inst

Changing logical volume names in volume group descriptor area.

Fixing LV control blocks...

Fixing file system superblocks...

Bootlist is set to the boot disk: hdisk2 blv=hd5

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 12.

+-----------------------------------------------------------------------------+

Cleaning up alt_disk_migration on the NIM master.

Cleaning up alt_disk_migration on client lparaix01.

root@nim1 : / #

 

Our nimadm operation is finished. The NIM client is rebooted. We verify it is now running AIX 6.1. And the correct version of the vendor’s device fileset is installed.

 

lparaix01 : / # oslevel -s

6100-06-05-1115

 

lparaix01 : / # lslpp -l devices.fcp.disk.VendorXYZ.mpio.rte 

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  devices.fcp.disk.VendorXYZ.mpio.rte

                                1.4.0.0  COMMITTED  AIX MPIO Support for VendorXYZ Disk

 

lparaix01 : / # lsdev -Cc disk

hdisk1 Available 20-T1-01 VendorXYZ MPIO Disk

hdisk2 Available 20-T1-01 VendorXYZ MPIO Disk

hdisk3 Available 20-T1-01 VendorXYZ MPIO Disk

hdisk4 Available 20-T1-01 VendorXYZ MPIO Disk

 

And there you have it, an example of using a post migration script with nimadm.

 

nimadm Command

http://pic.dhe.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.cmds/doc/aixcmds4/nimadm.htm

Make sure you are monitoring file system space on your VIOS.

 

Why?

 

If you run out of space in the root file system, odd things can happen when you try to map virtual devices to virtual adapters with mkvdev.

 

For example, a colleague of mine was attempting to map a new hdisk to a vhost adapter on a pair of VIOS. The VIOS was running a recent version of code. He received the following error message (see below). It wasn’t a very helpful message. At first I thought it was due to the fact that he had not set the reserve_policy attribute for the new disk to no_reserve on both VIOS. Changing the value for that attribute did not help.

 

$ ioslevel

2.2.1.3

 

$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1

*******************************************************************************

The command's response was not recognized.  This may or may not indicate a problem.

*******************************************************************************

 

I enabled debug output from the padmin command line and discovered that the mkdev command was failing for some reason.

 

$ export CLI_DEBUG=33

 

$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1

AIX: "lspv -l hdisk1 2>&1 | grep 0516-320"

AIX: "mkdev  -V hdisk1  -p vhost0  -l vhdisk1 "

*******************************************************************************

The command's response was not recognized.  This may or may not indicate a problem.

*******************************************************************************

 

I entered oem_setup_env and ran the mkdev command manually. I was presented with a very clear error message describing the problem. The root file system (/) was full!

 

$ oem_setup_env

 

# mkdev  -V hdisk1  -p vhost0  -l vhdisk1

mkdev: 0514-548 Cannot perform the requested function because

                the / filesystem is full or is out of inodes.

 

# df -g /

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4           0.50      0.00  100%     7728    75% /

 

I found a 45MB core file (generated by cimserver) in the root file system. I moved the file to another file system (with plenty of free space) for further investigation.

 

# mv core /tmp/core.cg

# df -m .

Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4         512.00     45.93   92%     7727    38% /

 

Then I ran the mkvdev command successfully.

 

$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1

AIX: "lspv -l hdisk1 2>&1 | grep 0516-320"

AIX: "mkdev  -V hdisk1  -p vhost0  -l vhdisk1 "

vhdisk1 Available

AIX: "lsattr -El hdisk1 -a max_transfer |cut -d" " -f2"

 

$ lsmap -all

AIX: "lsdev -c adapter -t IBM,v-scsi-host -s vdevice -F "name" | wc -l -c"

AIX: "lsdev -c adapter -t IBM,v-scsi-host -s vdevice -F "name""

AIX: "lsdev -C -l vhost0 -F "physloc""

AIX: "lsdev -p vhost0 -F "name" | wc -l -c"

AIX: "lsdev -p vhost0 -F "name""

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost0          U9119.FHA.87654B1-V6-C21                   0x0000002e

 

VTD                   vhdisk1

Status                Available

LUN                   0x8100000000000000

Backing device        hdisk1

Physloc               U5873.001.9SS005Y-P1-C9-T1-W50060E8006D0206A-L1000000000000

Mirrored              false

 

AIX: "lsdev -C -l vhost1 -F "physloc""

AIX: "lsdev -p vhost1 -F "name" | wc -l -c"

AIX: "lsdev -p vhost1 -F "name""

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost1          U9119.FHA.87654B1-V6-C22                   0x0000002g

 

VTD                   NO VIRTUAL TARGET DEVICE FOUND

 

I found the same issue on the second VIOS i.e. a full root file system due to a core file (from cimserver). I also found no trace of a full file system event in the error report. Perhaps someone had taken it upon themselves to “clean house” at some point and had removed entries from the VIOS error log.

 

Make sure you monitor file system space on your VIOS. Who knows what else might fail if you run out of space in a critical file system.

Are you backing up your AIX systems over Virtual Ethernet adapters? Of course you are, who isn’t right? Are your backup server and clients on the same physical POWER system? You are most likely backing up over Virtual Ethernet to another AIX LPAR that is running your enterprise backup software, such as TSM or Legato Networker for example. And you probably have a dedicated private virtual network (and adapters) on both the clients and the server to handle the traffic for the nightly backups. The next question is, have you tuned your Virtual Ethernet adapters?

 

There are several tips available for tuning your Virtual Ethernet adapters for better performance on AIX. These tips include changing settings such as MTU size, TCP window sizes, enabling largesend, etc. I highly recommend the following blog posts from Anthony English and Nigel Griffiths on this subject:

 

https://www.ibm.com/developerworks/mydeveloperworks/blogs/AIXDownUnder/entry/turbo_charge_inter_lpar_communications_using_jumbo_frames1?lang=en

 

https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/10gbit_ethernet_bad_assumption_and_best_practice_part_137?lang=en

 

https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/powervm_virtual_ethernet_speed_is_often_confused_with_vios_sea_ive_hea_speed?lang=en

 

OK, so you got everything humming along nicely, your backups are flying over the virtual network (across the POWER hypervisor) and everybody is happy. After a period of time, you notice that the backups have started to “slow down”. They are taking longer to finish. The overall throughput of a backup drops. Some backups start in the evening around 9pm and are still running the next morning at 7am! In some cases you need to kill the backups or even reboot the backup server LPAR for things to return to normal.

 

“What is going on!?” You cry.

 

Well, there are a number of reasons why this could be happening. For example, your shared processor pool may be overwhelmed during the backup window. As we know, Virtual Ethernet adapters require CPU to do their work. If the CPU pool is running low on available CPU resources, this could contribute to the problem. And of course there could be tuning issues with the Virtual Ethernet adapters or the AIX OS in general. Or there may be issues with other pieces of the infrastructure, like network and SAN switches, adapters, etc. Perhaps there’s an issue with the applications and/or databases on the AIX systems? They often have their own mechanisms/tools for backing up their data to your enterprise backup software. Is the backup server sized to cope with the load i.e. CPU, memory, disk layout and I/O, sufficient tape drives, disk storage pools, etc?

 

So assuming you’ve checked all of the above (and more), then perhaps you’ve hit a problem that I encountered recently. In my particular case, backups “over the hypervisor” were slowing down, without any discernible cause. Initially the backups would be “very fast” but after a month or so, things would start to slow down dramatically.

 

We noticed that there were very large (and increasing) values for “Packets Dropped”, “Hypervisor Send/Receive Failures” and “No Resource Errors” in the output from the netstat –v command.                              

                                                                       

ETHERNET STATISTICS (ent1) :                                           

Device Type: Virtual I/O Ethernet Adapter (l-lan)                      

Hardware Address: 41:ba:13:e7:25:0b                                    

Elapsed Time: 42 days 4 hours 3 minutes 34 seconds                    

                                                                       

Transmit Statistics:       Receive Statistics:                         

--------------------       -------------------                         

Packets: 5978589961        Packets: 26139832411                        

Bytes: 779465989202        Bytes: 711051516630458                      

Interrupts: 0              Interrupts: 6804561727                      

Transmit Errors: 0         Receive Errors: 0                           

Packets Dropped: 0         Packets Dropped: 86012309               

...                                                                                             

Max Collision Errors: 0    No Resource Errors: 46113807           

...                                                                       

Hypervisor Send Failures: 0                                            

  Receiver Failures: 0                                                  

  Send Errors: 0                                                       

Hypervisor Receive Failures: 46113807                             

                                                                       

After some discussion with IBM AIX support, we discovered that would should increase some of the buffer sizes for our Virtual Ethernet adapter (the entX device). This would alleviate the no resource issues we’d been experiencing. Looking at the output from the netstat -v command, we also noticed that the Medium, Large and Huge buffers had all reached their maximum values in the past.                                                           

                                                                       

...

Receive Information                                                    

  Receive Buffers                                                      

    Buffer Type              Tiny    Small   Medium    Large     Huge  

    Min Buffers               512      512      128       24       24  

    Max Buffers              2048     2048      256       64       64  

    Allocated                 513      535      148       28       64  

    Registered                512      510      127       24       13  

                                                                       

    History                                                            

      Max Allocated           576      951      256       64       64  

      Lowest Registered       502      502       64       12       11  

...

                                                                   

The advice from IBM support was to increase these buffers using the chdev command (they also advised that we should reboot for the changes to take effect): 

                                                                       

# chdev -l ent1 -a min_buf_medium=512 -a max_buf_medium=1024 –a min_buf_large=96 -a max_buf_large=256 -a min_buf_huge=96 –a max_buf_huge=128 -P             

                                       

# shutdown -Fr

 

Since implementing this tuning change (to the adapter on the backup server), we have not had a repeat of the problem. We will continue to monitor the performance and I’ll be sure to let everyone know if we have further issues.

This entry is similar in theme to one of my previous posts about verifying your hdisk queue_depth settings with kdb. This time we want to check if an attribute for a Virtual FC (VFC) adapter has been modified and whether or not AIX has been restarted since the change. The attribute I’m interested in is num_cmd_elems. This value is often changed from its default settings, in AIX environments, to improve I/O performance on SAN attached storage.

 

From kdb you can identify the VFC adapters configured on an AIX system using the vfcs subcommand. Not only does this tell you what adapters you have, but it also identifies the VIOS each adapter is connected to and the corresponding vfchost adapter. Nice!

 

(0)> vfcs

NAME      ADDRESS             STATE   HOST    HOST_ADAP  OPENED NUM_ACTIVE

fcs0      0xF1000A00103D4000  0x0008  vio1    vfchost10  0x01   0x0000

fcs1      0xF1000A00103D6000  0x0008  vio2    vfchost10  0x01   0x0000

 

You can view the current (running) configuration of a VFC adapter using the kdb vfcs subcommand and the name of the VFC adapter, for example fcs1:

 

(0)> vfcs fcs1

adapter: fcs0   partition_num: 0x14     partition_name: aixlpar1

num_cmd_elems: 0xC8     location_code: U9119.FHA.87654A1-V20-C10-T1

state: 0x8  opened: 0x1  flags: 0x1

host_name: vio1       host_device: vfchost10

maxCmds: FA     maxDMALength: 100000    SCSIid: 6F1E9A

portName: C050760399100168      nodeName: C050760399100168      nport_id:

6F1E9A

cmd_q.base_addr: 0xF1000A00102D0000     asyncq.base_addr: 0xF1000A0019761000

crq[0]: 0x8001000000000000      crq[1]: 0x8013960

ctl_head: 0x0   ctl_tail: 0x0

cmd_head: 0x0   cmd_tail: 0x0

pool_head: 0xF1000A001975BD08   pool_tail: 0xF1000A0019759D88

head_active: 0x0        tail_active: 0x0

cancel_head: 0xF1000A0019758090 cancel_tail: 0xF1000A0019758E58

num_cmds_allowed: 0xFA  num_cmds_active: 0x0    num_cmds_pending: 0x0

head_flush_q: 0x0       tail_flush_q: 0x0       cmd_flush_q: 0x0

no_elem: 0x0  no_dma: 0x0  no_sglist: 0x0  bad_mad: 0x0  qfull: 0x0

h_dropped: 0x0  link_down_cnt: 0x0  num_frames: 0x0  tx_lock: 0x0

rcv_lock: 0x0   ctl_lock: 0x0   lock: 0xFFFFFFFFFFFFFFFF

ctl_event: 0xFFFFFFFFFFFFFFFF   open_event: 0xFFFFFFFFFFFFFFFF

reset_cmd: 0x0  log_cmd: 0x0

link_cmd: 0x0   ctl_cmd->cmd_state: 0x0

inp_reqs: 0x12594       out_reqs: 0x1A7FD4      ctrl_reqs: 0x44

inp_bytes: 0x14973F90  out_bytes: 0xEC14EC18  npiv_logout_sent: 0x0

npiv_logout_rcvd: 0x0  act_hi_water: 0x10  pend_hi_water: 0x1

 

Using the output from this command we can determine the current (running) value for a number of VFC attributes, including num_cmd_elems.

 

So I start with an adapter with a num_cmd_elems value of 200. Both the lsattr command and kdb report 200 (C8 in hex) for num_cmd_elems.

 

# lsattr -El fcs1 -a num_cmd_elems

num_cmd_elems 200 Maximum Number of COMMAND Elements True

 

# echo vfcs fcs1 | kdb | grep num_cmd_elems

num_cmd_elems: 0xC8     location_code: U9119.FHA.87654A1-V20-C10-T1

 

I change num_cmd_elems to 400 with chdev –P (remember, the –P flag only updates the AIX ODM, and not the running configuration of the device in the AIX kernel. You must either reboot for this change to take effect or offline & online the device).

 

# chdev -l fcs1 -a num_cmd_elems=400 -P

fcs1 changed

 

Now the lsattr command reports num_cmd_elems is set to 400 in the ODM.

 

# lsattr -El fcs1 -a num_cmd_elems

num_cmd_elems 400 Maximum Number of COMMAND Elements True

 

But kdb still has num_cmd_elems set to C8 (200).

 

# echo vfcs fcs1 | kdb | grep num_cmd_elems

num_cmd_elems: 0xC8     location_code: U9119.FHA.87654A1-V20-C10-T1

 

Using this technique I now have a way of checking if an AIX system has been restarted since an attribute was changed on a VFC adapter.

 

At this point I could suggest a reboot of the system. Or I could take the adapter offline and then online for the changes to take effect. For example:

 

# lspath

Enabled hdisk0 fscsi0

Enabled hdisk1 fscsi0

Enabled hdisk0 fscsi1

Enabled hdisk1 fscsi1

 

# rmpath -l hdisk0 -p fscsi1

# rmpath -l hdisk1 -p fscsi1

 

# lspath

Enabled hdisk0 fscsi0

Enabled hdisk1 fscsi0

Defined hdisk0 fscsi1

Defined hdisk1 fscsi1

 

# rmdev -Rl fcs1

fscsi1 Defined

fcs1 Defined

 

# cfgmgr

 

# lspath

Enabled hdisk0 fscsi0

Enabled hdisk1 fscsi0

Enabled hdisk0 fscsi1

Enabled hdisk1 fscsi1

 

# lsattr -El fcs1 –a num_cmd_elems

num_cmd_elems 400      Maximum Number of COMMAND Elements True

 

# echo vfcs fcs1 | kdb | grep num_cmd_elems

num_cmd_elems: 0x190    location_code: U9119.FHA.87654A1-V20-C10-T1

 

;; 0x190 hex = 400 decimal

 

Please note: Be careful when using the kdb command. If used incorrectly, you can crash an AIX system!

Starting with AIX 7.1, CSM is no longer supported or available. It has been replaced by Distributed Systems Managment (DSM).   Section 5.2 of the IBM AIX 7.1 Differences Guide Redbook provides details of the new DSM capabilities.

 

Fortunately DSM still provides access to the dsh command.  I’ve written about how I’ve used this utility in the past. The new dsh command (and other tools) are provided in the new DSM filesets named dsm.core and dsm.dsh.

 

root@nim# lslpp -L dsm*

  Fileset                      Level  State  Type  Description (Uninstaller)

  ----------------------------------------------------------------------------

  dsm.core                   7.1.1.2    A     F    Distributed Systems Management

                                                   Core

  dsm.dsh                    7.1.0.0    C     F    Distributed Systems Management

                                                   Dsh

 

root@nim# lslpp -w /usr/bin/dsh

  File                                        Fileset               Type

  ----------------------------------------------------------------------------

  /usr/bin/dsh                                dsm.dsh               Symlink

 

 

These filesets are NOT installed by default. You must manually install them. They can be found on your AIX 7.1 media.

 

If dsh is something you use, then I recommend you read the section on DSM in the Redbook. Also take a look at section 5.2.7 Using DSM and NIM, in which it describes how you can integrate DSM and NIM and completely automate the installation of AIX:

 

“The AIX Network Installation Manager (NIM) has been enhanced to work with the Distributed System Management (DSM) commands. This integration enables the automatic installation of new AIX systems that are either currently powered on or off.”

 

Although I’ve written about the dsh command before, there’s one usage I’ve not covered.  And that is using dsh to manage users across a group of LPARs. In particular, changing a user’s password.

 

Before I go any further, I should state that for the following to work you must first configure ssh keys on your NIM master (or central mgmt AIX system) so that you can communicate with all of your AIX systems via SSH, as root,  without being prompted for a password. Read my article on dsh to find out how to do this if necessary.

 

In the following example, I use dsh from my NIM master. It is my central point of control for my AIX environment.

 

My ssh keys for root on my NIM master have been generated and distributed to all of my LPARs.

 

root@nim# ssh-keygen -d

Generating public/private dsa key pair.

Enter file in which to save the key (/.ssh/id_dsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /.ssh/id_dsa.

Your public key has been saved in /.ssh/id_dsa.pub.

The key fingerprint is:

ed:18:e9:00:37:13:7c:7c:74:6a:a9:e0:ad:c0:09:a9 root@nim

The key's randomart image is:

+--[ DSA 1024]----+

|     ... .. .    |

|   .  ..o .+     |

|  o . =. .+      |

| . o = = =       |

|E   + o S .      |

|     . + +       |

|      . o .      |

|                 |

|                 |

+-----------------+

 

root@nim# ls -ltra

total 40

-rw-------    1 root     system          214 17 Sep 2010  authorized_keys

drwxr-xr-x    7 root     system         4096 16 Nov 11:43 ..

-rw-r--r--    1 root     system         3615 16 Nov 12:04 known_hosts

-rw-r--r--    1 root     system          601 16 Nov 12:06 id_dsa.pub

-rw-------    1 root     system          672 16 Nov 12:06 id_dsa

drwx------    2 root     system          256 16 Nov 12:06 .

 

On my AIX LPARs, the authorized_keys file has been updated with the public ssh key from my NIM master:

 

root@aixlpar1 [/.ssh]# chmod 600 known_hosts authorized_keys

root@aixlpar1 [/.ssh]# ls -ltr

total 0

-rw-------    1 root     system          406 Sep 29 15:30 known_hosts

-rw-------    1 root     system          601 Nov 16 12:07 authorized_keys

 

root@aixlpar1 [/.ssh]# cat authorized_keys

ssh-dss AAAAB3NzaC1kc3MAAACBAPJpekVQgV2CTu5tCOsukwXwMrkfBabLAN7vwjgPk3S2p65COJkFwaDRWwBoK87cnMrrc7/GuOLdk/RiqYw1K8Z0d9RWnsymYrNm0MBALWioGGXwQ+YlLiYHIQsSArnajZOH6Ye36qz81uWNoj6u4xIxi/iGSYoU4x3UE/c/H833AAAAFQDKEtWVFk3G+KAyxUsI6FcpbJZqDQAAAIAVUVRak/s+X0mMElCaEs8Rh96GDfhorXWV1YzI0nbRo8QX2AMix+3LSdEmYCRn8TYpk1Udln3GG4gzy5oI/3Rd1RTWgAo+067LwytR6IJfwYt219Pwh2ktrm5iZu3c+yDBqQj6hsga2YIun5lxqYFy39CZNtslnR1wTx9uCcYkowAAAIEAuVFTsW9ugVM76CgluspNo0faz9xFo4DCOjCgye+k/ZqSDaNq6Xrvxktw+dkHuKHbhvofPgeiMrlqhdbrE0ovLUr8aadxArRhitnu10CFYbRwlrjAxyAaRwjD2pXG8M3a5D1XXAWDCDm6bEs5yxJqcl1wq+8UAhpKiS5yj2pwApk= root@nim

 

 

On the NIM master, the root user was configured for the DSH environment. The following entry was placed in roots .profile:

 

root@nim# cat /.profile

ENV=$HOME/.kshrc

 

The following entry was placed in roots .kshrc file:

 

root@nim# cat /.kshrc

export DSH_NODE_RSH=/usr/bin/ssh

export DSH_NODE_LIST=/usr/local/etc/nodes

 

A /usr/local/etc/nodes file was created on the NIM master. This file contains a list of each of the nodes that dsh can communicate with from NIM:

 

root@nim# cat /usr/local/etc/nodes

aixlpar1

aixlpar2

aixlpar3

aixlpar4

aixlpar5

aixlpar6

aixlpar7

aixlpar8

aixlpar9

aixlpar10

aixlpar11

 

The first time that the dsh command is run against a new host, the following message will be displayed. dsh uses the FQDN, and the FQDN needs to be added to the known_hosts file for ssh. Therefore you must make an ssh connection first with FQDN to the host:

 

root@nim# dsh uptime

aixlpar1.cg.com.au : Host key verification failed.

dsh:  2617-009 aixlpar1.cg.com.au  remote shell had exit code 255

 

It is necessary to ssh directly to each node using its FQDN. This step is only required once for each node. For example:

 

root@nim# ssh aixlpar1.cg.com.au

The authenticity of host 'aixlpar1.cg.com.au  (172.1.6.17)' can't be established.

RSA key fingerprint is 83:13:1d:cb:97:ee:00:7a:04:75:62:98:bd:e8:30:79.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'aixlpar1.cg.com.au ' (RSA) to the list of known hosts.

Last unsuccessful login: Tue Sep 27 03:04:33 GMT+10:00 2011 on ssh from 172.1.6.15

Last login: Wed Nov 16 12:08:08 GMT+11:00 2011 on /dev/pts/3 from nim

 

 

root@aixlpar1 [/root]# exit

Connection to aixlpar1.cg.com.au  closed.

 

Now we can use the dsh command to run a command across all the nodes defined in the DSH_NODE_LIST. For example, the following command will run the uptime command on all nodes:

 

root@nim# dsh uptime

aixlpar1.cg.com.au :   12:14PM   up 10 days,  22:32,  3 users,  load average: 0.47, 0.53, 0.64

aixlpar2.cg.com.au :   12:14PM   up 21 days,  21:13,  1 users,  load average: 0.11, 0.31, 0.51

aixlpar3.cg.com.au :   12:14PM   up 15 days,  20:33,  5 users,  load average: 0.17, 0.43, 0.54

...etc...

 

With dsh now configured, we can use it to perform several admin tasks from a central location. For example, the following commands will do the following:

 

1.       Create a user, named cg, on all nodes.

2.        Set cg’s password to abc123 using the chpasswd command.

3.        Remove the ADMCHG flag for the user cg. This will stop them from being prompted to change their password on their first login attempt.

 

First I check that user does not exist on the AIX systems.

 

root@nim# dsh lsuser cg

aixlpar1: 3004-687 User "cg" does not exist.

aixlpar2: 3004-687 User "cg" does not exist.

aixlpar3: 3004-687 User "cg" does not exist.

aixlpar4: 3004-687 User "cg" does not exist.

aixlpar5: 3004-687 User "cg" does not exist.

...etc...

 

I create the user with the mkuser command.

 

root@nim# dsh mkuser cg

 

I confirm that the user has been created on all the AIX systems, with the lsuser command.

 

root@nim# dsh lsuser cg

aixlpar1: cg id=204 pgrp=staff groups=staff home=/home/cg shell=/usr/bin/ksh login=true su=true rlogin=true daemon=true admin=false sugroups=ALL admgroups= tpath=nosak ttys=ALL expires=0 auth1=SYSTEM auth2=NONE umask=22 registry=files SYSTEM=compat logintimes= loginretries=0 pwdwarntime=0 account_locked=false minage=0 maxage=0 maxexpired=-1 minalpha=0 minother=0 mindiff=0 maxrepeats=8 minlen=0 histexpire=0 histsize=0 pwdchecks= dictionlist= fsize=-1 cpu=-1 data=262144 stack=65536 core=2097151 rss=65536 nofiles=2000 roles=

aixlpar2: cg id=204 pgrp=staff groups=staff home=/home/cg shell=/usr/bin/ksh login=true su=true rlogin=true daemon=true admin=false sugroups=ALL admgroups= tpath=nosak ttys=ALL expires=0 auth1=SYSTEM auth2=NONE umask=22 registry=files SYSTEM=compat logintimes= loginretries=0 pwdwarntime=0 account_locked=false minage=0 maxage=0 maxexpired=-1 minalpha=0 minother=0 mindiff=0 maxrepeats=8 minlen=0 histexpire=0 histsize=0 pwdchecks= dictionlist= fsize=-1 cpu=-1 data=262144 stack=65536 core=2097151 rss=65536 nofiles=2000 roles=

aixlpar3: cg id=204 pgrp=staff groups=staff home=/home/cg shell=/usr/bin/ksh login=true su=true rlogin=true daemon=true admin=false sugroups=ALL admgroups= tpath=nosak ttys=ALL expires=0 auth1=SYSTEM auth2=NONE umask=22 registry=files SYSTEM=compat logintimes= loginretries=0 pwdwarntime=0 account_locked=false minage=0 maxage=0 maxexpired=-1 minalpha=0 minother=0 mindiff=0 maxrepeats=8 minlen=0 histexpire=0 histsize=0 pwdchecks= dictionlist= fsize=-1 cpu=-1 data=262144 stack=65536 core=2097151 rss=65536 nofiles=2000 roles=

 

I set the users password to abc123, using the chpasswd utility. I also remove the ADMCHG flag so that the user is not prompted to change their password on their first logon attempt.

 

root@nim# dsh 'echo cg:abc123 | chpasswd -c'

 

I confirm that I can logon with the new user with the specified password, on one of the AIX LPARs.

 

root@nim# ssh cg@aixlpar1

cg@aixlpar1’s password:

Last login: Thu Mar  1 20:05:01 CST 2012 on /dev/pts/1 from aix71

$ id

uid=204(cg) gid=1(staff)

 

Another nice feature of dsh is the dshbak utility. This utility presents formatted output from the dsh command. For example:

 

root@nim 520 [/.ssh]# dsh errpt | dshbak

HOST: aixlpar1.cg.com.au

------------------------------------

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION

AA8AB241   1116110811 T O OPERATOR       OPERATOR NOTIFICATION

A6DF45AA   1104135011 I O RMCdaemon      The daemon is started.

2BFA76F6   1104134111 T S SYSPROC        SYSTEM SHUTDOWN BY USER

9DBCFDEE   1104134111 T O errdemon       ERROR LOGGING TURNED ON

 

HOST: aixlpar2.cg.com.au

-------------------------------

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION

DE9A52D1   1111012611 I S rmt10          AAA1

4865FA9B   1111012211 P H rmt10          TAPE OPERATION ERROR

DE9A52D1   1110233511 I S rmt0           AAA1

4865FA9B   1110225511 P H rmt0           TAPE OPERATION ERROR

DE9A52D1   1109180311 I S rmt0           AAA1

4865FA9B   1109180011 P H rmt0           TAPE OPERATION ERROR

DE9A52D1   1108180411 I S rmt2           AAA1

4865FA9B   1108180211 P H rmt2           TAPE OPERATION ERROR

DE9A52D1   1108165711 I S rmt6           AAA1

4865FA9B   1108165111 P H rmt6           TAPE OPERATION ERROR

A2205861   1102085311 P S SYSPROC        Excessive interrupt disablement time

F7FA22C9   1031134111 I O SYSJ2          UNABLE TO ALLOCATE SPACE IN FILE SYSTEM

DE9A52D1   1030163411 I S rmt0           AAA1

4865FA9B   1030163411 P H rmt0           TAPE OPERATION ERROR

....etc....

 

WARNING: Please be VERY CAREFUL when using the dsh command. Issuing the wrong command can cause damage to all your AIX LPARS!

 

The dsm.dsh package contains the following utilities:

 

# lslpp -f dsm.dsh | grep /usr/bin

                        /usr/bin/dcp -> /opt/ibm/sysmgt/dsm/bin/dcp

                        /usr/bin/dsh -> /opt/ibm/sysmgt/dsm/bin/dsh

                        /usr/bin/dping -> /opt/ibm/sysmgt/dsm/bin/dping

                        /usr/bin/dshbak -> /opt/ibm/sysmgt/dsm/bin/dshbak

 

If you are a fan of the dping command, you are going to be disappointed. Although the command is currently included in the dsm.dsh fileset, it probably won’t be for much longer.

 

The command works, “sort of”:

 

root@nim# dping aixlpar1

aixlpar1: ping (alive)

 

But if you run ‘dping –a’:

 

root@nim# dping -a

dping: 2651-095 CSM license has expired or has not been accepted. Run csmconfig -L if you have installed a new release.

 

According to the developers, dping is no longer supported and will eventually be removed from the DSM package. The response from the  developers was as follows:

"The reason "dping -a" is failing with the license check is because the command is calling “/usr/bin/runact-api –c IBM.DmsCtrl::::isLicenseValid" and the license is not set. So the command fails.  Since CSM is not supported anymore and went end of life. "

“... please consider the dping command as being "deprecated" code pending removal from the dsm.dsh package.”

When I asked why the command was listed in the AIX 7.1 online documentation if it was no longer available, I was informed: “We are  in the process of working with component owner regarding the DOCs and updating them.”. At this stage I’ve not been able to find an alternative command (in AIX). If I find one, I’ll update this post.

 

If you are planning on migrating to AIX 7.1 please be aware that CSM is no longer supported or available with AIX 7.1. CSM is now ‘end of life’.

 

 

http://www.ibmsystemsmag.com/aix/tipstechniques/systemsmanagement/DSH-Simplifies-the-Task-of-Managing-Multiple-Syste/

 

http://www.redbooks.ibm.com/abstracts/sg247910.html?Open

 

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.cmds/doc/aixcmds1/chpasswd.htm

 

So you’re configuring a new PowerHA 6.1 cluster and you decide to check if the clinfoES subsystem is running. You run the following command:

 

# lssrc -a | grep cluster

clstrmgrES       cluster          2883628      active

 

Hmmm, clinfoES is not listed. To ensure that is clinfoES is shown in lssrc output, in both inactive and active states, run the following command on all nodes in the cluster:

 

# chssys -s clinfoES –d

 

And now clinfoES is shown:

 

#lssrc -a | grep cluster

clstrmgrES       cluster          2883628      active

clinfoES         cluster                       inoperative

 

There was also a time when you had to do the same for clstrmgrES, however this subsystem is now always active (very rarely/never inactive) on a HA node e.g.

 

# chssys -s clstrmgrES -d

 

The chssys command can make changes to a subsystems definition on AIX:

 

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.cmds/doc/aixcmds1/chssys.htm

 

The –d flag to the chssys command specifies that an inactive subsystem is displayed in lssrc output.

 

I found a very old reference to something similar in this old HACMP/ES Redbook, but only the clinfoES recommendation still applies:

 

For a better understanding and to avoid problem determination, it is recommended to make all relevant HACMP subsystems visible with the lssrc command, independent of whether they are active or not.

 

# chssys -s clinfoES -d # modify subsystem definition

# chssys -s cllockdES -d

# chssys -s clsmuxpdES -d

# chssys -s clvmd –d

 

http://ps-2.kev009.com/rs6000/redbook-cd/SG245526.pdf

 

 

Prior to AIX 5.3 TL7 and AIX 6.1, there was an 8 character limit on AIX user passwords. If you need passwords of greater than 8 characters then you must enable one of the supplied Loadable Password Algorithms (LPAs). The following table lists the available algorithms and the limitations of each:

 

 

For example, to enable the MD5 algorithm I can modify /etc/security/login.cfg file with the chsec command as follows:

 

# chsec -f /etc/security/login.cfg -s usw -a pwd_algorithm=smd5

 

# tail -2 /etc/security/login.cfg

        pwd_algorithm = smd5

 

This algorithm (smd5) will allow a password limit of 255 characters. Each of the available algorithms is listed in the /etc/security/pwdalg.cfg file.

 

# head -50  /etc/security/pwdalg.cfg

*******************************************************************************

*

* Loadable Password Algorithm (LPA)

*

* The format of a LPA stanza is:

*

* lpa:

*       lpa_module = /any/program

*       lpa_options = options_list

*

* lpa           Corresponds to a password hashing algorithm that can be

*               specified in the "usw" stanza of /etc/security/login.cfg

*               as the system-wide password hashing algorithm.

*

* lpa_module    Specifies the path of the password hashing load module.

*               In the above example, /any/program is the path to the load

*               module that is used to do the password hashing in 32 bit

*               mode. The load module that is used to run in 64 bit mode

*               is /any/program_64.

*

* lpa_options   Is an optional attribute that provides a means of

*               communicating run-time configuration options to the load

*               module. The value is a comma separated list of items.

*               Please refer to the documentation for the load module for

*               appropriate values.

*

*******************************************************************************

*

* /usr/lib/security/smd5 is a password hashing load module using

* the MD5 algorithm.

*

* It supports password length up to 255 characters.

*

* To generate smd5 password hash compatible to standard salted MD5,

* add the following option line for smd5 stanza.

*       lpa_options = std_hash=true

*

* Note : password hash generated with this option won't be compatible with

* hash generated without this option.

*

*******************************************************************************

smd5:

        lpa_module = /usr/lib/security/smd5

 

*******************************************************************************

*

* /usr/lib/security/ssha is a password hashing load module using SHA and

* SHA2 algorithms. It supports password length up to 255 characters.

*

* This LPA accepts three options. The options are separated by commas.

...etc...

 

 

Once you’ve enabled the LPA of your choice, and you set/change a users’ password, you’ll notice that the /etc/security/passwd stanza for that user will look different when compared to the stanzas of users that have not had their password set/changed using the new LPA:

 

fred:

        password = E7nOaTrrz9Q16

        lastupdate = 1330986703

        flags = ADMCHG

 

joe:

        password = {smd5}z9JrHDJB$Oq/cZXr0jUyAWvfFyjt161

        lastupdate = 1330987903

        flags = ADMCHG

 

 

 

In the example above, user joe’s password has been set using the smd5 algorithm.

 

For those of you who run PowerHA (HACMP) and are thinking about using one of the LPAs with the clpasswd utility, you may want to review this APAR first:

 

IZ40362: HACMP PASSWD UTILITY FAILS WHEN LONG PASSWD ALGORITHM IS ENABLED

https://www-304.ibm.com/support/docview.wss?uid=isg1IZ40362

 

The APAR states “HACMP cluster-wide C-SPOC password administration does not support use of the feature allowing passwords longer than 8 characters which became available with the Loadable Password Algorithm as part of AIX 53 TL 7.”

 

The last time I tested this with PowerHA, the problem was that the password entry in /etc/security/passwd was corrupted/truncated when a users password was changed using the clpasswd utility.

 

For example, if the passwd utility is linked to clpasswd and I changed a users password, the password field appeared to be corrupted/truncated and the user could not log in successfully:

 

# ls -l /usr/bin/passwd

lrwxrwxrwx   1 root     system           39 Dec 22 11:30 /usr/bin/passwd -> /usr/es/sbin/cluster/utilities/clpasswd

 

# passwd tom

Changing password for "tom"

tom's New password:

Enter the new password again:

 

# grep -p tom passwd

tom:

        password = {smd5}FU2MWa.N.

        lastupdate = 1229910298

        flags =

 

 

Login no good!

tom@aix16's password:

Access denied

 

But if we use the original passwd command (stored in /usr/es/sbin/cluster/etc/clpasswd), the users password is changed successfully and the user can log in OK.

 

root@aix16 /usr/es/sbin/cluster/etc/clpasswd # ./usr_bin_passwd.orig tom

Changing password for "tom"

tom's New password:

Enter the new password again:

 

root@aix16 /usr/es/sbin/cluster/etc/clpasswd # grep -p tom /etc/security/passwd

tom:

        password = {smd5}PcBK/5QW$RE09Q0xz4TGQFp98KjxSs.

        lastupdate = 1229910818

        flags =                   ADMCHG

 

Login OK!

$ id

uid=10003(tom) gid=1(staff)

 

usr_bin_passwd.orig  password = {smd5}HcMHTTu3$BWkU8JVCWgT8szAN8TGCF1

clpasswd             password = {smd5}6z61fiIn/070

 

I’ve not tried this again recently but I am curious if the same behaviour can be expected on a PowerHA system today. When I first encountered this problem (in 2008) I opened a PMR for the issue. In that call I was told that the “clpasswd utility is corrupting the encrypted password when distributing to the nodes, so that a login fails”. I’ll configure a HA cluster soon and try it again with PowerHA 6.1 and AIX 6.1 and report back with the results.

 

UPDATE: I built a HA 6.1 cluster (on AIX 6.1) this afternoon in my lab and tested this successfully. Based on the tests I’ve performed so far, it appears that this limitation no longer exists. Thanks to hafeedbk@us.ibm.com for the help on this one.

 

The following IBM tech note has more information on the available Loadable Password Algorithms and support for longer than 8 character passwords on AIX:

 

AIX support for passwords greater than 8 characters

http://www-01.ibm.com/support/docview.wss?rs=111&context=SWG10&q1=pwd_algorithm&uid=isg3T1010741&loc=en_US&cs=utf-8&lang=en

 

 

It’s no secret that I like NIM. I’ve written about NIM on several occasions, in Redbooks, technical articles and on my blog. It’s a great tool for installing and maintaining AIX. Occasionally though, it can throw you an unusual error or two when things go wrong. For the uninitiated resolving these errors can be challenging.

 

Here’s a couple of NIM related questions I received recently. Both are troubleshooting queries.

 

“I'm getting the following error when trying to create a NIM lpp_source for AIX 6.1”:

 

                               Define a Resource

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                                                        [Entry Fields]

* Resource Name                               [6100-06_lpp]

* Resource Type                                lpp_source

* Server of Resource                          [master]                +

* Location of Resource                        [/export/eznim]          /

  NFS Client Security Method                  []                      +

  NFS Version Access                          []                      +

  Architecture of Resource                    []                      +

  Source of Install Images                    [/dev/cd0]              +/

  Names of Option Packages                    []

  Show Progress                               [yes]                   +

  Comments                                    []

 

...

Preparing to copy install images (this will take several minutes)...

 

0042-001 nim: processing error encountered on "master":

   0042-081 m_mk_lpp_source: a resource already exists on "master" at

location "/export/eznim/bid_ow"; due to NFS export restrictions, the

        new location "/export/eznim" may not be used

...

 

I could see what could be causing the issue but I first asked to see the contents of the /etc/exports file. Just to make sure that /export/eznim was not explicitly exported. It wasn’t.

 

# grep eznim /etc/exports

/export/eznim/vios/mksysb_image -ro,root=vio1,access=vio1

/export/eznim/bid_ow -ro,root=nim1,access=nim1

 

I suggested they re-try the operation again but this time specify the full path to the lpp_source directory, like so:

 

                               Define a Resource

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                                                        [Entry Fields]

* Resource Name                               [6100-06_lpp]

* Resource Type                                lpp_source

* Server of Resource                          [master]                +

* Location of Resource                        [/export/eznim/6100-06_lpp]          /

  NFS Client Security Method                  []                      +

  NFS Version Access                          []                      +

  Architecture of Resource                    []                      +

  Source of Install Images                    [/dev/cd0]              +/

  Names of Option Packages                    []

  Show Progress                               [yes]                   +

  Comments                                    []

 

This worked fine. And the lpp_source was created successfully.

 

#  lsnim -l 6100-06_lpp

6100-06_lpp:

   class       = resources

   type        = lpp_source

   locked      = 26738726

   Rstate      = unavailable for use

   prev_state  =

   location    = /export/eznim/6100-06_lpp

   alloc_count = 0

   server      = master

 

The second issue related to down-level filesets after an AIX 6.1 migration with nimadm.  There was also an issue with disk space that generated an error during NIM operations.

 

“Hi Chris,

 

Thanks for replying back to my post. I am a newbie to AIX/pSeries world.

I was following your document on upgrading AIX 5.3 to 6.1 using nimadm, it seemed to have worked partially for me, as I got to AIX 6100-00-00. The mksysb and the SPOT I created were for AIX 6100-06-05.

 

Before I re-attempt the upgrade, is there anything I need to be clean up on AIX 5.3 side that could have perhaps caused issues with the upgrade.

 

Thanks”

 

I recommended that the following command be run on the NIM client, to determine which filesets were down-level after the migration.

 

# oslevel -s

# instfix -i | grep AIX

# instfix -i | grep SP

 

If the output contained messages like "Not all filesets for 6100-06_AIX_ML were found", or something similar, then they should run the following command against each line (there could be more than one).

 

# instfix -icqk 6100-06_AIX_ML | grep ":-:"

 

This will tell you which filesets were not updated.

 

The output from ‘oslevel –s’and ‘instfix –i | grep AIX’ looked OK.

 

# oslevel -s

6100-06-04-1112

 

# instfix -i | grep AIX

    All filesets for 6100-00_AIX_ML were found.

    All filesets for 6100-01_AIX_ML were found.

    All filesets for 6100-02_AIX_ML were found.

    All filesets for 6100-03_AIX_ML were found.

    All filesets for 6100-04_AIX_ML were found.

    All filesets for 6100-05_AIX_ML were found.

    All filesets for 6100-06_AIX_ML were found.

 

But the output from ‘instfix –i  | grep SP’ indicated an issue with 61-06-051115_SP.

 

# instfix -i | grep SP

    All filesets for 61-00-010748_SP were found.

    All filesets for 61-00-020750_SP were found.

    All filesets for 61-00-030808_SP were found.

    All filesets for 61-00-040815_SP were found.

    All filesets for 61-01-010823_SP were found.

    All filesets for 61-00-050822_SP were found.

    All filesets for 61-00-060834_SP were found.

    All filesets for 61-01-020834_SP were found.

    All filesets for 61-00-070846_SP were found.

    All filesets for 61-01-030846_SP were found.

    All filesets for 61-02-010847_SP were found.

    All filesets for 61-02-020849_SP were found.

    All filesets for 61-00-080909_SP were found.

    All filesets for 61-01-040909_SP were found.

    All filesets for 61-02-030909_SP were found.

    All filesets for 61-03-010921_SP were found.

    All filesets for 61-00-090920_SP were found.

    All filesets for 61-01-050920_SP were found.

    All filesets for 61-02-040920_SP were found.

    All filesets for 61-00-100939_SP were found.

    All filesets for 61-01-060939_SP were found.

    All filesets for 61-02-050939_SP were found.

    All filesets for 61-03-020939_SP were found.

    All filesets for 61-04-010944_SP were found.

    All filesets for 61-04-021007_SP were found.

    All filesets for 61-04-031009_SP were found.

    All filesets for 61-00-110943_SP were found.

    All filesets for 61-01-081014_SP were found.

    All filesets for 61-02-060943_SP were found.

    All filesets for 61-03-030943_SP were found.

    All filesets for 61-03-041014_SP were found.

    All filesets for 61-04-041014_SP were found.

    All filesets for 61-05-011016_SP were found.

    All filesets for 61-01-091015_SP were found.

    All filesets for 61-02-081015_SP were found.

    All filesets for 61-03-051015_SP were found.

    All filesets for 61-04-051015_SP were found.

    All filesets for 61-01-070943_SP were found.

    All filesets for 61-02-071014_SP were found.

    All filesets for 61-02-091034_SP were found.

    All filesets for 61-03-061034_SP were found.

    All filesets for 61-04-061034_SP were found.

    All filesets for 61-05-021034_SP were found.

    All filesets for 61-06-011043_SP were found.

    All filesets for 61-02-101036_SP were found.

    All filesets for 61-03-071036_SP were found.

    All filesets for 61-04-071036_SP were found.

    All filesets for 61-05-031036_SP were found.

    All filesets for 61-06-021044_SP were found.

    All filesets for 61-06-031048_SP were found.

    All filesets for 61-05-041048_SP were found.

    All filesets for 61-04-081048_SP were found.

    All filesets for 61-03-081048_SP were found.

    All filesets for 61-03-091112_SP were found.

    All filesets for 61-04-091112_SP were found.

    All filesets for 61-05-051112_SP were found.

    All filesets for 61-06-041112_SP were found.

    Not all filesets for 61-06-051115_SP were found.

 

The culprit was the csm.hc_utils fileset from TL6 SP5. It was still at level 1.7.1.0 instead of the expected 1.7.1.1.

 

# instfix -ciqk "61-06-051115_SP" |grep ":-:"

61-06-051115_SP:csm.hc_utils:1.7.1.1:1.7.1.0:-:AIX 6100-06-05 Service Pack

 

I recommended that the csm.hc_utils 1.7.1.1 fileset be added to the AIX 6.1 lpp_source. The migration was attempted again and the client migrated to AIX 6.1 TLP6 SP5 successfully.  I also suggested they determine if this fileset was really used and/or needed (CSM is no longer supported/available starting with AIX 7.1 anyway).

 

And just when I thought all his problems were resolved, he replied with: “however upon trying to install these from the NIM server, I get errors on NIM”.

 

*********************************************************

errors:

0042-001 nim: processing error encountered on "master":                |

 |      0042-001 m_showres: processing error encountered on "master":       |

 |      0042-185 c_ch_nfsexp: Failed to link or copy /etc/.tmp.nim.exports. |

 |   12451894 to /etc/exports                                               |

 |           Check permissions and filesystem space

*********************************************************

 

Fortunately a quick look at the output from the df command highlighted the problem immediately.

He was out of space in the root file system. The very place that /etc/exports, and the NIM database (/etc/objrepos), reside.

 

# df -g

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4           2.50      0.00  100%     3388     1% /

/dev/hd2           4.50      1.74   62%    42150     4% /usr

/dev/hd9var        1.00      0.52   48%     3262     2% /var

/dev/hd3           4.00      3.84    5%       74     1% /tmp

/dev/hd1           0.50      0.37   27%      479     1% /home

/proc                 -         -    -         -     -  /proc

/dev/hd10opt       0.75      0.21   73%     5698     3% /opt

/dev/livedump      0.25      0.24    4%       17     1% /var/adm/ras/livedump

/dev/exportlv    136.25    124.03    9%    20128     1% /export

 

After cleaning up unwanted files in / and making sure there was plenty of free space, the error no longer appeared in future NIM operations.

 

 

When attempting to change the max_transfer attribute on a VSCSI hdisk, the following error was displayed:

 

# chdev -l hdisk0 -a max_transfer=0x100000 -P

Method error (/etc/methods/chgdisk):

        0514-018 The values specified for the following attributes

                 are not valid:

     max_transfer     Maximum TRANSFER Size

 

The desired value, 0x100000, was an allowed value for this type of device (as shown in the lsattr output below):

 

# lsattr -El hdisk0

PCM             PCM/friend/vscsi                 Path Control Module        False

algorithm       fail_over                        Algorithm                  True

hcheck_cmd      test_unit_rdy                    Health Check Command       True

hcheck_interval 60                               Health Check Interval      True

hcheck_mode     nonactive                        Health Check Mode          True

max_transfer    0x40000                          Maximum TRANSFER Size      True

pvid            00f6a83aeb1948950000000000000000 Physical volume identifier False

queue_depth     256                              Queue DEPTH                True

reserve_policy  no_reserve                       Reserve Policy             True

 

# lsattr -Rl hdisk0 -a max_transfer

0x20000

0x40000

0x80000

0x100000

0x200000

0x400000

0x800000

0x1000000

 

After some discussion with IBM support, it was discovered that the virtual optical device, assigned to this LPAR, was causing the problem. They advised that we should perform the following steps:

 

1. Remove the cd0 device from the LPAR i.e.

   # rmdev –dl cd0

 

2. Remove the vtopt0 device assigned to the LPAR on the VIOS i.e

   $ rmvdev –vtd vtopt0

 

3. Shutdown the LPAR.

 

4. Reboot the VIOS.

 

5. Restart the LPAR.

 

6. Change the max_transfer value for the hdisk on the LPAR.

 

Sure enough, after following these steps, we were able to change max_transfer to the desired value!

 

# chdev -l hdisk0 -a max_transfer=0x100000

hdisk0 changed

 

# lsattr -El hdisk0

PCM             PCM/friend/vscsi                 Path Control Module        False

algorithm       fail_over                        Algorithm                  True

hcheck_cmd      test_unit_rdy                    Health Check Command       True

hcheck_interval 0                                Health Check Interval      True

hcheck_mode     nonactive                        Health Check Mode          True

max_transfer    0x100000                         Maximum TRANSFER Size      True

pvid            00f6a83aeedf5ffb0000000000000000 Physical volume identifier False

queue_depth     3                                Queue DEPTH                True

reserve_policy  no_reserve                       Reserve Policy             True

 

The following explanation was given by IBM support:

 

“The virtual optical device seems to use max_transfer size as 0x40000 (256 KB) internally, which is not allowed to be modified. Virtual optical CD's max_transfer seems to limit vhost adapter's max_transfer size. So if you want to increase max_transfer size for virtual SCSI disks, you should separate them from virtual optical SCSI CD by configuring those two groups under different vhost adapters”

When you upgrade PowerHA (formerly HACMP) don’t forget to check/update your TSM client as well!

 

We successfully upgraded from PowerHA 5.4 to PowerHA 6.1 SP6 first.

 

# halevel -s

6.1.0 SP6

 

We found that our DB2 backups to TSM were no longer working.

 

$ db2adutl query

 

Error: Initialize environment failed with TSM return code 4602

 

In the TSM error log file we also saw the following message repeated over and over:

 

# tail -1 dsmerror_db2_ha.log

10/17/11   12:42:57 ANS1478E CLUSTERNODE is set to YES but the cluster software is not available on this system.

 

Obviously cluster services were most certainly installed and running! So we suspected the TSM client was in need of an update or two! This was no surprise to us.

 

The current installed TSM client version was 5.5.2.

 

# lslpp -L tivoli*

  Fileset                      Level  State  Type  Description (Uninstaller)

  ----------------------------------------------------------------------------

  tivoli.tivguid             1.3.2.1    C     F    IBM Tivoli GUID on AIX

  tivoli.tsm.client.api.32bit

                             5.5.2.0    C     F    TSM Client - Application

                                                   Programming Interface

  tivoli.tsm.client.api.64bit

                             5.5.2.0    C     F    TSM Client - 64bit Application

                                                   Programming Interface

  tivoli.tsm.client.ba.32bit.base

                             5.5.2.0    C     F    TSM Client - Backup/Archive

                                                   Base Files

  tivoli.tsm.client.ba.32bit.common

                             5.5.2.0    C     F    TSM Client - Backup/Archive

                                                   Common Files

  tivoli.tsm.client.ba.32bit.web

                             5.5.2.0    C     F    TSM Client - Backup/Archive

                                                   WEB Client

We upgraded the TSM client to version 6.2.

 

# lslpp -l tiv*

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  tivoli.tivguid             1.3.2.1  COMMITTED  IBM Tivoli GUID on AIX

  tivoli.tsm.client.api.32bit

                             6.2.2.3  COMMITTED  TSM Client - Application

                                                 Programming Interface

  tivoli.tsm.client.api.64bit

                             6.2.2.3  COMMITTED  TSM Client - 64bit Application

                                                 Programming Interface

  tivoli.tsm.client.ba.64bit.base

                             6.2.2.3  COMMITTED  TSM Client 64 - Backup/Archive

                                                 Base Files

  tivoli.tsm.client.ba.64bit.common

                             6.2.2.3  COMMITTED  TSM Client 64 - Backup/Archive

                                                 Common Files

  tivoli.tsm.client.ba.64bit.image

                             6.2.2.3  COMMITTED  TSM Client 64 - IMAGE Backup

                                                 Client

  tivoli.tsm.client.ba.64bit.web

                             6.2.2.3  COMMITTED  TSM Client 64 - Backup/Archive

                                                 Java GUI & WEB Client

 

Path: /etc/objrepos

  tivoli.tivguid             1.3.2.1  COMMITTED  IBM Tivoli GUID on AIX

 

 

Now our DB2 connection to the TSM server was working again.

 

$ db2adutl query

 

Query for database C01DCVBD1

 

 

Retrieving FULL DATABASE BACKUP information.

    1 Time: 20111024221807  Oldest log: S0093112.LOG  DB Partition Number: 0    Sessions: 1

    2 Time: 20111023215632  Oldest log: S0093111.LOG  DB Partition Number: 0    Sessions: 1

    3 Time: 20111022220109  Oldest log: S0093109.LOG  DB Partition Number: 0    Sessions: 1

    4 Time: 20111021221842  Oldest log: S0093108.LOG  DB Partition Number: 0    Sessions: 1

    5 Time: 20111020223428  Oldest log: S0093106.LOG  DB Partition Number: 0    Sessions: 1

    6 Time: 20111019221958  Oldest log: S0093105.LOG  DB Partition Number: 0    Sessions: 1

    7 Time: 20111018221802  Oldest log: S0093103.LOG  DB Partition Number: 0    Sessions: 1

 

 

Retrieving INCREMENTAL DATABASE BACKUP information.

  No INCREMENTAL DATABASE BACKUP images found for C01DCVBD1

 

 

Retrieving DELTA DATABASE BACKUP information.

  No DELTA DATABASE BACKUP images found for C01DCVBD1

 

 

Retrieving TABLESPACE BACKUP information.

  No TABLESPACE BACKUP images found for C01DCVBD1

 

 

Retrieving INCREMENTAL TABLESPACE BACKUP information.

  No INCREMENTAL TABLESPACE BACKUP images found for C01DCVBD1

 

 

Retrieving DELTA TABLESPACE BACKUP information.

  No DELTA TABLESPACE BACKUP images found for C01DCVBD1

 

 

Retrieving LOAD COPY information.

  No LOAD COPY images found for C01DCVBD1

 

 

Retrieving LOG ARCHIVE information.

   Log file: S0093099.LOG, Chain Num: 9, DB Partition Number: 0, Taken at: 2011-10-15-04.03.52

 

We also discovered that we needed to comment out the following entries in the TSM client dsm.sys file (note the MAILPROG variable appeared twice in our dsm.sys file):

 

*MAILPROG                /usr/bin/mailx  root

*CLUSTERNODE             YES

 

The latest TSM client no longer supports the MAILPROG option, as we discovered when we ran the TSM client from the command line:

 

# dsmc q sess

ANS1677W The MAILPROG option is ignored because it is no longer supported.

IBM Tivoli Storage Manager

Command Line Backup/Archive Client Interface

  Client Version 5, Release 5, Level 2.0

  Client date/time: 17-10-2011 12:01:34

(c) Copyright by IBM Corporation and other(s) 1990, 2009. All Rights Reserved.

 

Node Name: AIX1

Session established with server TSM1: AIX-RS/6000

  Server Version 5, Release 5, Level 5.0

  Server date/time: 17-10-2011 12:01:38  Last access: 17-10-2011 09:44:26

 

TSM Server Connection Information

 

Server Name.............: TSM1

Server Type.............: AIX-RS/6000

Archive Retain Protect..: "No"

Server Version..........: Ver. 5, Rel. 5, Lev. 5.0

Last Access Date........: 17-10-2011 09:44:26

Delete Backup Files.....: "No"

Delete Archive Files....: "Yes"

 

Node Name...............: AIX1

User Name...............: root

 

The CLUSTERNODE option was previously required when using TSM in a PowerHA (HACMP) environment. But not anymore:

 

Question

Do tsm client 6.1 and 6.2 support PowerHA cluster system?

 

Answer

TSM client supports all versions of PowerHA. In the past, TSM client used to publish information in the client books about the HACMP versions that it supported. The reason for this was, at that time TSM client was invoking the HACMP API in order to support it. From TSM version 6.1, TSM client does not use the HACMP API. As long as TSM client is configured with HACMP as documented in the client books and the HACMP documentation, TSM will be able to support any version of PowerHA (formerly known as HACMP).

 

https://www-304.ibm.com/support/docview.wss?uid=swg21468887

 

IBM Tivoli Storage Manager Requirements for the IBM® AIX® Backup-Archive and API Client

http://www-01.ibm.com/support/docview.wss?rs=663&tc=SSGSG7&uid=swg21052226

 

After reading about the latest AIX updates here: http://t.co/bkIpnXkS I decided to download and install the latest TL & SP for AIX 6.1 and 7.1 and take a peek at some of the latest features.

 

Here’s what I found so far.

 

There appears to be some new integration between NIM and the VIOS. The nim command now has an updateios option e.g. nim –o updateios.

So you can update your VIO servers from NIM now. This is nice.

 

On my lab NIM master I checked the nim man page and found the following new information:

 

NIM [/] # oslevel -s

6100-07-02-1150

 

NIM [/] # man nim

...

updateios

Performs software customization and maintenance on a virtual input output server (VIOS) management server that is of the vios or ivm type.

 

updateios

 

1    To install fixes or to update VIOS with the vioserver1 NIM object name to the latest maintenance level, type:

 

nim -o updateios -a lpp_source=lpp_source1 -a preview=no vioserver1

 

The updates are stored in lpp_source and lpp_source1 files. Note: The updateios operation runs a preview during installation. Running the updateios operation from NIM runs a preview unless the preview flag is set to no. During the installation, you must run a preview when using the updateios operation with updatios_flags=-install. With the preview, you can check if the installation is running accurately before proceeding with the VIOS update.

 

2    To reject fixes for a VIOS with the vioserver1 NIM object name, type:

 

nim -o updateios -a updateios_flags=-reject vioserver1

 

3    To clean up partially installed updates for a VIOS with the vioserver1 NIM object name, type:

 

nim -o updateios -a updateios_flags=-cleanup vioserver1

 

4    To commit updates for a VIOS with the vioserver1 NIM object name, type:

 

nim -o updateios -a updateios_flags=-commit vioserver1

 

5    To remove a specific update such as update1 for a VIOS with the vioserver1 NIM object name, type:

 

nim -o updateios -a updateios_flags=-remove-a filesets="update1" vioserver1

 

6    To remove updates for a VIOS with the vioserver1 NIM object name by using an installp_bundle bundle1, where bundle1 contains the updates to be removed, type:

 

nim -o updateios -a updateios_flags=remove -a installp_bundle=bundle1 vioserver1

 

===

 

There’s also mention of a new resource type, specifically for VIOS mksysbs. This resource type is called ios_mksysb:

...

ios_mksysb

Represents a backup image taken from a VIOS management server that is of the vios or ivm type.

 

26 To define a ios_mksysb resource such as ios_mksysb1, and create the ios_mksysb image of the vios client as vios1, during the resource definition where the image is located in /export/nim/ios_mksysb on the master, type:

 

nim -o define -t ios_mksysb -a server=master \

-a location=/export/nim/ios_mksysb -a source=vios1 \

-a mk_image=yes ios_mksysb1

 

This is all starting to come together now, since the introduction of the new “management” object class, vios, with AIX 6.1 TL3.

 

# smit nim

 

Manage Control Objects

Define a Management Object

 

 

                                                                 +--------------------------------------------------------------------------+

                                                                 |                          Management Object Type                          |

                                                                 |                                                                          |

                                                                 | Move cursor to desired item and press Enter. Use arrow keys to scroll.   |

                                                                 |                                                                          |

                                                                 |   hmc  = specifies the management object type for hmc hardware           |

                                                                 |   cec  = specifies the management object type for cec hardware           |

                                                                 |   vios = specifies the virtual hardware server object                    |

                                                                 |   ivm  = specifies the integrated virtual manager object                 |

                                                                 |   bcmm = specifies the management object type for management module hard |

 

 

Next I thought I’d take a look at the TCP Fast Loopback option. This new option should help to reduce TCP/IP (CPU) overhead when two (TCP) communication end points reside in the same LPAR. This could be useful where you have an LPAR running a database and application in the same LPAR e.g. SAP and Oracle in the same LPAR. It can also be used when two or more WPARs, in the same LPAR need to communicate with each other over TCP/IP.

 

I turned on this new feature on my AIX 7.1 LPAR.

 

AIX7[/] # oslevel -s

7100-01-02-1150

 

 

AIX7[/] # netstat -p tcp | grep fastpath

        0 fastpath loopback connection

        0 fastpath loopback sent packet (0 byte)

        0 fastpath loopback received packet (0 byte)

 

AIX7[/] # no -p -o tcp_fastlo=1

Setting tcp_fastlo to 1

Setting tcp_fastlo to 1 in nextboot file

Change to tunable tcp_fastlo, will only be effective for future connections

 

AIX7[/] # no -p -o tcp_fastlo_crosswpar=1

Setting tcp_fastlo to 1

Setting tcp_fastlo to 1 in nextboot file

Change to tunable tcp_fastlo, will only be effective for future connections

 

AIX7[/] # no -a | grep tcp_fast

               tcp_fastlo = 1

     tcp_fastlo_crosswpar = 1

 

Initially I did not see any traffic via the fastpath.

 

AIX7[/] # netstat -s -p tcp | grep fastpath

        0 fastpath loopback connection

        0 fastpath loopback sent packet (0 byte)

        0 fastpath loopback received packet (0 byte)

 

 

So I created two WPARs in the same LPAR and started transferring files between them via FTP.

 

 

AIX7[/] # lswpar -N

Name   Interface  Address(6)      Mask/Prefix    Broadcast

----------------------------------------------------------------

wpar1  en0        172.29.152.235  255.255.192.0  172.29.191.255

wpar2  en0        172.29.152.236  255.255.192.0  172.29.191.255

 

And sure enough I started to see some traffic.

 

AIX7[/] # netstat -s -p tcp | grep fastpath

        7072 fastpath loopback connection

        171102 fastpath loopback sent packet (5374011130 byte)

        171102 fastpath loopback received packet (5374011130 byte)

 

Next stop: JFS2 Remount Support. Apparently you can now change certain mount options dynamically. These particular options can influence (among other things) filesystem caching.

 

NIM [/] # oslevel -s

6100-07-02-1150

 

So I start with a standard JFS2 filesystem, without any additional mount options.

 

NIM [/] # mount | grep cg

         /dev/cglv        /cg              jfs2   Jan 18 21:14 rw,log=/dev/hd8

 

Then I dynamically remounted it with the rbr option. This option will prevent user data pages from being cached after a file is read from this filesystem.

 

NIM [/] # mount -o remount,rbr /cg

NIM [/] # mount | grep cg

         /dev/cglv        /cg              jfs2   Jan 18 21:14 rw,rbr,log=/dev/hd8

 

Still can’t dynamically mount a filesystem with CIO however. But that’s OK.

 

NIM [/] # mount -o remount,cio /cg

mount: cio is not valid with the remount option.

 

According to the presentation, there are several options that can now be changed dynamically e.g. atime,rbr,rbw,suid,dev, etc. Take a look at the presentation if you are interested in this new functionality.

 

By the way, just to make sure, I tried changing the same mount option, dynamically, on an AIX 6.1 TL6 system, and it failed as expected. I’d have to umount and mount the filesystem to do this on TL6 (or lower).

 

# oslevel -s

6100-06-04-1112

 

# mount -o remount,rbr /cg

mount: remount,rbr,log=/dev/loglv00 is not valid with the remount option.

 

 

OK, let’s look at the new LVM Infinite Retry Capability. Designed to improve system availability, by allowing LVM to recover from transient failures of storage devices. Sounds interesting!

 

AIX7[/] # oslevel -s

7100-01-02-1150

 

The man page for mkvg states the following:

 

       -O y / n

            Enables the infinite retry option of the logical volume.

              n

                   The infinite retry option of the logical volume is not enabled. The failing I/O of the logical volume is not retried. This is the default value.

              y

                   The infinite retry option of the logical volume is enabled. The failed I/O request is retried until it is successful.

 

I think “logical volume” should be “volume group”. But anyway, I get the idea.

 

So let’s create a new VG with infinite retry enabled.

 

AIX7[/] # mkvg -O y -S -y cgvg hdisk6

cgvg

AIX7[/] # lsvg cgvg

VOLUME GROUP:       cgvg                     VG IDENTIFIER:  00f6048800004c0000000134f4236851

VG STATE:           active                   PP SIZE:        128 megabyte(s)

VG PERMISSION:      read/write               TOTAL PPs:      1599 (204672 megabytes)

MAX LVs:            256                      FREE PPs:       1599 (204672 megabytes)

LVs:                0                        USED PPs:       0 (0 megabytes)

OPEN LVs:           0                        QUORUM:         2 (Enabled)

TOTAL PVs:          1                        VG DESCRIPTORS: 2

STALE PVs:          0                        STALE PPs:      0

ACTIVE PVs:         1                        AUTO ON:        yes

MAX PPs per VG:     32768                    MAX PVs:        1024

LTG size (Dynamic): 128 kilobyte(s)          AUTO SYNC:      no

HOT SPARE:          no                       BB POLICY:      relocatable

MIRROR POOL STRICT: off

PV RESTRICTION:     none                     INFINITE RETRY: yes

AIX7[/] #

 

Now, let’s disable it.

 

AIX7[/] # chvg -On cgvg

AIX7[/] # lsvg cgvg

VOLUME GROUP:       cgvg                     VG IDENTIFIER:  00f6048800004c0000000134f4236851

VG STATE:           active                   PP SIZE:        128 megabyte(s)

VG PERMISSION:      read/write               TOTAL PPs:      1599 (204672 megabytes)

MAX LVs:            256                      FREE PPs:       1599 (204672 megabytes)

LVs:                0                        USED PPs:       0 (0 megabytes)

OPEN LVs:           0                        QUORUM:         2 (Enabled)

TOTAL PVs:          1                        VG DESCRIPTORS: 2

STALE PVs:          0                        STALE PPs:      0

ACTIVE PVs:         1                        AUTO ON:        yes

MAX PPs per VG:     32768                    MAX PVs:        1024

LTG size (Dynamic): 128 kilobyte(s)          AUTO SYNC:      no

HOT SPARE:          no                       BB POLICY:      relocatable

MIRROR POOL STRICT: off

PV RESTRICTION:     none                     INFINITE RETRY: no

AIX7[/] #

 

The man page for mklv states the following:

 

       -O y / n

            Enables the infinite retry option of the logical volume.

              n

                   The infinite retry option of the logical volume is not enabled. The failing I/O of the logical volume is not retried. This is the default value.

              y

                   The infinite retry option of the logical volume is enabled. The failed I/O request is retried until it is successful.

 

Standby, creating a logical volume with infinite retry enabled.

 

AIX7[/] # mklv -tjfs2 -Oy -y mylv cgvg 100

mylv

AIX7[/] # lslv mylv

LOGICAL VOLUME:     mylv                   VOLUME GROUP:   cgvg

LV IDENTIFIER:      00f6048800004c0000000134f4236851.1 PERMISSION:     read/write

VG STATE:           active/complete        LV STATE:       closed/syncd

TYPE:               jfs2                   WRITE VERIFY:   off

MAX LPs:            512                    PP SIZE:        128 megabyte(s)

COPIES:             1                      SCHED POLICY:   parallel

LPs:                100                    PPs:            100

STALE PPs:          0                      BB POLICY:      relocatable

INTER-POLICY:       minimum                RELOCATABLE:    yes

INTRA-POLICY:       middle                 UPPER BOUND:    1024

MOUNT POINT:        N/A                    LABEL:          None

DEVICE UID:         0                      DEVICE GID:     0

DEVICE PERMISSIONS: 432

MIRROR WRITE CONSISTENCY: on/ACTIVE

EACH LP COPY ON A SEPARATE PV ?: yes

Serialize IO ?:     NO

INFINITE RETRY:     yes

DEVICESUBTYPE:      DS_LVZ

COPY 1 MIRROR POOL: None

COPY 2 MIRROR POOL: None

COPY 3 MIRROR POOL: None

AIX7[/] #

 

Can I disable it, once I have a mounted filesystem on the LV? No. You must unmount the filesystem first. OK, no problem.

 

AIX7[/] # chlv -On mylv

0516-012 lchangelv: Logical volume must be closed.  If the logical

        volume contains a filesystem, the umount command will close

        the LV device.

0516-704 chlv: Unable to change logical volume mylv.

 

AIX7[/] # umount /myfs

AIX7[/] # chlv -On mylv

AIX7[/] # mount /myfs

AIX7[/] # lslv mylv

LOGICAL VOLUME:     mylv                   VOLUME GROUP:   cgvg

LV IDENTIFIER:      00f6048800004c0000000134f4236851.1 PERMISSION:     read/write

VG STATE:           active/complete        LV STATE:       opened/syncd

TYPE:               jfs2                   WRITE VERIFY:   off

MAX LPs:            512                    PP SIZE:        128 megabyte(s)

COPIES:             1                      SCHED POLICY:   parallel

LPs:                100                    PPs:            100

STALE PPs:          0                      BB POLICY:      relocatable

INTER-POLICY:       minimum                RELOCATABLE:    yes

INTRA-POLICY:       middle                 UPPER BOUND:    1024

MOUNT POINT:        /myfs                  LABEL:          /myfs

DEVICE UID:         0                      DEVICE GID:     0

DEVICE PERMISSIONS: 432

MIRROR WRITE CONSISTENCY: on/ACTIVE

EACH LP COPY ON A SEPARATE PV ?: yes

Serialize IO ?:     NO

INFINITE RETRY:     no

DEVICESUBTYPE:      DS_LVZ

COPY 1 MIRROR POOL: None

COPY 2 MIRROR POOL: None

COPY 3 MIRROR POOL: None

 

 

And last, but not least, let’s take a brief look at Active System Optimiser (ASO).  To be honest, I’m still not entirely sure how ASO works. But I have no doubt that more information will be available from IBM soon. According to the presentation material, ASO can “increase system performance by autonomously tuning system configuration”. Wow, cool! It focuses on optimizing cache and memory affinity. Hmmm, interesting. How the heck does it do that!? Only works with POWER7 and AIX 7.1.

 

So can I enable this on my p7 LPAR? Let’s give it a try!

 

AIX7[/] # oslevel -s

7100-01-02-1150

 

AIX7[/var/log/aso] # asoo -a

aso_active = 0

 

AIX7[/var/log/aso] # asoo -p -o aso_active=1

Setting aso_active to 1 in nextboot file

Setting aso_active to 1

 

AIX7[/var/log/aso] # asoo -a

aso_active = 1

 

Is the aso daemon running already? Nope.

 

AIX7[/] # ps -ef | grep aso

AIX7[/] # lssrc -a | grep aso

 aso                                            inoperative

 

Can I start it now? Nope.

 

AIX7[/var/log] # startsrc -s aso

0513-059 The aso Subsystem has been started. Subsystem PID is 7209122.

 

AIX7[/var/log] # lssrc -a | grep aso

 aso                                            inoperative

 

Let’s check the ASO log file. Oh no, my VLP LPAR doesn’t have enough CPU entitlement for ASO to start. Oh well.

 

AIX7[/var/log] # cd /var/log/aso

AIX7[/var/log/aso] # ls -ltr

total 16

-rw-r--r--    1 root     system         1143 Jan 18 22:14 aso_process.log

-rw-r--r--    1 root     system         1143 Jan 18 22:14 aso.log

 

AIX7[/var/log/aso] # cat aso.log

Jan 18 22:14:00 l488pp011_pub aso:notice aso[7209122]: /var/run/aso locked with pid 7209122

Jan 18 22:14:00 l488pp011_pub aso:notice aso[7209122]: [STOP] Maximum system entitlement is 0.5 CPUs, but must be at least 2.0 CPUs for ASO to operate.  Stopping.

Jan 18 22:14:00 l488pp011_pub aso:notice aso[7209122]: [STOP] Unsupported partition configuration detected; ASO will not run.

Just say you change the queue_depth on a hdisk with chdev –P. This updates the devices ODM information only, not its running configuration. The new value will take effect next time I reboot the system. So now I have a different queue_depth in the ODM compared to the devices current running config (in the kernel).

 

What if I forget that I’ve made this change to the ODM and forget to reboot the system for many months? Someone complains of an I/O performance issue....I check the queue_depths and find they appear to be set appropriately but I still see disk queue full conditions on my hdisks. But have I rebooted since changing the values?

 

How do I know if the ODM matches the devices running configuration?

 

For example, I start with a queue_depth of 3, which is confirmed by looking at lsattr (ODM) and kdb (running config) output:

 

# lsattr -El hdisk6 -a queue_depth

queue_depth 3 Queue DEPTH True

 

 

# echo scsidisk hdisk6 | kdb | grep queue_depth

    ushort queue_depth  = 0x3;                                                   < In Hex.

 

Now I change the queue_depth using chdev –P i.e. only updating the ODM.

 

# chdev -l hdisk6 -a queue_depth=256 -P

hdisk6 changed

 

# lsattr -El hdisk6 -a queue_depth

queue_depth 256 Queue DEPTH True

         

kdb reports that the disks running configuration still has a queue_depth of 3.

 

# echo scsidisk hdisk6 | kdb | grep queue_depth

    ushort queue_depth  = 0x3;

 

Now if I varyoff the VG and change the disk queue_depth, both lsattr (ODM) and kdb (the running config) show the same value:

 

 

# umount /test

# varyoffvg testvg

 

# chdev -l hdisk6 -a queue_depth=256

hdisk6 changed

 

# varyonvg testvg

# mount /test

 

# lsattr -El hdisk6 -a queue_depth

queue_depth 256 Queue DEPTH True

 

# echo scsidisk hdisk6 | kdb | grep queue_depth

    ushort queue_depth  = 0x100;                                    < In Hex = Dec 256.

 

# echo "ibase=16 ; 100" | bc

256

 

This is one way of checking you’ve rebooted since you changed your queue_depth attributes. I’ve tried this on AIX 6.1 and 7.1 only.

 

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.kdb/doc/kdb/kdb_pdf.pdf

 

 

I’ve been working with a customer recently on an issue with nimadm. They were attempting to migrate a system from AIX 5.3 to 7.1 using nimadm. The NIM client AIX level was 5.3 TL12 SP4 and the NIM master was running AIX 7.1 TL1 SP1.

 

lpar1 : / # oslevel -s

5300-12-04-1119

 

root@nim1 : / # oslevel -s

7100-01-01-1141

 

They were running the following nimadm command:

 

# nimadm -j nimadmvg -c lpar1 -s spotaix710101 -l lpp_sourceaix710101 -d hdisk1 -Y

 

The nimadm operation would always fail at phase 11:

 

+-----------------------------------------------------------------------------+
Executing nimadm phase 11.
+-----------------------------------------------------------------------------+
Cloning altinst_rootvg on client, Phase 3.
Client alt_disk_install command: alt_disk_copy -j -M 7.1 -P3 -d "hdisk1"
## Phase 3 ###################
Verifying altinst_rootvg...
alt_disk_copy: 0505-218 ATTENTION: init_multibos() returned an unexpected result.
Cleaning up.
forced unmount of /alt_inst/var/log
forced unmount of /alt_inst/var/log
forced unmount of /alt_inst/var
forced unmount of /alt_inst/var
forced unmount of /alt_inst/usr/local
forced unmount of /alt_inst/usr/local
forced unmount of /alt_inst/usr
forced unmount of /alt_inst/usr
forced unmount of /alt_inst/tmp
forced unmount of /alt_inst/tmp
forced unmount of /alt_inst/opt
forced unmount of /alt_inst/opt
forced unmount of /alt_inst/home
forced unmount of /alt_inst/home
forced unmount of /alt_inst/admin
forced unmount of /alt_inst/admin
forced unmount of /alt_inst
forced unmount of /alt_inst
0505-187 nimadm: Error cloning altinst_rootvg on client.
Cleaning up alt_disk_migration on the NIM master.
Cleaning up alt_disk_migration on client lpar1.
Client alt_disk_install command: alt_disk_install -M 7.1 -X
Bootlist is set to the boot disk: hdisk0 blv=hd5

 

We also found the init_multibos error in the /var/adm/ras/alt_disk_inst.log file on the NIM client:

 

Tue Nov 22 14:48:12 EETDT 2011

cmd: /ALT_MIG_SPOT/sbin/alt_disk_copy -j -M 7.1 -P3 -d hdisk1

Verifying altinst_rootvg...

alt_disk_copy: 0505-218 ATTENTION: init_multibos() returned an unexpected result.

Cleaning up.

 

Given that the error appeared to be related to init_multibos, we assumed the failure was due to some multibos checks being performed by alt_disk_copy on the client. The client system did not have an existing multibos standby instance. So, we tried two things: First we created a standby instance on the client (multibos –s –X) and re-tried the nimadm operation. This failed. Next we removed the standby instance (multibos –R) and re-tried the nimadm operation. This worked and the client then migrated to AIX 7.1 successfully. We re-tried the same operations (i.e. create standby instance, remove standby instance & nimadm) several times and each worked as expected.

 

So it appeared that the unofficial work around to this problem would be to create and then remove a standby multibos instance prior to the nimadm migrate. However, the customer has over 200 LPARs that they need to migrate to AIX 7.1. If possible they would really rather avoid this extra step in the AIX 7.1 migration plan. We’ve made contact with IBM support and are hoping they can assist us in identifying the root cause of the issue and provide us with an official solution to the problem.

 

And just yesterday we hit the same problem when migrating from AIX 6.1 to 7.1 using nimadm. I’ll update my blog with any progress we make with this problem. In the meantime, our unofficial work around will get us “out of hot water”!

 

UPDATE (14/12/2011): The simple fix is to remove the /bos_inst directory before attempting the AIX migration. i.e.

 

# rm –r /bos_inst

 

In my nimadm article I wrote:

• If you have a multibos image in rootvg, remove it. AIX migrations are not supported with multibos enabled systems. Ensure all rootvg LVs are renamed to their legacy names. If necessary, create a new instance of rootvg and reboot the LPAR. For example:

# multibos –sXp         # multibos –sX         # shutdown –Fr

 

Confirm the legacy LV names are now in use that is, not bos_.

# lsvg -l rootvg | grep hd | grep open         hd6           paging     80      160     2    open/syncd    N/A         hd8           jfs2log    1       2       2    open/syncd    N/A         hd4           jfs2       1       2       2    open/syncd    /         hd2           jfs2       7       14      2    open/syncd    /usr         hd3           jfs2       16      32      2    open/syncd    /tmp         hd1           jfs2       1       2       2    open/syncd    /home         hd9var        jfs2       8       16      2    open/syncd    /var         hd7           sysdump    8       8       1    open/syncd    N/A         hd7a          sysdump    8       8       1    open/syncd    N/A         hd10opt       jfs2       8       16      2    open/syncd    /opt

 

Remove the old multibos instance.

# multibos -R

 

Unfortunately, it appears that ‘multibos –R’ may not clean up the /bos_inst directory. If this directory exists the nimadm operation will most likely fail.