Chris's AIX Blog

Last month I applied the latest fix pack for VIOS 2.1 i.e. FP22, to one of our lab VIOS.

I came across the following error while doing so: 

 

Renaming Media Repository Logical Volume:

 

VMLibrary_LV --> VMLibrary

umount: error unmounting /dev/VMLibrary_LV: Device busy

 

Media Repository failed to unmount. We cannot continue

until it can be successfully unmounted. Please address

and retry.

 

update:  Failed while executing the ios.cli.rte.post_u script.

 

installp:  The installation has FAILED for the "root" part

        of the following filesets:

        ios.cli.rte 6.1.4.1

 

 

Despite the error, the ioslevel command reported that the fix pack had been applied. This was misleading, as a critical fileset had not been updated i.e. ios.cli.rte.

 

$ ioslevel

2.1.2.10-FP-22

 

To workaround this problem I removed the Virtual Media Repository on the VIOS.

 

$ rmrep

 

 

As the ios.cli.rte fileset had not installed properly, I had to reinstall the base fileset and then apply the update (fix pack) again.

 

  ----

    1  Total to be installed

 

+-----------------------------------------------------------------------------+

                         Installing Software...

+-----------------------------------------------------------------------------+

 

installp: APPLYING software for:

        ios.cli.rte 6.1.2.0

 

 

. . . . . << Copyright notice for ios.cli >> . . . . . . .

 Licensed Materials - Property of IBM

 

 5765G3400

   Copyright International Business Machines Corp. 2004, 2008.

 

 All rights reserved.

 US Government Users Restricted Rights - Use, duplication or disclosure

 restricted by GSA ADP Schedule Contract with IBM Corp.

. . . . . << End of copyright notice for ios.cli >>. . . .

 

0 * * * * /usr/ios/cli/cron_mail_check.sh 2>/dev/null

0 * * * * /usr/ios/cli/cron_mail_check.sh 2>/dev/null

0 * * * * /usr/ios/cli/cron_mail_check.sh 2>/dev/null

root

padmin

Finished processing all filesets.  (Total time:  18 secs).

 

+-----------------------------------------------------------------------------+

                                Summaries:

+-----------------------------------------------------------------------------+

 

Installation Summary

--------------------

Name                        Level           Part        Event       Result

-------------------------------------------------------------------------------

ios.cli.rte                 6.1.2.0         USR         APPLY       SUCCESS

ios.cli.rte                 6.1.2.0         ROOT        APPLY       SUCCESS

 

 

# lslpp -l ios.cli.rte

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  ios.cli.rte                6.1.4.1  COMMITTED  Virtual I/O Technologies

 

Path: /etc/objrepos

  ios.cli.rte                6.1.4.1  COMMITTED  Virtual I/O Technologies

 

 

Then I recreated the media repository with mkrep.

 

$ mkrep -sp rootvg -size 10G

 

At the time I thought perhaps I had done something wrong. I forgot all about it, until now. Today (a month after my experience), IBM updated the VIOS support page with the following information:

 

http://www14.software.ibm.com/webapp/set2/sas/f/vios/download/fbmr.html

 

Known issues and limitations

File Backed Optical Media Repository: If you use the File Backed Optical Media Repository, you must unload any media images before you apply Fix Pack 22.1. If you have any media images loaded, the fix pack installation fails for the

ios.cli.rte fileset. The failure may not be apparent, because the ioslevel will be updated. Refer to this page for information to help you resolve this problem.

 

And they  re-released FP22 as FP22.1.

 

http://www14.software.ibm.com/webapp/set2/sas/f/vios/download/fixpack.html

 

Fixes in Fix Pack 22.1 (VIOS V2.1.2.10 FP-22.1) include:

            Fixed problem with Electronic Service Agent in IVM

            Fixed problem in bus device driver for NPIV

            Fixed problem in device driver pertaining to reading SSD disks (MPIO fixes)

            Fixed problem in xmtopas (install problem)

            Fixed packaging problem with the Media Repository

            Fixed problem with backupios ? fails to give adequate details on stdout errors

 

 

 

Upon reading through the release notes for Fix Pack 22 for VIOS 2.1, I stumbled across a new command called chkdev. According to the notes, this command was “added to help you check devices for virtual provisioning capability”. This fired my curiosity and as soon as I had applied FP22 to one of my VIOS, I started using this command to see what it could do for me! 

 

First, I checked the man page for chkdev. It stated that chkdev can determine whether or not a device can be used in physical to virtual (p2v) migration on a VIOS.

 

chkdev command

 

Purpose

 

       Checks devices for virtual device provisioning capability in the Virtual I/O Server.

 

Syntax

 

       chkdev [-dev Name] [-verbose] [-field FieldName ] [-fmt delimiter]

 

Description

 

The chkdev command determines whether a device can be used in physical to virtual transitions in the Virtual I/O Server. Devices include all physical volumes and multipath representations of logical units. If the -dev option is used, information is shown for the device specified. If the -verbose option is used, detailed information is shown. With the -field option, the user can specify which fields to display. With the -fmt option, the user can format the output to be separated by a given delimiter.

 

Capability in a virtual environment refers to the capability for a device to move between a physical environment to a virtual environment. The command displays these fields: Phys2Virt_Capable, Virt2NPIV_Capable, and Virt2Phys_Capable.

 

For a device to be capable of moving from a physical to a virtual environment, it must pass the address compatibility check. If the physical volume can be moved from a physical to a virtual environment, the Phys2Virt_Capable field has a value of YES. If the device is not capable of this move, the field has a value of NO. If the device is already in use by Virtual Small Computer System Interface (VSCSI), the field has a value of NA.

 

If a device is capable of moving from a VSCSI LUN environment to an N_Port ID Virtualization (NPIV) environment, it must pass the address-compatibility check. If this device can be moved, the Virt2NPIV_Capable field has a value of YES. If the device is not capable of this move, the field has a value of NO. If the device is not in use by the virtual target device (VTD), this field has a value of NA. This command is not sufficient to guarantee that the client can read the data after the VSCSI logical unit number (LUN) is removed. The storage area network (SAN) configuration must also be checked by the SAN administrator to ensure that it is mapped to a client NPIV.

 

Determination of whether a device is capable of moving from a virtual environment to a physical environment is applicable for devices that are mapped as VTDs. For devices to be capable of moving from a virtual environment to a physical environment, it must pass the address capability test.

 

If the physical volume is capable of being transferred from a virtual environment to a physical environment, the Virt2Phys_Capable field has a value of YES. If the device is not capable of this move, the field has a value of NO. If the device is not in use by VTD, the field has a value of NA.

..etc..

 

So, I ran this command on my VIOS and focused on the attributes of two different disk types. The first, hdisk1, was a (internal) SAS drive and the second was a FC attached SAN disk (IBM DS8100).

 

$ hostname

vio1

 

$ ioslevel

2.1.2.10-FP-22

 

$ lsdev -type disk

name             status      description

hdisk0           Available   SAS Disk Drive

hdisk1           Available   IBM MPIO FC 2107

hdisk2           Available   IBM MPIO FC 2107

hdisk3           Available   IBM MPIO FC 2107

hdisk4           Available   IBM MPIO FC 2107

hdisk5           Available   IBM MPIO FC 2107

hdisk6           Available   IBM MPIO FC 2107

name             status      description

vtscsi0          Available   Virtual Target Device - Disk

vtscsi1          Available   Virtual Target Device - Disk

vtscsi2          Available   Virtual Target Device – Disk

 

$ lsmap -all

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost0          U7998.61X.10071FA-V1-C11                     0x00000003

 

VTD                   vtscsi0

Status                Available

LUN                   0x8200000000000000

Backing device        hdisk1

Physloc               U78A5.001.WIH074C-P1-C6-T1-W500507630608059A-L4000402D00000000

..etc..

 

$ chkdev

NAME:                hdisk0

IDENTIFIER:          2F1135000C500063973B70BST9146802SS08IBM-ESXSsas

PHYS2VIRT_CAPABLE:   YES

VIRT2NPIV_CAPABLE:   NA

VIRT2PHYS_CAPABLE:   NA

 

NAME:                hdisk1

IDENTIFIER:          200B75W2261002D07210790003IBMfcp

PHYS2VIRT_CAPABLE:   NA

VIRT2NPIV_CAPABLE:   YES

VIRT2PHYS_CAPABLE:   YES

..etc..

 

From the chkdev output, I could determine that hdisk0 could be virtualised and presented to an LPAR (PHYS2VIRT_CAPABLE=YES). The VIRT2NPIV_CAPABLE and the VIRT2PHYS_CAPABLE attributes were both set to N/A, indicating that this device is not currently mapped to a VTD on the VIOS.

 

The attributes for hdisk1 showed that this disk had already been successfully mapped to an LPAR from the VIOS i.e. PHYS2VIRT_CAPABLE=N/A. The disk was also capable of moving from a VSCSI environment to an NPIV (N_Port ID Virtualisation) environment (VIRT2NPIV_CAPABLE=YES).

 

The -verbose flag provides additional attributes about a disk device. You can obtain the PVID, UDID, IEEE and VTD attributes, which may be useful in identifying and validating disks on a VIOS. For example looking at the output below, I can easily verify which VTD hdisk1 is currently mapped to on the VIOS. And I can quickly confirm that hdisk0 is not currently mapped to a VTD.

 

 

$ chkdev -dev hdisk0 -verbose

NAME:                hdisk0

IDENTIFIER:          2F1135000C500063973B70BST9146802SS08IBM-ESXSsas

PHYS2VIRT_CAPABLE:   YES

VIRT2NPIV_CAPABLE:   NA

VIRT2PHYS_CAPABLE:   NA

PVID:                000071facb8888a20000000000000000

UDID:                2F1135000C500063973B70BST9146802SS08IBM-ESXSsas

IEEE:

VTD:

 

$ chkdev -dev hdisk1 -verbose

NAME:                hdisk1

IDENTIFIER:          200B75W2261002D07210790003IBMfcp

PHYS2VIRT_CAPABLE:   NA

VIRT2NPIV_CAPABLE:   YES

VIRT2PHYS_CAPABLE:   YES

PVID:                000071fabe8efb250000000000000000

UDID:                200B75W2261002D07210790003IBMfcp

IEEE:

VTD:                 vtscsi0

 

This command is also mentioned in a new Redbook called “PowerVM Migration from Physical to Virtual Storage”. I recommend you take a look at it and so does Rob McNelly!

I stumbled across this IBM support tech note today. It relates to controlling which network to use during a Live Partition Mobility operation.

 

http://www-01.ibm.com/support/docview.wss?uid=isg3T7000117&myns=phw&mynp=OCHW182&mync=E

 

By being given the choice of which network to use we can select network adapters that are more suited to this type of activity i.e. ones that are one or more of the following: 

high-speed, under utilized and/or dedicated to non-application data (for example). Using a “high speed” network may improve the performance of an LPM operation and reduce 

the overall time required to perform the memory copy between systems. 

The diagram in the tech note shows a dedicated Gigabit network between the source and destination VIOS. I found the following point quite interesting:

 

"NOTE:
 Currently transfer of data for 1 partition does not scale much 
beyond 1 Gbit speed, 10Gibt does not provide benefit for 
parallel migrations."

 

For those of us who were planning on using their 10GB network adapters for LPM traffic (like I was), we may not need bother at the moment.

I wonder if this will change in the future?

Recently a colleague contacted me with a question relating to hostname resolution and DNS on AIX 6.1. I thought it was an interesting discussion so I thought I’d share it with you here.

 

His question was basically this:

 

“In AIX 6.1, as you know, the resolv.conf has some additional options. Do you know what would happen if I have two nameservers in my file and the target hostname isn't found, will the second nameserver necessarily be looked up? The man page says:

 

If more than one name server is listed, the resolver routines query each name server (in the order listed) until either the query succeeds or the maximum number of attempts have been made.

 

but the rotate option seems to be set for that purpose:

 

Enables the resolver to use all the nameservers in the resolv.conf file, not just the first one.

 

If I have multiple name servers in /etc/resolv.conf, and the first one is available but the query fails, will the name resolution inevitably go to the second nameserver?

 

(My /etc/netsvc.conf entry says:

hosts = local4, bind4

and I don't have NSORDER set).

 

Any thoughts on how I could test this?”

 

A while back I mentioned that there were some new options available in the resolv.conf file for AIX 6.1, including the rotate feature. So, my response to this question was as follows.

 

By default, if the first nameserver is able to answer the query, either by returning the IP address for the target hostname OR a 'host does not exist', then this equates to a successful lookup. Only if the first nameserver does not respond and/or times out will the resolver routine send the query to the next nameserver in the list.

 

To debug this you could use the RES_OPTIONS environment variable and examine the output to see what nameservers are being called and when and in what order. For example:

 

- In the following test, my resolv.conf file has what you would typically configure i.e. a couple of nameservers and a domain entry. Note that I have two nameservers listed in this file.

 

# cat /etc/resolv.conf

nameserver 10.1.50.201

nameserver 10.1.50.202

domain cg.com

 

- I then perform a lookup of a host that is known to DNS and returns an IP address. The output indicates that only one nameserver is queried not both. As expected.

 

# RES_OPTIONS=debug host mygoodhostname | grep Query

;; Querying server (# 1) address = 10.1.50.201

 

- Likewise if I perform a lookup on a hostname that is not known to DNS, I receive a reply from the first nameserver in the list only. Again, as expected.

 

# RES_OPTIONS=debug host mybadhostname | grep Query

host: 0827-801 Host name mybadhostname does not exist.

;; Querying server (# 1) address = 10.1.50.201

 

- Now, if I add the new rotate option to my resolv.conf file, I observe different behaviour. Both nameservers are queried, regardless.

 

# cat /etc/resolv.conf

nameserver 10.1.50.201

nameserver 10.1.50.202

domain cg.com

options rotate

 

-    Both nameservers are queried to lookup the hostname of a host known to DNS.

 

# RES_OPTIONS=debug host mygoodhostname  | grep Query 

;; Querying server (# 1) address = 10.1.50.202

;; Querying server (# 2) address = 10.1.50.201

 

- Again, both nameservers are queried to lookup the hostname of a host not known to DNS. In this case, the second nameserver (10.1.50.202) is bogus and it is actually the first nameserver, 10.1.50.201, that replies i.e. Query #1.

 

# RES_OPTIONS=debug host mybadhostname  | grep Query

host: 0827-801 Host name mybadhostname does not exist.

;; Querying server (# 1) address = 10.1.50.202

;; Querying server (# 2) address = 10.1.50.201

;; Querying server (# 1) address = 10.1.50.201

 

I finished off my response by stating that this approach was probably good practice, but might have the potential to slow down hostname lookups if there are several (max. of 3) nameservers to query. I expect the performance impact would be minimal. If he was concerned with the performance hit, he could always enable the netcd daemon to cache DNS lookups locally, which might speed things up for hosts that were referenced frequently.

 

Which brings me to the netcd daemon. This was first introduced with AIX 6.1 and is included in the bos.net.tcp.client fileset.

 

# lslpp -f bos.net.tcp.client | grep netcd

                        /usr/sbin/netcdctrl

                        /usr/sbin/netcd

                        /usr/samples/tcpip/netcd.conf

 

This new subsystem can be enabled to help improve network performance and reduce network traffic. You can configure this daemon to cache answers from DNS, NIS and other server queries. This daemon is not activated by default in AIX 6.1.

 

The netcd daemon can cache resolver lookups to a network resource such as a DNS server. It will populate its cache with the result of each query. Negative answers are cached as well. When an entry is inserted to the cache, a TTL is associated to it. For DNS queries, the TTL value returned by the DNS server is used (with the default settings). The daemon will also check periodically for expired entries and remove them.

 

There are a number of configurable options for netcd. However, on my test LPAR, I simply ran the following command to start the daemon and test it. I used the lssrc command to get an overview of the active configuration.

 

# startsrc –s netcd

# lssrc -ls netcd

Subsystem         Group            PID          Status

 netcd            netcd            569432       active

Debug                     Inactive

Configuration File        /etc/netcd.conf

Configured Cache         local services

Configured Cache         local protocols

Configured Cache         local hosts

Configured Cache         local networks

Configured Cache         local netgroup

Configured Cache         dns services

Configured Cache         dns protocols

Configured Cache         dns hosts

Configured Cache         dns networks

Configured Cache         dns netgroup

Configured Cache         nisplus services

Configured Cache         nisplus protocols

Configured Cache         nisplus hosts

Configured Cache         nisplus networks

Configured Cache         nisplus netgroup

Configured Cache         nis services

Configured Cache         nis protocols

Configured Cache         nis hosts

Configured Cache         nis networks

Configured Cache         nis netgroup

 yp passwd.byname

 yp passwd.byuid

 yp group.byname

 yp group.bygid

 yp netid.byname

 yp passwd.adjunct.byname

Configured Cache         ulm services

Configured Cache         ulm protocols

Configured Cache         ulm hosts

Configured Cache         ulm networks

Configured Cache         ulm netgroup

 

If you would like the daemon to start automatically on a system restart, uncomment the following entry from the /etc/rc.tcpip file.

 

#start /usr/sbin/netcd "$src_running"

 

By default, if you start the daemon without configuring it’s associated configuration file (/etc/netcd.conf), then it will start with its default values. So just about everything is cached. If you want to trim down the configuration you can create your own /etc/netcd.conf file. There is a sample file located in /usr/samples/tcpip/netcd.conf. You can copy the file to the /etc/ directory and use it as a template for your configuration.

 

The netcdctrl command can be used to control and manage the netcd cache(s). You can dump the current contents of a cache, flush a cache, change the logging level and view statistics. To verify that netcd was caching DNS lookups on my test system, I performed the following.

 

- First I dumped the DNS cache to a file. The contents did not contain any cached DNS lookups at this point in time.

 

# netcdctrl -t dns -e hosts -a /tmp/dns.out

# cat /tmp/out1

CACHE dns, hosts, name

END CACHE dns, hosts, name

CACHE dns, hosts, address

END CACHE dns, hosts, address

 

- Next performed a DNS lookup of an internet host, ibm.com.

 

# host ibm.com

ibm.com is 129.42.17.103

 

- Again, I dumped the contents of the cache. Now I could see a cached entry for ibm.com.

 

# netcdctrl -t dns -e hosts -a /tmp/dns.out

# cat /tmp/dns.out

CACHE dns, hosts, name

>>>>>>>>>>>>>>>>>>>>>>>>>>>> ELEM #1

Expiration date : Wed Jan 27 07:50:24 2010

Ulm or resolver name : dns

Query type : 10100002

Query length : 7

Answer (0: positive; otherwise : negative) : 0

Query key : 1264134311

String used in query : ibm.com

Additional parameters in query:

        query param1 : 2

        query param2 : 0

Length of cached element : 37

################### hostent

Number of aliases = 0

Number of addresses = 3

Type = 2

Length = 4

Host name = ibm.com

Alias = 

Address = 129.42.17.103

Address = 129.42.18.103

Address = 129.42.16.103

#################### end of hostent

>>>>>>>>>>>>>>>>>>>>>>>>>>>> END ELEM #1

 

END CACHE dns, hosts, name

CACHE dns, hosts, address

END CACHE dns, hosts, address

 

It is also possible to flush the cache if something is stale and needs to be refreshed manually.

 

# netcdctrl -t dns -e hosts -f

 

The netcd daemon can cache lookups for all sorts of resolver queries (not just DNS). Some of these include local (/etc/hosts), NIS, NIS+ and YP.

 

http://www-01.ibm.com/support/docview.wss?uid=isg3T1010925

 

http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds4/netcd.htm

 

http://www.redbooks.ibm.com/abstracts/sg247559.html

 

 

Follow me on twitter: http://twitter.com/cgibbo

 

 

Came across the following IBM Wiki site which is dedicated to all things related to Live Partition Mobility (LPM) on IBMs Power6 (and Power7) platform.

Just the other day, I needed to use the AIX splitvg command in order to copy some data from one system to another.

I thought I’d share the experience here.

The splitvg command can split a single mirror copy of a fully mirrored volume group into a separate “snapshot” volume group.

From the man page:

The original volume group VGname will stop using the disks that are now part of the snapshot volume group SnapVGname. Both volume groups will keep track of the writes within the volume group so that when the snapshot volume group is rejoined with the original volume group consistent data is maintained across the rejoined mirrors copies. Notes:

1         To split a volume group, all logical volumes in the volume group must have the target mirror copy and the mirror must exist on a disk or set of disks. Only the target mirror copy must exist on the target disk or disks.

 

2         The splitvg command will fail if any of the disks to be split are not active within the original volume group.

 

3         In the unlikely event of a system crash or loss of quorum while running this command, the joinvg command must be run to rejoin the disks back to the original volume group.

 

4         There is no concurrent or enhanced concurrent mode support for creating snapshot volume groups.

 

5         New logical volumes and file system mount points will be created in the snapshot volume group.

 

6         The splitvg command is not supported for the rootvg.

 

7         The splitvg command is not supported for a volume group that has an active paging space.

 

8         When the splitvg command targets a concurrent-capable volume group which is varied on in non-concurrent mode, the new volume group that is created will not be varied on when the splitvg command completes. The new volume group must be varied on manually.

 

So, looking at point 4, above, if you are using enhanced concurrent volume groups (for example with PowerHA), then you will not be able to use the splitvg command. This is disappointing as this would have been very handy in some of the large PowerHA systems I have worked with…..perhaps this will be supported in the future?

Anyway, back to my example. I had wanted to break-off one of the mirrors of a mirrored volume group and then assign the “split” volume group to another host to copy some data off.

The volume group datavg contained two disks, hdisk0 and hdisk3, as shown in the lspv output below.

 

# lspv
hdisk1          00c01c705bdc6136                    old_rootvg
hdisk0          00c01c70c050810f                    datavg              active
hdisk2          00c01c7018a47201                    rootvg              active
hdisk3          00c01c70fed9e41a                    datavg              active

 

There were only two logical volumes (loglvl00, jfs2 log and fslv00, data) and a single file system (/data) in this volume group.

 

# lsvg -l datavg
datavg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
loglv00             jfs2log    1       2       2    open/syncd    N/A
fslv00              jfs2       16      32      2    open/syncd    /data

 

The volume group datavg, was mirrored across hdisk0 and hdisk3, as shown in the lsvg output below.

 

# lsvg -p datavg
datavg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            583         566         117..100..116..116..117
hdisk3            active            583         566         117..100..116..116..117

 

The logical volumes for the /data file system and the JFS2 log were both mirrored, as shown in the lslv/lspv output.

 

# lslv fslv00
LOGICAL VOLUME:     fslv00                 VOLUME GROUP:   datavg
LV IDENTIFIER:      00c01c7000004c0000000122e9038480.2 PERMISSION:     read/write
VG STATE:           active/complete        LV STATE:       opened/syncd
TYPE:               jfs2                   WRITE VERIFY:   off
MAX LPs:            512                    PP SIZE:        256 megabyte(s)
COPIES:             2                      SCHED POLICY:   parallel
LPs:                16                     PPs:            32
STALE PPs:          0                      BB POLICY:      relocatable
INTER-POLICY:       minimum                RELOCATABLE:    yes
INTRA-POLICY:       middle                 UPPER BOUND:    32
MOUNT POINT:        /data                    LABEL:          /data
MIRROR WRITE CONSISTENCY: on/ACTIVE
EACH LP COPY ON A SEPARATE PV ?: yes
Serialize IO ?:     NO

 

# lslv loglv00
LOGICAL VOLUME:     loglv00                VOLUME GROUP:   datavg
LV IDENTIFIER:      00c01c7000004c0000000122e9038480.1 PERMISSION:     read/write
VG STATE:           active/complete        LV STATE:       opened/syncd
TYPE:               jfs2log                WRITE VERIFY:   off
MAX LPs:            512                    PP SIZE:        256 megabyte(s)
COPIES:             2                      SCHED POLICY:   parallel
LPs:                1                      PPs:            2
STALE PPs:          0                      BB POLICY:      relocatable
INTER-POLICY:       minimum                RELOCATABLE:    yes
INTRA-POLICY:       middle                 UPPER BOUND:    32
MOUNT POINT:        N/A                    LABEL:          None
MIRROR WRITE CONSISTENCY: on/ACTIVE
EACH LP COPY ON A SEPARATE PV ?: yes
Serialize IO ?:     NO

 

# lspv -l hdisk0
hdisk0:
LV NAME               LPs     PPs     DISTRIBUTION          MOUNT POINT
fslv00                16      16      00..16..00..00..00    /data
loglv00               1       1       00..01..00..00..00    N/A

 

# lspv -l hdisk3
hdisk3:
LV NAME               LPs     PPs     DISTRIBUTION          MOUNT POINT
fslv00                16      16      00..16..00..00..00    /data
loglv00               1       1       00..01..00..00..00    N/A

 

Using the splitvg command I was able to break-off one of the disks from the mirrored pair. This created a new volume group on hdisk3 called vg00.

 

# splitvg -c1 datavg
# lspv
hdisk1          00c01c705bdc6136                    old_rootvg
hdisk0          00c01c70c050810f                    datavg            active
hdisk2          00c01c7018a47201                    rootvg          active
hdisk3          00c01c70fed9e41a                    vg00            active

 

The new volume group contains a new logical volume (pre-fixed with fs i.e. fsfslv00) and a file system (pre-fixed with /fs i.e. /fs/data). I can mount this file system and access the data in the file system and create and/or modify files (as shown below).

 

# mount /fs/data
Replaying log for /dev/fsfslv00.

# cd /fs/data
# ls -ltr
total 112
drwxrwxrwx    2 root     system        53248 Jan 30 2009  AIX61TL2SP2
drwxr-xr-x    2 root     system          256 Aug 05 15:24 lost+found
-rw-r--r--    1 root     system            0 Sep 15 10:29 2

# touch 3
# ls -ltr
total 112
drwxrwxrwx    2 root     system        53248 Jan 30 2009  AIX61TL2SP2
drwxr-xr-x    2 root     system          256 Aug 05 15:24 lost+found
-rw-r--r--    1 root     system            0 Sep 15 10:29 2
-rw-r--r--    1 root     system            0 Sep 15 10:46 3

 

At this point I was able to export the volume group and import it on another system. I had to re-map the Virtual SCSI disk on my VIOS first.  

 

# cd
# umount /fs/data

# varyoffvg vg00
# exportvg vg00

# rmdev –dl hdisk3

; Re-map the disk at the VIOS layer to the other AIX LPAR.

; Configure the disk on the other LPAR (with cfgmgr) and import the volume group (with importvg).

 

Once I was finished with vg00 on the other LPAR, I re-mapped the disk to the original AIX LPAR, configured the disk and then re-joined the disk to the datavg volume group, with the joinvg command.

 

; Export the VG and remove the disk on the other AIX LPAR.

; Re-map the disk at the VIOS layer to the original AIX LPAR.

# cfgmgr

# lsdev –Cc | grep hdisk3

hdisk3 Available  Virtual SCSI Disk Drive

 

# joinvg datavg
# lspv
hdisk1          00c01c705bdc6136                    old_rootvg
hdisk0          00c01c70c050810f                    datavg            active
hdisk2          00c01c7018a47201                    rootvg          active
hdisk3          00c01c70fed9e41a                    datavg            active

 

The volume group is now fully mirrored again. However the partitions are stale and need to be sync’ed. I manually re-sync the volume group with the syncvg command.

 

# lsvg -l datavg
datavg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
loglv00             jfs2log    1       2       2    open/stale    N/A
fslv00              jfs2       16      32      2    open/stale    /data

# syncvg –v datavg

# lsvg -l datavg
datavg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
loglv00             jfs2log    1       2       2    open/syncd    N/A
fslv00              jfs2       16      32      2    open/syncd    /data

 

More information can be found here: http://www-01.ibm.com/support/docview.wss?uid=isg3T1010934

 

 

Today, IBM announced AIX version 7.  http://www-03.ibm.com/systems/power/software/aix/v71/preview.html

Several new features were mentioned in the launch, but there were two new features that I found particularly interesting:

 

- AIX 5.2 WPARs for AIX 7

- Cluster Aware AIX

 

I thought I would briefly describe each feature in this post.

 

AIX 5.2 WPARs for AIX 7

 

In AIX version 7, administrators will now have the capability to create Workload Partitions (WPARs) that can run AIX 5.2, inside an AIX 7 operating system instance. This will be supported on the POWER7 server platform.  This is pretty cool. IBM have done this to allow some customers, that are unable to migrate to later generations of AIX and Power, to move to POWER7 whilst keeping their legacy AIX 5.2 systems operational. So for those clients that MUST stay on AIX 5.2 (for various reasons such as Application support) but would like to run their systems on POWER7, this feature may be very attractive. It will help to reduce the effort required when consolidating older AIX 5.2 systems onto newer hardware. It may also reduce some of the risk associated with migrating applications from one version of the AIX operating system to another.

 

To migrate an existing AIX 5.2 system to an AIX 7 WPAR, administrators will first need to take a mksysb of the existing system. Then they can simply restore the mksysb image inside the AIX 7 WPAR. IBM will also offer limited defect and how-to support for the AIX 5.2 operating system in an AIX 7 WPAR. These WPARs can, of course, be managed via IBM Systems Director with the Workload Partitions Manager plug-in.

 

The following figure provides a visualization of how these AIX 5.2 systems will fit into an AIX 7 WPAR. The WPARs in blue are native AIX 7 WPARs, while the WPARs in orange are AIX 5.2 WPARs running in the same AIX 7 instance. Pretty amazing really!

I can only speculate as to what other versions of AIX will be supported in this manner in the future. Just imagine an AIX 5.3 WPAR inside AIX7?

 

 

Cluster Aware AIX

 

Another very interesting feature of AIX 7 is a new technology known as “Cluster Aware AIX”. Believe it or not, administrators will now be able to create a cluster of AIX systems using features of the new AIX 7 kernel. IBM have introduced this “in built” clustering to the AIX OS in order to simplify the configuration and management of highly available clusters. This new AIX clustering has been designed to allow for:

 

- The easy creation of clusters of AIX instances for scale-out computing or high availability.

- Significantly simplify cluster configuration, construction, and maintenance.
- Improve availability by reducing the time to discover failures.

-  Capabilities such as common device naming to help simplify administration.

- Built in event management and monitoring.

- A foundation for future AIX capabilities and the next generation of PowerHA SystemMirror.

 

This does not replace PowerHA but it does change the way in which AIX traditionally integrates with cluster software like HACMP and PowerHA. A lot of the HA cluster functionality is now available in the AIX 7 kernel itself. However, the mature RSCT technology is still a component of the AIX and PowerHA configuration. I’m looking forward to reading more about this new technology and it’s capabilities.

 

These are just two of the many features introduced in AIX 7. I’m eagerly looking forward to what these features and others mean for the future of the AIX operating system. It’s exciting to watch this operating system grow and strengthen over time. I can’t wait to get my hands on an AIX 7 system so that I can trial these new features.

 

And speaking of trialing AIX 7, there is good news. IBM plan on running another AIX Open Beta program for AIX 7 mid 2010. Just as they did with AIX Version 6, customers will be given the opportunity to download a beta version of AIX 7 and trial it on their own systems in their own environment. This is very exciting and I’m really looking forward to it.

 

I encourage you to read the official AIX 7 announcement to learn more about the future of the AIX operating system and what it can do for you and your organization…..for many years to come!

 

Back in 2008, VIOS version 1.5.2.1, Fix Pack 11.1 was released. A new command was introduced with this Fix Pack, called seastat.

 

This handy little utility can be used to view Shared Ethernet Adapter (SEA) statistics per virtual I/O client.

 

This is not to be confused with a certain shell script that Nigel Griffiths wrote a while back!

 

From the man page:

 

“The seastat command generates a report to view, per client, shared ethernet adapter statistics.

To gather network statistics at a per-client level, advanced accounting can be enabled on the Shared Ethernet Adapter to provide more information about its network traffic.

 

To enable per-client statistics, the VIOS administrator can set the Shared Ethernet Adapter accounting attribute to enabled. The default value is disabled.

When advanced accounting is enabled, the Shared Ethernet Adapter keeps track of the hardware (MAC) addresses of all of the packets it receives from the LPAR  clients, and increments packet and byte counts for each client independently. After advanced accounting is enabled on the Shared Ethernet Adapter, the VIOS administrator can generate a report to view per-client statistics by running the seastat command.”

 

Before using the command, you must first enable “advanced accounting” on the SEA. To enable advanced accounting on the SEA, enter the following command:

 

$ chdev -dev ent10 -attr accounting=enabled

 

When advanced accounting is enabled, the SEA keeps track of the hardware (MAC) addresses of all of the packets it receives from the LPAR clients, and increments packet and byte counts for each client independently.

 

After advanced accounting is enabled on the SEA, the VIOS administrator can generate a report to view per-client statistics by running the seastat command.

 

You can confirm that accounting is enabled by checking the SEA device attributes as shown below:

 

$ lsdev -dev ent10 –attr accounting

 

attribute  value    description user_settable

accounting enabled  Enable per-client accounting of network statistics True

 

To display Shared Ethernet Adapter statistics for ent10, I would enter the following command:

 

$ seastat -d ent10

===========================================================================

 

Advanced Statistics for SEA

Device Name: ent10

 

===========================================================================

MAC: 4A:73:E0:00:A0:11

----------------------

 

VLAN: None

VLAN Priority: None

Hostname: hvio7

IP: 10.3.73.113

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 18319573                      Packets: 10909416

Bytes: 27063241270                     Bytes: 915289249

 

===========================================================================

MAC: 4A:73:E0:00:A0:11

----------------------

 

VLAN: None

VLAN Priority: None

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 76667                         Packets: 1038372

Bytes: 4532682                         Bytes: 98065990

 

===========================================================================

MAC: 4A:73:E0:00:B0:11

----------------------

 

VLAN: None

VLAN Priority: None

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 76462                         Packets: 1038169

Bytes: 4520904                         Bytes: 98054070

 

===========================================================================

MAC: 4A:73:E0:00:B0:11

----------------------

 

VLAN: None

VLAN Priority: None

IP: 10.3.73.115

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 18262275                      Packets: 10813342

Bytes: 27015331139                     Bytes: 908259917

 

===========================================================================

MAC: 4A:73:E0:00:E0:0B

----------------------

 

VLAN: None

VLAN Priority: None

IP: 10.3.73.123

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 822646                        Packets: 1407153

Bytes: 55271389                        Bytes: 1951251791

 

===========================================================================

MAC: 4A:73:E0:00:E0:0B

----------------------

 

VLAN: None

VLAN Priority: None

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 292                           Packets: 9901

Bytes: 17130                           Bytes: 875286

 

===========================================================================

MAC: 4A:73:E0:01:00:0B

----------------------

 

VLAN: None

VLAN Priority: None

IP: 10.3.73.118

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 32145973                      Packets: 28000976

Bytes: 31146952180                     Bytes: 21631824293

 

===========================================================================

MAC: 4A:73:E0:01:00:0B

----------------------

 

VLAN: None

VLAN Priority: None

IP: 10.3.30.66

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 7439                          Packets: 962069

Bytes: 515632                          Bytes: 93477510

 

===========================================================================

MAC: 4A:73:E0:01:00:0B

----------------------

 

VLAN: None

VLAN Priority: None

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 78994                         Packets: 1035678

Bytes: 4720434                         Bytes: 97904350

 

===========================================================================

MAC: 4A:73:E0:01:10:0B

----------------------

 

VLAN: None

VLAN Priority: None

IP: 10.3.73.119

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 24864143                      Packets: 22249795

Bytes: 26229038743                     Bytes: 15952109586

 

===========================================================================

MAC: 4A:73:E0:01:10:0B

----------------------

 

VLAN: None

VLAN Priority: None

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 79016                         Packets: 1036886

Bytes: 4690722                         Bytes: 97976763

 

===========================================================================

MAC: 4A:73:E0:01:10:0B

----------------------

 

VLAN: None

VLAN Priority: None

IP: 10.3.30.67

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 7318                          Packets: 959949

Bytes: 504241                          Bytes: 93299959

 

===========================================================================

MAC: 4A:73:E0:01:20:0B

----------------------

 

VLAN: None

VLAN Priority: None

IP: 10.3.73.120

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 41236479                      Packets: 51720944

Bytes: 27649217018                     Bytes: 59572685172

 

===========================================================================

MAC: 4A:73:E0:01:20:0B

----------------------

 

VLAN: None

VLAN Priority: None

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 77521                         Packets: 1035290

Bytes: 4637346                         Bytes: 97881070

 

===========================================================================

MAC: 4A:73:E0:01:20:0B

----------------------

 

VLAN: None

VLAN Priority: None

IP: 10.3.30.68

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 7344                          Packets: 962069

Bytes: 505885                          Bytes: 93477510

 

===========================================================================

MAC: 4A:73:E0:01:30:0B

----------------------

 

VLAN: None

VLAN Priority: None

IP: 10.3.73.117

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 254647796                     Packets: 146507863

Bytes: 365619876921                    Bytes: 29550419740

 

===========================================================================

MAC: 4A:73:E0:01:30:0B

----------------------

 

VLAN: None

VLAN Priority: None

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 78876                         Packets: 1036670

Bytes: 4685868                         Bytes: 97963870

 

 

The seastat output provides several interesting bits of information for each VIOC communicating via a particular SEA. Such as the IP address of the client

LPAR, the MAC address, the VLAN tag and the packets transmitted/received.

 

 

 

This is a great way to monitor traffic/activity over a particular SEA. It can be very useful when determining if an SEA is currently being used  i.e. during troubleshooting network connectivity issues between client LPARs and an external network that is bridged via a SEA.

 

More information can be found here:

 

http://publib.boulder.ibm.com/infocenter/powersys/v3r1m5/index.jsp?topic=/iphcg/seastat.htm

 

 

 

 

 

 

A quick tip for monitoring Multiple Shared Processor Pools with topas.

Execute topas from the AIX command line. Press the ‘p’ key to toggle on ‘pool metrics’. You will see the default pool (pool id 0) and any other pools defined on your system. In the example below, there are two pools, the default (0) and a user defined pool (1). The default display shows the LPARs in the default pool (0), which in this case is sys06_partition8.

Using the up and down arrow keys, I place the cursor on pool ID number one, in the pool column on the left-hand side and then press the ‘f’ key to toggle the focus. This changes what is displayed in the lower portion of the output. The LPAR associated with pool 1 (sys06_partition9) is shown.

# topas –C

  

 

  

The nohup command in AIX has the ability to “nohup” an existing process and protect if from hangup signals. Using the -p flag, an existing, running process can be “nohup’ed” and told to ignore SIGHUPs.

 

"Why is this useful?" I hear you ask.

 

Well, I'll often start a "long running" process on a system at work, like a backup for instance. I'll sometimes forget to use the nohup command, so that my long running job will ignore any SIGHUPs and continue running, say for example if my ssh session to the server dies. The nohup command also generates a nohup.out file to capture any messages (errors or otherwise) generated by the process e.g.

 

$ nohup ./processname &

$ ls –l nohup.out

-rw-------    1 cgibson  staff  0 May 26 19:42 nohup.out

 

In older releases of AIX, if I forgot to use nohup, I had two choices. I could either stop the process and start it again with nohup  (often not desirable or possible) or I could leave my ssh session logged in on my work PC and hope that it didn't power down or crash or that my ssh connection did not drop over night.

 

Fortunately with more recent releases of AIX, this is not a problem anymore. If I forget to “nohup” a process, I can use the -p flag to inform the already running process that it should ignore all hangup (SIGHUP) signals. This will allow the process to continue running if my ssh session (terminal) dies and will also capture any output from the process into the nohup.out file, so I don’t lose any important output from the process.

 

Here’s an example. I’m at work, it's 5pm and I've started a "long running" process named mycmd. This was started via an ssh (Putty) session from my PC.

 

$ ./mycmd

 

; Partial output from topas showing mycmd running.

Name      PID      CPU%  

mycmd    823432    98.8       

 

 

; proctree output displaying PIDs associcated with mycmd and the associated ssh session (tty).

 

$ proctree 823432

520394    -ksh

   585738    -ksh

      823432    /home/cgibson/mycmd

 

I've walked out the door and I'm now on my way home for the evening.

 

Oh no! I've forgotten to start the process with nohup! I don't want to restart my process but I want to protect it from SIGHUPs, while it executes after hours.

 

No problem, I'll use nohup to modify the running process, so I can rest assured it will continue to run, even if my ssh connection drops over night. From home I start a new ssh session on the server and run  nohup against the running process:

 

$ nohup -p 823432

 

At work the next morning, I discover that my original Putty session has indeed dropped! But, my process is still happily running! :)

 

; Partial output from topas showing mycmd still running.

 

Name      PID      CPU%  

mycmd    823432    99.3  

$ proctree 823432

520394    -ksh

   585738    -ksh

      823432    /home/cgibson/mycmd

 

You might find this useful if you ever forget to “nohup” a process and don’t want to restart it again.

 

http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds4/nohup.htm

 

 

 

Have you ever encountered this error message on AIX before?

 

# man -k loopmount
man: 0703-310 file man not found.
Create the whatis database using the <catman -w> command

Of course you have! If you’ve ever installed AIX from scratch then you have undoubtedly seen this message.

 

And of course, you dutifully executed the ‘catman –w’ command to rectify the situation. Didn’t you?

But what if  ‘catman –w’  doesn’t work as expected?

 

I recently came across this minor issue. Running catman produced the following error message:

 

# catman -w    

catman: 0703-102 /usr/lbin/mkwhatis: The error number is 1.

 

The first place to check was the MANPATH environment variable (which didn’t look right).

 

# echo $MANPATH

:/opt/csm/man

 

So I unset it and ran catman again. This time, with ‘great success’.

 

# unset MANPATH

# catman -w  

 

Now I could search for keywords against the /usr/share/man/whatis database that contain the string “loopmount”.

 

# man -k loopmount

loopmount(1)    - Associate an image file to a loopback device.

 

Another interesting issue I’ve also come across is when catman runs with apparent success but I still can’t find keyword entries when I run  ‘man –k’.

 

# catman -w

# man -k loopmount

loopmount: Cannot find matching entry.

 

I found that manually setting MANPATH and re-running catman rectifies this situation.

 

# export MANPATH=/usr/share/man

# catman -w

# man -k lsvg

loopmount(1)    - Associate an image file to a loopback device.

 

Of course the whatis command can also perform a similar function. Only difference is that you need to know the command beforehand.

 

# whatis loopmount

loopmount(1)    - Associate an image file to a loopback device.

The catman command:

http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds1/catman.htm

 The whatis command:

http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds6/whatis.htm

Locating a command by keyword:

http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.baseadmn/doc/baseadmndita/HT_locating_command_keyword.htm

The /usr/share directory:

 http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.baseadmn/doc/baseadmndita/usr_share_dir.htm