Have you ever encountered this error message on AIX before?

 

# man -k loopmount
man: 0703-310 file man not found.
Create the whatis database using the <catman -w> command

Of course you have! If you’ve ever installed AIX from scratch then you have undoubtedly seen this message.

 

And of course, you dutifully executed the ‘catman –w’ command to rectify the situation. Didn’t you?

But what if  ‘catman –w’  doesn’t work as expected?

 

I recently came across this minor issue. Running catman produced the following error message:

 

# catman -w    

catman: 0703-102 /usr/lbin/mkwhatis: The error number is 1.

 

The first place to check was the MANPATH environment variable (which didn’t look right).

 

# echo $MANPATH

:/opt/csm/man

 

So I unset it and ran catman again. This time, with ‘great success’.

 

# unset MANPATH

# catman -w  

 

Now I could search for keywords against the /usr/share/man/whatis database that contain the string “loopmount”.

 

# man -k loopmount

loopmount(1)    - Associate an image file to a loopback device.

 

Another interesting issue I’ve also come across is when catman runs with apparent success but I still can’t find keyword entries when I run  ‘man –k’.

 

# catman -w

# man -k loopmount

loopmount: Cannot find matching entry.

 

I found that manually setting MANPATH and re-running catman rectifies this situation.

 

# export MANPATH=/usr/share/man

# catman -w

# man -k lsvg

loopmount(1)    - Associate an image file to a loopback device.

 

Of course the whatis command can also perform a similar function. Only difference is that you need to know the command beforehand.

 

# whatis loopmount

loopmount(1)    - Associate an image file to a loopback device.

The catman command:

http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds1/catman.htm

 The whatis command:

http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds6/whatis.htm

Locating a command by keyword:

http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.baseadmn/doc/baseadmndita/HT_locating_command_keyword.htm

The /usr/share directory:

 http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.baseadmn/doc/baseadmndita/usr_share_dir.htm

 

 

 

If you are a system administrator that is responsible for managing AIX systems that run SAP, then you’ve probably had an experience similar to the following?

 

OK, so one day my SAP Basis administrator contacts me and says “I can’t start saposcol......can you please reboot the system?” I quickly reply, “Was there an error message when trying to restart saposcol?”. He replies, “No”. Again I return very quickly, “OK, have you checked to see if there are any shared memory segments left for saposcol?”.  Just as quick, he replies “How do I do that?”

 

Together we try starting saposcol and what we find is that it thinks it’s already running (as shown below, PID 327924). But there is no such process!

 

$ saposcol -l

-l

09:37:24 22.01.2009   LOG: Effective User Id is root

***********************************************************************

* This is Saposcol Version COLL 20.95 700 - AIX v11.15 5L-64 bit 080317

* Usage:  saposcol -l: Start OS Collector

*         saposcol -k: Stop  OS Collector

*         saposcol -d: OS Collector Dialog Mode

*         saposcol -s: OS Collector Status

* The OS Collector (PID 327924) is already running .....

************************************************************************

$ ps -fp 327924

     UID     PID    PPID   C    STIME    TTY  TIME CMD

 

So we try to stop saposcol. This of course fails, as there is no PID 327924 running!

 

$ saposcol -k

 

Setting Stop Flag :

09:19:29 22.01.2009   LOG: ==== Stop Flag was set by saposcol (kill_collector()).

 

09:19:29 22.01.2009   LOG: ====  The collection process will stop as soon as possible

 

********************

can't kill process 327924.

kill: No such process

ERROR:No reaction from collecting process 327924.

Please kill collecting process.

 

My conclusion is that there must be a shared memory segment still allocated for saposcol. There were many other SAP processes still running happily, so there were several shared memory segments to sift through. So, what shared memory ID does saposcol use?

 

Now, according to the following website, shared memory key 4dbe is used by saposcol on AIX.

 

http://www.saptechies.com/os-collector-saposcol/

7. Question: How do I remove the shared memory key of saposcol?

Answer: Sometimes it may be necessary to remove the shared memory key of saposcol (see point 6). Caution: Please be very careful! This procedure should be performed only after checking that saposcol is really not running (see point 4) and only in cases when other options (see point 6) really do not work! For this, execute command “ipcs -ma” and note the line that contains saposcol key “4dbe”. You need the shared memory ID. After that, execute command ipcrm -m ID. Now the command “saposcol -s” should show that saposcol is not running and that the shared memory is not attached. The shared memory key will be created automatically by the saposcol when the collector is next started: “saposcol -l”.

 

So I run ipcs to check for the existence of 4dbe. And I find an entry for this key.

 

There are several process id’s ‘attached’ to this segment. However, only one of them actually exists (PID 2293794).

 

# ipcs -ma | grep 4dbe

m   2097156 0x00004dbe --rw-rw-rw-     root   sapsys     root   sapsys      0   1870188 467164 2293794  9:39:43  9:39:43  8:12:43

 

# ps -fp 1870188

     UID     PID    PPID   C    STIME    TTY  TIME CMD

 

# ps -fp 467164

     UID     PID    PPID   C    STIME    TTY  TIME CMD

 

# ps -fp 2293794

     UID     PID    PPID   C    STIME    TTY  TIME CMD

  sapadm 2293794       1   0 19:26:41      -  1:07 sapccm4x -DCCMS pf=/sapmnt/SAP/profile/SAP_DVEBMGS00_aix01

 

I ask the SAP admin to stop this process, which he does. Now I remove the shared memory segment and there is no evidence of 4dbe in the ipcs output.

 

# ipcrm -m 2097156

# ipcs -ma | grep 4dbe

 

We were then able to start saposcol again with success. The process is running and the shared memory segment 4dbe has returned.

 

# ipcs -ma | grep 4dbe

m   3145732 0x00004dbe --rw-rw-rw-     root   sapsys     root   sapsys      1   1870188 2293818 2207892 10:14:25 10:14:25 10:12:47

 

# ps -ef | grep oscol

  sapadm 2064616       1   0 10:12:53      -  0:00 saposcol -l

   

# /usr/sap/SAP/SYS/exe/run/saposcol -s

**************************************************************

Collector Versions :

  running : COLL 20.95 700 - AIX v11.15 5L-64 bit 080317

  dialog  : COLL 20.95 700 - AIX v11.15 5L-64 bit 080317

Shared Memory       : attached

Number of records   : 17640

Active Flag         : active (01)

Operating System    : AIX aix01 3 5 00C01C704C00

Collector PID       : 2064616 (001F80E8)

Collector           : running

Start time coll.    : Thu Jan 22 10:12:53 2009

 

Current Time        : Thu Jan 22 10:17:46 2009

 

Last write access   : Thu Jan 22 10:17:38 2009

 

Last Read  Access   : Thu Jan 22 10:16:16 2009

 

Collection Interval : 10 sec (next delay).

Collection Interval : 10 sec (last ).

Status              : free

Collect Details     : required

Refresh             : required

 

Header Extention Structure

Number of x-header      Records : 1

Number of Communication Records : 60

Number of free Com.     Records : 60

Resulting offset to 1.data rec. : 61

 

Trace level             : 2

 

Collector in IDLE - mode ? : NO

  become idle after 300 sec without read access.

  Length of Idle Interval  : 60 sec

  Length of norm.Interval  : 10 sec

**************************************************************

 

Problem solved and no reboot required.....this is AIX after all!  :)

 

OK, it’s official. The AIX 7 Open Beta is now available for testing!

 

An overview of the program is available here:

 

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/aix7ob/index.shtml

 

Essentially this program is designed to give IBM customers, ISVs and IBM BPs the opportunity to gain early experience with the latest release of AIX prior to general availability. This is a great time to join forces and help IBM mould the next generation of the AIX OS.

 

I got involved in the AIX 6 Open Beta back in 2007. It was a worthwhile experience. The time I spent learning new features like WPARs and RBAC, put me in a good position when it came time to actually implement these outside of my lab environment. It was also a good opportunity to provide feedback to the IBM AIX development community. Several AIX developers monitored the comments/questions in the Beta Forum and provided advice (and sometimes fixes) for known (and unknown!) issues with the beta release. It also provided the developers with plenty of real world feedback that they could take back to the labs, long before the product was officially released. This certainly helped fix bugs and improve certain enhancements before customers starting using the OS in their computing environments.

 

The AIX 7 beta code is available for download from here:

 

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/aix7ob/download.shtml

 

The user forum, where you can provide feedback and ask questions, is here:

 

https://www.ibm.com/developerworks/forums/forum.jspa?forumID=2255                        

 

I recommend you download the documentation and start by reading the Getting Started guide. The documentation is available here:

 

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/aix7ob/library.shtml

 

The Getting Started guide provides useful information that you will need to know before attempting to install the OS. For example, the beta code will run on any IBM System p, eServer pSeries or POWER system that is based on PPC970, POWER4, POWER5, POWER6 or POWER7 processors.

 

The guide also describes what new functionality has been included in this release of the beta. If this program is anything like the AIX 6 beta, there may be more than one release of the code, with further enhancements available in each release. The new function in this release includes:

 

A.    AIX 5.2 Workload Partitions for AIX 7.

B.    Removal of WPAR local storage device restrictions.

C.    Domain support in Role Based Access Control (RBAC).

D.    Etherchannel enhancements across stacked switches in IEEE 802.3ad mode.

E.    NIM thin server support for IPv6 and NFSv4.

 

I’m interested in testing A, B and D. Here’s why.

 

A.  AIX 5.2 Workload Partitions for AIX 7.

AIX 5.2 Workload Partitions for AIX 7 provides the capability to create a WPAR running AIX 5.2 TL10 SP8. This allows a migration path for an AIX 5.2 system running on old hardware to move to POWER7. All that is required is to create a mksysb image of the AIX 5.2 system and then provide this image when creating the WPAR. The WPAR must be created on a system running AIX 7 on POWER7 hardware. This is a very interesting feature, one that I am eager to test.

 

B.  Removal of WPAR local storage device restrictions.

AIX 7 will allow for exporting a virtual or physical fibre channel adapter to a WPAR. The WPAR will essentially own the physical adapter and its child devices. This will allow for SAN storage devices to be directly assigned to the WPAR's FC adapter(s). This means it will not be necessary to provision the storage in the Global environment first and then export it to the WPAR. This is also interesting as we may now be able to assign SAN disk to a WPAR for both rootvg and data volume groups. Maybe even FC tape devices within a WPAR will work?

 

D.  Etherchannel enhancements in 802.3ad mode.

There are some enhancements to AIX 7 EtherChannel support for 802.3AD mode. The enhancement makes sure that the link is LACP ready before sending data packets. If I’m interpreting this correctly, this will ensure that the aggregated link is configured appropriately. If it’s not, it will provide an error in the AIX errpt stating that the link is not configured correctly. This can help avoid situations where the AIX EtherChannel is configured but the Network Switch is not. At present there is very little an AIX administrator can do as the link will appear to be functioning even if the Switch end has not been configured for an aggregated link.

 

And the most important point in the Getting Started guide has to be, how to install the AIX beta code! An ISO image of the code is provided for download. The installation steps are straightforward as the image can be installed via a DVD device. Using a media repository on a VIO server could be one way to accomplish this task. Unfortunately, there is no mention of NIM install support yet. Here are the basic steps from the guide:

 

Installing the AIX Open Beta Driver

The AIX 7 Open Beta driver is delivered by restoring a system backup (mksysb) of the code downloaded via DVD ISO image from the AIX Open Beta web-site.

 

Once you have downloaded and created the AIX 7 Open Beta media (as described above) follow the following steps to install the ‘mksysb’.

 

1.     Put the DVD of the AIX Open Beta in the DVD drive. A series of screens/menus will be displayed. Follow the instruction on the screens and make the following selections:

• Type 1 and press Enter to have English during install.

• Type 1 to continue with the install.

• Type 1 to Start Install Now with Default Setting.

 

2.     The system will start installing the AIX 7.0 BETA.

 

3.     Upon completion of the install, the system will reboot. You can then login as “root”, no password is required.

 

Next I recommend that you take a look at the Release Notes. It provides a few bits of information that may come in handy when planning for the install, such as:

 

·         The Open Beta code is being delivered via an “mksysb” install image. Migration installation is not supported with the open beta driver.

 

·         The open beta driver does not support IBM Systems Director Agent.

 

·         When installing in a disk smaller than 15.36 GB, the following warning is displayed: A disk of size 15360 was specified in the bosinst.data file, but there is not a disk of at least that size on the system. You can safely ignore this warning.

 

·         The image is known to install without issues on an 8 GB disk.

 

·         oslevel output shows V7BETA.

 

By the way, if you are unable to find a spare system or LPAR on which to install the beta, perhaps you can consider using the IBM Virtual Loaner Program (VLP). They are planning to support LPARs running the AIX 7 Open Beta starting from July 17^th. I use the VLP all the time and found it be a fantastic way to try new things without the need for, or expense of, my own IBM POWER system. There are some drawbacks, such as not having access to your own dedicated hardware, HMC, VIO server, NIM master, but still it’s great if you just want to test something on an AIX system.

 

I’ll report back once I’ve got my AIX 7 Open Beta system up and running!

I like lsof.

 

Whenever I’m building a new AIX system I always make sure to install it. I really like the fact that I can quickly list processes that are connected to TCP and UDP ports on my system. For example, to check for the current SSH connections on my system I can run lsof and check port 22 (SSH). Immediately I have a good idea of the existing SSH sessions/connections. I can also check to see if the SSH server (sshd daemon) is running and listening (LISTEN) on my AIX partition.

 

# lsof -i tcp:22

Value of I :77   np:0

COMMAND    PID     USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME

sshd    340040     root    3u  IPv4 0xf100020002e4a3b0 0t277394  TCP aix01:ssh->172.29.131.16:49948 (ESTABLISHED)

sshd    385184 u0008904    3u  IPv4 0xf100020002e4a3b0 0t277394  TCP aix01:ssh->172.29.131.16:49948 (ESTABLISHED)

sshd    409808     root    3u  IPv4 0xf100020001c48bb0      0t0  TCP *:ssh (LISTEN)

 

But sometimes I work on systems that don’t have lsof installed. It may not be practical or appropriate for me to install it either. So I have to find another tool (or tools) that will do something similar.

 

Of course, I could use netstat to check that a server daemon was listening on a particular TCP port and view any established connections. But this doesn’t give me the associated process id’s.

 

$ netstat -a | grep -i ssh

tcp4       0      0  *.ssh                  *.*            LISTEN

tcp4       0     48  aix01.ssh      172.29.131.16.50284    ESTABLISHED

 

Fortunately, the rmsock command can provide that information. So if I wanted to find the process id for the sshd daemon that is listening on my system I’d do the following. First I need to find the socket id using netstat*.

 

# netstat -@aA | grep -i ssh | grep LIST | grep Global

Global  f1000700049303b0 tcp4       0      0  *.ssh     *.*                LISTEN

 

Then I can use rmsock to discover the process id associated with the sockect. In this case it’s PID 282700.

 

$ rmsock f1000200003e9bb0 tcpcb

The socket 0x3e9808 is being held by proccess 282700 (sshd).

 

Unlike what its name implies, rmsock does not remove the socket, if it is being used by a process. It just reports the process holding the socket. Note that the second argument of rmsock is the protocol. It's tcpcb in this example to indicate that the protocol is TCP. The results of the command are also logged to /var/adm/ras/rmsock.log.

 

# tail /var/adm/ras/rmsock.log

socket 0xf100020001c45008 held by process 434420 (writesrv) can't be removed.

socket 0xf100020000663008 held by process 418040 (java) can't be removed.

socket 0xf1000200012ad008 held by process 418040 (java) can't be removed.

socket 0xf100020000dec008 held by process 163840 (inetd) can't be removed.

socket 0xf100020000deb008 held by process 163840 (inetd) can't be removed.

socket 0xf10002000016f808 held by process 192554 (snmpdv3ne) can't be removed.

socket 0xf100020001c51808 held by process 442596 (dtlogin) can't be removed.

socket 0xf1000200012a4008 held by process 418040 (java) can't be removed.

socket 0xf100020000666008 held by process 315640 (java) can't be removed.

socket 0xf100020000deb808 held by process 163840 (inetd) can't be removed.

 

The following links have more information on the rmsock command:

 

http://www-01.ibm.com/support/docview.wss?rs=71&uid=swg21264632

http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds4/rmsock.htm

http://www.ibmsystemsmag.com/print/print.aspx?print_page=%2Faix%2Ftipstechniques%2F6666printp1.aspx&string_referer=/aix/tipstechniques/6666p1.aspx

http://aixblogs.blogspot.com/2009/03/using-netstat-and-rmsock-to-identify.html

 

 

*Note: In my example I specified the @ symbol with the netstat command. I also grep’ed for the string Global. You may have to do the same if you have WPARs running on your system. In my case I have two active WPARs who both have their own sshd process. My Global environment also has an sshd process. So in total there are three sshd daemons that I can view from the Global environment. By specifiying the @ symbol with netstat, I can quickly determine which process belongs to the Global environment and those that exist within each WPAR.

 

# netstat -aA | grep -i ssh | grep LISTEN

f1000700049303b0 tcp4       0      0  *.ssh              *.*                LISTEN

f100070000198bb0 tcp4       0      0  *.ssh              *.*                LISTEN

f100070000194bb0 tcp4       0      0  *.ssh              *.*                LISTEN

 

# netstat -@aA | grep -i ssh | grep LISTEN

Global  f1000700049303b0 tcp4       0      0  *.ssh       *.*                LISTEN

wpar1   f100070000198bb0 tcp4       0      0  *.ssh       *.*                LISTEN

wpar2   f100070000194bb0 tcp4       0      0  *.ssh       *.*                LISTEN

 

I created my first ‘versioned’ AIX 5.2 WPAR on AIX 7 today. To get me started, I followed the steps in the following guide from the AIX 7 Open Beta On-line Information Centre:

 

http://publib.boulder.ibm.com/infocenter/aix/v7r1/index.jsp?topic=/com.ibm.aix.cre/creconfig-create-wpar.htm

 

In order to create the WPAR, I needed an AIX 5.2 mksysb file to supply to the mkwpar command.

 

Fortunately, I just happened to have an old AIX 5.2 mksysb image in my archives!

 

I then executed the following command to build the WPAR:

 

# mkwpar -n wpar1 -C -B /home/cgibson/AIX5202_64bit-mksysb

 

The flags to the command are:

 

-n wparname

Specifies the name for the workload partition to be created. You must specify a name, either using the -n flag or in a specification file using the -f flag, unless the -p name or both –w and -o flags are used.

 

-B wparbackupdevice

Specifies a device containing a workload partition backup image. This image is used to populate the workload partition file systems. The wparBackupDevice parameter is a workload partition image that is created with the savewpar, mkcd, or mkdvd command. The -B flag is used by the restwpar command as part of the process of creating a workload partition from a backup image.

 

-C

Creates a versioned workload partition. This option is valid only when additional versioned workload partition software has been installed.

 

 

I was then able to start my new AIX 5.2 WPAR successfully!

 

# startwpar -v wpar1

Starting workload partition wpar1.

Mounting all workload partition file systems.

Mounting /wpars/wpar1

Mounting /wpars/wpar1/home

Mounting /wpars/wpar1/mksysb

Mounting /wpars/wpar1/nre/opt

Mounting /wpars/wpar1/nre/sbin

Mounting /wpars/wpar1/nre/usr

Mounting /wpars/wpar1/opt

Mounting /wpars/wpar1/proc

Mounting /wpars/wpar1/tmp

Mounting /wpars/wpar1/usr

Mounting /wpars/wpar1/usr/local

Mounting /wpars/wpar1/var

Mounting /wpars/wpar1/var/log

Mounting /wpars/wpar1/var/tsm/log

Loading workload partition.

Exporting workload partition devices.

Exporting workload partition kernel extensions.

Starting workload partition subsystem cor_wpar1.

0513-059 The cor_wpar1 Subsystem has been started. Subsystem PID is 8388822.

Verifying workload partition startup.

Return Status = SUCCESS.

 

The WPAR was now in an active state and the associated file systems were mounted (as shown from the Global environment).

 

# lswpar

Name   State  Type  Hostname  Directory     RootVG WPAR

--------------------------------------------------------

wpar1  A      S     wpar1     /wpars/wpar1  no

 

 

# mount | grep wpar

         /dev/lv00        /wpars/wpar1     jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /dev/lv01        /wpars/wpar1/home jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /dev/lv02        /wpars/wpar1/mksysb jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /opt             /wpars/wpar1/nre/opt namefs Jul 26 20:13 ro

         /sbin            /wpars/wpar1/nre/sbin namefs Jul 26 20:13 ro

         /usr             /wpars/wpar1/nre/usr namefs Jul 26 20:13 ro

         /dev/lv03        /wpars/wpar1/opt jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /proc            /wpars/wpar1/proc namefs Jul 26 20:13 rw

         /dev/lv04        /wpars/wpar1/tmp jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /dev/lv05        /wpars/wpar1/usr jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /dev/lv06        /wpars/wpar1/usr/local jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /dev/lv07        /wpars/wpar1/var jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /dev/lv08        /wpars/wpar1/var/log jfs    Jul 26 20:13 rw,log=/dev/loglv00

         /dev/lv09        /wpars/wpar1/var/tsm/log jfs    Jul 26 20:13 rw,log=/dev/loglv00

 

I was curious what the WPAR environment was going to look like, so I used clogin to access it and run a few commands.

 

From the Global environment I confirmed I was indeed on an AIX 7 system.

 

# uname -W

0

 

# oslevel

V7BETA

 

From within the WPAR, I confirmed that I was indeed running AIX 5.2! Wow!

 

# clogin wpar1

wpar1 : / # oslevel

5.2.0.0

 

And I could see all 8 logical CPUs (4 hardware threads per POWER7 processor i.e. SMT-4).

 

wpar1 : / # sar -P ALL 1 5

 

AIX wpar1 2 5 00F602734C00    07/26/10

 

wpar1 configuration: @lcpu=8  @mem=4096MB  @ent=0.50

 

20:22:20 cpu    %usr    %sys    %wio   %idle   physc   %entc

20:22:21  0        7      78       1      14    0.01     0.0

          1        1      70       0      29    0.01     0.0

          2        0       1       0      99    0.00     0.0

          3        0       0       0     100    0.01     0.0

          4        0      35       0      65    0.00     0.0

          7        0      28       0      72    0.00     0.0

          U        -       -       0      93    0.47    93.9

          -        0       3       0      97    0.03     0.0

 

I noticed an interesting device in the lscfg output.

 

wpar1 : / # lscfg

INSTALLED RESOURCE LIST

 

The following resources are installed on the machine.

+/- = Added or deleted from Resource List.

*   = Diagnostic support not available.

 

  Model Architecture: chrp

  Model Implementation: Multiple Processor, PCI bus

 

+ sys0               System Object

* wio0               WPAR I/O Subsystem

 

Also noticed some new and interesting mount points, for example /nre/opt.

 

wpar1 : / # df

Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on

Global            131072     99928   24%     1424     5% /

Global            131072    126704    4%       70     1% /home

Global           1048576   1015560    4%       17     1% /mksysb

Global            786432    428904   46%     7331    14% /nre/opt

Global            458752     88400   81%    10020    47% /nre/sbin

Global           4980736     24872  100%    53698    87% /nre/usr

Global            131072     63800   52%     1640    11% /opt

Global                 -         -    -         -     -  /proc

Global            131072    125080    5%       52     1% /tmp

Global           1572864    165744   90%    23183    12% /usr

Global            524288    494464    6%      154     1% /usr/local

Global            131072    111512   15%      493     4% /var

Global            262144    253744    4%       28     1% /var/log

Global            131072    126832    4%       20     1% /var/tsm/log

 

 

I did have one minor problem when I first tried to start my WPAR, but that issue was quickly resolved by the AIX developers on the AIX 7 Open Beta Forum.

 

https://www.ibm.com/developerworks/forums/thread.jspa?threadID=335415&tstart=0

 

Such a fast response to my query demonstrates IBMs commitment to the Open Beta program.

 

I was notified this morning that I’ve been selected to work on an upcoming AIX Redbook in Austin, Texas. I’m really looking forward to it.

 

AIX Differences Guide, PW-0001-R01: http://www.redbooks.ibm.com/residents.nsf/50da6a28780ffa688525701b004a4f21/9032e219ecda947c852576f900691af5?OpenDocument

 

Follow me on Twitter: http://twitter.com/cgibbo

 

IBM made some announcements today relating to their latest POWER7 server offerings. The new line of systems includes new entry level systems and the highly anticipated high-end system, the POWER7 795!  They also officially outlined some of the new features available in AIX 7.1. You can review the details here. I’ve discussed some of these new features here and here. The official AIX 7.1 announcement details are available here.

 

The announcement got me thinking about my recent customer engagements and why some have chosen to deploy AIX into their IBM POWER environments, while others are considering a Linux on POWER solution.

 

I’ve found that it usually comes down to a skills decision more than anything else. Most customers are happy to either continue working with AIX (if they are existing AIX users) or migrate from another UNIX OS to AIX. I’ve seen very few customers actually migrate to Linux on POWER, but I’ve worked with several that have seriously considered it. Those that have chosen to deploy Linux are doing so purely because they have in-house Linux skills. They are concerned that migrating to AIX may be too big a jump for their technical staff. I find this thinking interesting, as most of the customers I’ve dealt with who run other UNIX OS’s like Tru64, Solaris or HP-UX are more than happy to migrate to AIX. They believe the move is relatively minor and doesn’t require massive re-training of their UNIX admins. I tend to agree.

 

For me, AIX is my preferred “Enterprise class” UNIX Operating System. Notice I’m prefacing this with the words Enterprise class. Don’t get me wrong, I have worked with Linux systems in both small and large customer environments. It is a great OS. But I’ve found that it really only fits into environments that have a relatively small number of users and where significant downtime can be tolerated for things like operating system maintenance. This doesn’t fit the Enterprise class of UNIX server OS’s that I’m thinking of here. When I contemplate the word Enterprise, I think of servers and operating systems that can respond to business demands in terms of performance, reliability, stability and availability. An Enterprise UNIX can provide all of these things without compromise. Linux can offer performance and reliability (in my opinion). However, from what I’ve seen, it lacks features & functions in the areas of stability and availability. AIX on the other hand ticks all the boxes. Again, this is just my opinion based on my experiences with both AIX and Linux in the Enterprise landscape. Others will no doubt have their own experiences that may or may not match my own.

 

So when I’m designing an Enterprise UNIX server environment for a customer, I always start with an AIX on POWER base. If the customer wants Linux, sure I can look at that too, but I strongly recommend AIX as my preferred choice for large systems. Most of my customers are running relatively large SAP/Oracle systems. AIX on POWER is a great combination for large Enterprise systems. If you need to deploy large database systems that must service tens of thousands of users (like a big SAP system), then I believe AIX is the perfect OS on which to provide a platform for these large scale systems.

 

AIX is a very mature and powerful UNIX OS. It has been a major player in the UNIX server market for over 20 years (as shown below). Some people are just not aware of how mature, robust and stable the AIX OS has become over the years. There are many impressive aspects of the OS in the areas of performance, scalability, reliability, management and administration.

 

 

Just looking at some of the administration capabilities built into AIX are enough for me to always recommend AIX over Linux (or any other UNIX OS), when it comes to large Enterprise servers.

 

For example, the System Management Interface Tool (SMIT) can make the UNIX admins life a lot simpler. This is an interactive tool that is part of the AIX OS. Almost all tasks that an AIX administrator may need to perform can be executed using this tool. It is a text-based tool (there is also an X interface but I recommend sticking with the text-based menus). Everything it does, it does through standard AIX commands and Korn shell functions. This feature is especially useful when you need to automate a repetitive task; you can have SMIT create the proper command-line sequence, and you can then use those commands in your own script. My compatriot, Anthony English, has a nice intro to SMIT on his AIX blog.

 

The AIX Logical Volume Manager (LVM) is built into the OS, for free. AIX LVM helps UNIX system administrators manage their storage in a very flexible manner. LVM allows logical volumes to span multiple physical volumes. Data on logical volumes appears to be contiguous to the user, but might not be contiguous on the physical volume. This allows file systems, paging space, and other logical volumes to be resized or relocated, span multiple physical volumes, and have their contents replicated for greater flexibility and availability. It provides capabilities for mirroring data across disks, migrating data across disks & storage subsystems, expand/shrink filesystems and more.......all of which can be performed dynamically.....no downtime required. The concept, implementation and interface to the AIX LVM is one of a kind. All of its features support the ‘continuous availability’ philosophy.

 

One of the biggest reasons that I love AIX over Linux is the mksysb. It’s built into the OS and allows you to create a bootable image of your AIX system. This image can be used to restore a broken AIX system or for cloning other systems. The cloning feature is truly amazing. You can take an image created on a low-end system and deploy it on any POWER system, all the way up to the high-end POWER boxes. This simplifies the installation and cloning processing when you need to install and manage many AIX LPARs. By using an SOE mksysb image you can deploy consistent AIX images across your Enterprise POWER server environment.

 

This brings me to another wonderful feature of AIX, the Network Installation Manager (NIM). NIM is powerful network installation tool (comparable to Linux Kickstart). Using NIM you can backup/restore, update and upgrade one or more AIX systems either individually or simultaneously. This can all be achieved over a network connection, removing the need for handling physical installation media forever.

 

Another fine example of AIXs superior OS management tools is multibos. This tool allows an AIX administrator to create and maintain two separate, bootable instances of the AIX OS within the same root volume group (rootvg). This second instance of rootvg is known as a standby Base Operating System (BOS) and is an extremely handy tool for performing AIX TL and Service Pack (SP) updates. Multibos lets you install, update and customize a standby instance of the AIX OS without impacting the running and active production instance of the AIX OS. This is valuable in environments with tight maintenance windows.

 

When it comes to upgrading the OS to a new release of AIX, the nimadm utility can assist the administrator greatly in this task. The nimadm utility offers several advantages. For example, a system administrator can use nimadm to create a copy of a NIM client's rootvg and migrate the disk to a newer version or release of AIX. All of this can be done without disruption to the client (there is no outage required to perform the migration). After the migration is finished, the only downtime required will be a single scheduled reboot of the system.

 

AIX 6.1 introduced new capability that most UNIX operating systems are still working on. Concurrent Updates of the AIX Kernel.....without a reboot! IBM is always working hard at making AIX an OS that can provide continuous availability, even if it needs to be patched. AIX now has the ability to update certain kernel components and kernel extensions in place, without needing a system reboot. In addition, concurrent updates can be removed from the system without needing a reboot. Can you do this with other UNIX OSs?

 

And that’s just some of the features that make AIX the only UNIX OS that I recommend for Enterprise systems. There are many more tools and features that I couldn’t live without (like alt_disk_install, savevg, installp, WPARs, etc, the list goes on). If you are new to AIX and you are considering what your next UNIX OS should be, then I recommend you take a very close look at AIX.

 

And finally, the support provided by IBM, is first class. Whenever I’ve needed assistance with an AIX issue or query, I have always received timely, professional and useful advice. On the rare occasions where I’ve uncovered a new bug, IBM AIX support have always been quick to provide me with an interim fix to resolve or workaround a problem. That’s the sort of support you’d expect for an Enterprise UNIX OS, isn’t it? What’s the support like from your current UNIX (or Linux) OS vendor?

 

Linux is still a viable UNIX operating system. However, I think it’s more suited to certain workloads like small to medium size mail, web and other utility servers and services. AIX, however, would be my platform of choice for my 10TB Oracle database running SAP ERP, not just for performance reasons, but primarily because of the system administration features of AIX that allow me to support and manage the system without impacting my customers or enforcing reboots/outages whenever I need to change something on the system.

 

All IBM need to do is create their own Linux distribution (Blue Linux perhaps?) that has all the features of AIX built in and then I’m sold! But why would they? We already have AIX.

 

Follow me on Twitter: http://twitter.com/cgibbo

 

I remember reading this interesting document about some challenges renaming AIX devices.

 

http://gibsonnet.net/aix/ibm/The_Fun_of_Oracle_RAC_ASM_and_Device_Names.doc

 

This poor guy had to jump through some hoops to rename some disk and network devices in a clustered AIX (Oracle RAC) environment.

 

This comment stayed with me:

 

“The easy way to get network adapters and disk names to match between systems would be to run something like

 

# chdev –l old_name –n new_name

 

Except, this command option does NOT exist.”

 

Since then I’ve always been on the “look out” for an easy way to rename devices in AIX.

 

In particular in large clustered environments where it is sometimes very important to have the same disk and network device names in sync across all nodes in a cluster. And besides, it’s a lot easier  to verify a cluster configuration if the hdisk names are all the same. Matching PVIDs works but it requires a lot more effort! For example, knowing that hdisk123 is the same device on all nodes makes life easier than scanning lspv output for a PVID like 00f6048868b4gead.  Of course you can script things to make this easier but it would be great if you didn’t need to do this and that there was a way to rename devices as needed, without resorting to unsupported methods.

 

Well, this is no longer an issue for AIX.

 

Starting with AIX 7.1, you can now easily rename devices. A new command called rendev was introduced to allow AIX administrators to rename devices as required.

 

From the man page:

 

The rendev command enables devices to be renamed. The device to be renamed, is specified with the -l flag, and the new desired name is specified with the -n flag.

 

The new desired name must not exceed 15 characters in length. If the name has already been used or is present in the /dev directory, the operation fails. If the name formed by appending  the new name after the character r is already used as a device name, or appears in the /dev directory, the operation fails.

 

 If the device is in the Available state, the rendev command must unconfigure the device before renaming it. This is similar to the operation performed by the rmdev -l Name command. If the unconfigure operation fails, the renaming will also fail. If the unconfigure succeeds, the rendev command will configure the device, after renaming it, to restore it to the Available state. The -u flag may be used to prevent the device from being configured again after it is renamed.

 

 Some devices may have special requirements on their names in order for other devices or applications to use them. Using the rendev command to rename such a device may result in the device being unusable. Note: To protect the configuration database, the rendev command cannot be interrupted once it has started. Trying to stop this command before completion, could result in a corrupted database.

 

This functionality was required to help simplify cluster device configuration with the new Cluster Aware AIX (CAA) feature in AIX 7.1.

 

I’ll be blogging about CAA in a future post.

 

Here are some examples of using the rendev command on AIX 7.1 (GA) system. In the first example I will rename hdisk3 to hdisk300. Note: hdisk3 is not in use (busy).

If the disk had been allocated to a volume group, I would have needed to varyoff the volume group first.

 

 

# lspv

hdisk0          00f61ab2f73e46e2                    rootvg          active

hdisk1          00f61ab20bf28ac6                    None

hdisk2          00f61ab2202f7c0b                    None

hdisk4          00f61ab20b97190d                    None

hdisk3          00f61ab2202f93ab                    None

 

# rendev -l hdisk3 -n hdisk300

 

# lspv

hdisk0          00f61ab2f73e46e2                    rootvg          active

hdisk1          00f61ab20bf28ac6                    None

hdisk2          00f61ab2202f7c0b                    None

hdisk4          00f61ab20b97190d                    None

hdisk300        00f61ab2202f93ab                    None

 

Easy!

 

Next, I’ll rename a virtual SCSI adapter. I renamed vscsi0 to vscsi2. Note: I placed the adapter, vscsi0, in a Defined state before renaming the device.

 

# rmdev -Rl vscsi0

 

# lsdev -Cc adapter

ent0   Available  Virtual I/O Ethernet Adapter (l-lan)

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi0 Defined    Virtual SCSI Client Adapter

vscsi1 Available  Virtual SCSI Client Adapter

 

# rendev -l vscsi0 -n vscsi2

 

# lsdev -Cc adapter

ent0   Available  Virtual I/O Ethernet Adapter (l-lan)

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi1 Available  Virtual SCSI Client Adapter

vscsi2 Defined    Virtual SCSI Client Adapter

 

Now I’ll rename a network adapter from ent0 to ent10. I bring down the interface before changing the device name

 

# lsdev -Cc adapter

ent0   Available  Virtual I/O Ethernet Adapter (l-lan)

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi1 Available  Virtual SCSI Client Adapter

vscsi2 Defined    Virtual SCSI Client Adapter

 

# ifconfig en0

en0: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>

        inet 10.1.20.19 netmask 0xffff0000 broadcast 10.153.255.255

         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

 

# ifconfig en0 down detach

 

# rendev -l ent0 -n ent10

 

# lsdev -Cc adapter

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

ent10  Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi1 Available  Virtual SCSI Client Adapter

vscsi2 Defined    Virtual SCSI Client Adapter

 

# rendev -l en0 -n en10

 

# chdev -l en10 -a state=up

en10 changed

# mkdev -l inet0

inet0 Available

# ifconfig en10

en10: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>

        inet 10.1.20.19 netmask 0xffff0000 broadcast 10.153.255.255

         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

 

If you want to be creative you can rename devices to anything you like (as long as it’s not more than 15 characters). For example I’ll rename vscsi2 to myvscsiadapter.

 

 

# rendev -l vscsi2 -n myvscsiadapter

# lsdev -Cc adapter

ent1           Available  Virtual I/O Ethernet Adapter (l-lan)

myadapter      Available  Virtual I/O Ethernet Adapter (l-lan)

myvscsiadapter Defined    Virtual SCSI Client Adapter

vsa0           Available  LPAR Virtual Serial Adapter

vscsi1         Available  Virtual SCSI Client Adapter

 

And in the last example I’ll demonstrate changing virtual SCSI adapter device names on a live system.

 

This is single disk system (hdisk0), with two vscsi adapters.

 

# lspv

hdisk0          00f6048868b4deee                    rootvg          active

 

# lspath

Enabled hdisk0 vscsi0

Enabled hdisk0 vscsi1

 

# lsdev -Cc adapter

ent0   Available  Virtual I/O Ethernet Adapter (l-lan)

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi0 Available  Virtual SCSI Client Adapter

vscsi1 Available  Virtual SCSI Client Adapter

 

We ensure the adapter is in a Defined state before renaming it. This will fail otherwise.

 

# rmdev -Rl vscsi1

vscsi1 Defined

# lsdev -Cc adapter | grep vscsi

vscsi0 Available  Virtual SCSI Client Adapter

vscsi1 Defined    Virtual SCSI Client Adapter

 

Now we rename the adapter vscsi1 to vscsi3.

 

# rendev -l vscsi1 -n vscsi3

 

# lsdev -Cc adapter | grep vscsi

vscsi0 Available  Virtual SCSI Client Adapter

vscsi3 Defined    Virtual SCSI Client Adapter

 

That was easy enough. Now I need to bring the adapter and path online with cfgmgr. The lspath output displays an additional path to vscsi3.

 

# lspath

Enabled hdisk0 vscsi0

Defined hdisk0 vscsi1

 

# cfgmgr

Method error (/etc/methods/cfgscsidisk -l hdisk0 ):

        0514-082 The requested function could only be performed for some

                 of the specified paths.

 

# lspath

Enabled hdisk0 vscsi0

Defined hdisk0 vscsi1

Enabled hdisk0 vscsi3

 

Now I need to remove the old path to vscsi1. The path to vscsi3 is now Enabled. The adapter, vscsi3, is in an Available state. All is good.

 

# rmpath -l hdisk0 -p vscsi1 -d

path Deleted

 

# lspath

Enabled hdisk0 vscsi0

Enabled hdisk0 vscsi3

 

# lsdev -Cc adapter | grep vscsi

vscsi0 Available  Virtual SCSI Client Adapter

vscsi3 Available  Virtual SCSI Client Adapter

 

The same steps need to be repeated for the vscsi0 adapter. This is renamed to vscsi2.

 

# rmdev -Rl vscsi0

vscsi0 Defined

# lsdev -Cc adapter | grep vscsi

vscsi0 Defined    Virtual SCSI Client Adapter

vscsi3 Available  Virtual SCSI Client Adapter

 

 

# rendev -l vscsi0 -n vscsi2

 

# lsdev -Cc adapter | grep vscsi

vscsi2 Defined    Virtual SCSI Client Adapter

vscsi3 Available  Virtual SCSI Client Adapter

 

 

# lspath

Defined hdisk0 vscsi0

Enabled hdisk0 vscsi3

 

# cfgmgr

Method error (/etc/methods/cfgscsidisk -l hdisk0 ):

        0514-082 The requested function could only be performed for some

                 of the specified paths.

 

# lspath

Defined hdisk0 vscsi0

Enabled hdisk0 vscsi2

Enabled hdisk0 vscsi3

 

# rmpath -l hdisk0 -p vscsi0 -d

path Deleted

 

# cfgmgr

# lspath

Enabled hdisk0 vscsi2

Enabled hdisk0 vscsi3

 

That’s it. Both adapters have been renamed while the system was in use. No downtime required.

 

# lsdev -Cc adapter | grep vscsi

vscsi2 Available  Virtual SCSI Client Adapter

vscsi3 Available  Virtual SCSI Client Adapter

 

# lspath

Enabled hdisk0 vscsi2

Enabled hdisk0 vscsi3

 

Please refer to the AIX 7.1 command reference for more information on this new command:

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.cmds/doc/aixcmds4/rendev.htm

 

And by the way, here’s a draft of what I’ve been working on for the last 5 weeks:

 

IBM AIX Version 7.1 Differences Guide:

http://www.redbooks.ibm.com/Redbooks.nsf/RedpieceAbstracts/sg247910.html?Open

 

Follow me on Twitter:

http://twitter.com/cgibbo

 

I just had to share this entertaining presentation with everyone.

 

http://gibsonnet.net/AIX/ibm/Journey_in_VLAN_world.pps

 

It was written by Bernard Lemal. He works for IBM GTS Belgium in AIX support.

 

It’s a humorous look at the life of an IP packet and its journey in the world of VLANs, VLAN tagging and Virtual I/O.

 

Plenty of interesting and informative commentary along the way.

 

The part of the IP packet is portrayed by Bernard himself. Take a look at the presentation to see what I mean!

 

I wish I’d seen a presentation like this when I was first introduced to the concept of VLAN tagging, it might have answered several questions early on.

 

Enjoy!

Interested in learning more about the next generation of Virtual I/O Server? Yes? Then take a look at this presentation.

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/resource/VM19.pdf

 

The following diagram gives you a high-level overview of the classic VIOS compared with the NextGen VIOS. Should be enough to spark your interest!

Happy reading!

 

 

 

A colleague of mine contacted me during the week to ask how one could determine if a POWER7 system was capable of Active Memory Expansion. I thought I'd share my response with everyone who follows my blog.

 

You can check if the system is capable of providing Active Memory Expansion (AME), from the HMC command line using the lssyscfg command, as shown here:
 
hscroot@hmc1:~> lssyscfg -r sys -m Server-8233-E8B-SN1000 -F active_mem_expansion_capable
1

Alternatively you can view the capabilities of the managed system from the HMC GUI, under System properties/Capabilities, as shown in the following image. 

 
 

Once you’ve concluded that your POWER7 system is AME capable, the next step is to check if AME is enabled or disabled for an LPAR. This is easily done from the AIX command line with the lparstat and/or amepat commands, as shown in the following example:
 
;Running lparstat and amepat on a LPAR with AME disabled.

 

$ lparstat -i| grep Exp
Target Memory Expansion Factor             : -
Target Memory Expansion Size               : -
 
$ amepat | grep Expan
WARNING: Running in no modeling mode.
Active Memory Expansion        : Disabled

 

;Running lparstat and amepat on a LPAR with AME enabled.

 

$ lparstat -i | grep -i exp

Memory Mode                                : Dedicated-Expanded

Target Memory Expansion Factor             : 1.00

Target Memory Expansion Size               : 6144 MB

 

$ amepat | grep -i exp

WARNING: Running in no modeling mode.

Active Memory Expansion        : Enabled

Target Expanded Memory Size    : 6.00 GB

Target Memory Expansion factor : 1.00
 
Note that I deliberately ran these commands as a non-root user to highlight the fact that you don’t need root access to ascertain whether or not AME is active on an LPAR.

 

By the way, did you know that AME is now supported in SAP production environments running on AIX and POWER7? Both SAP application servers, as well as database servers running DB2 LUW are supported. Unfortunately there’s no support for LPARs with Oracle RDBMS at this stage. Thanks to George Manousos from IBM Australia for providing me with this update.

Please refer to the following SAP note.

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/resource/1464605%28Power7AME%29.pdf

 

The SAP note also states:

 

Specific tests have been successfully executed with SAP ERP 6.0 (ABAP)

workload. The results of those test can be found in the following whitepaper:

 

ftp://ftp.software.ibm.com/common/ssi/sa/wh/n/pow03038usen/POW03038USEN.PDF

 

Restrictions

As AME is not recommended with large page support, AME will disable AIX

64KB pages by default.

 

AME monitoring capability in CCMS

Monitoring capabilities are available with saposcol v12.46 for more details see SAP note 710975.

 

 

It looks as though SAP have been quick to support AME on AIX/POWER7. This is a great benefit to SAP customers running with PowerVM on the IBM POWER platform. The following screenshot shows the SAP CCMS with AME statistics being reported.

 

PowerVM Integration into SAP CCMS Monitoring expanded to include Virtualized Memory Metrics

The SAP AIX porting team continues in further improving the integration of PowerVM into SAP system monitoring. While in the past already most processor and AIX virtualization metrics could be monitored via CCMS, now POWER7 Active Memory Expansion (AME)  has been included.  Customers will find a new memory section in the respective CCMS-panel as depicted in this screen shot.

For those who are not familiar with what AME actually is and how it can benefit you, I suggest you take a look at the IBM AME Wiki site first:

 

http://www.ibm.com/developerworks/wikis/display/WikiPtype/IBM+Active+Memory+Expansion

 

And watch Nigel Griffiths AME movie:

 

http://www.ibm.com/developerworks/wikis/display/wikiptype/movies#Movies-ame

 

 

 

 

I received an email this week from a colleague that worked with me on the NIM Redbook back in 2006. He was experiencing an issue with DSM and NIM. He was attempting to use the dgetmacs command to obtain the MAC address of the network adapters on an LPAR. The command was failing to return the right information.

 

This was the output from his attempt with dgetmacs:

 

# dgetmacs -n lpar1

Using an adapter type of "ent".

Could not dsh to node lpar1.

Attempting to use openfirmware method to collect MAC addresses.

Acquiring adapter information from Open Firmware for node lpar1.

dgetmacs: 2760-185 Issuing the command "/opt/ibm/sysmgt/dsm/dsmbin/lpar_netboot -M -n -t ent -J hmc1 -j hmc -F /etc/ibm/sysmgt/dsm/config/my_password_file –A 4 10582FP 8202-E4B" gave a return code of 1.  The routine will continue.

No adapters found for node lpar1.

Could not get mac address for this aix node "lpar1"

 

I experienced this very issue during the writing of the AIX 7.1 Differences Guide Redbook. And given that I was in Austin, sitting in the same building as the AIX development team, I was able to speak with the developers directly about the issue. At that time they provided me with the following workaround.

 

First they asked me to check the size of the /usr/lib/nls/msg/en_US/IBMhsc.netboot.cat message catalog file.

 

# ls –l /usr/lib/nls/msg/en_US/IBMhsc.netboot.cat

-rw-r--r--    1 bin      bin            3905 Aug 08 09:54

 

They were surprised to find that the file appeared to be “too small”. They promptly sent me the catalog file from one of their development AIX 7.1 systems.  I replaced the file as follows:

 

# cd  /usr/lib/nls/msg/en_US/

# ls -ltr IBMhsc*

-rw-r--r--    1 bin      bin            3905 Aug 08 09:54 IBMhsc.netboot.cat

# cp -p IBMhsc.netboot.cat IBMhsc.netboot.cat.old

# cp /tmp/lpar1/IBMhsc.netboot.cat.new IBMhsc.netboot.cat

# ls -ltr IBMhsc*

-rw-r--r--    1 bin      bin         3905 Aug 08 09:54 IBMhsc.netboot.cat.old

-rw-r--r--    1 bin      bin        26374 Dec 23 11:24 IBMhsc.netboot.cat

 

This fixed the problem for me during the residency.

 

So I asked my friend to do the same (after I sent him the message catalog file). He ran the dgetmacs command again and this time it returned the MAC address for all the network adapters in his LPAR. Success!

 

# dgetmacs -n lpar1

Using an adapter type of "ent".

Could not dsh to node lpar1.

Attempting to use openfirmware method to collect MAC addresses.

Acquiring adapter information from Open Firmware for node lpar1.

 

# Node::adapter_type::interface_name::MAC_address::location::media_speed::adapter_duplex::UNUSED::install_gateway::ping_status::machine_type::netaddr::subnet_mask

 

lpar1::ent_v::::4A7009608816::U8202.E4B.10582FP-V4-C22-T1::auto::auto::::::n/a::secondary::::

 

lpar1::ent_v::::4A7009608818::U8202.E4B.10582FP-V4-C24-T1::auto::auto::::::n/a::secondary::::

 

This then allowed him to network boot his LPAR and install AIX 7.1 via NIM.

 

# nim -o bos_inst -a bosinst_data=noprompt -a source=rte -a installp_flags=agX -a accept_licenses=yes -a spot=spotaix7tl0sp2 -a lpp_source=aix7tl0sp2 lpar1

dnetboot Status: Invoking /opt/ibm/sysmgt/dsm/dsmbin/lpar_netboot lpar1

dnetboot Status: Was successful network booting node lpar1.

 

I thought this might have been resolved in the latest service pack for AIX 7.1. However, both of us have the latest service pack installed (i.e. TL0 SP2 for AIX 7.1).

 

# oslevel -s

7100-00-02-1041

 

If anyone encounters this problem, feel free to contact me (aix@gibsonnet.net).

 

For more information, please refer to section 5.2.7 Using DSM and NIM, in the AIX 7.1 Differences Guide Redbook:

 

http://www.redbooks.ibm.com/Redbooks.nsf/RedpieceAbstracts/sg247910.html?Open

 

 

Last year I wrote an article on using nimadm to migrate to AIX 6.1.

The procedures outlined how to migrate a live system with nimadm and a spare disk.

Another method that I have found to be extremely useful is the migration of mksysb images from one version of AIX to another using (you guessed it) nimadm.

In the following example I migrated an AIX 6.1 mksysb image to AIX 7.1 and then used that image to install a new LPAR via NIM.

The source LPAR was running AIX 6.1 TL6 SP3. I took a mksysb of this system to my NIM master.

# oslevel -s

6100-06-03-1048

#

 

The NIM mksysb resource was named cg-aix61

 

root@nim1 / # lsnim -t mksysb

vio1-mksysb     resources       mksysb

cg-aix61        resources       mksysb

 

root@nim1 / # lsnim -l cg-aix61

cg-aix61:

   class         = resources

   type          = mksysb

   arch          = power

   Rstate        = ready for use

   prev_state    = unavailable for use

   location      = /export/mksysb/cg-aix61

   version       = 6

   release       = 1

   mod           = 6

   oslevel_r     = 6100-06

   alloc_count   = 0

   server        = master

   creation_date = Thu Jan 6 10:00:22 2011

   source_image  = cg

root@nim1 / #

 

To migrate the mksysb image from AIX 6.1 to AIX 7.1, I ran the following nimadm command:

 

root@nim1 / # nimadm -T cg-aix61 -O /export/mksysb/cg-aix71 -s spotaix7tl0sp2 -l aix7tl0sp2 -j nimadmvg -Y -N cg-aix71

 

The flags indicate the following:

 

-T specifies the existing AIX 6.1 NIM mksysb resource.

-O specifies the output location for the migrated mksysb resource.

-s indicates the AIX 7.1 NIM SPOT resource for the migration.

-l indicates the AIX 7.1 NIM lpp_source resource for the migration.

-j identifies the volume group that will be used on the NIM master to create file systems.

-Y Agrees to required software license agreements for software to be installed.

-N specifies the name of the new AIX 7.1 NIM mksysb resource to be created after migration.

 

Before running this command I made sure that I had installed the bos.alt_disk_install.rte fileset into my AIX 7.1 SPOT. If this fileset was not installed I would have received the following message immediately after issuing the nimadm command:

 

0042-001 nim: processing error encountered on "master":

   /usr/bin/lslpp: Fileset bos.alt_disk_install.rte not installed.

 

0505-204 nimadm: SPOT spotaix7tl0sp2 does not have bos.alt_disk_install.rte installed.

0505-205 nimadm: The level of bos.alt_disk_install.rte installed in SPOT

spotaix7tl0sp2 (0.0.0.0) does not match the NIM master's level (7.1.0.2).

Cleaning up alt_disk_migration on the NIM master.

 

I also had an AIX 7.1 lpp_source and SPOT ready and waiting on my NIM master.

 

root@nim1 / # lsnim -t lpp_source

aix61tl6sp3     resources       lpp_source

openssh         resources       lpp_source

aix7tl0sp2      resources       lpp_source

 

root@nim1 / # lsnim -t spot

vio1-spot           resources       spot

spotaix61tl6sp3     resources       spot

spotaix7tl0sp2      resources       spot

 

OK, back to my nimadm operation. Upon initiating the migration I saw the following output on my screen (click here for a copy of the complete log file):

 

root@nim1 / # nimadm -T cg-aix61 -O /export/mksysb/cg-aix71 -s spotaix7tl0sp2 -l aix7tl0sp2 -j nimadmvg -Y -N cg-aix71

Initializing the NIM master.

Verifying alt_disk_migration eligibility.

Initializing log: /var/adm/ras/alt_mig/cg-aix61_alt_mig.log

Starting Alternate Disk Migration.

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 1.

+-----------------------------------------------------------------------------+

Processing target mksysb resource cg-aix61.

Restoring /image.data from mksysb image.

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 2.

+-----------------------------------------------------------------------------+

Creating nimadm cache file systems on volume group nimadmvg.

Checking for initial required migration space.

Creating cache file system /cg-aix61_mm_alt/alt_inst

Creating cache file system /cg-aix61_mm_alt/alt_inst/admin

Creating cache file system /cg-aix61_mm_alt/alt_inst/home

Creating cache file system /cg-aix61_mm_alt/alt_inst/opt

Creating cache file system /cg-aix61_mm_alt/alt_inst/tmp

Creating cache file system /cg-aix61_mm_alt/alt_inst/usr

Creating cache file system /cg-aix61_mm_alt/alt_inst/var

Creating cache file system /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 3.

+-----------------------------------------------------------------------------+

Syncing mksysb data to cache ...

 

.....etc......

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 8.

+-----------------------------------------------------------------------------+

Creating client boot image.

bosboot: Boot image is 45709 512 byte blocks.

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 9.

+-----------------------------------------------------------------------------+

Adjusting client file system sizes ...

Adjusting size for /

Adjusting size for /admin

Adjusting size for /home

Adjusting size for /opt

Adjusting size for /tmp

Adjusting size for /usr

Adjusting size for /var

Adjusting size for /var/adm/ras/livedump

Backing up cache data to mksysb file /export/mksysb/cg-aix71 ...

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 10.

+-----------------------------------------------------------------------------+

Unmounting client mounts on the NIM master.

forced unmount of /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

forced unmount of /cg-aix61_mm_alt/alt_inst/var

forced unmount of /cg-aix61_mm_alt/alt_inst/usr

forced unmount of /cg-aix61_mm_alt/alt_inst/tmp

forced unmount of /cg-aix61_mm_alt/alt_inst/opt

forced unmount of /cg-aix61_mm_alt/alt_inst/home

forced unmount of /cg-aix61_mm_alt/alt_inst/admin

forced unmount of /cg-aix61_mm_alt/alt_inst

Removing nimadm cache file systems.

Removing cache file system /cg-aix61_mm_alt/alt_inst

Removing cache file system /cg-aix61_mm_alt/alt_inst/admin

Removing cache file system /cg-aix61_mm_alt/alt_inst/home

Removing cache file system /cg-aix61_mm_alt/alt_inst/opt

Removing cache file system /cg-aix61_mm_alt/alt_inst/tmp

Removing cache file system /cg-aix61_mm_alt/alt_inst/usr

Removing cache file system /cg-aix61_mm_alt/alt_inst/var

Removing cache file system /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 11.

+-----------------------------------------------------------------------------+

Defining NIM mksysb resource ...

New NIMmksysb resource name is cg-aix71

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 12.

+-----------------------------------------------------------------------------+

Cleaning up alt_disk_migration on the NIM master.

root@nim1 / #

 

For a description of all the phases in a nimadm operation, I suggest you refer to my previous nimadm article or the AIX infocenter (links below).

 

Once the migration had completed, I found a new mksysb file in /export/mksysb (as expected). The file was named cg-aix71. And a new NIM mksysb resource was now available.

 

root@nim1 / # ls -ltr /export/mksysb/

total 30793928

drwxr-xr-x    2 root     system          256 Dec 21 10:40 lost+found

-rw-r--r--    1 root     system   4893952000 Dec 21 14:42 vio1-mksysb

-rw-r--r--    1 root     system   2323097600 Jan 06 10:00 cg-aix61

-rw-r--r--    1 root     system   1967411200 Jan 06 11:00 cg-aix71

 

root@nim1 / # lsnim -t mksysb

vio1-mksysb     resources       mksysb

cg-aix61        resources       mksysb

cg1-mksysb      resources       mksysb

cg-aix71        resources       mksysb

 

root@nim1 / # lsnim -l cg-aix71

cg-aix71:

   class         = resources

   type          = mksysb

   comments      = Created by nimadm on Thu Jan 6 11:00:51 EETDT 2011

   Rstate        = ready for use

   prev_state    = unavailable for use

   location      = /export/mksysb/cg-aix71

   version       = 7

   release       = 1

   mod           = 0

   oslevel_r     = 7100-00

   alloc_count   = 0

   server        = master

   creation_date = Thu Jan 6 11:00:54 2011

 

During the migration I noticed the following file systems were mounted on the NIM master.

 

root@nim1 / # df

Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4         1572864   1149640   27%    10471     8% /

/dev/hd2         5242880   1597696   70%    42812    19% /usr

/dev/hd9var      1048576    250456   77%     8260    23% /var

/dev/hd3         8912896   8287752    8%      111     1% /tmp

/dev/hd1          524288    523520    1%        5     1% /home

/dev/hd11admin     524288    523488    1%        5     1% /admin

/proc                  -         -    -         -     -  /proc

/dev/hd10opt     1048576    664176   37%     7262     9% /opt

/dev/livedump     524288    523552    1%        4     1% /var/adm/ras/livedump

/dev/tftpbootlv     524288    419656   20%       10     1% /tftpboot

/dev/exportlv     524288    523408    1%       13     1% /export

/dev/lppsrclv   25690112   4383632   83%     3114     1% /export/lppsource

/dev/spotlv      8912896   4848464   46%    43450     8% /export/spot

/dev/mksysblv   38273024  11315200   71%        9     1% /export/mksysb

/dev/lv00         425984    412432    4%       25     1% /cg-aix61_mm_alt/alt_inst

/dev/lv01          65536     63376    4%       18     1% /cg-aix61_mm_alt/alt_inst/admin

/dev/lv02          65536     63384    4%       17     1% /cg-aix61_mm_alt/alt_inst/home

/dev/lv03         557056    414720   26%      201     1% /cg-aix61_mm_alt/alt_inst/opt

/dev/lv04         294912    285184    4%       21     1% /cg-aix61_mm_alt/alt_inst/tmp

/dev/lv05        4653056   4506904    4%       17     1% /cg-aix61_mm_alt/alt_inst/usr

/dev/lv06         720896    698160    4%       20     1% /cg-aix61_mm_alt/alt_inst/var

/dev/lv07          65536     63384    4%       17     1% /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

 

And in my nimadmvg volume group, there were several new logical volumes.

 

root@nim1 / # lsvg -l nimadmvg

nimadmvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

loglv02             jfslog     1       1       1    open/syncd    N/A

lv00                jfs        13      13      1    open/syncd    /cg-aix61_mm_alt/alt_inst

lv01                jfs        2       2       1    open/syncd    /cg-aix61_mm_alt/alt_inst/admin

lv02                jfs        2       2       1    open/syncd    /cg-aix61_mm_alt/alt_inst/home

lv03                jfs        17      17      1    open/syncd    /cg-aix61_mm_alt/alt_inst/opt

lv04                jfs        9       9       1    open/syncd    /cg-aix61_mm_alt/alt_inst/tmp

lv05                jfs        142     142     1    open/syncd    /cg-aix61_mm_alt/alt_inst/usr

lv06                jfs        22      22      1    open/syncd    /cg-aix61_mm_alt/alt_inst/var

lv07                jfs        2       2       1    open/syncd    /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

root@nim1 / #

 

These file systems were used to extract the mksysb locally on the NIM master and then migrate the image to AIX 7.1. I noticed the restbyname process consuming some CPU during the restore process.

 

Topas Monitor for host:nim1                     EVENTS/QUEUES    FILE/TTY

Thu Jan  6 10:15:38 2011   Interval:2           Cswitch    1155  Readch  2655.6K

                                                Syscall    3143  Writech 2664.7K

CPU     User% Kern% Wait% Idle%   Physc  Entc%  Reads       454  Rawin         0

Total     6.1  22.0   8.2  63.7    0.09  47.04  Writes     1168  Ttyout      578

                                                Forks        12  Igets        85

Network    BPS  I-Pkts  O-Pkts    B-In   B-Out  Execs         0  Namei       521

Total    7.74K    5.50    1.50   5.74K   2.00K  Runqueue      0  Dirblk       97

                                                Waitqueue   1.0

Disk    Busy%      BPS     TPS  B-Read  B-Writ                   MEMORY

Total     0.0    5.21M   383.0   2.02M   3.19M  PAGING           Real,MB    2048

                                                Faults      776  % Comp     67

FileSystem          BPS    TPS  B-Read  B-Writ  Steals     1118  % Noncomp  32

Total             2.03M  85.50   2.03M       0  PgspIn        0  % Client   19

                                                PgspOut       0

Name           PID  CPU%  PgSp Owner            PageIn      517  PAGING SPACE

restbyna   15532232 16.4 1.21M root             PageOut     816  Size,MB     512

vmmd         458766  5.5 1.19M root             Sios        969  % Used      3

lrud         262152  1.5  512K root                              % Free     97

syncd       2097296  1.1  592K root             NFS (calls/sec)

topas      16973864  0.5 1.98M root             SerV2         0  WPAR Activ    0

xmgc         851994  0.5  448K root             CliV2         0  WPAR Total    0

nimesis    12320772  0.3 2.41M root             SerV3         0  Press: "h"-help

getty       7798832  0.1  580K root             CliV3         0         "q"-quit

xmtopas     8847610  0.1  816K root

nfsd        5111964  0.1 4.75M root

xmtopasa    6947052  0.1  916K root

gil         1441836  0.0  960K root

java        4784136  0.0 57.5M pconsole

java        8716308  0.0 70.7M root

rpc.lock    6029528  0.0 1.19M root

clcomd      6160578  0.0 1.69M root

pilegc       786456  0.0  640K root

aixmibd     3997882  0.0 1.24M root

netm        1376298  0.0  448K root

topasrec    8454202  0.0 1.01M root

 

 

root@nim1 / # ps -ef | grep rest

    root 15270132  3342336   1 10:15:55  pts/1  0:00 grep rest

    root 15532232 10747910  37 10:14:02  pts/0  0:02 restbyname -x -q -f /export/mksysb/cg-aix61

root@nim1 / #

 

I didn’t encounter any issues with my nimadm operation but if I had I could have reviewed the nimadm log file located in /var/adm/ras:

 

root@nim1 /var/adm/ras/alt_mig # ls -ltr

total 856

-rw-r--r--    1 root     system       435009 Jan 06 11:00 cg-aix61_alt_mig.log

 

 

If you ever encounter problems with nimadm, I highly recommend that you re-run the operation with the –V and –D flags. This will enable verbose output, as well as debugging information which can assist you greatly in determining the root cause of your issue.

 

I was able to use the new, migrated, AIX 7.1 mksysb image to install a new LPAR.

 

AIX Version 7

Copyright IBM Corporation, 1982, 2010.

login: root

root's Password:

*******************************************************************************

*                                                                             *

*                                                                             *

*  Welcome to AIX Version 7.1!                                                *

*                                                                             *

*                                                                             *

*  Please see the README file in /usr/lpp/bos for information pertinent to    *

*  this release of the AIX Operating System.                                  *

*                                                                             *

*                                                                             *

*******************************************************************************

Last login: Thu Jan  6 13:21:04 EETDT 2011 on /dev/vty0

 

# oslevel -s

7100-00-02-1041

#

 

You’ve gotta love NIM! Take a look at the nimadm documentation for more information:

 

http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds4/nimadm.htm

 

By the way, if you are interested in migrating from AIX 4.3 to AIX 7.1, then you’ll need to consider the mksysb migration method. I’ve not done this myself (i.e. 4.3 to 7.1 migration) but the feedback I received from the AIX development team indicates that it does work as expected.

 

“... you had expressed some interest in mksysb migration.  Since we were not aware of anyone having attempted it with 7.1.  I went ahead and created a 4.3.3.75 system and created a mksysb and then used that mksysb in conjunction with nim and successfully installed another system using the 4.3 mksysb and 7.1 nim resources to perform a mksysb migration to 7.1.  I just followed the steps as noted in the Installation guide for mksysb migration. “

Refer to the following document for further information on mksysb migration with NIM.

http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.install/doc/insgdrf/mksysbmigration.htm

 

NIM from A to Z in AIX 5L Redbook.

http://www.redbooks.ibm.com/redpieces/abstracts/sg247296.html?Open

I’ve written about multibos before, here and here. But recently I started experimenting with multibos mksysb migration. A customer asked me how this worked and apart from a high-level view I wasn’t able to provide any real world experience, so I thought I’d give it a try. What follows is just a ‘brain dump’ from my quick test.

 

First of all this isn’t really a migration. It just simply populates a second instance of AIX with a higher-version. It doesn’t really migrate (or merge) your existing configuration into the second instance. So I’m not sure how useful this feature really is right now.

 

Starting with 5.3 TL9 you can add a 6.1 TL2 (or above) instance. This is done with the new –M flag. You must be running with the 64bit kernel.

This isn’t really a migration because it populates the second instance using a mksysb based on the new release.

In 6.1 TL2 a new flag (-M) was added to the mksysb command which allows you to create a mksysb for use with multibos. It creates a backup of BOS (/, /usr, /var, /opt).
bos.alt_disk_install.boot_images must be installed.

To preview the multibos migration
multibos –M <location of mksysb> -spX

To perform the multibos migration
multibos –M <location of mksysb> -sX

It is not advised to run in this environment for an extended period of time. There could be problems if tfactor or maps are used. Be aware that 6.1 specific attributes may not be reflected in the standby instance.

 

So in my lab environment I have two AIX LPARs. One is running AIX 6.1 and the other running AIX 7.1.

 

First I take a mksysb (with the –M flag) of the AIX 7.1 system to a file. This file will be called by multibos to populate the second instance.

 

aix7[/] > mksysb -Mie /data/aix7-mksysb

 

Creating information file (/image.data) for rootvg.

 

Creating list of files to back up.

 

Backing up 71643 files.....

 

71643 of 71643 files (100%)

0512-038 mksysb: Backup Completed Successfully.

 

aix7[/] > ls -ltr /data

total 4276112

drwxr-xr-x    2 root     system          256 Feb 21 20:59 lost+found

-rw-r--r--    1 root     system   2189363200 Feb 21 21:06 aix7-mksysb

 

I copied this file over to my AIX 6.1 system. This was the system that was to be ‘migrated’. The next step was to perform a preview of the multibos operation.

 

root@aix6 /# multibos -sXp -M /data/aix7-mksysb

Initializing multibos methods ...

Initializing log /etc/multibos/logs/op.alog ...

Gathering system information ...

 

+-----------------------------------------------------------------------------+

Preview

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

Validating install images location ./aix7-mksysb

Processing preview information ...

 

ACTIVE LV:          hd4

STANDBY LV:         bos_hd4

TYPE:               jfs2

ACTIVE FS:          /

STANDBY FS:         /bos_inst

ACTION:             Setup

STATE:              mounted

 

ACTIVE LV:          hd2

STANDBY LV:         bos_hd2

TYPE:               jfs2

ACTIVE FS:          /usr

STANDBY FS:         /bos_inst/usr

ACTION:             Setup

STATE:              mounted

 

ACTIVE LV:          hd9var

STANDBY LV:         bos_hd9var

TYPE:               jfs2

ACTIVE FS:          /var

STANDBY FS:         /bos_inst/var

ACTION:             Setup

STATE:              mounted

 

ACTIVE LV:          hd10opt

STANDBY LV:         bos_hd10opt

TYPE:               jfs2

ACTIVE FS:          /opt

STANDBY FS:         /bos_inst/opt

ACTION:             Setup

STATE:              mounted

 

ACTIVE LV:          hd5

STANDBY LV:         bos_hd5

TYPE:               boot

ACTIVE FS:          None

STANDBY FS:         None

ACTION:             Setup

STATE:              closed

 

ACTIVE LV:          hd8

STANDBY LV:         bos_hd8

TYPE:               jfs2log

ACTIVE FS:          None

STANDBY FS:         None

ACTION:             Setup

STATE:              open

 

Log file is /etc/multibos/logs/op.alog

Return Status = SUCCESS

 

With a successful preview out of the way, I initiated the real multibos operation.

 

# multibos -sX -M /data/aix7-mksysb

 

Gathering system information ...

 

+-----------------------------------------------------------------------------+

Setup Operation

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

Validating install images location ./aix7-mksysb

New volume on /data/aix7-mksysb:

Cluster size is 51200 bytes (100 blocks).

The volume number is 1.

The backup date is: Mon Feb 21 21:05:39 CST 2011

Files are backed up by name.

The user is root.

x        13007 ./image.data

x       168589 ./tmp/vgdata/rootvg/backup.data

The total size is 181596 bytes.

The number of restored files is 2.

 

+-----------------------------------------------------------------------------+

Logical Volumes

+-----------------------------------------------------------------------------+

Creating standby BOS logical volume bos_hd5

Creating standby BOS logical volume bos_hd4

Creating standby BOS logical volume bos_hd2

Creating standby BOS logical volume bos_hd9var

Creating standby BOS logical volume bos_hd10opt

Creating standby BOS logical volume bos_hd8

 

+-----------------------------------------------------------------------------+

File Systems

+-----------------------------------------------------------------------------+

Creating all standby BOS file systems ...

Creating standby BOS file system /bos_inst on logical volume bos_hd4

Creating standby BOS file system /bos_inst/usr on logical volume bos_hd2

Creating standby BOS file system /bos_inst/var on logical volume bos_hd9var

Creating standby BOS file system /bos_inst/opt on logical volume bos_hd10opt

 

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Mounting all standby BOS file systems ...

Mounting /bos_inst

Mounting /bos_inst/usr

Mounting /bos_inst/var

Mounting /bos_inst/opt

 

+-----------------------------------------------------------------------------+

BOS Files

+-----------------------------------------------------------------------------+

Including files for file system /

Including files for file system /usr

Including files for file system /var

Including files for file system /opt

 

Copying files using backup/restore utilities ...

New volume on /data/aix7-mksysb:

Cluster size is 51200 bytes (100 blocks).

The volume number is 1.

The backup date is: Mon Feb 21 21:05:39 CST 2011

Files are backed up by name.

The user is root.

x         6086 ./bosinst.data

x           11 ./tmp/vgdata/rootvg/image.info

x        13007 ./image.data

x       168589 ./tmp/vgdata/rootvg/backup.data

x            0 ./

x           47 ./.profile

x           63 ./.rhosts

x        13192 ./.sh_history

x            0 ./.ssh

x         1647 ./.ssh/authorized_keys

x            0 ./.topasrecrc

x            0 ./.topasrecrc/topasrec.lock

x       141722 ./.topasrecrc/topasrec.log1

x            0 ./.topasrecrc/topasrec.log2

x          108 ./.vi_history

x            0 ./admin

x            0 ./audit

x            8 ./bin

x            0 ./cdat

x           59 ./cdat/cdat.xml

x            0 ./cg

x            0 ./cgps

x            0 ./dev

x            0 ./dev/.SRC-unix

x            0 ./dev/IPL_rootvg

x            0 ./dev/__pv17.0.10617024

x            0 ./dev/__vg10

x            0 ./dev/audit

.....BLAH.....

x ./dev/zero

Creating standby BOS boot image on boot logical volume bos_hd5

254+0 records in.

254+0 records out.

179+1 records in.

177+1 records out.

saving to '/dev/bos_hd5'

44 CuDv objects to be saved

130 CuAt objects to be saved

21 CuDep objects to be saved

7 CuVPD objects to be saved

351 CuDvDr objects to be saved

2 CuPath objects to be saved

0 CuPathAt objects to be saved

0 CuData objects to be saved

Number of bytes of data to save = 18716

Compressing data

Compressed data size is = 6393

        bi_start     = 0x400

        bi_size      = 0x1677200

        bd_size      = 0x1657200

        ram FS start = 0x8c3570

        ram FS size  = 0xd93abe

        sba_start    = 0x1657600

        sba_size     = 0x20000

        sbd_size     = 0x0

Checking boot image size:

        new save base byte cnt = 0x18fd

Wrote 6397 bytes

Successful completion

 

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Unmounting all standby BOS file systems ...

Unmounting /bos_inst/opt

Unmounting /bos_inst/var

Unmounting /bos_inst/usr

Unmounting /bos_inst

 

+-----------------------------------------------------------------------------+

Bootlist Processing

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

Setting bootlist to logical volume bos_hd5 on hdisk0.

ATTENTION: firmware recovery string for standby BLV (bos_hd5):

boot /vdevice/v-scsi@30000016/disk@8100000000000000:4

ATTENTION: firmware recovery string for standby BLV (bos_hd5):

boot /vdevice/v-scsi@30000017/disk@8100000000000000:4

ATTENTION: firmware recovery string for active BLV (hd5):

boot /vdevice/v-scsi@30000016/disk@8100000000000000:2

ATTENTION: firmware recovery string for active BLV (hd5):

boot /vdevice/v-scsi@30000017/disk@8100000000000000:2

 

Log file is /etc/multibos/logs/op.alog

Return Status = SUCCESS

 

Now that the second instance had been created, I thought I’d try to start a multibos shell and access the AIX 7.1 environment. I didn’t think this would work but I tried my luck anyway!

 

root@aix6 /# multibos -S

Initializing multibos methods ...

Initializing log /etc/multibos/logs/op.alog ...

Gathering system information ...

 

+-----------------------------------------------------------------------------+

Multibos Shell Operation

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

 

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Mounting all standby BOS file systems ...

Mounting /bos_inst

Mounting /bos_inst/usr

Mounting /bos_inst/var

Mounting /bos_inst/opt

 

+-----------------------------------------------------------------------------+

Multibos Root Shell

+-----------------------------------------------------------------------------+

Starting multibos root shell ...

multibos: 0565-110 Cannot start multibos shell due to incompatible levels of active and standby BOS.

 

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Unmounting all standby BOS file systems ...

Unmounting /bos_inst/opt

Unmounting /bos_inst/var

Unmounting /bos_inst/usr

Unmounting /bos_inst

 

Log file is /etc/multibos/logs/op.alog

Return Status: FAILURE

 

Upon checking my bootlist output, I noticed (as expected) that the list now contained two extra entries for bos_hd5. These were the boot logical volume entries for the second instance. If I was to boot from this LV I’d be booting into AIX 7.1. Cool.

 

root@aix6 /# bootlist -m normal -o

hdisk0 blv=bos_hd5

hdisk0 blv=bos_hd5

hdisk0 blv=hd5

hdisk0 blv=hd5

 

So at this point, I’d created a second instance of AIX running 7.1. My current version of (running) AIX was AIX 6.1. All I had to do now was reboot the LPAR and let it restart as an AIX 7.1 system.

 

root@aix6 /# oslevel -s

6100-01-05-0920

 

root@aix6 / # shutdown –Fr

 

The LPAR rebooted successfully and I found I was now running AIX 7.1, just as I’d hoped.

 

aix6[/] > oslevel -s

7100-00-01-1037

 

If I wanted to go back to AIX 6.1, I would change my bootlist setting again and restart the LPAR.

 

aix6[/] > bootlist -m normal -o

hdisk0 blv=bos_hd5 pathid=0

hdisk0 blv=bos_hd5 pathid=1

hdisk0 blv=hd5 pathid=0

hdisk0 blv=hd5 pathid=1

 

aix6[/] > bootlist -m normal hdisk0 blv=hd5 pathid=0 hdisk0 blv=hd5 pathid=1 hdisk0 blv=bos_hd5 pathid=0 hdisk0 blv=bos_hd5 pathid=1

aix6[/] > bootlist -m normal -o

hdisk0 blv=hd5 pathid=0

hdisk0 blv=hd5 pathid=1

hdisk0 blv=bos_hd5 pathid=0

hdisk0 blv=bos_hd5 pathid=1

 

aix6[/] > shutdown –Fr

 

root@aix6 /# oslevel -s

6100-01-05-0920

 

Now that I’ve actually tried this method of migration, I’m not sure I’d actually use it in its current form.

 

Although the migration keeps my hostname and IP address, the file systems are not shared between instances. Most of the target systems configuration is not retained. For example, any user accounts I create on my AIX 6.1 system would also need to be created on the existing AIX.7.1 system which I used to create the AIX 7.1 mksysb image. It reminds me a little of a preservation install.

 

More information on multibos:

 

http://www.aixmind.com/?p=1192

 

I received an email from one of my customers recently that simply said:

 

“mate …on lpar30, for some reason the IBM.DRM and others are failing to start .. um .. any chance you could have a quick look at that ?”

 

So I asked, “OK, so this use to work right?”. To which I received a relatively confused reply, ”.....yep it did....actually no....it’s never worked....or has it???...I’m not sure...”.

 

Based on my experience, the most common issue that prevents DLPAR operations from working are network problems. Before diving into the deep end and trying to debug RSCT, it’s always best to start with the basics. For example, can you ping the HMC from the LPAR? Can you ping the LPAR from the HMC? If either of these tests fails, check the network configuration on both components before doing anything else.

 

On the HMC check the network settings first e.g.

 

·         Click on HMC Configuration and then Customize Network Settings.

– Verify the IP address, netmask, default gateway, network routes, DNS server are all set correctly.

– Check the LPAR communications box in HMC configuration screen for LAN adapter that is used for HMC-to-LPAR communication.

– By the way, unlike POWER4 systems, LPARs on POWER5 and POWER6 systems do not depend on host name resolution for DLPAR operations.

 

·         Check routing on the LPAR and the HMC.

– Use ping and the HMC’s Test Network Connectivity task to verify the LPAR and the HMC can communicate with each other.

 

If you check the network and you are happy that the LPAR and the HMC can communicate, then perhaps you need to re-initialise the RMC subsystems on the AIX LPAR. Run the following commands:

  

# /usr/sbin/rsct/bin/rmcctrl –z

# /usr/sbin/rsct/bin/rmcctrl –A

# /usr/sbin/rsct/bin/rmcctrl –p

 

Wait up to 5 minutes before trying DLPAR again. If DLPAR still doesn’t work i.e. the HMC is still reporting no values for DCaps, and the IBM.DRM subsystem still won’t start, try using the recfgct command.

 

hscroot@hmc1:~> lspartition -dlpar

.....

<#5> LPAR:<24*9117-MMB*10284FP, , 192.168.1.15>

       Active:<1>, OS:<AIX, 6.1, 6100-05-01-1016>, DCaps:<0x0>, CmdCaps:<0x0, 0x0>, PinnedMem:<768>

.....

 

# /usr/sbin/rsct/install/bin/recfgct

 

Wait 5 minutes. This should resolve your DLPAR issue. The IBM.DRM subsystem should now be active and there should be good (non-zero) values for DCaps:

 

# lssrc -g rsct_rm

Subsystem         Group            PID          Status

 IBM.DRM          rsct_rm          6881300      active

 IBM.CSMAgentRM   rsct_rm          7274530      active

 IBM.ServiceRM    rsct_rm          6029480      active

 IBM.AuditRM      rsct_rm          6357058      active

 IBM.ERRM         rsct_rm          4456566      active

 IBM.LPRM         rsct_rm          6946986      active

 

hscroot@hmc1:~> lspartition -dlpar

....

<#5> LPAR:<24*9117-MMB*10284FP, , 192.168.1.15>

       Active:<1>, OS:<AIX, 6.1, 6100-05-01-1016>, DCaps:<0xc5f>, CmdCaps:<0x1b, 0x1b>, PinnedMem:<994>

....

 

Only run the rmcctrl and recfgct commands if you believe something has become corrupt in the RMC configuration of the LPAR. The fastest way to fix a broken configuration or to clear out the RMC ACL files after cloning (via alt_disk migration) is to use the recfgct command.

 

These daemons should work “out of the box” and are not typically the cause of DLPAR issues. However, you can try stopping and starting the daemons when troubleshooting DLPAR issues.

 

The rmcctrl -z command just stops the daemons. The rmcctrl -A command ensures that the subsystem group (rsct) and the subsystem (ctrmc) objects are added to the SRC, and an appropriate entry added to the end of /etc/inittab and it starts the daemons.

 

The rmcctrl –p command enables the daemons for remote client connections i.e. from the HMC to the LPAR and vice versa.

 

If you are familiar with the System Resource Controller (SRC) you might be tempted to use stopsrc and startsrc commands to stop and start these daemons.

Do not do it; use the rmcctrl commands instead.

 

If /var is 100% full, use chfs to expand it. If there is no more space available, examine subdirectories and remove unnecessary files (for example, trace.*, core, and so forth). If /var is full, RMC subsystems may fail to function correctly.

 

The polling interval for the RMC daemons on the LPAR to check with the HMC daemons is 5-7 minutes; so you need to wait long enough for the daemons to start up and synchronize.

 

The Resource Monitoring and Control (RMC) daemons are part of the Reliable, Scalable Cluster Technology (RSCT) and are controlled by the System Resource Controller (SRC). These daemons run in all LPARs and communicate with equivalent RMC daemons running on the HMC. The daemons start automatically when the operating system starts and synchronize with the HMC RMC daemons.

 

The daemons in the LPARs and the daemons on the HMC must be able to communicate over the network for DLPAR operations to succeed. This is not the network connection between the managed system (FSP) and the HMC; it is the network connection between the operating system (AIX) in each LPAR and the HMC.

 

Note: Apart from rebooting, there is no way to stop and start the RMC daemons on the HMC.

 

The following links also contain some (out dated) information relating to DLPAR verification and troubleshooting. Even though it is quite old some of it is still relevant today and is good a place to start.

 

The most common reasons for failures with Dynamic Logical LPARing

http://www-01.ibm.com/support/docview.wss?rs=111&context=SWG10&q1=dcaps&uid=isg3T1010615&loc=en_US&cs=utf-8&lang=en

 

Dynamic LPAR tips and checklists for RMC authentication and authorization

http://www.ibm.com/developerworks/systems/articles/DLPARchecklist.html

 

Setting up the HMC/LPARs hostname and network (old but interesting)

http://computerstec.com/aixhowto/aixh18.htm

 

lsLPAR -dlpar, DCAPs values (old but still applies)

http://community.livejournal.com/eserver/18517.html

 

The previous link (above) provides some information relating to the values for DCaps and what they mean (also out dated):

 

0 - DR CPU capable  (can move CPUs)

1 - DR MEM capable  (can move memory)

2 - DR I/O capable  (can move I/O resources)

3 - DR PCI Bridge   (can move PCI bridges)

4 - DR Entitlement  (POWER 5 can change shared entitlement)

5 - Multiple DR CPU (AIX 5.3 can move 2+ CPUs at once)

0x3f = max, and 0xf is common for AIX 5.2

 

If you are interested in how HMC and LPAR authentication works with DLPAR, then read on. Otherwise, happy DLPARing!

 

HMC and LPAR authentication (RSCT authentication)

 

The diagram below outlines how the HMC and an LPAR authenticate with each other in order for DLPAR operations to work. RSCT authentication is used to ensure the HMC is communicating with the correct LPAR.

 

Authentication is the process of ensuring that another party is who it claims to be. Authorization is the process by which a cluster software component grants or denies resources based on certain criteria. The RSCT component that implements authorization is RMC. It uses access control list (ACL) files to control user access to resources.

 

The RMC component subsystem uses cluster security services to map the operating system user identifiers, specified in the ACL file, to network security identifiers to determine if the user has the correct permissions. This is performed by the identity mapping service, which uses information stored in the identity mapping files ctsec_map.global and ctsec_map.local.

 

 

 

The RSCT authorization process in detail:

 

1.    On the HMC: DMSRM pushes down the secret key and HMC IP address to NVRAM when it detects a new CEC; this process is repeated every five minutes. Each time an HMC is rebooted or DMSRM is restarted, a new key is used.

 

2.       On the AIX LPAR: CSMAgentRM, through RTAS (Run-time Abstraction Services), reads the key and HMC IP address out from NVRAM. It will then authenticate the HMC. This process is repeated every five minutes on a LPAR to detect a new HMCs and if the key has changed. An HMC with a new key is treated as a new HMC and will go though the authentication and authorization processes again.

 

3.       On the AIX LPAR: After authenticating the HMC, CSMAgentRM will contact the DMSRM on the HMC to create a ManagedNode resource in order to identify itself as a LPAR of this HMC. CSMAgentRM then creates a compatible ManagementServer resource on AIX. This can be displayed on AIX with the lssrsrc command. e.g.

 

root@aix6 / # lsrsrc "IBM.ManagementServer"

Resource Persistent Attributes for IBM.ManagementServer

resource 1:

        Name             = "192.168.1.244"

        Hostname         = "192.168.1.244"

        ManagerType      = "HMC"

        LocalHostname    = "10.153.3.133"

        ClusterTM        = "9078-160"

        ClusterSNum      = ""

        ActivePeerDomain = ""

        NodeNameList     = {"aix6"}

 

4.       On the AIX LPAR: After the creation of the ManagedNode and ManagementServer resources on the HMC and AIX respectively, CSMAgentRM grants HMC permission to access necessary resource classes on the LPAR. After granting the HMC permission, CSMAgentRM will change its ManagedNode, on the HMC, Status to 1. (It should be noted that without proper permission on AIX, the HMC would be able to establish a session with the LPAR but will not be able to query for OS information, DLPAR capabilities, or execute DLPAR commands afterwards.)

 

5.       On the HMC: After the ManagedNode Status is changed to 1, LparCmdRM establishes a session with the LPAR, queries for operating system information and DLPAR capabilities, notifies CIMOM about the DLPAR capabilities of the LPAR, and then waits for the DLPAR commands from users.