Interested in paid support for LPAR2RRD tool?

 

Let’s all support Pavel and spread the word!

 

From: Pavel Hampl <pavel.hampl@cz.ibm.com> Date: 21 July 2011 3:00:19 AM AEST To: undisclosed-recipients:; Subject: LPAR2RRD professional support announcement Hi Everyone, You are receiving that mail because we have been either in touch in regards of LPA2RRD tool about year ago or you are on LPAR2RRD mailing list. I’ve decided to make professional paid support of LPAR2RRD. Here are my reasons which led me to that thought: - I have not touched the source code for nearly 1.5 year (looks like the last version is quite stable :) ). No time at all and lack of motivation... - Only what I trying to keep is support of the tool, but my response times are not what I am proud of - the tool itself would need big time investment to implement new functionalities, rewrite the web face (would need to hire a web designer for that) etc. Only the solution what I see how to keep continuity and do not let project slowly died is paid support. Basically I feel that as the last chance for any further development or support from my side. I simply cannot manage to do everything I actually do around and still work on LPAR2RRD. If I would get some money from the project then I will be able to prioritize my other activities in favour of LPAR2RRD development and support. From that reason I’ve created new LPAR2RRD home here: http://www.lpar2rrd.com I am about to found a company which would be responsible for support. Based on your feedback I will decide go or no-go. I could even imagine working on LPAR2RRD full time in case of enough support subscribers. I do not intend to force you ordering support. If you are happy with the actual version, you have no issues etc ... that this makes me also happy. Make people happy was the main reason why I spent my free time by developing of the tool since 2006. I do not regret that at all! It was fun. Support details can be found here: http://www.lpar2rrd.com/support.htm Let me know who would be inetrested in that offer. Cheers, Pavel P.S.: For those of you who have installed LPAR2RRD for your customers please distribute this information to them. Thanks.

 

 

OK, so I had a NIM master with two network interfaces, en0 and en1. The en0 interface was connected to the 192.168.10.0 network and the en1 interface was  connected to the 10.1.1.0 network. When the master was initially configured, we chose en0 as the primary install interface. The master could now install NIM clients on the 192.168.10.0 network, without any additional NIM network configuration.

 

root@nim1 / # ifconfig –a

en0: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>

        inet 192.168.10.10 netmask 0xffffff00 broadcast 192.168.10.255

         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

en1: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>

        inet 10.1.1.10 netmask 0xffffff00 broadcast 10.1.1.255

         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>

        inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255

        inet6 ::1%1/0

         tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1

 

There was a single NIM network definition for the 192.168.10.0 network (net_192_168_10) in the NIM database.

 

root@nim1 / # lsnim -c networks

net_192_168_10     networks       ent

 

Sure enough, the time came when we needed to install NIM clients on the 10.1.1.0 network. Rather than adding NIM routes, we chose to configure the NIM master with an additional install interface on the 10.1.1.0 network. Fortunately, the NIM master was already “directly” connected to the 10.1.1.0 network on its en1 interface. So all we had to do was update the NIM configuration.

 

First we added a new entry in the /etc/hosts file for the NIM master’s 10.1.1.0 network address. The hostname for this interface was nim1i.

 

root@nim1 / # grep nim1 /etc/hosts

192.168.10.10  nim1

10.1.1.10 nim1i

 

root@nim1 / # host nim1i

nim1i is 10.1.1.10

 

Next we defined a new network install interface for the NIM master via the smit fastpath, ‘smit nim_mac_if’.

 

root@nim1 / # smit nim_mac_if

 

Define a Network Install Interface

 

 

                                                            +--------------------------------------------------------------------------+

                                                            |                               Machine Name                               |

                                                            |                                                                          |

                                                            | Move cursor to desired item and press Enter.                             |

                                                            |                                                                          |

                                                            |   master     machines       master                                       |

                                                            |   lpar14     machines       standalone                                   |

                                                            |   lpar15     machines       standalone                                   |

                                                            |   lpar16     machines       standalone                                   |

                                                            |                                                                          |

                                                            | F1=Help                 F2=Refresh              F3=Cancel                |

                                                            | F8=Image                F10=Exit                Enter=Do                 |

F1=Help                                          F2=Refresh | /=Find                  n=Find Next                                      |           F8=Image

F9=Shell                                         F10=Exit   +--------------------------------------------------------------------------+

 

 

 

We entered the hostname of the NIM master on the 10.1.1.0 network i.e. nim1i.

 

                                                                                 Define a Network Install Interface

 

Type or select a value for the entry field.

Press Enter AFTER making all desired changes.

 

                                                        [Entry Fields]

* Host Name of Network Install Interface             [nim1i]

 

We named the new NIM network, net_10_1_1 and entered the appropriate subnet mask for this network. We did not enter a default gateway for the client or the NIM master (it is unnecessary, as the master is “directly” connected to this network).

 

                                                                                 Define a Network Install Interface

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                                                        [Entry Fields]

* Machine Name                                        master

  Network Install Interface

*   Cable Type                                        N/A          +

    Network Speed Setting                            []            +

    Network Duplex Setting                           []            +

*   NIM Network                                      [net_10_1_1]

*     Network Type                                    ent

*     Ethernet Type                                   Standard     +

*     Subnetmask                                     [255.255.255.0]

      Default Gateway Used by Machine                []

      Default Gateway Used by Master                 []

*   Host Name                                         nim1i

    Network Adapter Hardware Address                 [0]

    Network Adapter Logical Device Name              []

 

There were now two network definitions in our NIM database. The Nstate attribute was set to ‘ready for use’, so our definition was OK to use.

 

root@nim1 / # lsnim -c networks

net_192_!68_10     networks       ent

net_10_1_1          networks       ent

 

root@nim1 / # lsnim -l net_10_1_1

net_10_1_1:

   class      = networks

   type       = ent

   comments   = Generated during definition of nim1i

   Nstate     = ready for use

   prev_state = information is missing from this object's definition

   net_addr   = 10.1.1.0

   snm        = 255.255.255.0

 

The NIM master has two network interfaces defined.

 

root@nim1 / # lsnim -l master | grep if

   if_defined          = chrp.64.ent

   if1                 = net_192_168_10 nim1 0

   if2                 = net_10_1_1 nim1i 0

 

Now we could install new NIM clients using this network. We verified this by adding a new NIM client to the NIM database.

 

# smit nim_mkmac

                                                                                          Define a Machine

 

Type or select a value for the entry field.

Press Enter AFTER making all desired changes.

 

                                                        [Entry Fields]

* Host Name of Machine                               [lpar22i]

    (Primary Network Install Interface)

 

After pressing enter on the previous smit panel, we were immediately presented with a panel to modify the attributes of the NIM client and (more importantly) the correct network and hostname were automatically selected as part of the clients definition.

 

                                                                                          Define a Machine

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                                                        [Entry Fields]

* NIM Machine Name                                   [lpar22]

* Machine Type                                       [standalone]                                                                                                                                 +

* Hardware Platform Type                             [chrp]       +

  Kernel to use for Network Boot                     [64]         +

  Communication Protocol used by client              [nimsh]      +

  Primary Network Install Interface

*   Cable Type                                        N/A         +

    Network Speed Setting                            []           +

    Network Duplex Setting                           []           +

*   NIM Network                                       net_10_1_1

*   Host Name                                         lpar22i

    Network Adapter Hardware Address                 [0]

    Network Adapter Logical Device Name              []

  IPL ROM Emulation Device                           []           +/

  CPU Id                                             []

  Machine Group                                      []           +

 

  Managing System Information

  WPAR Options

    Managing System                                  []

    -OR-

    LPAR Options

      Identity                                       []

      Management Source                              []            +

 

  Comments                                           []

 

 

If we had configured NIM incorrectly i.e. it was not aware of the 10.1.1.0 network, then we would have been prompted for network information before we could configure the NIM client (as shown below).

 

                                                                                          Define a Machine

 

Type or select a value for the entry field.

Press Enter AFTER making all desired changes.

 

                                                        [Entry Fields]

* Host Name of Machine                               [lpar22i]

    (Primary Network Install Interface)

 

                                                            +--------------------------------------------------------------------------+

                                                            |      Type of Network Attached to Primary Network Install Interface       |

                                                            |                                                                          |

                                                            | Move cursor to desired item and press Enter.                             |

                                                            |                                                                          |

                                                            |   tok     = token ring network                                           |

                                                            |   ent     = ethernet network                                             |

                                                            |   fddi    = FDDI network                                                 |

                                                            |   generic = generic network (no network boot capability)                 |

                                                            |   atm     = ATM network                                                  |

                                                            |   ent6    = IPv6 ethernet network                                        |

                                                            |   hfi     = HFI network                                                  |

                                                            |                                                                          |

                                                            | F1=Help                 F2=Refresh              F3=Cancel                |

 

If you have the AIX iSCSI Software Initiator (iscsi0) configured on your system, you may find that nmon reports some “funny” information under the Disk Adapter (a) and Disk I/O (d) screens, such as:

 

┌─topas_nmon──4=Top-by-RAM-use───Host=aix1 Refresh=2 secs───15:09.41──────────────────────────┐

│ Disk-Adapter-I/O ─────────────────────────────────────────────────────────────────────────────────────│

│Name          %busy     read    write        xfers Disks Adapter-Type                                  │

│vscsi1          0.0      0.0      0.0 KB/s     0.0 4     Virtual SCSI Client A                         │

│vscsi0          0.0      0.0      0.0 KB/s     0.0 4     Virtual SCSI Client A                         │

│Adapter name "not available" returned by AIX API                                                       │

│vscsi3          0.0      0.0      0.0 KB/s     0.0 12    Virtual SCSI Client A                         │

│vscsi2          0.0      0.0      0.0 KB/s     0.0 12    Virtual SCSI Client A                         │

│TOTALS  5 adapters       0.0      0.0 KB/s     0.0 32    TOTAL(MB/s)=0.0                               │

│ Disk-KBytes/second-(K=1024,M=1024*1024) ──────────────────────────────────────────────────────────────│

│Disk        Size  Free Disk  Disk    Volume      Disk                                                  │

│ Name         GB    GB Paths Adapter Group       Description                                           │

│hdisk11       29     0   1    vscsi1 localscratchvg Virtual SCSI Disk Drive                            │

│hdisk1        50    20   1    vscsi1 rootvg        Virtual SCSI Disk Drive                             │

│hdisk9        25     0   1    vscsi1 localscratchvg Virtual SCSI Disk Drive                            │

│hdisk3        50    34   1    vscsi1 datavg        Virtual SCSI Disk Drive                             │

│hdisk2        50    34   1    vscsi0 datavg        Virtual SCSI Disk Drive                             │

│hdisk8        25     0   1    vscsi0 localscratchvg Virtual SCSI Disk Drive                            │

│hdisk10       29     0   1    vscsi0 localscratchvg Virtual SCSI Disk Drive                            │

│hdisk0        50    20   1    vscsi0 rootvg        Virtual SCSI Disk Drive                             │

│hdisk6         0     0   1  not available None          MPIO NetApp iSCSI Disk                         │

│hdisk16        0     0   1  not available None          MPIO NetApp iSCSI Disk                         │

│hdisk17        0     0   1  not available None          MPIO NetApp iSCSI Disk                         │

│hdisk12        0     0   1  not available None          MPIO NetApp iSCSI Disk                         │

│hdisk7         0     0   1  not available None          MPIO NetApp iSCSI Disk                         │

│hdisk13        0     0   1  not available None          MPIO NetApp iSCSI Disk                         │

│hdisk14        0     0   1  not available None          MPIO NetApp iSCSI Disk                         │

│hdisk15        0     0   1  not available None          MPIO NetApp iSCSI Disk                         │

│hdisk4        50     0   1  not available projvg        MPIO NetApp iSCSI Disk                         │

│hdisk5       750     0   1  not available ipscratchvg   MPIO NetApp iSCSI Disk                         │

│hdisk18      104    54   2    vscsi2 projectsvg    Virtual SCSI Disk Drive                             │

│hdisk19      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk20      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk21      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk22      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk23      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk24      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk25      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk26      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk27      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk28      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│hdisk29      104     0   2    vscsi2 scratchvg     Virtual SCSI Disk Drive                             │

│Totals(MB/s) Read=0.0   Write=0.0   Size(GB)=2354  Free(GB)=161                                        │

│───────────────────────────────────────────────────────────────────────────────────────────────────────│

 

I don’t know if there are any plans to fix this but I wanted to share this information in case anyone else encounters this issue. iSCSI is not all that popular on AIX systems (perhaps this is a good thing?), so it may not impact enough people to warrant a quick fix.

Need to update the running attributes of a virtual SCSI adapter on an AIX system? Try these steps.

 

I want to change value for the  vscsi_err_recov attribute to fast_fail. I can’t do this if the adapter is busy i.e. has open/available paths to a disk device.

 

# lsattr -El vscsi2

vscsi_err_recov delayed_fail N/A                       True

vscsi_path_to   0            Virtual SCSI Path Timeout True

 

# chdev -l vscsi2 -a vscsi_err_recov=fast_fail

Method error (/usr/lib/methods/chggen):

        0514-029 Cannot perform the requested function because a

                 child device of the specified device is not in a correct state.

 

 

I have two paths to my disk devices. This will allow me to take one path offline while I change the attributes of the adapter.

 

# lspath | grep vscsi2

Enabled hdisk4 vscsi2

Enabled hdisk5 vscsi2

 

# rmpath -l hdisk4 -p vscsi2

path Defined

 

# rmpath -l hdisk5 -p vscsi2

path Defined

 

# lspath | grep vscsi2

Defined hdisk4 vscsi2

Defined hdisk5 vscsi2

 

With the path offline, I can now change the attributes of the device and then bring the path online again.

 

# chdev -l vscsi2 -a vscsi_err_recov=fast_fail

vscsi2 changed

 

# lsattr -El vscsi2 -a vscsi_err_recov

vscsi_err_recov fast_fail N/A True

 

# cfgmgr                                        < Bring the path online.

 

# lspath | grep vscsi2

Enabled hdisk4 vscsi2

Enabled hdisk5 vscsi2

 

Now I can do the same for the other VSCSI adapter.

 

# lsattr -El vscsi3 -a vscsi_err_recov

vscsi_err_recov delayed_fail N/A True

 

# chdev -l vscsi3 -a vscsi_err_recov=fast_fail

Method error (/usr/lib/methods/chggen):

        0514-029 Cannot perform the requested function because a

                 child device of the specified device is not in a correct state.

 

# lspath | grep vscsi3

Enabled hdisk4 vscsi3

Enabled hdisk5 vscsi3

 

# rmpath -l hdisk4 -p vscsi3

path Defined

 

# rmpath -l hdisk5 -p vscsi3

path Defined

 

# lspath | grep vscsi3

Defined hdisk4 vscsi3

Defined hdisk5 vscsi3

 

# chdev -l vscsi3 -a vscsi_err_recov=fast_fail

vscsi3 changed

 

# lsattr -El vscsi3 -a vscsi_err_recov

vscsi_err_recov fast_fail N/A True

 

# cfgmgr

# lspath | grep vscsi3

Enabled hdisk4 vscsi3

Enabled hdisk5 vscsi3

 

All paths are available again and the adapters have been updated. All of this was done without taking an outage.

 

# lspath

Enabled hdisk4 vscsi2

Enabled hdisk4 vscsi3

Enabled hdisk5 vscsi2

Enabled hdisk5 vscsi3

 

# lsattr -El vscsi2 -a vscsi_err_recov

vscsi_err_recov fast_fail N/A True

 

# lsattr -El vscsi3 -a vscsi_err_recov

vscsi_err_recov fast_fail N/A True

 

If I’d chosen not to take the paths offline, one at a time, I would have needed to change the adapters attributes by specifying the –P flag with the chdev command. Then I would need to reboot the system.

 

 -P  

Changes the device's characteristics permanently in the Customized Devices object class without actually changing the device. This is useful for devices that cannot be made unavailable and cannot be changed while in the available state. The change is made to the database, and the changes are applied to the device when the system is rebooted. This flag cannot be used with the -T flag. Not all devices support the -P flag.

 

A question I received this week from a customer.

 

Hi Chris,

 

Is there  a way to find on which processor a process is executing?

 

If I’ve used binprocessor, ps –emo THREAD, would show which processor a process is bound too. But when that’s not been used, what are the options?

 

The ps command will only help if we bind a process to a specific CPU. For example, here we can see that the bootpd process has been bound (and then un-bound) on processor 3:

 

# bindprocessor -q

The available processors are:  0 1 2 3

 

# bindprocessor 8585302 3

 

# ps -o THREAD -p 8585302

    USER     PID    PPID       TID ST  CP PRI SC    WCHAN        F     TT BND COMMAND

    root 8585302 4063394         - A    0  60  1        -   200001      -   3 bootpd /etc/bootptab

 

# bindprocessor -u 8585302

 

# ps -o THREAD -p 8585302

    USER     PID    PPID       TID ST  CP PRI SC    WCHAN        F     TT BND COMMAND

    root 8585302 4063394         - A    0  60  1        -   200001      -   - bootpd /etc/bootptab

 

The trace command will tell us what CPU a process is scheduled on.*

 

# /usr/bin/trace –J’curt' -P'p' '-a'

 

# sleep 30

 

# trcstop

 

# trcrpt -O'exec=y' -O'pid=n' -O'tid=n' -O'wparname=off' -O'cid=off' -O'svc=y' -O'timestamp=1'  | /usr/lib/ras/trace_smutil cat Z'/tmp/trace.out'

 

# vi /tmp/trace.out                 (I’m interested in PID 904428 in this case)

 

106  ksh 0.033107          dispatch:   cmd=ksh pid=9044208 tid=9175205 priority=135 old_tid=9175205 old_priority=135 CPUID=2 [302 usec]

101D ksh 0.033108                   Home SRAD=-1 Home ref1=-1 Dispatch SRAD=-1 Dispatch ref1=-1

200  ksh 0.033108                   resume  ksh iar=100180CC cpuid=02

 

 

 

*Note: If you are running AIX 5.3 try these options with trcrpt (tested on an AIX 5.3 TL11 SP3 system):

 

# trcrpt -O'exec=y' -O'pid=n' -O'tid=n' -O'svc=y' -O'timestamp=1'  | /usr/lib/ras/trace_smutil cat Z'/tmp/trace.out'

 

Perhaps there are other methods of monitoring this type of activity online, rather than offline?

 

 

 

Has your IBM XL C/C++ for AIX software evaluation license expired?

 

You can work around this little problem temporarily by placing the following environment variable in /etc/profile.

 

XLC_EXTEND_EVAL=yes

 

According to the IBM support site, this will extend the XLC license for another 74 days.

 

https://www-304.ibm.com/support/docview.wss?uid=swg21498670

 

It appears to work.

 

Before the change, I was unable to compile anything. Note the highlighted text below.

 

# /usr/vac/bin/xlc cgc.c

The license for the Evaluation version of IBM XL C/C++ for AIX V11.1 compiler product has expired. Please send an email to compiler@ca.ibm.com for information on purchasing the product.  The evaluation license can be extended to 74 days in total by either a) setting an environment variable XLC_EXTEND_EVAL=yes; or, b) specifying a compiler command line option -qxflag=extend_eval.  The extended evaluation license will expire on Sat Jun 18 10:26:33 2011. Use of the Program continues to be subject to the terms and conditions of the International License Agreement for Evaluation of Programs, including the accompanying License Information document (the "Agreement").  A copy of the Agreement can be found in the "LicAgree.pdf" and "LicInfo.pdf" files residing in the root directory of the installation media.  If you do not agree to the terms and conditions of the Agreement, you may not use or access the Program.

 

With the environment variable in place, I was able to compile again.

 

# /usr/vac/bin/xlc cgc.c

# ./a.out

Hello World!

 

Of course I could also export the environment variable as required instead e.g.

 

# export  XLC_EXTEND_EVAL=yes

# /usr/vac/bin/xlc cgc.c

# ./a.out

Hello World!

 

If you do place this environment variable in /etc/profile, you may need to restart any processes on the system that need to call the compiler.

 

 

 

Here’s a problem I’ve encountered more than once.

 

Someone installs SAP onto an AIX system and decides to use TCP port 3901 as an SAP service port. This is the same port used by nimsh. In some rare cases, nimsh may not be active on the LPAR, which makes it easy for the SAP installation to hijack port 3901. If nimsh is active, the person installing SAP may consciously stop nimsh and use port 3901 for SAP anyway. Hopefully that doesn’t happen. Hopefully, they will talk to the AIX administrator and discuss the best way forward. Hopefully...

 

In either case, if the port is taken by SAP, nimsh will no longer work. If you love using NIM as much as I do, this is a real problem! We could revert back to using rsh but no-one will do this anymore because of concerns around security. And rightfully so!

 

The ports used by nimsh (3901 and 3902) are registered to Internet Assigned Number Authority (IANA).  These port numbers appear in the /etc/services file.

 

nimsh                   3901/tcp               # NIM Service Handler

nimsh                   3901/udp               # NIM Service Handler

nimaux                  3902/tcp                # NIMsh Auxiliary Port

nimaux                  3902/udp                # NIMsh Auxiliary Port

 

Considering these port numbers are registered with IANA, we can usually persuade our SAP colleagues to change their SAP installation to use a different port number. However, depending on the skills/experience of the SAP resource, one of two things usually happens 1) They take an outage, re-install SAP and choose a different port number or 2) The more experienced/confident SAP basis resource will take an outage and modify the instance to use a different port: without reinstalling SAP.

 

Perhaps SAP need to include a warning in their install notes, advising customers not to use port 3901 on AIX systems (i.e. best practice)?

 

Now, if you must change nimsh to use a different port number, it is possible. But not recommended.

 

To do this, you must change the /etc/services file on the NIM master and the NIM client to reflect the same port numbers for nimsh. This will work until the NIM master or the NIM client have their services file overwritten by way of install or fileset updates. After which, the default values for nimsh will be reinstated.

 

You would also need to change the services file on all of your NIM clients. Every time you performed a NIM fileset update, you would need to remember to change the /etc/services file again. This is painful and bound to catch someone out eventually!

 

In the following example I’ll demonstrate how to change the port number used by nimsh.

 

We start with a typical nimsh configuration using port 3901. On the NIM client, nimsh is listening on port 3901.

nimaix / # lsof -i tcp:3901

Value of I :77   np:0

COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME

nimsh   2883740 root    4u  IPv6 0xf1000e0006f843b0      0t0  TCP *:nimsh (LISTEN)

 

On the NIM master we can connect to the NIM client on port 3901.

 

nimmast / # nim -o lslpp nimaix | head -10

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  Firefox.base.rte          2.0.0.20  COMMITTED  Firefox Web Browser

  ICU4C.rte                  6.1.4.0  COMMITTED  International Components for

                                                 Unicode

                             6.1.4.1  COMMITTED  International Components for

                                                 Unicode

                             6.1.6.0  COMMITTED  International Components for

                                                 Unicode

...

 

The /etc/services file on both the NIM client and the master have the same port number for nimsh.

 

nimmast / # grep nimsh /etc/services

nimsh                    3901/tcp               # NIM Service Handler

nimsh                    3901/udp               # NIM Service Handler

 

nimaix / # grep nimsh /etc/services

nimsh                    3901/tcp               # NIM Service Handler

nimsh                    3901/udp               # NIM Service Handler

 

To change the port number for nimsh, first we select an unused port number, in this case 39011.

 

nimaix / # lsof -i tcp:39011

Value of I :77   np:0

 

We update the /etc/services file on the master and the client to reflect the new port number.

 

nimaix / # grep nimsh /etc/services

#nimsh                   3901/tcp               # NIM Service Handler

#nimsh                   3901/udp               # NIM Service Handler

nimsh                   39011/tcp               # NIM Service Handler

nimsh                   39011/udp               # NIM Service Handler

 

nimmast / # grep nimsh /etc/services

#nimsh                   3901/tcp               # NIM Service Handler

#nimsh                   3901/udp               # NIM Service Handler

nimsh                   39011/tcp               # NIM Service Handler

nimsh                   39011/udp               # NIM Service Handler

 

We stop and start nimsh on the client and confirm that nimsh is now listening on the new port number, 39011.

 

nimaix / # stopsrc -s nimsh

0513-044 The nimsh Subsystem was requested to stop.

 

nimaix / # startsrc -s nimsh

0513-059 The nimsh Subsystem has been started. Subsystem PID is 2883742.

 

nimaix / # lsof -i tcp:39011

Value of I :77   np:0

COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME

nimsh   2883742 root    4u  IPv6 0xf1000e0006f85bb0      0t0  TCP *:nimsh (LISTEN)

 

From the NIM master we connect to the NIM client using the nim command.

 

nimmast / # nim -o lslpp nimaix | head -10

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  Firefox.base.rte          2.0.0.20  COMMITTED  Firefox Web Browser

  ICU4C.rte                  6.1.4.0  COMMITTED  International Components for

                                                 Unicode

                             6.1.4.1  COMMITTED  International Components for

                                                 Unicode

                             6.1.6.0  COMMITTED  International Components for

                                                 Unicode

....

 

We can confirm that we have connected to the NIM client on port 39011 by looking at the output from lsof and netstat. There is a TCP session established between the master and the client on port 39011.

 

nimmast / # lsof -i tcp:39011

Value of I :93   np:0

COMMAND      PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME

m_showlog 430080 root    8u  IPv4 0xf1000700047693b0      0t5  TCP nimmast:1023->nimaix:nimsh (ESTABLISHED)

 

 

nimmast / # netstat -na | grep 39011

tcp4       0      0  172.29.144.167.1023    172.29.152.241.39011   ESTABLISHED

 

nimaix / # lsof -itcp:39011

Value of I :75   np:0

COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME

nimsh   2293836 root    0u  IPv6 0xf1000e0004d313b0      0t5  TCP nimaix:nimsh->nimmast:1023 (ESTABLISHED)

nimsh   2293836 root    1u  IPv6 0xf1000e0004d313b0      0t5  TCP nimaix:nimsh->nimmast:1023 (ESTABLISHED)

nimsh   2293836 root    2u  IPv6 0xf1000e0004d313b0      0t5  TCP nimaix:nimsh->nimmast:1023 (ESTABLISHED)

nimsh   2293836 root    4u  IPv6 0xf1000e0006f85bb0      0t0  TCP *:nimsh (LISTEN)

nimsh   2293836 root    5u  IPv6 0xf1000e0004d313b0      0t5  TCP nimaix:nimsh->nimmast:1023 (ESTABLISHED)

nimsh   2883742 root    4u  IPv6 0xf1000e0006f85bb0      0t0  TCP *:nimsh (LISTEN)

nimsh   2883742 root    5u  IPv6 0xf1000e0004d313b0      0t5  TCP nimaix:nimsh->nimmast:1023 (ESTABLISHED)

 

If the /etc/services file on either the NIM client or master is incorrect we would see error messages similar to the following:

 

   0042-006 m_lslpp: (From_Master) connect A remote host refused an attempted connect operation.

 

nconn: connect() failed, errno is 79nimaix: A remote host refused an attempted connect operation.

 

Would I recommend this approach? No. This is not a good, permanent solution to the problem.

 

Do I think IBM would support this configuration? No.

 

Why am I writing about it then? Because it might help someone caught in a bind. You could use this method temporarily until you’ve sorted out your SAP port problem.

A colleague was attempting to recreate a WPAR. The WPAR had previously been removed using the WPAR Manager Web GUI. This worked, or so he tells me!

 

Anyway, when he tried to create the WPAR again from the GUI, using the same WPAR name (wpar20), it failed. The error claimed that /wpars/wpar20 already existed. But it didn’t!

 

Now, before I go any further, looking back at the issue now, it is pretty obvious what the problem might have been. However, the solution was not evident at first. This proves one thing for certain: some problems cannot be resolved without TWO cups of coffee in the morning.

 

Of course, I resorted to the command line to see what was REALLY happening.

 

The mkwpar command was indeed complaining that /wpars/wpar20 already existed.

 

# mkwpar -n wpar20

mkwpar: 0960-288 The /wpars/wpar20 file system already exists.

mkwpar: 0960-417 Specify -p to create this workload partition using the existing file system data.

 

I could not find any directory or file system with that name, anywhere under /wpars or elsewhere.

 

# ls -l /wpars/wpar20

/wpars/wpar20 not found

 

# ls -l /wpars

total 0

#

 

# df –g | grep –i wpar

#

 

# lsvg | lsvg –il | grep –i wpar

#

 

I ran the mkwpar with truss to try to find out where it was finding the reference to /wpars/wpar20. At this point you have probably already determined where the problem might be!

 

The truss output pointed me in the direction of /etc/filesystems.....of course! Another “facepalm“ moment!

 

# truss mkwpar -n wpar20 > /tmp/wpar20 2>&1

# vi /tmp/wpar220

 

statx("/wpars/wpar20", 0x30008AB8, 128, 010)    = -1

kopen("/etc/filesystems", O_RDONLY|O_LARGEFILE) = 4

 

Whatever my learned colleague had done when he “removed” the WPAR with the GUI, it left entries for the WPARs file systems in /etc/filesystems.

 

# grep wpar20 /etc/filesystems | grep -v type

/wpars/wpar20:

/wpars/wpar20/home:

/wpars/wpar20/opt:

/wpars/wpar20/proc:

/wpars/wpar20/tmp:

/wpars/wpar20/usr:

/wpars/wpar20/var:

 

And after removing these entries with the rmfs command, like so:

 

# grep wpar20 /etc/filesystems | grep -v type | sed -e 's/://'

/wpars/wpar20

/wpars/wpar20/home

/wpars/wpar20/opt

/wpars/wpar20/proc

/wpars/wpar20/tmp

/wpars/wpar20/usr

/wpars/wpar20/var

 

# grep wpar20 /etc/filesystems | grep -v type | sed -e 's/://'

# cg="/wpars/wpar20

> /wpars/wpar20/home

> /wpars/wpar20/opt

> /wpars/wpar20/proc

> /wpars/wpar20/tmp

> /wpars/wpar20/usr

> /wpars/wpar20/var

> "

 

# echo $cg

/wpars/wpar20 /wpars/wpar20/home /wpars/wpar20/opt /wpars/wpar20/proc /wpars/wpar20/tmp /wpars/wpar20/usr /wpars/wpar20/var

 

# for i in $cg

> do

> rmfs $i

> done

 

We were able to create the WPAR once more without a problem.

 

# mkwpar -n wpar20

mkwpar: Creating file systems...

/

/home

/opt

/proc

/tmp

/usr

/var

Mounting all workload partition file systems.

x ./usr

x ./lib

x ./admin

x ./admin/tmp

x ./audit

x ./dev

x ./etc

x ./etc/check_config.files

x ./etc/consdef

x ./etc/cronlog.conf

x ./etc/csh.cshrc

x ./etc/csh.login

x ./etc/dlpi.conf

x ./etc/dumpdates

x ./etc/environment

x ./etc/ewlm

x ./etc/ewlm/limits

x ./etc/ewlm/trc

x ./etc/ewlm/trc/config_schema.xsd

x ./etc/ewlm/trc/output_schema.xsd

x ./etc/filesystems

x ./etc/group

x ./etc/inittab

x ./etc/magic

...etc...

 

Lessons I learnt from this were twofold, truss is (still) useful and drink more coffee before tackling a problem!

I was working at a client site today, on a NIM master that I configured a month or so ago. I was there to install the TSM backup client software on about 30 or so LPARs. Of course I was going to use NIM to accomplish this task.

 

The software install via NIM worked for the majority of the LPARs but I noticed a few of them were failing. This was very odd, as the last time I’d use the same NIM method to install software, everything was fine.

 

I suspected that perhaps something had changed on the client LPARs...maybe with their /etc/niminfo file for instance. So I performed the following steps to reconfigure the /etc/niminfo file and configure the nimsh subsystem on the client LPAR.

 

lpar1# mv /etc/niminfo /etc/niminfo.old

lpar1# niminit -a master=nim1 -a name=`hostname`

lpar1# stopsrc -s nimsh

lpar1# smit nim_config_services

                                                                               Configure Client Communication Services

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                                                        [Entry Fields]

* Communication Protocol used by client              [nimsh]             +

 

  NIM Service Handler Options

*   Enable Cryptographic Authentication              [disable]           +

      for client communication?

 

    Install Secure Socket Layer Software (SSLv3)?    [no]                +

      Absolute path location for INSTALLP package    [/dev/cd0]          /

                  -OR-

      lpp_source which contains INSTALLP package     []                  +

 

    Alternate Port Range for Secondary Connections

       (reserved values will be used if left blank)

      Secondary Port Number                          []                  #

      Port Increment Range                           []                 +#

 

The last step failed with the following error message:

 

0042-358 niminit: The connect attribute may only be assigned  a service value of "shell" or "nimsh".

 

I checked the NIM client and confirmed it was configured for nimsh and it was fine. However, I did notice something odd when I ran the following command:

 

lpar1# egrep 'nimsh|nimaux' /etc/services

lpar1#

 

The entries for nimsh were missing from the /etc/services file!

 

Somebody had decided that these entries were not required and had simply removed them! Gee, thanks so much for that!

 

After adding the following entries back into the services file, everything started working again!

 

nimsh                    3901/tcp               # NIM Service Handler

nimsh                    3901/udp               # NIM Service Handler

nimaux                  3902/tcp                # NIMsh Auxiliary Port

nimaux                  3902/udp                # NIMsh Auxiliary Port

 

I’ve also encountered this error when there is another process (other than nimsh) using port 3901 or 3902.

 

Another error message you might confront, if those entries are either missing or commented out, is on the NIM master:

 

nimmast# nim -o showlog -a log_type=lppchk lpar1

 

0042-001 nim: processing error encountered on "master":

   0042-006 m_showlog: (From_Master) connect Error 0

 

poll: setup failure

 

I thought I’d also mention another error message that can potentially drive you insane (especially if you haven’t had your morning coffee!). The error doesn’t relate to nimsh at all but I thought I’d describe it anyway. The message appears when running the nim –o showlog command against a client LPAR.

 

nimmast# nim -o showlog  lpar1

0042-001 nim: processing error encountered on "master":

   0042-006 m_showlog: (From_Master) connect Error 0

 

0042-008 nimsh: Request denied – wronghostname

 

I’ve modified the output a little to make it easier to identify the problem. Can you see it? I thought so! Upon investigation you may find that the IP address for the NIM master is resolving to a different hostname on the client. For example:

 

On the NIM master:

nimmast# host nimmast

nimmast is 172.29.150.177

 

nimmast# host 172.29.150.177

nimmast is 172.29.150.177

 

nimmast# grep 177 /etc/hosts

172.29.150.177  nimmast

 

On the NIM client:

lpar1# host nimmast

nimmast is 172.29.150.177

 

lpar1# host 172.29.150.177

wronghostname is 172.29.150.177

 

lpar1# grep 172.29.150.177 /etc/hosts

172.29.150.177  wronghostname

172.29.150.177  nimmast

 

In this example, someone placed two host entries in /etc/hosts with the same IP address. The client was resolving the IP address to an incorrect hostname. This resulted in our nim –o showlog command failing.

I finally got access to a system where I could test RootVG WPARs a little more extensively than I have in the past.

 

If you are not sure what a RootVG WPAR is then I suggest you take a look at the following IBM documentation:

 

http://publib.boulder.ibm.com/infocenter/aix/v6r1/topic/com.ibm.aix.wpar/rootvg-wpars.htm

 

I performed the following test today, just to prove a theory. The result was interesting, so I thought I would share it with you on my blog.

 

First, I started with an AIX 7.1 (7100-00-01-1037) system that had one spare vscsi disk, hdisk5.

 

Global# lspv

hdisk4          00f6050a2cd79ef8                    rootvg          active

hdisk5          00f602736c1a2ad8                    None

 

I then created a volume group, logical volume and file system on hdisk5.

 

Global# mkvg -fy cgvg hdisk5

cgvg

 

Global# mklv -t jfs2 -ycglv cgvg 1

cglv

 

Global# crfs -vjfs2 -Ayes -d cglv -m /cg

File system created successfully.

65328 kilobytes total disk space.

New File System size is 131072

 

Global# mount /cg

 

Global# df -m /cg

Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on

/dev/cglv         64.00     63.67    1%        4     1% /cg

 

Global# lsvg -l cgvg

cgvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

cglv                jfs2       1       1       1    open/syncd    /cg

loglv00             jfs2log    1       1       1    open/syncd    N/A

 

I attempted to create a RootVG WPAR on hdisk5. Expecting (hoping) this would fail, as I still had an active file system in the VG.

 

It failed stating it was unable to overwrite hdisk5. Good.

 

Global# lswpar

 

Global# mkwpar -D rootvg=yes devname=hdisk5 -n rootvgwpar1

Creating workload partition's rootvg. Please wait...

mkwpar: 0960-621 Failed to create a workload partition's rootvg. Please use -O flag to overwrite hdisk5.

If restoring a workload partition, target disks should be in available state.

 

So I tried the command again, this time with the –O flag as suggested in the error message. It failed again stating that it could not remove cgvg from hdisk5. This was also good. At this point, an experienced AIX admin would look for the existence of a volume group on hdisk5. However, a junior AIX admin might not! :)

 

Global# mkwpar -O -D rootvg=yes devname=hdisk5 -n rootvgwpar1

mkwpar: 0960-620 Failed to remove cgvg on disk hdisk5.

 

Global# lsvg -l cgvg

cgvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

cglv                jfs2       1       1       1    open/syncd    /cg

loglv00             jfs2log    1       1       1    open/syncd    N/A

 

But what if the file systems were already unmounted prior to running mkwpar with the –O flag. What would happen?

 

So, to test my theory, I unmounted my file system so that the logical volumes in cgvg were all now closed.

 

Global# umount /cg

 

Global# lsvg -l cgvg

cgvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

cglv                jfs2       1       1       1    closed/syncd  /cg

loglv00             jfs2log    1       1       1    closed/syncd  N/A

 

This time the mkwpar command (with –O) happily blew away my logical volume, file system and volume group! Whoops! :)

 

Global#  mkwpar -O -D rootvg=yes devname=hdisk5 -n rootvgwpar1

Creating workload partition's rootvg. Please wait...

mkwpar: Creating file systems...

/

/admin

...etc...

devices.common.IBM.scsi.rte 7.1.0.0         ROOT        APPLY       SUCCESS

devices.fcp.disk.array.rte  7.1.0.0         ROOT        APPLY       SUCCESS

devices.fcp.disk.rte        7.1.0.0         ROOT        APPLY       SUCCESS

devices.fcp.tape.rte        7.1.0.0         ROOT        APPLY       SUCCESS

devices.scsi.disk.rte       7.1.0.0         ROOT        APPLY       SUCCESS

devices.tty.rte             7.1.0.0         ROOT        APPLY       SUCCESS

bos.mp64                    7.1.0.0         ROOT        APPLY       SUCCESS

bos.mp64                    7.1.0.1         ROOT        APPLY       SUCCESS

bos.net.tcp.client          7.1.0.0         ROOT        APPLY       SUCCESS

bos.net.tcp.client          7.1.0.1         ROOT        APPLY       SUCCESS

bos.perf.tune               7.1.0.0         ROOT        APPLY       SUCCESS

perfagent.tools             7.1.0.0         ROOT        APPLY       SUCCESS

bos.net.nfs.client          7.1.0.0         ROOT        APPLY       SUCCESS

bos.wpars                   7.1.0.0         ROOT        APPLY       SUCCESS

bos.wpars                   7.1.0.1         ROOT        APPLY       SUCCESS

bos.net.ncs                 7.1.0.0         ROOT        APPLY       SUCCESS

wio.common                  7.1.0.0         ROOT        APPLY       SUCCESS

Finished populating scratch file systems.

Workload partition rootvgwpar1 created successfully.

mkwpar: 0960-390 To start the workload partition, execute the following as root: startwpar [-v] rootvgwpar1

 

Global# lsvg

rootvg

 

Global# lspv

hdisk4          00f6050a2cd79ef8                    rootvg          active

hdisk5          00f602736c1a2ad8                    None

 

Global# lswpar -D

Name         Device Name      Type    Virtual Device  RootVG  Status

-----------------------------------------------------------------------

rootvgwpar1  /dev/null        pseudo                          EXPORTED

rootvgwpar1  /dev/tty         pseudo                          EXPORTED

rootvgwpar1  /dev/console     pseudo                          EXPORTED

rootvgwpar1  /dev/zero        pseudo                          EXPORTED

rootvgwpar1  /dev/clone       pseudo                          EXPORTED

rootvgwpar1  /dev/sad         clone                           EXPORTED

rootvgwpar1  /dev/xti/tcp     clone                           EXPORTED

rootvgwpar1  /dev/xti/tcp6    clone                           EXPORTED

rootvgwpar1  /dev/xti/udp     clone                           EXPORTED

rootvgwpar1  /dev/xti/udp6    clone                           EXPORTED

rootvgwpar1  /dev/xti/unixdg  clone                           EXPORTED

rootvgwpar1  /dev/xti/unixst  clone                           EXPORTED

rootvgwpar1  /dev/error       pseudo                          EXPORTED

rootvgwpar1  /dev/errorctl    pseudo                          EXPORTED

rootvgwpar1  /dev/audit       pseudo                          EXPORTED

rootvgwpar1  /dev/nvram       pseudo                          EXPORTED

rootvgwpar1  /dev/kmem        pseudo                          EXPORTED

rootvgwpar1  hdisk5           disk    hdisk0          yes     EXPORTED

 

Global# startwpar -v rootvgwpar1

Starting workload partition rootvgwpar1.

Mounting all workload partition file systems.

Mounting /wpars/rootvgwpar1

Mounting /wpars/rootvgwpar1/etc/objrepos/wboot

Mounting /wpars/rootvgwpar1/opt

Mounting /wpars/rootvgwpar1/usr

Loading workload partition.

Exporting workload partition devices.

hdisk5 Defined

Exporting workload partition kernel extensions.

Starting workload partition subsystem cor_rootvgwpar1.

0513-059 The cor_rootvgwpar1 Subsystem has been started. Subsystem PID is 2818228.

Verifying workload partition startup.

Return Status = SUCCESS.

 

Global# lswpar

Name         State  Type  Hostname     Directory           RootVG WPAR

-----------------------------------------------------------------------

rootvgwpar1  A      S     rootvgwpar1  /wpars/rootvgwpar1  yes

 

 

Global# clogin rootvgwpar1

 

rootvgwpar1# lspv

hdisk0          00f602736c1a2ad8                    rootvg          active

 

rootvgwpar1# lsvg -l rootvg

rootvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

hd4                 jfs2       2       2       1    open/syncd    /

hd11admin           jfs2       1       1       1    open/syncd    /admin

hd1                 jfs2       1       1       1    open/syncd    /home

hd3                 jfs2       2       2       1    open/syncd    /tmp

hd9var              jfs2       2       2       1    open/syncd    /var

 

This could be a trap for “first time” users of RootVG WPARs! So look out! :)

 

Apparently this is working as designed, as the –O flag is really only meant to be called by WPAR tools such as the WPAR Manager.

The man page for mkwpar states:

 

-O This flag is used to force the overwrite of an existing volume group on the given set of devices specified with the -D rootvg=yes flag directive. If not specified, the overwrite value defaults to FALSE. This flag should only be specified once, for its setting will be applied to all devices specified with the -D rootvg=yes flag directive.

 

Please take care when using the –O flag with the mkwpar command.

 

I was collecting a list of WWPNs for a bunch of new AIX LPARs I was installing from scratch. There were two ways I could find the WWPN for a virtual Fibre Channel adapter on a new LPAR i.e. one that did not yet have an operating system installed.

 

I started by checking the LPAR properties from the HMC (as shown below).

 

 

 

To speed things up I moved to the HMC command line tool, lssyscfg, to display the WWPNs (as shown below).

 

hscroot@hmc1:~> lssyscfg -r prof -m 750-2 -F virtual_fc_adapters --filter lpar_names=lpar1

"""32/client/2/vio2/30/c0507603a2410000,c0507603a2410001/0"",""33/client/2/vio2/31/c0507603a2410002,c0507603a2410003/0"",""30/client/1/vio1/30/c0507603a2410004,c0507603a2410005/0"",""31/client/1/vio1/31/c0507603a2410006,c0507603a2410007/0""" """30/client/1/vio1/30/c0507603a2410004,c0507603a2410005/0"",""31/client/1/vio1/31/c0507603a2410006,c0507603a2410007/0"""

 

 

This was good. I now had a list of WWPNs for each LPAR.

 

# cat lpar1_wwpns.txt

c0507603a292007c

c0507603a292007e

c0507603a2920078

c0507603a292007a

 

I gave these WWPNS to my SAN administrator so that he could manually “zone in” the LPARs on the SAN switches and allocate storage to each. He then, half-jokingly said, “Gee, it would be nice if you could insert the colons into the WWPNS for me! J”. Of course, this got me thinking and after a few minutes of playing with sed, I came up with a way to do this quickly.

 

# cat lpar1_wwpns.txt | sed 's/../&:/g;s/:$//'

c0:50:76:03:a2:92:00:7c

c0:50:76:03:a2:92:00:7e

c0:50:76:03:a2:92:00:78

c0:50:76:03:a2:92:00:7a

 

Now my list of WWPNs was ready to be cut’n’paste by my SAN admin. He was happy.

This is something that I experience on all new Power/AIX systems that I install:

 

System migrated to AIX 5.3 (or later) might experience double boot

When booting AIX Version 5.3 (or later) on a system that has previously been running an earlier release of AIX, you may notice that the system automatically reboots and restarts the boot process. This is how the firmware processes changed information in the boot image. This reboot also occurs if the process is reversed. A system previously running AIX 5.3 (or later) that is booting a release of AIX prior to 5.3 goes through the same process. This ″double boot″ occurs only once; if the stored value does not change, then the second boot does not occur. If you install AIX 5.3 (or later) and continue to use only that version, this double boot occurs once, and it occurs only if your system was running a pre-AIX 5.3 release before you boot AIX 5.3 (or later). Systems that are preinstalled with AIX 5.3 (or later) and use only that version do not experience the ″double boot.″

 

http://publib.boulder.ibm.com/infocenter/aix/v6r1/topic/com.ibm.aix.resources/RELNOTES/610_base_relnotes.pdf

 

Some will panic when this happens, but it’s not an issue. It only happens once, on the first installation of the first AIX or VIOS LPAR you install. After that you will never see it again.

I received an email from one of my customers recently that simply said:

 

“mate …on lpar30, for some reason the IBM.DRM and others are failing to start .. um .. any chance you could have a quick look at that ?”

 

So I asked, “OK, so this use to work right?”. To which I received a relatively confused reply, ”.....yep it did....actually no....it’s never worked....or has it???...I’m not sure...”.

 

Based on my experience, the most common issue that prevents DLPAR operations from working are network problems. Before diving into the deep end and trying to debug RSCT, it’s always best to start with the basics. For example, can you ping the HMC from the LPAR? Can you ping the LPAR from the HMC? If either of these tests fails, check the network configuration on both components before doing anything else.

 

On the HMC check the network settings first e.g.

 

·         Click on HMC Configuration and then Customize Network Settings.

– Verify the IP address, netmask, default gateway, network routes, DNS server are all set correctly.

– Check the LPAR communications box in HMC configuration screen for LAN adapter that is used for HMC-to-LPAR communication.

– By the way, unlike POWER4 systems, LPARs on POWER5 and POWER6 systems do not depend on host name resolution for DLPAR operations.

 

·         Check routing on the LPAR and the HMC.

– Use ping and the HMC’s Test Network Connectivity task to verify the LPAR and the HMC can communicate with each other.

 

If you check the network and you are happy that the LPAR and the HMC can communicate, then perhaps you need to re-initialise the RMC subsystems on the AIX LPAR. Run the following commands:

  

# /usr/sbin/rsct/bin/rmcctrl –z

# /usr/sbin/rsct/bin/rmcctrl –A

# /usr/sbin/rsct/bin/rmcctrl –p

 

Wait up to 5 minutes before trying DLPAR again. If DLPAR still doesn’t work i.e. the HMC is still reporting no values for DCaps, and the IBM.DRM subsystem still won’t start, try using the recfgct command.

 

hscroot@hmc1:~> lspartition -dlpar

.....

<#5> LPAR:<24*9117-MMB*10284FP, , 192.168.1.15>

       Active:<1>, OS:<AIX, 6.1, 6100-05-01-1016>, DCaps:<0x0>, CmdCaps:<0x0, 0x0>, PinnedMem:<768>

.....

 

# /usr/sbin/rsct/install/bin/recfgct

 

Wait 5 minutes. This should resolve your DLPAR issue. The IBM.DRM subsystem should now be active and there should be good (non-zero) values for DCaps:

 

# lssrc -g rsct_rm

Subsystem         Group            PID          Status

 IBM.DRM          rsct_rm          6881300      active

 IBM.CSMAgentRM   rsct_rm          7274530      active

 IBM.ServiceRM    rsct_rm          6029480      active

 IBM.AuditRM      rsct_rm          6357058      active

 IBM.ERRM         rsct_rm          4456566      active

 IBM.LPRM         rsct_rm          6946986      active

 

hscroot@hmc1:~> lspartition -dlpar

....

<#5> LPAR:<24*9117-MMB*10284FP, , 192.168.1.15>

       Active:<1>, OS:<AIX, 6.1, 6100-05-01-1016>, DCaps:<0xc5f>, CmdCaps:<0x1b, 0x1b>, PinnedMem:<994>

....

 

Only run the rmcctrl and recfgct commands if you believe something has become corrupt in the RMC configuration of the LPAR. The fastest way to fix a broken configuration or to clear out the RMC ACL files after cloning (via alt_disk migration) is to use the recfgct command.

 

These daemons should work “out of the box” and are not typically the cause of DLPAR issues. However, you can try stopping and starting the daemons when troubleshooting DLPAR issues.

 

The rmcctrl -z command just stops the daemons. The rmcctrl -A command ensures that the subsystem group (rsct) and the subsystem (ctrmc) objects are added to the SRC, and an appropriate entry added to the end of /etc/inittab and it starts the daemons.

 

The rmcctrl –p command enables the daemons for remote client connections i.e. from the HMC to the LPAR and vice versa.

 

If you are familiar with the System Resource Controller (SRC) you might be tempted to use stopsrc and startsrc commands to stop and start these daemons.

Do not do it; use the rmcctrl commands instead.

 

If /var is 100% full, use chfs to expand it. If there is no more space available, examine subdirectories and remove unnecessary files (for example, trace.*, core, and so forth). If /var is full, RMC subsystems may fail to function correctly.

 

The polling interval for the RMC daemons on the LPAR to check with the HMC daemons is 5-7 minutes; so you need to wait long enough for the daemons to start up and synchronize.

 

The Resource Monitoring and Control (RMC) daemons are part of the Reliable, Scalable Cluster Technology (RSCT) and are controlled by the System Resource Controller (SRC). These daemons run in all LPARs and communicate with equivalent RMC daemons running on the HMC. The daemons start automatically when the operating system starts and synchronize with the HMC RMC daemons.

 

The daemons in the LPARs and the daemons on the HMC must be able to communicate over the network for DLPAR operations to succeed. This is not the network connection between the managed system (FSP) and the HMC; it is the network connection between the operating system (AIX) in each LPAR and the HMC.

 

Note: Apart from rebooting, there is no way to stop and start the RMC daemons on the HMC.

 

The following links also contain some (out dated) information relating to DLPAR verification and troubleshooting. Even though it is quite old some of it is still relevant today and is good a place to start.

 

The most common reasons for failures with Dynamic Logical LPARing

http://www-01.ibm.com/support/docview.wss?rs=111&context=SWG10&q1=dcaps&uid=isg3T1010615&loc=en_US&cs=utf-8&lang=en

 

Dynamic LPAR tips and checklists for RMC authentication and authorization

http://www.ibm.com/developerworks/systems/articles/DLPARchecklist.html

 

Setting up the HMC/LPARs hostname and network (old but interesting)

http://computerstec.com/aixhowto/aixh18.htm

 

lsLPAR -dlpar, DCAPs values (old but still applies)

http://community.livejournal.com/eserver/18517.html

 

The previous link (above) provides some information relating to the values for DCaps and what they mean (also out dated):

 

0 - DR CPU capable  (can move CPUs)

1 - DR MEM capable  (can move memory)

2 - DR I/O capable  (can move I/O resources)

3 - DR PCI Bridge   (can move PCI bridges)

4 - DR Entitlement  (POWER 5 can change shared entitlement)

5 - Multiple DR CPU (AIX 5.3 can move 2+ CPUs at once)

0x3f = max, and 0xf is common for AIX 5.2

 

If you are interested in how HMC and LPAR authentication works with DLPAR, then read on. Otherwise, happy DLPARing!

 

HMC and LPAR authentication (RSCT authentication)

 

The diagram below outlines how the HMC and an LPAR authenticate with each other in order for DLPAR operations to work. RSCT authentication is used to ensure the HMC is communicating with the correct LPAR.

 

Authentication is the process of ensuring that another party is who it claims to be. Authorization is the process by which a cluster software component grants or denies resources based on certain criteria. The RSCT component that implements authorization is RMC. It uses access control list (ACL) files to control user access to resources.

 

The RMC component subsystem uses cluster security services to map the operating system user identifiers, specified in the ACL file, to network security identifiers to determine if the user has the correct permissions. This is performed by the identity mapping service, which uses information stored in the identity mapping files ctsec_map.global and ctsec_map.local.

 

 

 

The RSCT authorization process in detail:

 

1.    On the HMC: DMSRM pushes down the secret key and HMC IP address to NVRAM when it detects a new CEC; this process is repeated every five minutes. Each time an HMC is rebooted or DMSRM is restarted, a new key is used.

 

2.       On the AIX LPAR: CSMAgentRM, through RTAS (Run-time Abstraction Services), reads the key and HMC IP address out from NVRAM. It will then authenticate the HMC. This process is repeated every five minutes on a LPAR to detect a new HMCs and if the key has changed. An HMC with a new key is treated as a new HMC and will go though the authentication and authorization processes again.

 

3.       On the AIX LPAR: After authenticating the HMC, CSMAgentRM will contact the DMSRM on the HMC to create a ManagedNode resource in order to identify itself as a LPAR of this HMC. CSMAgentRM then creates a compatible ManagementServer resource on AIX. This can be displayed on AIX with the lssrsrc command. e.g.

 

root@aix6 / # lsrsrc "IBM.ManagementServer"

Resource Persistent Attributes for IBM.ManagementServer

resource 1:

        Name             = "192.168.1.244"

        Hostname         = "192.168.1.244"

        ManagerType      = "HMC"

        LocalHostname    = "10.153.3.133"

        ClusterTM        = "9078-160"

        ClusterSNum      = ""

        ActivePeerDomain = ""

        NodeNameList     = {"aix6"}

 

4.       On the AIX LPAR: After the creation of the ManagedNode and ManagementServer resources on the HMC and AIX respectively, CSMAgentRM grants HMC permission to access necessary resource classes on the LPAR. After granting the HMC permission, CSMAgentRM will change its ManagedNode, on the HMC, Status to 1. (It should be noted that without proper permission on AIX, the HMC would be able to establish a session with the LPAR but will not be able to query for OS information, DLPAR capabilities, or execute DLPAR commands afterwards.)

 

5.       On the HMC: After the ManagedNode Status is changed to 1, LparCmdRM establishes a session with the LPAR, queries for operating system information and DLPAR capabilities, notifies CIMOM about the DLPAR capabilities of the LPAR, and then waits for the DLPAR commands from users.

I’ve written about multibos before, here and here. But recently I started experimenting with multibos mksysb migration. A customer asked me how this worked and apart from a high-level view I wasn’t able to provide any real world experience, so I thought I’d give it a try. What follows is just a ‘brain dump’ from my quick test.

 

First of all this isn’t really a migration. It just simply populates a second instance of AIX with a higher-version. It doesn’t really migrate (or merge) your existing configuration into the second instance. So I’m not sure how useful this feature really is right now.

 

Starting with 5.3 TL9 you can add a 6.1 TL2 (or above) instance. This is done with the new –M flag. You must be running with the 64bit kernel.

This isn’t really a migration because it populates the second instance using a mksysb based on the new release.

In 6.1 TL2 a new flag (-M) was added to the mksysb command which allows you to create a mksysb for use with multibos. It creates a backup of BOS (/, /usr, /var, /opt).
bos.alt_disk_install.boot_images must be installed.

To preview the multibos migration
multibos –M <location of mksysb> -spX

To perform the multibos migration
multibos –M <location of mksysb> -sX

It is not advised to run in this environment for an extended period of time. There could be problems if tfactor or maps are used. Be aware that 6.1 specific attributes may not be reflected in the standby instance.

 

So in my lab environment I have two AIX LPARs. One is running AIX 6.1 and the other running AIX 7.1.

 

First I take a mksysb (with the –M flag) of the AIX 7.1 system to a file. This file will be called by multibos to populate the second instance.

 

aix7[/] > mksysb -Mie /data/aix7-mksysb

 

Creating information file (/image.data) for rootvg.

 

Creating list of files to back up.

 

Backing up 71643 files.....

 

71643 of 71643 files (100%)

0512-038 mksysb: Backup Completed Successfully.

 

aix7[/] > ls -ltr /data

total 4276112

drwxr-xr-x    2 root     system          256 Feb 21 20:59 lost+found

-rw-r--r--    1 root     system   2189363200 Feb 21 21:06 aix7-mksysb

 

I copied this file over to my AIX 6.1 system. This was the system that was to be ‘migrated’. The next step was to perform a preview of the multibos operation.

 

root@aix6 /# multibos -sXp -M /data/aix7-mksysb

Initializing multibos methods ...

Initializing log /etc/multibos/logs/op.alog ...

Gathering system information ...

 

+-----------------------------------------------------------------------------+

Preview

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

Validating install images location ./aix7-mksysb

Processing preview information ...

 

ACTIVE LV:          hd4

STANDBY LV:         bos_hd4

TYPE:               jfs2

ACTIVE FS:          /

STANDBY FS:         /bos_inst

ACTION:             Setup

STATE:              mounted

 

ACTIVE LV:          hd2

STANDBY LV:         bos_hd2

TYPE:               jfs2

ACTIVE FS:          /usr

STANDBY FS:         /bos_inst/usr

ACTION:             Setup

STATE:              mounted

 

ACTIVE LV:          hd9var

STANDBY LV:         bos_hd9var

TYPE:               jfs2

ACTIVE FS:          /var

STANDBY FS:         /bos_inst/var

ACTION:             Setup

STATE:              mounted

 

ACTIVE LV:          hd10opt

STANDBY LV:         bos_hd10opt

TYPE:               jfs2

ACTIVE FS:          /opt

STANDBY FS:         /bos_inst/opt

ACTION:             Setup

STATE:              mounted

 

ACTIVE LV:          hd5

STANDBY LV:         bos_hd5

TYPE:               boot

ACTIVE FS:          None

STANDBY FS:         None

ACTION:             Setup

STATE:              closed

 

ACTIVE LV:          hd8

STANDBY LV:         bos_hd8

TYPE:               jfs2log

ACTIVE FS:          None

STANDBY FS:         None

ACTION:             Setup

STATE:              open

 

Log file is /etc/multibos/logs/op.alog

Return Status = SUCCESS

 

With a successful preview out of the way, I initiated the real multibos operation.

 

# multibos -sX -M /data/aix7-mksysb

 

Gathering system information ...

 

+-----------------------------------------------------------------------------+

Setup Operation

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

Validating install images location ./aix7-mksysb

New volume on /data/aix7-mksysb:

Cluster size is 51200 bytes (100 blocks).

The volume number is 1.

The backup date is: Mon Feb 21 21:05:39 CST 2011

Files are backed up by name.

The user is root.

x        13007 ./image.data

x       168589 ./tmp/vgdata/rootvg/backup.data

The total size is 181596 bytes.

The number of restored files is 2.

 

+-----------------------------------------------------------------------------+

Logical Volumes

+-----------------------------------------------------------------------------+

Creating standby BOS logical volume bos_hd5

Creating standby BOS logical volume bos_hd4

Creating standby BOS logical volume bos_hd2

Creating standby BOS logical volume bos_hd9var

Creating standby BOS logical volume bos_hd10opt

Creating standby BOS logical volume bos_hd8

 

+-----------------------------------------------------------------------------+

File Systems

+-----------------------------------------------------------------------------+

Creating all standby BOS file systems ...

Creating standby BOS file system /bos_inst on logical volume bos_hd4

Creating standby BOS file system /bos_inst/usr on logical volume bos_hd2

Creating standby BOS file system /bos_inst/var on logical volume bos_hd9var

Creating standby BOS file system /bos_inst/opt on logical volume bos_hd10opt

 

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Mounting all standby BOS file systems ...

Mounting /bos_inst

Mounting /bos_inst/usr

Mounting /bos_inst/var

Mounting /bos_inst/opt

 

+-----------------------------------------------------------------------------+

BOS Files

+-----------------------------------------------------------------------------+

Including files for file system /

Including files for file system /usr

Including files for file system /var

Including files for file system /opt

 

Copying files using backup/restore utilities ...

New volume on /data/aix7-mksysb:

Cluster size is 51200 bytes (100 blocks).

The volume number is 1.

The backup date is: Mon Feb 21 21:05:39 CST 2011

Files are backed up by name.

The user is root.

x         6086 ./bosinst.data

x           11 ./tmp/vgdata/rootvg/image.info

x        13007 ./image.data

x       168589 ./tmp/vgdata/rootvg/backup.data

x            0 ./

x           47 ./.profile

x           63 ./.rhosts

x        13192 ./.sh_history

x            0 ./.ssh

x         1647 ./.ssh/authorized_keys

x            0 ./.topasrecrc

x            0 ./.topasrecrc/topasrec.lock

x       141722 ./.topasrecrc/topasrec.log1

x            0 ./.topasrecrc/topasrec.log2

x          108 ./.vi_history

x            0 ./admin

x            0 ./audit

x            8 ./bin

x            0 ./cdat

x           59 ./cdat/cdat.xml

x            0 ./cg

x            0 ./cgps

x            0 ./dev

x            0 ./dev/.SRC-unix

x            0 ./dev/IPL_rootvg

x            0 ./dev/__pv17.0.10617024

x            0 ./dev/__vg10

x            0 ./dev/audit

.....BLAH.....

x ./dev/zero

Creating standby BOS boot image on boot logical volume bos_hd5

254+0 records in.

254+0 records out.

179+1 records in.

177+1 records out.

saving to '/dev/bos_hd5'

44 CuDv objects to be saved

130 CuAt objects to be saved

21 CuDep objects to be saved

7 CuVPD objects to be saved

351 CuDvDr objects to be saved

2 CuPath objects to be saved

0 CuPathAt objects to be saved

0 CuData objects to be saved

Number of bytes of data to save = 18716

Compressing data

Compressed data size is = 6393

        bi_start     = 0x400

        bi_size      = 0x1677200

        bd_size      = 0x1657200

        ram FS start = 0x8c3570

        ram FS size  = 0xd93abe

        sba_start    = 0x1657600

        sba_size     = 0x20000

        sbd_size     = 0x0

Checking boot image size:

        new save base byte cnt = 0x18fd

Wrote 6397 bytes

Successful completion

 

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Unmounting all standby BOS file systems ...

Unmounting /bos_inst/opt

Unmounting /bos_inst/var

Unmounting /bos_inst/usr

Unmounting /bos_inst

 

+-----------------------------------------------------------------------------+

Bootlist Processing

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

Setting bootlist to logical volume bos_hd5 on hdisk0.

ATTENTION: firmware recovery string for standby BLV (bos_hd5):

boot /vdevice/v-scsi@30000016/disk@8100000000000000:4

ATTENTION: firmware recovery string for standby BLV (bos_hd5):

boot /vdevice/v-scsi@30000017/disk@8100000000000000:4

ATTENTION: firmware recovery string for active BLV (hd5):

boot /vdevice/v-scsi@30000016/disk@8100000000000000:2

ATTENTION: firmware recovery string for active BLV (hd5):

boot /vdevice/v-scsi@30000017/disk@8100000000000000:2

 

Log file is /etc/multibos/logs/op.alog

Return Status = SUCCESS

 

Now that the second instance had been created, I thought I’d try to start a multibos shell and access the AIX 7.1 environment. I didn’t think this would work but I tried my luck anyway!

 

root@aix6 /# multibos -S

Initializing multibos methods ...

Initializing log /etc/multibos/logs/op.alog ...

Gathering system information ...

 

+-----------------------------------------------------------------------------+

Multibos Shell Operation

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

 

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Mounting all standby BOS file systems ...

Mounting /bos_inst

Mounting /bos_inst/usr

Mounting /bos_inst/var

Mounting /bos_inst/opt

 

+-----------------------------------------------------------------------------+

Multibos Root Shell

+-----------------------------------------------------------------------------+

Starting multibos root shell ...

multibos: 0565-110 Cannot start multibos shell due to incompatible levels of active and standby BOS.

 

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Unmounting all standby BOS file systems ...

Unmounting /bos_inst/opt

Unmounting /bos_inst/var

Unmounting /bos_inst/usr

Unmounting /bos_inst

 

Log file is /etc/multibos/logs/op.alog

Return Status: FAILURE

 

Upon checking my bootlist output, I noticed (as expected) that the list now contained two extra entries for bos_hd5. These were the boot logical volume entries for the second instance. If I was to boot from this LV I’d be booting into AIX 7.1. Cool.

 

root@aix6 /# bootlist -m normal -o

hdisk0 blv=bos_hd5

hdisk0 blv=bos_hd5

hdisk0 blv=hd5

hdisk0 blv=hd5

 

So at this point, I’d created a second instance of AIX running 7.1. My current version of (running) AIX was AIX 6.1. All I had to do now was reboot the LPAR and let it restart as an AIX 7.1 system.

 

root@aix6 /# oslevel -s

6100-01-05-0920

 

root@aix6 / # shutdown –Fr

 

The LPAR rebooted successfully and I found I was now running AIX 7.1, just as I’d hoped.

 

aix6[/] > oslevel -s

7100-00-01-1037

 

If I wanted to go back to AIX 6.1, I would change my bootlist setting again and restart the LPAR.

 

aix6[/] > bootlist -m normal -o

hdisk0 blv=bos_hd5 pathid=0

hdisk0 blv=bos_hd5 pathid=1

hdisk0 blv=hd5 pathid=0

hdisk0 blv=hd5 pathid=1

 

aix6[/] > bootlist -m normal hdisk0 blv=hd5 pathid=0 hdisk0 blv=hd5 pathid=1 hdisk0 blv=bos_hd5 pathid=0 hdisk0 blv=bos_hd5 pathid=1

aix6[/] > bootlist -m normal -o

hdisk0 blv=hd5 pathid=0

hdisk0 blv=hd5 pathid=1

hdisk0 blv=bos_hd5 pathid=0

hdisk0 blv=bos_hd5 pathid=1

 

aix6[/] > shutdown –Fr

 

root@aix6 /# oslevel -s

6100-01-05-0920

 

Now that I’ve actually tried this method of migration, I’m not sure I’d actually use it in its current form.

 

Although the migration keeps my hostname and IP address, the file systems are not shared between instances. Most of the target systems configuration is not retained. For example, any user accounts I create on my AIX 6.1 system would also need to be created on the existing AIX.7.1 system which I used to create the AIX 7.1 mksysb image. It reminds me a little of a preservation install.

 

More information on multibos:

 

http://www.aixmind.com/?p=1192

 

Recently I had the pleasure of configuring a couple of POWER7 720s for a customer. Each 720 was to host roughly 12 LPARs each. There were two VIO servers and a NIM server per 720.

 

Everything went along nicely and according to plan. In a few days we had both systems built. My steps for building the systems were almost identical to those described by Rob McNelly in a recent post.

All four VIO servers were running the latest VIOS code i.e. 2.2.0.10-FP-24 SP-01. All the client LPARs were running AIX 6.1 TL6 SP3. Each VIOS was configured with two FC paths to the SAN and the SAN storage device was an IBM DS5020.

 

Native AIX MPIO was in use on the VIOS and the AIX LPARs. I did not deploy SDDPCM on the VIOS as this is currently unsupported with the DS5020.

Once the LPAR builds were complete we performed a number of “integration tests”. These typically involve disconnecting network and SAN cables from each VIOS and observing how the VIOS and LPARs respond to and recover from these types of conditions.

 

One of the integration tests required that BOTH fibre cables be disconnected from the first VIOS and confirm that the client LPARs were not impacted i.e. that all I/O travelled via the second VIOS.

During the test we notice the following:

 

• I/O on the client LPAR would hang for approximately 5 minutes. Our test was simple enough, create a file in a file system. Eventually the I/O would continue after the 5 minute delay.

 

• The VIOS became sluggish and took some time to respond. The lspath command would take a very long time to return (or in some cases it would never return). During one test, the VIOS actually hung and had to be restarted (however we were not able to reproduce this issue again).

 

• When we reconnected both the cables, the paths would recover on the first VIOS after another 5 minutes. On the client however, it took roughly 20 minutes for the paths to recover. And yes, the hcheck_interval was set to 60 on all disks.

What's even more puzzling was that if we simply rebooted the first VIOS, everything worked as expected i.e. the client LPARs were not impacted, I/O continued as normal and when the first VIOS was back up, the paths on the client LPARs recovered quickly.

After doing some research, we discovered the following post on the IBM developerWorks AIX forum:

 

http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14472352&tstart=0

 

This post highlighted a number of things we needed to check and also confirmed several decisions we’d made during the design process, such as SDDPCM not being supported with DS5020 storage and VIOS (this was good as some people were starting to believe we should have installed SDDPCM to resolve this problem. I’d only be happy to do this if it was a supported combination and it’s not).

 

Finally we found the following IBM tech note that related directly to our issue.

 
IZ66020: ACTIVE/PASSIVE PCM CONTROLLER HCHECK SUPPORT
https://www-304.ibm.com/support/docview.wss?uid=isg1IZ66020

 

The following statement seemed to match our exact problem.

 

"For active/passive storage device, such as DS3K, DS4K, or DS5K if complete access is lost to the storage device, then it may take greater than 5 minutes to fail I/O. This feature is for Active/Passive storage devices, which are running with the AIX Default A/P PCM. This includes DS3K, DS4K, and DS5K family of devices.”

The new feature was described as follows.

 

“Added feature which health checks controllers, when an enabled path becomes unavailable, due to transport problems.  By default this feature is DISABLED.  To enable this feature set the following ODM attributes for the active/passive storage device.  Enabling this feature, results in faster I/O failure times.                                                      

cntl_delay_time:                                                        
Is the amount of time in seconds the storage device's controller(s) will be health checked after a transport failure.  At the end of this period, if no paths are detected as good, then all pending and subsequent I/O to the device will be failed, until the device health checker detects a failed path has returned.                                             
                                                                        
cntl_hcheck_int:                                                        
The first controller health check will only be issued after a storage fabric transport failure had been detected. cntl_hcheck_int is the amount of time in seconds, which the next controller health check command will be issued. This value must be less than the cntl_delay_time  (unless set to "0", disabled).                                                  

If you wish to allow the storage device 30 seconds to come back on the fabric (after leaving the fabric),  then you can set cntl_delay_time=30 and cntl_hcheck_int=2.     

         
The device, /dev/hdisk#, must not be in use, when setting the ODM values (or the chdev "-P" option must be used,  which requires a reboot).                                               
                                                                        
CAUTION: There are cases where the storage device may  reboot both of the controllers and become inaccessible for a period of time. If the controller health check sequence is enabled, then this may result in an I/O failure.  It is recommended to to make sure you have an mirrored volume           
to failover to, if you are running with controller health check enabled (especially with under 60 second cntl_delay_time). “

                     

And as I suspected the issue was related to the type of storage we were using. It appears the I/O delay was attributed to the following attributes on the DS5020 hdisks on the VIOS:

 

cntl_delay_time 0  Controller Delay Time            True
cntl_hcheck_int 0  Controller Health Check Interval True

 

Based on the tech note, I attempted several tests with various values for both parameters e.g.

 

$ chdev -dev hdiskX -attr cntl_delay_time=30                        
$ chdev -dev hdiskX -attr cntl_hcheck_int=2  

 

After making the changes to the hdisks on all VIOS, I performed the same test i.e. disconnected BOTH fibre cables from the first VIOS and continued to write a file to a file system on the client LPARs. By modifying these values on all the DS5020 disks, on all the VIO servers, the I/O delay was reduced to seconds rather than five minutes!

 

The following attributes were used for the hdisks and adapters in the final configuration.

 

On the VIO servers:

 

$ lsdev -type disk | grep DS5020

hdisk3           Available   MPIO DS5020 Disk

hdisk4           Available   MPIO DS5020 Disk

hdisk5           Available   MPIO DS5020 Disk

hdisk6           Available   MPIO DS5020 Disk

hdisk7           Available   MPIO DS5020 Disk

hdisk8           Available   MPIO DS5020 Disk

hdisk9           Available   MPIO DS5020 Disk

hdisk10          Available   MPIO DS5020 Disk

hdisk11          Available   MPIO DS5020 Disk

hdisk12          Available   MPIO DS5020 Disk

hdisk13          Available   MPIO DS5020 Disk

hdisk14          Available   MPIO DS5020 Disk

hdisk15          Available   MPIO DS5020 Disk

hdisk16          Available   MPIO DS5020 Disk

hdisk17          Available   MPIO DS5020 Disk

hdisk18          Available   MPIO DS5020 Disk

hdisk19          Available   MPIO DS5020 Disk

hdisk20          Available   MPIO DS5020 Disk

hdisk21          Available   MPIO DS5020 Disk

hdisk22          Available   MPIO DS5020 Disk

hdisk23          Available   MPIO DS5020 Disk

hdisk24          Available   MPIO DS5020 Disk

hdisk25          Available   MPIO DS5020 Disk

hdisk26          Available   MPIO DS5020 Disk

hdisk27          Available   MPIO DS5020 Disk

hdisk28          Available   MPIO DS5020 Disk

hdisk29          Available   MPIO DS5020 Disk

hdisk30          Available   MPIO DS5020 Disk

hdisk31          Available   MPIO DS5020 Disk

hdisk32          Available   MPIO DS5020 Disk

 

 

$ lsdev -dev hdisk3 -attr

attribute       value                                                          description                      user_settable

 

PCM             PCM/friend/otherapdisk                                         Path Control Module              False

PR_key_value    none                                                           Persistant Reserve Key Value     True

algorithm       fail_over                                                      Algorithm                        True

autorecovery    no                                                             Path/Ownership Autorecovery      True

clr_q           no                                                             Device CLEARS its Queue on error True

cntl_delay_time 15                                                             Controller Delay Time            True

cntl_hcheck_int 2                                                              Controller Health Check Interval True

dist_err_pcnt   0                                                              Distributed Error Percentage     True

dist_tw_width   50                                                             Distributed Error Sample Time    True

hcheck_cmd      inquiry                                                        Health Check Command             True

hcheck_interval 60                                                             Health Check Interval            True

hcheck_mode     nonactive                                                      Health Check Mode                True

location                                                                       Location Label                   True

lun_id          0x11000000000000                                               Logical Unit Number ID           False

lun_reset_spt   yes                                                            LUN Reset Supported              True

max_retry_delay 60                                                             Maximum Quiesce Time             True

max_transfer    0x40000                                                        Maximum TRANSFER Size            True

node_name       0x20040080e5187564                                             FC Node Name                     False

pvid            00f6482f7869e92d0000000000000000                               Physical volume identifier       False

q_err           yes                                                            Use QERR bit                     True

q_type          simple                                                         Queuing TYPE                     True

queue_depth     24                                                             Queue DEPTH                      True

reassign_to     120                                                            REASSIGN time out value          True

reserve_policy  no_reserve                                                     Reserve Policy                   True

rw_timeout      30                                                             READ/WRITE time out value        True

scsi_id         0x11600                                                        SCSI ID                          False

start_timeout   60                                                             START unit time out value        True

unique_id       3E21360080E50001875DE000005224D2CD63B0F1814      FAStT03IBMfcp Unique device identifier         False

ww_name         0x20150080e5187564                                             FC World Wide Name               False

 

 

$ lsdev -dev fscsi0 -attr

attribute    value     description                           user_settable

 

attach       switch    How this adapter is CONNECTED         False

dyntrk       yes       Dynamic Tracking of FC Devices        True

fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True

scsi_id      0x11400   Adapter SCSI ID                       False

sw_fc_class  3         FC Class for Fabric                   True

 

 

$ r oem

oem_setup_env

# manage_disk_drivers

Device           Present Driver     Driver Options

2810XIV          AIX_AAPCM          AIX_AAPCM,AIX_non_MPIO

DS4100           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4200           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4300           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4500           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4700           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4800           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS3950           AIX_APPCM          AIX_APPCM

DS5020           AIX_APPCM          AIX_APPCM

DS5100/DS5300AIX_APPCM AIX_APPCM          AIX_APPCM

DS3500           AIX_APPCM          AIX_APPCM

Usage :

        manage_disk_drivers [-l]

        manage_disk_drivers -d device -o driver_option

        For entries with multiple model names use the first one listed.

        Ex. DS5100/DS5300 use DS5100.

        manage_disk_drivers –h

 

# mpio_get_config -Av

Frame id 0:

    Storage Subsystem worldwide name: 609e50018345de00004da7998

    Controller count: 2

    Partition count: 1

    Partition 0:

    Storage Subsystem Name = 'MyApp-DS5020'

        hdisk      LUN #   Ownership          User Label

        hdisk3        17   B (preferred)      LPAR1

        hdisk4        18   A (preferred)      LPAR1datavg

        hdisk5        19   A (preferred)      LPAR1appvg

        hdisk6        16   B (preferred)      LPAR2

        hdisk7        20   A (preferred)      LPAR2datavg

        hdisk8        21   B (preferred)      LPAR2data3vg

        hdisk9        15   A (preferred)      LPAR3

        hdisk10       14   B (preferred)      LPAR4

        hdisk11       22   B (preferred)      LPAR5

        hdisk12       23   A (preferred)      LPAR6

        hdisk13       13   B (preferred)      LPAR6datavg

        hdisk14       12   A (preferred)      LPAR7

        hdisk15       24   B (preferred)      LPAR7datavg

        hdisk16       25   A (preferred)      LPAR8

        hdisk17       11   B (preferred)      LPAR8datavg

        hdisk18       10   A (preferred)      LPAR8datavg

        hdisk19        9   B (preferred)      LPAR8datavg

        hdisk20        8   A (preferred)      LPAR8datavg

        hdisk21       26   A (preferred)      LPAR9

        hdisk22        7   A (preferred)      LPAR9datavg

        hdisk23        6   B (preferred)      LPAR9datavg

        hdisk24       27   B (preferred)      LPAR9appvg

        hdisk25       28   A (preferred)      LPAR9binvg

        hdisk26        5   A (preferred)      LPAR10

        hdisk27        4   B (preferred)      LPAR10datavg

        hdisk28       29   B (preferred)      LPAR10datavg

        hdisk29       30   A (preferred)      LPAR11

        hdisk30        3   A (preferred)      LPAR11datavg

        hdisk31       32   B (preferred)      LPAR12

        hdisk32       50   B (preferred)      LPAR12datavg

 

On the VIO clients:

 

- Changed the following attributes for all virtual SCSI disks to:

 

# lsattr -El hdisk0

PCM             PCM/friend/vscsi                 Path Control Module        False

algorithm       fail_over                        Algorithm                  True

hcheck_cmd      test_unit_rdy                    Health Check Command       True

hcheck_interval 60                               Health Check Interval      True

hcheck_mode     nonactive                        Health Check Mode          True

max_transfer    0x40000                          Maximum TRANSFER Size      True

pvid            00f6482f7869e92d0000000000000000 Physical volume identifier False

queue_depth     24                               Queue DEPTH                True

reserve_policy  no_reserve                       Reserve Policy             True

 

- Changed the following attributes for all virtual SCSI adapters to:

 

# lsattr -El vscsi0

vscsi_err_recov fast_fail N/A                       True

vscsi_path_to   30        Virtual SCSI Path Timeout True

 

Note: With regard to SDDPCM support for DS5020 with VIOS, I’ve been referring to the following IBM website:

 

http://www-01.ibm.com/support/docview.wss?uid=ssg1S4000201

 

The site has a link to the SDDPCM readme file for DS5020 storage (under SDDPCM Package for DS5000):

 

ftp://ftp.software.ibm.com/storage/subsystem/aix/2.5.2.0/sddpcm.readme.2.5.2.0.txt

 

The readme states:

 

Note: VIOS is not supported with SDDPCM on DS4000/DS5000/DS5020/DS3950 subsystem devices.

 

During our research we had this confirmed by IBM support that, at this time, SDDPCM is not supported
on VIOS with DS5000.

 

I hope this helps others that may be about to implement this type of storage with a VIOS.

Last year I wrote an article on using nimadm to migrate to AIX 6.1.

The procedures outlined how to migrate a live system with nimadm and a spare disk.

Another method that I have found to be extremely useful is the migration of mksysb images from one version of AIX to another using (you guessed it) nimadm.

In the following example I migrated an AIX 6.1 mksysb image to AIX 7.1 and then used that image to install a new LPAR via NIM.

The source LPAR was running AIX 6.1 TL6 SP3. I took a mksysb of this system to my NIM master.

# oslevel -s

6100-06-03-1048

#

 

The NIM mksysb resource was named cg-aix61

 

root@nim1 / # lsnim -t mksysb

vio1-mksysb     resources       mksysb

cg-aix61        resources       mksysb

 

root@nim1 / # lsnim -l cg-aix61

cg-aix61:

   class         = resources

   type          = mksysb

   arch          = power

   Rstate        = ready for use

   prev_state    = unavailable for use

   location      = /export/mksysb/cg-aix61

   version       = 6

   release       = 1

   mod           = 6

   oslevel_r     = 6100-06

   alloc_count   = 0

   server        = master

   creation_date = Thu Jan 6 10:00:22 2011

   source_image  = cg

root@nim1 / #

 

To migrate the mksysb image from AIX 6.1 to AIX 7.1, I ran the following nimadm command:

 

root@nim1 / # nimadm -T cg-aix61 -O /export/mksysb/cg-aix71 -s spotaix7tl0sp2 -l aix7tl0sp2 -j nimadmvg -Y -N cg-aix71

 

The flags indicate the following:

 

-T specifies the existing AIX 6.1 NIM mksysb resource.

-O specifies the output location for the migrated mksysb resource.

-s indicates the AIX 7.1 NIM SPOT resource for the migration.

-l indicates the AIX 7.1 NIM lpp_source resource for the migration.

-j identifies the volume group that will be used on the NIM master to create file systems.

-Y Agrees to required software license agreements for software to be installed.

-N specifies the name of the new AIX 7.1 NIM mksysb resource to be created after migration.

 

Before running this command I made sure that I had installed the bos.alt_disk_install.rte fileset into my AIX 7.1 SPOT. If this fileset was not installed I would have received the following message immediately after issuing the nimadm command:

 

0042-001 nim: processing error encountered on "master":

   /usr/bin/lslpp: Fileset bos.alt_disk_install.rte not installed.

 

0505-204 nimadm: SPOT spotaix7tl0sp2 does not have bos.alt_disk_install.rte installed.

0505-205 nimadm: The level of bos.alt_disk_install.rte installed in SPOT

spotaix7tl0sp2 (0.0.0.0) does not match the NIM master's level (7.1.0.2).

Cleaning up alt_disk_migration on the NIM master.

 

I also had an AIX 7.1 lpp_source and SPOT ready and waiting on my NIM master.

 

root@nim1 / # lsnim -t lpp_source

aix61tl6sp3     resources       lpp_source

openssh         resources       lpp_source

aix7tl0sp2      resources       lpp_source

 

root@nim1 / # lsnim -t spot

vio1-spot           resources       spot

spotaix61tl6sp3     resources       spot

spotaix7tl0sp2      resources       spot

 

OK, back to my nimadm operation. Upon initiating the migration I saw the following output on my screen (click here for a copy of the complete log file):

 

root@nim1 / # nimadm -T cg-aix61 -O /export/mksysb/cg-aix71 -s spotaix7tl0sp2 -l aix7tl0sp2 -j nimadmvg -Y -N cg-aix71

Initializing the NIM master.

Verifying alt_disk_migration eligibility.

Initializing log: /var/adm/ras/alt_mig/cg-aix61_alt_mig.log

Starting Alternate Disk Migration.

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 1.

+-----------------------------------------------------------------------------+

Processing target mksysb resource cg-aix61.

Restoring /image.data from mksysb image.

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 2.

+-----------------------------------------------------------------------------+

Creating nimadm cache file systems on volume group nimadmvg.

Checking for initial required migration space.

Creating cache file system /cg-aix61_mm_alt/alt_inst

Creating cache file system /cg-aix61_mm_alt/alt_inst/admin

Creating cache file system /cg-aix61_mm_alt/alt_inst/home

Creating cache file system /cg-aix61_mm_alt/alt_inst/opt

Creating cache file system /cg-aix61_mm_alt/alt_inst/tmp

Creating cache file system /cg-aix61_mm_alt/alt_inst/usr

Creating cache file system /cg-aix61_mm_alt/alt_inst/var

Creating cache file system /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 3.

+-----------------------------------------------------------------------------+

Syncing mksysb data to cache ...

 

.....etc......

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 8.

+-----------------------------------------------------------------------------+

Creating client boot image.

bosboot: Boot image is 45709 512 byte blocks.

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 9.

+-----------------------------------------------------------------------------+

Adjusting client file system sizes ...

Adjusting size for /

Adjusting size for /admin

Adjusting size for /home

Adjusting size for /opt

Adjusting size for /tmp

Adjusting size for /usr

Adjusting size for /var

Adjusting size for /var/adm/ras/livedump

Backing up cache data to mksysb file /export/mksysb/cg-aix71 ...

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 10.

+-----------------------------------------------------------------------------+

Unmounting client mounts on the NIM master.

forced unmount of /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

forced unmount of /cg-aix61_mm_alt/alt_inst/var

forced unmount of /cg-aix61_mm_alt/alt_inst/usr

forced unmount of /cg-aix61_mm_alt/alt_inst/tmp

forced unmount of /cg-aix61_mm_alt/alt_inst/opt

forced unmount of /cg-aix61_mm_alt/alt_inst/home

forced unmount of /cg-aix61_mm_alt/alt_inst/admin

forced unmount of /cg-aix61_mm_alt/alt_inst

Removing nimadm cache file systems.

Removing cache file system /cg-aix61_mm_alt/alt_inst

Removing cache file system /cg-aix61_mm_alt/alt_inst/admin

Removing cache file system /cg-aix61_mm_alt/alt_inst/home

Removing cache file system /cg-aix61_mm_alt/alt_inst/opt

Removing cache file system /cg-aix61_mm_alt/alt_inst/tmp

Removing cache file system /cg-aix61_mm_alt/alt_inst/usr

Removing cache file system /cg-aix61_mm_alt/alt_inst/var

Removing cache file system /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 11.

+-----------------------------------------------------------------------------+

Defining NIM mksysb resource ...

New NIMmksysb resource name is cg-aix71

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 12.

+-----------------------------------------------------------------------------+

Cleaning up alt_disk_migration on the NIM master.

root@nim1 / #

 

For a description of all the phases in a nimadm operation, I suggest you refer to my previous nimadm article or the AIX infocenter (links below).

 

Once the migration had completed, I found a new mksysb file in /export/mksysb (as expected). The file was named cg-aix71. And a new NIM mksysb resource was now available.

 

root@nim1 / # ls -ltr /export/mksysb/

total 30793928

drwxr-xr-x    2 root     system          256 Dec 21 10:40 lost+found

-rw-r--r--    1 root     system   4893952000 Dec 21 14:42 vio1-mksysb

-rw-r--r--    1 root     system   2323097600 Jan 06 10:00 cg-aix61

-rw-r--r--    1 root     system   1967411200 Jan 06 11:00 cg-aix71

 

root@nim1 / # lsnim -t mksysb

vio1-mksysb     resources       mksysb

cg-aix61        resources       mksysb

cg1-mksysb      resources       mksysb

cg-aix71        resources       mksysb

 

root@nim1 / # lsnim -l cg-aix71

cg-aix71:

   class         = resources

   type          = mksysb

   comments      = Created by nimadm on Thu Jan 6 11:00:51 EETDT 2011

   Rstate        = ready for use

   prev_state    = unavailable for use

   location      = /export/mksysb/cg-aix71

   version       = 7

   release       = 1

   mod           = 0

   oslevel_r     = 7100-00

   alloc_count   = 0

   server        = master

   creation_date = Thu Jan 6 11:00:54 2011

 

During the migration I noticed the following file systems were mounted on the NIM master.

 

root@nim1 / # df

Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4         1572864   1149640   27%    10471     8% /

/dev/hd2         5242880   1597696   70%    42812    19% /usr

/dev/hd9var      1048576    250456   77%     8260    23% /var

/dev/hd3         8912896   8287752    8%      111     1% /tmp

/dev/hd1          524288    523520    1%        5     1% /home

/dev/hd11admin     524288    523488    1%        5     1% /admin

/proc                  -         -    -         -     -  /proc

/dev/hd10opt     1048576    664176   37%     7262     9% /opt

/dev/livedump     524288    523552    1%        4     1% /var/adm/ras/livedump

/dev/tftpbootlv     524288    419656   20%       10     1% /tftpboot

/dev/exportlv     524288    523408    1%       13     1% /export

/dev/lppsrclv   25690112   4383632   83%     3114     1% /export/lppsource

/dev/spotlv      8912896   4848464   46%    43450     8% /export/spot

/dev/mksysblv   38273024  11315200   71%        9     1% /export/mksysb

/dev/lv00         425984    412432    4%       25     1% /cg-aix61_mm_alt/alt_inst

/dev/lv01          65536     63376    4%       18     1% /cg-aix61_mm_alt/alt_inst/admin

/dev/lv02          65536     63384    4%       17     1% /cg-aix61_mm_alt/alt_inst/home

/dev/lv03         557056    414720   26%      201     1% /cg-aix61_mm_alt/alt_inst/opt

/dev/lv04         294912    285184    4%       21     1% /cg-aix61_mm_alt/alt_inst/tmp

/dev/lv05        4653056   4506904    4%       17     1% /cg-aix61_mm_alt/alt_inst/usr

/dev/lv06         720896    698160    4%       20     1% /cg-aix61_mm_alt/alt_inst/var

/dev/lv07          65536     63384    4%       17     1% /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

 

And in my nimadmvg volume group, there were several new logical volumes.

 

root@nim1 / # lsvg -l nimadmvg

nimadmvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

loglv02             jfslog     1       1       1    open/syncd    N/A

lv00                jfs        13      13      1    open/syncd    /cg-aix61_mm_alt/alt_inst

lv01                jfs        2       2       1    open/syncd    /cg-aix61_mm_alt/alt_inst/admin

lv02                jfs        2       2       1    open/syncd    /cg-aix61_mm_alt/alt_inst/home

lv03                jfs        17      17      1    open/syncd    /cg-aix61_mm_alt/alt_inst/opt

lv04                jfs        9       9       1    open/syncd    /cg-aix61_mm_alt/alt_inst/tmp

lv05                jfs        142     142     1    open/syncd    /cg-aix61_mm_alt/alt_inst/usr

lv06                jfs        22      22      1    open/syncd    /cg-aix61_mm_alt/alt_inst/var

lv07                jfs        2       2       1    open/syncd    /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

root@nim1 / #

 

These file systems were used to extract the mksysb locally on the NIM master and then migrate the image to AIX 7.1. I noticed the restbyname process consuming some CPU during the restore process.

 

Topas Monitor for host:nim1                     EVENTS/QUEUES    FILE/TTY

Thu Jan  6 10:15:38 2011   Interval:2           Cswitch    1155  Readch  2655.6K

                                                Syscall    3143  Writech 2664.7K

CPU     User% Kern% Wait% Idle%   Physc  Entc%  Reads       454  Rawin         0

Total     6.1  22.0   8.2  63.7    0.09  47.04  Writes     1168  Ttyout      578

                                                Forks        12  Igets        85

Network    BPS  I-Pkts  O-Pkts    B-In   B-Out  Execs         0  Namei       521

Total    7.74K    5.50    1.50   5.74K   2.00K  Runqueue      0  Dirblk       97

                                                Waitqueue   1.0

Disk    Busy%      BPS     TPS  B-Read  B-Writ                   MEMORY

Total     0.0    5.21M   383.0   2.02M   3.19M  PAGING           Real,MB    2048

                                                Faults      776  % Comp     67

FileSystem          BPS    TPS  B-Read  B-Writ  Steals     1118  % Noncomp  32

Total             2.03M  85.50   2.03M       0  PgspIn        0  % Client   19

                                                PgspOut       0

Name           PID  CPU%  PgSp Owner            PageIn      517  PAGING SPACE

restbyna   15532232 16.4 1.21M root             PageOut     816  Size,MB     512

vmmd         458766  5.5 1.19M root             Sios        969  % Used      3

lrud         262152  1.5  512K root                              % Free     97

syncd       2097296  1.1  592K root             NFS (calls/sec)

topas      16973864  0.5 1.98M root             SerV2         0  WPAR Activ    0

xmgc         851994  0.5  448K root             CliV2         0  WPAR Total    0

nimesis    12320772  0.3 2.41M root             SerV3         0  Press: "h"-help

getty       7798832  0.1  580K root             CliV3         0         "q"-quit

xmtopas     8847610  0.1  816K root

nfsd        5111964  0.1 4.75M root

xmtopasa    6947052  0.1  916K root

gil         1441836  0.0  960K root

java        4784136  0.0 57.5M pconsole

java        8716308  0.0 70.7M root

rpc.lock    6029528  0.0 1.19M root

clcomd      6160578  0.0 1.69M root

pilegc       786456  0.0  640K root

aixmibd     3997882  0.0 1.24M root

netm        1376298  0.0  448K root

topasrec    8454202  0.0 1.01M root

 

 

root@nim1 / # ps -ef | grep rest

    root 15270132  3342336   1 10:15:55  pts/1  0:00 grep rest

    root 15532232 10747910  37 10:14:02  pts/0  0:02 restbyname -x -q -f /export/mksysb/cg-aix61

root@nim1 / #

 

I didn’t encounter any issues with my nimadm operation but if I had I could have reviewed the nimadm log file located in /var/adm/ras:

 

root@nim1 /var/adm/ras/alt_mig # ls -ltr

total 856

-rw-r--r--    1 root     system       435009 Jan 06 11:00 cg-aix61_alt_mig.log

 

 

If you ever encounter problems with nimadm, I highly recommend that you re-run the operation with the –V and –D flags. This will enable verbose output, as well as debugging information which can assist you greatly in determining the root cause of your issue.

 

I was able to use the new, migrated, AIX 7.1 mksysb image to install a new LPAR.

 

AIX Version 7

Copyright IBM Corporation, 1982, 2010.

login: root

root's Password:

*******************************************************************************

*                                                                             *

*                                                                             *

*  Welcome to AIX Version 7.1!                                                *

*                                                                             *

*                                                                             *

*  Please see the README file in /usr/lpp/bos for information pertinent to    *

*  this release of the AIX Operating System.                                  *

*                                                                             *

*                                                                             *

*******************************************************************************

Last login: Thu Jan  6 13:21:04 EETDT 2011 on /dev/vty0

 

# oslevel -s

7100-00-02-1041

#

 

You’ve gotta love NIM! Take a look at the nimadm documentation for more information:

 

http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds4/nimadm.htm

 

By the way, if you are interested in migrating from AIX 4.3 to AIX 7.1, then you’ll need to consider the mksysb migration method. I’ve not done this myself (i.e. 4.3 to 7.1 migration) but the feedback I received from the AIX development team indicates that it does work as expected.

 

“... you had expressed some interest in mksysb migration.  Since we were not aware of anyone having attempted it with 7.1.  I went ahead and created a 4.3.3.75 system and created a mksysb and then used that mksysb in conjunction with nim and successfully installed another system using the 4.3 mksysb and 7.1 nim resources to perform a mksysb migration to 7.1.  I just followed the steps as noted in the Installation guide for mksysb migration. “

Refer to the following document for further information on mksysb migration with NIM.

http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.install/doc/insgdrf/mksysbmigration.htm

 

NIM from A to Z in AIX 5L Redbook.

http://www.redbooks.ibm.com/redpieces/abstracts/sg247296.html?Open

Here’s a nice little matrix of current VIOS levels and the corresponding AIX level.

 

 

The following link also has some fantastic information relating to NIM installation and backup of a Virtual I/O Server (VIOS).

https://www-304.ibm.com/support/docview.wss?uid=isg3T1011386

It has an official answer for one of the questions I’m asked the most:

Q: Do I need to use an lpp_source to install the VIO server using NIM?

A: No. All of the device drivers and software packages needed to install the VIO server are in the VIO server mksysb.

I also recommend Rob McNelly’s latest blog on installing VIO servers.

http://ibmsystemsmag.blogs.com/aixchange/2011/01/an-unusual-vios-install.html

 

I received an email this week from a colleague that worked with me on the NIM Redbook back in 2006. He was experiencing an issue with DSM and NIM. He was attempting to use the dgetmacs command to obtain the MAC address of the network adapters on an LPAR. The command was failing to return the right information.

 

This was the output from his attempt with dgetmacs:

 

# dgetmacs -n lpar1

Using an adapter type of "ent".

Could not dsh to node lpar1.

Attempting to use openfirmware method to collect MAC addresses.

Acquiring adapter information from Open Firmware for node lpar1.

dgetmacs: 2760-185 Issuing the command "/opt/ibm/sysmgt/dsm/dsmbin/lpar_netboot -M -n -t ent -J hmc1 -j hmc -F /etc/ibm/sysmgt/dsm/config/my_password_file –A 4 10582FP 8202-E4B" gave a return code of 1.  The routine will continue.

No adapters found for node lpar1.

Could not get mac address for this aix node "lpar1"

 

I experienced this very issue during the writing of the AIX 7.1 Differences Guide Redbook. And given that I was in Austin, sitting in the same building as the AIX development team, I was able to speak with the developers directly about the issue. At that time they provided me with the following workaround.

 

First they asked me to check the size of the /usr/lib/nls/msg/en_US/IBMhsc.netboot.cat message catalog file.

 

# ls –l /usr/lib/nls/msg/en_US/IBMhsc.netboot.cat

-rw-r--r--    1 bin      bin            3905 Aug 08 09:54

 

They were surprised to find that the file appeared to be “too small”. They promptly sent me the catalog file from one of their development AIX 7.1 systems.  I replaced the file as follows:

 

# cd  /usr/lib/nls/msg/en_US/

# ls -ltr IBMhsc*

-rw-r--r--    1 bin      bin            3905 Aug 08 09:54 IBMhsc.netboot.cat

# cp -p IBMhsc.netboot.cat IBMhsc.netboot.cat.old

# cp /tmp/lpar1/IBMhsc.netboot.cat.new IBMhsc.netboot.cat

# ls -ltr IBMhsc*

-rw-r--r--    1 bin      bin         3905 Aug 08 09:54 IBMhsc.netboot.cat.old

-rw-r--r--    1 bin      bin        26374 Dec 23 11:24 IBMhsc.netboot.cat

 

This fixed the problem for me during the residency.

 

So I asked my friend to do the same (after I sent him the message catalog file). He ran the dgetmacs command again and this time it returned the MAC address for all the network adapters in his LPAR. Success!

 

# dgetmacs -n lpar1

Using an adapter type of "ent".

Could not dsh to node lpar1.

Attempting to use openfirmware method to collect MAC addresses.

Acquiring adapter information from Open Firmware for node lpar1.

 

# Node::adapter_type::interface_name::MAC_address::location::media_speed::adapter_duplex::UNUSED::install_gateway::ping_status::machine_type::netaddr::subnet_mask

 

lpar1::ent_v::::4A7009608816::U8202.E4B.10582FP-V4-C22-T1::auto::auto::::::n/a::secondary::::

 

lpar1::ent_v::::4A7009608818::U8202.E4B.10582FP-V4-C24-T1::auto::auto::::::n/a::secondary::::

 

This then allowed him to network boot his LPAR and install AIX 7.1 via NIM.

 

# nim -o bos_inst -a bosinst_data=noprompt -a source=rte -a installp_flags=agX -a accept_licenses=yes -a spot=spotaix7tl0sp2 -a lpp_source=aix7tl0sp2 lpar1

dnetboot Status: Invoking /opt/ibm/sysmgt/dsm/dsmbin/lpar_netboot lpar1

dnetboot Status: Was successful network booting node lpar1.

 

I thought this might have been resolved in the latest service pack for AIX 7.1. However, both of us have the latest service pack installed (i.e. TL0 SP2 for AIX 7.1).

 

# oslevel -s

7100-00-02-1041

 

If anyone encounters this problem, feel free to contact me (aix@gibsonnet.net).

 

For more information, please refer to section 5.2.7 Using DSM and NIM, in the AIX 7.1 Differences Guide Redbook:

 

http://www.redbooks.ibm.com/Redbooks.nsf/RedpieceAbstracts/sg247910.html?Open

 

 

A colleague of mine contacted me during the week to ask how one could determine if a POWER7 system was capable of Active Memory Expansion. I thought I'd share my response with everyone who follows my blog.

 

You can check if the system is capable of providing Active Memory Expansion (AME), from the HMC command line using the lssyscfg command, as shown here:
 
hscroot@hmc1:~> lssyscfg -r sys -m Server-8233-E8B-SN1000 -F active_mem_expansion_capable
1

Alternatively you can view the capabilities of the managed system from the HMC GUI, under System properties/Capabilities, as shown in the following image. 

 
 

Once you’ve concluded that your POWER7 system is AME capable, the next step is to check if AME is enabled or disabled for an LPAR. This is easily done from the AIX command line with the lparstat and/or amepat commands, as shown in the following example:
 
;Running lparstat and amepat on a LPAR with AME disabled.

 

$ lparstat -i| grep Exp
Target Memory Expansion Factor             : -
Target Memory Expansion Size               : -
 
$ amepat | grep Expan
WARNING: Running in no modeling mode.
Active Memory Expansion        : Disabled

 

;Running lparstat and amepat on a LPAR with AME enabled.

 

$ lparstat -i | grep -i exp

Memory Mode                                : Dedicated-Expanded

Target Memory Expansion Factor             : 1.00

Target Memory Expansion Size               : 6144 MB

 

$ amepat | grep -i exp

WARNING: Running in no modeling mode.

Active Memory Expansion        : Enabled

Target Expanded Memory Size    : 6.00 GB

Target Memory Expansion factor : 1.00
 
Note that I deliberately ran these commands as a non-root user to highlight the fact that you don’t need root access to ascertain whether or not AME is active on an LPAR.

 

By the way, did you know that AME is now supported in SAP production environments running on AIX and POWER7? Both SAP application servers, as well as database servers running DB2 LUW are supported. Unfortunately there’s no support for LPARs with Oracle RDBMS at this stage. Thanks to George Manousos from IBM Australia for providing me with this update.

Please refer to the following SAP note.

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/resource/1464605%28Power7AME%29.pdf

 

The SAP note also states:

 

Specific tests have been successfully executed with SAP ERP 6.0 (ABAP)

workload. The results of those test can be found in the following whitepaper:

 

ftp://ftp.software.ibm.com/common/ssi/sa/wh/n/pow03038usen/POW03038USEN.PDF

 

Restrictions

As AME is not recommended with large page support, AME will disable AIX

64KB pages by default.

 

AME monitoring capability in CCMS

Monitoring capabilities are available with saposcol v12.46 for more details see SAP note 710975.

 

 

It looks as though SAP have been quick to support AME on AIX/POWER7. This is a great benefit to SAP customers running with PowerVM on the IBM POWER platform. The following screenshot shows the SAP CCMS with AME statistics being reported.

 

PowerVM Integration into SAP CCMS Monitoring expanded to include Virtualized Memory Metrics

The SAP AIX porting team continues in further improving the integration of PowerVM into SAP system monitoring. While in the past already most processor and AIX virtualization metrics could be monitored via CCMS, now POWER7 Active Memory Expansion (AME)  has been included.  Customers will find a new memory section in the respective CCMS-panel as depicted in this screen shot.

For those who are not familiar with what AME actually is and how it can benefit you, I suggest you take a look at the IBM AME Wiki site first:

 

http://www.ibm.com/developerworks/wikis/display/WikiPtype/IBM+Active+Memory+Expansion

 

And watch Nigel Griffiths AME movie:

 

http://www.ibm.com/developerworks/wikis/display/wikiptype/movies#Movies-ame

 

 

 

 

Interested in learning more about the next generation of Virtual I/O Server? Yes? Then take a look at this presentation.

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/resource/VM19.pdf

 

The following diagram gives you a high-level overview of the classic VIOS compared with the NextGen VIOS. Should be enough to spark your interest!

Happy reading!

 

 

 

I just had to share this entertaining presentation with everyone.

 

http://gibsonnet.net/AIX/ibm/Journey_in_VLAN_world.pps

 

It was written by Bernard Lemal. He works for IBM GTS Belgium in AIX support.

 

It’s a humorous look at the life of an IP packet and its journey in the world of VLANs, VLAN tagging and Virtual I/O.

 

Plenty of interesting and informative commentary along the way.

 

The part of the IP packet is portrayed by Bernard himself. Take a look at the presentation to see what I mean!

 

I wish I’d seen a presentation like this when I was first introduced to the concept of VLAN tagging, it might have answered several questions early on.

 

Enjoy!

Using Cluster Aware AIX with AIX 7.1 you can create a cluster of AIX nodes. This interconnected cluster of nodes can immediately provide several capabilities, such as:

 

·         Clusterwide event management.

·         Communication and storage events:

o   Node UP and node DOWN

o   Network adapter UP and DOWN

o   Network address change

o   Point-of-contact UP and DOWN

o   Disk UP and DOWN

o   Predefined and user-defined events

·         Clusterwide storage naming service

·         Clusterwide command distribution

·         Clusterwide communication making use of networking and storage connectivity

 

All of the capabilities of CAA are built into the AIX Version 7.1 operating system. CAA is essentially a set of tools and services embedded into AIX that help to manage a cluster of AIX nodes and assist in running cluster software on AIX. Cluster products, such as PowerHA and RSCT from IBM, can utilise CAA to simplify the configuration, management and monitoring of an AIX cluster.

 

CAA does not form a cluster by itself. It is a tool set. There is no notion of quorum. (If 20 nodes of a 21 node cluster are down, CAA still runs on the remaining node). CAA does not eject nodes from a cluster. CAA provides tools to fence a node but never fences a node and will continue to run on a fenced node.

 

As you can see in the following diagram, cluster products, like PowerHA, will integrate with CAA to help form and manage highly available clusters.

 

 

So you will still need some form of cluster product, either from IBM or another vendor, in order to build a cluster that provides high availability capabilities like node failover/takeover.

 

What follows are some of the most important “snippets” from the CAA documentation (IMHO), available in the AIX 7.1 Information Centre (see the link at the bottom of the page).

 

Just like any cluster, each node that is added to a cluster by using CAA must have common storage devices available, for example via SAN storage device. These storage devices are used for the cluster repository disk and for any clustered shared data disks.

 

The Storage Naming Service provides a global device view across all the nodes in the cluster. The Storage Naming Service also provides a single global device name for a disk from any node in the cluster. The global device name, for example, cldisk1, refers to the same physical disk from any node in the cluster.

 

The cluster repository disk is used as the central repository for the cluster configuration data. The cluster repository disk must be accessible from all nodes in the cluster and is a minimum of 10 GB in size. Given the importance of the cluster configuration data, the cluster repository disk should be backed up by a redundant and highly available storage configuration. Even though the cluster repository disk is visible as a disk device to AIX, it should be treated as a special device for the cluster. The use of LVM commands is not supported when used on a cluster repository disk. The AIX LVM commands are designed as single node administrative commands, and are not applicable in a clustered configuration.

 

The cluster repository disk is renamed to a private device name (caa_private0). A raw section of the disk and a section of the disk that contains a special volume group and special logical volumes are used during cluster operations.

 

A multicast address is used for cluster communications between the nodes in the cluster.

These are configured automatically during the creation of the cluster. These nodes support cluster monitoring of events and cluster configuration attributes.

 

Scalable reliable multicasting is implemented in the cluster with a special gossip protocol over the multicast address. The gossip protocol determines the node configuration and then transmits the gossip packets over all available networking and storage communication interfaces (either Fibre Channel and/or SAS adapters). If no storage communication interfaces are configured, only the traditional networking interfaces are used.

 

When you first configure CAA, the following actions are performed:

 

•             The cluster is created using the mkcluster command.

•             The cluster configuration is written to the raw section of the cluster repository disk.

•             Primary and secondary database nodes are selected from the list of candidate nodes in the mkcluster command.

•             Special volume groups and logical volumes are created on the cluster repository disk.

•             Cluster file systems are created on the special volume group.

•             The cluster repository database is created on both primary and secondary nodes.

•             The cluster repository database is started.

•             Cluster services are made available to other functions in the operating system, such as Reliable Scalable Cluster Technology (RSCT) and PowerHA.

•             Storage framework register lists are created on the cluster repository disk.

•             A global device namespace is created and interaction with LVM starts for handling associated volume group events.

•             A clusterwide multicast address is established.

•             The node discovers all of the available communication interfaces.

•             The cluster interface monitoring starts.

•             The cluster interacts with Autonomic Health Advisory File System (AHAFS) for clusterwide event distribution.

•             The cluster exports cluster messaging and cluster socket services to other functions in the operating system, such as Reliable Scalable Cluster Technology (RSCT) and PowerHA.

 

In the following example I created a two node cluster using CAA tools. First of all, I had to use the mkcluster command to define the cluster nodes, shared storage and repository disk. The node names are 7502lp01 and 7502lp02. The shared data storage disks are hdisk2, hdisk3 and hdisk4. The repository disk, used to house the cluster configuration data, is hdisk1.

 

# mkcluster -n mycluster -r hdisk1 -d hdisk2,hdisk3,hdisk4 -m 7502lp01,7502lp02

mkcluster: Cluster shared disks are automatically renamed to names such as

        cldisk1, [cldisk2, ...] on all cluster nodes.  However, this cannot

        take place while a disk is busy or on a node which is down or not

        reachable.  If any disks cannot be renamed now, they will be renamed

        later by the clconfd daemon, when the node is available and the disks

        are not busy.

 

You’ll notice that the mkcluster command informed me that the cluster shared disks are automatically renamed to cluster disk names like cldisk1. After I’d run the command I noticed something quite interesting and impressive. Prior to configuring the cluster the shared disks on both nodes had names like hdiskX. Afterwards the shared disks had been renamed across both nodes. Now these disks had the same name!

 

This is going to simplify cluster configuration and management. No more will I need to remove and recreate disks in order to resolve disk naming inconsistencies in a cluster. The lspv output (shown below) from both nodes shows that I have three shared disks (cldisk1, cldisk2 and cldisk3). These would be used for shared data. The disk named caa_private0 is my cluster repository disk. This is used to store and share the cluster configuration data.

 

7502lp01:

# lspv | sort

caa_private0    00f61ab20b97190d                    caavg_private   active

cldisk1         00f61ab20bf28ac6                    None

cldisk2         none                                None

cldisk3         none                                None

hdisk0          00f61ab2f73e46e2                    rootvg          active

 

7502lp02:

# lspv | sort

caa_private0    00f61ab20b97190d                    caavg_private   active

cldisk1         00f61ab20bf28ac6                    None

cldisk2         none                                None

cldisk3         none                                None

hdisk0          00f61ab2895e4cbe                    rootvg          active

 

Cluster Aware AIX tells you what nodes are in the cluster plus information on those nodes, including state. A special “gossip” protocol is used over the multicast address to determine node information and implement scalable reliable multicast. No traditional heartbeat mechanism is employed. Gossip packets travel over all interfaces, including storage. Immediately after running mkcluster, I was able to query the status of the nodes in my cluster without any further configuration!

 

# lscluster -m

Calling node query for all nodes

Node query number of nodes examined: 2

 

 

        Node name: 7502lp01

        Cluster shorthand id for node: 1

        uuid for node: 3cd9cb00-bf55-11df-b015-6e8dd0af6304

        State of node:  UP

        Smoothed rtt to node: 7

        Mean Deviation in network rtt to node: 3

        Number of zones this node is a member in: 0

        Number of clusters node is a member in: 1

        CLUSTER NAME       TYPE  SHID   UUID

        mycluster          local        267ce7fc-bf55-11df-a3b9-6e8dd877b814

 

        Number of points_of_contact for node: 1

        Point-of-contact interface & contact state

         en0  UP

 

------------------------------

 

        Node name: 7502lp02

        Cluster shorthand id for node: 2

        uuid for node: d1a46164-bf46-11df-94b3-6e8dd877b814

        State of node:  UP  NODE_LOCAL

        Smoothed rtt to node: 0

        Mean Deviation in network rtt to node: 0

        Number of zones this node is a member in: 0

        Number of clusters node is a member in: 1

        CLUSTER NAME       TYPE  SHID   UUID

        mycluster          local        267ce7fc-bf55-11df-a3b9-6e8dd877b814

 

        Number of points_of_contact for node: 0

        Point-of-contact interface & contact state

         n/a

 

CAA tells you what interfaces have been discovered on a node plus information on those interfaces, including state.

 

# lscluster –i –n

Node 7502lp01

Node uuid = 110b2422-7efc-11df-aed7-1612a0003002

Number of interfaces discovered = 1

        Interface number 1 en0

                ifnet type = 6 ndd type = 7

                Mac address length = 6

                Mac address =  16.12.a0.0.30.2

                Smoothed rrt across interface = 7

                Mean Deviation in network rrt across interface = 3

                Probe interval for interface = 100 ms

                ifnet flags for interface = 0x1e080863

                ndd flags for interface = 0x21081b

                Interface state  UP

                Number of regular addresses configured on interface = 1

                IPV4 ADDRESS: 9.3.28.136  broadcast 9.3.28.159  netmask 255.255.255.224

                Number of cluster multicast addresses configured on interface = 1

                IPV4 MULTICAST ADDRESS: 228.8.8.8  broadcast 0.0.0.0  netmask 0.0.0.0

 

CAA tells you what disks are in the cluster plus information on those disks, including the state and type.

 

# lscluster –d

 

Storage Interface Query

Cluster Name:  mycluster

Cluster uuid: a1bde416-83b9-11df-b438-1612a0004002

Number of nodes reporting = 2

Number of nodes expected = 2

Node oscar-dev8.austin.ibm.com

Node uuid = 9807465b-3194-f1e5-8a04-1f044bc82593

Number of disk discovered = 2

        cldisk1

          state : UP

          uDid  : 533E3E213600A0B8000475C200000E7114BC825930F1818      FAStT03IB

Mfcp05VDASD03AIXvscsi

             type  : CLUSDISK

          uUid  : 9807465b-3194-f1e5-8a04-1f044bc82593

        hdisk1

          state : UP

          uDid  :

          uUid  : 600a0b80-0047-5d0a-0000-e6094bc82690

          type  : REPDISK

 

You can run cluster wide commands on all nodes immediately after configuring your cluster using the clcmd command. No further configuration required.

 

# clcmd ps -ef

 

 

-------------------------------

NODE 7502lp01

-------------------------------

     UID     PID         PPID      C    STIME    TTY  TIME CMD

    root       1           0       0   Aug 25      -  0:00 /etc/init

    root  655526           1       0   Aug 25      -  0:18 [cimserve]

    root 1310840           1       0   Aug 25      -  0:00 /usr/ccs/bin/shlap64

    root 2490482           1       0   Aug 25      -  0:42 /usr/sbin/srcmstr

    root 3145874         3801262   0 11:17:13      -  0:00 telnetd -a

    root 3276994         2490482   0   Aug 25      -  0:10 /usr/sbin/snmpd

 

-------------------------------

NODE 7502lp02

-------------------------------

     UID     PID         PPID   C    STIME    TTY  TIME CMD

    root       1           0   1   Aug 28      -  0:00 /etc/init

    root 1441934           1   0   Aug 28      -  0:39 /usr/sbin/syncd 60

    root 1638496           1   0   Aug 28      -  0:00 /usr/ccs/bin/shlap64

    root 2228252     2556054   0   Aug 28      -  0:00 /usr/sbin/hostmibd

   

I believe that PowerHA SystemMirror 7.1 will be the first cluster product from IBM to provide high availability on AIX using the new Cluster Aware AIX features. An IBM Redbook Residency recently covered the integration of CAA and PowerHA. I’m eagerly awaiting the first draft of this book so that I can learn how these two components will “play” together.

 

Refer to the AIX 7.1 Information Centre for more information on Cluster Aware AIX:

 

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.clusteraware/claware_main.htm

 

 

Refer to the AIX 7.1 Information Centre for more information on AHAFS and the new AIX Event Infrastructure:

 

http://publib.boulder.ibm.com/infocenter/aix/v7r1/index.jsp?topic=/com.ibm.aix.baseadmn/doc/baseadmndita/aix_ev.htm

 

 

 

I remember reading this interesting document about some challenges renaming AIX devices.

 

http://gibsonnet.net/aix/ibm/The_Fun_of_Oracle_RAC_ASM_and_Device_Names.doc

 

This poor guy had to jump through some hoops to rename some disk and network devices in a clustered AIX (Oracle RAC) environment.

 

This comment stayed with me:

 

“The easy way to get network adapters and disk names to match between systems would be to run something like

 

# chdev –l old_name –n new_name

 

Except, this command option does NOT exist.”

 

Since then I’ve always been on the “look out” for an easy way to rename devices in AIX.

 

In particular in large clustered environments where it is sometimes very important to have the same disk and network device names in sync across all nodes in a cluster. And besides, it’s a lot easier  to verify a cluster configuration if the hdisk names are all the same. Matching PVIDs works but it requires a lot more effort! For example, knowing that hdisk123 is the same device on all nodes makes life easier than scanning lspv output for a PVID like 00f6048868b4gead.  Of course you can script things to make this easier but it would be great if you didn’t need to do this and that there was a way to rename devices as needed, without resorting to unsupported methods.

 

Well, this is no longer an issue for AIX.

 

Starting with AIX 7.1, you can now easily rename devices. A new command called rendev was introduced to allow AIX administrators to rename devices as required.

 

From the man page:

 

The rendev command enables devices to be renamed. The device to be renamed, is specified with the -l flag, and the new desired name is specified with the -n flag.

 

The new desired name must not exceed 15 characters in length. If the name has already been used or is present in the /dev directory, the operation fails. If the name formed by appending  the new name after the character r is already used as a device name, or appears in the /dev directory, the operation fails.

 

 If the device is in the Available state, the rendev command must unconfigure the device before renaming it. This is similar to the operation performed by the rmdev -l Name command. If the unconfigure operation fails, the renaming will also fail. If the unconfigure succeeds, the rendev command will configure the device, after renaming it, to restore it to the Available state. The -u flag may be used to prevent the device from being configured again after it is renamed.

 

 Some devices may have special requirements on their names in order for other devices or applications to use them. Using the rendev command to rename such a device may result in the device being unusable. Note: To protect the configuration database, the rendev command cannot be interrupted once it has started. Trying to stop this command before completion, could result in a corrupted database.

 

This functionality was required to help simplify cluster device configuration with the new Cluster Aware AIX (CAA) feature in AIX 7.1.

 

I’ll be blogging about CAA in a future post.

 

Here are some examples of using the rendev command on AIX 7.1 (GA) system. In the first example I will rename hdisk3 to hdisk300. Note: hdisk3 is not in use (busy).

If the disk had been allocated to a volume group, I would have needed to varyoff the volume group first.

 

 

# lspv

hdisk0          00f61ab2f73e46e2                    rootvg          active

hdisk1          00f61ab20bf28ac6                    None

hdisk2          00f61ab2202f7c0b                    None

hdisk4          00f61ab20b97190d                    None

hdisk3          00f61ab2202f93ab                    None

 

# rendev -l hdisk3 -n hdisk300

 

# lspv

hdisk0          00f61ab2f73e46e2                    rootvg          active

hdisk1          00f61ab20bf28ac6                    None

hdisk2          00f61ab2202f7c0b                    None

hdisk4          00f61ab20b97190d                    None

hdisk300        00f61ab2202f93ab                    None

 

Easy!

 

Next, I’ll rename a virtual SCSI adapter. I renamed vscsi0 to vscsi2. Note: I placed the adapter, vscsi0, in a Defined state before renaming the device.

 

# rmdev -Rl vscsi0

 

# lsdev -Cc adapter

ent0   Available  Virtual I/O Ethernet Adapter (l-lan)

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi0 Defined    Virtual SCSI Client Adapter

vscsi1 Available  Virtual SCSI Client Adapter

 

# rendev -l vscsi0 -n vscsi2

 

# lsdev -Cc adapter

ent0   Available  Virtual I/O Ethernet Adapter (l-lan)

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi1 Available  Virtual SCSI Client Adapter

vscsi2 Defined    Virtual SCSI Client Adapter

 

Now I’ll rename a network adapter from ent0 to ent10. I bring down the interface before changing the device name

 

# lsdev -Cc adapter

ent0   Available  Virtual I/O Ethernet Adapter (l-lan)

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi1 Available  Virtual SCSI Client Adapter

vscsi2 Defined    Virtual SCSI Client Adapter

 

# ifconfig en0

en0: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>

        inet 10.1.20.19 netmask 0xffff0000 broadcast 10.153.255.255

         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

 

# ifconfig en0 down detach

 

# rendev -l ent0 -n ent10

 

# lsdev -Cc adapter

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

ent10  Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi1 Available  Virtual SCSI Client Adapter

vscsi2 Defined    Virtual SCSI Client Adapter

 

# rendev -l en0 -n en10

 

# chdev -l en10 -a state=up

en10 changed

# mkdev -l inet0

inet0 Available

# ifconfig en10

en10: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>

        inet 10.1.20.19 netmask 0xffff0000 broadcast 10.153.255.255

         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

 

If you want to be creative you can rename devices to anything you like (as long as it’s not more than 15 characters). For example I’ll rename vscsi2 to myvscsiadapter.

 

 

# rendev -l vscsi2 -n myvscsiadapter

# lsdev -Cc adapter

ent1           Available  Virtual I/O Ethernet Adapter (l-lan)

myadapter      Available  Virtual I/O Ethernet Adapter (l-lan)

myvscsiadapter Defined    Virtual SCSI Client Adapter

vsa0           Available  LPAR Virtual Serial Adapter

vscsi1         Available  Virtual SCSI Client Adapter

 

And in the last example I’ll demonstrate changing virtual SCSI adapter device names on a live system.

 

This is single disk system (hdisk0), with two vscsi adapters.

 

# lspv

hdisk0          00f6048868b4deee                    rootvg          active

 

# lspath

Enabled hdisk0 vscsi0

Enabled hdisk0 vscsi1

 

# lsdev -Cc adapter

ent0   Available  Virtual I/O Ethernet Adapter (l-lan)

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi0 Available  Virtual SCSI Client Adapter

vscsi1 Available  Virtual SCSI Client Adapter

 

We ensure the adapter is in a Defined state before renaming it. This will fail otherwise.

 

# rmdev -Rl vscsi1

vscsi1 Defined

# lsdev -Cc adapter | grep vscsi

vscsi0 Available  Virtual SCSI Client Adapter

vscsi1 Defined    Virtual SCSI Client Adapter

 

Now we rename the adapter vscsi1 to vscsi3.

 

# rendev -l vscsi1 -n vscsi3

 

# lsdev -Cc adapter | grep vscsi

vscsi0 Available  Virtual SCSI Client Adapter

vscsi3 Defined    Virtual SCSI Client Adapter

 

That was easy enough. Now I need to bring the adapter and path online with cfgmgr. The lspath output displays an additional path to vscsi3.

 

# lspath

Enabled hdisk0 vscsi0

Defined hdisk0 vscsi1

 

# cfgmgr

Method error (/etc/methods/cfgscsidisk -l hdisk0 ):

        0514-082 The requested function could only be performed for some

                 of the specified paths.

 

# lspath

Enabled hdisk0 vscsi0

Defined hdisk0 vscsi1

Enabled hdisk0 vscsi3

 

Now I need to remove the old path to vscsi1. The path to vscsi3 is now Enabled. The adapter, vscsi3, is in an Available state. All is good.

 

# rmpath -l hdisk0 -p vscsi1 -d

path Deleted

 

# lspath

Enabled hdisk0 vscsi0

Enabled hdisk0 vscsi3

 

# lsdev -Cc adapter | grep vscsi

vscsi0 Available  Virtual SCSI Client Adapter

vscsi3 Available  Virtual SCSI Client Adapter

 

The same steps need to be repeated for the vscsi0 adapter. This is renamed to vscsi2.

 

# rmdev -Rl vscsi0

vscsi0 Defined

# lsdev -Cc adapter | grep vscsi

vscsi0 Defined    Virtual SCSI Client Adapter

vscsi3 Available  Virtual SCSI Client Adapter

 

 

# rendev -l vscsi0 -n vscsi2

 

# lsdev -Cc adapter | grep vscsi

vscsi2 Defined    Virtual SCSI Client Adapter

vscsi3 Available  Virtual SCSI Client Adapter

 

 

# lspath

Defined hdisk0 vscsi0

Enabled hdisk0 vscsi3

 

# cfgmgr

Method error (/etc/methods/cfgscsidisk -l hdisk0 ):

        0514-082 The requested function could only be performed for some

                 of the specified paths.

 

# lspath

Defined hdisk0 vscsi0

Enabled hdisk0 vscsi2

Enabled hdisk0 vscsi3

 

# rmpath -l hdisk0 -p vscsi0 -d

path Deleted

 

# cfgmgr

# lspath

Enabled hdisk0 vscsi2

Enabled hdisk0 vscsi3

 

That’s it. Both adapters have been renamed while the system was in use. No downtime required.

 

# lsdev -Cc adapter | grep vscsi

vscsi2 Available  Virtual SCSI Client Adapter

vscsi3 Available  Virtual SCSI Client Adapter

 

# lspath

Enabled hdisk0 vscsi2

Enabled hdisk0 vscsi3

 

Please refer to the AIX 7.1 command reference for more information on this new command:

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.cmds/doc/aixcmds4/rendev.htm

 

And by the way, here’s a draft of what I’ve been working on for the last 5 weeks:

 

IBM AIX Version 7.1 Differences Guide:

http://www.redbooks.ibm.com/Redbooks.nsf/RedpieceAbstracts/sg247910.html?Open

 

Follow me on Twitter:

http://twitter.com/cgibbo