Chris's AIX Blog

Just say you change the queue_depth on a hdisk with chdev –P. This updates the devices ODM information only, not its running configuration. The new value will take effect next time I reboot the system. So now I have a different queue_depth in the ODM compared to the devices current running config (in the kernel).

 

What if I forget that I’ve made this change to the ODM and forget to reboot the system for many months? Someone complains of an I/O performance issue....I check the queue_depths and find they appear to be set appropriately but I still see disk queue full conditions on my hdisks. But have I rebooted since changing the values?

 

How do I know if the ODM matches the devices running configuration?

 

For example, I start with a queue_depth of 3, which is confirmed by looking at lsattr (ODM) and kdb (running config) output:

 

# lsattr -El hdisk6 -a queue_depth

queue_depth 3 Queue DEPTH True

 

 

# echo scsidisk hdisk6 | kdb | grep queue_depth

    ushort queue_depth  = 0x3;                                                   < In Hex.

 

Now I change the queue_depth using chdev –P i.e. only updating the ODM.

 

# chdev -l hdisk6 -a queue_depth=256 -P

hdisk6 changed

 

# lsattr -El hdisk6 -a queue_depth

queue_depth 256 Queue DEPTH True

         

kdb reports that the disks running configuration still has a queue_depth of 3.

 

# echo scsidisk hdisk6 | kdb | grep queue_depth

    ushort queue_depth  = 0x3;

 

Now if I varyoff the VG and change the disk queue_depth, both lsattr (ODM) and kdb (the running config) show the same value:

 

 

# umount /test

# varyoffvg testvg

 

# chdev -l hdisk6 -a queue_depth=256

hdisk6 changed

 

# varyonvg testvg

# mount /test

 

# lsattr -El hdisk6 -a queue_depth

queue_depth 256 Queue DEPTH True

 

# echo scsidisk hdisk6 | kdb | grep queue_depth

    ushort queue_depth  = 0x100;                                    < In Hex = Dec 256.

 

# echo "ibase=16 ; 100" | bc

256

 

This is one way of checking you’ve rebooted since you changed your queue_depth attributes. I’ve tried this on AIX 6.1 and 7.1 only.

 

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.kdb/doc/kdb/kdb_pdf.pdf

 

Using Cluster Aware AIX with AIX 7.1 you can create a cluster of AIX nodes. This interconnected cluster of nodes can immediately provide several capabilities, such as:

 

·         Clusterwide event management.

·         Communication and storage events:

o   Node UP and node DOWN

o   Network adapter UP and DOWN

o   Network address change

o   Point-of-contact UP and DOWN

o   Disk UP and DOWN

o   Predefined and user-defined events

·         Clusterwide storage naming service

·         Clusterwide command distribution

·         Clusterwide communication making use of networking and storage connectivity

 

All of the capabilities of CAA are built into the AIX Version 7.1 operating system. CAA is essentially a set of tools and services embedded into AIX that help to manage a cluster of AIX nodes and assist in running cluster software on AIX. Cluster products, such as PowerHA and RSCT from IBM, can utilise CAA to simplify the configuration, management and monitoring of an AIX cluster.

 

CAA does not form a cluster by itself. It is a tool set. There is no notion of quorum. (If 20 nodes of a 21 node cluster are down, CAA still runs on the remaining node). CAA does not eject nodes from a cluster. CAA provides tools to fence a node but never fences a node and will continue to run on a fenced node.

 

As you can see in the following diagram, cluster products, like PowerHA, will integrate with CAA to help form and manage highly available clusters.

 

 

So you will still need some form of cluster product, either from IBM or another vendor, in order to build a cluster that provides high availability capabilities like node failover/takeover.

 

What follows are some of the most important “snippets” from the CAA documentation (IMHO), available in the AIX 7.1 Information Centre (see the link at the bottom of the page).

 

Just like any cluster, each node that is added to a cluster by using CAA must have common storage devices available, for example via SAN storage device. These storage devices are used for the cluster repository disk and for any clustered shared data disks.

 

The Storage Naming Service provides a global device view across all the nodes in the cluster. The Storage Naming Service also provides a single global device name for a disk from any node in the cluster. The global device name, for example, cldisk1, refers to the same physical disk from any node in the cluster.

 

The cluster repository disk is used as the central repository for the cluster configuration data. The cluster repository disk must be accessible from all nodes in the cluster and is a minimum of 10 GB in size. Given the importance of the cluster configuration data, the cluster repository disk should be backed up by a redundant and highly available storage configuration. Even though the cluster repository disk is visible as a disk device to AIX, it should be treated as a special device for the cluster. The use of LVM commands is not supported when used on a cluster repository disk. The AIX LVM commands are designed as single node administrative commands, and are not applicable in a clustered configuration.

 

The cluster repository disk is renamed to a private device name (caa_private0). A raw section of the disk and a section of the disk that contains a special volume group and special logical volumes are used during cluster operations.

 

A multicast address is used for cluster communications between the nodes in the cluster.

These are configured automatically during the creation of the cluster. These nodes support cluster monitoring of events and cluster configuration attributes.

 

Scalable reliable multicasting is implemented in the cluster with a special gossip protocol over the multicast address. The gossip protocol determines the node configuration and then transmits the gossip packets over all available networking and storage communication interfaces (either Fibre Channel and/or SAS adapters). If no storage communication interfaces are configured, only the traditional networking interfaces are used.

 

When you first configure CAA, the following actions are performed:

 

•             The cluster is created using the mkcluster command.

•             The cluster configuration is written to the raw section of the cluster repository disk.

•             Primary and secondary database nodes are selected from the list of candidate nodes in the mkcluster command.

•             Special volume groups and logical volumes are created on the cluster repository disk.

•             Cluster file systems are created on the special volume group.

•             The cluster repository database is created on both primary and secondary nodes.

•             The cluster repository database is started.

•             Cluster services are made available to other functions in the operating system, such as Reliable Scalable Cluster Technology (RSCT) and PowerHA.

•             Storage framework register lists are created on the cluster repository disk.

•             A global device namespace is created and interaction with LVM starts for handling associated volume group events.

•             A clusterwide multicast address is established.

•             The node discovers all of the available communication interfaces.

•             The cluster interface monitoring starts.

•             The cluster interacts with Autonomic Health Advisory File System (AHAFS) for clusterwide event distribution.

•             The cluster exports cluster messaging and cluster socket services to other functions in the operating system, such as Reliable Scalable Cluster Technology (RSCT) and PowerHA.

 

In the following example I created a two node cluster using CAA tools. First of all, I had to use the mkcluster command to define the cluster nodes, shared storage and repository disk. The node names are 7502lp01 and 7502lp02. The shared data storage disks are hdisk2, hdisk3 and hdisk4. The repository disk, used to house the cluster configuration data, is hdisk1.

 

# mkcluster -n mycluster -r hdisk1 -d hdisk2,hdisk3,hdisk4 -m 7502lp01,7502lp02

mkcluster: Cluster shared disks are automatically renamed to names such as

        cldisk1, [cldisk2, ...] on all cluster nodes.  However, this cannot

        take place while a disk is busy or on a node which is down or not

        reachable.  If any disks cannot be renamed now, they will be renamed

        later by the clconfd daemon, when the node is available and the disks

        are not busy.

 

You’ll notice that the mkcluster command informed me that the cluster shared disks are automatically renamed to cluster disk names like cldisk1. After I’d run the command I noticed something quite interesting and impressive. Prior to configuring the cluster the shared disks on both nodes had names like hdiskX. Afterwards the shared disks had been renamed across both nodes. Now these disks had the same name!

 

This is going to simplify cluster configuration and management. No more will I need to remove and recreate disks in order to resolve disk naming inconsistencies in a cluster. The lspv output (shown below) from both nodes shows that I have three shared disks (cldisk1, cldisk2 and cldisk3). These would be used for shared data. The disk named caa_private0 is my cluster repository disk. This is used to store and share the cluster configuration data.

 

7502lp01:

# lspv | sort

caa_private0    00f61ab20b97190d                    caavg_private   active

cldisk1         00f61ab20bf28ac6                    None

cldisk2         none                                None

cldisk3         none                                None

hdisk0          00f61ab2f73e46e2                    rootvg          active

 

7502lp02:

# lspv | sort

caa_private0    00f61ab20b97190d                    caavg_private   active

cldisk1         00f61ab20bf28ac6                    None

cldisk2         none                                None

cldisk3         none                                None

hdisk0          00f61ab2895e4cbe                    rootvg          active

 

Cluster Aware AIX tells you what nodes are in the cluster plus information on those nodes, including state. A special “gossip” protocol is used over the multicast address to determine node information and implement scalable reliable multicast. No traditional heartbeat mechanism is employed. Gossip packets travel over all interfaces, including storage. Immediately after running mkcluster, I was able to query the status of the nodes in my cluster without any further configuration!

 

# lscluster -m

Calling node query for all nodes

Node query number of nodes examined: 2

 

 

        Node name: 7502lp01

        Cluster shorthand id for node: 1

        uuid for node: 3cd9cb00-bf55-11df-b015-6e8dd0af6304

        State of node:  UP

        Smoothed rtt to node: 7

        Mean Deviation in network rtt to node: 3

        Number of zones this node is a member in: 0

        Number of clusters node is a member in: 1

        CLUSTER NAME       TYPE  SHID   UUID

        mycluster          local        267ce7fc-bf55-11df-a3b9-6e8dd877b814

 

        Number of points_of_contact for node: 1

        Point-of-contact interface & contact state

         en0  UP

 

------------------------------

 

        Node name: 7502lp02

        Cluster shorthand id for node: 2

        uuid for node: d1a46164-bf46-11df-94b3-6e8dd877b814

        State of node:  UP  NODE_LOCAL

        Smoothed rtt to node: 0

        Mean Deviation in network rtt to node: 0

        Number of zones this node is a member in: 0

        Number of clusters node is a member in: 1

        CLUSTER NAME       TYPE  SHID   UUID

        mycluster          local        267ce7fc-bf55-11df-a3b9-6e8dd877b814

 

        Number of points_of_contact for node: 0

        Point-of-contact interface & contact state

         n/a

 

CAA tells you what interfaces have been discovered on a node plus information on those interfaces, including state.

 

# lscluster –i –n

Node 7502lp01

Node uuid = 110b2422-7efc-11df-aed7-1612a0003002

Number of interfaces discovered = 1

        Interface number 1 en0

                ifnet type = 6 ndd type = 7

                Mac address length = 6

                Mac address =  16.12.a0.0.30.2

                Smoothed rrt across interface = 7

                Mean Deviation in network rrt across interface = 3

                Probe interval for interface = 100 ms

                ifnet flags for interface = 0x1e080863

                ndd flags for interface = 0x21081b

                Interface state  UP

                Number of regular addresses configured on interface = 1

                IPV4 ADDRESS: 9.3.28.136  broadcast 9.3.28.159  netmask 255.255.255.224

                Number of cluster multicast addresses configured on interface = 1

                IPV4 MULTICAST ADDRESS: 228.8.8.8  broadcast 0.0.0.0  netmask 0.0.0.0

 

CAA tells you what disks are in the cluster plus information on those disks, including the state and type.

 

# lscluster –d

 

Storage Interface Query

Cluster Name:  mycluster

Cluster uuid: a1bde416-83b9-11df-b438-1612a0004002

Number of nodes reporting = 2

Number of nodes expected = 2

Node oscar-dev8.austin.ibm.com

Node uuid = 9807465b-3194-f1e5-8a04-1f044bc82593

Number of disk discovered = 2

        cldisk1

          state : UP

          uDid  : 533E3E213600A0B8000475C200000E7114BC825930F1818      FAStT03IB

Mfcp05VDASD03AIXvscsi

             type  : CLUSDISK

          uUid  : 9807465b-3194-f1e5-8a04-1f044bc82593

        hdisk1

          state : UP

          uDid  :

          uUid  : 600a0b80-0047-5d0a-0000-e6094bc82690

          type  : REPDISK

 

You can run cluster wide commands on all nodes immediately after configuring your cluster using the clcmd command. No further configuration required.

 

# clcmd ps -ef

 

 

-------------------------------

NODE 7502lp01

-------------------------------

     UID     PID         PPID      C    STIME    TTY  TIME CMD

    root       1           0       0   Aug 25      -  0:00 /etc/init

    root  655526           1       0   Aug 25      -  0:18 [cimserve]

    root 1310840           1       0   Aug 25      -  0:00 /usr/ccs/bin/shlap64

    root 2490482           1       0   Aug 25      -  0:42 /usr/sbin/srcmstr

    root 3145874         3801262   0 11:17:13      -  0:00 telnetd -a

    root 3276994         2490482   0   Aug 25      -  0:10 /usr/sbin/snmpd

 

-------------------------------

NODE 7502lp02

-------------------------------

     UID     PID         PPID   C    STIME    TTY  TIME CMD

    root       1           0   1   Aug 28      -  0:00 /etc/init

    root 1441934           1   0   Aug 28      -  0:39 /usr/sbin/syncd 60

    root 1638496           1   0   Aug 28      -  0:00 /usr/ccs/bin/shlap64

    root 2228252     2556054   0   Aug 28      -  0:00 /usr/sbin/hostmibd

   

I believe that PowerHA SystemMirror 7.1 will be the first cluster product from IBM to provide high availability on AIX using the new Cluster Aware AIX features. An IBM Redbook Residency recently covered the integration of CAA and PowerHA. I’m eagerly awaiting the first draft of this book so that I can learn how these two components will “play” together.

 

Refer to the AIX 7.1 Information Centre for more information on Cluster Aware AIX:

 

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.clusteraware/claware_main.htm

 

 

Refer to the AIX 7.1 Information Centre for more information on AHAFS and the new AIX Event Infrastructure:

 

http://publib.boulder.ibm.com/infocenter/aix/v7r1/index.jsp?topic=/com.ibm.aix.baseadmn/doc/baseadmndita/aix_ev.htm

 

 

 

In this post I’ll demonstrate how to create a Cluster Aware AIX (CAA) cluster on AIX 7.1. I’ll create a simple two-node CAA cluster. I’ll then check the cluster status and, finally, remove the cluster.

 

Before I start, I make sure that both nodes have entries in /etc/hosts on both systems.

 

node1[/] > grep node /etc/hosts

172.29.134.169  node1

172.29.152.160  node2

 

node2[/] > grep node /etc/hosts

172.29.134.169  node1

172.29.152.160  node2

 

I verify that cluster services are not active on both nodes.

 

node1[/tmp/cg] > lscluster -i

Cluster services are not active.

 

node2[/] >  lscluster -i

Cluster services are not active.

 

I’ll now create a cluster named mycluster. My CAA repository disk is named hdisk1.

 

node1[/] > mkcluster -n mycluster -m node1,node2 -r hdisk1

 

My two node CAA cluster is now operational.

 

node1[/] > lscluster -c

Cluster query for cluster mycluster returns:

Cluster uuid: 2a476f06-be50-11e0-9781-66da93d16b18

Number of nodes in cluster = 2

        Cluster id for node node2 is 1

        Primary IP address for node node2 is 172.29.152.160

        Cluster id for node node1 is 2

        Primary IP address for node node1 is 172.29.134.169

Number of disks in cluster = 0

Multicast address for cluster is 228.29.134.169

 

I can immediately execute commands on both nodes using the clcmd utility.

 

node1[/] > clcmd uptime

 

-------------------------------

NODE node1

-------------------------------

  11:16PM   up   1:16,  1 user,  load average: 1.10, 1.18, 1.36

 

 

-------------------------------

NODE node2

-------------------------------

  11:16PM   up 45 mins,  1 user,  load average: 0.97, 0.75, 0.64

 

Prior to cluster creation, the disks on my system were named as follows:

 

node1[/] > lspv

hdisk1          00f6048892b78f9f                    None

hdisk0          00f6050a2cd79ef8                    rootvg          active

 

After cluster creation, hdisk1 is renamed to caa_private0 and is assigned to the CAA repository volume group named caavg_private.

 

node1[/] > lspv

caa_private0    00f6048892b78f9f                    caavg_private   active

hdisk0          00f6050a2cd79ef8                    rootvg          active

 

node1[/] > lsvg caavg_private

VOLUME GROUP:       caavg_private            VG IDENTIFIER:  00f6048800004c0000000131930c1af3

VG STATE:           active                   PP SIZE:        64 megabyte(s)

VG PERMISSION:      read/write               TOTAL PPs:      799 (51136 megabytes)

MAX LVs:            256                      FREE PPs:       784 (50176 megabytes)

LVs:                6                        USED PPs:       15 (960 megabytes)

OPEN LVs:           2                        QUORUM:         2 (Enabled)

TOTAL PVs:          1                        VG DESCRIPTORS: 2

STALE PVs:          0                        STALE PPs:      0

ACTIVE PVs:         1                        AUTO ON:        yes

MAX PPs per VG:     32512

MAX PPs per PV:     1016                     MAX PVs:        32

LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no

HOT SPARE:          no                       BB POLICY:      relocatable

PV RESTRICTION:     none

 

node1[/] > lsvg -l caavg_private

caavg_private:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

caalv_private1      boot       1       1       1    closed/syncd  N/A

caalv_private2      boot       1       1       1    closed/syncd  N/A

caalv_private3      boot       4       4       1    open/syncd    N/A

fslv00              jfs2       4       4       1    open/syncd    /clrepos_private1

fslv01              jfs2       4       4       1    closed/syncd  /clrepos_private2

powerha_crlv        boot       1       1       1    closed/syncd  N/A

 

node1[/tmp/cg] > df -g

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4           0.25      0.07   74%    10118    37% /

/dev/hd2           2.50      0.19   93%    45192    48% /usr

/dev/hd9var        0.38      0.02   95%     5995    47% /var

/dev/hd3           0.50      0.38   24%      187     1% /tmp

/dev/hd1           0.12      0.12    1%       11     1% /home

/dev/hd11admin      0.12      0.12    1%        5     1% /admin

/proc                 -         -    -         -     -  /proc

/dev/hd10opt       0.38      0.13   65%     8021    21% /opt

/dev/livedump      0.25      0.25    1%        4     1% /var/adm/ras/livedump

/aha                  -         -    -        37     1% /aha

/dev/fslv00        0.25      0.25    2%       15     1% /clrepos_private1

 

Several services are started during cluster creation. In particular you will notice the following subsystems active (lssrc –a output).

 

cld              caa              7340104      active

solidhac         caa              8912986      active

solid            caa              8519794      active

clconfd          caa              9306188      active

cthags           cthags           9830450      active

ctrmc            rsct             3342554      active

IBM.DRM          rsct_rm          8716462      active

IBM.ServiceRM    rsct_rm          9240724      active

IBM.StorageRM    rsct_rm          8061146      active

IBM.MgmtDomainRM rsct_rm          8454200      active

 

node1[/tmp/cg] > lssrc -g caa

Subsystem         Group            PID          Status

 clcomd           caa              6553810      active

 cld              caa              6619252      active

 solidhac         caa              8650856      active

 solid            caa              8781824      active

 clconfd          caa              10092598     active

 

Now I will remove the cluster.

 

node1[/] > rmcluster -n mycluster

 

rmcluster: Removed cluster shared disks are automatically renamed to names such

        as hdisk10, [hdisk11, ...] on all cluster nodes.  However, this cannot

        take place while a disk is busy or on a node which is down or not

        reachable.  If any disks cannot be renamed now, you must manually

        rename them by removing them from the ODM database and then running

        the cfgmgr command to recreate them with default names.  For example:

        rmdev -l cldisk1 -d

        rmdev -l cldisk2 -d

        cfgmgr

 

CAA cluster services are no longer active.

 

node1[/] > lscluster -c

Cluster services are not active.

 

And caa_private0 is renamed back to hdisk1 and the caavg_private volume group is removed also.

 

node1[/] > lspv

 

hdisk1          00f6048892b78f9f                    None

hdisk0          00f6050a2cd79ef8                    rootvg          active

 

node1[/] > df

Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4          524288    139136   74%    10097    37% /

/dev/hd2         5242880    405088   93%    45192    48% /usr

/dev/hd9var       786432     88040   89%     5986    35% /var

/dev/hd3         1048576    806528   24%      186     1% /tmp

/dev/hd1          262144    261352    1%       11     1% /home

/dev/hd11admin     262144    261416    1%        5     1% /admin

/proc                  -         -    -         -     -  /proc

/dev/hd10opt      786432    279912   65%     8021    21% /opt

/dev/livedump     524288    523552    1%        4     1% /var/adm/ras/livedump

/aha                   -         -    -        32     1% /aha

node1[/] >

 

 

 

 

Whilst working with one of my AIX customers recently I discovered a problem with NIMSH and SSL. The customer had updated their NIM master from AIX 7.1 TL4 SP1 to AIX 7.1 TL4 SP3. After the SP update, any attempt to connect to a NIM client (over NIMSH+SSL), from the NIM master, would simply hang. For example, we tried to list the filesets on the NIM client with this command, which never returned any output.

 

[root@750lpar4]/ # nim -o lslpp 750lpar9

 

The /var/adm/ras/nimsh.log file, on the NIM client, showed that the NIMSH session stopped here:

 

…

Thu Jan 12 14:31:49 2017        Loading certificates..

Thu Jan 12 14:31:49 2017        Loading private key file..

Thu Jan 12 14:31:49 2017        create BIO

…

 

NIM master: 750lpar4

7100-04-03-1543

 

NIM client: 750lpar9

7100-04-02-1614

 

[root@750lpar4]/ # lsnim -l 750lpar9

750lpar9:

   class          = machines

   type           = standalone

   connect        = nimsh (secure)

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar9 0

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00

   Cstate_result  = success

 

The root cause of the problem become apparent when we ran truss against the nim –o command.

 

[root@750lpar4]/ # truss -adef -o truss.lsnim.out -w all nim -o lslpp 750lpar9

 

[root@750lpar4]/ # cat truss.lsnim.out

…

13959372: C o u l d   n o t   l o a d   m o d u l e   / u s r / l i b / l

13959372: i b s s l . s o .\n S y s t e m   e r r o r :   N o   s u c h

13959372: f i l e   o r   d i r e c t o r y

…

19267612: C o u l d   n o t   l o a d   m o d u l e   / u s r / l i b / l

19267612: i b c r y p t o . s o .\n S y s t e m   e r r o r :   N o   s u

19267612: c h   f i l e   o r   d i r e c t o r y

 

The required shared library object files were missing on the NIM master.

 

[root@750lpar4]/usr/lib # ls -ltr libssl.so libcrypto.so

libssl.so not found

libcrypto.so not found

 

We fixed this issue by extracting the missing files from the (existing) /usr/lib/libssl.a and /usr/lib/libcrypto.a archives.

 

[root@750lpar4]/usr/lib # slibclean

 

[root@750lpar4]/usr/lib # /bin/ar -v -x /usr/lib/libssl.a /usr/lib/libssl.so

x - /usr/lib/libssl.so

 

[root@750lpar4]/usr/lib # /bin/ar -v -x /usr/lib/libcrypto.a /usr/lib/libcrypto.so

x - /usr/lib/libcrypto.so

 

[root@750lpar4]/usr/lib # ls -ltr libssl.so libcrypto.so

-rwxr-xr-x    1 root     system       724913 Jan 18 09:08 libssl.so

-rwxr-xr-x    1 root     system      3031337 Jan 18 09:08 libcrypto.so

 

After that, the nim –o commands started working again.

 

[root@750lpar4]/usr/lib # nim -o showlog 750lpar9

HELLO

 

So, the question is why did this happen? Well, in the past, the libssl.so.0.9.8 shared object was extracted by NIM, but more recent updates by the OpenSSL version has forced IBM to move to libssl.so. Usually, the extracted shared library object is added (if not currently present) when nimconfig -c is run.  But given that this is an existing NIM master, we did not want to run this again (as we would lose all of the current SSL key access). So extracting the objects is preferred. The problem is due to the fact that the libssl.so and libcrypto.so files are not populated when the AIX 7100-04-03 update is applied. This is a bug and will be officially addressed, soon, under APAR IV93152 NIM push operation to client hang on nimsh over SSL.

 

I believe this issue may also occur when you migrate your NIM master from AIX 7.1 to 7.2 (with nimadm for example). But I need to do more testing to reproduce and confirm the issue.

 

Here’s one good reason to setup NIMSH over SSL.

 

NIMSH, SSL and LPM

https://www.ibm.com/developerworks/community/blogs/cgaix/entry/NIMSH_SSL_and_LPM?lang=en

 

The following link is a great reference guide for configuring NIMSH over SSL.

 

NIMSH over SSL

http://www-01.ibm.com/support/docview.wss?uid=isg3T1024615

 

 

 

 

I’ve been working with a customer recently on an issue with nimadm. They were attempting to migrate a system from AIX 5.3 to 7.1 using nimadm. The NIM client AIX level was 5.3 TL12 SP4 and the NIM master was running AIX 7.1 TL1 SP1.

 

lpar1 : / # oslevel -s

5300-12-04-1119

 

root@nim1 : / # oslevel -s

7100-01-01-1141

 

They were running the following nimadm command:

 

# nimadm -j nimadmvg -c lpar1 -s spotaix710101 -l lpp_sourceaix710101 -d hdisk1 -Y

 

The nimadm operation would always fail at phase 11:

 

+-----------------------------------------------------------------------------+
Executing nimadm phase 11.
+-----------------------------------------------------------------------------+
Cloning altinst_rootvg on client, Phase 3.
Client alt_disk_install command: alt_disk_copy -j -M 7.1 -P3 -d "hdisk1"
## Phase 3 ###################
Verifying altinst_rootvg...
alt_disk_copy: 0505-218 ATTENTION: init_multibos() returned an unexpected result.
Cleaning up.
forced unmount of /alt_inst/var/log
forced unmount of /alt_inst/var/log
forced unmount of /alt_inst/var
forced unmount of /alt_inst/var
forced unmount of /alt_inst/usr/local
forced unmount of /alt_inst/usr/local
forced unmount of /alt_inst/usr
forced unmount of /alt_inst/usr
forced unmount of /alt_inst/tmp
forced unmount of /alt_inst/tmp
forced unmount of /alt_inst/opt
forced unmount of /alt_inst/opt
forced unmount of /alt_inst/home
forced unmount of /alt_inst/home
forced unmount of /alt_inst/admin
forced unmount of /alt_inst/admin
forced unmount of /alt_inst
forced unmount of /alt_inst
0505-187 nimadm: Error cloning altinst_rootvg on client.
Cleaning up alt_disk_migration on the NIM master.
Cleaning up alt_disk_migration on client lpar1.
Client alt_disk_install command: alt_disk_install -M 7.1 -X
Bootlist is set to the boot disk: hdisk0 blv=hd5

 

We also found the init_multibos error in the /var/adm/ras/alt_disk_inst.log file on the NIM client:

 

Tue Nov 22 14:48:12 EETDT 2011

cmd: /ALT_MIG_SPOT/sbin/alt_disk_copy -j -M 7.1 -P3 -d hdisk1

Verifying altinst_rootvg...

alt_disk_copy: 0505-218 ATTENTION: init_multibos() returned an unexpected result.

Cleaning up.

 

Given that the error appeared to be related to init_multibos, we assumed the failure was due to some multibos checks being performed by alt_disk_copy on the client. The client system did not have an existing multibos standby instance. So, we tried two things: First we created a standby instance on the client (multibos –s –X) and re-tried the nimadm operation. This failed. Next we removed the standby instance (multibos –R) and re-tried the nimadm operation. This worked and the client then migrated to AIX 7.1 successfully. We re-tried the same operations (i.e. create standby instance, remove standby instance & nimadm) several times and each worked as expected.

 

So it appeared that the unofficial work around to this problem would be to create and then remove a standby multibos instance prior to the nimadm migrate. However, the customer has over 200 LPARs that they need to migrate to AIX 7.1. If possible they would really rather avoid this extra step in the AIX 7.1 migration plan. We’ve made contact with IBM support and are hoping they can assist us in identifying the root cause of the issue and provide us with an official solution to the problem.

 

And just yesterday we hit the same problem when migrating from AIX 6.1 to 7.1 using nimadm. I’ll update my blog with any progress we make with this problem. In the meantime, our unofficial work around will get us “out of hot water”!

 

UPDATE (14/12/2011): The simple fix is to remove the /bos_inst directory before attempting the AIX migration. i.e.

 

# rm –r /bos_inst

 

In my nimadm article I wrote:

• If you have a multibos image in rootvg, remove it. AIX migrations are not supported with multibos enabled systems. Ensure all rootvg LVs are renamed to their legacy names. If necessary, create a new instance of rootvg and reboot the LPAR. For example:

# multibos –sXp         # multibos –sX         # shutdown –Fr

 

Confirm the legacy LV names are now in use that is, not bos_.

# lsvg -l rootvg | grep hd | grep open         hd6           paging     80      160     2    open/syncd    N/A         hd8           jfs2log    1       2       2    open/syncd    N/A         hd4           jfs2       1       2       2    open/syncd    /         hd2           jfs2       7       14      2    open/syncd    /usr         hd3           jfs2       16      32      2    open/syncd    /tmp         hd1           jfs2       1       2       2    open/syncd    /home         hd9var        jfs2       8       16      2    open/syncd    /var         hd7           sysdump    8       8       1    open/syncd    N/A         hd7a          sysdump    8       8       1    open/syncd    N/A         hd10opt       jfs2       8       16      2    open/syncd    /opt

 

Remove the old multibos instance.

# multibos -R

 

Unfortunately, it appears that ‘multibos –R’ may not clean up the /bos_inst directory. If this directory exists the nimadm operation will most likely fail.

 

 

 

 

 

I remember reading this interesting document about some challenges renaming AIX devices.

 

http://gibsonnet.net/aix/ibm/The_Fun_of_Oracle_RAC_ASM_and_Device_Names.doc

 

This poor guy had to jump through some hoops to rename some disk and network devices in a clustered AIX (Oracle RAC) environment.

 

This comment stayed with me:

 

“The easy way to get network adapters and disk names to match between systems would be to run something like

 

# chdev –l old_name –n new_name

 

Except, this command option does NOT exist.”

 

Since then I’ve always been on the “look out” for an easy way to rename devices in AIX.

 

In particular in large clustered environments where it is sometimes very important to have the same disk and network device names in sync across all nodes in a cluster. And besides, it’s a lot easier  to verify a cluster configuration if the hdisk names are all the same. Matching PVIDs works but it requires a lot more effort! For example, knowing that hdisk123 is the same device on all nodes makes life easier than scanning lspv output for a PVID like 00f6048868b4gead.  Of course you can script things to make this easier but it would be great if you didn’t need to do this and that there was a way to rename devices as needed, without resorting to unsupported methods.

 

Well, this is no longer an issue for AIX.

 

Starting with AIX 7.1, you can now easily rename devices. A new command called rendev was introduced to allow AIX administrators to rename devices as required.

 

From the man page:

 

The rendev command enables devices to be renamed. The device to be renamed, is specified with the -l flag, and the new desired name is specified with the -n flag.

 

The new desired name must not exceed 15 characters in length. If the name has already been used or is present in the /dev directory, the operation fails. If the name formed by appending  the new name after the character r is already used as a device name, or appears in the /dev directory, the operation fails.

 

 If the device is in the Available state, the rendev command must unconfigure the device before renaming it. This is similar to the operation performed by the rmdev -l Name command. If the unconfigure operation fails, the renaming will also fail. If the unconfigure succeeds, the rendev command will configure the device, after renaming it, to restore it to the Available state. The -u flag may be used to prevent the device from being configured again after it is renamed.

 

 Some devices may have special requirements on their names in order for other devices or applications to use them. Using the rendev command to rename such a device may result in the device being unusable. Note: To protect the configuration database, the rendev command cannot be interrupted once it has started. Trying to stop this command before completion, could result in a corrupted database.

 

This functionality was required to help simplify cluster device configuration with the new Cluster Aware AIX (CAA) feature in AIX 7.1.

 

I’ll be blogging about CAA in a future post.

 

Here are some examples of using the rendev command on AIX 7.1 (GA) system. In the first example I will rename hdisk3 to hdisk300. Note: hdisk3 is not in use (busy).

If the disk had been allocated to a volume group, I would have needed to varyoff the volume group first.

 

 

# lspv

hdisk0          00f61ab2f73e46e2                    rootvg          active

hdisk1          00f61ab20bf28ac6                    None

hdisk2          00f61ab2202f7c0b                    None

hdisk4          00f61ab20b97190d                    None

hdisk3          00f61ab2202f93ab                    None

 

# rendev -l hdisk3 -n hdisk300

 

# lspv

hdisk0          00f61ab2f73e46e2                    rootvg          active

hdisk1          00f61ab20bf28ac6                    None

hdisk2          00f61ab2202f7c0b                    None

hdisk4          00f61ab20b97190d                    None

hdisk300        00f61ab2202f93ab                    None

 

Easy!

 

Next, I’ll rename a virtual SCSI adapter. I renamed vscsi0 to vscsi2. Note: I placed the adapter, vscsi0, in a Defined state before renaming the device.

 

# rmdev -Rl vscsi0

 

# lsdev -Cc adapter

ent0   Available  Virtual I/O Ethernet Adapter (l-lan)

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi0 Defined    Virtual SCSI Client Adapter

vscsi1 Available  Virtual SCSI Client Adapter

 

# rendev -l vscsi0 -n vscsi2

 

# lsdev -Cc adapter

ent0   Available  Virtual I/O Ethernet Adapter (l-lan)

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi1 Available  Virtual SCSI Client Adapter

vscsi2 Defined    Virtual SCSI Client Adapter

 

Now I’ll rename a network adapter from ent0 to ent10. I bring down the interface before changing the device name

 

# lsdev -Cc adapter

ent0   Available  Virtual I/O Ethernet Adapter (l-lan)

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi1 Available  Virtual SCSI Client Adapter

vscsi2 Defined    Virtual SCSI Client Adapter

 

# ifconfig en0

en0: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>

        inet 10.1.20.19 netmask 0xffff0000 broadcast 10.153.255.255

         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

 

# ifconfig en0 down detach

 

# rendev -l ent0 -n ent10

 

# lsdev -Cc adapter

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

ent10  Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi1 Available  Virtual SCSI Client Adapter

vscsi2 Defined    Virtual SCSI Client Adapter

 

# rendev -l en0 -n en10

 

# chdev -l en10 -a state=up

en10 changed

# mkdev -l inet0

inet0 Available

# ifconfig en10

en10: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>

        inet 10.1.20.19 netmask 0xffff0000 broadcast 10.153.255.255

         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

 

If you want to be creative you can rename devices to anything you like (as long as it’s not more than 15 characters). For example I’ll rename vscsi2 to myvscsiadapter.

 

 

# rendev -l vscsi2 -n myvscsiadapter

# lsdev -Cc adapter

ent1           Available  Virtual I/O Ethernet Adapter (l-lan)

myadapter      Available  Virtual I/O Ethernet Adapter (l-lan)

myvscsiadapter Defined    Virtual SCSI Client Adapter

vsa0           Available  LPAR Virtual Serial Adapter

vscsi1         Available  Virtual SCSI Client Adapter

 

And in the last example I’ll demonstrate changing virtual SCSI adapter device names on a live system.

 

This is single disk system (hdisk0), with two vscsi adapters.

 

# lspv

hdisk0          00f6048868b4deee                    rootvg          active

 

# lspath

Enabled hdisk0 vscsi0

Enabled hdisk0 vscsi1

 

# lsdev -Cc adapter

ent0   Available  Virtual I/O Ethernet Adapter (l-lan)

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi0 Available  Virtual SCSI Client Adapter

vscsi1 Available  Virtual SCSI Client Adapter

 

We ensure the adapter is in a Defined state before renaming it. This will fail otherwise.

 

# rmdev -Rl vscsi1

vscsi1 Defined

# lsdev -Cc adapter | grep vscsi

vscsi0 Available  Virtual SCSI Client Adapter

vscsi1 Defined    Virtual SCSI Client Adapter

 

Now we rename the adapter vscsi1 to vscsi3.

 

# rendev -l vscsi1 -n vscsi3

 

# lsdev -Cc adapter | grep vscsi

vscsi0 Available  Virtual SCSI Client Adapter

vscsi3 Defined    Virtual SCSI Client Adapter

 

That was easy enough. Now I need to bring the adapter and path online with cfgmgr. The lspath output displays an additional path to vscsi3.

 

# lspath

Enabled hdisk0 vscsi0

Defined hdisk0 vscsi1

 

# cfgmgr

Method error (/etc/methods/cfgscsidisk -l hdisk0 ):

        0514-082 The requested function could only be performed for some

                 of the specified paths.

 

# lspath

Enabled hdisk0 vscsi0

Defined hdisk0 vscsi1

Enabled hdisk0 vscsi3

 

Now I need to remove the old path to vscsi1. The path to vscsi3 is now Enabled. The adapter, vscsi3, is in an Available state. All is good.

 

# rmpath -l hdisk0 -p vscsi1 -d

path Deleted

 

# lspath

Enabled hdisk0 vscsi0

Enabled hdisk0 vscsi3

 

# lsdev -Cc adapter | grep vscsi

vscsi0 Available  Virtual SCSI Client Adapter

vscsi3 Available  Virtual SCSI Client Adapter

 

The same steps need to be repeated for the vscsi0 adapter. This is renamed to vscsi2.

 

# rmdev -Rl vscsi0

vscsi0 Defined

# lsdev -Cc adapter | grep vscsi

vscsi0 Defined    Virtual SCSI Client Adapter

vscsi3 Available  Virtual SCSI Client Adapter

 

 

# rendev -l vscsi0 -n vscsi2

 

# lsdev -Cc adapter | grep vscsi

vscsi2 Defined    Virtual SCSI Client Adapter

vscsi3 Available  Virtual SCSI Client Adapter

 

 

# lspath

Defined hdisk0 vscsi0

Enabled hdisk0 vscsi3

 

# cfgmgr

Method error (/etc/methods/cfgscsidisk -l hdisk0 ):

        0514-082 The requested function could only be performed for some

                 of the specified paths.

 

# lspath

Defined hdisk0 vscsi0

Enabled hdisk0 vscsi2

Enabled hdisk0 vscsi3

 

# rmpath -l hdisk0 -p vscsi0 -d

path Deleted

 

# cfgmgr

# lspath

Enabled hdisk0 vscsi2

Enabled hdisk0 vscsi3

 

That’s it. Both adapters have been renamed while the system was in use. No downtime required.

 

# lsdev -Cc adapter | grep vscsi

vscsi2 Available  Virtual SCSI Client Adapter

vscsi3 Available  Virtual SCSI Client Adapter

 

# lspath

Enabled hdisk0 vscsi2

Enabled hdisk0 vscsi3

 

Please refer to the AIX 7.1 command reference for more information on this new command:

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.cmds/doc/aixcmds4/rendev.htm

 

And by the way, here’s a draft of what I’ve been working on for the last 5 weeks:

 

IBM AIX Version 7.1 Differences Guide:

http://www.redbooks.ibm.com/Redbooks.nsf/RedpieceAbstracts/sg247910.html?Open

 

Follow me on Twitter:

http://twitter.com/cgibbo

 

After updating my NIM master to AIX 7.1 TL2 SP1 (7100-02-01-1245), I noticed a problem. Whenever I installed a new AIX partition using NIM, the resources allocated to the NIM client were not being de-allocated, even though the installation was completing successfully. Also, if I tried to run my usual ‘NIM client reset’ script (below), the resources were still allocated.

 

#!/usr/bin/ksh

# Reset a NIM client.

 

if [[ "$1" = "" ]] ; then

  echo Please specify a NIM client to reset e.g. aixlpar1.

else

  if lsnim -l $1 > /dev/null 2>&1 ; then

    nim -o reset -F $1

    nim -Fo de-allocate  -a subclass=all $1

    nim -Fo change -a cpuid= $1

  else

    echo Not a valid NIM client!?

  fi

fi

                                                                     

For example, here’s my NIM client with the lpp_source, mksysb and SPOT resources assigned to it (even though the AIX install completed OK).

                                                                       

root@nim1 : / # lsnim -l aixlpar1                                

aixlpar1:                                                           

   class           = machines                                          

   type            = standalone                                        

   connect         = shell                                             

   platform        = chrp                                              

   netboot_kernel  = 64                                                

   if1             = network1 aixlpar1 0                             

   cable_type1     = N/A                                               

   Cstate          = ready for a NIM operation                         

   prev_state      = not running                                       

   Mstate          = currently running                                 

   boot            = boot                                              

   lpp_source      = lpp_sourceaix710105                               

   mksysb          = aixlpar1-71                                 

   nim_script      = nim_script                                        

   spot            = spotaix710105                                     

   cpuid           = 00C453C75C00                                      

   control         = master                                            

   Cstate_result   = success                                           

   installed_image = aixlpar1-71                  

                                                                        

My workaround was to use 'smit nim_mac_res' to manually de-allocate  resources from the client:                               

                                                                       

====                                                                    

De-allocate  Network Install Resources                                   

aixlpar1     machines       standalone                              

                                                                       

>lpp_sourceaix710105     lpp_source                                    

>spotaix710105           spot                                          

>aixlpar1-71       mksysb                                        

====                                                                    

                                                                                                                                             

It appears that others were also experiencing this problem. I found the following thread on the IBM developerWorks AIX user forum:     

                                             

NIM de-allocate  with subclass=all doesn't work:

https://www.ibm.com/developerworks/forums/thread.jspa?threadID=464710

 

Someone even provided their own version of a ‘NIM client reset’ script that offered another workaround to the problem:

 

nim -Fo reset $1

for i in bosinst_data fb_script image_data installp_bundle lpp_source mksysb nim_script script spot ; do

  ressources_to_clean=$(lsnim -l $1 | grep $i | awk '{print $NF}')

  for j in ${ressources_to_clean} ; do

    nim -Fo de-allocate  -a $i=$j $1

  done

done

                                                                   

I reported the issue to IBM AIX support and discovered that this is a known issue with 7.1 TL2.

 

“This is a known issue which recently has been documented in 2 open     

APARs:  IV32570 and IV32670:  UPDATE NIM BINARIES FOR UPDATES.         

                                                                       

Error description                                                   

One known symptom (of this issue) is that "nim -Fo               

de-allocate  -a subclass=all <client>" returns successfully but the

resource will still be allocated.                                 

                                                                       

The problem occured because the "update" pack that brings you from     

7.1.1.x to 7.1.2.0 missed a piece that effected the bos.sysmgt.7.1.2.0 

fileset.                                                               

                                                                       

Currently there is no fix available and no projected timeframe when this

issue will be fixed (i assume it will be with the next SP for AIX      

7100-02)                                                               

                                                                       

However, a workaround is allready documented in our incidents          

database - you simply need to force reinstall the bos.sysmgt.nim.master

7.1.2.0 fileset using your AIX 7.1 TL02 media.                         

                                                                        

This is not a backleveling of the fileset, you are simply force        

overwriting the full base level at the same version that you are       

currently at.  It will still be at 7100-02.                            

                                                                        

The workaround is a force overwrite of the bos.sysmgt.nim.master 7.1.2.0

from media." 

 

The AIX support team suggested that I download the necessary install fileset from the following location:

 

ftp://testcase.software.ibm.com/fromibm/aix/bos.sysmgt.7.1.2.0.I  

 

I backed up my NIM database before re-installing the fileset, just in case.

 

---------------------------------------------------------

Starting NIM Database Backup on nim1 at Wed Dec 12 12:39:43 EETDT 2012

 

a ./etc/objrepos/nim_attr 32 blocks.

a ./etc/objrepos/nim_attr.vc 112 blocks.

a ./etc/objrepos/nim_object 8 blocks.

a ./etc/objrepos/nim_object.vc 24 blocks.

a ./etc/NIM.level 1 blocks.

a ./etc/niminfo 1 blocks.

a ./etc/NIM.primary.cpuid 1 blocks.

 

Finished NIM Database Backup on nim1 at Wed Dec 12 12:39:44 EETDT 2012

---------------------------------------------------------

 

I placed the fileset into a separate directory on the NIM master and ran inutoc.

 

root@nim1 : /home/cgibson/NIMPMR # ls -ltr

total 36170

-rw-r--r--    1 cgibson  staff      18512896 Dec 12 12:21 bos.sysmgt.7.1.2.0.I

-rw-r--r--    1 root     system         4778 Dec 12 12:35 .toc

 

Then I re-installed the fileset (force OVERWRITE) on the NIM master:

 

                             Install Software

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                                                        [Entry Fields]

* INPUT device / directory for software               .

* SOFTWARE to install                                [@ 7.1.2.0  Network Install Manager - Master Tools                      @@I:bos.sysmgt.nim.master 7>

  PREVIEW only? (install operation will NOT occur)    no

  COMMIT software updates?                            yes

  SAVE replaced files?                                no

  AUTOMATICALLY install requisite software?           no 

  EXTEND file systems if space needed?                yes

  OVERWRITE same or newer versions?                   yes

  VERIFY install and check file sizes?                no 

  Include corresponding LANGUAGE filesets?            yes 

  DETAILED output?                                    yes 

  Process multiple volumes?                           yes 

  ACCEPT new license agreements?                      yes  

  Preview new LICENSE agreements?                     no      

 

 

The fileset re-installed without a problem.

 

Installation Summary

--------------------

Name                        Level           Part        Event       Result

-------------------------------------------------------------------------------

bos.sysmgt.nim.master       7.1.2.0         USR         APPLY       SUCCESS

 

root@nim1 : /home/cgibson/NIMPMR # lslpp -l bos.sysmgt.nim.master

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  bos.sysmgt.nim.master      7.1.2.0  COMMITTED  Network Install Manager -

                                                 Master Tools

 

I re-tried installing AIX on a NIM client.

 

root@nim1 : / # lsnim -l aixlpar1

aixlpar1:

   class           = machines

   type            = standalone

   connect         = shell

   platform        = chrp

   netboot_kernel  = 64

   if1             = network1 aixlpar1 0

   cable_type1     = N/A

   Cstate          = Base Operating System installation is being performed

   prev_state      = BOS installation has been enabled

   Mstate          = in the process of booting

   info            = verifying_data_files

   boot            = boot

   lpp_source      = lpp_sourceaix710105

   mksysb          = aixlpar1-71

   nim_script      = nim_script

   spot            = spotaix710105

   cpuid           = 00C453C75C00

   control         = master

   Cstate_result   = success

   installed_image = aixlpar1-71

 

Once the install completed I found that all allocated NIM resources (lpp_source, mksysb and SPOT) were de-allocated. Problem solved.

 

root@nim1 : / # lsnim -l aixlpar1

aixlpar1:

   class           = machines

   type            = standalone

   connect         = shell

   platform        = chrp

   netboot_kernel  = 64

   if1             = network1 aixlpar1 0

   cable_type1     = N/A

   Cstate          = ready for a NIM operation

   prev_state      = in the process of booting

   Mstate          = not running

   cpuid           = 00C453C75C00

   Cstate_result   = success

   installed_image = aixlpar1-71

 

So if you are planning on updating your NIM master to 7.1 TL2, please take note and make sure you implement the workaround.