Chris's AIX Blog

So you’re configuring a new PowerHA 6.1 cluster and you decide to check if the clinfoES subsystem is running. You run the following command:

 

# lssrc -a | grep cluster

clstrmgrES       cluster          2883628      active

 

Hmmm, clinfoES is not listed. To ensure that is clinfoES is shown in lssrc output, in both inactive and active states, run the following command on all nodes in the cluster:

 

# chssys -s clinfoES –d

 

And now clinfoES is shown:

 

#lssrc -a | grep cluster

clstrmgrES       cluster          2883628      active

clinfoES         cluster                       inoperative

 

There was also a time when you had to do the same for clstrmgrES, however this subsystem is now always active (very rarely/never inactive) on a HA node e.g.

 

# chssys -s clstrmgrES -d

 

The chssys command can make changes to a subsystems definition on AIX:

 

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.cmds/doc/aixcmds1/chssys.htm

 

The –d flag to the chssys command specifies that an inactive subsystem is displayed in lssrc output.

 

I found a very old reference to something similar in this old HACMP/ES Redbook, but only the clinfoES recommendation still applies:

 

For a better understanding and to avoid problem determination, it is recommended to make all relevant HACMP subsystems visible with the lssrc command, independent of whether they are active or not.

 

# chssys -s clinfoES -d # modify subsystem definition

# chssys -s cllockdES -d

# chssys -s clsmuxpdES -d

# chssys -s clvmd –d

 

http://ps-2.kev009.com/rs6000/redbook-cd/SG245526.pdf

 

 

Starting with AIX 7.1, CSM is no longer supported or available. It has been replaced by Distributed Systems Managment (DSM).   Section 5.2 of the IBM AIX 7.1 Differences Guide Redbook provides details of the new DSM capabilities.

 

Fortunately DSM still provides access to the dsh command.  I’ve written about how I’ve used this utility in the past. The new dsh command (and other tools) are provided in the new DSM filesets named dsm.core and dsm.dsh.

 

root@nim# lslpp -L dsm*

  Fileset                      Level  State  Type  Description (Uninstaller)

  ----------------------------------------------------------------------------

  dsm.core                   7.1.1.2    A     F    Distributed Systems Management

                                                   Core

  dsm.dsh                    7.1.0.0    C     F    Distributed Systems Management

                                                   Dsh

 

root@nim# lslpp -w /usr/bin/dsh

  File                                        Fileset               Type

  ----------------------------------------------------------------------------

  /usr/bin/dsh                                dsm.dsh               Symlink

 

 

These filesets are NOT installed by default. You must manually install them. They can be found on your AIX 7.1 media.

 

If dsh is something you use, then I recommend you read the section on DSM in the Redbook. Also take a look at section 5.2.7 Using DSM and NIM, in which it describes how you can integrate DSM and NIM and completely automate the installation of AIX:

 

“The AIX Network Installation Manager (NIM) has been enhanced to work with the Distributed System Management (DSM) commands. This integration enables the automatic installation of new AIX systems that are either currently powered on or off.”

 

Although I’ve written about the dsh command before, there’s one usage I’ve not covered.  And that is using dsh to manage users across a group of LPARs. In particular, changing a user’s password.

 

Before I go any further, I should state that for the following to work you must first configure ssh keys on your NIM master (or central mgmt AIX system) so that you can communicate with all of your AIX systems via SSH, as root,  without being prompted for a password. Read my article on dsh to find out how to do this if necessary.

 

In the following example, I use dsh from my NIM master. It is my central point of control for my AIX environment.

 

My ssh keys for root on my NIM master have been generated and distributed to all of my LPARs.

 

root@nim# ssh-keygen -d

Generating public/private dsa key pair.

Enter file in which to save the key (/.ssh/id_dsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /.ssh/id_dsa.

Your public key has been saved in /.ssh/id_dsa.pub.

The key fingerprint is:

ed:18:e9:00:37:13:7c:7c:74:6a:a9:e0:ad:c0:09:a9 root@nim

The key's randomart image is:

+--[ DSA 1024]----+

|     ... .. .    |

|   .  ..o .+     |

|  o . =. .+      |

| . o = = =       |

|E   + o S .      |

|     . + +       |

|      . o .      |

|                 |

|                 |

+-----------------+

 

root@nim# ls -ltra

total 40

-rw-------    1 root     system          214 17 Sep 2010  authorized_keys

drwxr-xr-x    7 root     system         4096 16 Nov 11:43 ..

-rw-r--r--    1 root     system         3615 16 Nov 12:04 known_hosts

-rw-r--r--    1 root     system          601 16 Nov 12:06 id_dsa.pub

-rw-------    1 root     system          672 16 Nov 12:06 id_dsa

drwx------    2 root     system          256 16 Nov 12:06 .

 

On my AIX LPARs, the authorized_keys file has been updated with the public ssh key from my NIM master:

 

root@aixlpar1 [/.ssh]# chmod 600 known_hosts authorized_keys

root@aixlpar1 [/.ssh]# ls -ltr

total 0

-rw-------    1 root     system          406 Sep 29 15:30 known_hosts

-rw-------    1 root     system          601 Nov 16 12:07 authorized_keys

 

root@aixlpar1 [/.ssh]# cat authorized_keys

ssh-dss AAAAB3NzaC1kc3MAAACBAPJpekVQgV2CTu5tCOsukwXwMrkfBabLAN7vwjgPk3S2p65COJkFwaDRWwBoK87cnMrrc7/GuOLdk/RiqYw1K8Z0d9RWnsymYrNm0MBALWioGGXwQ+YlLiYHIQsSArnajZOH6Ye36qz81uWNoj6u4xIxi/iGSYoU4x3UE/c/H833AAAAFQDKEtWVFk3G+KAyxUsI6FcpbJZqDQAAAIAVUVRak/s+X0mMElCaEs8Rh96GDfhorXWV1YzI0nbRo8QX2AMix+3LSdEmYCRn8TYpk1Udln3GG4gzy5oI/3Rd1RTWgAo+067LwytR6IJfwYt219Pwh2ktrm5iZu3c+yDBqQj6hsga2YIun5lxqYFy39CZNtslnR1wTx9uCcYkowAAAIEAuVFTsW9ugVM76CgluspNo0faz9xFo4DCOjCgye+k/ZqSDaNq6Xrvxktw+dkHuKHbhvofPgeiMrlqhdbrE0ovLUr8aadxArRhitnu10CFYbRwlrjAxyAaRwjD2pXG8M3a5D1XXAWDCDm6bEs5yxJqcl1wq+8UAhpKiS5yj2pwApk= root@nim

 

 

On the NIM master, the root user was configured for the DSH environment. The following entry was placed in roots .profile:

 

root@nim# cat /.profile

ENV=$HOME/.kshrc

 

The following entry was placed in roots .kshrc file:

 

root@nim# cat /.kshrc

export DSH_NODE_RSH=/usr/bin/ssh

export DSH_NODE_LIST=/usr/local/etc/nodes

 

A /usr/local/etc/nodes file was created on the NIM master. This file contains a list of each of the nodes that dsh can communicate with from NIM:

 

root@nim# cat /usr/local/etc/nodes

aixlpar1

aixlpar2

aixlpar3

aixlpar4

aixlpar5

aixlpar6

aixlpar7

aixlpar8

aixlpar9

aixlpar10

aixlpar11

 

The first time that the dsh command is run against a new host, the following message will be displayed. dsh uses the FQDN, and the FQDN needs to be added to the known_hosts file for ssh. Therefore you must make an ssh connection first with FQDN to the host:

 

root@nim# dsh uptime

aixlpar1.cg.com.au : Host key verification failed.

dsh:  2617-009 aixlpar1.cg.com.au  remote shell had exit code 255

 

It is necessary to ssh directly to each node using its FQDN. This step is only required once for each node. For example:

 

root@nim# ssh aixlpar1.cg.com.au

The authenticity of host 'aixlpar1.cg.com.au  (172.1.6.17)' can't be established.

RSA key fingerprint is 83:13:1d:cb:97:ee:00:7a:04:75:62:98:bd:e8:30:79.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'aixlpar1.cg.com.au ' (RSA) to the list of known hosts.

Last unsuccessful login: Tue Sep 27 03:04:33 GMT+10:00 2011 on ssh from 172.1.6.15

Last login: Wed Nov 16 12:08:08 GMT+11:00 2011 on /dev/pts/3 from nim

 

 

root@aixlpar1 [/root]# exit

Connection to aixlpar1.cg.com.au  closed.

 

Now we can use the dsh command to run a command across all the nodes defined in the DSH_NODE_LIST. For example, the following command will run the uptime command on all nodes:

 

root@nim# dsh uptime

aixlpar1.cg.com.au :   12:14PM   up 10 days,  22:32,  3 users,  load average: 0.47, 0.53, 0.64

aixlpar2.cg.com.au :   12:14PM   up 21 days,  21:13,  1 users,  load average: 0.11, 0.31, 0.51

aixlpar3.cg.com.au :   12:14PM   up 15 days,  20:33,  5 users,  load average: 0.17, 0.43, 0.54

...etc...

 

With dsh now configured, we can use it to perform several admin tasks from a central location. For example, the following commands will do the following:

 

1.       Create a user, named cg, on all nodes.

2.        Set cg’s password to abc123 using the chpasswd command.

3.        Remove the ADMCHG flag for the user cg. This will stop them from being prompted to change their password on their first login attempt.

 

First I check that user does not exist on the AIX systems.

 

root@nim# dsh lsuser cg

aixlpar1: 3004-687 User "cg" does not exist.

aixlpar2: 3004-687 User "cg" does not exist.

aixlpar3: 3004-687 User "cg" does not exist.

aixlpar4: 3004-687 User "cg" does not exist.

aixlpar5: 3004-687 User "cg" does not exist.

...etc...

 

I create the user with the mkuser command.

 

root@nim# dsh mkuser cg

 

I confirm that the user has been created on all the AIX systems, with the lsuser command.

 

root@nim# dsh lsuser cg

aixlpar1: cg id=204 pgrp=staff groups=staff home=/home/cg shell=/usr/bin/ksh login=true su=true rlogin=true daemon=true admin=false sugroups=ALL admgroups= tpath=nosak ttys=ALL expires=0 auth1=SYSTEM auth2=NONE umask=22 registry=files SYSTEM=compat logintimes= loginretries=0 pwdwarntime=0 account_locked=false minage=0 maxage=0 maxexpired=-1 minalpha=0 minother=0 mindiff=0 maxrepeats=8 minlen=0 histexpire=0 histsize=0 pwdchecks= dictionlist= fsize=-1 cpu=-1 data=262144 stack=65536 core=2097151 rss=65536 nofiles=2000 roles=

aixlpar2: cg id=204 pgrp=staff groups=staff home=/home/cg shell=/usr/bin/ksh login=true su=true rlogin=true daemon=true admin=false sugroups=ALL admgroups= tpath=nosak ttys=ALL expires=0 auth1=SYSTEM auth2=NONE umask=22 registry=files SYSTEM=compat logintimes= loginretries=0 pwdwarntime=0 account_locked=false minage=0 maxage=0 maxexpired=-1 minalpha=0 minother=0 mindiff=0 maxrepeats=8 minlen=0 histexpire=0 histsize=0 pwdchecks= dictionlist= fsize=-1 cpu=-1 data=262144 stack=65536 core=2097151 rss=65536 nofiles=2000 roles=

aixlpar3: cg id=204 pgrp=staff groups=staff home=/home/cg shell=/usr/bin/ksh login=true su=true rlogin=true daemon=true admin=false sugroups=ALL admgroups= tpath=nosak ttys=ALL expires=0 auth1=SYSTEM auth2=NONE umask=22 registry=files SYSTEM=compat logintimes= loginretries=0 pwdwarntime=0 account_locked=false minage=0 maxage=0 maxexpired=-1 minalpha=0 minother=0 mindiff=0 maxrepeats=8 minlen=0 histexpire=0 histsize=0 pwdchecks= dictionlist= fsize=-1 cpu=-1 data=262144 stack=65536 core=2097151 rss=65536 nofiles=2000 roles=

 

I set the users password to abc123, using the chpasswd utility. I also remove the ADMCHG flag so that the user is not prompted to change their password on their first logon attempt.

 

root@nim# dsh 'echo cg:abc123 | chpasswd -c'

 

I confirm that I can logon with the new user with the specified password, on one of the AIX LPARs.

 

root@nim# ssh cg@aixlpar1

cg@aixlpar1’s password:

Last login: Thu Mar  1 20:05:01 CST 2012 on /dev/pts/1 from aix71

$ id

uid=204(cg) gid=1(staff)

 

Another nice feature of dsh is the dshbak utility. This utility presents formatted output from the dsh command. For example:

 

root@nim 520 [/.ssh]# dsh errpt | dshbak

HOST: aixlpar1.cg.com.au

------------------------------------

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION

AA8AB241   1116110811 T O OPERATOR       OPERATOR NOTIFICATION

A6DF45AA   1104135011 I O RMCdaemon      The daemon is started.

2BFA76F6   1104134111 T S SYSPROC        SYSTEM SHUTDOWN BY USER

9DBCFDEE   1104134111 T O errdemon       ERROR LOGGING TURNED ON

 

HOST: aixlpar2.cg.com.au

-------------------------------

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION

DE9A52D1   1111012611 I S rmt10          AAA1

4865FA9B   1111012211 P H rmt10          TAPE OPERATION ERROR

DE9A52D1   1110233511 I S rmt0           AAA1

4865FA9B   1110225511 P H rmt0           TAPE OPERATION ERROR

DE9A52D1   1109180311 I S rmt0           AAA1

4865FA9B   1109180011 P H rmt0           TAPE OPERATION ERROR

DE9A52D1   1108180411 I S rmt2           AAA1

4865FA9B   1108180211 P H rmt2           TAPE OPERATION ERROR

DE9A52D1   1108165711 I S rmt6           AAA1

4865FA9B   1108165111 P H rmt6           TAPE OPERATION ERROR

A2205861   1102085311 P S SYSPROC        Excessive interrupt disablement time

F7FA22C9   1031134111 I O SYSJ2          UNABLE TO ALLOCATE SPACE IN FILE SYSTEM

DE9A52D1   1030163411 I S rmt0           AAA1

4865FA9B   1030163411 P H rmt0           TAPE OPERATION ERROR

....etc....

 

WARNING: Please be VERY CAREFUL when using the dsh command. Issuing the wrong command can cause damage to all your AIX LPARS!

 

The dsm.dsh package contains the following utilities:

 

# lslpp -f dsm.dsh | grep /usr/bin

                        /usr/bin/dcp -> /opt/ibm/sysmgt/dsm/bin/dcp

                        /usr/bin/dsh -> /opt/ibm/sysmgt/dsm/bin/dsh

                        /usr/bin/dping -> /opt/ibm/sysmgt/dsm/bin/dping

                        /usr/bin/dshbak -> /opt/ibm/sysmgt/dsm/bin/dshbak

 

If you are a fan of the dping command, you are going to be disappointed. Although the command is currently included in the dsm.dsh fileset, it probably won’t be for much longer.

 

The command works, “sort of”:

 

root@nim# dping aixlpar1

aixlpar1: ping (alive)

 

But if you run ‘dping –a’:

 

root@nim# dping -a

dping: 2651-095 CSM license has expired or has not been accepted. Run csmconfig -L if you have installed a new release.

 

According to the developers, dping is no longer supported and will eventually be removed from the DSM package. The response from the  developers was as follows:

"The reason "dping -a" is failing with the license check is because the command is calling “/usr/bin/runact-api –c IBM.DmsCtrl::::isLicenseValid" and the license is not set. So the command fails.  Since CSM is not supported anymore and went end of life. "

“... please consider the dping command as being "deprecated" code pending removal from the dsm.dsh package.”

When I asked why the command was listed in the AIX 7.1 online documentation if it was no longer available, I was informed: “We are  in the process of working with component owner regarding the DOCs and updating them.”. At this stage I’ve not been able to find an alternative command (in AIX). If I find one, I’ll update this post.

 

If you are planning on migrating to AIX 7.1 please be aware that CSM is no longer supported or available with AIX 7.1. CSM is now ‘end of life’.

 

 

http://www.ibmsystemsmag.com/aix/tipstechniques/systemsmanagement/DSH-Simplifies-the-Task-of-Managing-Multiple-Syste/

 

http://www.redbooks.ibm.com/abstracts/sg247910.html?Open

 

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.cmds/doc/aixcmds1/chpasswd.htm

 

After updating my NIM master to AIX 7.1 TL2 SP1 (7100-02-01-1245), I noticed a problem. Whenever I installed a new AIX partition using NIM, the resources allocated to the NIM client were not being de-allocated, even though the installation was completing successfully. Also, if I tried to run my usual ‘NIM client reset’ script (below), the resources were still allocated.

 

#!/usr/bin/ksh

# Reset a NIM client.

 

if [[ "$1" = "" ]] ; then

  echo Please specify a NIM client to reset e.g. aixlpar1.

else

  if lsnim -l $1 > /dev/null 2>&1 ; then

    nim -o reset -F $1

    nim -Fo de-allocate  -a subclass=all $1

    nim -Fo change -a cpuid= $1

  else

    echo Not a valid NIM client!?

  fi

fi

                                                                     

For example, here’s my NIM client with the lpp_source, mksysb and SPOT resources assigned to it (even though the AIX install completed OK).

                                                                       

root@nim1 : / # lsnim -l aixlpar1                                

aixlpar1:                                                           

   class           = machines                                          

   type            = standalone                                        

   connect         = shell                                             

   platform        = chrp                                              

   netboot_kernel  = 64                                                

   if1             = network1 aixlpar1 0                             

   cable_type1     = N/A                                               

   Cstate          = ready for a NIM operation                         

   prev_state      = not running                                       

   Mstate          = currently running                                 

   boot            = boot                                              

   lpp_source      = lpp_sourceaix710105                               

   mksysb          = aixlpar1-71                                 

   nim_script      = nim_script                                        

   spot            = spotaix710105                                     

   cpuid           = 00C453C75C00                                      

   control         = master                                            

   Cstate_result   = success                                           

   installed_image = aixlpar1-71                  

                                                                        

My workaround was to use 'smit nim_mac_res' to manually de-allocate  resources from the client:                               

                                                                       

====                                                                    

De-allocate  Network Install Resources                                   

aixlpar1     machines       standalone                              

                                                                       

>lpp_sourceaix710105     lpp_source                                    

>spotaix710105           spot                                          

>aixlpar1-71       mksysb                                        

====                                                                    

                                                                                                                                             

It appears that others were also experiencing this problem. I found the following thread on the IBM developerWorks AIX user forum:     

                                             

NIM de-allocate  with subclass=all doesn't work:

https://www.ibm.com/developerworks/forums/thread.jspa?threadID=464710

 

Someone even provided their own version of a ‘NIM client reset’ script that offered another workaround to the problem:

 

nim -Fo reset $1

for i in bosinst_data fb_script image_data installp_bundle lpp_source mksysb nim_script script spot ; do

  ressources_to_clean=$(lsnim -l $1 | grep $i | awk '{print $NF}')

  for j in ${ressources_to_clean} ; do

    nim -Fo de-allocate  -a $i=$j $1

  done

done

                                                                   

I reported the issue to IBM AIX support and discovered that this is a known issue with 7.1 TL2.

 

“This is a known issue which recently has been documented in 2 open     

APARs:  IV32570 and IV32670:  UPDATE NIM BINARIES FOR UPDATES.         

                                                                       

Error description                                                   

One known symptom (of this issue) is that "nim -Fo               

de-allocate  -a subclass=all <client>" returns successfully but the

resource will still be allocated.                                 

                                                                       

The problem occured because the "update" pack that brings you from     

7.1.1.x to 7.1.2.0 missed a piece that effected the bos.sysmgt.7.1.2.0 

fileset.                                                               

                                                                       

Currently there is no fix available and no projected timeframe when this

issue will be fixed (i assume it will be with the next SP for AIX      

7100-02)                                                               

                                                                       

However, a workaround is allready documented in our incidents          

database - you simply need to force reinstall the bos.sysmgt.nim.master

7.1.2.0 fileset using your AIX 7.1 TL02 media.                         

                                                                        

This is not a backleveling of the fileset, you are simply force        

overwriting the full base level at the same version that you are       

currently at.  It will still be at 7100-02.                            

                                                                        

The workaround is a force overwrite of the bos.sysmgt.nim.master 7.1.2.0

from media." 

 

The AIX support team suggested that I download the necessary install fileset from the following location:

 

ftp://testcase.software.ibm.com/fromibm/aix/bos.sysmgt.7.1.2.0.I  

 

I backed up my NIM database before re-installing the fileset, just in case.

 

---------------------------------------------------------

Starting NIM Database Backup on nim1 at Wed Dec 12 12:39:43 EETDT 2012

 

a ./etc/objrepos/nim_attr 32 blocks.

a ./etc/objrepos/nim_attr.vc 112 blocks.

a ./etc/objrepos/nim_object 8 blocks.

a ./etc/objrepos/nim_object.vc 24 blocks.

a ./etc/NIM.level 1 blocks.

a ./etc/niminfo 1 blocks.

a ./etc/NIM.primary.cpuid 1 blocks.

 

Finished NIM Database Backup on nim1 at Wed Dec 12 12:39:44 EETDT 2012

---------------------------------------------------------

 

I placed the fileset into a separate directory on the NIM master and ran inutoc.

 

root@nim1 : /home/cgibson/NIMPMR # ls -ltr

total 36170

-rw-r--r--    1 cgibson  staff      18512896 Dec 12 12:21 bos.sysmgt.7.1.2.0.I

-rw-r--r--    1 root     system         4778 Dec 12 12:35 .toc

 

Then I re-installed the fileset (force OVERWRITE) on the NIM master:

 

                             Install Software

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                                                        [Entry Fields]

* INPUT device / directory for software               .

* SOFTWARE to install                                [@ 7.1.2.0  Network Install Manager - Master Tools                      @@I:bos.sysmgt.nim.master 7>

  PREVIEW only? (install operation will NOT occur)    no

  COMMIT software updates?                            yes

  SAVE replaced files?                                no

  AUTOMATICALLY install requisite software?           no 

  EXTEND file systems if space needed?                yes

  OVERWRITE same or newer versions?                   yes

  VERIFY install and check file sizes?                no 

  Include corresponding LANGUAGE filesets?            yes 

  DETAILED output?                                    yes 

  Process multiple volumes?                           yes 

  ACCEPT new license agreements?                      yes  

  Preview new LICENSE agreements?                     no      

 

 

The fileset re-installed without a problem.

 

Installation Summary

--------------------

Name                        Level           Part        Event       Result

-------------------------------------------------------------------------------

bos.sysmgt.nim.master       7.1.2.0         USR         APPLY       SUCCESS

 

root@nim1 : /home/cgibson/NIMPMR # lslpp -l bos.sysmgt.nim.master

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  bos.sysmgt.nim.master      7.1.2.0  COMMITTED  Network Install Manager -

                                                 Master Tools

 

I re-tried installing AIX on a NIM client.

 

root@nim1 : / # lsnim -l aixlpar1

aixlpar1:

   class           = machines

   type            = standalone

   connect         = shell

   platform        = chrp

   netboot_kernel  = 64

   if1             = network1 aixlpar1 0

   cable_type1     = N/A

   Cstate          = Base Operating System installation is being performed

   prev_state      = BOS installation has been enabled

   Mstate          = in the process of booting

   info            = verifying_data_files

   boot            = boot

   lpp_source      = lpp_sourceaix710105

   mksysb          = aixlpar1-71

   nim_script      = nim_script

   spot            = spotaix710105

   cpuid           = 00C453C75C00

   control         = master

   Cstate_result   = success

   installed_image = aixlpar1-71

 

Once the install completed I found that all allocated NIM resources (lpp_source, mksysb and SPOT) were de-allocated. Problem solved.

 

root@nim1 : / # lsnim -l aixlpar1

aixlpar1:

   class           = machines

   type            = standalone

   connect         = shell

   platform        = chrp

   netboot_kernel  = 64

   if1             = network1 aixlpar1 0

   cable_type1     = N/A

   Cstate          = ready for a NIM operation

   prev_state      = in the process of booting

   Mstate          = not running

   cpuid           = 00C453C75C00

   Cstate_result   = success

   installed_image = aixlpar1-71

 

So if you are planning on updating your NIM master to 7.1 TL2, please take note and make sure you implement the workaround.

 

 

 

 

 

Last year I was working for a customer that was upgrading several AIX 5.3 systems to AIX 6.1. The migrations were successful for the most part, but we did encounter one issue that took a little time to resolve.

 

The customer was using nimadm to migrate. This process worked fine, however, on a couple of systems a strange error was encountered after the migration. The LPAR was booted into AIX 6.1 and everything came up fine. The applications were started and users started accessing the system.

 

It was several days later, when the AIX administrator attempted to configure new storage on the AIX system, when the first sign of trouble appeared.  He had asked his Storage guy to assign a couple of new disks to his LPAR (via NPIV/VFC). As soon as the Storage admin had completed the assignment, the AIX admin ran cfgmgr to detect and configure the new hdisks. Immediately, cfgmgr reported the following error:

 

Method error (/usr/lib/methods/cfgscsidisk):

        0514-023 The specified device does not exist in the

                 customized device configuration database.

 

Initially, the AIX team suspected there was some fault with either the storage device or the zoning of the disk. Both of these items were checked and doubled-checked and were found to be OK. Our next step was to run cfgmgr again, but this time we wanted a greater level of detail captured. To do this we used the following environment variable to force cfgmgr to be ‘more verbose’.

 

# export CFGLOG="cmd,meth,lib,verbosity:9"

 

We ran cfgmgr and went to the /var/adm/ras/cfglog file to view the results with the alog command. However, we noticed that the cfglog file had a size of zero (0) and contained no data.

 

# cd /var/adm/ras

# ls –l cfglog

-rw-r-----    1 root     system      0 May 16 13:22 cfglog

 

We decided to recreate the cfglog alog file and run mkdev again to reproduce the disk configuration error.

 

# rm cfglog

# echo "Create cfglog `date`"|alog -t cfg

# mkdev -l hdisk0

Method error (/usr/lib/methods/cfgscsidisk):

        0514-023 The specified device does not exist in the

                 customized device configuration database.

 

This time we found some useful data in the cfglog file.

 

# alog -t cfg -o

MS 31981804 28835876 /usr/lib/methods/cfgscsidisk -l hdisk39

M4 31981804 Parallel mode = 0

M4 31981804 Get CuDv for hdisk39

M4 31981804 Get device PdDv, uniquetype=disk/fcp/htcvspmpio

M4 31981804 Get parent CuDv, name=fscsi0

M4 31981804 ..is_mpio_capable()

M4 31981804 Device is MPIO

M4 31981804 ..get_paths()

M4 31981804 Getting CuPaths for name='hdisk39'

M4 31981804 Found 1 paths

M0 31981804 cfgcommon.c 225 mpio_init error, rc=23

MS 28835892 31981568 /usr/lib/methods/cfgscsidisk -l hdisk0

M4 28835892 Parallel mode = 0

M4 28835892 Get CuDv for hdisk0

M4 28835892 Get device PdDv, uniquetype=disk/fcp/htcvspmpio

M4 28835892 Get parent CuDv, name=fscsi0

M4 28835892 ..is_mpio_capable()

M4 28835892 Device is MPIO

M4 28835892 ..get_paths()

M4 28835892 Getting CuPaths for name='hdisk0'

M4 28835892 Found 2 paths

M0 28835892 cfgcommon.c 225 mpio_init error, rc=23

MS 25690326 27328608 /usr/lib/methods/cfgscsidisk -l hdisk0

M4 25690326 Parallel mode = 0

M4 25690326 Get CuDv for hdisk0

M4 25690326 Get device PdDv, uniquetype=disk/fcp/htcvspmpio

M4 25690326 Get parent CuDv, name=fscsi0

M4 25690326 ..is_mpio_capable()

M4 25690326 Device is MPIO

M4 25690326 ..get_paths()

M4 25690326 Getting CuPaths for name='hdisk0'

M4 25690326 Found 2 paths

M0 25690326 cfgcommon.c 225 mpio_init error, rc=23

 

The configuration method was attempting to configure a disk device type of htcvspmpio (which was correct) but it was unable to configure the device paths (mpio_init error rc=23). We suspected that the system was missing some sort device driver support for the type of storage in use.

 

Cutting a very long story short, we determined, with the help of the IBM AIX support team, that the issue stemmed from “old” AIX installation media used to create the AIX 6.1 TL6 SP5 SPOT and lppsource on the NIM master. Old AIX 6.1 media was originally used (several years ago) to create the NIM resources and was gradually updated over time, all the way up to TL6 SP5.

 

IBM support identified that the older install media contained a liblpp.a file that was missing the necessary PdPathAt ODM files. Newer install media contained a fix to add the appropriate entries to the bos.rte.cfgfiles. e.g.

                                                                      

SPOT created using OLD install media.

======================================

# ar xv /export/spot/spotaix610605_OLD/usr/lpp/bos/liblpp.a bos.rte.cfgfiles   

x - bos.rte.cfgfiles                                                   

                                                                        

# grep PdPathAt bos.rte.cfgfiles

#                                       

 

SPOT created using NEW install media.

======================================

# ar xv /export/spot/spotaix610605_NEW/usr/lpp/bos/liblpp.a bos.rte.cfgfiles   

x - bos.rte.cfgfiles                                                   

                                                                        

# grep PdPathAt bos.rte.cfgfiles                                       

/usr/lib/objrepos/PdPathAt              v4preserve                     

/usr/lib/objrepos/PdPathAt.vc           v4preserve 

 

We recreated the SPOT and lppsource resources using newer media on the NIM master. We were then able to migrate the AIX 5.3 LPAR to 6.1 without encountering the issues faced previously.

 

Before I describe the sudo problem I hit, I will tell you that the root cause of the problem is evident in the following output:

 

# mount

  node       mounted        mounted over    vfs       date        options

-------- ---------------  ---------------  ------ ------------ ---------------

         /dev/hd4         /                jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/hd2         /usr             jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/hd9var      /var             jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/hd3         /tmp             jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/hd11admin   /admin           jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/mksysblv    /export/mksysb   jfs2   Aug 14 12:00 rw,log=/dev/loglv00

         /dev/fslv00      /export/nim      jfs2   Aug 14 12:00 rw,log=/dev/loglv00

         /dev/hd1         /home            jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/hd10opt     /opt             jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/swlv        /software        jfs2   Aug 14 12:00 rw,log=INLINE

         /dev/fslv01      /tftpboot        jfs2   Aug 14 12:00 rw,log=/dev/loglv00

         /dev/livedump    /var/adm/ras/livedump jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/fixelv      /fixes           jfs2   Aug 14 12:00 rw,log=INLINE

         /dev/lpmautolv   /lpmauto         jfs2   Aug 14 12:00 rw,log=INLINE

         /dev/promonlv    /promon          jfs2   Aug 14 12:00 rw,log=INLINE

         /aha             /aha             ahafs  Aug 14 12:00 rw

 

The output above is from the mount command on an AIX host.

 

Can you see any problems that might impact sudo's ability to run?

 

No?

 

Problem Description:

 

A customer was attempting to use sudo to run some commands on an AIX host. Each time, sudo would fail with the following error message:

 

$ sudo -l

sudo: no tty present and no askpass program specified

 

The only information we had was that the host had been rebooted at some point in the last day or so.

 

We spent some time checking and verifying a bunch of things (specifically making sure that the /etc/sudoers file was intact and did not contain any bad entries) but none of them provided any real clues as to the root cause of the problem.

 

Problem Debugging and Investigation

 

We enabled debug for sudo. This involved creating an /etc/sudo.conf file with the following entries:

 

# vi sudo.conf

Debug sudo /var/log/sudo_debug.log all@debug

Debug sudoers.so /var/log/sudo_debug.log all@debug

 

Then, as the non-root user, we attempted to run sudo again. This generated some useful data in the /var/log/sudo_debug.log file. You'll notice we grep'ed for the string tty specifically, given that this was part of the sudo error message provided to us.

 

 

$ sudo -l

sudo: no tty present and no askpass program specified

 

# grep tty /var/log/sudo_debug.log

Aug 10 16:20:26 sudo[14352830] -> get_process_ttyname @ ./ttyname.c:417

Aug 10 16:20:26 sudo[14352830] unable to resolve tty via /proc/14352830/psinfo: A file or directory in the path name does not exist. @ get_process_ttyname() ./ttyname.c:442

Aug 10 16:20:26 sudo[14352830] <- get_process_ttyname @ ./ttyname.c:444 := (null)

Aug 10 16:20:26 sudo[14352830] -> sudo_get_ttysize_v1 @ ./ttysize.c:59

Aug 10 16:20:26 sudo[14352830] -> get_ttysize_ioctl @ ./ttysize.c:38

Aug 10 16:20:26 sudo[14352830] <- get_ttysize_ioctl @ ./ttysize.c:44 := 0

Aug 10 16:20:26 sudo[14352830] <- sudo_get_ttysize_v1 @ ./ttysize.c:75

Aug 10 16:20:26 sudo[14352830] searching for tty time stamp record @ timestamp_lock() ./timestamp.c:603

Aug 10 16:20:26 sudo[14352830] appending new tty time stamp record @ timestamp_lock() ./timestamp.c:611

Aug 10 16:20:26 sudo[14352830] tty time stamp position is 112 @ timestamp_lock() ./timestamp.c:617

Aug 10 16:20:26 sudo[14352830] -> tty_present @ ./tgetpass.c:360

Aug 10 16:20:26 sudo[14352830] <- tty_present @ ./tgetpass.c:361 := false

Aug 10 16:20:26 sudo[14352830] no tty present and no askpass program specified @ tgetpass() ./tgetpass.c:107

 

The following information was of great interest to us! It appeared to be that some information was "missing" from the /proc file system on the host?

 

# grep tty /var/log/sudo_debug.log | grep unable

Aug 10 16:20:26 sudo[14352830] unable to resolve tty via /proc/14352830/psinfo: A file or directory in the path name does not exist. @ get_process_ttyname() ./ttyname.c:442

Aug 18 11:19:59 sudo[21889328] unable to resolve tty via /proc/21889328/psinfo: A file or directory in the path name does not exist. @ get_process_ttyname() ./ttyname.c:442

 

Upon further investigation, we found that /proc did, in fact, appear to be completely empty!! This was not expected or normal on a modern AIX system.

 

# ls -ltr /proc

total 0

 

And, after running the mount command, we discovered that, yes, the /proc file system was NOT mounted!

 

# mount | grep proc

#

 

Resolution

 

Next, we attempted to mount /proc. But we couldn't, as it was not a known file system? Curious indeed.

 

# mount /proc

mount: 0506-334 /proc is not a known file system.

 

The /proc file system stanza was missing from /etc/filesystems. So, we added it back in and mounted the file system successfully.

 

 

# grep -p proc /etc/filesystems

#                                  << No /proc entry found in /etc/filesystems!

 

# grep -p proc /etc/filesystems

/proc:

        dev       = /proc

        vol       = "/proc"

        mount     = true

        check     = false

        free      = false

        vfs       = procfs

 

# mount /proc

# mount | grep proc

         /proc            /proc            procfs Aug 18 11:31 rw

             

# ls -ltr /proc | head -10

total 0

dr-xr-xr-x    8 root     system            0 Aug 10 16:21 sys

-r--r--r--    1 root     system           91 Aug 10 16:21 version

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 0

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 1

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 260

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 65798

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 131336

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 196874

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 262412

 

As soon as /proc was mounted again, the non-root user could run sudo without any issues.

 

$ sudo -l

Password:

User cgibson may run the following commands on 750lpar4:

    (ALL) /usr/sbin/slibclean

$

 

The /proc file system is different from other file systems. It does not reside on disk. The /proc file system resides in AIX memory and is referred to as a virtual file system. The files maintained in /proc represent the running processes on the system.

http://www-01.ibm.com/support/docview.wss?uid=isg3T1012015

https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.files/proc.htm