Chris's AIX Blog

If you are a system administrator that is responsible for managing AIX systems that run SAP, then you’ve probably had an experience similar to the following?

 

OK, so one day my SAP Basis administrator contacts me and says “I can’t start saposcol......can you please reboot the system?” I quickly reply, “Was there an error message when trying to restart saposcol?”. He replies, “No”. Again I return very quickly, “OK, have you checked to see if there are any shared memory segments left for saposcol?”.  Just as quick, he replies “How do I do that?”

 

Together we try starting saposcol and what we find is that it thinks it’s already running (as shown below, PID 327924). But there is no such process!

 

$ saposcol -l

-l

09:37:24 22.01.2009   LOG: Effective User Id is root

***********************************************************************

* This is Saposcol Version COLL 20.95 700 - AIX v11.15 5L-64 bit 080317

* Usage:  saposcol -l: Start OS Collector

*         saposcol -k: Stop  OS Collector

*         saposcol -d: OS Collector Dialog Mode

*         saposcol -s: OS Collector Status

* The OS Collector (PID 327924) is already running .....

************************************************************************

$ ps -fp 327924

     UID     PID    PPID   C    STIME    TTY  TIME CMD

 

So we try to stop saposcol. This of course fails, as there is no PID 327924 running!

 

$ saposcol -k

 

Setting Stop Flag :

09:19:29 22.01.2009   LOG: ==== Stop Flag was set by saposcol (kill_collector()).

 

09:19:29 22.01.2009   LOG: ====  The collection process will stop as soon as possible

 

********************

can't kill process 327924.

kill: No such process

ERROR:No reaction from collecting process 327924.

Please kill collecting process.

 

My conclusion is that there must be a shared memory segment still allocated for saposcol. There were many other SAP processes still running happily, so there were several shared memory segments to sift through. So, what shared memory ID does saposcol use?

 

Now, according to the following website, shared memory key 4dbe is used by saposcol on AIX.

 

http://www.saptechies.com/os-collector-saposcol/

7. Question: How do I remove the shared memory key of saposcol?

Answer: Sometimes it may be necessary to remove the shared memory key of saposcol (see point 6). Caution: Please be very careful! This procedure should be performed only after checking that saposcol is really not running (see point 4) and only in cases when other options (see point 6) really do not work! For this, execute command “ipcs -ma” and note the line that contains saposcol key “4dbe”. You need the shared memory ID. After that, execute command ipcrm -m ID. Now the command “saposcol -s” should show that saposcol is not running and that the shared memory is not attached. The shared memory key will be created automatically by the saposcol when the collector is next started: “saposcol -l”.

 

So I run ipcs to check for the existence of 4dbe. And I find an entry for this key.

 

There are several process id’s ‘attached’ to this segment. However, only one of them actually exists (PID 2293794).

 

# ipcs -ma | grep 4dbe

m   2097156 0x00004dbe --rw-rw-rw-     root   sapsys     root   sapsys      0   1870188 467164 2293794  9:39:43  9:39:43  8:12:43

 

# ps -fp 1870188

     UID     PID    PPID   C    STIME    TTY  TIME CMD

 

# ps -fp 467164

     UID     PID    PPID   C    STIME    TTY  TIME CMD

 

# ps -fp 2293794

     UID     PID    PPID   C    STIME    TTY  TIME CMD

  sapadm 2293794       1   0 19:26:41      -  1:07 sapccm4x -DCCMS pf=/sapmnt/SAP/profile/SAP_DVEBMGS00_aix01

 

I ask the SAP admin to stop this process, which he does. Now I remove the shared memory segment and there is no evidence of 4dbe in the ipcs output.

 

# ipcrm -m 2097156

# ipcs -ma | grep 4dbe

 

We were then able to start saposcol again with success. The process is running and the shared memory segment 4dbe has returned.

 

# ipcs -ma | grep 4dbe

m   3145732 0x00004dbe --rw-rw-rw-     root   sapsys     root   sapsys      1   1870188 2293818 2207892 10:14:25 10:14:25 10:12:47

 

# ps -ef | grep oscol

  sapadm 2064616       1   0 10:12:53      -  0:00 saposcol -l

   

# /usr/sap/SAP/SYS/exe/run/saposcol -s

**************************************************************

Collector Versions :

  running : COLL 20.95 700 - AIX v11.15 5L-64 bit 080317

  dialog  : COLL 20.95 700 - AIX v11.15 5L-64 bit 080317

Shared Memory       : attached

Number of records   : 17640

Active Flag         : active (01)

Operating System    : AIX aix01 3 5 00C01C704C00

Collector PID       : 2064616 (001F80E8)

Collector           : running

Start time coll.    : Thu Jan 22 10:12:53 2009

 

Current Time        : Thu Jan 22 10:17:46 2009

 

Last write access   : Thu Jan 22 10:17:38 2009

 

Last Read  Access   : Thu Jan 22 10:16:16 2009

 

Collection Interval : 10 sec (next delay).

Collection Interval : 10 sec (last ).

Status              : free

Collect Details     : required

Refresh             : required

 

Header Extention Structure

Number of x-header      Records : 1

Number of Communication Records : 60

Number of free Com.     Records : 60

Resulting offset to 1.data rec. : 61

 

Trace level             : 2

 

Collector in IDLE - mode ? : NO

  become idle after 300 sec without read access.

  Length of Idle Interval  : 60 sec

  Length of norm.Interval  : 10 sec

**************************************************************

 

Problem solved and no reboot required.....this is AIX after all!  :)

 

The nohup command in AIX has the ability to “nohup” an existing process and protect if from hangup signals. Using the -p flag, an existing, running process can be “nohup’ed” and told to ignore SIGHUPs.

 

"Why is this useful?" I hear you ask.

 

Well, I'll often start a "long running" process on a system at work, like a backup for instance. I'll sometimes forget to use the nohup command, so that my long running job will ignore any SIGHUPs and continue running, say for example if my ssh session to the server dies. The nohup command also generates a nohup.out file to capture any messages (errors or otherwise) generated by the process e.g.

 

$ nohup ./processname &

$ ls –l nohup.out

-rw-------    1 cgibson  staff  0 May 26 19:42 nohup.out

 

In older releases of AIX, if I forgot to use nohup, I had two choices. I could either stop the process and start it again with nohup  (often not desirable or possible) or I could leave my ssh session logged in on my work PC and hope that it didn't power down or crash or that my ssh connection did not drop over night.

 

Fortunately with more recent releases of AIX, this is not a problem anymore. If I forget to “nohup” a process, I can use the -p flag to inform the already running process that it should ignore all hangup (SIGHUP) signals. This will allow the process to continue running if my ssh session (terminal) dies and will also capture any output from the process into the nohup.out file, so I don’t lose any important output from the process.

 

Here’s an example. I’m at work, it's 5pm and I've started a "long running" process named mycmd. This was started via an ssh (Putty) session from my PC.

 

$ ./mycmd

 

; Partial output from topas showing mycmd running.

Name      PID      CPU%  

mycmd    823432    98.8       

 

 

; proctree output displaying PIDs associcated with mycmd and the associated ssh session (tty).

 

$ proctree 823432

520394    -ksh

   585738    -ksh

      823432    /home/cgibson/mycmd

 

I've walked out the door and I'm now on my way home for the evening.

 

Oh no! I've forgotten to start the process with nohup! I don't want to restart my process but I want to protect it from SIGHUPs, while it executes after hours.

 

No problem, I'll use nohup to modify the running process, so I can rest assured it will continue to run, even if my ssh connection drops over night. From home I start a new ssh session on the server and run  nohup against the running process:

 

$ nohup -p 823432

 

At work the next morning, I discover that my original Putty session has indeed dropped! But, my process is still happily running! :)

 

; Partial output from topas showing mycmd still running.

 

Name      PID      CPU%  

mycmd    823432    99.3  

$ proctree 823432

520394    -ksh

   585738    -ksh

      823432    /home/cgibson/mycmd

 

You might find this useful if you ever forget to “nohup” a process and don’t want to restart it again.

 

http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds4/nohup.htm

 

 

 

Just the other day, I needed to use the AIX splitvg command in order to copy some data from one system to another.

I thought I’d share the experience here.

The splitvg command can split a single mirror copy of a fully mirrored volume group into a separate “snapshot” volume group.

From the man page:

The original volume group VGname will stop using the disks that are now part of the snapshot volume group SnapVGname. Both volume groups will keep track of the writes within the volume group so that when the snapshot volume group is rejoined with the original volume group consistent data is maintained across the rejoined mirrors copies. Notes:

1         To split a volume group, all logical volumes in the volume group must have the target mirror copy and the mirror must exist on a disk or set of disks. Only the target mirror copy must exist on the target disk or disks.

 

2         The splitvg command will fail if any of the disks to be split are not active within the original volume group.

 

3         In the unlikely event of a system crash or loss of quorum while running this command, the joinvg command must be run to rejoin the disks back to the original volume group.

 

4         There is no concurrent or enhanced concurrent mode support for creating snapshot volume groups.

 

5         New logical volumes and file system mount points will be created in the snapshot volume group.

 

6         The splitvg command is not supported for the rootvg.

 

7         The splitvg command is not supported for a volume group that has an active paging space.

 

8         When the splitvg command targets a concurrent-capable volume group which is varied on in non-concurrent mode, the new volume group that is created will not be varied on when the splitvg command completes. The new volume group must be varied on manually.

 

So, looking at point 4, above, if you are using enhanced concurrent volume groups (for example with PowerHA), then you will not be able to use the splitvg command. This is disappointing as this would have been very handy in some of the large PowerHA systems I have worked with…..perhaps this will be supported in the future?

Anyway, back to my example. I had wanted to break-off one of the mirrors of a mirrored volume group and then assign the “split” volume group to another host to copy some data off.

The volume group datavg contained two disks, hdisk0 and hdisk3, as shown in the lspv output below.

 

# lspv
hdisk1          00c01c705bdc6136                    old_rootvg
hdisk0          00c01c70c050810f                    datavg              active
hdisk2          00c01c7018a47201                    rootvg              active
hdisk3          00c01c70fed9e41a                    datavg              active

 

There were only two logical volumes (loglvl00, jfs2 log and fslv00, data) and a single file system (/data) in this volume group.

 

# lsvg -l datavg
datavg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
loglv00             jfs2log    1       2       2    open/syncd    N/A
fslv00              jfs2       16      32      2    open/syncd    /data

 

The volume group datavg, was mirrored across hdisk0 and hdisk3, as shown in the lsvg output below.

 

# lsvg -p datavg
datavg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            583         566         117..100..116..116..117
hdisk3            active            583         566         117..100..116..116..117

 

The logical volumes for the /data file system and the JFS2 log were both mirrored, as shown in the lslv/lspv output.

 

# lslv fslv00
LOGICAL VOLUME:     fslv00                 VOLUME GROUP:   datavg
LV IDENTIFIER:      00c01c7000004c0000000122e9038480.2 PERMISSION:     read/write
VG STATE:           active/complete        LV STATE:       opened/syncd
TYPE:               jfs2                   WRITE VERIFY:   off
MAX LPs:            512                    PP SIZE:        256 megabyte(s)
COPIES:             2                      SCHED POLICY:   parallel
LPs:                16                     PPs:            32
STALE PPs:          0                      BB POLICY:      relocatable
INTER-POLICY:       minimum                RELOCATABLE:    yes
INTRA-POLICY:       middle                 UPPER BOUND:    32
MOUNT POINT:        /data                    LABEL:          /data
MIRROR WRITE CONSISTENCY: on/ACTIVE
EACH LP COPY ON A SEPARATE PV ?: yes
Serialize IO ?:     NO

 

# lslv loglv00
LOGICAL VOLUME:     loglv00                VOLUME GROUP:   datavg
LV IDENTIFIER:      00c01c7000004c0000000122e9038480.1 PERMISSION:     read/write
VG STATE:           active/complete        LV STATE:       opened/syncd
TYPE:               jfs2log                WRITE VERIFY:   off
MAX LPs:            512                    PP SIZE:        256 megabyte(s)
COPIES:             2                      SCHED POLICY:   parallel
LPs:                1                      PPs:            2
STALE PPs:          0                      BB POLICY:      relocatable
INTER-POLICY:       minimum                RELOCATABLE:    yes
INTRA-POLICY:       middle                 UPPER BOUND:    32
MOUNT POINT:        N/A                    LABEL:          None
MIRROR WRITE CONSISTENCY: on/ACTIVE
EACH LP COPY ON A SEPARATE PV ?: yes
Serialize IO ?:     NO

 

# lspv -l hdisk0
hdisk0:
LV NAME               LPs     PPs     DISTRIBUTION          MOUNT POINT
fslv00                16      16      00..16..00..00..00    /data
loglv00               1       1       00..01..00..00..00    N/A

 

# lspv -l hdisk3
hdisk3:
LV NAME               LPs     PPs     DISTRIBUTION          MOUNT POINT
fslv00                16      16      00..16..00..00..00    /data
loglv00               1       1       00..01..00..00..00    N/A

 

Using the splitvg command I was able to break-off one of the disks from the mirrored pair. This created a new volume group on hdisk3 called vg00.

 

# splitvg -c1 datavg
# lspv
hdisk1          00c01c705bdc6136                    old_rootvg
hdisk0          00c01c70c050810f                    datavg            active
hdisk2          00c01c7018a47201                    rootvg          active
hdisk3          00c01c70fed9e41a                    vg00            active

 

The new volume group contains a new logical volume (pre-fixed with fs i.e. fsfslv00) and a file system (pre-fixed with /fs i.e. /fs/data). I can mount this file system and access the data in the file system and create and/or modify files (as shown below).

 

# mount /fs/data
Replaying log for /dev/fsfslv00.

# cd /fs/data
# ls -ltr
total 112
drwxrwxrwx    2 root     system        53248 Jan 30 2009  AIX61TL2SP2
drwxr-xr-x    2 root     system          256 Aug 05 15:24 lost+found
-rw-r--r--    1 root     system            0 Sep 15 10:29 2

# touch 3
# ls -ltr
total 112
drwxrwxrwx    2 root     system        53248 Jan 30 2009  AIX61TL2SP2
drwxr-xr-x    2 root     system          256 Aug 05 15:24 lost+found
-rw-r--r--    1 root     system            0 Sep 15 10:29 2
-rw-r--r--    1 root     system            0 Sep 15 10:46 3

 

At this point I was able to export the volume group and import it on another system. I had to re-map the Virtual SCSI disk on my VIOS first.  

 

# cd
# umount /fs/data

# varyoffvg vg00
# exportvg vg00

# rmdev –dl hdisk3

; Re-map the disk at the VIOS layer to the other AIX LPAR.

; Configure the disk on the other LPAR (with cfgmgr) and import the volume group (with importvg).

 

Once I was finished with vg00 on the other LPAR, I re-mapped the disk to the original AIX LPAR, configured the disk and then re-joined the disk to the datavg volume group, with the joinvg command.

 

; Export the VG and remove the disk on the other AIX LPAR.

; Re-map the disk at the VIOS layer to the original AIX LPAR.

# cfgmgr

# lsdev –Cc | grep hdisk3

hdisk3 Available  Virtual SCSI Disk Drive

 

# joinvg datavg
# lspv
hdisk1          00c01c705bdc6136                    old_rootvg
hdisk0          00c01c70c050810f                    datavg            active
hdisk2          00c01c7018a47201                    rootvg          active
hdisk3          00c01c70fed9e41a                    datavg            active

 

The volume group is now fully mirrored again. However the partitions are stale and need to be sync’ed. I manually re-sync the volume group with the syncvg command.

 

# lsvg -l datavg
datavg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
loglv00             jfs2log    1       2       2    open/stale    N/A
fslv00              jfs2       16      32      2    open/stale    /data

# syncvg –v datavg

# lsvg -l datavg
datavg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
loglv00             jfs2log    1       2       2    open/syncd    N/A
fslv00              jfs2       16      32      2    open/syncd    /data

 

More information can be found here: http://www-01.ibm.com/support/docview.wss?uid=isg3T1010934

 

 

 

Upon reading through the release notes for Fix Pack 22 for VIOS 2.1, I stumbled across a new command called chkdev. According to the notes, this command was “added to help you check devices for virtual provisioning capability”. This fired my curiosity and as soon as I had applied FP22 to one of my VIOS, I started using this command to see what it could do for me! 

 

First, I checked the man page for chkdev. It stated that chkdev can determine whether or not a device can be used in physical to virtual (p2v) migration on a VIOS.

 

chkdev command

 

Purpose

 

       Checks devices for virtual device provisioning capability in the Virtual I/O Server.

 

Syntax

 

       chkdev [-dev Name] [-verbose] [-field FieldName ] [-fmt delimiter]

 

Description

 

The chkdev command determines whether a device can be used in physical to virtual transitions in the Virtual I/O Server. Devices include all physical volumes and multipath representations of logical units. If the -dev option is used, information is shown for the device specified. If the -verbose option is used, detailed information is shown. With the -field option, the user can specify which fields to display. With the -fmt option, the user can format the output to be separated by a given delimiter.

 

Capability in a virtual environment refers to the capability for a device to move between a physical environment to a virtual environment. The command displays these fields: Phys2Virt_Capable, Virt2NPIV_Capable, and Virt2Phys_Capable.

 

For a device to be capable of moving from a physical to a virtual environment, it must pass the address compatibility check. If the physical volume can be moved from a physical to a virtual environment, the Phys2Virt_Capable field has a value of YES. If the device is not capable of this move, the field has a value of NO. If the device is already in use by Virtual Small Computer System Interface (VSCSI), the field has a value of NA.

 

If a device is capable of moving from a VSCSI LUN environment to an N_Port ID Virtualization (NPIV) environment, it must pass the address-compatibility check. If this device can be moved, the Virt2NPIV_Capable field has a value of YES. If the device is not capable of this move, the field has a value of NO. If the device is not in use by the virtual target device (VTD), this field has a value of NA. This command is not sufficient to guarantee that the client can read the data after the VSCSI logical unit number (LUN) is removed. The storage area network (SAN) configuration must also be checked by the SAN administrator to ensure that it is mapped to a client NPIV.

 

Determination of whether a device is capable of moving from a virtual environment to a physical environment is applicable for devices that are mapped as VTDs. For devices to be capable of moving from a virtual environment to a physical environment, it must pass the address capability test.

 

If the physical volume is capable of being transferred from a virtual environment to a physical environment, the Virt2Phys_Capable field has a value of YES. If the device is not capable of this move, the field has a value of NO. If the device is not in use by VTD, the field has a value of NA.

..etc..

 

So, I ran this command on my VIOS and focused on the attributes of two different disk types. The first, hdisk1, was a (internal) SAS drive and the second was a FC attached SAN disk (IBM DS8100).

 

$ hostname

vio1

 

$ ioslevel

2.1.2.10-FP-22

 

$ lsdev -type disk

name             status      description

hdisk0           Available   SAS Disk Drive

hdisk1           Available   IBM MPIO FC 2107

hdisk2           Available   IBM MPIO FC 2107

hdisk3           Available   IBM MPIO FC 2107

hdisk4           Available   IBM MPIO FC 2107

hdisk5           Available   IBM MPIO FC 2107

hdisk6           Available   IBM MPIO FC 2107

name             status      description

vtscsi0          Available   Virtual Target Device - Disk

vtscsi1          Available   Virtual Target Device - Disk

vtscsi2          Available   Virtual Target Device – Disk

 

$ lsmap -all

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost0          U7998.61X.10071FA-V1-C11                     0x00000003

 

VTD                   vtscsi0

Status                Available

LUN                   0x8200000000000000

Backing device        hdisk1

Physloc               U78A5.001.WIH074C-P1-C6-T1-W500507630608059A-L4000402D00000000

..etc..

 

$ chkdev

NAME:                hdisk0

IDENTIFIER:          2F1135000C500063973B70BST9146802SS08IBM-ESXSsas

PHYS2VIRT_CAPABLE:   YES

VIRT2NPIV_CAPABLE:   NA

VIRT2PHYS_CAPABLE:   NA

 

NAME:                hdisk1

IDENTIFIER:          200B75W2261002D07210790003IBMfcp

PHYS2VIRT_CAPABLE:   NA

VIRT2NPIV_CAPABLE:   YES

VIRT2PHYS_CAPABLE:   YES

..etc..

 

From the chkdev output, I could determine that hdisk0 could be virtualised and presented to an LPAR (PHYS2VIRT_CAPABLE=YES). The VIRT2NPIV_CAPABLE and the VIRT2PHYS_CAPABLE attributes were both set to N/A, indicating that this device is not currently mapped to a VTD on the VIOS.

 

The attributes for hdisk1 showed that this disk had already been successfully mapped to an LPAR from the VIOS i.e. PHYS2VIRT_CAPABLE=N/A. The disk was also capable of moving from a VSCSI environment to an NPIV (N_Port ID Virtualisation) environment (VIRT2NPIV_CAPABLE=YES).

 

The -verbose flag provides additional attributes about a disk device. You can obtain the PVID, UDID, IEEE and VTD attributes, which may be useful in identifying and validating disks on a VIOS. For example looking at the output below, I can easily verify which VTD hdisk1 is currently mapped to on the VIOS. And I can quickly confirm that hdisk0 is not currently mapped to a VTD.

 

 

$ chkdev -dev hdisk0 -verbose

NAME:                hdisk0

IDENTIFIER:          2F1135000C500063973B70BST9146802SS08IBM-ESXSsas

PHYS2VIRT_CAPABLE:   YES

VIRT2NPIV_CAPABLE:   NA

VIRT2PHYS_CAPABLE:   NA

PVID:                000071facb8888a20000000000000000

UDID:                2F1135000C500063973B70BST9146802SS08IBM-ESXSsas

IEEE:

VTD:

 

$ chkdev -dev hdisk1 -verbose

NAME:                hdisk1

IDENTIFIER:          200B75W2261002D07210790003IBMfcp

PHYS2VIRT_CAPABLE:   NA

VIRT2NPIV_CAPABLE:   YES

VIRT2PHYS_CAPABLE:   YES

PVID:                000071fabe8efb250000000000000000

UDID:                200B75W2261002D07210790003IBMfcp

IEEE:

VTD:                 vtscsi0

 

This command is also mentioned in a new Redbook called “PowerVM Migration from Physical to Virtual Storage”. I recommend you take a look at it and so does Rob McNelly!

TL4 for AIX 6.1 now allows you to rename logical volumes (LV) while they are still online i.e. open.

 

I confirmed this on a test system today.

 

 

# oslevel -s

6100-04-01-0944

 

 

I renamed the logical volume for the /home/nmon filesystem. The original LV name was nmonlv.

 

 

# df -m /home/nmon

Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on

/dev/nmonlv      256.00    254.73    1%       23     1% /home/nmon

 

# lsvg -l rootvg | grep nmon

nmonlv              jfs2       1       1       1    open/syncd    /home/nmon

 

 

I changed the LV name to nmon_lv.

 

# chlv -n nmon_lv nmonlv

 

 

The df command did not show any change i.e. it still displayed the old LV name. As expected.

 

# df -m /home/nmon

Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on

/dev/nmonlv      256.00    254.73    1%       23     1% /home/nmon

 

 

However, the lsvg command confirmed that the LV name had indeed changed. As did the stanza in /etc/filesystems.

 

# lsvg -l rootvg | grep nmon

nmon_lv             jfs2       1       1       1    open/syncd    /home/nmon

 

# grep -p nmon /etc/filesystems

/home/nmon:

        dev             = /dev/nmon_lv

        vfs             = jfs2

        log             = /dev/hd8

        mount           = true

        options         = rw

        account         = false

        quota           = no

 

 

After re-mounting the filesystem, the df command now displayed the new LV name.

 

# umount /home/nmon

# mount /home/nmon

# df /home/nmon

Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on

/dev/nmon_lv      524288    521696    1%       23     1% /home/nmon

 

 

 

Not a big deal you say? Well, prior to TL4, I would have to unmount the filesystem first and then change the LV name. e.g.

 

# oslevel -s

6100-02-02-0849

 

# chlv -n nmon_lv nmonlv

0516-708 chlv: Cannot rename open logical volume. If LV

        contains a filesystem, unmount to close logical volume nmonlv.

 

 

Just another enhancement to the usability features of the AIX operating system.

 

Not as impressive as being able to shrink a filesystem ‘on-the-fly’, but it’s certainly making life easier for AIX administrators. J

 

Before I describe the sudo problem I hit, I will tell you that the root cause of the problem is evident in the following output:

 

# mount

  node       mounted        mounted over    vfs       date        options

-------- ---------------  ---------------  ------ ------------ ---------------

         /dev/hd4         /                jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/hd2         /usr             jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/hd9var      /var             jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/hd3         /tmp             jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/hd11admin   /admin           jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/mksysblv    /export/mksysb   jfs2   Aug 14 12:00 rw,log=/dev/loglv00

         /dev/fslv00      /export/nim      jfs2   Aug 14 12:00 rw,log=/dev/loglv00

         /dev/hd1         /home            jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/hd10opt     /opt             jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/swlv        /software        jfs2   Aug 14 12:00 rw,log=INLINE

         /dev/fslv01      /tftpboot        jfs2   Aug 14 12:00 rw,log=/dev/loglv00

         /dev/livedump    /var/adm/ras/livedump jfs2   Aug 14 12:00 rw,log=/dev/hd8

         /dev/fixelv      /fixes           jfs2   Aug 14 12:00 rw,log=INLINE

         /dev/lpmautolv   /lpmauto         jfs2   Aug 14 12:00 rw,log=INLINE

         /dev/promonlv    /promon          jfs2   Aug 14 12:00 rw,log=INLINE

         /aha             /aha             ahafs  Aug 14 12:00 rw

 

The output above is from the mount command on an AIX host.

 

Can you see any problems that might impact sudo's ability to run?

 

No?

 

Problem Description:

 

A customer was attempting to use sudo to run some commands on an AIX host. Each time, sudo would fail with the following error message:

 

$ sudo -l

sudo: no tty present and no askpass program specified

 

The only information we had was that the host had been rebooted at some point in the last day or so.

 

We spent some time checking and verifying a bunch of things (specifically making sure that the /etc/sudoers file was intact and did not contain any bad entries) but none of them provided any real clues as to the root cause of the problem.

 

Problem Debugging and Investigation

 

We enabled debug for sudo. This involved creating an /etc/sudo.conf file with the following entries:

 

# vi sudo.conf

Debug sudo /var/log/sudo_debug.log all@debug

Debug sudoers.so /var/log/sudo_debug.log all@debug

 

Then, as the non-root user, we attempted to run sudo again. This generated some useful data in the /var/log/sudo_debug.log file. You'll notice we grep'ed for the string tty specifically, given that this was part of the sudo error message provided to us.

 

 

$ sudo -l

sudo: no tty present and no askpass program specified

 

# grep tty /var/log/sudo_debug.log

Aug 10 16:20:26 sudo[14352830] -> get_process_ttyname @ ./ttyname.c:417

Aug 10 16:20:26 sudo[14352830] unable to resolve tty via /proc/14352830/psinfo: A file or directory in the path name does not exist. @ get_process_ttyname() ./ttyname.c:442

Aug 10 16:20:26 sudo[14352830] <- get_process_ttyname @ ./ttyname.c:444 := (null)

Aug 10 16:20:26 sudo[14352830] -> sudo_get_ttysize_v1 @ ./ttysize.c:59

Aug 10 16:20:26 sudo[14352830] -> get_ttysize_ioctl @ ./ttysize.c:38

Aug 10 16:20:26 sudo[14352830] <- get_ttysize_ioctl @ ./ttysize.c:44 := 0

Aug 10 16:20:26 sudo[14352830] <- sudo_get_ttysize_v1 @ ./ttysize.c:75

Aug 10 16:20:26 sudo[14352830] searching for tty time stamp record @ timestamp_lock() ./timestamp.c:603

Aug 10 16:20:26 sudo[14352830] appending new tty time stamp record @ timestamp_lock() ./timestamp.c:611

Aug 10 16:20:26 sudo[14352830] tty time stamp position is 112 @ timestamp_lock() ./timestamp.c:617

Aug 10 16:20:26 sudo[14352830] -> tty_present @ ./tgetpass.c:360

Aug 10 16:20:26 sudo[14352830] <- tty_present @ ./tgetpass.c:361 := false

Aug 10 16:20:26 sudo[14352830] no tty present and no askpass program specified @ tgetpass() ./tgetpass.c:107

 

The following information was of great interest to us! It appeared to be that some information was "missing" from the /proc file system on the host?

 

# grep tty /var/log/sudo_debug.log | grep unable

Aug 10 16:20:26 sudo[14352830] unable to resolve tty via /proc/14352830/psinfo: A file or directory in the path name does not exist. @ get_process_ttyname() ./ttyname.c:442

Aug 18 11:19:59 sudo[21889328] unable to resolve tty via /proc/21889328/psinfo: A file or directory in the path name does not exist. @ get_process_ttyname() ./ttyname.c:442

 

Upon further investigation, we found that /proc did, in fact, appear to be completely empty!! This was not expected or normal on a modern AIX system.

 

# ls -ltr /proc

total 0

 

And, after running the mount command, we discovered that, yes, the /proc file system was NOT mounted!

 

# mount | grep proc

#

 

Resolution

 

Next, we attempted to mount /proc. But we couldn't, as it was not a known file system? Curious indeed.

 

# mount /proc

mount: 0506-334 /proc is not a known file system.

 

The /proc file system stanza was missing from /etc/filesystems. So, we added it back in and mounted the file system successfully.

 

 

# grep -p proc /etc/filesystems

#                                  << No /proc entry found in /etc/filesystems!

 

# grep -p proc /etc/filesystems

/proc:

        dev       = /proc

        vol       = "/proc"

        mount     = true

        check     = false

        free      = false

        vfs       = procfs

 

# mount /proc

# mount | grep proc

         /proc            /proc            procfs Aug 18 11:31 rw

             

# ls -ltr /proc | head -10

total 0

dr-xr-xr-x    8 root     system            0 Aug 10 16:21 sys

-r--r--r--    1 root     system           91 Aug 10 16:21 version

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 0

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 1

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 260

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 65798

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 131336

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 196874

dr-xr-xr-x    1 root     system            0 Aug 10 16:21 262412

 

As soon as /proc was mounted again, the non-root user could run sudo without any issues.

 

$ sudo -l

Password:

User cgibson may run the following commands on 750lpar4:

    (ALL) /usr/sbin/slibclean

$

 

The /proc file system is different from other file systems. It does not reside on disk. The /proc file system resides in AIX memory and is referred to as a virtual file system. The files maintained in /proc represent the running processes on the system.

http://www-01.ibm.com/support/docview.wss?uid=isg3T1012015

https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.files/proc.htm

 

The AIX minidump facility was introduced with AIX 5.3 TL3. A mini dump is a small compressed dump that is stored to NVRAM when a system crashes or a dump is initiated, and then written to the AIX error log on reboot. It can be used to see some of the system’s state and do some debugging when a full dump is not available. It can also be used to get a quick snapshot of a crash without having to transfer the entire dump from the crashed system to IBM support.

 

Please refer to the following, official guide, on "How to examine a minidump in AIX".

 

"Using this crash stack IBM support personnel can then search through the database to find what the fault may mean".

"The RAS effort mentioned..is part of an ongoing effort by AIX to increase stability and to make more information available for troubleshooting when a problem occurs. The ability to look at minidump data has helped solve many issues that would otherwise go unresolved".

 

http://www-01.ibm.com/support/docview.wss?uid=isg3T1010709

 

Of course, minidumps have their limitations and do not replace the need for a full system dump in many cases.

 

"Limitations of minidumps
A minidump is of limited or no use in situations where a server has hung. In this situation we can use mdmprpt to see what was running on each cpu. Practically speaking a full dump is usually needed to determine how and why a server has hung."

 

Here are some examples of how I've used minidump to assist me (and IBM support) in diagnosing the root cause of a system crash.

 

The first example is from a customer that found that one of their AIX partitions would crash when they ran optmem DPO (Dynamic Platform Optimiser) against one of their new POWER8 E880s. Immediately after optmem ran, one specific LPAR would crash. The LPAR reference LED code would show "888 102 300 C20". This LPAR was installed with AIX 7.1 TL3 SP4. When we tried to restart the partition, it would crash (and dump) several times before it would start successfully. Once the partition booted successfully, we noticed that there were several system dumps (SYSDUMP) and minidumps (COMPRESSED MINIMAL DUMP) shown in the AIX error report.

 

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION

A6DF45AA   0411163117 I O RMCdaemon      The daemon is started.

2BFA76F6   0411163117 T S SYSPROC        SYSTEM SHUTDOWN BY USER

9DBCFDEE   0411163117 T O errdemon       ERROR LOGGING TURNED ON

192AC071   0411162917 T O errdemon       ERROR LOGGING TURNED OFF

A6DF45AA   0411162517 I O RMCdaemon      The daemon is started.

2BFA76F6   0411162517 T S SYSPROC        SYSTEM SHUTDOWN BY USER

9DBCFDEE   0411162517 T O errdemon       ERROR LOGGING TURNED ON

192AC071   0411162217 T O errdemon       ERROR LOGGING TURNED OFF

A6DF45AA   0411161117 I O RMCdaemon      The daemon is started.

67145A39   0411161017 U S SYSDUMP        SYSTEM DUMP                                  << CRASH!

F48137AC   0411161017 U O minidump       COMPRESSED MINIMAL DUMP                      << Minidump information

9D035E4D   0411161017 P S SYSVMM         DATA STORAGE INTERRUPT, PROCESSOR

9DBCFDEE   0411161017 T O errdemon       ERROR LOGGING TURNED ON

A6DF45AA   0411155917 I O RMCdaemon      The daemon is started.

67145A39   0411155817 U S SYSDUMP        SYSTEM DUMP                                  << CRASH!

F48137AC   0411155817 U O minidump       COMPRESSED MINIMAL DUMP                      << Minidump information

9D035E4D   0411155817 P S SYSVMM         DATA STORAGE INTERRUPT, PROCESSOR

9DBCFDEE   0411155817 T O errdemon       ERROR LOGGING TURNED ON

A6DF45AA   0411155717 I O RMCdaemon      The daemon is started.

67145A39   0411155717 U S SYSDUMP        SYSTEM DUMP                                  << CRASH!

F48137AC   0411155717 U O minidump       COMPRESSED MINIMAL DUMP                      << Minidump information

9D035E4D   0411155717 P S SYSVMM         DATA STORAGE INTERRUPT, PROCESSOR            << CRASH!

9DBCFDEE   0411155717 T O errdemon       ERROR LOGGING TURNED ON

6AEB31F5   0410143317 I H sysplanar0     Platform Resource Reassignment.              << DPO

 

I used the minidump reporting tool (mdmprpt) to perform a quick analysis of the cause of the crash, by reviewing the "Stack Trace" information. The mdmprpt tool has a couple of options.

 

 

# mdmprpt -?

Usage:

mdmprpt [[-l seq_no] [-i filename]] | [ -b filename ] [ -F ] [-r]

 

Process error log entries from the supplied file(s).

-l seq_no Format the minidump at the specified sequence number in the error log.

-i filename  Uses the error log file specified by the filename param.

-b filename  Uses the binary file specified by the filename param.

-F prints only dump failure information.

-r print the raw minidump, without formatting.

 

If no parameters are specified, the most recently logged minidump is shown. You may not specify -b with -i or -l.

 

Typically, the most common usage is to simply run mdmprpt and redirect the output to a file for review.

 

# mdmprpt > /tmp/minidumpreport.out

 

The most relevant section of the report is the "Stack Trace" section(s).

 

# vi /tmp/minidumpreport.out

 

MINIDUMP VERSION 4D33

***************************************************

64-bit Kernel, 57 Entries

 

Last Error Log Entry:

         Error ID: 9D035E4D      Resource Name: SYSVMM

         Detail Data: 0000000000000000 00007FFFFFFFD080 00000007000000C7 0000000000000115

 

Symptom Information:

        Crash Location: [00000000000ADE98] trc_generate_ea+78

        Component: COMP  Exception Type: 277

 

Data From CPU #1 (Faulting CPU)

***************************************************

...

Stack Trace:

[00000000000ADE98] trc_generate_ea+78

        0000000000000002 F1000A002033C000 C0C0C0C0C0C0C0C1 8000002841900000

        00000000025BBC00 00000000000034C8 00000000009F3998 0000000003322000

[00000000000ACF64] trc_inmem_recor+144

        0000000000000000 F1000815B0052D00 0000000000007800 000000000000F000

 ...

 

In this case, the stack trace showed messages relating to trc_generate and trc_inmem_recor. Using this information, I was able to find several hits (both internally to IBM and using my preferred internet search tool) for the potential cause of the crash. The problem was a known issue and related to the following APAR.

 

IV69116: AIX CAN CRASH IBM POWER8 E880 (9119-MHE) WITH > THAN 128 CORES APPLIES TO AIX 7100-03

http://www-01.ibm.com/support/docview.wss?uid=isg1IV69116

 

 

We verified our findings with IBM support (who performed their own internal search) and we concluded that an update would be required. We chose to update the AIX system to AIX 7.1 TL4 SP3, and the problem went away.

 

In the next example, the customer had updated one of their AIX partitions to AIX 7.1 TL4 SP3 (previously on AIX 7.1 TL4 SP1). This LPAR also housed a single, AIX 5.3 versioned WPAR. The update was successful but whenever they ran the stopwpar command, the entire partition would crash (dump). So, once again, I employed the mdmprpt utiliy to check the stack traces.

 

MINIDUMP VERSION 4D33

***************************************************

64-bit Kernel, 48 Entries

 

Last Error Log Entry:

         Error ID: 9D035E4D      Resource Name: SYSVMM

         Detail Data: 000000000A000000 00000000C000C400 F1000000C01ED500 0000000000000086

 

Symptom Information:

        Crash Location: [F1000000C012B968] pcmUserGetDevIn+E8

        Component: COMP  Exception Type: 134

...

Stack Trace:

[F1000000C012B968] pcmUserGetDevIn+E8

       000000002FF44EAC 00000000DEADBEEF 00000000A28F47CB 0000000000000005

       F1000A000036A500 0000000000010000 F1000000C01D3000 000000002FEF56E0

[F1000000C01D1808] sddUserInterfac+588

       0000000000000002 00000000DEADBEEF 00000000A28F47CB 0000000000000088

...

 

The report showed us pcmUserGetDevIn and sddUserInterfac in the stack trace. Both are related to the IBM sddpcm device driver. They appeared to be the likely culprits. Searching for these presented us with a couple of potential reasons for the crash, such as too many paths configured for a sddpcm device (which we checked and confirmed was far less than the maximum of 32). So, my next question was, what version of sddpcm was installed? We found that the customer had (unexpectedly) two different versions of sddpcm installed. In the Global LPAR, the sddpcm version was 2.6.9.0 (the latest) and in the vWPAR, 2.6.6.0. During the TL update, the customer had updated sddpcm to the latest level available, but had forgotten to update sddpcm inside the vWPAR as well. We then discovered, that if they simply ran "pcmpath query device" from inside the vWPAR, this would crash the partition. The latest sddpcm readme file contained the following information "5557  Fix for AIX 7.1 crash during pcmpath query device"! This was indeed the cause of the system crash. After the customer updated sddpcm, inside the vWPAR, the system was stable and the problem was resolved.

 

If you encounter a system crash in the future, and there's minidump data available, why not consider using the minidump reporting tool to analyse the issue? It might just help speed up the root cause analysis of the problem.

 

 

 

The latest release of the VIOS provides a way to automatically backup your virtual device configuration whenever it changes. From the IBM Knowledge Center for POWER8:

 

https://www.ibm.com/support/knowledgecenter/POWER8/p8hcg/p8hcg_viosbr.htm

 

"The viosbr command automatically creates a backup, whenever there are any configuration changes. This functionality is known as the autoviosbr backup. It is triggered every hour, and checks if there are any configuration changes, or any other changes. If it detects any changes, a backup is created. Otherwise, no action is taken. The backup files resulting from the autoviosbr backup are located under the default path /home/padmin/cfgbackups with the names autoviosbr_SSP.<cluster_name>.tar.gz for cluster level and autoviosbr_<hostname>.tar.gz for node level. The cluster-level backup file is present only in the default path of the database node.

 

The -autobackup flag is provided for the autoviosbr backup functionality. By default, autoviosbr backup is enabled on the system. To disable the autoviosbr backup, use the stop parameter and to enable it you can use the start parameter. When the autoviosbr backupis disabled, no autoviosbr related tar.gz file is generated.

 

To check if the autoviosbr backup file, present in the default path is up to date, you can use the status parameter. To access the cluster-level backup file on any node of the cluster, use the save parameter. This action is necessary as the cluster-level backup file is present in the default path of the database node only.

 

If the node is a part of cluster, you can use the -type flag to specify the parameter. The parameter can be either cluster or node, depending on if it is a cluster-level or a node-level backup.

Flags""

 

On your VIOS, under oem_setup_env (as root), you'll find the following new entry in root's crontab:

 

# crontab -l | grep autoviosbr

0 * * * * /usr/ios/sbin/autoviosbr -start 1>/dev/null 2>/dev/null

 

This entry will check for any configuration changes and generate a new backup if necessary. Here's an example from my VIOS, running 2.2.5.10.

 

$ ioslevel

2.2.5.10

 

I create a new virtual optical device, which should trigger a new backup the next time the autoviosbr script runs (once per hour). Prior to creating the device, viosbr shows the autobackup status as Complete (no changes). I ensure that autobackup is configured by stopping and starting it with the viosbr command.

 

$ viosbr -autobackup stop -type node

Autobackup stopped successfully.

 

$ viosbr -autobackup start -type node

Autobackup started successfully.

 

$ viosbr -autobackup status -type node

Node configuration changes:Complete.

 

$ mkvdev -fbo -vadapter vhost34

vtopt13 Available

 

Immediately after the vtopt device is created, the autobackup status displays as Pending (something has changed but has not yet been backed up).

 

$ viosbr -autobackup status -type node

Node configuration changes:Pending.

 

The autoviosbr file is created in /home/padmin/cfgbackups.

 

$ r oem

oem_setup_env

 

# cd /home/padmin/cfgbackups

# ls -tlr

total 72

-rw-r--r--    1 root     staff         12189 May 10 14:00 autoviosbr_s824vio2.tar.gz

 

On the hour, the autoviosbr script runs, notices that the configuration has changed and generates a new viosbr backup file. The viosbr autobackup status changes to Complete.

 

$ viosbr -autobackup status -type node

Node configuration changes:Complete.

 

# ls -ltr

total 88

-rw-r--r--    1 root     staff         12218 May 10 15:00 autoviosbr_s824vio2.tar.gz

 

What's inside the tar file?

 

# cp /home/padmin/cfgbackups/autoviosbr_s824vio2.tar.gz /tmp/cg/

# cd /tmp/cg

# gzip -d autoviosbr_s824vio2.tar.gz

# tar -xvf autoviosbr_s824vio2.tar

x ArtexRules_MTM8286-42A02214F58VP2.xml, 20549 bytes, 41 media blocks.

x autoviosbr_s824vio2, 198554 bytes, 388 media blocks.

# ls -ltr

total 888

-rw-r--r--   1 root staff  198554 May 10 15:00 autoviosbr_s824vio2

-rw-r--r--   1 root staff  20549 May 10 15:00 ArtexRules_MTM8286-42A02214F58VP2.xml

-rw-r--r--   1 root staff  225280 May 10 15:00 autoviosbr_s824vio2.tar

 

The ArtexRules file contains device specific configuration attributes.

 

# head ArtexRules_MTM8286-42A02214F58VP2.xml

<?xml version="1.0" encoding="UTF-8"?>

<Profile origin="get" version="3.0.0" date="2017-05-10T04:10:54Z">

 <Catalog id="devParam.disk.fcp.mpioosdisk" version="3.0">

  <Parameter name="reserve_policy" value="single_path" applyType="nextboot" reboot="true">

   <Target class="device" instance="disk/fcp/mpioosdisk"/>

  </Parameter>

 </Catalog>

 <Catalog id="devParam.disk.fcp.mpioapdisk" version="3.0">

  <Parameter name="reserve_policy" value="single_path" applyType="nextboot" reboot="true">

   <Target class="device" instance="disk/fcp/mpioapdisk"/>

#

 

The autoviosbr file contains virtual device configuration information. In this file I can confirm that the configuration of my new virtual optical device (vtopt13) has been backed up.

 

# grep vtopt13 autoviosbr_s824vio2

                <name>vtopt13</name>

                        <vtdName>vtopt13</vtdName>

                        <value>vtopt13</value>

                                          

# vi autoviosbr_s824vio2

...

        <virtDevOPT>

                <name>vtopt13</name>

                <state>AVAILABLE</state>

                <locCode>U8286.42A.214F58V-V2-C101-L2</locCode>

                <unique_type>virtual_target/vtdev/fbopt</unique_type>

                <type>BDOPFBOPT</type>

                <fbOpt>

                </fbOpt>

                <CuAtcount>2</CuAtcount>

                <CuAt name="LogicalUnitAddr" value="0x8200000000000000" action="FALSE" />

                <CuAt name="udid" value="19cc3cb76ad94674c3" action="TRUE" />

        </virtDevOPT>

...

 

If you want to force a backup to run, before the next hour, manually run the following as root (oem_setup_env):

 

$ oem_setup_env

# /usr/ios/sbin/autoviosbr -start

 

On more than one occasion I’ve written articles and blogs on how to migrate AIX systems to a new version of the AIX operating system. You can find a list these articles (and others) here. Each of the NIM articles focuses on migrating AIX systems (which are configured as NIM clients). The approach is always the same; use a NIM master to migrate the NIM clients to the next release of AIX. What they don’t discuss is how to migrate your NIM master to a new version of the OS. This is the very first step you should take before you consider migrating your clients. The NIM master should always be at the same or higher AIX level than the level you are migrating to on a client. So, in this latest article, I’ll discuss, step by step, how to migrate your NIM master from AIX 7.1 to AIX 7.2 (the latest release of the AIX OS).

 

https://www.ibm.com/developerworks/community/blogs/cgaix/resource/Migrating_NIMMaster_AIX72_VML.pdf