Chris's AIX Blog

Is there a way to find on which processor a process is executing?

cggibbo Tags: chris trace gibson aix curt 4 Comments 9,167 Views

A question I received this week from a customer.

Hi Chris,

Is there a way to find on which processor a process is executing?

If I’ve used binprocessor, ps –emo THREAD, would show which processor a process is bound too. But when that’s not been used, what are the options?

The ps command will only help if we bind a process to a specific CPU. For example, here we can see that the bootpd process has been bound (and then un-bound) on processor 3:

# bindprocessor -q

The available processors are: 0 1 2 3

# bindprocessor 8585302 3

# ps -o THREAD -p 8585302

USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMAND

root 8585302 4063394 - A 0 60 1 - 200001 - 3 bootpd /etc/bootptab

# bindprocessor -u 8585302

# ps -o THREAD -p 8585302

USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMAND

root 8585302 4063394 - A 0 60 1 - 200001 - - bootpd /etc/bootptab

The trace command will tell us what CPU a process is scheduled on.*

# /usr/bin/trace –J’curt' -P'p' '-a'

# sleep 30

# trcstop

# trcrpt -O'exec=y' -O'pid=n' -O'tid=n' -O'wparname=off' -O'cid=off' -O'svc=y' -O'timestamp=1' | /usr/lib/ras/trace_smutil cat Z'/tmp/trace.out'

# vi /tmp/trace.out (I’m interested in PID 904428 in this case)

106 ksh 0.033107 dispatch: cmd=ksh pid=9044208 tid=9175205 priority=135 old_tid=9175205 old_priority=135 CPUID=2 [302 usec]

101D ksh 0.033108 Home SRAD=-1 Home ref1=-1 Dispatch SRAD=-1 Dispatch ref1=-1

200 ksh 0.033108 resume ksh iar=100180CC cpuid=02

*Note: If you are running AIX 5.3 try these options with trcrpt (tested on an AIX 5.3 TL11 SP3 system):

# trcrpt -O'exec=y' -O'pid=n' -O'tid=n' -O'svc=y' -O'timestamp=1' | /usr/lib/ras/trace_smutil cat Z'/tmp/trace.out'

Perhaps there are other methods of monitoring this type of activity online, rather than offline?

0516-966 reorgvg: Unable to create internal map and LVM logging.

cggibbo 4 Comments 12,000 Views

I was performing a volume group re-org i.e. changing the INTER-POLICY of a logical volume from minimum to maximum.

# lslv fixeslv| grep INTER

INTER-POLICY: minimum RELOCATABLE: yes

# chlv -e x fixeslv

# lslv fixeslv| grep INTER

INTER-POLICY: maximum RELOCATABLE: yes

I attempted to run the reorgvg command. I was greeted by the following error message!

# reorgvg tempvg fixeslv

0516-966 reorgvg: Unable to create internal map.

I ran the command again, this time with truss. I found that the /usr/sbin/allocp command was being called and was failing. I determined this must be because of a lack of space at the logical volume layer.

# /usr/sbin/allocp -?

/usr/sbin/allocp: Not a recognized flag: ?

0516-422 allocp: [-i LVid] [-t Type] [-c Copies] [-s Size]

[-k] [-u UpperBound>] [-e InterPolicy] [-a InterPolicy

The truss output showed:

statx("/usr/sbin/allocp", 0x2FF21ED8, 76, 0) = 0

statx("/usr/sbin/allocp", 0x20009E70, 176, 020) = 0

kioctl(2, 22528, 0x00000000, 0x00000000) Err#25 ENOTTY

kfork() = 3735812

_sigaction(20, 0x00000000, 0x2FF21F20) = 0

_sigaction(20, 0x2FF21F20, 0x2FF21F30) = 0

kwaitpid(0x2FF21F90, -1, 6, 0x00000000, 0x00000000) = 3735812

And yes, my volume group was indeed out of free PPs!

# lsvg tempvg

VOLUME GROUP: tempvg VG IDENTIFIER: 00f6027300004c0000000130773bdb73

VG STATE: active PP SIZE: 512 megabyte(s)

VG PERMISSION: read/write TOTAL PPs: 99 (50688 megabytes)

MAX LVs: 256 FREE PPs: 0 (0 megabytes)

LVs: 1 USED PPs: 99 (50688 megabytes)

OPEN LVs: 1 QUORUM: 2 (Enabled)

TOTAL PVs: 2 VG DESCRIPTORS: 3

STALE PVs: 0 STALE PPs: 0

ACTIVE PVs: 1 AUTO ON: yes

MAX PPs per VG: 32768 MAX PVs: 1024

LTG size (Dynamic): 256 kilobyte(s) AUTO SYNC: no

HOT SPARE: no BB POLICY: relocatable

PV RESTRICTION: none

cgaix7[/opt] >

Silly me, it clearly states in the reorgvg man page that there must be at least one free PP in the volume group for the command to run successfully.

2 At least one free physical partition (PP) must exist on the specified volume group for the reorgvg command to run successfully. For mirrored logical volumes, one free PP per physical volume (PV) is required in order for the reorgvg command to maintain logical volume strictness during execution; otherwise the reorgvg command still runs, but moves both copies of a logical partition to the same disk during its execution.

So I shrunk the file system in question (there was a large amount of allocated but unused file system space, so it was safe to shrink it).

# chfs -a size=-40G /fixes

Filesystem size changed to 19922944

Inlinelog size changed to 38 MB.

Now I had plenty of free PPs in my volume group.

# lsvg tempvg

VOLUME GROUP: tempvg VG IDENTIFIER: 00f6027300004c0000000130773bdb73

VG STATE: active PP SIZE: 512 megabyte(s)

VG PERMISSION: read/write TOTAL PPs: 99 (50688 megabytes)

MAX LVs: 256 FREE PPs: 80 (40960 megabytes)

LVs: 1 USED PPs: 19 (9728 megabytes)

OPEN LVs: 1 QUORUM: 2 (Enabled)

TOTAL PVs: 1 VG DESCRIPTORS: 2

STALE PVs: 0 STALE PPs: 0

ACTIVE PVs: 1 AUTO ON: yes

MAX PPs per VG: 32768 MAX PVs: 1024

LTG size (Dynamic): 256 kilobyte(s) AUTO SYNC: no

HOT SPARE: no BB POLICY: relocatable

PV RESTRICTION: none

And reorgvg now ran with success.

# reorgvg tempvg fixeslv

0516-962 reorgvg: Logical volume fixeslv migrated.

I found some interesting information in the lvmcfg alog file.

# alog -o -t lvmcfg | grep reorgvg

[S 2359470 3080260 06/15/11-19:26:36:915 reorgvg.sh 580] reorgvg tempvg fixeslv

[E 2359470 0:964 reorgvg.sh 23] reorgvg: exited with rc=1

[S 3145780 3080260 06/15/11-19:28:19:645 reorgvg.sh 580] reorgvg tempvg fixeslv

[E 3145780 190:611 reorgvg.sh 23] reorgvg: exited with rc=0

I can see when my reorgvg failed (rc=1) and when it succeded (rc=0). This is also a good way of determining when a reorgvg command is issued and when it finished. Of course, an easier way would be to start the reorgvg command with the time command. It will produce a nice little summary of the time taken.

# time reorgvg tempvg fixeslv

0516-962 reorgvg: Logical volume fixeslv migrated.

real 3m12.94s

user 0m1.52s

sys 0m4.60s

But if I forgot to use the time command, I can look at the lvmcfg alog file for an answer. In the following example, the reorgvg.sh process is started at 23:49. The entry in the log file begins with an uppercase S. The entry that starts with an uppercase E indicates the end of the reorgvg.sh process. It is the information in the third field that tells me how long the process ran for in seconds:milliseconds.

# alog -o -t lvmcfg | grep reorgvg | tail -3

[S 3735990 3080260 06/15/11-23:49:58:491 reorgvg.sh 580] reorgvg tempvg fixeslv

[E 3735990 192:969 reorgvg.sh 23] reorgvg: exited with rc=0

In this case, my “reorgvg” took 192 seconds, 969 milliseconds (192:969), which equates to a little over 3 mins and matches the time reported from the time command (above).

Here is the format of an end (E) entry in the lvmcfg logs:

[E 237694 1:733 mklv.sh 71] mklv: exited with rc=0

· E = Verbosity

· 237694 = pid

· 1:733 = time elapsed since start entry(s:ms)

· mklv.sh = file writing to log

· 71 = line number where log is being written

· mklv: = command exiting

· exited with rc=0 = exit code of command

More information on the new LVM logging techniques can be found at the following link:

http://www.aixmind.com/?p=809

Collecting WWPNs for new Virtual Fibre Channel Adapters

cggibbo Tags: wwpn vfc sed lssyscfg aix fibre gibson channel hmc chris virtual 4 Comments 21,214 Views

I was collecting a list of WWPNs for a bunch of new AIX LPARs I was installing from scratch. There were two ways I could find the WWPN for a virtual Fibre Channel adapter on a new LPAR i.e. one that did not yet have an operating system installed.

I started by checking the LPAR properties from the HMC (as shown below).

To speed things up I moved to the HMC command line tool, lssyscfg, to display the WWPNs (as shown below).

hscroot@hmc1:~> lssyscfg -r prof -m 750-2 -F virtual_fc_adapters --filter lpar_names=lpar1

"""32/client/2/vio2/30/c0507603a2410000,c0507603a2410001/0"",""33/client/2/vio2/31/c0507603a2410002,c0507603a2410003/0"",""30/client/1/vio1/30/c0507603a2410004,c0507603a2410005/0"",""31/client/1/vio1/31/c0507603a2410006,c0507603a2410007/0""" """30/client/1/vio1/30/c0507603a2410004,c0507603a2410005/0"",""31/client/1/vio1/31/c0507603a2410006,c0507603a2410007/0"""

This was good. I now had a list of WWPNs for each LPAR.

# cat lpar1_wwpns.txt

c0507603a292007c

c0507603a292007e

c0507603a2920078

c0507603a292007a

I gave these WWPNS to my SAN administrator so that he could manually “zone in” the LPARs on the SAN switches and allocate storage to each. He then, half-jokingly said, “Gee, it would be nice if you could insert the colons into the WWPNS for me! J”. Of course, this got me thinking and after a few minutes of playing with sed, I came up with a way to do this quickly.

# cat lpar1_wwpns.txt | sed 's/../&:/g;s/:$//'

c0:50:76:03:a2:92:00:7c

c0:50:76:03:a2:92:00:7e

c0:50:76:03:a2:92:00:78

c0:50:76:03:a2:92:00:7a

Now my list of WWPNs was ready to be cut’n’paste by my SAN admin. He was happy.

AIX multibos mksysb migration

cggibbo Tags: mksysb aix migration multibos chris gibson 4 Comments 17,485 Views

I’ve written about multibos before, here and here. But recently I started experimenting with multibos mksysb migration. A customer asked me how this worked and apart from a high-level view I wasn’t able to provide any real world experience, so I thought I’d give it a try. What follows is just a ‘brain dump’ from my quick test.

First of all this isn’t really a migration. It just simply populates a second instance of AIX with a higher-version. It doesn’t really migrate (or merge) your existing configuration into the second instance. So I’m not sure how useful this feature really is right now.

Starting with 5.3 TL9 you can add a 6.1 TL2 (or above) instance. This is done with the new –M flag. You must be running with the 64bit kernel.

This isn’t really a migration because it populates the second instance using a mksysb based on the new release.

In 6.1 TL2 a new flag (-M) was added to the mksysb command which allows you to create a mksysb for use with multibos. It creates a backup of BOS (/, /usr, /var, /opt).
bos.alt_disk_install.boot_images must be installed.

To preview the multibos migration
multibos –M <location of mksysb> -spX

To perform the multibos migration
multibos –M <location of mksysb> -sX

It is not advised to run in this environment for an extended period of time. There could be problems if tfactor or maps are used. Be aware that 6.1 specific attributes may not be reflected in the standby instance.

So in my lab environment I have two AIX LPARs. One is running AIX 6.1 and the other running AIX 7.1.

First I take a mksysb (with the –M flag) of the AIX 7.1 system to a file. This file will be called by multibos to populate the second instance.

aix7[/] > mksysb -Mie /data/aix7-mksysb

Creating information file (/image.data) for rootvg.

Creating list of files to back up.

Backing up 71643 files.....

71643 of 71643 files (100%)

0512-038 mksysb: Backup Completed Successfully.

aix7[/] > ls -ltr /data

total 4276112

drwxr-xr-x 2 root system 256 Feb 21 20:59 lost+found

-rw-r--r-- 1 root system 2189363200 Feb 21 21:06 aix7-mksysb

I copied this file over to my AIX 6.1 system. This was the system that was to be ‘migrated’. The next step was to perform a preview of the multibos operation.

root@aix6 /# multibos -sXp -M /data/aix7-mksysb

Initializing multibos methods ...

Initializing log /etc/multibos/logs/op.alog ...

Gathering system information ...

+-----------------------------------------------------------------------------+

Preview

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

Validating install images location ./aix7-mksysb

Processing preview information ...

ACTIVE LV: hd4

STANDBY LV: bos_hd4

TYPE: jfs2

ACTIVE FS: /

STANDBY FS: /bos_inst

ACTION: Setup

STATE: mounted

ACTIVE LV: hd2

STANDBY LV: bos_hd2

TYPE: jfs2

ACTIVE FS: /usr

STANDBY FS: /bos_inst/usr

ACTION: Setup

STATE: mounted

ACTIVE LV: hd9var

STANDBY LV: bos_hd9var

TYPE: jfs2

ACTIVE FS: /var

STANDBY FS: /bos_inst/var

ACTION: Setup

STATE: mounted

ACTIVE LV: hd10opt

STANDBY LV: bos_hd10opt

TYPE: jfs2

ACTIVE FS: /opt

STANDBY FS: /bos_inst/opt

ACTION: Setup

STATE: mounted

ACTIVE LV: hd5

STANDBY LV: bos_hd5

TYPE: boot

ACTIVE FS: None

STANDBY FS: None

ACTION: Setup

STATE: closed

ACTIVE LV: hd8

STANDBY LV: bos_hd8

TYPE: jfs2log

ACTIVE FS: None

STANDBY FS: None

ACTION: Setup

STATE: open

Log file is /etc/multibos/logs/op.alog

Return Status = SUCCESS

With a successful preview out of the way, I initiated the real multibos operation.

# multibos -sX -M /data/aix7-mksysb

Gathering system information ...

+-----------------------------------------------------------------------------+

Setup Operation

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

Validating install images location ./aix7-mksysb

New volume on /data/aix7-mksysb:

Cluster size is 51200 bytes (100 blocks).

The volume number is 1.

The backup date is: Mon Feb 21 21:05:39 CST 2011

Files are backed up by name.

The user is root.

x 13007 ./image.data

x 168589 ./tmp/vgdata/rootvg/backup.data

The total size is 181596 bytes.

The number of restored files is 2.

+-----------------------------------------------------------------------------+

Logical Volumes

+-----------------------------------------------------------------------------+

Creating standby BOS logical volume bos_hd5

Creating standby BOS logical volume bos_hd4

Creating standby BOS logical volume bos_hd2

Creating standby BOS logical volume bos_hd9var

Creating standby BOS logical volume bos_hd10opt

Creating standby BOS logical volume bos_hd8

+-----------------------------------------------------------------------------+

File Systems

+-----------------------------------------------------------------------------+

Creating all standby BOS file systems ...

Creating standby BOS file system /bos_inst on logical volume bos_hd4

Creating standby BOS file system /bos_inst/usr on logical volume bos_hd2

Creating standby BOS file system /bos_inst/var on logical volume bos_hd9var

Creating standby BOS file system /bos_inst/opt on logical volume bos_hd10opt

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Mounting all standby BOS file systems ...

Mounting /bos_inst

Mounting /bos_inst/usr

Mounting /bos_inst/var

Mounting /bos_inst/opt

+-----------------------------------------------------------------------------+

BOS Files

+-----------------------------------------------------------------------------+

Including files for file system /

Including files for file system /usr

Including files for file system /var

Including files for file system /opt

Copying files using backup/restore utilities ...

New volume on /data/aix7-mksysb:

Cluster size is 51200 bytes (100 blocks).

The volume number is 1.

The backup date is: Mon Feb 21 21:05:39 CST 2011

Files are backed up by name.

The user is root.

x 6086 ./bosinst.data

x 11 ./tmp/vgdata/rootvg/image.info

x 13007 ./image.data

x 168589 ./tmp/vgdata/rootvg/backup.data

x 0 ./

x 47 ./.profile

x 63 ./.rhosts

x 13192 ./.sh_history

x 0 ./.ssh

x 1647 ./.ssh/authorized_keys

x 0 ./.topasrecrc

x 0 ./.topasrecrc/topasrec.lock

x 141722 ./.topasrecrc/topasrec.log1

x 0 ./.topasrecrc/topasrec.log2

x 108 ./.vi_history

x 0 ./admin

x 0 ./audit

x 8 ./bin

x 0 ./cdat

x 59 ./cdat/cdat.xml

x 0 ./cg

x 0 ./cgps

x 0 ./dev

x 0 ./dev/.SRC-unix

x 0 ./dev/IPL_rootvg

x 0 ./dev/__pv17.0.10617024

x 0 ./dev/__vg10

x 0 ./dev/audit

.....BLAH.....

x ./dev/zero

Creating standby BOS boot image on boot logical volume bos_hd5

254+0 records in.

254+0 records out.

179+1 records in.

177+1 records out.

saving to '/dev/bos_hd5'

44 CuDv objects to be saved

130 CuAt objects to be saved

21 CuDep objects to be saved

7 CuVPD objects to be saved

351 CuDvDr objects to be saved

2 CuPath objects to be saved

0 CuPathAt objects to be saved

0 CuData objects to be saved

Number of bytes of data to save = 18716

Compressing data

Compressed data size is = 6393

bi_start = 0x400

bi_size = 0x1677200

bd_size = 0x1657200

ram FS start = 0x8c3570

ram FS size = 0xd93abe

sba_start = 0x1657600

sba_size = 0x20000

sbd_size = 0x0

Checking boot image size:

new save base byte cnt = 0x18fd

Wrote 6397 bytes

Successful completion

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Unmounting all standby BOS file systems ...

Unmounting /bos_inst/opt

Unmounting /bos_inst/var

Unmounting /bos_inst/usr

Unmounting /bos_inst

+-----------------------------------------------------------------------------+

Bootlist Processing

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

Setting bootlist to logical volume bos_hd5 on hdisk0.

ATTENTION: firmware recovery string for standby BLV (bos_hd5):

boot /vdevice/v-scsi@30000016/disk@8100000000000000:4

ATTENTION: firmware recovery string for standby BLV (bos_hd5):

boot /vdevice/v-scsi@30000017/disk@8100000000000000:4

ATTENTION: firmware recovery string for active BLV (hd5):

boot /vdevice/v-scsi@30000016/disk@8100000000000000:2

ATTENTION: firmware recovery string for active BLV (hd5):

boot /vdevice/v-scsi@30000017/disk@8100000000000000:2

Log file is /etc/multibos/logs/op.alog

Return Status = SUCCESS

Now that the second instance had been created, I thought I’d try to start a multibos shell and access the AIX 7.1 environment. I didn’t think this would work but I tried my luck anyway!

root@aix6 /# multibos -S

Initializing multibos methods ...

Initializing log /etc/multibos/logs/op.alog ...

Gathering system information ...

+-----------------------------------------------------------------------------+

Multibos Shell Operation

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Mounting all standby BOS file systems ...

Mounting /bos_inst

Mounting /bos_inst/usr

Mounting /bos_inst/var

Mounting /bos_inst/opt

+-----------------------------------------------------------------------------+

Multibos Root Shell

+-----------------------------------------------------------------------------+

Starting multibos root shell ...

multibos: 0565-110 Cannot start multibos shell due to incompatible levels of active and standby BOS.

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Unmounting all standby BOS file systems ...

Unmounting /bos_inst/opt

Unmounting /bos_inst/var

Unmounting /bos_inst/usr

Unmounting /bos_inst

Log file is /etc/multibos/logs/op.alog

Return Status: FAILURE

Upon checking my bootlist output, I noticed (as expected) that the list now contained two extra entries for bos_hd5. These were the boot logical volume entries for the second instance. If I was to boot from this LV I’d be booting into AIX 7.1. Cool.

root@aix6 /# bootlist -m normal -o

hdisk0 blv=bos_hd5

hdisk0 blv=hd5

So at this point, I’d created a second instance of AIX running 7.1. My current version of (running) AIX was AIX 6.1. All I had to do now was reboot the LPAR and let it restart as an AIX 7.1 system.

root@aix6 /# oslevel -s

6100-01-05-0920

root@aix6 / # shutdown –Fr

The LPAR rebooted successfully and I found I was now running AIX 7.1, just as I’d hoped.

aix6[/] > oslevel -s

7100-00-01-1037

If I wanted to go back to AIX 6.1, I would change my bootlist setting again and restart the LPAR.

aix6[/] > bootlist -m normal -o

hdisk0 blv=bos_hd5 pathid=0

hdisk0 blv=bos_hd5 pathid=1

hdisk0 blv=hd5 pathid=0

hdisk0 blv=hd5 pathid=1

aix6[/] > bootlist -m normal hdisk0 blv=hd5 pathid=0 hdisk0 blv=hd5 pathid=1 hdisk0 blv=bos_hd5 pathid=0 hdisk0 blv=bos_hd5 pathid=1

aix6[/] > bootlist -m normal -o

hdisk0 blv=hd5 pathid=0

hdisk0 blv=hd5 pathid=1

hdisk0 blv=bos_hd5 pathid=0

hdisk0 blv=bos_hd5 pathid=1

aix6[/] > shutdown –Fr

root@aix6 /# oslevel -s

6100-01-05-0920

Now that I’ve actually tried this method of migration, I’m not sure I’d actually use it in its current form.

Although the migration keeps my hostname and IP address, the file systems are not shared between instances. Most of the target systems configuration is not retained. For example, any user accounts I create on my AIX 6.1 system would also need to be created on the existing AIX.7.1 system which I used to create the AIX 7.1 mksysb image. It reminds me a little of a preservation install.

More information on multibos:

http://www.aixmind.com/?p=1192

Virtual I/O Server and DS5020 disk.

cggibbo Tags: ds5020 aix vios 4 Comments 19,378 Views

Recently I had the pleasure of configuring a couple of POWER7 720s for a customer. Each 720 was to host roughly 12 LPARs each. There were two VIO servers and a NIM server per 720.

Everything went along nicely and according to plan. In a few days we had both systems built. My steps for building the systems were almost identical to those described by Rob McNelly in a recent post.

All four VIO servers were running the latest VIOS code i.e. 2.2.0.10-FP-24 SP-01. All the client LPARs were running AIX 6.1 TL6 SP3. Each VIOS was configured with two FC paths to the SAN and the SAN storage device was an IBM DS5020.

Native AIX MPIO was in use on the VIOS and the AIX LPARs. I did not deploy SDDPCM on the VIOS as this is currently unsupported with the DS5020.

Once the LPAR builds were complete we performed a number of “integration tests”. These typically involve disconnecting network and SAN cables from each VIOS and observing how the VIOS and LPARs respond to and recover from these types of conditions.

One of the integration tests required that BOTH fibre cables be disconnected from the first VIOS and confirm that the client LPARs were not impacted i.e. that all I/O travelled via the second VIOS.

During the test we notice the following:

I/O on the client LPAR would hang for approximately 5 minutes. Our test was simple enough, create a file in a file system. Eventually the I/O would continue after the 5 minute delay.

The VIOS became sluggish and took some time to respond. The lspath command would take a very long time to return (or in some cases it would never return). During one test, the VIOS actually hung and had to be restarted (however we were not able to reproduce this issue again).

When we reconnected both the cables, the paths would recover on the first VIOS after another 5 minutes. On the client however, it took roughly 20 minutes for the paths to recover. And yes, the hcheck_interval was set to 60 on all disks.

What's even more puzzling was that if we simply rebooted the first VIOS, everything worked as expected i.e. the client LPARs were not impacted, I/O continued as normal and when the first VIOS was back up, the paths on the client LPARs recovered quickly.

After doing some research, we discovered the following post on the IBM developerWorks AIX forum:

http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14472352&tstart=0

This post highlighted a number of things we needed to check and also confirmed several decisions we’d made during the design process, such as SDDPCM not being supported with DS5020 storage and VIOS (this was good as some people were starting to believe we should have installed SDDPCM to resolve this problem. I’d only be happy to do this if it was a supported combination and it’s not).

Finally we found the following IBM tech note that related directly to our issue.

IZ66020: ACTIVE/PASSIVE PCM CONTROLLER HCHECK SUPPORT
https://www-304.ibm.com/support/docview.wss?uid=isg1IZ66020

The following statement seemed to match our exact problem.

"For active/passive storage device, such as DS3K, DS4K, or DS5K if complete access is lost to the storage device, then it may take greater than 5 minutes to fail I/O. This feature is for Active/Passive storage devices, which are running with the AIX Default A/P PCM. This includes DS3K, DS4K, and DS5K family of devices.”

The new feature was described as follows.

“Added feature which health checks controllers, when an enabled path becomes unavailable, due to transport problems. By default this feature is DISABLED. To enable this feature set the following ODM attributes for the active/passive storage device. Enabling this feature, results in faster I/O failure times.

cntl_delay_time:
Is the amount of time in seconds the storage device's controller(s) will be health checked after a transport failure. At the end of this period, if no paths are detected as good, then all pending and subsequent I/O to the device will be failed, until the device health checker detects a failed path has returned.

cntl_hcheck_int:
The first controller health check will only be issued after a storage fabric transport failure had been detected. cntl_hcheck_int is the amount of time in seconds, which the next controller health check command will be issued. This value must be less than the cntl_delay_time (unless set to "0", disabled).

If you wish to allow the storage device 30 seconds to come back on the fabric (after leaving the fabric), then you can set cntl_delay_time=30 and cntl_hcheck_int=2.

The device, /dev/hdisk#, must not be in use, when setting the ODM values (or the chdev "-P" option must be used, which requires a reboot).

CAUTION: There are cases where the storage device may reboot both of the controllers and become inaccessible for a period of time. If the controller health check sequence is enabled, then this may result in an I/O failure. It is recommended to to make sure you have an mirrored volume
to failover to, if you are running with controller health check enabled (especially with under 60 second cntl_delay_time). “

And as I suspected the issue was related to the type of storage we were using. It appears the I/O delay was attributed to the following attributes on the DS5020 hdisks on the VIOS:

cntl_delay_time 0 Controller Delay Time True
cntl_hcheck_int 0 Controller Health Check Interval True

Based on the tech note, I attempted several tests with various values for both parameters e.g.

$ chdev -dev hdiskX -attr cntl_delay_time=30
$ chdev -dev hdiskX -attr cntl_hcheck_int=2

After making the changes to the hdisks on all VIOS, I performed the same test i.e. disconnected BOTH fibre cables from the first VIOS and continued to write a file to a file system on the client LPARs. By modifying these values on all the DS5020 disks, on all the VIO servers, the I/O delay was reduced to seconds rather than five minutes!

The following attributes were used for the hdisks and adapters in the final configuration.

On the VIO servers:

$ lsdev -type disk | grep DS5020

hdisk3 Available MPIO DS5020 Disk

hdisk4 Available MPIO DS5020 Disk

hdisk5 Available MPIO DS5020 Disk

hdisk6 Available MPIO DS5020 Disk

hdisk7 Available MPIO DS5020 Disk

hdisk8 Available MPIO DS5020 Disk

hdisk9 Available MPIO DS5020 Disk

hdisk10 Available MPIO DS5020 Disk

hdisk11 Available MPIO DS5020 Disk

hdisk12 Available MPIO DS5020 Disk

hdisk13 Available MPIO DS5020 Disk

hdisk14 Available MPIO DS5020 Disk

hdisk15 Available MPIO DS5020 Disk

hdisk16 Available MPIO DS5020 Disk

hdisk17 Available MPIO DS5020 Disk

hdisk18 Available MPIO DS5020 Disk

hdisk19 Available MPIO DS5020 Disk

hdisk20 Available MPIO DS5020 Disk

hdisk21 Available MPIO DS5020 Disk

hdisk22 Available MPIO DS5020 Disk

hdisk23 Available MPIO DS5020 Disk

hdisk24 Available MPIO DS5020 Disk

hdisk25 Available MPIO DS5020 Disk

hdisk26 Available MPIO DS5020 Disk

hdisk27 Available MPIO DS5020 Disk

hdisk28 Available MPIO DS5020 Disk

hdisk29 Available MPIO DS5020 Disk

hdisk30 Available MPIO DS5020 Disk

hdisk31 Available MPIO DS5020 Disk

hdisk32 Available MPIO DS5020 Disk

$ lsdev -dev hdisk3 -attr

attribute value description user_settable

PCM PCM/friend/otherapdisk Path Control Module False

PR_key_value none Persistant Reserve Key Value True

algorithm fail_over Algorithm True

autorecovery no Path/Ownership Autorecovery True

clr_q no Device CLEARS its Queue on error True

cntl_delay_time 15 Controller Delay Time True

cntl_hcheck_int 2 Controller Health Check Interval True

dist_err_pcnt 0 Distributed Error Percentage True

dist_tw_width 50 Distributed Error Sample Time True

hcheck_cmd inquiry Health Check Command True

hcheck_interval 60 Health Check Interval True

hcheck_mode nonactive Health Check Mode True

location Location Label True

lun_id 0x11000000000000 Logical Unit Number ID False

lun_reset_spt yes LUN Reset Supported True

max_retry_delay 60 Maximum Quiesce Time True

max_transfer 0x40000 Maximum TRANSFER Size True

node_name 0x20040080e5187564 FC Node Name False

pvid 00f6482f7869e92d0000000000000000 Physical volume identifier False

q_err yes Use QERR bit True

q_type simple Queuing TYPE True

queue_depth 24 Queue DEPTH True

reassign_to 120 REASSIGN time out value True

reserve_policy no_reserve Reserve Policy True

rw_timeout 30 READ/WRITE time out value True

scsi_id 0x11600 SCSI ID False

start_timeout 60 START unit time out value True

unique_id 3E21360080E50001875DE000005224D2CD63B0F1814 FAStT03IBMfcp Unique device identifier False

ww_name 0x20150080e5187564 FC World Wide Name False

$ lsdev -dev fscsi0 -attr

attribute value description user_settable

attach switch How this adapter is CONNECTED False

dyntrk yes Dynamic Tracking of FC Devices True

fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True

scsi_id 0x11400 Adapter SCSI ID False

sw_fc_class 3 FC Class for Fabric True

$ r oem

oem_setup_env

# manage_disk_drivers

Device Present Driver Driver Options

2810XIV AIX_AAPCM AIX_AAPCM,AIX_non_MPIO

DS4100 AIX_APPCM AIX_APPCM,AIX_fcparray

DS4200 AIX_APPCM AIX_APPCM,AIX_fcparray

DS4300 AIX_APPCM AIX_APPCM,AIX_fcparray

DS4500 AIX_APPCM AIX_APPCM,AIX_fcparray

DS4700 AIX_APPCM AIX_APPCM,AIX_fcparray

DS4800 AIX_APPCM AIX_APPCM,AIX_fcparray

DS3950 AIX_APPCM AIX_APPCM

DS5020 AIX_APPCM AIX_APPCM

DS5100/DS5300AIX_APPCM AIX_APPCM AIX_APPCM

DS3500 AIX_APPCM AIX_APPCM

Usage :

manage_disk_drivers [-l]

manage_disk_drivers -d device -o driver_option

For entries with multiple model names use the first one listed.

Ex. DS5100/DS5300 use DS5100.

manage_disk_drivers –h

# mpio_get_config -Av

Frame id 0:

Storage Subsystem worldwide name: 609e50018345de00004da7998

Controller count: 2

Partition count: 1

Partition 0:

Storage Subsystem Name = 'MyApp-DS5020'

hdisk LUN # Ownership User Label

hdisk3 17 B (preferred) LPAR1

hdisk4 18 A (preferred) LPAR1datavg

hdisk5 19 A (preferred) LPAR1appvg

hdisk6 16 B (preferred) LPAR2

hdisk7 20 A (preferred) LPAR2datavg

hdisk8 21 B (preferred) LPAR2data3vg

hdisk9 15 A (preferred) LPAR3

hdisk10 14 B (preferred) LPAR4

hdisk11 22 B (preferred) LPAR5

hdisk12 23 A (preferred) LPAR6

hdisk13 13 B (preferred) LPAR6datavg

hdisk14 12 A (preferred) LPAR7

hdisk15 24 B (preferred) LPAR7datavg

hdisk16 25 A (preferred) LPAR8

hdisk17 11 B (preferred) LPAR8datavg

hdisk18 10 A (preferred) LPAR8datavg

hdisk19 9 B (preferred) LPAR8datavg

hdisk20 8 A (preferred) LPAR8datavg

hdisk21 26 A (preferred) LPAR9

hdisk22 7 A (preferred) LPAR9datavg

hdisk23 6 B (preferred) LPAR9datavg

hdisk24 27 B (preferred) LPAR9appvg

hdisk25 28 A (preferred) LPAR9binvg

hdisk26 5 A (preferred) LPAR10

hdisk27 4 B (preferred) LPAR10datavg

hdisk28 29 B (preferred) LPAR10datavg

hdisk29 30 A (preferred) LPAR11

hdisk30 3 A (preferred) LPAR11datavg

hdisk31 32 B (preferred) LPAR12

hdisk32 50 B (preferred) LPAR12datavg

On the VIO clients:

- Changed the following attributes for all virtual SCSI disks to:

# lsattr -El hdisk0

PCM PCM/friend/vscsi Path Control Module False

algorithm fail_over Algorithm True

hcheck_cmd test_unit_rdy Health Check Command True

hcheck_interval 60 Health Check Interval True

hcheck_mode nonactive Health Check Mode True

max_transfer 0x40000 Maximum TRANSFER Size True

pvid 00f6482f7869e92d0000000000000000 Physical volume identifier False

queue_depth 24 Queue DEPTH True

reserve_policy no_reserve Reserve Policy True

- Changed the following attributes for all virtual SCSI adapters to:

# lsattr -El vscsi0

vscsi_err_recov fast_fail N/A True

vscsi_path_to 30 Virtual SCSI Path Timeout True

Note: With regard to SDDPCM support for DS5020 with VIOS, I’ve been referring to the following IBM website:

http://www-01.ibm.com/support/docview.wss?uid=ssg1S4000201

The site has a link to the SDDPCM readme file for DS5020 storage (under SDDPCM Package for DS5000):

ftp://ftp.software.ibm.com/storage/subsystem/aix/2.5.2.0/sddpcm.readme.2.5.2.0.txt

The readme states:

Note: VIOS is not supported with SDDPCM on DS4000/DS5000/DS5020/DS3950 subsystem devices.

During our research we had this confirmed by IBM support that, at this time, SDDPCM is not supported
on VIOS with DS5000.

I hope this helps others that may be about to implement this type of storage with a VIOS.

Setting Up A Network To Improve Live Partition Mobility Performance

cggibbo 4 Comments 7,175 Views

I stumbled across this IBM support tech note today. It relates to controlling which network to use during a Live Partition Mobility operation.

http://www-01.ibm.com/support/docview.wss?uid=isg3T7000117&myns=phw&mynp=OCHW182&mync=E

By being given the choice of which network to use we can select network adapters that are more suited to this type of activity i.e. ones that are one or more of the following:

high-speed, under utilized and/or dedicated to non-application data (for example). Using a “high speed” network may improve the performance of an LPM operation and reduce

the overall time required to perform the memory copy between systems.

The diagram in the tech note shows a dedicated Gigabit network between the source and destination VIOS. I found the following point quite interesting:

"NOTE:  Currently transfer of data for 1 partition does not scale much  beyond 1 Gbit speed, 10Gibt does not provide benefit for  parallel migrations."

For those of us who were planning on using their 10GB network adapters for LPM traffic (like I was), we may not need bother at the moment.

I wonder if this will change in the future?

Using NIM to distribute files to your AIX servers.

cggibbo Tags: resource files server copy master sync chris blog client aix gibson nim 3 Comments 3,919 Views

At the IBM Technical Symposium in Sydney last week, a person approached me to discuss NIM and some of its capabilities. During the conversation we discussed how NIM could be used to copy files from the NIM master to its NIM clients. I promised to send them some information on how to achieve this ASAP. They were smart and followed up with an email the next day! They’d even tried to configure this on their systems but hit a small problem.

"Hello Chris,

So trying to push out a new netbackup tar file to one of our nim clients (usually we do all sort of stuff via our SSH deployment server). But decided to use NIM.

So I created the resource file_res called netbackup. Where the tar file is placed on the master

# nim -o define -t file_res -a location=/tmp/netbackup -a dest_dir=/tmp/netbackup -a server=master netbackup

# lsnim -l netbackup
netbackup:
class = resources
type = file_res
Rstate = ready for use
prev_state = unavailable for use
location = /tmp/netbackup
alloc_count = 0
server = master
dest_dir = /tmp/netbackup

But for the life of me I cannot find anywhere under NIM to allocate the resource to the client i.e. to push the file out..
I though it would be under install software, but the resource is not listed only lpp_sources.

Do you have any ideas, this one I am stuck on.

Can I actually push files out by themselves? Thanks"

The answer is yes! You can define a file_res resource on the NIM master. From the AIX 7.2 Knowledge Centre:

"A file_res resource is where NIM allows for resource files to be stored on the server. When the resource is allocated to a client, a copy of the directory contents is placed on the client at a location that is specified by the dest_dir attribute."

"-a location=Value Specifies the full path name of the directory on the NIM server. This path is used as a source directory among clients.
-a dest_dir=Value Specifies the full path name of the directory on the NIM client. This path is where the source directory is recursively copied into.

Notes: If the target directory does not exist on the destination machine, the entire source directory contents are copied (including the hidden files in the top-level directory). If the target directory exists on the destination machine, the source directory contents are copied (excluding the hidden files in the top-level directory)."

http://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.install/defining_file_res_resource.htm

Essentially, you can place the files you need to distribute to your NIM clients, into a directory on the NIM master. Then you can create a file_res NIM resource that points to this directory. After that, you can allocate this resource to the NIM client or NIM machine group, and run a NIM customisation operation against the client (or machine group). This will copy the directory and all its files to the NIM client. Pretty cool really!

Here's an example. I want to distribute (copy) all of the files located in /usr/local/etc, to a NIM client. The original (source) directory and files reside on the NIM master.

root@MASTER / # ls –l /usr/local/etc/
total 16
-rw-r--r-- 1 root system 542 Aug 22 09:53 logger.conf
-rw-r--r-- 1 root system 3219 Aug 22 10:41 fugazi.cf

First, I'll create a new file_res NIM resource, called soe_conf.

root@MASTER / # nim -o define -t file_res -a location=/usr/local/etc -a dest_dir=/usr/local/etc -a server=master soe_conf

root@MASTER / # lsnim -t file_res
soe_conf resources file_res

# lsnim -l soe_conf
soe_conf:
class = resources
type = file_res
dest_dir = /usr/local/etc
Rstate = ready for use
prev_state = unavailable for use
location = /usr/local/etc
alloc_count = 0
server = master

Now I can allocate the new resource to the NIM client named AIXmig.

root@MASTER / # nim -o allocate -a file_res=soe_conf AIXmig

root@MASTER / # lsnim -l AIXmig
AIXmig:
class = machines
type = standalone
connect = nimsh
file_res = soe_conf
platform = chrp
netboot_kernel = 64
if1 = VLAN50 AIXmig 0
cable_type1 = N/A
Cstate = ready for a NIM operation
prev_state = not running
Mstate = currently running
cpuid = 00F94F584C00
control = master

Before I customise the client, the /usr/local/etc directory does NOT exist on the NIM client.

root@AIXmig / # ls -ltr /usr/local/etc
ls: 0653-341 The file /usr/local/etc does not exist.

On the NIM master, I issue the "nim -o cust" operation against the NIM client AIXmig.

root@MASTER /# nim -o cust AIXmig

Looking again on the NIM client, I now find the directory and all of its files have been copied to the correct location on the host.

root@AIXmig / # ls –l /usr/local/etc
total 16
-rw-r--r-- 1 root system 542 Aug 22 09:53 logger.conf
-rw-r--r-- 1 root system 3219 Aug 22 10:41 fugazi.cf

The file_res resource, is automatically deallocated from the NIM client.

root@MASTER / # lsnim -l AIXmig
AIXmig:
class = machines
type = standalone
connect = nimsh
platform = chrp
netboot_kernel = 64
if1 = VLAN50 AIXmig 0
cable_type1 = N/A
Cstate = ready for a NIM operation
prev_state = customization is being performed
Mstate = currently running
cpuid = 00F94F584C00
Cstate_result = success

If you’re looking for a fast and cheap way of copying a bunch of files from a central location to one or more servers, then this is a very good option for AIX administrators. It’s like a “poor mans” file collections; similar to what we once had with Cluster System Management (CSM, which is now discontinued) and PowerHA File Collections (but not as powerful or configurable).

Modified by cggibbo

Testing "Scaled Throughput" mode with AIX 7.1 on POWER7.

cggibbo Tags: smt scaled_throughput chris_gibson aix power7 vpm_throughput_mode 3 Comments 18,133 Views

Starting with AIX 6.1 TL08 and AIX 7.1 TL02 there’s a new AIX CPU tuning feature called “Scaled Throughput” mode. This is supported on POWER7 and POWER7+ processors only (do not try this on POWER6!). This new mode has the ability to dispatch workload to more SMT threads per VP, avoiding the need to unfold additional VPs. I’ve heard it described as being more “POWER6 like”. I’m not suggesting that you use this feature. This post simply discusses what this new mode can do.

By default AIX (on POWER7) operates in “Raw Throughput” mode. This mode provides the best performance per thread per core. It offers the best response times. It utilises more cores (VPs) to process a systems workload. By comparison, “Scaled Throughput” provides a greater level of per core throughput (processing) by dispatching more SMT threads on a core. This has the effect of utilising fewer VPs/cores. In this mode, more (or all) SMT threads per core will be utilised before dispatching workload to other VPs/core in the system.

The schedo tuning command can be used to enable the new mode using a new parameter called vpm_throughput_mode. e.g.

# schedo –p –o vpm_throughput_mode=X

This tunable can be set to one of the following values:

0 = Legacy Raw mode (default).

1= Scaled or “Enhanced Raw” mode with a higher threshold than legacy.

2 = Scaled mode, use primary and secondary SMT threads.

4 = Scaled mode, use all four SMT threads.

At this stage, this tunable is not restricted, but if you plan on experimenting with it please be careful; make sure you understand how this tuning may impact your system; always test new tuning in a non-production environment first!

I performed a couple of quick tests today, just to see what impact tuning the parameter would have on an AIX 7.1 TL2 system.

aixlpar1 : / # oslevel -s

7100-02-01-1245

aixlpar1 : / # lsconf | grep Mode

System Model: IBM,9119-FHB

Processor Implementation Mode: POWER 7

I started some CPU intensive workload.

Name PID CPU% PgSp Owner CliV2 0 WPAR Total 0

ncpu 4718826 15.7 108K db SerV3 0 Press: "h"-help

ncpu 7143668 15.7 108K db CliV3 0 "q"-quit

ncpu 8126488 15.7 108K db

ncpu 5832920 15.6 108K db

The vpm_throughput_mode parameter was left at the default value (0).

# schedo -a | grep vpm_throughput_mode

vpm_throughput_mode = 0

As expected, the workload was evenly dispatched across each of the primary SMT threads of the 4 VPs assigned to the partition i.e. logical CPU 0, 4, 8 and 12. None of the secondary or tertiary SMT threads were active. This is the default mode and will provide the greatest raw throughput (performance) per VP (as there’s no overhead associated with enabling secondary or tertiary SMT threads).

Topas Monitor for host:aixlpar1 EVENTS/QUEUES FILE/TTY

Fri Dec 21 11:06:34 2012 Interval:2 Cswitch 184 Readch 864

Syscall 299 Writech 651

CPU User% Kern% Wait% Idle% Physc Reads 111 Rawin 0

0 99.7 0.3 0.0 0.0 0.63 Writes 1 Ttyout 233

1 0.2 0.5 0.0 99.3 0.12 Forks 0 Igets 0

2 0.0 0.0 0.0 100.0 0.12 Execs 0 Namei 9

3 0.0 0.0 0.0 100.0 0.12 Runqueue 4.00 Dirblk 0

4 100.0 0.0 0.0 0.0 0.63 Waitqueue 0.0

5 0.0 0.0 0.0 100.0 0.12 MEMORY

6 0.0 0.0 0.0 100.0 0.12 PAGING Real,MB 4096

7 0.0 0.0 0.0 100.0 0.12 Faults 0 % Comp 22

8 100.0 0.0 0.0 0.0 0.63 Steals 0 % Noncomp 2

9 0.0 0.0 0.0 100.0 0.12 PgspIn 0 % Client 2

10 0.0 0.0 0.0 100.0 0.12 PgspOut 0

11 0.0 0.0 0.0 100.0 0.12 PageIn 0 PAGING SPACE

12 100.0 0.0 0.0 0.0 0.63 PageOut 0 Size,MB 2048

13 0.0 0.0 0.0 100.0 0.12 Sios 0 % Used 0

14 0.0 0.0 0.0 100.0 0.12 % Free 100

15 0.0 0.0 0.0 100.0 0.12 NFS (calls/sec)

Next, we enabled scaled throughput mode (2). The workload slowly migrated to logical CPUs 4, 5, 8 and 9. So now only two primary SMT threads were active (lcpu’s 4 and 8) and two secondary threads were active (lcpu’s 5 and 9). All the processing was being performed by fewer VPs (almost like POWER6).

# schedo -p -o vpm_throughput_mode=2

Topas Monitor for host:aixlpar1 EVENTS/QUEUES FILE/TTY

Fri Dec 21 11:07:36 2012 Interval:2 Cswitch 179 Readch 935

Syscall 301 Writech 794

CPU User% Kern% Wait% Idle% Physc Reads 112 Rawin 0

0 14.1 60.3 0.0 25.6 0.00 Writes 2 Ttyout 304

1 5.7 32.8 0.0 61.5 0.00 Forks 0 Igets 0

2 0.0 1.6 0.0 98.4 0.00 Execs 0 Namei 10

3 0.0 3.0 0.0 97.0 0.00 Runqueue 4.00 Dirblk 0

4 100.0 0.0 0.0 0.0 0.47 Waitqueue 0.0

5 100.0 0.0 0.0 0.0 0.47 MEMORY

6 0.0 0.0 0.0 100.0 0.03 PAGING Real,MB 4096

7 0.0 0.0 0.0 100.0 0.03 Faults 0 % Comp 22

8 100.0 0.0 0.0 0.0 0.47 Steals 0 % Noncomp 2

9 100.0 0.0 0.0 0.0 0.47 PgspIn 0 % Client 2

10 0.0 0.0 0.0 100.0 0.03 PgspOut 0

11 0.0 0.0 0.0 100.0 0.03 PageIn 0 PAGING SPACE

12 0.0 50.6 0.0 49.4 0.00 PageOut 0 Size,MB 2048

15 0.0 13.3 0.0 86.7 0.00 Sios 0 % Used 0

14 0.0 0.8 0.0 99.2 0.00 % Free 100

15 0.0 0.8 0.0 99.2 0.00 NFS (calls/sec)

And finally, I tried scaled mode with all four SMT threads (4). All of the workload was migrated to a single VP but all 4 SMT threads were being utilised (primary SMT thread lcpu 0, secondary/tertiary SMT threads 1, 2 &3). This mode offers lower overall core consumption but has the (possibly negative) side effect of enabling more SMT threads on a single VP/core, which may not perform as well as the same workload evenly dispatched to 4 individual VPs/cores (on primary SMT threads).

# schedo -p -o vpm_throughput_mode=4

Topas Monitor for host:aixlpar1 EVENTS/QUEUES FILE/TTY

Fri Dec 21 11:08:30 2012 Interval:2 Cswitch 199 Readch 385

Syscall 149 Writech 769

CPU User% Kern% Wait% Idle% Physc Reads 2 Rawin 0

0 99.7 0.3 0.0 0.0 0.25 Writes 1 Ttyout 375

1 99.9 0.1 0.0 0.0 0.25 Forks 0 Igets 0

2 100.0 0.0 0.0 0.0 0.25 Execs 0 Namei 6

3 100.0 0.0 0.0 0.0 0.25 Runqueue 4.00 Dirblk 0

4 0.0 48.8 0.0 51.2 0.00 Waitqueue 0.0

5 0.0 5.5 0.0 94.5 0.00 MEMORY

6 0.0 3.6 0.0 96.4 0.00 PAGING Real,MB 4096

7 0.0 3.3 0.0 96.7 0.00 Faults 0 % Comp 22

8 0.0 77.1 0.0 22.9 0.00 Steals 0 % Noncomp 2

9 0.0 52.9 0.0 47.1 0.00 PgspIn 0 % Client 2

10 0.0 44.8 0.0 55.2 0.00 PgspOut 0

11 0.0 38.5 0.0 61.5 0.00 PageIn 0 PAGING SPACE

12 0.0 51.7 0.0 48.3 0.00 PageOut 0 Size,MB 2048

13 0.0 8.9 0.0 91.1 0.00 Sios 0 % Used 0

14 0.0 7.0 0.0 93.0 0.00 % Free 100

15 0.0 7.4 0.0 92.6 0.00 NFS (calls/sec)

For more information on “Scaled Throughput” mode, take a look at the following presentation:

http://t.co/jVtJ40Ty

Using a post-migration script with nimadm

cggibbo Tags: script aix nim post-migration chris_gibson nimadm 3 Comments 19,884 Views

I’ve received a couple of requests for an example of using a post migration script with nimadm. What follows is a simple example of using such a resource with NIM. If you are not familiar with the nimadm tool then perhaps you’d like to start first by reading my article on using nimadm to migrate to AIX 6.1.

The nimadm utility can perform both pre and post migration tasks. This is accomplished by running NIM scripts either before or after a migration. The tool accepts the following flags for pre and post migration script resources:

-a PreMigrationScript Specifies the pre-migration NIM script resource.

-z PostMigrationScript Specifies the post-migration NIM script resource.

pre-migration

This script resource that is run on the NIM master, but in the environment of the client's alt_inst file system that is mounted on the master (this is done by using the chroot command). This script is run before the migration begins.

post-migration

This script resource is similar to the pre-migration script, but it is executed after the migration is complete.

We are going to focus on post-migration only, although the configuration is the same for both.

In this example I need to uninstall and install a 3^rd party device fileset for a storage device. I need to perform this task as part of the migration process. To protect the innocent, I have not named the storage vendor in this post. But I will say that it was not IBM storage we are dealing with in this case.

Before we start, first we collect all the necessary device filesets that provide support for this type of storage on AIX. We place them into a local directory on my NIM master. Along with the software, I also place a copy of my NIM script in the same directory on the NIM master. The script name is XYZpost.ksh.

root@nim1 : /usr/local/XYZ # ls -ltr

total 544

-r-xr-xr-x 1 root system 51200 Mar 11 2011 MPIO_1001I

-r-xr-xr-x 1 root system 51200 Mar 11 2011 MPIO_1002U

-r-xr-xr-x 1 root system 51200 Mar 11 2011 MPIO_1003U

-r-xr-xr-x 1 root system 51200 Mar 11 2011 MPIO_1004U

-r-xr-xr-x 1 root system 51200 May 18 16:39 MPIO_1005U

-r-xr-xr-x 1 root system 715 May 24 16:57 XYZpost.ksh

-rw-r--r-- 1 root system 2310 May 25 14:57 .toc

The contents of my script are simple. This script will de-install the old device fileset and then immediately install the latest version of the VendorXYZ’s device fileset. The script will then change the attributes for the vendor’s storage to more appropriate default values.

#!/usr/bin/ksh

echo “Uninstalling XYZ fileset: devices.fcp.disk.VendorXYZ.mpio.rte.”

installp -u devices.fcp.disk.VendorXYZ.mpio.rte

echo Return code is: $?

echo echo “Installing XYZ ODM fileset: devices.fcp.disk.VendorXYZ.mpio.rte.”

cd /usr/local/XYZ/

installp -aXd . devices.fcp.disk.VendorXYZ.mpio.rte

echo Return code is: $?

echo “Changing default ODM (PdAt) attributes for VendorXYZ storage.”

set -x

odmget -q uniquetype="disk/fcp/xyzmpio and attribute=queue_depth" PdAt |sed s/deflt\ =\ \"2\"/deflt\ =\ \"32\"/g >/usr/local/XYZ/chg_pdat_qd

odmchange -o PdAt -q uniquetype="disk/fcp/xyzmpio and attribute=queue_depth" /usr/local/XYZ/chg_pdat_qd

odmget -q uniquetype="disk/fcp/xyzmpio and attribute=reserve_policy" PdAt |sed s/deflt\ =\ \"single_path\"/deflt\ =\ \"no_reserve\"/g >/usr/local/XYZ/chg_pdat_rp

odmchange -o PdAt -q uniquetype="disk/fcp/xyzmpio and attribute=reserve_policy" /usr/local/XYZ/chg_pdat_rp

odmget -q uniquetype="PCM/friend/VendorXYZ and attribute=algorithm" PdAt |sed s/deflt\ =\ \"fail_over\"/deflt\ =\ \"round_robin\"/g >/usr/local/XYZ/chg_pdat_al

odmchange -o PdAt -q uniquetype="PCM/friend/VendorXYZ and attribute=algorithm" /usr/local/XYZ/chg_pdat_al

set +x

cd /

At this point I copy the same directory and all of its contents to the NIM client.

root@nim1 : /usr/local # scp –pr XYZ lparaix01:/usr/local/

…etc…

lparaix01 : /usr/local/XYZ # ls -ltr

total 0

-r-xr-xr-x 1 root system 51200 Mar 11 2011 MPIO_1001I

-r-xr-xr-x 1 root system 51200 Mar 11 2011 MPIO_1002U

-r-xr-xr-x 1 root system 51200 Mar 11 2011 MPIO_1003U

-r-xr-xr-x 1 root system 51200 Mar 11 2011 MPIO_1004U

-r-xr-xr-x 1 root system 51200 May 18 16:39 MPIO_1005U

-r-xr-xr-x 1 root system 715 May 24 16:57 XYZpost.ksh

-rw-r--r-- 1 root system 2310 May 25 14:57 .toc

Make sure that any scripts you write for use with nimadm start with an appropriate ‘hashbang’ to announce it is a shell script and the shell that must be used to execute it e.g. #!/usr/bin/ksh. If you forget to do this nimadm will fail to execute your script and will report an error message similar to the following:

+-----------------------------------------------------------------------------+

Executing nimadm phase 7.

+-----------------------------------------------------------------------------+

Executing user chroot script /usr/local/XYZ/XYZpost.ksh.

/lparaix01_alt/alt_inst/tmp/.alt_mig_chroot_script.11731036: Cannot run a file that does not have a valid format.

The next step is to define the script as a NIM resource so that nimadm can call the resource during the migration process. I’ve decided to call this new NIM resource, XYZPOST.

This is easily achieved using smit nim_mkres:

root@nim1 : / # smit nim_mkres

| script = an executable file which is executed on a client |

Define a Resource

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Resource Name [XYZPOST]

* Resource Type script

* Server of Resource [master] +

* Location of Resource [/usr/local/XYZ/XYZpost.ksh] /

We can confirm that the NIM script resource is now available using the lsnim command.

root@nim1 : / # lsnim -t script

XYZPOST resources script

root@nim1 : / # lsnim -l XYZPOST

XYZPOST:

class = resources

type = script

Rstate = ready for use

prev_state = unavailable for use

location = /usr/local/XYZ/XYZpost.ksh

alloc_count = 0

server = master

Now that the script is in place, and defined to NIM, we are ready to test it. We will migrate the system from AIX 5.3 to AIX 6.1 using nimadm. Once the migration phase is complete (phases 1 to 6), the post-migration script will be executed in the NIM clients nimadm (chroot) environment on the NIM master. Once this is finished the NIM clients data is synced back to the NIM clients alternate disk and the boot image is created. The migration process is then complete.

We add the –z flag to our nimadm command line options to specify the post migration resource.

In normal operation we would simply let nimadm run all phases in sequence with the following command.

root@nim1 : / # nimadm -j nimadmvg -c lparaix01 -s spotaix610605 -l lpp_sourceaix610605 -d hdisk2 -z XYZPOST -Y

In our case we would like to test the post-migration script phase as a separate step.

We will start by running only the first 6 phases. We will stop at the completion of phase 6 and run phase 7 (post-migration script) manually, before moving on to phases 8 through 12.

You’ll observe that we have specified the –z option with the name of the NIM script resource and the –P flag with phases 1 to 6 only.

root@nim1 : / # nimadm -j nimadmvg -c lparaix01 -s spotaix610605 -l lpp_sourceaix610605 -d hdisk2 -z XYZPOST -P1,2,3,4,5,6 -Y

Initializing the NIM master.

Initializing NIM client lparaix01.

Verifying alt_disk_migration eligibility.

Initializing log: /var/adm/ras/alt_mig/lparaix01_alt_mig.log

Starting Alternate Disk Migration.

+-----------------------------------------------------------------------------+

Executing nimadm phase 1.

+-----------------------------------------------------------------------------+

Cloning altinst_rootvg on client, Phase 1.

Client alt_disk_install command: alt_disk_copy -j -M 6.1 -P1 -d "hdisk2"

Calling mkszfile to create new /image.data file.

...etc...

Filesets processed: 1 of 2 (Total time: 3 secs).

installp: COMMITTING software for:

X11.compat.lib.X11R6_motif 6.1.6.15

Finished processing all filesets. (Total time: 4 secs).

...etc...

install_all_updates: Checking for recommended maintenance level 6100-06.

install_all_updates: Executing /usr/bin/oslevel -rf, Result = 6100-06

install_all_updates: Verification completed.

install_all_updates: Log file is /var/adm/ras/install_all_updates.log

install_all_updates: Result = SUCCESS

Known Recommended Maintenance Levels

------------------------------------

Restoring device ODM database.

root@nim1 : / #

Phase 6 has completed successfully. The NIM clients rootvg data has been migrated from AIX 5.3 to 6.1 on the NIM master. The data has not yet been synced back to the NIM client.

At this stage we can now run phase 7 separately and ensure that it performs the required task. We expect it will de-install the device fileset, install the latest version and change the ODM default attributes for the device type. Again, you’ll notice that we specify the –P flag for phase 7 only.

Let’s test it.

root@nim1 : / # nimadm -j nimadmvg -c lparaix01 -s spotaix610605 -l lpp_sourceaix610605 -d hdisk2 -z XYZPOST -P7 -Y

Initializing the NIM master.

Initializing NIM client lparaix01.

Initializing log: /var/adm/ras/alt_mig/lparaix01_alt_mig.log

Starting Alternate Disk Migration.

+-----------------------------------------------------------------------------+

Executing nimadm phase 7.

+-----------------------------------------------------------------------------+

Executing user chroot script /usr/local/XYZ/XYZpost.ksh.

Uninstalling XYZ ODM fileset: devices.fcp.disk.VendorXYZ.mpio.rte

+-----------------------------------------------------------------------------+

Pre-deinstall Verification...

+-----------------------------------------------------------------------------+

Verifying selections...done

Verifying requisites...done

Results...

FILESET STATISTICS

------------------

1 Selected to be deinstalled, of which:

1 Passed pre-deinstall verification

----

1 Total to be deinstalled

SUCCESSES

---------

Filesets listed in this section passed pre-deinstall verification

and will be removed.

Selected Filesets

-----------------

devices.fcp.disk.VendorXYZ.mpio.rte 1.2.0.0 # AIX MPIO Support for VendorXYZ Disk

<< End of Success Section >>

+-----------------------------------------------------------------------------+

Deinstalling Software...

+-----------------------------------------------------------------------------+

installp: DEINSTALLING software for:

devices.fcp.disk.VendorXYZ.mpio.rte 1.2.0.0

...etc...

Installation Summary

--------------------

Name Level Part Event Result

-------------------------------------------------------------------------------

devices.fcp.disk.VendorXYZ.mpio 1.2.0.0 USR DEINSTALL SUCCESS

Return code is: 0

Installing XYZ ODM fileset: devices.fcp.disk.VendorXYZ.mpio.rte

+-----------------------------------------------------------------------------+

Pre-installation Verification...

+-----------------------------------------------------------------------------+

Verifying selections...done

Verifying requisites...done

Results...

FILESET STATISTICS

------------------

1 Selected to be installed, of which:

1 Passed pre-installation verification

----

1 Total to be installed

SUCCESSES

---------

Filesets listed in this section passed pre-installation verification

and will be installed.

Selected Filesets

-----------------

devices.fcp.disk.VendorXYZ.mpio.rte 1.4.0.0 # AIX MPIO Support for VendorXYZ Disk

<< End of Success Section >>

+-----------------------------------------------------------------------------+

Installing Software...

+-----------------------------------------------------------------------------+

installp: APPLYING software for:

devices.fcp.disk.VendorXYZ.mpio.rte 1.4.0.0

. . . . . << Copyright notice for devices.fcp.disk.VendorXYZ.mpio >> . . . . . . .

...etc...

Installation Summary

--------------------

Name Level Part Event Result

-------------------------------------------------------------------------------

devices.fcp.disk.VendorXYZ.mpio 1.4.0.0 USR APPLY SUCCESS

Return code is: 0

Changing default ODM (PdAt) attributes for VendorXYZ storage

+ odmget -q uniquetype=disk/fcp/xyzmpio and attribute=queue_depth PdAt

+ sed s/deflt = "2"/deflt = "32"/g

+ 1> /usr/local/XYZ/chg_pdat_qd

+ odmchange -o PdAt -q uniquetype=disk/fcp/xyzmpio and attribute=queue_depth /usr/local/XYZ/chg_pdat_qd

+ odmget -q uniquetype=disk/fcp/xyzmpio and attribute=reserve_policy PdAt

+ sed s/deflt = "single_path"/deflt = "no_reserve"/g

+ 1> /usr/local/XYZ/chg_pdat_rp

+ odmchange -o PdAt -q uniquetype=disk/fcp/xyzmpio and attribute=reserve_policy /usr/local/XYZ/chg_pdat_rp

+ odmget -q uniquetype=PCM/friend/VendorXYZ and attribute=algorithm PdAt

+ sed s/deflt = "fail_over"/deflt = "round_robin"/g

+ 1> /usr/local/XYZ/chg_pdat_al

+ odmchange -o PdAt -q uniquetype=PCM/friend/VendorXYZ and attribute=algorithm /usr/local/XYZ/chg_pdat_al

root@nim1 : / #

Great news! Our script has worked as expected. The old fileset was de-installed, the new fileset was installed and the PdAt default attributes were changed successfully.

Note: You can also review the post migration script output at a later date if you wish. All nimadm activities are logged, on the NIM master, to /var/adm/ras/alt_mig/NIMclientname_alt_mig.log (where NIMclientname is the name of the NIM client being migrated by nimadm).

With regard to nimadm log files, please be aware that if you choose to run nimadm in phases (as I’ve shown in this example) that each run will generate a new log file. So in my case, when I ran phases 1 to 6, this created a log file named lparaix01_alt_mig.log. When I ran phase 7, the original log file was moved to lparaix01_alt_mig.log.prev. A new log file was created and used for phase 7. Then when I ran phases 8 to 12, the phase 7 log file was moved to lparaix01_alt_mig.log.prev and a new log file was used for phases 8-12. For this reason you may want to backup each log file to a unique file name as you execute each phase group, so that you do not lose any of the information logged to the .log or .log.prev files.

Now we can complete the rest of the migration and execute the remaining phases, 8 through 12.

root@nim1 : / # nimadm -j nimadmvg -c lparaix01 -s spotaix610605 -l lpp_sourceaix610605 -d hdisk2 -z XYZPOST -P8,9,10,11,12 -Y

Initializing the NIM master.

Initializing NIM client lparaix01.

Initializing log: /var/adm/ras/alt_mig/lparaix01_alt_mig.log

Starting Alternate Disk Migration.

+-----------------------------------------------------------------------------+

Executing nimadm phase 8.

+-----------------------------------------------------------------------------+

Creating client boot image.

bosboot: Boot image is 46989 512 byte blocks.

Writing boot image to client's alternate boot disk hdisk2.

+-----------------------------------------------------------------------------+

Executing nimadm phase 9.

+-----------------------------------------------------------------------------+

Adjusting client file system sizes ...

Adjusting size for /

Adjusting size for /admin

Adjusting size for /home

Adjusting size for /opt

Adjusting size for /tmp

Adjusting size for /usr

Expanding /alt_inst/usr client filesystem.

Filesystem size changed to 4325376

Adjusting size for /usr/local

Adjusting size for /usr/local/chksys

Adjusting size for /var

Adjusting size for /var/log

Adjusting size for /var/log/audit

Adjusting size for /var/tsm/log

Syncing cache data to client ...

+-----------------------------------------------------------------------------+

Executing nimadm phase 10.

+-----------------------------------------------------------------------------+

Unmounting client mounts on the NIM master.

forced unmount of /lparaix01_alt/alt_inst/var/log/audit

forced unmount of /lparaix01_alt/alt_inst/var/log

forced unmount of /lparaix01_alt/alt_inst/var

forced unmount of /lparaix01_alt/alt_inst/usr/local

forced unmount of /lparaix01_alt/alt_inst/usr

forced unmount of /lparaix01_alt/alt_inst/tmp

forced unmount of /lparaix01_alt/alt_inst/opt

forced unmount of /lparaix01_alt/alt_inst/home

forced unmount of /lparaix01_alt/alt_inst/admin

forced unmount of /lparaix01_alt/alt_inst

Removing nimadm cache file systems.

Removing cache file system /lparaix01_alt/alt_inst

Removing cache file system /lparaix01_alt/alt_inst/admin

Removing cache file system /lparaix01_alt/alt_inst/home

Removing cache file system /lparaix01_alt/alt_inst/opt

Removing cache file system /lparaix01_alt/alt_inst/tmp

Removing cache file system /lparaix01_alt/alt_inst/usr

Removing cache file system /lparaix01_alt/alt_inst/usr/local

Removing cache file system /lparaix01_alt/alt_inst/var

Removing cache file system /lparaix01_alt/alt_inst/var/log

Removing cache file system /lparaix01_alt/alt_inst/var/log/audit

+-----------------------------------------------------------------------------+

Executing nimadm phase 11.

+-----------------------------------------------------------------------------+

Cloning altinst_rootvg on client, Phase 3.

Client alt_disk_install command: alt_disk_copy -j -M 6.1 -P3 -d "hdisk2"

## Phase 3 ###################

Verifying altinst_rootvg...

Modifying ODM on cloned disk.

forced unmount of /alt_inst/var/log/audit

forced unmount of /alt_inst/var/log

forced unmount of /alt_inst/var

forced unmount of /alt_inst/usr/local

forced unmount of /alt_inst/usr

forced unmount of /alt_inst/tmp

forced unmount of /alt_inst/opt

forced unmount of /alt_inst/home

forced unmount of /alt_inst/admin

forced unmount of /alt_inst

Changing logical volume names in volume group descriptor area.

Fixing LV control blocks...

Fixing file system superblocks...

Bootlist is set to the boot disk: hdisk2 blv=hd5

+-----------------------------------------------------------------------------+

Executing nimadm phase 12.

+-----------------------------------------------------------------------------+

Cleaning up alt_disk_migration on the NIM master.

Cleaning up alt_disk_migration on client lparaix01.

root@nim1 : / #

Our nimadm operation is finished. The NIM client is rebooted. We verify it is now running AIX 6.1. And the correct version of the vendor’s device fileset is installed.

lparaix01 : / # oslevel -s

6100-06-05-1115

lparaix01 : / # lslpp -l devices.fcp.disk.VendorXYZ.mpio.rte

Fileset Level State Description

----------------------------------------------------------------------------

Path: /usr/lib/objrepos

devices.fcp.disk.VendorXYZ.mpio.rte

1.4.0.0 COMMITTED AIX MPIO Support for VendorXYZ Disk

lparaix01 : / # lsdev -Cc disk

hdisk1 Available 20-T1-01 VendorXYZ MPIO Disk

hdisk2 Available 20-T1-01 VendorXYZ MPIO Disk

hdisk3 Available 20-T1-01 VendorXYZ MPIO Disk

hdisk4 Available 20-T1-01 VendorXYZ MPIO Disk

And there you have it, an example of using a post migration script with nimadm.

nimadm Command

http://pic.dhe.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.cmds/doc/aixcmds4/nimadm.htm

Can't change max_transfer for a VSCSI hdisk? Check your vtopt devices!

cggibbo Tags: rmdev max_transfer vtopt0 rmvdev aix vios chdev hdisk cd0 virtual_optical vio 3 Comments 19,952 Views

When attempting to change the max_transfer attribute on a VSCSI hdisk, the following error was displayed:

# chdev -l hdisk0 -a max_transfer=0x100000 -P

Method error (/etc/methods/chgdisk):

0514-018 The values specified for the following attributes

are not valid:

max_transfer Maximum TRANSFER Size

The desired value, 0x100000, was an allowed value for this type of device (as shown in the lsattr output below):

# lsattr -El hdisk0

PCM PCM/friend/vscsi Path Control Module False

algorithm fail_over Algorithm True

hcheck_cmd test_unit_rdy Health Check Command True

hcheck_interval 60 Health Check Interval True

hcheck_mode nonactive Health Check Mode True

max_transfer 0x40000 Maximum TRANSFER Size True

pvid 00f6a83aeb1948950000000000000000 Physical volume identifier False

queue_depth 256 Queue DEPTH True

reserve_policy no_reserve Reserve Policy True

# lsattr -Rl hdisk0 -a max_transfer

0x20000

0x40000

0x80000

0x100000

0x200000

0x400000

0x800000

0x1000000

After some discussion with IBM support, it was discovered that the virtual optical device, assigned to this LPAR, was causing the problem. They advised that we should perform the following steps:

1. Remove the cd0 device from the LPAR i.e.

# rmdev –dl cd0

2. Remove the vtopt0 device assigned to the LPAR on the VIOS i.e

$ rmvdev –vtd vtopt0

3. Shutdown the LPAR.

4. Reboot the VIOS.

5. Restart the LPAR.

6. Change the max_transfer value for the hdisk on the LPAR.

Sure enough, after following these steps, we were able to change max_transfer to the desired value!

# chdev -l hdisk0 -a max_transfer=0x100000

hdisk0 changed

# lsattr -El hdisk0

PCM PCM/friend/vscsi Path Control Module False

algorithm fail_over Algorithm True

hcheck_cmd test_unit_rdy Health Check Command True

hcheck_interval 0 Health Check Interval True

hcheck_mode nonactive Health Check Mode True

max_transfer 0x100000 Maximum TRANSFER Size True

pvid 00f6a83aeedf5ffb0000000000000000 Physical volume identifier False

queue_depth 3 Queue DEPTH True

reserve_policy no_reserve Reserve Policy True

The following explanation was given by IBM support:

“The virtual optical device seems to use max_transfer size as 0x40000 (256 KB) internally, which is not allowed to be modified. Virtual optical CD's max_transfer seems to limit vhost adapter's max_transfer size. So if you want to increase max_transfer size for virtual SCSI disks, you should separate them from virtual optical SCSI CD by configuring those two groups under different vhost adapters”

AIX performance tool enhancements - Filtered output and XML reports

cggibbo 3 Comments 8,768 Views

High End POWER7 systems, like the POWER7 795, support configurations of up to 256 physical processors. Most of us can only dream of working on a system of this size. For those of us that are fortunate enough to play with these big boxes, you may find collecting and reviewing CPU related performance data a challenging task.

With 256 physical POWER7 processors, where each processor has 4 threads enabled (SMT 4), there will be up to 1024 logical CPUs active on the system. As a consequence, a lot of performance monitoring data will be generated, which presents challenges to the OS to filter and effectively present relevant performance data to the administrator.

Many AIX performance tools provide per-CPU statistics for the administrator to analyse application and system performance. With the introduction of 1024 logical CPUs on a system, these tools are impacted. All standard AIX performance tools have been modified in AIX 7.1 (and AIX 6.1 TL6) to support AIX partitions running as many as 1024 logical CPUs on a POWER7 system.

The following tools have been modified to support the increase in logical CPUs and now provide options for filtering and sorting data: mpstat, sar and topas.

Both the mpstat and sar commands have been updated to allow for the sorting and filtering of output. The –O flag provides this new feature. The following options can be supplied to the –O flag (from the man page):

-OOptions

Specifies the command option.

-O options=value...

Following are the supported options:

* sortcolumn = Name of the metrics in the mpstat command output

* sortorder = [asc|desc]

* topcount = Number of CPUs to be displayed in the mpstat command sorted output

For example, to see the sorted mpstat output for the cs column you would enter the following command.

# mpstat -d -O sortcolumn=cs 1 3

Another example, to see the list of the top 10 CPUs, you would enter the following command.

# mpstat -a -O sortcolumn=min,sortorder=desc,topcount=10 1 3

Some examples of this new option with the sar command are shown below.

-OOptions

Allows users to specify the command option.

-O options=value...

Following are the supported options:

* sortcolumn = Name of the metrics in the sar command output

* sortorder = [asc|desc]

* topcount = Number of CPUs to be displayed in the sar command sorted output

To display the sorted sar output for the column cswch/s with the -w flag, you would enter the following command:

# sar -w -P ALL -O sortcolumn=cswch/s 1 3

To list the top ten CPUs, sorted on the scall/s column, you would enter the following command:

# sar -c -O sortcolumn=scall/s,sortorder=desc,topcount=10 -P ALL 1 3

Some performance tools have also been enhanced to provide the capability to generate XML reports. The following tools now have this feature: sar, mpstat, vmstat, iostat and lparstat. Specifying the –X flag with these commands will produce XML output. You can specify an output file with the –o flag. If you don’t specify an output file, a default file is generated with the following naming convention, command_DDMMYYHHMM.xml.

Here are some examples of using the –X flag with the updated commands:

# lparstat -X

# sar -X 1 5

# mpstat -X 1 5

# vmstat -X 1 3

# iostat -X 1 3

*** stack smashing detected ***: program terminated

IOT/Abort trap(coredump)

# ls -lr *.xml

-rw-r--r-- 1 root system 50027 Nov 25 22:00 vmstat.xml

-rw-r--r-- 1 root system 50852 Nov 25 21:59 sar_2511102159.xml

-rw-r--r-- 1 root system 44570 Nov 25 21:59 mpstat_2511102159.xml

-rw-r--r-- 1 root system 18926 Nov 25 21:59 lparstat_2511102159.xml

In my tests I noticed a couple of things. First the vmstat output is named vmstat.xml, this doesn’t match the expected naming convention. This could be a bug? Also, the iostat command dumped core. This is meant to be supported so this was totally unexpected. And yes I was running the latest service pack for AIX 7.1 (7100-00-01-1037). This could also be a bug. Anyway, I digress!

Here are some samples of the content of the generated XML files.

# head -20 lparstat_2511102159.xml

<?xml version="1.0" encoding="UTF-8"?>

<Command>lparstat</Command>

1290743958

</TimeStamp>

Thu Nov 25 21:59:18 2010

</TimeStamp>

</ProcessingHeader>

# tail -20 sar_2511102159.xml

</CPUUtil>

</CPUStats>

</SyswideContextSwitch>

</AvgPerProcessorContextSwitch>

</AvgTTYActivity>

</SarAverageData>

</CollectionDataSet>

</PerformanceMeasurement>

# head -40 mpstat_2511102159.xml

<?xml version="1.0" encoding="UTF-8"?>

<Command>mpstat</Command>

1290743975

</TimeStamp>

Thu Nov 25 21:59:35 2010

</TimeStamp>

</ProcessingHeader>

</TaggedNotes>

</HardwareDescription>

</HardwareDescription>

<LogicalPartitionType>Shared</LogicalPartitionType>

<CappedAttribute>Uncapped</CappedAttribute>

</LogicalPartitionConfiguration>

And one last feature that I thought I’d mention. The topas command has been enhanced with two new useful features

· topas panel freezing [ 'Space Bar' is used a toggle for freezing]

Topas Monitor for host:l273pp007 EVENTS/QUEUES FILE/TTY

Thu Nov 25 21:54:56 2010 Interval:FROZEN Cswitch 15.2G Readch 578.1G

Syscall 26.2G Writech 103.4G

CPU User% Kern% Wait% Idle% Physc Entc% Reads 439.4M Rawin 170.6K

Total 37.6 62.4 0.0 0.0 3.10 620.80 Writes 533.6M Ttyout 791.8M

Forks 5996.3K Igets 2658.0K

Network BPS I-Pkts O-Pkts B-In B-Out Execs 7654.3K Namei 2048.0M

Total 19.9K 0 178.5 0 19.9K Runqueue 12.0M Dirblk 18.1M

Waitqueue 17938.1

Disk Busy% BPS TPS B-Read B-Writ MEMORY

Total 0.0 0 0 0 0 PAGING Real,MB 6144

Faults 1753.2M % Comp 29

FileSystem BPS TPS B-Read B-Writ Steals 0 % Noncomp 12

Total 624.7 89.24 624.7 0 PgspIn 0 % Client 11

PgspOut 0

WLM-Class (Active) CPU% Mem% Blk-I/O% PageIn 18.9M PAGING SPACE

System 1 21 0 PageOut 50.4M Size,MB 1536

Default 0 2 0 Sios 47.1M % Used 1

% Free 99

Name PID CPU% PgSp Class NFS (calls/sec)

sendmail 7340250 0.0 924K 52wpar SerV2 0 WPAR Activ 3

cron 7471202 0.0 324K 52wpar CliV2 0 WPAR Total 3

IBM.ERrm 10092604 0.0 1.83M 52wpar SerV3 0 Press: "h"-help

syslogd 7733322 0.0 452K wpar52 CliV3 713.8K "q"-quit

sshd 7864478 0.0 796K System

srcmstr 7929858 0.0 1.10M wpar1

inetd 8126494 0.0 1016K wpar1

init 8192026 0.0 1.05M wpar1

lockd-1 8257544 0.0 1.19M System

pilegc 786456 0.0 640K System

· topas panel scrolling and sorting [PgUp/PgDn keys are be used for scrolling]

DLPAR event history on the HMC

cggibbo Tags: gibson aix lssvcevents dlpar hmc history events chris 2 Comments 3,095 Views

You can use the HMC lssvcevents command to view DLPAR event history. In the following example, I’m looking for “Add processor” related DLPAR events, over the last 30 days, for a specific LPAR.

$ lssvcevents -t console -d 30 | egrep 'Add processor|DLPAR' | grep s82861p22_nim

time=12/01/2016 00:48:33,text=HSCE2209 User name hscroot: DLPAR Add processor resources to partition s82861p22_nim succeeded on managed system Server-8286-42A-SN214F55V

Search for all DLPAR events over the last 30 days.

$ lssvcevents -t console -d 30 | grep -i dlpar

time=12/01/2016 00:52:27,text=HSCE2211 User name hscroot: DLPAR Remove processor resources from partition s82861p22_nim succeeded on managed system Server-8286-42A-SN214F55V

time=12/01/2016 00:48:33,text=HSCE2209 User name hscroot: DLPAR Add processor resources to partition s82861p22_nim succeeded on managed system Server-8286-42A-SN214F55V

HMC Manual Reference Pages - LSSVCEVENTS (1)

http://www.ibm.com/support/knowledgecenter/TI0003N/p8edm/lssvcevents.html

Modified by cggibbo

Hacking cloud-init on AIX.

cggibbo Tags: /etc/hosts update hack resolv.conf chris cloud-init aix gibson hostname 2 Comments 3,351 Views

There may be better ways to do this, and if there are, please let me know. But lately I've been "hacking around" with cloud-init on AIX and trying to make it behave the way I want it to. There were two problems I faced and solved.

My first challenge. The AIX /etc/hosts file isn't updated with the IP address and hostname of the new AIX VM after deployment from PowerVC.

To work-around this niggle*, I added the following short but effective shell script to the Activation Input in the PowerVC GUI.

#!/bin/ksh
myip=`lsattr -El en0 -a netaddr | awk '{print $2}'`
myname=`hostname`
echo Adding $myip $myname to /etc/hosts >> /tmp/mycloud.log
/usr/bin/hostent -a $myip -h $myname
date >> /tmp/mycloud.log

It's ugly but hey, it does the job!

And the second niggle. In my customers lab environment, where there's no DNS at all. Everything is /etc/hosts only. Every time they deployed an AIX VM, there was a significant delay for ssh sessions, even with netsvc.conf set to hosts=local4, the dodgy**, default, resolv.conf (search localdomain) forced DNS first....and ssh connections would hang for a minute or so before a login prompt appeared....waiting on name resolution to complete.

Some people told me that I could change "manage_resolv_conf" to false in my cloud-init config file and this would prevent the resolv.conf file from being managed (over-written) by cloud-init. But changing that option did nothing. And I really didn't want a resolv.conf file at all anyway!

What I really wanted was for cloud-init to deploy the AIX VM and to NOT create an /etc/resolv.conf file. But how? Well, I managed to fudge it***. I made a change to the aix.py python script. With this change, the script now writes out an /etc/resolv.conf.cloud file instead. This works OK.

root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros # cat /etc/resolv.conf.cloud
search localdomain

root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros # mkdir cg
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros # cd cg
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg/
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg # cp ../aix.py .
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg # vi aix.py

Change resolve_conf_fn = "/etc/resolv.conf"
To resolve_conf_fn = "/etc/resolv.conf.cloud"

root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros # diff aix.py /tmp/cg/aix.py
35c35
< resolve_conf_fn = "/etc/resolv.conf.cloud"
---
> resolve_conf_fn = "/etc/resolv.conf"

root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg # python -m compileall .
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg # cp -p * ..
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros/cg # cd ..
root@vm1 /opt/freeware/lib/python2.7/site-packages/cloudinit/distros # rm -r cg

Now the customer can connect to new AIX VMs immediately after deployment.

# ls -ltr /etc/resolv.conf.cloud
-rw-r--r-- 1 root system 19 Aug 03 12:47 /etc/resolv.conf.cloud

It's a hack but it was fun.

And I guess the next time I install the latest release of cloud-init for AIX, I'll need to modify the script again. But I'm OK with this, as I expect the newer release may actually provide me with a fix to each of the problems I faced.

*niggle: To cause slight but persistent annoyance, discomfort, or anxiety.

**dodgy: dishonest or unreliable, of low quality.

***fudge it: put together clumsily. Perhaps an alteration of fadge "make suit, fit".

Modified by cggibbo

AIX LVM Mirroring enhancement for Flash Preferred Read

cggibbo 2 Comments 5,832 Views

As you are probably aware, in a lot of cases, customers have chosen to include IBM flash system storage as the preferred read LVM mirror on AIX. This gives them enhanced I/O read performance and resiliency. An AIX administrator can add the hdisks to the VG, mirror the data and set the LV scheduling policy to parallel/sequential indicating writes will be done to both copies in parallel and reads will come from the first LVM copy which should be on flash. This is discussed at length in section 5.5.2 Implementing preferred read, Preferred read with AIX, of the following Redbook.

http://www.redbooks.ibm.com/abstracts/sg248189.html?Open

I discovered today that both the mklv and chlv commands have recently been enhanced to support this configuration. Both now have an option for setting the preferred read to the required copy of a logical volume (usually on the flash disk).

From the man page for mklv and chlv, I found the following new information.

# man mklv

-R PreferredRead

Sets read preference to the copy of the logical volume. If the -R flag is specified and if the preferred copy is available, the read operation occurs from the preferred copy. If the preferred copy is not available, the read operations follow the scheduling policy of the logical volume. The PreferredRead variable can be set to a value in the range 0 -3. The default value is 0.

# man chlv

-R PreferredRead

Changes preferred read copy of the logical volume. Always reads from the preferred copy if the preferred copy is available. If the preferred copy is not in available then the reads follows the scheduling policy of the logical volume. The PreferredRead variable can be set to a value ranging from 0 to 3. Setting the PreferredRead variable to 0 disables the preferred read copy of the logical volume.

To my surprise, this is available in AIX 7.1 TL3 SP5.

# oslevel -s

7100-03-05-1524

Now I can specify the preferred copy during creation of the logical volume. And lslv will now show me the preferred read copy.

# mklv -tjfs2 -y lv1 -c 2 -R 2 cgvg 10

lv1

# lslv lv1

LOGICAL VOLUME: lv1 VOLUME GROUP: cgvg

LV IDENTIFIER: 00f603cd00004c000000014a0515f597.2 PERMISSION: read/write

VG STATE: active/complete LV STATE: closed/syncd

TYPE: jfs2 WRITE VERIFY: off

MAX LPs: 512 PP SIZE: 32 megabyte(s)

COPIES: 2 SCHED POLICY: parallel

LPs: 10 PPs: 20

STALE PPs: 0 BB POLICY: relocatable

INTER-POLICY: minimum RELOCATABLE: yes

INTRA-POLICY: middle UPPER BOUND: 1024

MOUNT POINT: N/A LABEL: None

DEVICE UID: 0 DEVICE GID: 0

DEVICE PERMISSIONS: 432

MIRROR WRITE CONSISTENCY: on/ACTIVE

EACH LP COPY ON A SEPARATE PV ?: yes

Serialize IO ?: NO

INFINITE RETRY: no PREFERRED READ: 2

DEVICESUBTYPE: DS_LVZ

COPY 1 MIRROR POOL: None

COPY 2 MIRROR POOL: None

COPY 3 MIRROR POOL: None

If I need to change the preferred read copy later, I can use chlv -R. This is a dynamic change.

# chlv -R 1 lv1

# lslv lv1

LOGICAL VOLUME: lv1 VOLUME GROUP: cgvg

LV IDENTIFIER: 00f603cd00004c000000014a0515f597.2 PERMISSION: read/write

VG STATE: active/complete LV STATE: closed/syncd

TYPE: jfs2 WRITE VERIFY: off

MAX LPs: 512 PP SIZE: 32 megabyte(s)

COPIES: 2 SCHED POLICY: parallel

LPs: 10 PPs: 20

STALE PPs: 0 BB POLICY: relocatable

INTER-POLICY: minimum RELOCATABLE: yes

INTRA-POLICY: middle UPPER BOUND: 1024

MOUNT POINT: N/A LABEL: None

DEVICE UID: 0 DEVICE GID: 0

DEVICE PERMISSIONS: 432

MIRROR WRITE CONSISTENCY: on/ACTIVE

EACH LP COPY ON A SEPARATE PV ?: yes

Serialize IO ?: NO

INFINITE RETRY: no PREFERRED READ: 1

DEVICESUBTYPE: DS_LVZ

COPY 1 MIRROR POOL: None

COPY 2 MIRROR POOL: None

COPY 3 MIRROR POOL: None

Previously, to set the preferred read to the flash storage, it would often require you to mirror a volume group to the flash disks, then un-mirror the volume group to make the flash disk the primary LV copy, then mirror the volume group again to make the non-flash disks the secondary LV copy. This is no longer necessary with the new –R option to the chlv command and can be changed as and when necessary.

http://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.cmds3/mklv.htm?lang=en

http://www-01.ibm.com/support/knowledgecenter/ssw_aix_61/com.ibm.aix.cmds1/chlv.htm?lang=en

http://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.cmds3/lslv.htm?lang=en

Modified by cggibbo

AIX Live Update - No Reboot Required! Non-disruptive OS Updates!

cggibbo Tags: chris kernel aix update live gibson 7.2 lku 2 Comments 11,261 Views

AIX Live Update

Starting with AIX Version 7.2, the AIX operating system provides the AIX Live Update function which eliminates downtime associated with patching the AIX operating system. Previous releases of AIX required systems to be rebooted after an interim fix was applied to a running system. This new feature allows workloads to remain active during a Live Update operation and the operating system can use the interim fix immediately without needing to restart the entire system. In the first release of this feature, AIX Live Update will allow customers to install interim fixes (ifixes) only. Ultimately it may be possible to use this function to install AIX Service Packs (SPs) and Technology Levels (TLs) without a reboot.

IBM delivers kernel fixes in the form of ifixes to resolve issues that are reported by customers. If a fix changes the AIX kernel or loaded kernel extensions that cannot be unloaded, the host logical partition (LPAR) must be rebooted. To address this issue, AIX Version 7.1, and earlier, provided concurrent update-enabled ifixes that allowed deployment of some limited kernel fixes to a running LPAR. Unfortunately not all ifixes could be delivered as “concurrent update-enabled”. The AIX Live Update solution is not constrained by the same limitations as in the case of concurrent update enabled ifixes. The AIX 7.2, Live Update feature will allow customers to install ifixes without needing to reboot their AIX systems, avoiding downtime for their mission critical, production workloads.

This article (in the link below) will discuss the high-level concepts relating to AIX Live Updates and then provide a real example of how to use the tool to patch a live AIX system. I was fortunate enough to take part in an Early Ship Program (ESP) for AIX 7.2. During the ESP I had the opportunity to test the AIX Live Update feature. I’ll share my experience using this tool in the example that follows.

https://www.ibm.com/developerworks/community/blogs/cgaix/resource/AIXLiveUpdateblog.pdf

Modified by cggibbo

Restoring a Versioned WPAR with NIM

cggibbo Tags: vwpar gibson chris iv68299 blog wpar mkwpar aix restore volume wparvg group rootvg nim 2 Comments 7,380 Views

I was contacted recently by a customer who was attempting to restore an AIX 5.3 Versioned WPAR (VWPAR) from backup using NIM. The restore worked OK but the data was restored to the wrong volume group!

When the VWPAR was created, the –g option was specified with mkwpar to force the creation of the VWPAR file systems in a separate volume group (named wparvg) rather than the default location of the Global root volume group (rootvg).

# mkwpar -g wparvg -n p8vw2 -B /cg/53gibbo.mksysb -C -O

Running lsvg against wparvg confirmed the file systems were in the right location after creation.

# lsvg -l wparvg

wparvg:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

loglv00 jfs2log 1 1 1 open/syncd N/A

fslv02 jfs2 4 4 1 open/syncd /wpars/p8vw2

fslv03 jfs2 2 2 1 open/syncd /wpars/p8vw2/home

fslv04 jfs2 12 12 1 open/syncd /wpars/p8vw2/opt

fslv05 jfs2 6 6 1 open/syncd /wpars/p8vw2/tmp

fslv06 jfs2 56 56 1 open/syncd /wpars/p8vw2/usr

fslv07 jfs2 12 12 1 open/syncd /wpars/p8vw2/var

Before handing the VWPAR over for production use, the customer wanted to ensure they could successfully backup and recover the VWPAR using NIM. First they took a backup of the VWPAR using NIM. From the NIM master, they created a “savewpar backup image”, as shown below.

Note: The WPAR was already defined as a NIM client. Refer to section “12.2 Adding a System WPAR to NIM” in the “Exploiting IBM AIX Workload Partitions (SG24-7955-00)” Redbook for information on how to perform this task.

# smit nim_mkres

savewpar = a workload partition backup image

Define a Resource

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Resource Name [p8vw2-backup]

* Resource Type savewpar

* Server of Resource [master] +

* Location of Resource [/export/vwpar/p8vw2-backup] /

NFS Client Security Method [] +

NFS Version Access [] +

Comments []

Source for Replication [] +

-OR-

System Backup Image Creation Options:

CREATE system backup image? yes +

WPAR to backup [p8vw2] +

....

# lsnim -l p8vw2

p8vw2:

class = machines

type = wpar

connect = shell

platform = chrp

if1 = 10_1_50 p8vw2 0

mgmt_profile1 = gibopvc1 p8vw2

Cstate = ready for a NIM operation

prev_state = not running

Mstate = currently running

cpuid = 00F94F584C00

Cstate_result = success

# lsnim -t savewpar

p8vw2-backup resources savewpar

In the Global environment, we then stopped and removed the VWPAR (p8vw2).

# stopwpar -F p8vw2

# rmwpar -F p8vw2

Back on the NIM master, we attempted to restore the VWPAR from the recently created backup image (p8vw2-backup).

# smit nim_wpar_create

p8vw2

Create a Managed Workload Partition

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Target Name [p8vw2]

Remain NIM client after install? [yes] +

Specification Resource [] +

WPAR Options

WPAR Name p8vw2

Resource for WPAR Backup Image [p8vw2-backup] +

Resource for System Backup Image [] +

Alternate DEVEXPORTS for installation [] +

Alternate SECATTRS for installation [] +

The restore completed successfully but to our surprise, the VWPAR file systems were in the Global rootvg not wparvg.

# lsvg –l rootvg

rootvg:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

hd5 boot 1 1 1 closed/syncd N/A

hd6 paging 8 8 1 open/syncd N/A

hd8 jfs2log 1 1 1 open/syncd N/A

hd4 jfs2 6 6 1 open/syncd /

hd2 jfs2 35 35 1 open/syncd /usr

hd9var jfs2 7 7 1 open/syncd /var

hd3 jfs2 2 2 1 open/syncd /tmp

hd1 jfs2 1 1 1 open/syncd /home

hd10opt jfs2 5 5 1 open/syncd /opt

hd11admin jfs2 2 2 1 open/syncd /admin

lg_dumplv sysdump 16 16 1 closed/syncd N/A

livedump jfs2 4 4 1 open/syncd /var/adm/ras/livedump

cglv jfs2 100 100 1 open/syncd /cg

fslv02 jfs2 2 2 1 open/syncd /wpars/p8vw2

fslv03 jfs2 1 1 1 open/syncd /wpars/p8vw2/home

fslv04 jfs2 6 6 1 open/syncd /wpars/p8vw2/opt

fslv05 jfs2 3 3 1 open/syncd /wpars/p8vw2/tmp

fslv06 jfs2 28 28 1 open/syncd /wpars/p8vw2/usr

fslv07 jfs2 6 6 1 open/syncd /wpars/p8vw2/var

# lsvg -l wparvg

wparvg:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

loglv00 jfs2log 1 1 1 closed/syncd N/A

We attempted the restore again but this time we explicitly included a WPAR “Specification Resource”. We did this to ensure that the restwpar process was using the correct specification file.

# smit nim_wpar_create

p8vw2

Create a Managed Workload Partition

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Target Name [p8vw2]

Remain NIM client after install? [yes] +

Specification Resource [p8vw2-spec] +

WPAR Options

WPAR Name p8vw2

Resource for WPAR Backup Image [p8vw2-backup] +

Resource for System Backup Image [] +

Alternate DEVEXPORTS for installation [] +

Alternate SECATTRS for installation [] +

We created a WPAR Specification NIM resource. The file was created in the Global environment, using the mkwpar command to write out the VWPAR specification details to a text file. This file was then copied to the NIM master to be used to create the NIM resource.

# mkwpar -e p8vw2 -w -o /tmp/cg/p8vw2_cg.cf

# lsnim -l p8vw2-spec

p8vw2-spec:

class = resources

type = wpar_spec

Rstate = ready for use

prev_state = unavailable for use

location = /tmp/cg/p8vw2_cg.cf

alloc_count = 1

server = master

# lsnim -t wpar_spec

p8vw2-spec resources wpar_spec

The specification file contained the volume group name (wparvg) where each of the VWPAR file systems where located.

# grep vg /tmp/cg/p8vw2_cg.cf

rootvgwpar = "no"

vg = "wparvg"

# grep -p vg /tmp/cg/p8vw2_cg.cf

general:

version = "1"

name = "p8vw2"

hostname = "p8vw2"

checkpointable = "no"

directory = "/wpars/p8vw2"

privateusr = "yes"

uuid = "3e7a2bfb-6060-4770-ad7e-4d6b2a84f657"

devices = "/etc/wpars/devexports"

architecture = "none"

ostype = "1024"

xwparipc = "no"

auto = "no"

rootvgwpar = "no"

preserve = "no"

routing = "no"

mount:

logname = "/dev/loglv00"

directory = "/home"

vfs = "jfs2"

vg = "wparvg"

size = "131072"

mount:

logname = "/dev/loglv00"

mountopts = "rw"

directory = "/opt"

vfs = "jfs2"

vg = "wparvg"

size = "786432"

mount:

logname = "/dev/loglv00"

directory = "/var"

vfs = "jfs2"

vg = "wparvg"

size = "786432"

mount:

logname = "/dev/loglv00"

directory = "/tmp"

vfs = "jfs2"

vg = "wparvg"

size = "393216"

mount:

logname = "/dev/loglv00"

directory = "/"

vfs = "jfs2"

vg = "wparvg"

size = "262144"

mount:

logname = "/dev/loglv00"

mountopts = "rw"

directory = "/usr"

vfs = "jfs2"

vg = "wparvg"

size = "3670016"

However, even with the specification file in place, the result was the same and the VWPAR file systems were created in rootvg rather than wparvg.

We eventually discovered that this behaviour was a bug.

IV68299: RESTWPAR OF VWPAR IS CREATED IN ROOTVG

http://www-01.ibm.com/support/docview.wss?uid=isg1IV68299

Error description

When restoring a backup of a VWPAR that resided in a

non-root volume group, the WPAR is created in rootvg,

even though the image.data file in the backup indicates the

desired volume group.

Local fix

Specify an alternate default volume group using the -M

flag of the restvg command. For example:

# restwpar -f backup_location -M "-g wparvg" -n wpar_name

Problem summary

When restoring a backup of a VWPAR that resided in a

non-root volume group, the WPAR is created in rootvg,

even though the image.data file in the backup indicates the

desired volume group.

Problem conclusion

Added flag in restwpar to call mkwpar with ignore_lvs=no by

default.

Note: Both the Global environment and the NIM master were running AIX 7100-03-04-1441.

We were able to request in ifix from AIX support. Once we installed the ifix, in the Global environment, the restore process via NIM worked as expected and the VWPAR file systems were recovered in wparvg. We did not need to use the WPAR specification NIM resource.

# emgr -l

ID STATE LABEL INSTALL TIME UPDATED BY ABSTRACT

=== ===== ========== ================= ========== ======================================

1 S IV68299s4a 04/20/15 01:08:20 RESTWPAR VWPAR CREATED IN ROOTVG

STATE codes:

S = STABLE

M = MOUNTED

U = UNMOUNTED

Q = REBOOT REQUIRED

B = BROKEN

I = INSTALLING

R = REMOVING

T = TESTED

P = PATCHED

N = NOT PATCHED

SP = STABLE + PATCHED

SN = STABLE + NOT PATCHED

QP = BOOT IMAGE MODIFIED + PATCHED

QN = BOOT IMAGE MODIFIED + NOT PATCHED

RQ = REMOVING + REBOOT REQUIRED

# diff /usr/emgrdata/efixdata/IV68299s4a/save/EFSAVE1 /usr/sbin/restwpar

459a460

> IGNORE_LVS="ignore_lvs=no"

660a662

> IGNORE_LVS=

664c666

< IMAGE_DATA_SETUP="$IMAGE_DATA_SETUP $SHRINK $IGNORE_MAPS"

---

> IMAGE_DATA_SETUP="$IMAGE_DATA_SETUP $SHRINK $IGNORE_MAPS $IGNORE_LVS"

# lsvg -l rootvg

rootvg:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

hd5 boot 1 1 1 closed/syncd N/A

hd6 paging 8 8 1 open/syncd N/A

hd8 jfs2log 1 1 1 open/syncd N/A

hd4 jfs2 6 6 1 open/syncd /

hd2 jfs2 35 35 1 open/syncd /usr

hd9var jfs2 7 7 1 open/syncd /var

hd3 jfs2 2 2 1 open/syncd /tmp

hd1 jfs2 1 1 1 open/syncd /home

hd10opt jfs2 5 5 1 open/syncd /opt

hd11admin jfs2 2 2 1 open/syncd /admin

lg_dumplv sysdump 16 16 1 closed/syncd N/A

livedump jfs2 4 4 1 open/syncd /var/adm/ras/livedump

cglv jfs2 100 100 1 open/syncd /cg

# lsvg -l wparvg

wparvg:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

loglv00 jfs2log 1 1 1 open/syncd N/A

fslv02 jfs2 4 4 1 open/syncd /wpars/p8wpar2

fslv03 jfs2 2 2 1 open/syncd /wpars/p8wpar2/home

fslv04 jfs2 12 12 1 open/syncd /wpars/p8wpar2/opt

fslv05 jfs2 6 6 1 open/syncd /wpars/p8wpar2/tmp

fslv06 jfs2 56 56 1 open/syncd /wpars/p8wpar2/usr

fslv07 jfs2 12 12 1 open/syncd /wpars/p8wpar2/var

# uname -W

# lswpar

Name State Type Hostname Directory RootVG WPAR

------------------------------------------------------------

p8vw2 A S p8wpar2 /wpars/p8wpar2 no

# clogin p8vw2

*******************************************************************************

* *

* Welcome to AIX Version 5.3! *

* *

* Please see the README file in /usr/lpp/bos for information pertinent to *

* this release of the AIX Operating System. *

* *

*******************************************************************************

# oslevel -s

5300-12-09-1341

SSP and Disaster Recovery (Tech Preview)

cggibbo Tags: dr vios aix ssp recovery chris gibson disaster 2 Comments 10,700 Views

I’ve been working with a customer recently that required a special kind of disaster recovery capability for their shared storage pool (SSP) environment. The customer had implemented a private cloud solution in their existing POWER environment. The solution consisted of an IBM SmartCloud deployment with IBM Systems Director and VMControl managing the PowerVM deployment of virtual machines (AIX partitions) across the landscape.

The decision to use shared storage pools was driven by the fact that the customer was using SAN storage that could not be managed by VMControl. However, shared storage pool support was available with VMControl. This meant that the customer could continue using their non-IBM disk in their private cloud. The customer quickly noticed the many advantages that SSPs bring to a POWER cloud environment, such as very fast virtual machine deployment using snapshots and linked-clones.

Traditionally, the customer’s disaster (data) recovery process relied on SAN disk replication between its production and disaster recovery sites. In the event of a disaster the customer could recover a production system using a replicated copy of the SAN disks at the disaster recovery site. This also allowed the customer to perform regular disaster recovery tests. In these tests, the disaster recovery site is “fenced” from the real production network and SAN. The customer is then able to “bring up” a complete copy of any or all (replicated) production systems. Because these “copies” of production are fenced off from the real production environment the customer can undertake lengthy disaster recovery test programs without impacting the running production systems.

The customer had used this process for many, many years. They were happy with it and the flexibility it gave them. They wanted to continue using this process for the production SSP based systems.

The customer wanted the ability to recover a (storage) replicated copy of their entire shared storage pool at their disaster recovery (DR) site. We discussed the use of SSP failure group mirroring to cater for their DR requirements of their production AIX partitions. The following diagram shows our initial proposal of SSP mirroring across the production and DR sites.

However, this did not meet their needs. They needed a way to recover their AIX production partitions in isolation to their real production environment. SSP mirroring would not provide them with this flexibility. Essentially the customer could not start a “copy” of any or all production partitions at the DR site. In a SSP failure group setup, the SSP spans both the production and DR sites (as shown in the previous diagram). Meaning the SSP is in production across both sites. SSP mirroring would protect the customer in the event that the production site was to fail; they could start their production systems at the DR site, however they could not start isolated copies of the production systems in the same SSP. This meant SSP mirroring would break their current DR processes and provide them with reduced capability when compared to their traditional DR method.

Prior to SSP mirroring, one of the major drawbacks of SSPs was the lack of resilience at the storage layer. We were not able to mirror the disks in the pool from one storage subsystem to another (either local or remote). Having a single disk storage subsystem in a shared storage pool was a rather large single point of failure (SPOF). Starting with VIOS 2.2.3.0, we were given the capability to mirror the disks in a SSP from one storage system to another. All of the mirroring was configured and managed from the VIOS in the SSP cluster. The VIO client partitions are not aware that their disks are ‘mirror protected’ in the SSP. There are no changes required in the client partitions as the SSP mirroring feature is completely transparent to them. This feature enabled SSPs to be considered ready for production use.

Some other considerations that we discussed with the customer regarding SSP mirroring were the fact that the disks in the SSP failure group need to be shared across all the SSP nodes at both the production and DR sites. This is not a viable option in the case of long distances between sites (more than 100KM apart). SSP mirroring uses synchronous writes. This means that any write that happens at the production site has to be written synchronously on the failure group disks at the DR site first then production. This will introduce a delay in completing I/O which will have an impact on performance. Fortunately the customer’s two sites were far less than 100KM apart, so this consideration did not apply to them, but I mention it simply to make others aware of this point.

After much discussion with our SSP development team at IBM, we were able to provide the customer with an acceptable method of recovering their SSP environment at their DR site using a replica of the storage. What follows is a preview of this new recovery mechanism. This feature is not currently available as part of the standard SSP capability with VIOS 2.2.3.X. It is hoped that we will see this feature officially offered and supported at some stage in 2015. Of course, this is not an announcement and plans are always subject to change.

The following diagram provides a pictorial view of what the customer needed for DR with their SSP systems. As you can see in the diagram, there is a production and DR site. At the DR site there is a single SSP (with two VIOS in the cluster). Both the production and DR VIOS are in two separate shared storage pools.

The SSP development team provided a special ifix for the customer to install on their DR virtual I/O servers (VIOS). They also provided a script and procedure for the customer to follow. We successfully tested this procedure at the customers DR site. This was a huge success from the customer’s perspective as they were able to recover their entire production SSP in a few minutes. They could then selectively start/stop clones of the production AIX partitions at their DR site, without impacting the “real” production partitions at the production site. They were also able to continue using their existing storage replication method (as they had always done for all of their systems) in a DR situation. Not having to dramatically change their existing DR process and procedures was very important to the customer.

I’ll walk through the procedure now and share some insights (from my lab tests) as we go. It goes without saying, that before we started, the customer ensured that all the production SSP disks (LUNs) were being replicated to their DR site using their standard storage replication facility. The SSP repository disk was not replicated to the DR site. A new disk was used as the repos disk at the DR site.

The process (described in detail below) consists of a) taking a backup of the production SSP cluster configuration, b) running a “customise” script to modify the saved backup file to replace the production VIOS and hdisk names with the DR VIOS and hdisk names, c) restoring the modified VIOS backup on the DR VIOS at the DR site and finally d) verifying that the cluster has been recovered using the replicated storage at DR.

The first step was to take a backup of the production SSP VIOS cluster using the viosbr command. At the primary (production) site we used viosbr to create a backup of the cluster configuration. This generated a backup file named /home/padmin/cfgbackups/sspbackup.mycluster.tar.gz.

$ viosbr -backup -clustername mycluster -file sspbackup

The viosbr backup file was then transferred to one of the DR VIOS. On the selected DR VIOS we copy the backup file (sspbackup.mycluster.tar.gz) to the /home/padmin/backups directory.

We start an oem_setup_env shell and verify the file is intact.

$ oem_setup_env

# cd /home/padmin/backups

#ls

sspbackup.mycluster.tar.gz

We create a new file called nodelist in the backups directory. This file contains a list of the DR VIOS hostnames. These hostnames are used to create the replicated cluster at the remote site.

sspnode1.abc.com
sspnode2.abc.com

Also in the backups directory, we create another new file called disklist. This file contains a list of each of the replicated SSP disks with their unique_ids. These disks are the replicated copies of the SSP pool disks from the primary site. All these disks must be accessible on all DR VIOS nodes.

$ pv -list -fmt , | awk -F, '{print $7}'

3E213600A0B80006E24E60000AB8F4F570A440F1818 FAStT03IBMfcp
3E213600A0B80006E250600001CD94F570E730F1818 FAStT03IBMfcp

The next step is to copy the necessary ifix file to the backups directory on the selected DR VIOS. We untar this file, which extracts two files. One file is the customize script and the other is the actual ifix we need to install on both VIOS.

./customize_ssp_backup
./ifix/sspDR_1.140627.epkg.Z

We install the ifix on all the DR VIOS which are going to be part of SSP setup.

$ updateios -install -dev /home/padmin/backups/ifix -accept

We run the script to customize the backup file for new setup at the DR site.

# ./customize_ssp_backup

Usage: customize_ssp_backup <backup_filename> <filename_with_hostnames> <filename_with_PoolDisks>

# ./customize_ssp_backup sspbackup.mycluster.tar.gz nodelist disklist

New configuration file created successfully: sspbackup_NEW.mycluster.tar.gz

We locate a free disk to be used as the new cluster repository disk on the selected DR VIOS.

$ lspv –free

We restore the cluster configuration using viosbr, specifying the customized viosbr backup file. We check the exit status is zero to confirm the restore process completed successfully.

$ viosbr -restore -clustername mycluster -repopvs hdisk5 -file /home/padmin/backups/sspbackup_NEW.mycluster.tar.gz -type cluster_remote

Backedup Devices that are unable to restore/change

==================================================

DEPLOYED or CHANGED devices:

============================

Dev name during BACKUP Dev name after RESTORE

---------------------- ----------------------

$ echo $?

We can now verify the new cluster has been configured and recovered correctly. We use the cluster and lu commands to check the cluster is up and that the logical units are available in the shared storage pool.

$ cluster -status -clustername mycluster

$ lu –list

The SSP is now ready for use at the DR site.

Before we tested this procedure in the customer’s environment, I tried it in my lab first. My lab had the following configuration:

1 x POWER7 750 with a single VIOS and a Shared Storage Pool. I considered this my “production” SSP, which I wanted to recover on another POWER system in my lab.
1 x POWER7 PS701 Blade with a single VIOS. No SSP. I treated this system as my “DR” environment on which I wanted to recover a complete copy of my “production” SSP.

My “production” SSP was configured with V7000 storage. I asked my storage administrator to create a copy (via flashcopy) of the nominated SSP disks from my 750 VIOS cluster. He then presented these new (flashcopied) LUNs to the “DR” VIOS on the PS701.

I then ran through the recovery steps (as outlined above). I started by taking a backup of the VIOS on the 750 using the viosbr command.

750vio1 (“production”)

-------------------------------

$ viosbr -backup -clustername VIOCLUSTER -file sspbackup

Backup of this node (750vio1) successful

I discovered that a SSP was already configured on my blade VIOS. I would need to delete the SSP cluster before I could recover the “production” cluster backup on this system. I obtained a list of the current disks in the SSP.

ps701-vios (“DR”)

-----------------------

$ pv -list

POOL_NAME: VIOSSP

TIER_NAME: SYSTEM

FG_NAME: Default

PV_NAME SIZE(MB) STATE UDID

hdisk2 51200 ONLINE 3321360050768019E027CC000000000000~

hdisk3 51200 ONLINE 3321360050768019E027CC000000000000~

I also took note of the existing repository disk.

$ lspv | grep caa

hdisk1 000a366a031ffa37 caavg_private active

I removed any existing client VSCSI mappings for SSP disk so that I could delete the logical units from the SSP and delete the cluster.

$ lu -remove -lu sspdisk1_lpar1

Logical unit sspdisk1_lpar1 with udid "27cf8eb804710aa5d5995c414b2fb329" is removed.

$ lu -list

Then I deleted the cluster and took note of the existing disks again.

$ cluster -delete -clustername VIOCLUSTER

Cluster VIOCLUSTER has been removed successfully.

$ lspv

NAME PVID VG STATUS

hdisk0 000a366a68640e58 rootvg active

hdisk1 000a366a031ffa37 None

hdisk2 none None

hdisk3 none None

hdisk4 none None

hdisk5 none None

hdisk6 none None

hdisk7 none None

hdisk8 none None

hdisk9 none None

hdisk10 none None

hdisk11 000a366a6b0d0d56 None

hdisk12 000a366a6b0d0d56 None

hdisk13 000a366a6b0d0d56 None

I also chose to “scrub” the existing repository disk using the cleandisk command. This would allow me to re-use this disk for my SSP recovery procedure.

$ cleandisk -r hdisk1

0967-112 Subsystem not configured.

rmcluster: This operation will scrub hdisk1, removing any volume groups and clearing cluster identifiers.

If another cluster is using this disk, that cluster will be destroyed.

Are you sure? (y/[n]) y

WARNING: Force continue.

ERROR: import caavg_private failed.

remove_cluster_repository: Force continue.

ERROR: Cannot varyonvg caavg_private. It does not exist.

ERROR: rc=1, remove caavg_private failed.

remove_cluster_repository: Force continue.

rmcluster: Successfully removed hdisk1.

I entered the oem_setup_env shell and ran cfgmgr to bring in the new (flashed) disks on the “DR” VIOS. Two new disks were configured. I would use these disks with the SSP recovery script and disklist file.

$ oem_setup_env

# cfgmgr

# lspv

hdisk0 000a366a68640e58 rootvg active

hdisk1 000a366a031ffa37 None

hdisk2 none None

hdisk3 none None

hdisk4 none None

hdisk5 none None

hdisk6 none None

hdisk7 none None

hdisk8 none None

hdisk9 none None

hdisk10 none None

hdisk11 000a366a6b0d0d56 None

hdisk12 000a366a6b0d0d56 None

hdisk13 000a366a6b0d0d56 None

hdisk14 none None <<<<<<<<

hdisk15 00f603cde9a7b15a None <<<<<<<<

I took note of the hdisks and the unique_ids and placed this information in a new file named disklist.

$ oem_setup_env

# odmget -qname=hdisk14 CuAt |grep -p unique_id |grep value |cut -d= -f2 | sed -e

's/"//g' | sed -e 's/ //g'

hdisk14 3321360050768019E027CC00000000000009D04214503IBMfcp

# odmget -qname=hdisk15 CuAt |grep -p unique_id |grep value |cut -d= -f2 | sed -e

's/"//g' | sed -e 's/ //g'

hdisk15 3321360050768019E027CC00000000000009E04214503IBMfcp

# cd backups

# cat disklist

3321360050768019E027CC00000000000009D04214503IBMfcp

3321360050768019E027CC00000000000009E04214503IBMfcp

I created the nodelist file, also in the backups directory.

# pwd

/home/padmin/backups

# cat nodelist

ps701-vios

I ensured that the necessary ifix was installed on my “DR” blade VIOS partition.

# emgr -l

ID STATE LABEL INSTALL TIME UPDATED BY ABSTRACT

=== ===== ========== ================= ========== ======================================

1 S sspDR_1 07/07/14 10:38:10 Fix to restore cluster on remote site

Then I ran the customise script to prepare the viosbr backup file for restoration on this “DR” VIOS.

# ./customize_ssp_backup sspbackup.tar.gz nodelist disklist

backupfile=sspbackup.tar.gz, hostfile=/home/padmin/backups/nodelist, pvfile=/home/padmin/backups/disklist

MTM=8406-71Y0310A366A, ID=1, nodefile=VIOCLUSTERMTM8406-71Y0310A366AP1.xml, clust=VIOCLUSTER, localnode=VIOCLUSTERMTM8406-71Y0310A366AP1.xml, newbackup=sspbackup_NEW.VIOCLUSTER.tar

cp: 0653-435 VIOCLUSTERMTM8406-71Y0310A366AP1.xml and VIOCLUSTERMTM8406-71Y0310A366AP1.xml are identical (not copied).

VIOCLUSTERDB/9fa498e4896611e38d72f603cdcd9c55/VIOCLUST.xml

/home/padmin/backups/temp/VIOCLUSTERDB/9fa498e4896611e38d72f603cdcd9c55/VIOCLUST.xml

CLUSTER CONF FILE=/home/padmin/backups/temp/VIOCLUSTERDB/9fa498e4896611e38d72f603cdcd9c55/VIOCLUST.xml

CLUSTER CONF FILE NEW=/home/padmin/backups/temp/VIOCLUSTERDB/9fa498e4896611e38d72f603cdcd9c55/VIOCLUST2.xml

New configuration file created successfully: sspbackup_NEW.VIOCLUSTER.tar.gz

I then restored the SSP from the customised viosbr backup file. I specified hdisk1 as the repos disk. The restore was successful (with an exit status of zero).

$ viosbr -restore -clustername VIOCLUSTER -repopvs hdisk1 -file /home/padmin/backups/sspbackup_NEW.VIOCLUSTER.tar.gz -type cluster_remote

"CLUSTER restore successful.

"Backedup Devices that are unable to restore/change

==================================================

DEPLOYED or CHANGED devices:

============================

Dev name during BACKUP Dev name after RESTORE

---------------------- ----------------------

$ echo $?

After the successful restore, I was able to display the cluster status and list the logical units in the recovered SSP.

$ cluster -list

CLUSTER_NAME: VIOCLUSTER

CLUSTER_ID: 9fa498e4896611e38d72f603cdcd9c55

$ lu -list

POOL_NAME: VIOSSP

TIER_NAME: SYSTEM

LU_NAME SIZE(MB) UNUSED(MB) UDID

sspdisk1_750lpar11 10240 7251 4f42b83c0ed7e826a4784b204b1c81ea

SNAPSHOTS

sspdisk1_750lpar11_snap1

$ cluster -status -clustername VIOCLUSTER

Cluster Name State

VIOCLUSTER OK

Node Name MTM Partition Num State Pool State

ps701-vios 8406-71Y0310A366A 1 OK OK

As expected, both of the “flashed” disks were now assigned to the recovered disk pool.

$ pv -list

POOL_NAME: VIOSSP

TIER_NAME: SYSTEM

FG_NAME: Default

PV_NAME SIZE(MB) STATE UDID

hdisk15 51200 ONLINE 3321360050768019E027CC000000000000~

hdisk14 51200 ONLINE 3321360050768019E027CC000000000000~

The existing hdisk1 had been successfully re-used for the repository disk of the “new” cluster.

$ lspv | grep caa

hdisk1 000a366a031ffa37 caavg_private active

The VIOS error report on the “DR” VIOS also confirmed that the SSP cluster was recovered and the services started correctly.

$ errlog

IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION

228C1EBE 0707110114 I S POOL Cluster state change has occurred

228C1EBE 0707110014 I S POOL Cluster state change has occurred

EDFF8E9B 0707110014 I O StorageRM IBM.StorageRM daemon has started.

3B16518D 0707110014 I S ConfigRM The node is online in the domain indicat

4BDDFBCC 0707110014 I S ConfigRM The operational quorum state of the acti

AFA89905 0707110014 I O cthags Group Services daemon started

At this point I was able to map the LU to my VIO client partition and boot it on the PS701. AIX started without an issue and I now had a complete clone of my existing AIX partition (from the 750) running on my POWER blade.

$ lu –map –lu sspdisk1_750lpar11 –vadapter vhost4

Assigning logical unit 'sspdisk1_750lpar11' as a backing device.

VTD:vtscsi4

$ lsmap -vadapter vhost4

SVSA Physloc Client Partition ID

--------------- -------------------------------------------- ------------------

vhost4 U8406.71Y.10A366A-V1-C19 0x00000006

VTD vtopt1

Status Available

LUN 0x8200000000000000

Backing device

Physloc

Mirrored N/A

VTD vtscsi4

Status Available

LUN 0x8100000000000000

Backing device sspdisk1_750lpar11.4f42b83c0ed7e826a4784b204b1c81ea

Physloc

Mirrored N/A

Shared storage pools already provide some fantastic capabilities that are attractive to most POWER customers. Features such as SSP mirroring, snapshots and linked-clones provide some very powerful and flexible configurations that will accelerate virtual machine deployment and management on IBM Power Systems.

I believe the enhancement I’ve discussed here will prove very popular with many of our enterprise AIX and POWER customers. It will provide one more compelling reason for customers to consider shared storage pools in their environments, particularly for disaster recovery situations. There may even be the capability (in the future) to completely automate the entire procedure that I’ve outlined in this article. I hope that I can provide further updates on this capability in early 2015.

Reference material:

Mirroring a shared storage pool

https://www-304.ibm.com/support/knowledgecenter/POWER7/p7hb1/iphb1_mirroring_storage_pool.htm?cp=POWER7%2F1-8-3-5-7-1-2-3

Modified by cggibbo

LPM and viosecure

cggibbo Tags: lpm ipsec firewall vios hscla230 2 Comments 8,660 Views

I came across this strange LPM issue recently. Thought I’d share it with you.

All the customers VIOS were configured with the viosecure level set to high.

When the VIOS is configured with a security profile set to high, PCI or DoD, a new feature is enabled during LPM. This new feature is called “Secure LPM”. As a result when you initiate an LPM operation, the “Secure LPM” feature automatically enables the VIOS firewall and configures a secure (ipsec) tunnel for all LPM traffic over the network.

Before we started LPM, the VIOS firewall is off:

$ viosecure -firewall view

IPv4 Firewall OFF

ALLOWED PORTS

Local Remote

Interface Port Port Service IPAddress Expiration

Time(seconds)

--------- ---- ---- ------- --------- ---------------

And it's activated after we start LPM:

$ viosecure -firewall view

IPv4 Firewall ON

ALLOWED PORTS

Local Remote

Interface Port Port Service IPAddress Expiration

Time(seconds)

--------- ---- ---- ------- --------- ---------------

In the customer’s case, we moved a partition from a 795 running a pair of VIOS at 2.2.1.3 to another 795 running another pair of VIOS at 2.2.2.1. This worked OK. When we attempted to move the partition back, we received an error stating that the source MSP had rejected the request.

HSCLA230

The mover service partition on the source managed system has rejected the request to stop the migration. Verify that the migration state of the partition is Migration Starting, and try the operation again.

After a while we discovered that when the LPM operation started, the source VIOS would enable its firewall and we would immediately loose connectivity (i.e. our SSH session to the VIOS would hang). We also found that RMC connectivity between the source VIOS and HMC also dropped. Somehow the VIOS firewall was blocking network connectivity to pretty much everything.

Eventually, after much digging we found that the source VIOS was the victim of an errant firewall rule (configured on the source VIOS itself).

# smitty ips4_conf_filter

List IP Security Filter Rules

1 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 no udp eq 4001 eq 4001 both both no all packets 0 all 0 none  Default Rule

2 *** Dynamic filter placement rule for IKE tunnels *** no

3 deny 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 y all any 0 any 0 all both inbound no yes 0 no 0 patt_none

0 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 yes all any 0 any 0 both both no all packets 0 all 0 none  Default Rule

We removed this rule (after some discussion with IBM support).

# smitty ips4_conf_filter

Delete IP Security Filter Rules

3 deny 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 y all any 0 any 0 all both inbound no yes 0 no 0 patt_none

# smitty ips4_conf_filter

List IP Security Filter Rules

1 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 no udp eq 4001 eq 4001 both both no all packets 0 all 0 none  Default Rule

2 *** Dynamic filter placement rule for IKE tunnels *** no

0 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 yes all any 0 any 0 both both no all packets 0 all 0 none  Default Rule

We were then able to move the partition back to the original frame. We are still searching for an answer to how this happened. The IBM support team believe the rule must have been created by an administrator at some point. Of course, the administrator claims that is untrue.

Modified by cggibbo

Using chhwres to free up partition resources.

cggibbo 2 Comments 11,491 Views

You can use the chhwres command, on the HMC, to “free up” allocated processor and memory resources from your logical partitions.

I may want to do this if I need to improve the affinity placement of a partition on a POWER7 system or if I need to free up resources from trial capacity on demand in order to re-use the resources with On/Off capacity instead.

Here’s an example of “freeing up” some processor resources from a partition.

I’d like to remove the processor allocation from my AIX partition named 750lpar6. This partition is currently active and has 0.5 processing units assigned to it.

hscroot@hmc1:~> lparls | grep 750lpar6

750lpar6 Running

hscroot@hmc1:~> lparlscpu | grep 750lpar6

750lpar6 0.5

I’ll stop the partition (shutdown –F) so that I may free up its processor resources. The partition is now inactive.

hscroot@hmc1:~> lparls | grep 750lpar6

750lpar6 Not Activated

Even though the partition is no longer running, the processor allocation of 0.5 cores remains assigned to this partition.

The number of processors ‘assigned to partitions’ is 6.5 units; as shown on the HMC GUI, under the properties of the managed system.

Now I’ll use the chhwres command to “free” the currently allocated cores from the partition.

hscroot@hmc1:~> chhwres -r proc -m 750-1 -o r --id 8 --procunits 0.5

The partition is now allocated a total of 0.0 cores and is no longer using any processor resources.

hscroot@hmc1:~> lparlscpu | grep 750lpar6

750lpar6 0.0

Looking at the managed system properties again, the number of processors ‘assigned to partitions’ has dropped to 6.0 units.

The partitions profile is NOT changed.

hscroot@hmc1:~> lparinfo 750lpar6

750lpar6

On frame 750-1

Current State: Not Activated

OS version: AIX 6.1 6100-05-01-1016

Profile properties:

Profile properties:
name=Default,lpar_name=750lpar6,lpar_id=8,lpar_env=aixlinux,all_resources=0,
min_mem=256,desired_mem=10240,max_mem=24576,min_num_huge_pages=0,desired_num_huge_pages=0,
max_num_huge_pages=0,mem_mode=ded,hpt_ratio=1:64,proc_mode=shared,min_proc_units=0.1,
desired_proc_units=0.5,max_proc_units=2.0,min_procs=1,desired_procs=4,max_procs=8,
sharing_mode=uncap,uncap_weight=128,shared

The partition is started again (activated) and it re-acquires 0.5 cores as it did prior to shut down.

hscroot@hmc1:~> lparpoweron 750lpar6

hscroot@hmc1:~> lparls | grep 750lpar6

750lpar6 Running

hscroot@hmc1:~> lparlscpu | grep 750lpar6

750lpar6 0.5

Modified by cggibbo

SMS SAN zoning support

cggibbo Tags: npiv virtual_fibre_channel powervm chris_gibson aix power_systems vios vfc 2 Comments 21,337 Views

Until recently, if you were configuring a new LPAR with virtual FC adapters you couldn’t force it to log into the SAN before an operating system (such as AIX) was installed. I’ve written about this before (see link below). I also offered a way to work around this issue.

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/entry/new_hmc_commands_chnportlogin_and_lsnportlogin_login_in_virtual_fibre_channel_adapters_for_npiv_client_lpars1?lang=en

As mentioned in my post, there are new HMC commands to force a new LPAR to connect to the SAN, so that SAN administrators can zone storage to the new WWPN(s).

There’s also another method available in the Power Systems SMS menu.

In recent levels of POWER systems firmware, there is a new option in the SMS menu called SAN Zoning Support. You’ll find it mentioned here:

http://pic.dhe.ibm.com/infocenter/powersys/v3r1m5/index.jsp?topic=/p7hb6l/usingsmsmenus_sanzoning.htm

I’ve successfully used this method on both POWER6 (595) and POWER7 (795) systems. After configuring a new LPAR profile with a single VFC adapter, the VIOS reported that the client was not logged into the SAN:

$ lsmap -vadapter vfchost41 -npiv

Name Physloc ClntID ClntName ClntOS

------------- ---------------------------------- ------ -------------- -------

vfchost41 U9119.FHA.99D99C1-V2-C199 99

Status:NOT_LOGGED_IN

FC name:fcs4 FC loc code:U5803.001.9912345-P2-C5-T1

Ports logged in:0

Flags:4<NOT_LOGGED>

VFC client name: VFC client DRC:

I forced this event by booting into SMS and selecting 5 (Boot Options).

PowerPC Firmware

Version EH350_120

-------------------------------------------------------------------------------

Main Menu

1. Select Language

2. Setup Remote IPL (Initial Program Load)

3. Change SCSI Settings

4. Select Console

5. Select Boot Options

-------------------------------------------------------------------------------

Navigation Keys:

X = eXit System Management Services

-------------------------------------------------------------------------------

Type menu item number and press Enter or select Navigation key:5

Then I selected 4 (SAN Zoning Support).

PowerPC Firmware

Version EH350_120

-------------------------------------------------------------------------------

Multiboot

1. Select Install/Boot Device

2. Configure Boot Device Order

3. Multiboot Startup <OFF>

4. SAN Zoning Support

-------------------------------------------------------------------------------

Navigation keys:

M = return to Main Menu

ESC key = return to previous screen X = eXit System Management Services

-------------------------------------------------------------------------------

Type menu item number and press Enter or select Navigation key:4

Next I choose the appropriate VFC adapter I’d like to connect on the SAN at this time.

PowerPC Firmware

Version EH350_120

-------------------------------------------------------------------------------

Select Media Adapter

1. U9119.FHA.83D99C3-V99-C20-T1 /vdevice/vfc-client@30000014

-------------------------------------------------------------------------------

Navigation keys:

M = return to Main Menu

ESC key = return to previous screen X = eXit System Management Services

-------------------------------------------------------------------------------

Type menu item number and press Enter or select Navigation key:1

A message appears stating that it may now be possible to perform SAN zoning to the adapters WWPN.

.-----------------------------------------------------------------------.

| The selected adapter has been opened. |

| Zoning of attached disks may now be possible. |

| Press any key to close the adapter and return to the previous menu. |

`-----------------------------------------------------------------------'

On the VIOS I observed the client vfchost adapter status changed to LOGGED_IN.

$ lsmap -vadapter vfchost41 -npiv

Name Physloc ClntID ClntName ClntOS

------------- ---------------------------------- ------ -------------- -------

vfchost41 U9119.FHA.99D99C1-V2-C199 99 aixlpar1 FIRMWARE

Status:LOGGED_IN

FC name:fcs4 FC loc code:U5803.001.9912345-P2-C5-T1

Ports logged in:1

Flags:a<LOGGED_IN,STRIP_MERGE>

VFC client name:Virtual Fibre Channel - Firmware driverVFC client DRC:U9119.FHA.83D99C3-V99-C20

It appears that the new SMS option is simply an interface to the Open Firmware Prompt and the IOINFO command. This was how we performed the task without the SMS menu in front:

https://www.ibm.com/developerworks/wikis/display/virtualization/vfclogin

So far I’ve found this new SMS option on POWER7 795 systems running firmware level AH730_078 and POWER6 595 systems running EH350_120.

Checking num_cmd_elems for VFC adapters with kdb

cggibbo Tags: -p odm kdb lsattr chris_gibson num_cmd_elems adapter vfcs aix chdev reboot vfc 2 Comments 24,057 Views

This entry is similar in theme to one of my previous posts about verifying your hdisk queue_depth settings with kdb. This time we want to check if an attribute for a Virtual FC (VFC) adapter has been modified and whether or not AIX has been restarted since the change. The attribute I’m interested in is num_cmd_elems. This value is often changed from its default settings, in AIX environments, to improve I/O performance on SAN attached storage.

From kdb you can identify the VFC adapters configured on an AIX system using the vfcs subcommand. Not only does this tell you what adapters you have, but it also identifies the VIOS each adapter is connected to and the corresponding vfchost adapter. Nice!

(0)> vfcs

NAME ADDRESS STATE HOST HOST_ADAP OPENED NUM_ACTIVE

fcs0 0xF1000A00103D4000 0x0008 vio1 vfchost10 0x01 0x0000

fcs1 0xF1000A00103D6000 0x0008 vio2 vfchost10 0x01 0x0000

You can view the current (running) configuration of a VFC adapter using the kdb vfcs subcommand and the name of the VFC adapter, for example fcs1:

(0)> vfcs fcs1

adapter: fcs0 partition_num: 0x14 partition_name: aixlpar1

num_cmd_elems: 0xC8 location_code: U9119.FHA.87654A1-V20-C10-T1

state: 0x8 opened: 0x1 flags: 0x1

host_name: vio1 host_device: vfchost10

maxCmds: FA maxDMALength: 100000 SCSIid: 6F1E9A

portName: C050760399100168 nodeName: C050760399100168 nport_id:

6F1E9A

cmd_q.base_addr: 0xF1000A00102D0000 asyncq.base_addr: 0xF1000A0019761000

crq[0]: 0x8001000000000000 crq[1]: 0x8013960

ctl_head: 0x0 ctl_tail: 0x0

cmd_head: 0x0 cmd_tail: 0x0

pool_head: 0xF1000A001975BD08 pool_tail: 0xF1000A0019759D88

head_active: 0x0 tail_active: 0x0

cancel_head: 0xF1000A0019758090 cancel_tail: 0xF1000A0019758E58

num_cmds_allowed: 0xFA num_cmds_active: 0x0 num_cmds_pending: 0x0

head_flush_q: 0x0 tail_flush_q: 0x0 cmd_flush_q: 0x0

no_elem: 0x0 no_dma: 0x0 no_sglist: 0x0 bad_mad: 0x0 qfull: 0x0

h_dropped: 0x0 link_down_cnt: 0x0 num_frames: 0x0 tx_lock: 0x0

rcv_lock: 0x0 ctl_lock: 0x0 lock: 0xFFFFFFFFFFFFFFFF

ctl_event: 0xFFFFFFFFFFFFFFFF open_event: 0xFFFFFFFFFFFFFFFF

reset_cmd: 0x0 log_cmd: 0x0

link_cmd: 0x0 ctl_cmd->cmd_state: 0x0

inp_reqs: 0x12594 out_reqs: 0x1A7FD4 ctrl_reqs: 0x44

inp_bytes: 0x14973F90 out_bytes: 0xEC14EC18 npiv_logout_sent: 0x0

npiv_logout_rcvd: 0x0 act_hi_water: 0x10 pend_hi_water: 0x1

Using the output from this command we can determine the current (running) value for a number of VFC attributes, including num_cmd_elems.

So I start with an adapter with a num_cmd_elems value of 200. Both the lsattr command and kdb report 200 (C8 in hex) for num_cmd_elems.

# lsattr -El fcs1 -a num_cmd_elems

num_cmd_elems 200 Maximum Number of COMMAND Elements True

# echo vfcs fcs1 | kdb | grep num_cmd_elems

num_cmd_elems: 0xC8 location_code: U9119.FHA.87654A1-V20-C10-T1

I change num_cmd_elems to 400 with chdev –P (remember, the –P flag only updates the AIX ODM, and not the running configuration of the device in the AIX kernel. You must either reboot for this change to take effect or offline & online the device).

# chdev -l fcs1 -a num_cmd_elems=400 -P

fcs1 changed

Now the lsattr command reports num_cmd_elems is set to 400 in the ODM.

# lsattr -El fcs1 -a num_cmd_elems

num_cmd_elems 400 Maximum Number of COMMAND Elements True

But kdb still has num_cmd_elems set to C8 (200).

# echo vfcs fcs1 | kdb | grep num_cmd_elems

num_cmd_elems: 0xC8 location_code: U9119.FHA.87654A1-V20-C10-T1

Using this technique I now have a way of checking if an AIX system has been restarted since an attribute was changed on a VFC adapter.

At this point I could suggest a reboot of the system. Or I could take the adapter offline and then online for the changes to take effect. For example:

# lspath

Enabled hdisk0 fscsi0

Enabled hdisk1 fscsi0

Enabled hdisk0 fscsi1

Enabled hdisk1 fscsi1

# rmpath -l hdisk0 -p fscsi1

# rmpath -l hdisk1 -p fscsi1

# lspath

Enabled hdisk0 fscsi0

Enabled hdisk1 fscsi0

Defined hdisk0 fscsi1

Defined hdisk1 fscsi1

# rmdev -Rl fcs1

fscsi1 Defined

fcs1 Defined

# cfgmgr

# lspath

Enabled hdisk0 fscsi0

Enabled hdisk1 fscsi0

Enabled hdisk0 fscsi1

Enabled hdisk1 fscsi1

# lsattr -El fcs1 –a num_cmd_elems

num_cmd_elems 400 Maximum Number of COMMAND Elements True

# echo vfcs fcs1 | kdb | grep num_cmd_elems

num_cmd_elems: 0x190 location_code: U9119.FHA.87654A1-V20-C10-T1

;; 0x190 hex = 400 decimal

Please note: Be careful when using the kdb command. If used incorrectly, you can crash an AIX system!

Does AIX support passwords longer than 8 characters? Yes!

cggibbo Tags: /etc/security/pwdalg.cfg characters smd5 pwd_algorithm /etc/security/login.cfg aix /usr/lib/security/smd5 support clpasswd gibson 8 greater chris than passwords for 2 Comments 32,019 Views

Prior to AIX 5.3 TL7 and AIX 6.1, there was an 8 character limit on AIX user passwords. If you need passwords of greater than 8 characters then you must enable one of the supplied Loadable Password Algorithms (LPAs). The following table lists the available algorithms and the limitations of each:

For example, to enable the MD5 algorithm I can modify /etc/security/login.cfg file with the chsec command as follows:

# chsec -f /etc/security/login.cfg -s usw -a pwd_algorithm=smd5

# tail -2 /etc/security/login.cfg

pwd_algorithm = smd5

This algorithm (smd5) will allow a password limit of 255 characters. Each of the available algorithms is listed in the /etc/security/pwdalg.cfg file.

# head -50 /etc/security/pwdalg.cfg

*******************************************************************************

* Loadable Password Algorithm (LPA)

* The format of a LPA stanza is:

* lpa:

* lpa_module = /any/program

* lpa_options = options_list

* lpa Corresponds to a password hashing algorithm that can be

* specified in the "usw" stanza of /etc/security/login.cfg

* as the system-wide password hashing algorithm.

* lpa_module Specifies the path of the password hashing load module.

* In the above example, /any/program is the path to the load

* module that is used to do the password hashing in 32 bit

* mode. The load module that is used to run in 64 bit mode

* is /any/program_64.

* lpa_options Is an optional attribute that provides a means of

* communicating run-time configuration options to the load

* module. The value is a comma separated list of items.

* Please refer to the documentation for the load module for

* appropriate values.

*******************************************************************************

* /usr/lib/security/smd5 is a password hashing load module using

* the MD5 algorithm.

* It supports password length up to 255 characters.

* To generate smd5 password hash compatible to standard salted MD5,

* add the following option line for smd5 stanza.

* lpa_options = std_hash=true

* Note : password hash generated with this option won't be compatible with

* hash generated without this option.

*******************************************************************************

smd5:

lpa_module = /usr/lib/security/smd5

*******************************************************************************

* /usr/lib/security/ssha is a password hashing load module using SHA and

* SHA2 algorithms. It supports password length up to 255 characters.

* This LPA accepts three options. The options are separated by commas.

...etc...

Once you’ve enabled the LPA of your choice, and you set/change a users’ password, you’ll notice that the /etc/security/passwd stanza for that user will look different when compared to the stanzas of users that have not had their password set/changed using the new LPA:

fred:

password = E7nOaTrrz9Q16

lastupdate = 1330986703

flags = ADMCHG

joe:

password = {smd5}z9JrHDJB$Oq/cZXr0jUyAWvfFyjt161

lastupdate = 1330987903

flags = ADMCHG

In the example above, user joe’s password has been set using the smd5 algorithm.

For those of you who run PowerHA (HACMP) and are thinking about using one of the LPAs with the clpasswd utility, you may want to review this APAR first:

IZ40362: HACMP PASSWD UTILITY FAILS WHEN LONG PASSWD ALGORITHM IS ENABLED

https://www-304.ibm.com/support/docview.wss?uid=isg1IZ40362

The APAR states “HACMP cluster-wide C-SPOC password administration does not support use of the feature allowing passwords longer than 8 characters which became available with the Loadable Password Algorithm as part of AIX 53 TL 7.”

The last time I tested this with PowerHA, the problem was that the password entry in /etc/security/passwd was corrupted/truncated when a users password was changed using the clpasswd utility.

For example, if the passwd utility is linked to clpasswd and I changed a users password, the password field appeared to be corrupted/truncated and the user could not log in successfully:

# ls -l /usr/bin/passwd

lrwxrwxrwx 1 root system 39 Dec 22 11:30 /usr/bin/passwd -> /usr/es/sbin/cluster/utilities/clpasswd

# passwd tom

Changing password for "tom"

tom's New password:

Enter the new password again:

# grep -p tom passwd

tom:

password = {smd5}FU2MWa.N.

lastupdate = 1229910298

flags =

tom@aix16's password:

Access denied

But if we use the original passwd command (stored in /usr/es/sbin/cluster/etc/clpasswd), the users password is changed successfully and the user can log in OK.

root@aix16 /usr/es/sbin/cluster/etc/clpasswd # ./usr_bin_passwd.orig tom

Changing password for "tom"

tom's New password:

Enter the new password again:

root@aix16 /usr/es/sbin/cluster/etc/clpasswd # grep -p tom /etc/security/passwd

tom:

password = {smd5}PcBK/5QW$RE09Q0xz4TGQFp98KjxSs.

lastupdate = 1229910818

flags = ADMCHG

Login OK!

$ id

uid=10003(tom) gid=1(staff)

usr_bin_passwd.orig password = {smd5}HcMHTTu3$BWkU8JVCWgT8szAN8TGCF1

clpasswd password = {smd5}6z61fiIn/070

I’ve not tried this again recently but I am curious if the same behaviour can be expected on a PowerHA system today. When I first encountered this problem (in 2008) I opened a PMR for the issue. In that call I was told that the “clpasswd utility is corrupting the encrypted password when distributing to the nodes, so that a login fails”. I’ll configure a HA cluster soon and try it again with PowerHA 6.1 and AIX 6.1 and report back with the results.

UPDATE: I built a HA 6.1 cluster (on AIX 6.1) this afternoon in my lab and tested this successfully. Based on the tests I’ve performed so far, it appears that this limitation no longer exists. Thanks to hafeedbk@us.ibm.com for the help on this one.

The following IBM tech note has more information on the available Loadable Password Algorithms and support for longer than 8 character passwords on AIX:

AIX support for passwords greater than 8 characters

http://www-01.ibm.com/support/docview.wss?rs=111&context=SWG10&q1=pwd_algorithm&uid=isg3T1010741&loc=en_US&cs=utf-8&lang=en

Check your WPAR filesets

cggibbo Tags: un-supported hdisk5 type. mkwpar: has wio.vscsi 0960-587 wio.fco wio.common subclass 2 Comments 6,454 Views

Got an issue with mkwpar? Have you checked your WPAR filesets?

If you attempt to create a rootvg WPAR with a VSCSI disk, on a system running AIX 6.1 TL6 and you encounter the following error:

# mkwpar –O -D rootvg=yes devname=hdisk5 -n rootvgwpar1

**********************************************************************

ERROR

mkwpar: 0960-587 hdisk5 has un-supported subclass type.

**********************************************************************

Then you should check that the following filesets are installed:

# lslpp -L | grep wio
wio.common    6.1.6.0    C F Common I/O Support for
wio.fcp 6.1.6.0    C F FC I/O Support for Workload
wio.vscsi 6.1.6.0    C F VSCSI I/O Support for Workload

Once these filesets are installed (from the AIX 6.1 install media), then you should be able to create a rootvg WPAR using a VSCSI disk.

Convert rootvg JFS file systems to JFS2 with alt_disk_copy

cggibbo Tags: alt_disk_copy chris convert jfs2 aix gibson jfs 2 Comments 13,249 Views

Starting with AIX 6.1 (TL4?), you can use the alt_disk_copy command to convert rootvg file systems from jfs to jfs2. The man page for the alt_disk_copy command states the following:

-T Indicates that you want to convert JFS file systems to JFS2 file systems during the process of recreating the rootvg volume group on target disks.

As an example, to clone a running systems rootvg to hdisk2, and convert the rootvg file systems from JFS to JFS2, you would run the following command:

# alt_disk_copy -T -d hdisk2

Good to know!

Has your IBM XL C/C++ for AIX software evaluation license expired?

cggibbo Tags: chris aix evaluation expired gibson xlc license 2 Comments 15,566 Views

Has your IBM XL C/C++ for AIX software evaluation license expired?

You can work around this little problem temporarily by placing the following environment variable in /etc/profile.

XLC_EXTEND_EVAL=yes

According to the IBM support site, this will extend the XLC license for another 74 days.

https://www-304.ibm.com/support/docview.wss?uid=swg21498670

It appears to work.

Before the change, I was unable to compile anything. Note the highlighted text below.

# /usr/vac/bin/xlc cgc.c

The license for the Evaluation version of IBM XL C/C++ for AIX V11.1 compiler product has expired. Please send an email to compiler@ca.ibm.com for information on purchasing the product. The evaluation license can be extended to 74 days in total by either a) setting an environment variable XLC_EXTEND_EVAL=yes; or, b) specifying a compiler command line option -qxflag=extend_eval. The extended evaluation license will expire on Sat Jun 18 10:26:33 2011. Use of the Program continues to be subject to the terms and conditions of the International License Agreement for Evaluation of Programs, including the accompanying License Information document (the "Agreement"). A copy of the Agreement can be found in the "LicAgree.pdf" and "LicInfo.pdf" files residing in the root directory of the installation media. If you do not agree to the terms and conditions of the Agreement, you may not use or access the Program.

With the environment variable in place, I was able to compile again.

# /usr/vac/bin/xlc cgc.c

# ./a.out

Hello World!

Of course I could also export the environment variable as required instead e.g.

# export XLC_EXTEND_EVAL=yes

# /usr/vac/bin/xlc cgc.c

# ./a.out

Hello World!

If you do place this environment variable in /etc/profile, you may need to restart any processes on the system that need to call the compiler.

VIOS/AIX Matrix

cggibbo Tags: nim vios aix 2 Comments 18,239 Views

Here’s a nice little matrix of current VIOS levels and the corresponding AIX level.

The following link also has some fantastic information relating to NIM installation and backup of a Virtual I/O Server (VIOS).

https://www-304.ibm.com/support/docview.wss?uid=isg3T1011386

It has an official answer for one of the questions I’m asked the most:

Q: Do I need to use an lpp_source to install the VIO server using NIM?

A: No. All of the device drivers and software packages needed to install the VIO server are in the VIO server mksysb.

I also recommend Rob McNelly’s latest blog on installing VIO servers.

http://ibmsystemsmag.blogs.com/aixchange/2011/01/an-unusual-vios-install.html

AIX 7 Open Beta is live - it's time to test!

cggibbo Tags: chris 7 beta gibson aix open 2 Comments 10,427 Views

OK, it’s official. The AIX 7 Open Beta is now available for testing!

An overview of the program is available here:

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/aix7ob/index.shtml

Essentially this program is designed to give IBM customers, ISVs and IBM BPs the opportunity to gain early experience with the latest release of AIX prior to general availability. This is a great time to join forces and help IBM mould the next generation of the AIX OS.

I got involved in the AIX 6 Open Beta back in 2007. It was a worthwhile experience. The time I spent learning new features like WPARs and RBAC, put me in a good position when it came time to actually implement these outside of my lab environment. It was also a good opportunity to provide feedback to the IBM AIX development community. Several AIX developers monitored the comments/questions in the Beta Forum and provided advice (and sometimes fixes) for known (and unknown!) issues with the beta release. It also provided the developers with plenty of real world feedback that they could take back to the labs, long before the product was officially released. This certainly helped fix bugs and improve certain enhancements before customers starting using the OS in their computing environments.

The AIX 7 beta code is available for download from here:

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/aix7ob/download.shtml

The user forum, where you can provide feedback and ask questions, is here:

https://www.ibm.com/developerworks/forums/forum.jspa?forumID=2255

I recommend you download the documentation and start by reading the Getting Started guide. The documentation is available here:

https://www14.software.ibm.com/iwm/web/cc/earlyprograms/websphere/aix7ob/library.shtml

The Getting Started guide provides useful information that you will need to know before attempting to install the OS. For example, the beta code will run on any IBM System p, eServer pSeries or POWER system that is based on PPC970, POWER4, POWER5, POWER6 or POWER7 processors.

The guide also describes what new functionality has been included in this release of the beta. If this program is anything like the AIX 6 beta, there may be more than one release of the code, with further enhancements available in each release. The new function in this release includes:

A. AIX 5.2 Workload Partitions for AIX 7.

B. Removal of WPAR local storage device restrictions.

C. Domain support in Role Based Access Control (RBAC).

D. Etherchannel enhancements across stacked switches in IEEE 802.3ad mode.

E. NIM thin server support for IPv6 and NFSv4.

I’m interested in testing A, B and D. Here’s why.

A. AIX 5.2 Workload Partitions for AIX 7.

AIX 5.2 Workload Partitions for AIX 7 provides the capability to create a WPAR running AIX 5.2 TL10 SP8. This allows a migration path for an AIX 5.2 system running on old hardware to move to POWER7. All that is required is to create a mksysb image of the AIX 5.2 system and then provide this image when creating the WPAR. The WPAR must be created on a system running AIX 7 on POWER7 hardware. This is a very interesting feature, one that I am eager to test.

B. Removal of WPAR local storage device restrictions.

AIX 7 will allow for exporting a virtual or physical fibre channel adapter to a WPAR. The WPAR will essentially own the physical adapter and its child devices. This will allow for SAN storage devices to be directly assigned to the WPAR's FC adapter(s). This means it will not be necessary to provision the storage in the Global environment first and then export it to the WPAR. This is also interesting as we may now be able to assign SAN disk to a WPAR for both rootvg and data volume groups. Maybe even FC tape devices within a WPAR will work?

D. Etherchannel enhancements in 802.3ad mode.

There are some enhancements to AIX 7 EtherChannel support for 802.3AD mode. The enhancement makes sure that the link is LACP ready before sending data packets. If I’m interpreting this correctly, this will ensure that the aggregated link is configured appropriately. If it’s not, it will provide an error in the AIX errpt stating that the link is not configured correctly. This can help avoid situations where the AIX EtherChannel is configured but the Network Switch is not. At present there is very little an AIX administrator can do as the link will appear to be functioning even if the Switch end has not been configured for an aggregated link.

And the most important point in the Getting Started guide has to be, how to install the AIX beta code! An ISO image of the code is provided for download. The installation steps are straightforward as the image can be installed via a DVD device. Using a media repository on a VIO server could be one way to accomplish this task. Unfortunately, there is no mention of NIM install support yet. Here are the basic steps from the guide:

Installing the AIX Open Beta Driver

The AIX 7 Open Beta driver is delivered by restoring a system backup (mksysb) of the code downloaded via DVD ISO image from the AIX Open Beta web-site.

Once you have downloaded and created the AIX 7 Open Beta media (as described above) follow the following steps to install the ‘mksysb’.

1. Put the DVD of the AIX Open Beta in the DVD drive. A series of screens/menus will be displayed. Follow the instruction on the screens and make the following selections:

• Type 1 and press Enter to have English during install.

• Type 1 to continue with the install.

• Type 1 to Start Install Now with Default Setting.

2. The system will start installing the AIX 7.0 BETA.

3. Upon completion of the install, the system will reboot. You can then login as “root”, no password is required.

Next I recommend that you take a look at the Release Notes. It provides a few bits of information that may come in handy when planning for the install, such as:

· The Open Beta code is being delivered via an “mksysb” install image. Migration installation is not supported with the open beta driver.

· The open beta driver does not support IBM Systems Director Agent.

· When installing in a disk smaller than 15.36 GB, the following warning is displayed: A disk of size 15360 was specified in the bosinst.data file, but there is not a disk of at least that size on the system. You can safely ignore this warning.

· The image is known to install without issues on an 8 GB disk.

· oslevel output shows V7BETA.

By the way, if you are unable to find a spare system or LPAR on which to install the beta, perhaps you can consider using the IBM Virtual Loaner Program (VLP). They are planning to support LPARs running the AIX 7 Open Beta starting from July 17^th. I use the VLP all the time and found it be a fantastic way to try new things without the need for, or expense of, my own IBM POWER system. There are some drawbacks, such as not having access to your own dedicated hardware, HMC, VIO server, NIM master, but still it’s great if you just want to test something on an AIX system.

I’ll report back once I’ve got my AIX 7 Open Beta system up and running!

New Features in AIX Version 7.

cggibbo 2 Comments 11,092 Views

Today, IBM announced AIX version 7. http://www-03.ibm.com/systems/power/software/aix/v71/preview.html

Several new features were mentioned in the launch, but there were two new features that I found particularly interesting:

- AIX 5.2 WPARs for AIX 7

- Cluster Aware AIX

I thought I would briefly describe each feature in this post.

AIX 5.2 WPARs for AIX 7

In AIX version 7, administrators will now have the capability to create Workload Partitions (WPARs) that can run AIX 5.2, inside an AIX 7 operating system instance. This will be supported on the POWER7 server platform. This is pretty cool. IBM have done this to allow some customers, that are unable to migrate to later generations of AIX and Power, to move to POWER7 whilst keeping their legacy AIX 5.2 systems operational. So for those clients that MUST stay on AIX 5.2 (for various reasons such as Application support) but would like to run their systems on POWER7, this feature may be very attractive. It will help to reduce the effort required when consolidating older AIX 5.2 systems onto newer hardware. It may also reduce some of the risk associated with migrating applications from one version of the AIX operating system to another.

To migrate an existing AIX 5.2 system to an AIX 7 WPAR, administrators will first need to take a mksysb of the existing system. Then they can simply restore the mksysb image inside the AIX 7 WPAR. IBM will also offer limited defect and how-to support for the AIX 5.2 operating system in an AIX 7 WPAR. These WPARs can, of course, be managed via IBM Systems Director with the Workload Partitions Manager plug-in.

The following figure provides a visualization of how these AIX 5.2 systems will fit into an AIX 7 WPAR. The WPARs in blue are native AIX 7 WPARs, while the WPARs in orange are AIX 5.2 WPARs running in the same AIX 7 instance. Pretty amazing really!

I can only speculate as to what other versions of AIX will be supported in this manner in the future. Just imagine an AIX 5.3 WPAR inside AIX7?

Cluster Aware AIX

Another very interesting feature of AIX 7 is a new technology known as “Cluster Aware AIX”. Believe it or not, administrators will now be able to create a cluster of AIX systems using features of the new AIX 7 kernel. IBM have introduced this “in built” clustering to the AIX OS in order to simplify the configuration and management of highly available clusters. This new AIX clustering has been designed to allow for:

- The easy creation of clusters of AIX instances for scale-out computing or high availability.

- Significantly simplify cluster configuration, construction, and maintenance.
- Improve availability by reducing the time to discover failures.

- Capabilities such as common device naming to help simplify administration.

- Built in event management and monitoring.

- A foundation for future AIX capabilities and the next generation of PowerHA SystemMirror.

This does not replace PowerHA but it does change the way in which AIX traditionally integrates with cluster software like HACMP and PowerHA. A lot of the HA cluster functionality is now available in the AIX 7 kernel itself. However, the mature RSCT technology is still a component of the AIX and PowerHA configuration. I’m looking forward to reading more about this new technology and it’s capabilities.

These are just two of the many features introduced in AIX 7. I’m eagerly looking forward to what these features and others mean for the future of the AIX operating system. It’s exciting to watch this operating system grow and strengthen over time. I can’t wait to get my hands on an AIX 7 system so that I can trial these new features.

And speaking of trialing AIX 7, there is good news. IBM plan on running another AIX Open Beta program for AIX 7 mid 2010. Just as they did with AIX Version 6, customers will be given the opportunity to download a beta version of AIX 7 and trial it on their own systems in their own environment. This is very exciting and I’m really looking forward to it.

I encourage you to read the official AIX 7 announcement to learn more about the future of the AIX operating system and what it can do for you and your organization…..for many years to come!

VIOS 2.1 FP22 and VMLibrary.

cggibbo 2 Comments 11,192 Views

Last month I applied the latest fix pack for VIOS 2.1 i.e. FP22, to one of our lab VIOS.

I came across the following error while doing so:

Renaming Media Repository Logical Volume:

VMLibrary_LV --> VMLibrary

umount: error unmounting /dev/VMLibrary_LV: Device busy

Media Repository failed to unmount. We cannot continue

until it can be successfully unmounted. Please address

and retry.

update: Failed while executing the ios.cli.rte.post_u script.

installp: The installation has FAILED for the "root" part

of the following filesets:

ios.cli.rte 6.1.4.1

Despite the error, the ioslevel command reported that the fix pack had been applied. This was misleading, as a critical fileset had not been updated i.e. ios.cli.rte.

$ ioslevel

2.1.2.10-FP-22

To workaround this problem I removed the Virtual Media Repository on the VIOS.

$ rmrep

As the ios.cli.rte fileset had not installed properly, I had to reinstall the base fileset and then apply the update (fix pack) again.

----

1 Total to be installed

+-----------------------------------------------------------------------------+

Installing Software...

+-----------------------------------------------------------------------------+

installp: APPLYING software for:

ios.cli.rte 6.1.2.0

. . . . . << Copyright notice for ios.cli >> . . . . . . .

Licensed Materials - Property of IBM

5765G3400

US Government Users Restricted Rights - Use, duplication or disclosure

restricted by GSA ADP Schedule Contract with IBM Corp.

. . . . . << End of copyright notice for ios.cli >>. . . .

0 * * * * /usr/ios/cli/cron_mail_check.sh 2>/dev/null

root

padmin

Finished processing all filesets. (Total time: 18 secs).

+-----------------------------------------------------------------------------+

Summaries:

+-----------------------------------------------------------------------------+

Installation Summary

--------------------

Name Level Part Event Result

-------------------------------------------------------------------------------

ios.cli.rte 6.1.2.0 USR APPLY SUCCESS

ios.cli.rte 6.1.2.0 ROOT APPLY SUCCESS

# lslpp -l ios.cli.rte

Fileset Level State Description

----------------------------------------------------------------------------

Path: /usr/lib/objrepos

ios.cli.rte 6.1.4.1 COMMITTED Virtual I/O Technologies

Path: /etc/objrepos

ios.cli.rte 6.1.4.1 COMMITTED Virtual I/O Technologies

Then I recreated the media repository with mkrep.

$ mkrep -sp rootvg -size 10G

At the time I thought perhaps I had done something wrong. I forgot all about it, until now. Today (a month after my experience), IBM updated the VIOS support page with the following information:

http://www14.software.ibm.com/webapp/set2/sas/f/vios/download/fbmr.html

Known issues and limitations

File Backed Optical Media Repository: If you use the File Backed Optical Media Repository, you must unload any media images before you apply Fix Pack 22.1. If you have any media images loaded, the fix pack installation fails for the

ios.cli.rte fileset. The failure may not be apparent, because the ioslevel will be updated. Refer to this page for information to help you resolve this problem.

And they re-released FP22 as FP22.1.

http://www14.software.ibm.com/webapp/set2/sas/f/vios/download/fixpack.html

Fixes in Fix Pack 22.1 (VIOS V2.1.2.10 FP-22.1) include:

Fixed problem with Electronic Service Agent in IVM

Fixed problem in bus device driver for NPIV

Fixed problem in device driver pertaining to reading SSD disks (MPIO fixes)

Fixed problem in xmtopas (install problem)

Fixed packaging problem with the Media Repository

Fixed problem with backupios ? fails to give adequate details on stdout errors

New resolv.conf options in AIX 6.1

cggibbo 2 Comments 14,355 Views

FYI.

There are new options for /etc/resolv.conf in AIX 6.1 (not in 5.3):

...
timeout
        Enables you to specify the initial timeout for a query to a nameserver. The default value is five seconds. The maximum value is 30 seconds. For
        the second and successive rounds of queries, the resolver doubles the initial timeout and is divided by the number of nameservers in the
        resolv.conf file.

attempts
Enables you to specify how many queries the resolver should send to each nameserver in the resolv.conf file before it stops execution. The
default value is 2. The maximum value is 5.

rotate
Enables the resolver to use all the nameservers in the resolv.conf file, not just the first one.
...

Who cored?

cggibbo 2 Comments 8,021 Views

Ever wondered what process just core dumped?

The AIX errpt command will tell you where the core file has been dumped and in most cases the program name:

# errpt -a | head -45
---------------------------------------------------------------------------
LABEL: CORE_DUMP
IDENTIFIER: C69F5C9B

Date/Time:       Wed Jul 22 13:41:52 EET 2009
Sequence Number: 2694
Machine Id:      00C01C704C00
Node Id:         hxaix14
Class:           S
Type:            PERM
Resource Name:   SYSPROC

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SOFTWARE PROGRAM

User Causes
USER GENERATED SIGNAL

Recommended Actions
CORRECT THEN RETRY

Failure Causes
SOFTWARE PROGRAM

        Recommended Actions
        RERUN THE APPLICATION PROGRAM
        IF PROBLEM PERSISTS THEN DO THE FOLLOWING
        CONTACT APPROPRIATE SERVICE REPRESENTATIVE

Detail Data
SIGNAL NUMBER
          11
USER'S PROCESS ID:
               1646842
FILE SYSTEM SERIAL NUMBER
          19
INODE NUMBER
       18736
CORE FILE NAME
/usr/sap/SMD/J98/SMDAgent/core
PROGRAM NAME
jlaunch

But what if the AIX error report has been cleared out and all you have is the core file. How do you know who generated the core file?

Once you've found the core file, run the following command to determine what command and/or process failed:

# cd /usr/sap/SMD/J98/SMDAgent/

# lquerypv -h core 6b0 64
000006B0   7FFFFFFF FFFFFFFF 7FFFFFFF FFFFFFFF |................|
000006C0   00000000 000007D0 7FFFFFFF FFFFFFFF |................|
000006D0   00120000 1A7743F8 00000000 00000003 |................|
000006E0   736C6565 70000000 00000000 00000000 |jlaunch.........|
000006F0   00000000 00000000 00000000 00000000 |................|
00000700   00000000 00000000 00000000 00000027 |...............'|
00000710   00000000 00000001 00000000 00000027 |...............'|

Online quorum change

cggibbo Tags: rootvg aix mirror quorum 2 Comments 10,457 Views

For those who weren't aware.....starting with AIX 5.3 TL7, quorum changes are allowed online without having to varyoff and varyon the volume group. This also means no reboot is required when changing quorum for rootvg. Of course this also applies to AIX 6.1.

What are those AIX fcstat error numbers all about?

cggibbo Tags: fcstat gibson errno chris vios aix 1 Comment 3,589 Views

When troubleshooting Fibre Channel adapter issues on AIX (or VIOS), I often use the fcstat command to assist me in the process. The command may display error numbers similar to the following:

$ fcstat fcs0

Error opening device: /dev/fscsi0

errno: 00000045

The error number (errno) displayed can (in some cases) be used to identify the root cause of a problem. In the example above, the error number is 45. This number is in hex. If we convert this to decimal, the number is 69. Now, if we look for 69 in the /usr/include/errno.h file (on AIX), we discover that this error number relates to a "Network is down" event.

# fcstat fcs0

Error opening device: /dev/fscsi0

errno: 00000045

# echo "ibase=16; 45"|bc

# grep 69 /usr/include/errno.h

#define ENETDOWN 69 /* Network is down */

The AIX error report (errpt or errlog on VIOS) also tells me that there's some type of link error. This helps me focus my investigation toward the most likely problem area. In this case, I suspect either a physical link or cable problem between my FC adapter and the SAN switch.

IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION

7BFEEA1F 1116121816 T H fcs0 LINK ERROR

# errpt -aN fcs0 | grep -p Desc

...

Description

LINK ERROR

We can use the same process for other errors displayed by the fcstat command. For example:

# fcstat fcs0

Error opening device: /dev/fscsi0

errno: 00000046

# echo "ibase=16; 46"|bc

# grep 70 /usr/include/errno.h

#define ENETUNREACH 70 /* Network is unreachable */

This particular error may direct us to the solution outlined in the following tech note:

Fibre channel workspace takes over 2 minutes to load but returns no data

http://www-01.ibm.com/support/docview.wss?uid=swg21600064

fcstat Command (AIX 6.1)

http://www.ibm.com/support/knowledgecenter/ssw_aix_61/com.ibm.aix.cmds2/fcstat.htm

fcstat Command (AIX 7.1)

http://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.cmds2/fcstat.htm

fcstat Command (AIX 7.2)

http://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.cmds2/fcstat.htm

Modified by cggibbo

Help! My mksysb is only backing up 4 files!

cggibbo Tags: chris up backing files blog gibson aix 4 mksysb 1 Comment 4,894 Views

I enjoy it when I open my email in the morning and find a new message with a subject line of “weird one….”! I immediately prepare myself for whatever challenge awaits. Fortunately I do delight in helping others with their AIX challenges so I usually open these emails first and start to diagnose and troubleshoot the problem!

This week I was contacted by someone that was having a little trouble with a mksysb backup on one of their AIX systems.

“Hi Chris,

This one has me stumped, any ideas? I’ll have to log a call I think as I’m not sure why this is happening. I run a mksysb and it just backs up 4 files! I also can’t do an alt_disk_copy that also fails.

My /etc/exclude.rootvg is empty.

# cat /etc/exclude.rootvg
# mksysb -i /mksysb/aixlpar1-mksysb

Creating information file (/image.data) for rootvg.

Creating list of files to back up.

Backing up 4 files

4 of 4 files (100%)
0512-038 mksysb: Backup Completed Successfully.

# lsmksysb -f /mksysb/aixlpar1-mksysb
New volume on /mksysb/aixlpar1-mksysb:
Cluster size is 51200 bytes (100 blocks).
The volume number is 1.
The backup date is: Wed Oct 21 22:12:04 EST 2015
Files are backed up by name.
The user is root.
5911 ./bosinst.data
11 ./tmp/vgdata/rootvg/image.info
11837 ./image.data
270567 ./tmp/vgdata/rootvg/backup.data
The total size is 288326 bytes.
The number of archived files is 4.”

Yep, that is a weird one! J I replied with the following tips.

“Hi,

A couple of things to check and try.....

1. Check the mount command is OK. i.e.

# ls -ltr /usr/sbin/mount
-r-sr-xr-x 3 root system 67040 Aug 18 15:52 /usr/sbin/mount << Not zero size?

# mount

2. Check the find command is OK i.e.

# ls -ltr /usr/bin/find
-r-xr-xr-x 1 bin bin 58608 Aug 18 15:44 /usr/bin/find << Not symlink'ed?

3. Delete the /etc/exclude.rootvg file. Try mksysb again. Does it work?

4. Riun mkszfile and check image.data. Are fs data stanzas listed for each file system?

5. Try with verbose and debug enabled on mksysb/mkszfile commands.

# cd /usr/bin
# cp -p mksysb mksysb.orig
# cp -p mkszfile mkszfile.orig

# vi mksysb

; search for main

#################### main ##########################

#
# set up environment
#
PATH=/usr/bin:/usr/sbin:/sbin:/etc: ; export PATH
export ODMDIR="/etc/objrepos"
NAME=`/usr/bin/basename $0`
PLATFORM=`/usr/sbin/bootinfo -a` # Needed for IA64 (value 4)

Add the following:

#################### main ##########################

set -x
for F in $(typeset +f); do typeset -ft $F; done

#
# set up environment
#

; Do the same (above) for mkszfile.

; Save the files and run mkszfile and mksysb with script (to capture the output).

# script /tmp/mksysb.out
# mkszfile
# mksysb -i -v /mksysb/aixlpar-mksysbimg
# exit”

A short time later I received another email stating that the problem had been resolved! Curiously, the find command had been (somehow) destroyed on this system.

“LLLLLLLLLLLEEEEEEEEEEEGGGGGGGGGGGGEEEEEEEEEEENNNNNNNNNNNDDDDDDDDDDDDDDDDD!!!!!!!!!!!!!!!!!!!!!!!!

/usr/bin/find was a zero size file!!! WTF!

I just copied the file from another LPAR and it worked perfect.

THANKS!!!!”

I thought I’d share this information here, just in case anyone else comes across a similar problem in the future.

Modified by cggibbo

MDS reports

cggibbo Tags: invscout device report file mup gibson adapter mic power aix firmware microcode mds catalog chris 1 Comment 11,218 Views

I like using the IBM Microcode Discovery Service (MDS) to perform spot checks on POWER system and adapter firmware levels. This tool gives me snapshot of a POWER systems current firmware levels. I can tell almost immediately if I need to update the firmware on the system or any device installed in the server. This report is especially useful when performing a health check on a customer’s system.

The tool is easy to use. First you need to download the latest inventory scout catalog file from the IBM website. The most current microcode catalog can be downloaded here:

ftp://ftp.software.ibm.com/software/server/firmware/catalog.mic

Once downloaded, copy the catalog file to an AIX (or VIOS) partition on the physical server. In most cases, a Virtual I/O Server (VIOS) is the most logical place to copy this file. The VIOS typically owns all the physical adapter devices on the system.

# scp catalog.mic padmin@vio1:

padmin@vio1's password:

catalog.mic 100% 263KB 263.2KB/s 00:00

Next, login to the VIOS and move the new catalog file into the correct location for use with the inventory scout (invscout) tool.

# ssh padmin@vio1

padmin@vio1's password:

$ ls -ltr

total 968

drwxr-xr-x 3 padmin staff 256 May 17 21:20 tivoli

-rw-r--r-- 1 padmin staff 2389 Aug 19 04:43 smit.transaction

-rw-r--r-- 1 padmin staff 1290 Aug 19 04:43 smit.script

-rw-r--r-- 1 padmin staff 11247 Aug 27 11:57 smit.log

drwxr-xr-x 2 padmin staff 4096 Sep 15 15:26 SV810_081

-rw-r--r-- 1 root system 5103 Sep 15 15:53 install.log

drwxr-xr-x 2 padmin staff 256 Sep 15 15:53 fixes

-rw-r--r-- 1 padmin staff 176510 Sep 15 18:34 errlog.out

drwxrwxr-- 2 root staff 256 Sep 16 13:25 config

-rw-r----- 1 padmin staff 269478 Sep 17 20:18 catalog.mic

-rw-r--r-- 1 root staff 5143 Sep 17 20:18 ioscli.log

$ r oem

oem_setup_env

# ls -ltr

total 8

-rw-r--r-- 1 root system 88 May 17 20:26 catalog.mic

# cd /var/adm/invscout/microcode

# mv catalog.mic catalog.mic.old

# cp /home/padmin/catalog.mic .

# ls -ltr

total 536

-rw-r--r-- 1 root system 88 May 17 20:26 catalog.mic.old

-rw-r----- 1 root staff 269478 Sep 17 20:20 catalog.mic

Now all we need to do is run the invscout command to create a new MDS upload file (MUP file).

# invscout

****** Command ---- V2.2.0.20

****** Logic Database V2.2.0.2

Initializing ...

Identifying the system ...

Working ...

Getting system microcode level(s) ...

Scanning for device microcode level(s) ...

117 devices detected; each dot (.)

represents 10 devices processed:

...........

Writing Microcode Survey upload file ...

Microcode Survey complete

The output files can be found at:

Upload file: /var/adm/invscout/vio1.mup

Report file: /var/adm/invscout/invs.mrp

Report file: /var/adm/invscout/invs.mrrup

To transfer the invscout 'Upload file' for microcode

comparison, see your service provider's web page.

The new MUP file (vio1.mup) can be uploaded to the MDS tool at the following URL:

https://www-304.ibm.com/webapp/set2/mds/fetch?page=mdsUpload.html

In the following screenshot I’ve selected the file to be uploaded. Then I clicked on ‘Upload File’.

I’m presented with the MDS report screen.

Immediately my eyes are drawn to the items highlighted in red. Both the PCIe2 4-Port (10GbE SFP+ & 1GbE RJ45) and PCIe3 RAID SAS Adapter have new firmware available for them. I should update the firmware on these devices! But what fixes are included in the new firmware? Well, directly below the devices report is the ‘Microcode by Type’ table.

I can simply click on the ‘Readme’ hyperlink in the report and a new browser window opens with the readme file for the adapters firmware.

So, should I install these fixes right away? How critical are they? Again, the MDS report provides you with a quick way to answer these questions. Under the ‘Severity’ column, you will see several terms that describe the severity of the related fix. If you click on the word ‘Severity’ you’ll be presented with an ‘Understanding the MDS Report’ window.

On this page you’ll find the definition and description of each severity type. This will make it easier for you to ascertain if a fix should be installed ASAP or if it can wait until the next maintenance window.

I often save the MDS report as a PDF and include it in my health check reports. Alternatively you can save the entire page as a HTML document and send this to the customer. This will ensure the hyperlinks still work and can be referenced by the customer as needed e.g.

file:///C:/data/MDS/CustomerName%20Microcode%20Discovery%20Service.htm.

By the way, if the microcode catalog file you used to generate the MDS report is not the latest, the MDS tool will warn you. If you see this message, simply download the latest file and send it over to the server where it’s needed and re-run invscout and upload the new MUP file e.g.

“WARNING: the microcode catalog used in surveying host vio1 is now out of date. Version 2014.09.10 was used for the survey and version 2014.09.17 is the most current. We recommend that you either download and install the latest microcode catalog, and run this program again, or run the MDS Applet and allow it to automatically update the catalog.”

MDS and Inventory Scout

https://www-304.ibm.com/webapp/set2/mds/fetch?page=invreadme.html

Modified by cggibbo

Thanks kdb but lsattr's got me covered!

cggibbo 1 Comment 7,572 Views

In 2011 I posted instructions on how to use kdb to compare a hdisk’s running queue depth value against the value stored in the AIX ODM. Something you often find if values are set but the system is not rebooted (or devices are not re-configured).

AIX 7.1 Technology Level 3 and AIX 6.1 Technology Level 9 introduce a new flag to the lsattr command. The new –P option will display the current ODM attribute settings i.e. this new option will display applied attributes that may not yet be in effect on the running system.

For example, we start with a hdisk, with a queue depth of 8. The lsattr command confirms that both the running and stored values match.

# lsattr -El hdisk1 -a queue_depth

queue_depth 8 Queue DEPTH True << ODM

# lsattr -Pl hdisk1 -a queue_depth

queue_depth 8 Queue DEPTH True << Running

The queue depth is then changed to 256 (using chdev –P, which only changes the stored ODM value). I can quickly detect this mismatch easily using lsattr. The command quickly highlights the differences.

# chdev -l hdisk1 -a queue_depth=256 -P

hdisk1 changed

# lsattr -El hdisk1 -a queue_depth

queue_depth 256 Queue DEPTH True << ODM

# lsattr -Pl hdisk1 -a queue_depth

queue_depth 8 Queue DEPTH True << Running

This method should work for other devices as well. For example, if I change the rw_timeout value on a VSCSI adapter, the lsattr command highlights the configuration differences before and after.

# lsattr -El vscsi0 -a rw_timeout

rw_timeout 120 Virtual SCSI Read/Write Command Timeout True

# lsattr -Pl vscsi0 -a rw_timeout

rw_timeout 120 Virtual SCSI Read/Write Command Timeout True

# chdev -l vscsi0 -a rw_timeout=3600 -P

vscsi0 changed

# lsattr -El vscsi0 -a rw_timeout

rw_timeout 3600 Virtual SCSI Read/Write Command Timeout True

# lsattr -Pl vscsi0 -a rw_timeout

rw_timeout 120 Virtual SCSI Read/Write Command Timeout True

Modified by cggibbo

Troubleshooting AIX hdisk configuration errors

cggibbo Tags: device specified liblpp.a does configuration odm hds the gibson not customized error vsp 0514-023 method database. exist /usr/lib/methods/cfgscsid... in pdpathat aix chris 1 Comment 13,348 Views

Last year I was working for a customer that was upgrading several AIX 5.3 systems to AIX 6.1. The migrations were successful for the most part, but we did encounter one issue that took a little time to resolve.

The customer was using nimadm to migrate. This process worked fine, however, on a couple of systems a strange error was encountered after the migration. The LPAR was booted into AIX 6.1 and everything came up fine. The applications were started and users started accessing the system.

It was several days later, when the AIX administrator attempted to configure new storage on the AIX system, when the first sign of trouble appeared. He had asked his Storage guy to assign a couple of new disks to his LPAR (via NPIV/VFC). As soon as the Storage admin had completed the assignment, the AIX admin ran cfgmgr to detect and configure the new hdisks. Immediately, cfgmgr reported the following error:

Method error (/usr/lib/methods/cfgscsidisk):

0514-023 The specified device does not exist in the

customized device configuration database.

Initially, the AIX team suspected there was some fault with either the storage device or the zoning of the disk. Both of these items were checked and doubled-checked and were found to be OK. Our next step was to run cfgmgr again, but this time we wanted a greater level of detail captured. To do this we used the following environment variable to force cfgmgr to be ‘more verbose’.

# export CFGLOG="cmd,meth,lib,verbosity:9"

We ran cfgmgr and went to the /var/adm/ras/cfglog file to view the results with the alog command. However, we noticed that the cfglog file had a size of zero (0) and contained no data.

# cd /var/adm/ras

# ls –l cfglog

-rw-r----- 1 root system 0 May 16 13:22 cfglog

We decided to recreate the cfglog alog file and run mkdev again to reproduce the disk configuration error.

# rm cfglog

# echo "Create cfglog `date`"|alog -t cfg

# mkdev -l hdisk0

Method error (/usr/lib/methods/cfgscsidisk):

0514-023 The specified device does not exist in the

customized device configuration database.

This time we found some useful data in the cfglog file.

# alog -t cfg -o

MS 31981804 28835876 /usr/lib/methods/cfgscsidisk -l hdisk39

M4 31981804 Parallel mode = 0

M4 31981804 Get CuDv for hdisk39

M4 31981804 Get device PdDv, uniquetype=disk/fcp/htcvspmpio

M4 31981804 Get parent CuDv, name=fscsi0

M4 31981804 ..is_mpio_capable()

M4 31981804 Device is MPIO

M4 31981804 ..get_paths()

M4 31981804 Getting CuPaths for name='hdisk39'

M4 31981804 Found 1 paths

M0 31981804 cfgcommon.c 225 mpio_init error, rc=23

MS 28835892 31981568 /usr/lib/methods/cfgscsidisk -l hdisk0

M4 28835892 Parallel mode = 0

M4 28835892 Get CuDv for hdisk0

M4 28835892 Get device PdDv, uniquetype=disk/fcp/htcvspmpio

M4 28835892 Get parent CuDv, name=fscsi0

M4 28835892 ..is_mpio_capable()

M4 28835892 Device is MPIO

M4 28835892 ..get_paths()

M4 28835892 Getting CuPaths for name='hdisk0'

M4 28835892 Found 2 paths

M0 28835892 cfgcommon.c 225 mpio_init error, rc=23

MS 25690326 27328608 /usr/lib/methods/cfgscsidisk -l hdisk0

M4 25690326 Parallel mode = 0

M4 25690326 Get CuDv for hdisk0

M4 25690326 Get device PdDv, uniquetype=disk/fcp/htcvspmpio

M4 25690326 Get parent CuDv, name=fscsi0

M4 25690326 ..is_mpio_capable()

M4 25690326 Device is MPIO

M4 25690326 ..get_paths()

M4 25690326 Getting CuPaths for name='hdisk0'

M4 25690326 Found 2 paths

M0 25690326 cfgcommon.c 225 mpio_init error, rc=23

The configuration method was attempting to configure a disk device type of htcvspmpio (which was correct) but it was unable to configure the device paths (mpio_init error rc=23). We suspected that the system was missing some sort device driver support for the type of storage in use.

Cutting a very long story short, we determined, with the help of the IBM AIX support team, that the issue stemmed from “old” AIX installation media used to create the AIX 6.1 TL6 SP5 SPOT and lppsource on the NIM master. Old AIX 6.1 media was originally used (several years ago) to create the NIM resources and was gradually updated over time, all the way up to TL6 SP5.

IBM support identified that the older install media contained a liblpp.a file that was missing the necessary PdPathAt ODM files. Newer install media contained a fix to add the appropriate entries to the bos.rte.cfgfiles. e.g.

SPOT created using OLD install media.

======================================

# ar xv /export/spot/spotaix610605_OLD/usr/lpp/bos/liblpp.a bos.rte.cfgfiles

x - bos.rte.cfgfiles

# grep PdPathAt bos.rte.cfgfiles

SPOT created using NEW install media.

======================================

# ar xv /export/spot/spotaix610605_NEW/usr/lpp/bos/liblpp.a bos.rte.cfgfiles

x - bos.rte.cfgfiles

# grep PdPathAt bos.rte.cfgfiles

/usr/lib/objrepos/PdPathAt v4preserve

/usr/lib/objrepos/PdPathAt.vc v4preserve

We recreated the SPOT and lppsource resources using newer media on the NIM master. We were then able to migrate the AIX 5.3 LPAR to 6.1 without encountering the issues faced previously.

Modified by cggibbo

seastat search option

cggibbo Tags: chris_gibson seastat vios powervm virtual_io_server vio 1 Comment 9,695 Views

It’s possible to search through seastat command output on the VIOS using the –s flag. You specify the following attributes with this flag: searchtype=value.

Valid search values include:

mac, priority, vlan, ip, host,

▼ Similar Blogs

▼ Similar Ideation Blogs

▼ Archive

▼ Blog Authors

Chris's AIX Blog

AIX support for passwords greater than 8 characters