Chris's AIX Blog

Tuning Virtual Ethernet adapters for (even) better backup performance.

cggibbo Tags: min_buf_huge=96 backup netstat slow hypervisor -a aix max_buf_huge=128 tsm chris_gibson min_buf_large=96 max_buf_large=256 networker min_buf_medium=512 -l adapter max_buf_medium=1024 virtual ent1 chdev –a ethernet 21 Comments 38,666 Views

Are you backing up your AIX systems over Virtual Ethernet adapters? Of course you are, who isn’t right? Are your backup server and clients on the same physical POWER system? You are most likely backing up over Virtual Ethernet to another AIX LPAR that is running your enterprise backup software, such as TSM or Legato Networker for example. And you probably have a dedicated private virtual network (and adapters) on both the clients and the server to handle the traffic for the nightly backups. The next question is, have you tuned your Virtual Ethernet adapters?

There are several tips available for tuning your Virtual Ethernet adapters for better performance on AIX. These tips include changing settings such as MTU size, TCP window sizes, enabling largesend, etc. I highly recommend the following blog posts from Anthony English and Nigel Griffiths on this subject:

https://www.ibm.com/developerworks/mydeveloperworks/blogs/AIXDownUnder/entry/turbo_charge_inter_lpar_communications_using_jumbo_frames1?lang=en

https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/10gbit_ethernet_bad_assumption_and_best_practice_part_137?lang=en

https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/powervm_virtual_ethernet_speed_is_often_confused_with_vios_sea_ive_hea_speed?lang=en

OK, so you got everything humming along nicely, your backups are flying over the virtual network (across the POWER hypervisor) and everybody is happy. After a period of time, you notice that the backups have started to “slow down”. They are taking longer to finish. The overall throughput of a backup drops. Some backups start in the evening around 9pm and are still running the next morning at 7am! In some cases you need to kill the backups or even reboot the backup server LPAR for things to return to normal.

“What is going on!?” You cry.

Well, there are a number of reasons why this could be happening. For example, your shared processor pool may be overwhelmed during the backup window. As we know, Virtual Ethernet adapters require CPU to do their work. If the CPU pool is running low on available CPU resources, this could contribute to the problem. And of course there could be tuning issues with the Virtual Ethernet adapters or the AIX OS in general. Or there may be issues with other pieces of the infrastructure, like network and SAN switches, adapters, etc. Perhaps there’s an issue with the applications and/or databases on the AIX systems? They often have their own mechanisms/tools for backing up their data to your enterprise backup software. Is the backup server sized to cope with the load i.e. CPU, memory, disk layout and I/O, sufficient tape drives, disk storage pools, etc?

So assuming you’ve checked all of the above (and more), then perhaps you’ve hit a problem that I encountered recently. In my particular case, backups “over the hypervisor” were slowing down, without any discernible cause. Initially the backups would be “very fast” but after a month or so, things would start to slow down dramatically.

We noticed that there were very large (and increasing) values for “Packets Dropped”, “Hypervisor Send/Receive Failures” and “No Resource Errors” in the output from the netstat –v command.

ETHERNET STATISTICS (ent1) :

Device Type: Virtual I/O Ethernet Adapter (l-lan)

Hardware Address: 41:ba:13:e7:25:0b

Elapsed Time: 42 days 4 hours 3 minutes 34 seconds

Transmit Statistics: Receive Statistics:

-------------------- -------------------

Packets: 5978589961 Packets: 26139832411

Bytes: 779465989202 Bytes: 711051516630458

Interrupts: 0 Interrupts: 6804561727

Transmit Errors: 0 Receive Errors: 0

Packets Dropped: 0 Packets Dropped: 86012309

...

Max Collision Errors: 0 No Resource Errors: 46113807

...

Hypervisor Send Failures: 0

Receiver Failures: 0

Send Errors: 0

Hypervisor Receive Failures: 46113807

After some discussion with IBM AIX support, we discovered that would should increase some of the buffer sizes for our Virtual Ethernet adapter (the entX device). This would alleviate the no resource issues we’d been experiencing. Looking at the output from the netstat -v command, we also noticed that the Medium, Large and Huge buffers had all reached their maximum values in the past.

...

Receive Information

Receive Buffers

Buffer Type Tiny Small Medium Large Huge

Min Buffers 512 512 128 24 24

Max Buffers 2048 2048 256 64 64

Allocated 513 535 148 28 64

Registered 512 510 127 24 13

History

Max Allocated 576 951 256 64 64

Lowest Registered 502 502 64 12 11

...

The advice from IBM support was to increase these buffers using the chdev command (they also advised that we should reboot for the changes to take effect):

# chdev -l ent1 -a min_buf_medium=512 -a max_buf_medium=1024 –a min_buf_large=96 -a max_buf_large=256 -a min_buf_huge=96 –a max_buf_huge=128 -P

# shutdown -Fr

Since implementing this tuning change (to the adapter on the backup server), we have not had a repeat of the problem. We will continue to monitor the performance and I’ll be sure to let everyone know if we have further issues.

Migrating to AIX 7.1 with nimadm.

cggibbo Tags: gibson migration chris nimadm aix 21 Comments 30,609 Views

Last year I wrote an article on using nimadm to migrate to AIX 6.1.

The procedures outlined how to migrate a live system with nimadm and a spare disk.

Another method that I have found to be extremely useful is the migration of mksysb images from one version of AIX to another using (you guessed it) nimadm.

In the following example I migrated an AIX 6.1 mksysb image to AIX 7.1 and then used that image to install a new LPAR via NIM.

The source LPAR was running AIX 6.1 TL6 SP3. I took a mksysb of this system to my NIM master.

# oslevel -s

6100-06-03-1048

The NIM mksysb resource was named cg-aix61

root@nim1 / # lsnim -t mksysb

vio1-mksysb resources mksysb

cg-aix61 resources mksysb

root@nim1 / # lsnim -l cg-aix61

cg-aix61:

class = resources

type = mksysb

arch = power

Rstate = ready for use

prev_state = unavailable for use

location = /export/mksysb/cg-aix61

version = 6

release = 1

mod = 6

oslevel_r = 6100-06

alloc_count = 0

server = master

creation_date = Thu Jan 6 10:00:22 2011

source_image = cg

root@nim1 / #

To migrate the mksysb image from AIX 6.1 to AIX 7.1, I ran the following nimadm command:

root@nim1 / # nimadm -T cg-aix61 -O /export/mksysb/cg-aix71 -s spotaix7tl0sp2 -l aix7tl0sp2 -j nimadmvg -Y -N cg-aix71

The flags indicate the following:

-T specifies the existing AIX 6.1 NIM mksysb resource.

-O specifies the output location for the migrated mksysb resource.

-s indicates the AIX 7.1 NIM SPOT resource for the migration.

-l indicates the AIX 7.1 NIM lpp_source resource for the migration.

-j identifies the volume group that will be used on the NIM master to create file systems.

-Y Agrees to required software license agreements for software to be installed.

-N specifies the name of the new AIX 7.1 NIM mksysb resource to be created after migration.

Before running this command I made sure that I had installed the bos.alt_disk_install.rte fileset into my AIX 7.1 SPOT. If this fileset was not installed I would have received the following message immediately after issuing the nimadm command:

0042-001 nim: processing error encountered on "master":

/usr/bin/lslpp: Fileset bos.alt_disk_install.rte not installed.

0505-204 nimadm: SPOT spotaix7tl0sp2 does not have bos.alt_disk_install.rte installed.

0505-205 nimadm: The level of bos.alt_disk_install.rte installed in SPOT

spotaix7tl0sp2 (0.0.0.0) does not match the NIM master's level (7.1.0.2).

Cleaning up alt_disk_migration on the NIM master.

I also had an AIX 7.1 lpp_source and SPOT ready and waiting on my NIM master.

root@nim1 / # lsnim -t lpp_source

aix61tl6sp3 resources lpp_source

openssh resources lpp_source

aix7tl0sp2 resources lpp_source

root@nim1 / # lsnim -t spot

vio1-spot resources spot

spotaix61tl6sp3 resources spot

spotaix7tl0sp2 resources spot

OK, back to my nimadm operation. Upon initiating the migration I saw the following output on my screen (click here for a copy of the complete log file):

root@nim1 / # nimadm -T cg-aix61 -O /export/mksysb/cg-aix71 -s spotaix7tl0sp2 -l aix7tl0sp2 -j nimadmvg -Y -N cg-aix71

Initializing the NIM master.

Verifying alt_disk_migration eligibility.

Initializing log: /var/adm/ras/alt_mig/cg-aix61_alt_mig.log

Starting Alternate Disk Migration.

+-----------------------------------------------------------------------------+

Executing nimadm phase 1.

+-----------------------------------------------------------------------------+

Processing target mksysb resource cg-aix61.

Restoring /image.data from mksysb image.

+-----------------------------------------------------------------------------+

Executing nimadm phase 2.

+-----------------------------------------------------------------------------+

Creating nimadm cache file systems on volume group nimadmvg.

Checking for initial required migration space.

Creating cache file system /cg-aix61_mm_alt/alt_inst

Creating cache file system /cg-aix61_mm_alt/alt_inst/admin

Creating cache file system /cg-aix61_mm_alt/alt_inst/home

Creating cache file system /cg-aix61_mm_alt/alt_inst/opt

Creating cache file system /cg-aix61_mm_alt/alt_inst/tmp

Creating cache file system /cg-aix61_mm_alt/alt_inst/usr

Creating cache file system /cg-aix61_mm_alt/alt_inst/var

Creating cache file system /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

+-----------------------------------------------------------------------------+

Executing nimadm phase 3.

+-----------------------------------------------------------------------------+

Syncing mksysb data to cache ...

.....etc......

+-----------------------------------------------------------------------------+

Executing nimadm phase 8.

+-----------------------------------------------------------------------------+

Creating client boot image.

bosboot: Boot image is 45709 512 byte blocks.

+-----------------------------------------------------------------------------+

Executing nimadm phase 9.

+-----------------------------------------------------------------------------+

Adjusting client file system sizes ...

Adjusting size for /

Adjusting size for /admin

Adjusting size for /home

Adjusting size for /opt

Adjusting size for /tmp

Adjusting size for /usr

Adjusting size for /var

Adjusting size for /var/adm/ras/livedump

Backing up cache data to mksysb file /export/mksysb/cg-aix71 ...

+-----------------------------------------------------------------------------+

Executing nimadm phase 10.

+-----------------------------------------------------------------------------+

Unmounting client mounts on the NIM master.

forced unmount of /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

forced unmount of /cg-aix61_mm_alt/alt_inst/var

forced unmount of /cg-aix61_mm_alt/alt_inst/usr

forced unmount of /cg-aix61_mm_alt/alt_inst/tmp

forced unmount of /cg-aix61_mm_alt/alt_inst/opt

forced unmount of /cg-aix61_mm_alt/alt_inst/home

forced unmount of /cg-aix61_mm_alt/alt_inst/admin

forced unmount of /cg-aix61_mm_alt/alt_inst

Removing nimadm cache file systems.

Removing cache file system /cg-aix61_mm_alt/alt_inst

Removing cache file system /cg-aix61_mm_alt/alt_inst/admin

Removing cache file system /cg-aix61_mm_alt/alt_inst/home

Removing cache file system /cg-aix61_mm_alt/alt_inst/opt

Removing cache file system /cg-aix61_mm_alt/alt_inst/tmp

Removing cache file system /cg-aix61_mm_alt/alt_inst/usr

Removing cache file system /cg-aix61_mm_alt/alt_inst/var

Removing cache file system /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

+-----------------------------------------------------------------------------+

Executing nimadm phase 11.

+-----------------------------------------------------------------------------+

Defining NIM mksysb resource ...

New NIMmksysb resource name is cg-aix71

+-----------------------------------------------------------------------------+

Executing nimadm phase 12.

+-----------------------------------------------------------------------------+

Cleaning up alt_disk_migration on the NIM master.

root@nim1 / #

For a description of all the phases in a nimadm operation, I suggest you refer to my previous nimadm article or the AIX infocenter (links below).

Once the migration had completed, I found a new mksysb file in /export/mksysb (as expected). The file was named cg-aix71. And a new NIM mksysb resource was now available.

root@nim1 / # ls -ltr /export/mksysb/

total 30793928

drwxr-xr-x 2 root system 256 Dec 21 10:40 lost+found

-rw-r--r-- 1 root system 4893952000 Dec 21 14:42 vio1-mksysb

-rw-r--r-- 1 root system 2323097600 Jan 06 10:00 cg-aix61

-rw-r--r-- 1 root system 1967411200 Jan 06 11:00 cg-aix71

root@nim1 / # lsnim -t mksysb

vio1-mksysb resources mksysb

cg-aix61 resources mksysb

cg1-mksysb resources mksysb

cg-aix71 resources mksysb

root@nim1 / # lsnim -l cg-aix71

cg-aix71:

class = resources

type = mksysb

comments = Created by nimadm on Thu Jan 6 11:00:51 EETDT 2011

Rstate = ready for use

prev_state = unavailable for use

location = /export/mksysb/cg-aix71

version = 7

release = 1

mod = 0

oslevel_r = 7100-00

alloc_count = 0

server = master

creation_date = Thu Jan 6 11:00:54 2011

During the migration I noticed the following file systems were mounted on the NIM master.

root@nim1 / # df

Filesystem 512-blocks Free %Used Iused %Iused Mounted on

/dev/hd4 1572864 1149640 27% 10471 8% /

/dev/hd2 5242880 1597696 70% 42812 19% /usr

/dev/hd9var 1048576 250456 77% 8260 23% /var

/dev/hd3 8912896 8287752 8% 111 1% /tmp

/dev/hd1 524288 523520 1% 5 1% /home

/dev/hd11admin 524288 523488 1% 5 1% /admin

/proc - - - - - /proc

/dev/hd10opt 1048576 664176 37% 7262 9% /opt

/dev/livedump 524288 523552 1% 4 1% /var/adm/ras/livedump

/dev/tftpbootlv 524288 419656 20% 10 1% /tftpboot

/dev/exportlv 524288 523408 1% 13 1% /export

/dev/lppsrclv 25690112 4383632 83% 3114 1% /export/lppsource

/dev/spotlv 8912896 4848464 46% 43450 8% /export/spot

/dev/mksysblv 38273024 11315200 71% 9 1% /export/mksysb

/dev/lv00 425984 412432 4% 25 1% /cg-aix61_mm_alt/alt_inst

/dev/lv01 65536 63376 4% 18 1% /cg-aix61_mm_alt/alt_inst/admin

/dev/lv02 65536 63384 4% 17 1% /cg-aix61_mm_alt/alt_inst/home

/dev/lv03 557056 414720 26% 201 1% /cg-aix61_mm_alt/alt_inst/opt

/dev/lv04 294912 285184 4% 21 1% /cg-aix61_mm_alt/alt_inst/tmp

/dev/lv05 4653056 4506904 4% 17 1% /cg-aix61_mm_alt/alt_inst/usr

/dev/lv06 720896 698160 4% 20 1% /cg-aix61_mm_alt/alt_inst/var

/dev/lv07 65536 63384 4% 17 1% /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

And in my nimadmvg volume group, there were several new logical volumes.

root@nim1 / # lsvg -l nimadmvg

nimadmvg:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

loglv02 jfslog 1 1 1 open/syncd N/A

lv00 jfs 13 13 1 open/syncd /cg-aix61_mm_alt/alt_inst

lv01 jfs 2 2 1 open/syncd /cg-aix61_mm_alt/alt_inst/admin

lv02 jfs 2 2 1 open/syncd /cg-aix61_mm_alt/alt_inst/home

lv03 jfs 17 17 1 open/syncd /cg-aix61_mm_alt/alt_inst/opt

lv04 jfs 9 9 1 open/syncd /cg-aix61_mm_alt/alt_inst/tmp

lv05 jfs 142 142 1 open/syncd /cg-aix61_mm_alt/alt_inst/usr

lv06 jfs 22 22 1 open/syncd /cg-aix61_mm_alt/alt_inst/var

lv07 jfs 2 2 1 open/syncd /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

root@nim1 / #

These file systems were used to extract the mksysb locally on the NIM master and then migrate the image to AIX 7.1. I noticed the restbyname process consuming some CPU during the restore process.

Topas Monitor for host:nim1 EVENTS/QUEUES FILE/TTY

Thu Jan 6 10:15:38 2011 Interval:2 Cswitch 1155 Readch 2655.6K

Syscall 3143 Writech 2664.7K

CPU User% Kern% Wait% Idle% Physc Entc% Reads 454 Rawin 0

Total 6.1 22.0 8.2 63.7 0.09 47.04 Writes 1168 Ttyout 578

Forks 12 Igets 85

Network BPS I-Pkts O-Pkts B-In B-Out Execs 0 Namei 521

Total 7.74K 5.50 1.50 5.74K 2.00K Runqueue 0 Dirblk 97

Waitqueue 1.0

Disk Busy% BPS TPS B-Read B-Writ MEMORY

Total 0.0 5.21M 383.0 2.02M 3.19M PAGING Real,MB 2048

Faults 776 % Comp 67

FileSystem BPS TPS B-Read B-Writ Steals 1118 % Noncomp 32

Total 2.03M 85.50 2.03M 0 PgspIn 0 % Client 19

PgspOut 0

Name PID CPU% PgSp Owner PageIn 517 PAGING SPACE

restbyna 15532232 16.4 1.21M root PageOut 816 Size,MB 512

vmmd 458766 5.5 1.19M root Sios 969 % Used 3

lrud 262152 1.5 512K root % Free 97

syncd 2097296 1.1 592K root NFS (calls/sec)

topas 16973864 0.5 1.98M root SerV2 0 WPAR Activ 0

xmgc 851994 0.5 448K root CliV2 0 WPAR Total 0

nimesis 12320772 0.3 2.41M root SerV3 0 Press: "h"-help

getty 7798832 0.1 580K root CliV3 0 "q"-quit

xmtopas 8847610 0.1 816K root

nfsd 5111964 0.1 4.75M root

xmtopasa 6947052 0.1 916K root

gil 1441836 0.0 960K root

java 4784136 0.0 57.5M pconsole

java 8716308 0.0 70.7M root

rpc.lock 6029528 0.0 1.19M root

clcomd 6160578 0.0 1.69M root

pilegc 786456 0.0 640K root

aixmibd 3997882 0.0 1.24M root

netm 1376298 0.0 448K root

topasrec 8454202 0.0 1.01M root

root@nim1 / # ps -ef | grep rest

root 15270132 3342336 1 10:15:55 pts/1 0:00 grep rest

root 15532232 10747910 37 10:14:02 pts/0 0:02 restbyname -x -q -f /export/mksysb/cg-aix61

root@nim1 / #

I didn’t encounter any issues with my nimadm operation but if I had I could have reviewed the nimadm log file located in /var/adm/ras:

root@nim1 /var/adm/ras/alt_mig # ls -ltr

total 856

-rw-r--r-- 1 root system 435009 Jan 06 11:00 cg-aix61_alt_mig.log

If you ever encounter problems with nimadm, I highly recommend that you re-run the operation with the –V and –D flags. This will enable verbose output, as well as debugging information which can assist you greatly in determining the root cause of your issue.

I was able to use the new, migrated, AIX 7.1 mksysb image to install a new LPAR.

AIX Version 7

root's Password:

*******************************************************************************

* *

* Welcome to AIX Version 7.1! *

* *

* Please see the README file in /usr/lpp/bos for information pertinent to *

* this release of the AIX Operating System. *

* *

*******************************************************************************

Last login: Thu Jan 6 13:21:04 EETDT 2011 on /dev/vty0

# oslevel -s

7100-00-02-1041

You’ve gotta love NIM! Take a look at the nimadm documentation for more information:

http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds4/nimadm.htm

By the way, if you are interested in migrating from AIX 4.3 to AIX 7.1, then you’ll need to consider the mksysb migration method. I’ve not done this myself (i.e. 4.3 to 7.1 migration) but the feedback I received from the AIX development team indicates that it does work as expected.

“... you had expressed some interest in mksysb migration. Since we were not aware of anyone having attempted it with 7.1. I went ahead and created a 4.3.3.75 system and created a mksysb and then used that mksysb in conjunction with nim and successfully installed another system using the 4.3 mksysb and 7.1 nim resources to perform a mksysb migration to 7.1. I just followed the steps as noted in the Installation guide for mksysb migration. “

Refer to the following document for further information on mksysb migration with NIM.

http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.install/doc/insgdrf/mksysbmigration.htm

NIM from A to Z in AIX 5L Redbook.

http://www.redbooks.ibm.com/redpieces/abstracts/sg247296.html?Open

Renaming devices in AIX 7.1

cggibbo Tags: 7.1 gibson aix chris rendev 20 Comments 45,195 Views

I remember reading this interesting document about some challenges renaming AIX devices.

http://gibsonnet.net/aix/ibm/The_Fun_of_Oracle_RAC_ASM_and_Device_Names.doc

This poor guy had to jump through some hoops to rename some disk and network devices in a clustered AIX (Oracle RAC) environment.

This comment stayed with me:

“The easy way to get network adapters and disk names to match between systems would be to run something like

# chdev –l old_name –n new_name

Except, this command option does NOT exist.”

Since then I’ve always been on the “look out” for an easy way to rename devices in AIX.

In particular in large clustered environments where it is sometimes very important to have the same disk and network device names in sync across all nodes in a cluster. And besides, it’s a lot easier to verify a cluster configuration if the hdisk names are all the same. Matching PVIDs works but it requires a lot more effort! For example, knowing that hdisk123 is the same device on all nodes makes life easier than scanning lspv output for a PVID like 00f6048868b4gead. Of course you can script things to make this easier but it would be great if you didn’t need to do this and that there was a way to rename devices as needed, without resorting to unsupported methods.

Well, this is no longer an issue for AIX.

Starting with AIX 7.1, you can now easily rename devices. A new command called rendev was introduced to allow AIX administrators to rename devices as required.

From the man page:

The rendev command enables devices to be renamed. The device to be renamed, is specified with the -l flag, and the new desired name is specified with the -n flag.

The new desired name must not exceed 15 characters in length. If the name has already been used or is present in the /dev directory, the operation fails. If the name formed by appending the new name after the character r is already used as a device name, or appears in the /dev directory, the operation fails.

If the device is in the Available state, the rendev command must unconfigure the device before renaming it. This is similar to the operation performed by the rmdev -l Name command. If the unconfigure operation fails, the renaming will also fail. If the unconfigure succeeds, the rendev command will configure the device, after renaming it, to restore it to the Available state. The -u flag may be used to prevent the device from being configured again after it is renamed.

Some devices may have special requirements on their names in order for other devices or applications to use them. Using the rendev command to rename such a device may result in the device being unusable. Note: To protect the configuration database, the rendev command cannot be interrupted once it has started. Trying to stop this command before completion, could result in a corrupted database.

This functionality was required to help simplify cluster device configuration with the new Cluster Aware AIX (CAA) feature in AIX 7.1.

I’ll be blogging about CAA in a future post.

Here are some examples of using the rendev command on AIX 7.1 (GA) system. In the first example I will rename hdisk3 to hdisk300. Note: hdisk3 is not in use (busy).

If the disk had been allocated to a volume group, I would have needed to varyoff the volume group first.

# lspv

hdisk0 00f61ab2f73e46e2 rootvg active

hdisk1 00f61ab20bf28ac6 None

hdisk2 00f61ab2202f7c0b None

hdisk4 00f61ab20b97190d None

hdisk3 00f61ab2202f93ab None

# rendev -l hdisk3 -n hdisk300

# lspv

hdisk0 00f61ab2f73e46e2 rootvg active

hdisk1 00f61ab20bf28ac6 None

hdisk2 00f61ab2202f7c0b None

hdisk4 00f61ab20b97190d None

hdisk300 00f61ab2202f93ab None

Easy!

Next, I’ll rename a virtual SCSI adapter. I renamed vscsi0 to vscsi2. Note: I placed the adapter, vscsi0, in a Defined state before renaming the device.

# rmdev -Rl vscsi0

# lsdev -Cc adapter

ent0 Available Virtual I/O Ethernet Adapter (l-lan)

ent1 Available Virtual I/O Ethernet Adapter (l-lan)

vsa0 Available LPAR Virtual Serial Adapter

vscsi0 Defined Virtual SCSI Client Adapter

vscsi1 Available Virtual SCSI Client Adapter

# rendev -l vscsi0 -n vscsi2

# lsdev -Cc adapter

ent0 Available Virtual I/O Ethernet Adapter (l-lan)

ent1 Available Virtual I/O Ethernet Adapter (l-lan)

vsa0 Available LPAR Virtual Serial Adapter

vscsi1 Available Virtual SCSI Client Adapter

vscsi2 Defined Virtual SCSI Client Adapter

Now I’ll rename a network adapter from ent0 to ent10. I bring down the interface before changing the device name

# lsdev -Cc adapter

ent0 Available Virtual I/O Ethernet Adapter (l-lan)

ent1 Available Virtual I/O Ethernet Adapter (l-lan)

vsa0 Available LPAR Virtual Serial Adapter

vscsi1 Available Virtual SCSI Client Adapter

vscsi2 Defined Virtual SCSI Client Adapter

# ifconfig en0

en0: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>

inet 10.1.20.19 netmask 0xffff0000 broadcast 10.153.255.255

tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

# ifconfig en0 down detach

# rendev -l ent0 -n ent10

# lsdev -Cc adapter

ent1 Available Virtual I/O Ethernet Adapter (l-lan)

ent10 Available Virtual I/O Ethernet Adapter (l-lan)

vsa0 Available LPAR Virtual Serial Adapter

vscsi1 Available Virtual SCSI Client Adapter

vscsi2 Defined Virtual SCSI Client Adapter

# rendev -l en0 -n en10

# chdev -l en10 -a state=up

en10 changed

# mkdev -l inet0

inet0 Available

# ifconfig en10

en10: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>

inet 10.1.20.19 netmask 0xffff0000 broadcast 10.153.255.255

tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

If you want to be creative you can rename devices to anything you like (as long as it’s not more than 15 characters). For example I’ll rename vscsi2 to myvscsiadapter.

# rendev -l vscsi2 -n myvscsiadapter

# lsdev -Cc adapter

ent1 Available Virtual I/O Ethernet Adapter (l-lan)

myadapter Available Virtual I/O Ethernet Adapter (l-lan)

myvscsiadapter Defined Virtual SCSI Client Adapter

vsa0 Available LPAR Virtual Serial Adapter

vscsi1 Available Virtual SCSI Client Adapter

And in the last example I’ll demonstrate changing virtual SCSI adapter device names on a live system.

This is single disk system (hdisk0), with two vscsi adapters.

# lspv

hdisk0 00f6048868b4deee rootvg active

# lspath

Enabled hdisk0 vscsi0

Enabled hdisk0 vscsi1

# lsdev -Cc adapter

ent0 Available Virtual I/O Ethernet Adapter (l-lan)

ent1 Available Virtual I/O Ethernet Adapter (l-lan)

vsa0 Available LPAR Virtual Serial Adapter

vscsi0 Available Virtual SCSI Client Adapter

vscsi1 Available Virtual SCSI Client Adapter

We ensure the adapter is in a Defined state before renaming it. This will fail otherwise.

# rmdev -Rl vscsi1

vscsi1 Defined

# lsdev -Cc adapter | grep vscsi

vscsi0 Available Virtual SCSI Client Adapter

vscsi1 Defined Virtual SCSI Client Adapter

Now we rename the adapter vscsi1 to vscsi3.

# rendev -l vscsi1 -n vscsi3

# lsdev -Cc adapter | grep vscsi

vscsi0 Available Virtual SCSI Client Adapter

vscsi3 Defined Virtual SCSI Client Adapter

That was easy enough. Now I need to bring the adapter and path online with cfgmgr. The lspath output displays an additional path to vscsi3.

# lspath

Enabled hdisk0 vscsi0

Defined hdisk0 vscsi1

# cfgmgr

Method error (/etc/methods/cfgscsidisk -l hdisk0 ):

0514-082 The requested function could only be performed for some

of the specified paths.

# lspath

Enabled hdisk0 vscsi0

Defined hdisk0 vscsi1

Enabled hdisk0 vscsi3

Now I need to remove the old path to vscsi1. The path to vscsi3 is now Enabled. The adapter, vscsi3, is in an Available state. All is good.

# rmpath -l hdisk0 -p vscsi1 -d

path Deleted

# lspath

Enabled hdisk0 vscsi0

Enabled hdisk0 vscsi3

# lsdev -Cc adapter | grep vscsi

vscsi0 Available Virtual SCSI Client Adapter

vscsi3 Available Virtual SCSI Client Adapter

The same steps need to be repeated for the vscsi0 adapter. This is renamed to vscsi2.

# rmdev -Rl vscsi0

vscsi0 Defined

# lsdev -Cc adapter | grep vscsi

vscsi0 Defined Virtual SCSI Client Adapter

vscsi3 Available Virtual SCSI Client Adapter

# rendev -l vscsi0 -n vscsi2

# lsdev -Cc adapter | grep vscsi

vscsi2 Defined Virtual SCSI Client Adapter

vscsi3 Available Virtual SCSI Client Adapter

# lspath

Defined hdisk0 vscsi0

Enabled hdisk0 vscsi3

# cfgmgr

Method error (/etc/methods/cfgscsidisk -l hdisk0 ):

0514-082 The requested function could only be performed for some

of the specified paths.

# lspath

Defined hdisk0 vscsi0

Enabled hdisk0 vscsi2

Enabled hdisk0 vscsi3

# rmpath -l hdisk0 -p vscsi0 -d

path Deleted

# cfgmgr

# lspath

Enabled hdisk0 vscsi2

Enabled hdisk0 vscsi3

That’s it. Both adapters have been renamed while the system was in use. No downtime required.

# lsdev -Cc adapter | grep vscsi

vscsi2 Available Virtual SCSI Client Adapter

vscsi3 Available Virtual SCSI Client Adapter

# lspath

Enabled hdisk0 vscsi2

Enabled hdisk0 vscsi3

Please refer to the AIX 7.1 command reference for more information on this new command:

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.cmds/doc/aixcmds4/rendev.htm

And by the way, here’s a draft of what I’ve been working on for the last 5 weeks:

IBM AIX Version 7.1 Differences Guide:

http://www.redbooks.ibm.com/Redbooks.nsf/RedpieceAbstracts/sg247910.html?Open

Follow me on Twitter:

http://twitter.com/cgibbo

DLPAR not doing it for you anymore?

cggibbo Tags: gibson chris dlpar aix 18 Comments 67,034 Views

I received an email from one of my customers recently that simply said:

“mate …on lpar30, for some reason the IBM.DRM and others are failing to start .. um .. any chance you could have a quick look at that ?”

So I asked, “OK, so this use to work right?”. To which I received a relatively confused reply, ”.....yep it did....actually no....it’s never worked....or has it???...I’m not sure...”.

Based on my experience, the most common issue that prevents DLPAR operations from working are network problems. Before diving into the deep end and trying to debug RSCT, it’s always best to start with the basics. For example, can you ping the HMC from the LPAR? Can you ping the LPAR from the HMC? If either of these tests fails, check the network configuration on both components before doing anything else.

On the HMC check the network settings first e.g.

· Click on HMC Configuration and then Customize Network Settings.

– Verify the IP address, netmask, default gateway, network routes, DNS server are all set correctly.

– Check the LPAR communications box in HMC configuration screen for LAN adapter that is used for HMC-to-LPAR communication.

– By the way, unlike POWER4 systems, LPARs on POWER5 and POWER6 systems do not depend on host name resolution for DLPAR operations.

· Check routing on the LPAR and the HMC.

– Use ping and the HMC’s Test Network Connectivity task to verify the LPAR and the HMC can communicate with each other.

If you check the network and you are happy that the LPAR and the HMC can communicate, then perhaps you need to re-initialise the RMC subsystems on the AIX LPAR. Run the following commands:

# /usr/sbin/rsct/bin/rmcctrl –z

# /usr/sbin/rsct/bin/rmcctrl –A

# /usr/sbin/rsct/bin/rmcctrl –p

Wait up to 5 minutes before trying DLPAR again. If DLPAR still doesn’t work i.e. the HMC is still reporting no values for DCaps, and the IBM.DRM subsystem still won’t start, try using the recfgct command.

hscroot@hmc1:~> lspartition -dlpar

.....

<#5> LPAR:<24*9117-MMB*10284FP, , 192.168.1.15>

Active:<1>, OS:<AIX, 6.1, 6100-05-01-1016>, DCaps:<0x0>, CmdCaps:<0x0, 0x0>, PinnedMem:<768>

.....

# /usr/sbin/rsct/install/bin/recfgct

Wait 5 minutes. This should resolve your DLPAR issue. The IBM.DRM subsystem should now be active and there should be good (non-zero) values for DCaps:

# lssrc -g rsct_rm

Subsystem Group PID Status

IBM.DRM rsct_rm 6881300 active

IBM.CSMAgentRM rsct_rm 7274530 active

IBM.ServiceRM rsct_rm 6029480 active

IBM.AuditRM rsct_rm 6357058 active

IBM.ERRM rsct_rm 4456566 active

IBM.LPRM rsct_rm 6946986 active

hscroot@hmc1:~> lspartition -dlpar

....

<#5> LPAR:<24*9117-MMB*10284FP, , 192.168.1.15>

Active:<1>, OS:<AIX, 6.1, 6100-05-01-1016>, DCaps:<0xc5f>, CmdCaps:<0x1b, 0x1b>, PinnedMem:<994>

....

Only run the rmcctrl and recfgct commands if you believe something has become corrupt in the RMC configuration of the LPAR. The fastest way to fix a broken configuration or to clear out the RMC ACL files after cloning (via alt_disk migration) is to use the recfgct command.

These daemons should work “out of the box” and are not typically the cause of DLPAR issues. However, you can try stopping and starting the daemons when troubleshooting DLPAR issues.

The rmcctrl -z command just stops the daemons. The rmcctrl -A command ensures that the subsystem group (rsct) and the subsystem (ctrmc) objects are added to the SRC, and an appropriate entry added to the end of /etc/inittab and it starts the daemons.

The rmcctrl –p command enables the daemons for remote client connections i.e. from the HMC to the LPAR and vice versa.

If you are familiar with the System Resource Controller (SRC) you might be tempted to use stopsrc and startsrc commands to stop and start these daemons.

Do not do it; use the rmcctrl commands instead.

If /var is 100% full, use chfs to expand it. If there is no more space available, examine subdirectories and remove unnecessary files (for example, trace.*, core, and so forth). If /var is full, RMC subsystems may fail to function correctly.

The polling interval for the RMC daemons on the LPAR to check with the HMC daemons is 5-7 minutes; so you need to wait long enough for the daemons to start up and synchronize.

The Resource Monitoring and Control (RMC) daemons are part of the Reliable, Scalable Cluster Technology (RSCT) and are controlled by the System Resource Controller (SRC). These daemons run in all LPARs and communicate with equivalent RMC daemons running on the HMC. The daemons start automatically when the operating system starts and synchronize with the HMC RMC daemons.

The daemons in the LPARs and the daemons on the HMC must be able to communicate over the network for DLPAR operations to succeed. This is not the network connection between the managed system (FSP) and the HMC; it is the network connection between the operating system (AIX) in each LPAR and the HMC.

Note: Apart from rebooting, there is no way to stop and start the RMC daemons on the HMC.

The following links also contain some (out dated) information relating to DLPAR verification and troubleshooting. Even though it is quite old some of it is still relevant today and is good a place to start.

The most common reasons for failures with Dynamic Logical LPARing

http://www-01.ibm.com/support/docview.wss?rs=111&context=SWG10&q1=dcaps&uid=isg3T1010615&loc=en_US&cs=utf-8&lang=en

Dynamic LPAR tips and checklists for RMC authentication and authorization

http://www.ibm.com/developerworks/systems/articles/DLPARchecklist.html

Setting up the HMC/LPARs hostname and network (old but interesting)

http://computerstec.com/aixhowto/aixh18.htm

lsLPAR -dlpar, DCAPs values (old but still applies)

http://community.livejournal.com/eserver/18517.html

The previous link (above) provides some information relating to the values for DCaps and what they mean (also out dated):

0 - DR CPU capable (can move CPUs)

1 - DR MEM capable (can move memory)

2 - DR I/O capable (can move I/O resources)

3 - DR PCI Bridge (can move PCI bridges)

4 - DR Entitlement (POWER 5 can change shared entitlement)

5 - Multiple DR CPU (AIX 5.3 can move 2+ CPUs at once)

0x3f = max, and 0xf is common for AIX 5.2

If you are interested in how HMC and LPAR authentication works with DLPAR, then read on. Otherwise, happy DLPARing!

HMC and LPAR authentication (RSCT authentication)

The diagram below outlines how the HMC and an LPAR authenticate with each other in order for DLPAR operations to work. RSCT authentication is used to ensure the HMC is communicating with the correct LPAR.

Authentication is the process of ensuring that another party is who it claims to be. Authorization is the process by which a cluster software component grants or denies resources based on certain criteria. The RSCT component that implements authorization is RMC. It uses access control list (ACL) files to control user access to resources.

The RMC component subsystem uses cluster security services to map the operating system user identifiers, specified in the ACL file, to network security identifiers to determine if the user has the correct permissions. This is performed by the identity mapping service, which uses information stored in the identity mapping files ctsec_map.global and ctsec_map.local.

The RSCT authorization process in detail:

1. On the HMC: DMSRM pushes down the secret key and HMC IP address to NVRAM when it detects a new CEC; this process is repeated every five minutes. Each time an HMC is rebooted or DMSRM is restarted, a new key is used.

2. On the AIX LPAR: CSMAgentRM, through RTAS (Run-time Abstraction Services), reads the key and HMC IP address out from NVRAM. It will then authenticate the HMC. This process is repeated every five minutes on a LPAR to detect a new HMCs and if the key has changed. An HMC with a new key is treated as a new HMC and will go though the authentication and authorization processes again.

3. On the AIX LPAR: After authenticating the HMC, CSMAgentRM will contact the DMSRM on the HMC to create a ManagedNode resource in order to identify itself as a LPAR of this HMC. CSMAgentRM then creates a compatible ManagementServer resource on AIX. This can be displayed on AIX with the lssrsrc command. e.g.

root@aix6 / # lsrsrc "IBM.ManagementServer"

Resource Persistent Attributes for IBM.ManagementServer

resource 1:

Name = "192.168.1.244"

Hostname = "192.168.1.244"

ManagerType = "HMC"

LocalHostname = "10.153.3.133"

ClusterTM = "9078-160"

ClusterSNum = ""

ActivePeerDomain = ""

NodeNameList = {"aix6"}

4. On the AIX LPAR: After the creation of the ManagedNode and ManagementServer resources on the HMC and AIX respectively, CSMAgentRM grants HMC permission to access necessary resource classes on the LPAR. After granting the HMC permission, CSMAgentRM will change its ManagedNode, on the HMC, Status to 1. (It should be noted that without proper permission on AIX, the HMC would be able to establish a session with the LPAR but will not be able to query for OS information, DLPAR capabilities, or execute DLPAR commands afterwards.)

5. On the HMC: After the ManagedNode Status is changed to 1, LparCmdRM establishes a session with the LPAR, queries for operating system information and DLPAR capabilities, notifies CIMOM about the DLPAR capabilities of the LPAR, and then waits for the DLPAR commands from users.

AIX 7.1 – DSM dgetmacs.

cggibbo Tags: nim chris aix 7.1 dgetmacs dsm gibson 12 Comments 10,854 Views

I received an email this week from a colleague that worked with me on the NIM Redbook back in 2006. He was experiencing an issue with DSM and NIM. He was attempting to use the dgetmacs command to obtain the MAC address of the network adapters on an LPAR. The command was failing to return the right information.

This was the output from his attempt with dgetmacs:

# dgetmacs -n lpar1

Using an adapter type of "ent".

Could not dsh to node lpar1.

Attempting to use openfirmware method to collect MAC addresses.

Acquiring adapter information from Open Firmware for node lpar1.

dgetmacs: 2760-185 Issuing the command "/opt/ibm/sysmgt/dsm/dsmbin/lpar_netboot -M -n -t ent -J hmc1 -j hmc -F /etc/ibm/sysmgt/dsm/config/my_password_file –A 4 10582FP 8202-E4B" gave a return code of 1. The routine will continue.

No adapters found for node lpar1.

Could not get mac address for this aix node "lpar1"

I experienced this very issue during the writing of the AIX 7.1 Differences Guide Redbook. And given that I was in Austin, sitting in the same building as the AIX development team, I was able to speak with the developers directly about the issue. At that time they provided me with the following workaround.

First they asked me to check the size of the /usr/lib/nls/msg/en_US/IBMhsc.netboot.cat message catalog file.

# ls –l /usr/lib/nls/msg/en_US/IBMhsc.netboot.cat

-rw-r--r-- 1 bin bin 3905 Aug 08 09:54

They were surprised to find that the file appeared to be “too small”. They promptly sent me the catalog file from one of their development AIX 7.1 systems. I replaced the file as follows:

# cd /usr/lib/nls/msg/en_US/

# ls -ltr IBMhsc*

-rw-r--r-- 1 bin bin 3905 Aug 08 09:54 IBMhsc.netboot.cat

# cp -p IBMhsc.netboot.cat IBMhsc.netboot.cat.old

# cp /tmp/lpar1/IBMhsc.netboot.cat.new IBMhsc.netboot.cat

# ls -ltr IBMhsc*

-rw-r--r-- 1 bin bin 3905 Aug 08 09:54 IBMhsc.netboot.cat.old

-rw-r--r-- 1 bin bin 26374 Dec 23 11:24 IBMhsc.netboot.cat

This fixed the problem for me during the residency.

So I asked my friend to do the same (after I sent him the message catalog file). He ran the dgetmacs command again and this time it returned the MAC address for all the network adapters in his LPAR. Success!

# dgetmacs -n lpar1

Using an adapter type of "ent".

Could not dsh to node lpar1.

Attempting to use openfirmware method to collect MAC addresses.

Acquiring adapter information from Open Firmware for node lpar1.

# Node::adapter_type::interface_name::MAC_address::location::media_speed::adapter_duplex::UNUSED::install_gateway::ping_status::machine_type::netaddr::subnet_mask

lpar1::ent_v::::4A7009608816::U8202.E4B.10582FP-V4-C22-T1::auto::auto::::::n/a::secondary::::

lpar1::ent_v::::4A7009608818::U8202.E4B.10582FP-V4-C24-T1::auto::auto::::::n/a::secondary::::

This then allowed him to network boot his LPAR and install AIX 7.1 via NIM.

# nim -o bos_inst -a bosinst_data=noprompt -a source=rte -a installp_flags=agX -a accept_licenses=yes -a spot=spotaix7tl0sp2 -a lpp_source=aix7tl0sp2 lpar1

dnetboot Status: Invoking /opt/ibm/sysmgt/dsm/dsmbin/lpar_netboot lpar1

dnetboot Status: Was successful network booting node lpar1.

I thought this might have been resolved in the latest service pack for AIX 7.1. However, both of us have the latest service pack installed (i.e. TL0 SP2 for AIX 7.1).

# oslevel -s

7100-00-02-1041

If anyone encounters this problem, feel free to contact me (aix@gibsonnet.net).

For more information, please refer to section 5.2.7 Using DSM and NIM, in the AIX 7.1 Differences Guide Redbook:

http://www.redbooks.ibm.com/Redbooks.nsf/RedpieceAbstracts/sg247910.html?Open

Replicated rootvg, ghostdev and NPIV

cggibbo Tags: chris hds recovery led rootvg fc support ghostdev replication lpar hung gibson 554 boot replicated chris_gibson vfc vsp aix san channel virtual fibre dr disaster npiv hang 11 Comments 26,645 Views

My team and I have recently been trying to stream line our AIX disaster recovery process. We’ve been looking for ways to reduce our overall recovery time. Several ideas were tossed around such as a) using a standby DR LPAR with AIX already installed and using rsync/scp to keep the Prod & DR LPARs in sync and b) using alt_disk_copy (with the –O flag for a device reset) to clone rootvg to an alternate disk which is then replicated to DR. These methods may work but are cumbersome to administer and (in the case of alt_disk_copy) require additional (permanent) resources on every production system. With over 120 production instances of AIX, the disk space requirements start to add up.

So far we’ve concluded that the best way to achieve our goal is by using SAN replicated rootvg volumes at our DR site.

Our current DR process relies on recovery of AIX systems from mksysb images from a NIM master. All our data (non-rootvg) LUNs are already replicated to our DR site. The aim was to change the process and ‘recover’ our AIX images using replicated rootvg LUNs. This will reduce our overall recovery time at DR (which is crucial if we are to meet the proposed recovery time objectives set by our business). Based on current IBM documentation we were relatively comfortable with the proposed approach. The following IBM developerWorks article (originally published in 2009 and updated in late 2010) describes “scenarios in which remapping, copying, and reuse of SAN disks is allowed and supported. More easily switch AIX environments from one system to another and help achieve higher availability and reduced down time. These scenarios also allow for fast deployment of new systems using cloning.”

The document focuses on fully virtualised environments that utilise shared processors and VIO servers. One area where this document is currently lacking in information is the use of NPIV and virtual fibre channel adapters in a DR scenario. We reached out to our contacts in the AIX development space and asked the following question:

“Hoping you can help us find some statements regarding support for a replicated rootvg environment using NPIV/Virtual Fibre Channel adapters?

The following IBM developerWorks article discusses VSCSI and we are looking for something similar for NPIV.

http://www.ibm.com/developerworks/aix/library/au-AIX_HA_SAN/index.html

My guess is that restrictions similar to those for physical FC adapters will apply here? But I'm hoping that given the adapters are virtual the limitations may be relaxed.

Are you aware of any statement regarding support (or not) for booting from another system using a disk subsystem image of rootvg replicated to another disk subsystem when using NPIV? And what, if any, additional requirements/restrictions may apply when using NPIV?”

We received the following responses:

“There are some additional considerations when using NPIV for booting from a replicated rootvg. With NPIV the client partitions has virtual Fibre Channel adapter ports, but has physical access to the actual (physical) disk devices. There may be an increased chance of needing to update the boot list via the Open Firmware SMS menu. Since the clients have access to the actual disks, you have the possibility of running multipathing software besides AIX MPIO. If you are using a multipathing software to manage the NPIV attached disks besides AIX MPIO, then you should contact the vendor that provided the software to check their support statement,

Since one or more of the physical devices will change when booting from an NPIV replicated rootvg, it is recommend to set the ghostdev attribute. The ghostdev attribute will trigger when it detects the AIX image is booting from either a different partition or server. Ghostdev attribute should not trigger during LPM operations (Live Partition Mobility). Once triggered, ghostdev will clear the customized ODM database. This will cause detected devices to be discovered as new devices (with default settings), and avoid the issue with missing/stale device entries in ODM. Since ghostdev does clear the entire customized ODM database, this will require you import your data (non-rootvg) volume groups again, and perform any (device) attribute customization. To set ghostdev, run "chdev -l sys0 -a ghostdev=1". Ghostdev must be set before the rootvg is replicated.

Similar with virtual devices, the client partition is booting an existing rootvg where the hardware may be different. It's possible some applications have dependency on tracking the actual physical devices (instead of data on the disks). For example PowerHA, may keep track of a disk for a cluster health checks. If you do have applications that have a dependency on tracking physical devices, then additional setup (of those applications) may be required after the first boot from the replicated rootvg.

We do have multiple customers using NPIV for such scenarios. I believe most of them worked with IBM Lab Based Services to assist with implementing such a configuration, and some of the customers required some custom scripts to further customize their system after booting from the replicated rootvg. Those customers set the ghostdev attribute, and had custom scripts to import their data (non-rootvg) volume groups, and update PowerHA to point to the new health check disk.

You should get support for such an NPIV setup with IBM as long as you follow the considerations listed in the WhitePaper.”

“Development has approved using NPIV to do this for one customer. Below are more detailed requirements for this DR strategy using NPIV. If a Disaster Recovery (DR) environment not using PowerHA Enterprise Edition is used, then we believe the white paper located http://www.ibm.com/developerworks/aix/library/au-AIX_HA_SAN/index.html provides the guidelines in regards to setup, prerequisites, and as well as the limitations of such a DR deployment. Deployments as detailed in the white paper are supported by IBM. However note that such a deployment has many manually instituted responsibilities on the customer to setup and maintain such an environment. IBM expects that the customer carefully manage these manual steps without any mistakes. The White paper does not currently cover using NPIV in such a DR scenario.

We have the following guidelines regarding the configuration, which includes NPIV as an option: All of the system configuration should be virtualized, with the possible exception of disk devices when using NPIV. If NPIV is used then AIX MPIO must be used as the multi-pathing solution. If multi-pathing software is used besides AIX MPIO, then the vendor of that software must be contacted regarding a support statement. Install AIX (at least minimum required TLs/SPs for desired AIX version) and software stack (Middleware and applications) on primary systems, which is compatible with systems at both sites. Primary and secondary sites should be using systems with similar hardware, same microcode levels, and same VIOS levels.

Many manual steps are needed to setup the virtual and physical devices accurately on secondary site VIOS. If Virtual SCSI Disks are being used, then discover the unique identification on the primary site and map the disks to the corresponding replication disks. Map the same appropriately on VIOS on secondary site. Level of VIOS should support attribute to open the secondary devices passively. This setting needs to be setup correctly on the VIOS on secondary site. Operating environment should not have subnet dependencies. Manage the replication relationships accurately. Manually may need to switch the secondary disks to primary node. Raw disk usage may cause problems. Some middle ware products may bypass Operating system and use the disk directly. They might have their own restrictions for this environment. (e.g. anything that is device location code or storage LUN unique ID dependant may have issues when the cloned image is restarted on the secondary system with replicated storage).

Set the "ghostdev" attribute using chdev command (must be done on the primary). This attribute can be set using the command "chdev -l sys0 –a ghostdev=1". The ghostdev attribute will delete customized ODM database on rootvg when AIX detects it has booted from a different LPAR or system. If the "ghostdev" attribute is not set, then the booting from the alternate site will result in devices in ODM showing up in "Defined" or "Missing" state.

After a failover to secondary site, may need to reset the boot device list for each LPAR before the boot of the LPAR using SMS menus of the firmware Note that this is not an exhaustive list of issues. Refer to the white paper and study the same as it applies to the environment. So as long as they are using MPIO, you're OK. If not using MPIO and some OEM storage, then the storage vendor must also support it.”

While these responses indicate that this form of recovery is supported by IBM, we were still looking to IBM for clarity on the support position. It has been noted that other IBM customers have had mixed responses when contacting AIX support for feedback and assistance with this type of DR procedure. And it’s not hard to see why when you read statements like this from the “Supported Methods of Duplicating an AIX System” document:

“Unsupported Methods

1. Using a bitwise copy of a rootvg disk to another disk.

This bitwise copy can be a one-time snapshot copy such as flashcopy, from one disk to another, or a continuously-updating copy method, such as Metro Mirror.

While these methods will give you an exact duplicate of the installed AIX operating system, the copy of the OS may not be bootable. A typical scenario where this is tried is when one system is a production host and there is a desire to create a duplicate system at a disaster recovery site in a remote location.

2. Removing the rootvg disks from one system and inserting into another.

This also applies to re-zoning SAN disks that contain the rootvg so another host can see them and attempt to boot from them.

Why don't these methods work?

The reason for this is there are many objects in an AIX system that are unique to it; Hardware location codes, World-Wide Port Names, partition identifiers, and Vital Product Data (VPD) to name a few. Most of these objects or identifiers are stored in the ODM and used by AIX commands.

If a disk containing the AIX rootvg in one system is copied bit-for-bit (or removed), then inserted in another system, the firmware in the second system will describe an entirely different device tree than the AIX ODM expects to find, because it is operating on different hardware. Devices that were previously seen will show missing or removed, and usually the system will typically fail to boot with LED 554 (unknown boot disk).”

So, as a secondary objective, we have been working closely with our local IBM representatives to obtain some surety from IBM that our proposed DR strategy for AIX is fully supported by both the AIX development and support teams.

With that in mind I’ll provide an overview of our new DR approach and hope that it offers others insight to alternative method for recovery and also to assist IBM in further understanding what some of the “larger” AIX customers are looking for in terms of simplified AIX disaster recovery.

What follows is a detailed description of our IBM AIX, PowerVM/Power Systems environment, the proposed recovery steps and other items for consideration.

Our Environment:

· - AIX 5.3*, 6.1 & 7.1 - 5300-12-04-1119, 6100-06-05-1115 and 7100-01-04-1216.

· - VIOS 2.2.1.3 – both production and DR.

· - VIOS physical FC adapters – both production and DR - 8Gb PCI Express Dual Port FC Adapter (5735) – Firmware level: 2.00X7.

· - All client LPARs utilise Virtual FC adapters (NPIV) for disk storage.

· - Production: POWER7 795 (9119-FHB). Firmware level: AH730_078.

· - DR: POWER6 595 (9119-FHA). Firmware level: EH350_120.

· - HDS VSP storage – at both production and DR sites.

· - AIX MPIO only – HDS ODM driver is installed (devices.fcp.disk.Hitachi.array.mpio.rte) – HDLM is not installed.

· - Production: Dual VIOS.

· - DR: Single VIOS.

· - Production: LUNs are mapped directly to client LPARs via NPIV, virtual FC adapter pass thru via dual VIOS. MPIO is in use i.e. one path per VIOS/Physical FC.

· - DR: LUNs are mapped directly to client LPARs via NPIV, virtual FC adapter pass thru via a single VIOS. i.e. single path only.

· - Please refer to the following table for a summary of the environment details.

Our Recovery Procedure:

1. - Change the sys0 ghostdev attribute value to 1 on the source production AIX system. Set the "ghostdev" attribute using chdev command (must be done on the primary). This attribute can be set using the command "chdev -l sys0 –a ghostdev=1". The ghostdev attribute will delete customized ODM database on rootvg when AIX detects it has booted from a different LPAR or system.

2. - Take note of the source systems rootvg hdisk PVID.

3. - Select the source production rootvg LUN for replication on the HDS VSP.

4. - Replicate the LUN from the production site to the DR HDS VSP.

5. - In a DR test, suspend HDS replication from production to DR.

6. - Assign the replicated LUN to the target LPAR on the DR POWER6 595 i.e. map the LUN to the WWPN of the virtual FC adapter on the DR LPAR.

7. - Attempt to boot the DR LPAR using the replicated rootvg LUN. If necessary, enter SMS menu to update the boot list i.e. select the correct boot disk, check for the same PVID as the source host.

8. - Once the LPAR has successfully booted, the AIX administrator would configure the necessary devices i.e. import data volume groups, configure network interfaces, etc. This may also be scripted for execution during the first boot process.

9. - Please refer to the following diagrams for a visual representation of the proposed process.

Some Notes/Caveats:

The following is a list of items that we understand are possible limitations and issues with our new DR process.

· - Booting from replicated rootvg disks may fail for several reasons, such as, a) there is unexpected corruption in the replicated LUN image due to rootvg not being quiesced during replication or b) there is a unidentified issue with the AIX system that is only apparent the next time the system is booted; this could be mis-configuration by the administrator or some other unforseen problem.

· - In the event that an LPAR fails to boot via a replicated rootvg LUN, a backup method is available for recovery. Switching back to manual NIM mksysb restore provides a sufficient backup should the replicated rootvg be unusable.

· - If the "ghostdev" attribute is not set, then booting from the DR site will result in devices in the ODM showing up in a "Defined" or "Missing" state.

· - Once a DR test is completed, the DR LPAR should be de-activated immediately so that SAN disk replication can be restarted between production and DR. Failure to perform this step may result in the DR LPAR failing as a result of file system corruption.

· - At present we are using AIX MPIO only. There is discussion of using HDLM in the future. We will contact HDS for a support statement regarding booting from replicated rootvg LUNs with HDLM installed.

· - The ghostdev attribute is not implemented in AIX 5.3. AIX 5.3 is no longer supported*.

So far all of our testing has been successful. We verified that we could replicate an SOE rootvg image of AIX 6.1 and 7.1 to DR and successfully boot an LPAR using the replicated disk. Based on these tests there doesn’t appear to be anything stopping us from using this method for DR purposes. The following table outlines the different versions of AIX we tested and the results.

Once the system was booted we needed to perform some post boot configuration tasks. These tasks were handled by two scripts that were called from /etc/inittab. On the source system we installed the new scripts (in /etc) and added new entries to the /etc/inittab file. These scripts only run if the systemid matches that of the DR systemid. Note: Only partial contents of each script are shown below…but you get the idea.

# mkitab -i srcmstr "AIXDRimportvg:2:once:/etc/AIX_DRimportvg.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot."

# mkitab -i srcmstr "AIXDRconfig:2:once:/etc/AIX_DRconfig.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot."

# lsitab -a | grep DR

AIXDRconfig:2:once:/etc/AIX_DRconfig.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot.

AIXDRimportvg:2:once:/etc/AIX_DRimportvg.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot.

AIX_DRconfig.ksh:

#!/bin/ksh

#set -xv

#################################################################

# This script will configure the AIX environment for DR purposes.

# This script will only run if the systemid matches the systemid of the DR system.

# aixdr : / # lsattr -EOl sys0 -a systemid | grep -v systemid | awk -F, '{print $2}'

# 0211D11C1 = DR POWER System

#################################################################

MYNAME=$(basename $0)

DR_SYSTEMID=0211D11C1

LSATTR_SYSTEMID_DR=`lsattr -EOl sys0 -a systemid | grep -v systemid | awk -F, '{print $2}'`

if [ "$LSATTR_SYSTEMID_DR" = "$DR_SYSTEMID" ]

then

#Set the DR hostname, IP address, netmask and gateway.

#mktcpip -h <hostname> -a <IP to use in DR> -m <network mask> -i en0 -g <gateway IP>

#en0:10.1.5.99:255.255.252.0:10.1.1.10,255.255.255.0 10.2.2.10,255.255.255.0

dr_hostname=`cat /usr/local/dr/dr_hostname.txt`

dr_defgw=`cat /usr/local/dr/dr_defgw.txt`

dr_intf=`cat /usr/local/dr/dr_en0.txt | awk -F: 'print $1}'`

dr_ip=`cat /usr/local/dr/dr_en0.txt | awk -F: 'print $2}'`

dr_netm=`cat /usr/local/dr/dr_en0.txt | awk -F: 'print $3}'`

echo "$MYNAME: Setting DR hostname and IP address."

mktcpip -h $dr_hostname -a $dr_ip -m $dr_netm -i $dr_intf -g $dr_defgw

#Configure IP aliases

#chdev -l en0 -a alias4=10.1.1.10,255.255.255.0

#Remove IP aliases

#chdev -l en0 -a delalias4=10.1.1.10,255.255.255.0

for i in `cat /usr/local/dr/dr_en0.txt | awk -F: '{print $4}'`

chdev -l $dr_intf -a alias4=$i

done

for inet in `cat dr_inet.txt | grep -v en0`

for i in `cat /usr/local/dr/dr_$inet.txt | awk -F: '{print $4}'`

chdev -l $inet -a alias4=$i

done

#Configure the AIX environment for DR.

echo "$MYNAME: Configure the AIX environment for DR."

echo

#Delete the MDC DNS entries

echo "$MYNAME: Configuring DNS for DR."

namerslv -d -i 10.1.6.21

namerslv -d -i 10.1.4.21

namerslv -d -i 10.1.5.21

#Add the DR DNS entry

namerslv -a -i 10.1.7.38

echo

echo "$MYNAME: Configuring bootlist for DR."

bootlist -m normal -o

bootlist -m normal hdisk0

bootlist -m normal -o

echo

echo "$MYNAME: Configuring /etc/inittab for DR."

lsitab -a | grep nim

chitab "nimclient:2:off:/usr/sbin/nimclient -S running > /dev/console 2>&1 # inform nim we're running"

lsitab -a | grep nim

echo

else

MYNAME=$(basename $0)

echo "$MYNAME: The systemid $LSATTR_SYSTEMID_DR does not match the expected DR systemid of $DR_SYSTEMID."

echo "$MYNAME: This script should only be executed at DR."

echo "$MYNAME: If you are not booting the system at the DR site, then you can ignore this message."

echo "$MYNAME: No changes have been perfomed. Script is exiting."

The ghostdev attribute essentially provides us with a clean ODM and allows the system to discover new devices and build the ODM from scratch. If you attempt to boot from a replicated rootvg disk without first setting the ghostdev attribute, your system may fail to boot (hang at LED 554) because of a new device tree and/or missing devices. You might be able to recover from this situation (without restoring from mksysb) by performing the steps outlined on pages 16-20 of the following document (thanks to Dominic Lancaster at IBM for the presentation).

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/resource/IBMSTGEducationSymposium_AX61_POWERVM_P2V.pdf

Cluster Aware AIX

cggibbo Tags: caa_private0 caa event aware ahafs cluster mkcluster aix cldisk1 7.1 lscluster 11 Comments 58,253 Views

Using Cluster Aware AIX with AIX 7.1 you can create a cluster of AIX nodes. This interconnected cluster of nodes can immediately provide several capabilities, such as:

· Clusterwide event management.

· Communication and storage events:

o Node UP and node DOWN

o Network adapter UP and DOWN

o Network address change

o Point-of-contact UP and DOWN

o Disk UP and DOWN

o Predefined and user-defined events

· Clusterwide storage naming service

· Clusterwide command distribution

· Clusterwide communication making use of networking and storage connectivity

All of the capabilities of CAA are built into the AIX Version 7.1 operating system. CAA is essentially a set of tools and services embedded into AIX that help to manage a cluster of AIX nodes and assist in running cluster software on AIX. Cluster products, such as PowerHA and RSCT from IBM, can utilise CAA to simplify the configuration, management and monitoring of an AIX cluster.

CAA does not form a cluster by itself. It is a tool set. There is no notion of quorum. (If 20 nodes of a 21 node cluster are down, CAA still runs on the remaining node). CAA does not eject nodes from a cluster. CAA provides tools to fence a node but never fences a node and will continue to run on a fenced node.

As you can see in the following diagram, cluster products, like PowerHA, will integrate with CAA to help form and manage highly available clusters.

So you will still need some form of cluster product, either from IBM or another vendor, in order to build a cluster that provides high availability capabilities like node failover/takeover.

What follows are some of the most important “snippets” from the CAA documentation (IMHO), available in the AIX 7.1 Information Centre (see the link at the bottom of the page).

Just like any cluster, each node that is added to a cluster by using CAA must have common storage devices available, for example via SAN storage device. These storage devices are used for the cluster repository disk and for any clustered shared data disks.

The Storage Naming Service provides a global device view across all the nodes in the cluster. The Storage Naming Service also provides a single global device name for a disk from any node in the cluster. The global device name, for example, cldisk1, refers to the same physical disk from any node in the cluster.

The cluster repository disk is used as the central repository for the cluster configuration data. The cluster repository disk must be accessible from all nodes in the cluster and is a minimum of 10 GB in size. Given the importance of the cluster configuration data, the cluster repository disk should be backed up by a redundant and highly available storage configuration. Even though the cluster repository disk is visible as a disk device to AIX, it should be treated as a special device for the cluster. The use of LVM commands is not supported when used on a cluster repository disk. The AIX LVM commands are designed as single node administrative commands, and are not applicable in a clustered configuration.

The cluster repository disk is renamed to a private device name (caa_private0). A raw section of the disk and a section of the disk that contains a special volume group and special logical volumes are used during cluster operations.

A multicast address is used for cluster communications between the nodes in the cluster.

These are configured automatically during the creation of the cluster. These nodes support cluster monitoring of events and cluster configuration attributes.

Scalable reliable multicasting is implemented in the cluster with a special gossip protocol over the multicast address. The gossip protocol determines the node configuration and then transmits the gossip packets over all available networking and storage communication interfaces (either Fibre Channel and/or SAS adapters). If no storage communication interfaces are configured, only the traditional networking interfaces are used.

When you first configure CAA, the following actions are performed:

• The cluster is created using the mkcluster command.

• The cluster configuration is written to the raw section of the cluster repository disk.

• Primary and secondary database nodes are selected from the list of candidate nodes in the mkcluster command.

• Special volume groups and logical volumes are created on the cluster repository disk.

• Cluster file systems are created on the special volume group.

• The cluster repository database is created on both primary and secondary nodes.

• The cluster repository database is started.

• Cluster services are made available to other functions in the operating system, such as Reliable Scalable Cluster Technology (RSCT) and PowerHA.

• Storage framework register lists are created on the cluster repository disk.

• A global device namespace is created and interaction with LVM starts for handling associated volume group events.

• A clusterwide multicast address is established.

• The node discovers all of the available communication interfaces.

• The cluster interface monitoring starts.

• The cluster interacts with Autonomic Health Advisory File System (AHAFS) for clusterwide event distribution.

• The cluster exports cluster messaging and cluster socket services to other functions in the operating system, such as Reliable Scalable Cluster Technology (RSCT) and PowerHA.

In the following example I created a two node cluster using CAA tools. First of all, I had to use the mkcluster command to define the cluster nodes, shared storage and repository disk. The node names are 7502lp01 and 7502lp02. The shared data storage disks are hdisk2, hdisk3 and hdisk4. The repository disk, used to house the cluster configuration data, is hdisk1.

# mkcluster -n mycluster -r hdisk1 -d hdisk2,hdisk3,hdisk4 -m 7502lp01,7502lp02

mkcluster: Cluster shared disks are automatically renamed to names such as

cldisk1, [cldisk2, ...] on all cluster nodes. However, this cannot

take place while a disk is busy or on a node which is down or not

reachable. If any disks cannot be renamed now, they will be renamed

later by the clconfd daemon, when the node is available and the disks

are not busy.

You’ll notice that the mkcluster command informed me that the cluster shared disks are automatically renamed to cluster disk names like cldisk1. After I’d run the command I noticed something quite interesting and impressive. Prior to configuring the cluster the shared disks on both nodes had names like hdiskX. Afterwards the shared disks had been renamed across both nodes. Now these disks had the same name!

This is going to simplify cluster configuration and management. No more will I need to remove and recreate disks in order to resolve disk naming inconsistencies in a cluster. The lspv output (shown below) from both nodes shows that I have three shared disks (cldisk1, cldisk2 and cldisk3). These would be used for shared data. The disk named caa_private0 is my cluster repository disk. This is used to store and share the cluster configuration data.

7502lp01:

# lspv | sort

caa_private0 00f61ab20b97190d caavg_private active

cldisk1 00f61ab20bf28ac6 None

cldisk2 none None

cldisk3 none None

hdisk0 00f61ab2f73e46e2 rootvg active

7502lp02:

# lspv | sort

caa_private0 00f61ab20b97190d caavg_private active

cldisk1 00f61ab20bf28ac6 None

cldisk2 none None

cldisk3 none None

hdisk0 00f61ab2895e4cbe rootvg active

Cluster Aware AIX tells you what nodes are in the cluster plus information on those nodes, including state. A special “gossip” protocol is used over the multicast address to determine node information and implement scalable reliable multicast. No traditional heartbeat mechanism is employed. Gossip packets travel over all interfaces, including storage. Immediately after running mkcluster, I was able to query the status of the nodes in my cluster without any further configuration!

# lscluster -m

Calling node query for all nodes

Node query number of nodes examined: 2

Node name: 7502lp01

Cluster shorthand id for node: 1

uuid for node: 3cd9cb00-bf55-11df-b015-6e8dd0af6304

State of node: UP

Smoothed rtt to node: 7

Mean Deviation in network rtt to node: 3

Number of zones this node is a member in: 0

Number of clusters node is a member in: 1

CLUSTER NAME TYPE SHID UUID

mycluster local 267ce7fc-bf55-11df-a3b9-6e8dd877b814

Number of points_of_contact for node: 1

Point-of-contact interface & contact state

en0 UP

------------------------------

Node name: 7502lp02

Cluster shorthand id for node: 2

uuid for node: d1a46164-bf46-11df-94b3-6e8dd877b814

State of node: UP NODE_LOCAL

Smoothed rtt to node: 0

Mean Deviation in network rtt to node: 0

Number of zones this node is a member in: 0

Number of clusters node is a member in: 1

CLUSTER NAME TYPE SHID UUID

mycluster local 267ce7fc-bf55-11df-a3b9-6e8dd877b814

Number of points_of_contact for node: 0

Point-of-contact interface & contact state

n/a

CAA tells you what interfaces have been discovered on a node plus information on those interfaces, including state.

# lscluster –i –n

Node 7502lp01

Node uuid = 110b2422-7efc-11df-aed7-1612a0003002

Number of interfaces discovered = 1

Interface number 1 en0

ifnet type = 6 ndd type = 7

Mac address length = 6

Mac address = 16.12.a0.0.30.2

Smoothed rrt across interface = 7

Mean Deviation in network rrt across interface = 3

Probe interval for interface = 100 ms

ifnet flags for interface = 0x1e080863

ndd flags for interface = 0x21081b

Interface state UP

Number of regular addresses configured on interface = 1

IPV4 ADDRESS: 9.3.28.136 broadcast 9.3.28.159 netmask 255.255.255.224

Number of cluster multicast addresses configured on interface = 1

IPV4 MULTICAST ADDRESS: 228.8.8.8 broadcast 0.0.0.0 netmask 0.0.0.0

CAA tells you what disks are in the cluster plus information on those disks, including the state and type.

# lscluster –d

Storage Interface Query

Cluster Name: mycluster

Cluster uuid: a1bde416-83b9-11df-b438-1612a0004002

Number of nodes reporting = 2

Number of nodes expected = 2

Node oscar-dev8.austin.ibm.com

Node uuid = 9807465b-3194-f1e5-8a04-1f044bc82593

Number of disk discovered = 2

cldisk1

state : UP

uDid : 533E3E213600A0B8000475C200000E7114BC825930F1818 FAStT03IB

Mfcp05VDASD03AIXvscsi

type : CLUSDISK

uUid : 9807465b-3194-f1e5-8a04-1f044bc82593

hdisk1

state : UP

uDid :

uUid : 600a0b80-0047-5d0a-0000-e6094bc82690

type : REPDISK

You can run cluster wide commands on all nodes immediately after configuring your cluster using the clcmd command. No further configuration required.

# clcmd ps -ef

-------------------------------

NODE 7502lp01

-------------------------------

UID PID PPID C STIME TTY TIME CMD

root 1 0 0 Aug 25 - 0:00 /etc/init

root 655526 1 0 Aug 25 - 0:18 [cimserve]

root 1310840 1 0 Aug 25 - 0:00 /usr/ccs/bin/shlap64

root 2490482 1 0 Aug 25 - 0:42 /usr/sbin/srcmstr

root 3145874 3801262 0 11:17:13 - 0:00 telnetd -a

root 3276994 2490482 0 Aug 25 - 0:10 /usr/sbin/snmpd

-------------------------------

NODE 7502lp02

-------------------------------

UID PID PPID C STIME TTY TIME CMD

root 1 0 1 Aug 28 - 0:00 /etc/init

root 1441934 1 0 Aug 28 - 0:39 /usr/sbin/syncd 60

root 1638496 1 0 Aug 28 - 0:00 /usr/ccs/bin/shlap64

root 2228252 2556054 0 Aug 28 - 0:00 /usr/sbin/hostmibd

I believe that PowerHA SystemMirror 7.1 will be the first cluster product from IBM to provide high availability on AIX using the new Cluster Aware AIX features. An IBM Redbook Residency recently covered the integration of CAA and PowerHA. I’m eagerly awaiting the first draft of this book so that I can learn how these two components will “play” together.

Refer to the AIX 7.1 Information Centre for more information on Cluster Aware AIX:

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.clusteraware/claware_main.htm

Refer to the AIX 7.1 Information Centre for more information on AHAFS and the new AIX Event Infrastructure:

http://publib.boulder.ibm.com/infocenter/aix/v7r1/index.jsp?topic=/com.ibm.aix.baseadmn/doc/baseadmndita/aix_ev.htm

fatal: PRNG is not seeded.

cggibbo 11 Comments 18,625 Views

The /dev/*random* devices were missing…….and I had to recreate them!

# tail /var/log/syslog

Oct 15 16:36:01 lpar1 auth|security:crit sshd[380976]: fatal: PRNG is not seeded

# odmget CuDvDr | grep -p random

CuDvDr:

resource = "ddins"

value1 = "random"

value2 = "32"

value3 = ""

# mknod /dev/random c 32 0

# mknod /dev/urandom c 32 1

# randomctl -l

# stopsrc -s sshd

# startsrc -s sshd

How much memory or CPU will that CoD code activate anyway?

cggibbo Tags: power cod gibson mod demand activation aix pod capacity chris code on powervm vet 10 Comments 24,912 Views

This little tip was passed on to me by a friendly IBM hardware engineer many years ago.

When entering a capacity on demand (CoD) code into a Power system, you can tell how many processors and how much memory will be activated, just by looking at the code you’ve given by IBM.

For example, the following codes, when entered for the appropriate Power system, will enable 4 processors (POD) and 64GB of memory (MOD). I can also tell* that once the VET code is entered, this system will be licensed for PowerVM Enterprise Edition (2C28).

System Type: 9117 Serial Number: 06-B719P
Type	Activation Code	Posted Date (MM/DD/YYYY)
VET	F2FB60ERC4FF3DB50CA1F00002C28004177	04/30/2010
POD	FA94E589F2067F0C54680000000400419B	04/30/2010
MOD	B154E7097C231A85821200000064004102	04/30/2010

Activation type definitions
POD:	CUoD Processor Activation Code
MOD:	CUoD Memory Activation Code
TCOD:	On/Off CoD Enablement Code
TMOD:	On/Off CoD Memory Enablement Code
VET:	Virtualization Technology Code
	(PowerVM, Enterprise Enablement, WWPN, Active Memory Expansion)
STDP:	Standard Trial CoD Processor Activation Code
STDM:	Standard Trial CoD Memory Activation Code
STME:	Standard Trial Active Memory Expansion Code
EXCP:	Exception Trial CoD Processor Activation Code
EXCM:	Exception Trial CoD Memory Activation Code
USTA:	Utility CoD Enablement Code
USTO:	Utility CoD Termination Code
URPT:	Utility CoD Reporting Code
PAID:	Utility CoD PrePaid Code

0000 = Reset to base defaults (No PowerVM)
1xxx = Express
2c00 = Standard
2c20 = Enterprise

*Refer to the following link for more information on “How to tell which PowerVM Editions feature was ordered”: https://www-304.ibm.com/support/docview.wss?uid=isg3T1010860

You can obtain your CoD codes from the following website: http://www-912.ibm.com/pod/pod

======

UPDATE 12/9/2011: Script from Chris-R that collects CoD information for a system from the IBM CoD Website. Thanks Chris-R, nice job!

Done!

$ ./activations.pl 9117 10 06ebp
02 processors activated on 05/23/2011
024 GB memory activated on 05/23/2011
02 processors activated on 03/11/2011
016 GB memory activated on 03/11/2011
055 GB memory activated on 03/02/2011
06 processors activated on 02/22/2011
096 GB memory activated on 02/22/2011
01 processors activated on 01/12/2011
016 GB memory activated on 01/12/2011
06 processors activated on 03/17/2010
072 GB memory activated on 03/17/2010
19 processors activated on 03/16/2010
228 GB memory activated on 03/16/2010
================
TOTAL CPU=36 RAM=519168MB (507GB)

$cat activations.pl
#!/usr/bin/perl
#
# PoC Script to go out on the net to the IBM POD site and tally up activations for a given i or p Series machine
# Chris-R rutherfc@gmail.com
# v1.0 9/9/2011
#
# todo - add logic for processor deactivation - who uses that? IBM dont even publish the code on the pod site ;)
#
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->agent("mozilla 8.0");
# $ua->proxy(['http'], 'http://proxy:8080/');
$ua->timeout(10);
use HTTP::Request::Common qw(POST);
if ($#ARGV != 2) {
print "usage: $0 MODEL XX XXXXX e.g $0 9119 83 9f6bf\n";
exit;
}
($model, $serial1, $serial2) = @ARGV;

##### main #####
get('http://www-912.ibm.com/pod/pod',"$serial2.htm");
html2txt("$serial2.htm","$serial2.txt");
total("$serial2.txt");
exit;

sub get # fakes a mozilla browser, fills in the CGI form and snags the returned page to a local html file
{
my $req = (POST $_[0],
["system_type" => $model,
"system_serial_number1" => $serial1,
"system_serial_number2" => $serial2 ]);
$request = $ua->request($req);
$activations = $request->content;
open(POD,">$_[1]");
print POD $activations;
close(POD);
}

sub html2txt # strips out the crap and converts the table to a local txt file to parse
{
open(HTML,"<$_[0]");
open(TXT,">$_[1]");
while (<HTML>) {
if (/<\/table>/) {$f = 0;};
if (/<th>Posted Date \(MM/) {$f = 1;};
if ($f == 1) {
# poor mans HTML::TableExtract - excuse my sed like perl....
s/<tr align="center">/:/g;
s/<[^>][^>]*>//g;
s/ //g;
s/\n//g;
s/:/\n/g;
print TXT $_;
};
};
close(TXT);
close(HTML);
}
sub total # totals up the de/activations to get totals
{
open(TXT,"<$_[0]");
$[ = 1; $\ = "\n";# set array base & output record separator
while (<TXT>) {
($code,$hex,$date) = split(' ', $_, -1);
if (/POD/) {
$p = substr($hex, 27, 2);
print $p . ' processors activated on ' . $date;
$pt = $pt + $p;
};
if (/MOD/) {
$r = substr($hex, 26, 3);
print $r . ' GB memory activated on ' . $date;
$rt = $rt + $r;
};
if (/RMEM/) {
$r = substr($hex, 27, 2);
print $r . ' GB memory activated on ' . $date;
$rt = $rt - $r;
};
};
print '================';
print 'TOTAL CPU=' . $pt . ' RAM=' . $rt*1024 . 'MB (' . $rt . 'GB)';
close(TXT);
}
</TXT></tr></th></HTML>

In a perfect world....rootvg would always reside on hdisk0.

cggibbo Tags: hdisk0 chris_gibson rendev rootvg aix 9 Comments 19,527 Views

In a perfect world, 99.9% of AIX administrators would prefer their systems to look like this:

# lspv | grep rootvg

hdisk0 00c342c68dfcbdfb rootvg active

However, in reality, 99.9% of AIX administrators live with systems that look something like this:

# lspv | grep rootvg

hdisk39 00c342c68dfcbdfb rootvg active

And 99.9% of them don’t have time to tidy up their systems so that rootvg resides on hdisk0.

Most of them have much bigger fish to fry, such as performance, virtualisation, automation, security, project delivery, TPS reports, etc!

If they did have time, they could use the mirrorvg and rendev commands to ‘bring order to the Universe’.

1. Add a new disk to the system.

# lspv

hdisk39 00c342c68dfcbdfb rootvg active

hdisk40 00c342c6161c6b47 None

2. Rename the new disk to hdisk0.

# rendev -l hdisk40 -n hdisk0

hdisk0

3. Add the disk to rootvg and mirror to it.

# extendvg rootvg hdisk0

# mirrorvg rootvg hdisk0

0516-1804 chvg: The quorum change takes effect immediately.

0516-1126 mirrorvg: rootvg successfully mirrored, user should perform

bosboot of system to initialize boot records. Then, user must modify

bootlist to include: hdisk0 hdisk39.

4. Create a boot image on hdisk0.

# bosboot –a –d /dev/hdisk0

bosboot: Boot image is 49180 512 byte blocks.

# ipl_varyon -i

[S 3670248 14942228 07/24/12-10:18:11:104 ipl_varyon.c 1270] ipl_varyon -i

PVNAME BOOT DEVICE PVID VOLUME GROUP ID

hdisk39 YES 00c342c68dfcbdfb0000000000000000 00c342c600004c00

hdisk0 YES 00c342c6161c6b470000000000000000 00c342c600004c00

[E 3670248 0:087 ipl_varyon.c 1410] ipl_varyon: exited with rc=0

5. Check and update the bootlist. Ensure hdisk0 is in the bootlist.

# bootlist –m normal hdisk0

# bootlist –m normal –o

hdisk0 blv=hd5 pathid=0

hdisk0 blv=hd5 pathid=1

6. Unmirror rootvg on hdisk39. Remove hdisk39 from rootvg.

# unmirrorvg rootvg hdisk39

0516-1246 rmlvcopy: If hd5 is the boot logical volume, please run 'chpv -c <diskname>'

as root user to clear the boot record and avoid a potential boot

off an old boot image that may reside on the disk from which this

logical volume is moved/removed.

0516-1804 chvg: The quorum change takes effect immediately.

0516-1144 unmirrorvg: rootvg successfully unmirrored, user should perform

bosboot of system to reinitialize boot records. Then, user must modify

bootlist to just include: hdisk0.

# lspv –l hdisk39

# reducevg rootvg hdisk39

7. Order in the Universe has been restored.

# lspv | grep rootvg

hdisk0 00c342c6161c6b47 rootvg active

Of course, all of this assumes that the name, hdisk0, is not already in use by a hdisk in another volume group on the system.

Again, if you have the time, then you could perform the following to rectify the situation.

1. datavg resides on hdisk0.

# lspv | grep datavg

hdisk0 00c342c6161c6b47 datavg active

2. Unmount all data file systems in this volume group. Varyoff the VG.

# unmount /datafs

# varyoffvg datavg

3. Rename hdisk0 to hdisk99 (or something other than hdisk0).

# rendev -l hdisk0 -n hdisk99

hdisk99

4. Varyon the volume group, datavg. Mount the data file systems in datavg.

# varyonvg datavg

# mount /datafs

5. datavg now resides on (the newly renamed) hdisk99.

# lspv | grep datavg

hdisk99 00c342c6161c6b47 datavg

Now that the name, hdisk0, is free, you can perform the steps, outlined above, to restore ‘Order in the Universe’ and put rootvg back on hdisk0!

nimadm - multibos error!?

cggibbo Tags: altinst_rootvg attention: client nimadm: cloning error returned multibos 6 unexpected init_multibos() 11 7.1 phase alt_disk_copy: nimadm on result an .1 0505-218 5.3 aix 0505-187 9 Comments 14,549 Views

I’ve been working with a customer recently on an issue with nimadm. They were attempting to migrate a system from AIX 5.3 to 7.1 using nimadm. The NIM client AIX level was 5.3 TL12 SP4 and the NIM master was running AIX 7.1 TL1 SP1.

lpar1 : / # oslevel -s

5300-12-04-1119

root@nim1 : / # oslevel -s

7100-01-01-1141

They were running the following nimadm command:

# nimadm -j nimadmvg -c lpar1 -s spotaix710101 -l lpp_sourceaix710101 -d hdisk1 -Y

The nimadm operation would always fail at phase 11:

+-----------------------------------------------------------------------------+
Executing nimadm phase 11.
+-----------------------------------------------------------------------------+
Cloning altinst_rootvg on client, Phase 3.
Client alt_disk_install command: alt_disk_copy -j -M 7.1 -P3 -d "hdisk1"
## Phase 3 ###################
Verifying altinst_rootvg...
alt_disk_copy: 0505-218 ATTENTION: init_multibos() returned an unexpected result.
Cleaning up.
forced unmount of /alt_inst/var/log
forced unmount of /alt_inst/var/log
forced unmount of /alt_inst/var
forced unmount of /alt_inst/var
forced unmount of /alt_inst/usr/local
forced unmount of /alt_inst/usr/local
forced unmount of /alt_inst/usr
forced unmount of /alt_inst/usr
forced unmount of /alt_inst/tmp
forced unmount of /alt_inst/tmp
forced unmount of /alt_inst/opt
forced unmount of /alt_inst/opt
forced unmount of /alt_inst/home
forced unmount of /alt_inst/home
forced unmount of /alt_inst/admin
forced unmount of /alt_inst/admin
forced unmount of /alt_inst
forced unmount of /alt_inst
0505-187 nimadm: Error cloning altinst_rootvg on client.
Cleaning up alt_disk_migration on the NIM master.
Cleaning up alt_disk_migration on client lpar1.
Client alt_disk_install command: alt_disk_install -M 7.1 -X
Bootlist is set to the boot disk: hdisk0 blv=hd5

We also found the init_multibos error in the /var/adm/ras/alt_disk_inst.log file on the NIM client:

Tue Nov 22 14:48:12 EETDT 2011

cmd: /ALT_MIG_SPOT/sbin/alt_disk_copy -j -M 7.1 -P3 -d hdisk1

Verifying altinst_rootvg...

alt_disk_copy: 0505-218 ATTENTION: init_multibos() returned an unexpected result.

Cleaning up.

Given that the error appeared to be related to init_multibos, we assumed the failure was due to some multibos checks being performed by alt_disk_copy on the client. The client system did not have an existing multibos standby instance. So, we tried two things: First we created a standby instance on the client (multibos –s –X) and re-tried the nimadm operation. This failed. Next we removed the standby instance (multibos –R) and re-tried the nimadm operation. This worked and the client then migrated to AIX 7.1 successfully. We re-tried the same operations (i.e. create standby instance, remove standby instance & nimadm) several times and each worked as expected.

So it appeared that the unofficial work around to this problem would be to create and then remove a standby multibos instance prior to the nimadm migrate. However, the customer has over 200 LPARs that they need to migrate to AIX 7.1. If possible they would really rather avoid this extra step in the AIX 7.1 migration plan. We’ve made contact with IBM support and are hoping they can assist us in identifying the root cause of the issue and provide us with an official solution to the problem.

And just yesterday we hit the same problem when migrating from AIX 6.1 to 7.1 using nimadm. I’ll update my blog with any progress we make with this problem. In the meantime, our unofficial work around will get us “out of hot water”!

UPDATE (14/12/2011): The simple fix is to remove the /bos_inst directory before attempting the AIX migration. i.e.

# rm –r /bos_inst

In my nimadm article I wrote:

If you have a multibos image in rootvg, remove it. AIX migrations are not supported with multibos enabled systems. Ensure all rootvg LVs are renamed to their legacy names. If necessary, create a new instance of rootvg and reboot the LPAR. For example:

# multibos –sXp

# multibos –sX

# shutdown –Fr

Confirm the legacy LV names are now in use that is, not bos_.

# lsvg -l rootvg | grep hd | grep open

hd6 paging 80 160 2 open/syncd N/A

hd8 jfs2log 1 2 2 open/syncd N/A

hd4 jfs2 1 2 2 open/syncd /

hd2 jfs2 7 14 2 open/syncd /usr

hd3 jfs2 16 32 2 open/syncd /tmp

hd1 jfs2 1 2 2 open/syncd /home

hd9var jfs2 8 16 2 open/syncd /var

hd7 sysdump 8 8 1 open/syncd N/A

hd7a sysdump 8 8 1 open/syncd N/A

hd10opt jfs2 8 16 2 open/syncd /opt

Remove the old multibos instance.

# multibos -R

Unfortunately, it appears that ‘multibos –R’ may not clean up the /bos_inst directory. If this directory exists the nimadm operation will most likely fail.

IVM and VLAN Tagging

cggibbo Tags: ivm chhwres vlan lshwres aix tag vios sea 9 Comments 25,478 Views

I was working with a customer recently on a Power Blade that was running the Integrated Virtualisation Manager (IVM). They’d installed a VIO partition onto the Blade and had hoped to install a couple of AIX LPARs on the system. However they didn’t get very far.

As soon as they attempted to NIM install the LPARs, they would get stuck at trying to ping the NIM master from the client. Basically, the Shared Ethernet Adapter (SEA) was not working properly and none of the LPARs could communicate with the external network. So they asked for some assistance.

The Blade server name was Server-8406-71Y-SN06BF99Z. The SEA was configured as ent7.

# lsdev -Cc adapter

ent0 Available Logical Host Ethernet Port (lp-hea)

ent1 Available Logical Host Ethernet Port (lp-hea)

ent2 Available Virtual I/O Ethernet Adapter (l-lan)

ent3 Available Virtual I/O Ethernet Adapter (l-lan)

ent4 Available Virtual I/O Ethernet Adapter (l-lan)

ent5 Available Virtual I/O Ethernet Adapter (l-lan)

ent6 Available EtherChannel / IEEE 802.3ad Link Aggregation

ent7 Available Shared Ethernet Adapter

ent9 Available Virtual I/O Ethernet Adapter (l-lan)

fcs0 Available 02-00 8Gb PCIe FC Blade Expansion Card (7710322577107601)

fcs1 Available 02-01 8Gb PCIe FC Blade Expansion Card (7710322577107601)

ibmvmc0 Available Virtual Management Channel

lhea0 Available Logical Host Ethernet Adapter (l-hea)

sissas0 Available 01-08 PCI-X266 Planar 3Gb SAS Adapter

usbhc0 Available 00-08 USB Host Controller (33103500)

usbhc1 Available 00-09 USB Host Controller (33103500)

usbhc2 Available 00-0a USB Enhanced Host Controller (3310e000)

vhost0 Available Virtual SCSI Server Adapter

vsa0 Available LPAR Virtual Serial Adapter

vts0 Available Virtual TTY Server Adapter

The SEA was configured with Port-VLAN ID (PVID) of 68 without any VLAN tags. This was the root cause of the problem.

# lsattr -El ent7

accounting enabled Enable per-client accounting of network statistics True

ctl_chan Control Channel adapter for SEA failover True

gvrp no Enable GARP VLAN Registration Protocol (GVRP) True

ha_mode disabled High Availability Mode True

jumbo_frames no Enable Gigabit Ethernet Jumbo Frames True

large_receive no Enable receive TCP segment aggregation True

largesend 0 Enable Hardware Transmit TCP Resegmentation True

netaddr 0 Address to ping True

pvid 68 PVID to use for the SEA device True

pvid_adapter ent9 Default virtual adapter to use for non-VLAN-tagged packets True

qos_mode disabled N/A True

real_adapter ent6 Physical adapter associated with the SEA True

thread 1 Thread mode enabled (1) or disabled (0) True

virt_adapters ent9 List of virtual adapters associated with the SEA (comma separated) True

$ entstat -all ent8 | grep -i vlan

VLAN Ids :

VLAN Extract: False

VLAN tagged filtering mode: Filter according to VLAN permit array

Max number of VLAN IDs per HEA port: 20

VLAN Extract: False

VLAN tagged filtering mode: Filter according to VLAN permit array

Max number of VLAN IDs per HEA port: 20

Invalid VLAN ID Packets: 5388

Port VLAN ID: 68

VLAN Tag IDs: None

On the network switch port, the native VLAN (PVID), was configured as 11, with VLAN tag 68 added as an allowed VLAN. If the client LPARs tried to access the network using a PVID of 68, instead of a VLAN TAG of 68, they would get stuck at the switch port i.e. the un-tagged packets for 10.1.68.X via PVID 11 would fail. The packets for 10.1.68.X needed to be tagged with VLAN id 68 in order for the switch to pass the traffic.

So the question was, how do we add VLAN tags in the IVM environment? If we’d been using a HMC, then this would be simple to fix. Just add the VLAN tags into the Virtual Ethernet Adapter used by the SEA and we’d be done.

We had to use the lshwres and chhwres commands to resolve this one. First we listed the virtual adapters known to the VIO server (IVM). At slot 12, we found our SEA adapter with port_vlan_id set to 68 and addl_vlan_ids set to none.

$ lshwres -r virtualio --rsubtype eth --level lpar

lpar_name=06-BF99Z,lpar_id=1,slot_num=3,state=1,ieee_virtual_eth=0,port_vlan_id=1,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62803

lpar_name=06-BF99Z,lpar_id=1,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=2,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62804

lpar_name=06-BF99Z,lpar_id=1,slot_num=5,state=1,ieee_virtual_eth=0,port_vlan_id=3,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62805

lpar_name=06-BF99Z,lpar_id=1,slot_num=6,state=1,ieee_virtual_eth=0,port_vlan_id=4,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62806

lpar_name=06-BF99Z,lpar_id=1,slot_num=12,state=1,ieee_virtual_eth=0,port_vlan_id=68,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA6280C

lpar_name=aixlpar1,lpar_id=2,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=68,addl_vlan_ids=none,is_trunk=0,trunk_priority=0,is_required=0,mac_addr=F67D5345AD04

We needed to change port_vlan_id to 11 and addl_vlan_ids to 68. We also required the ieee_virtual_eth value set to 1.

First we removed the existing SEA adapter, as we would not be able to make changes to it while it was “active”. We then removed the adapter from slot 12 and then re-added it, again at slot 12, with port_vlan_id and addl_vlan_ids set to the desired values.

$ chhwres -m Server-8406-71Y-SN06BF99Z -p 06-BF99Z -r virtualio --rsubtype eth -s 12 -o r

$ chhwres -m Server-8406-71Y-SN06BF99Z -p 06-BF99Z -r virtualio --rsubtype eth -s 12 -o a -a "ieee_virtual_eth=1,port_vlan_id=11,addl_vlan_ids=68,is_trunk=1,trunk_priority=1" -d 5

$ lshwres -r virtualio --rsubtype eth --level lpar

lpar_name=06-BF99Z,lpar_id=1,slot_num=3,state=1,ieee_virtual_eth=0,port_vlan_id=1,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62803

lpar_name=06-BF99Z,lpar_id=1,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=2,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62804

lpar_name=06-BF99Z,lpar_id=1,slot_num=5,state=1,ieee_virtual_eth=0,port_vlan_id=3,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62805

lpar_name=06-BF99Z,lpar_id=1,slot_num=6,state=1,ieee_virtual_eth=0,port_vlan_id=4,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62806

lpar_name=06-BF99Z,lpar_id=1,slot_num=12,state=1,ieee_virtual_eth=1,port_vlan_id=11,addl_vlan_ids=68,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA6280C

lpar_name=aixlpar1,lpar_id=2,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=68,addl_vlan_ids=none,is_trunk=0,trunk_priority=0,is_required=0,mac_addr=F67D5345AD04

Using the mkvdev command we created the SEA again. Then using the entstat command we found that the PVID and VLAN tags had been configured correctly.

$ mkvdev -sea ent6 -vadapter ent2 -default ent2 -defaultid 11

ent7 Available

en7

et7

$ entstat -all ent7 | grep -i vlan

VLAN Ids :

VLAN Extract: False

VLAN tagged filtering mode: Filter according to VLAN permit array

Max number of VLAN IDs per HEA port: 20

VLAN Extract: False

VLAN tagged filtering mode: Filter according to VLAN permit array

Max number of VLAN IDs per HEA port: 20

Invalid VLAN ID Packets: 5388

Port VLAN ID: 11

VLAN Tag IDs: 68

Once this was done, the client LPARs were able to ping the NIM master. The customer happily started installing AIX onto each of the blades Partitions.

A related link: http://www.aixmind.com/?p=2133

Tips and tricks with dsh on AIX

cggibbo Tags: dsh chris gibson there 0042-053 22 no different aix lsnim object dsh_remote_opts dsh_context named port nim is 8 Comments 9,685 Views

I received the following errors whilst running dsh on a NIM master recently.

root@nim1 : / # dsh -waixlpar1 date

0042-053 lsnim: there is no NIM object named "aixlpar1"

The node aixlpar1 is not defined in NIM database.

aixlpar1: Mon Aug 4 14:01:57 EET 2014

I had to set the following environment variable, shown below. By setting DSH_CONTEXT to DSH this prevented the dsh command from referring to the NIM database and instead forced it to query a user-defined node list.

root@nim1 : / # export DSH_CONTEXT=DSH

root@nim1 : / # dsh -waixlpar1 date

aixlpar1: Mon Aug 4 14:02:22 EET 2014

root@nim1 : / # env | grep -i dsh

DSH_CONTEXT=DSH

DSH_NODE_RSH=/usr/bin/ssh

root@nim1 : / # dsh -q

DSH:DCP_DEVICE_OPTS=

DSH:DCP_DEVICE_RCP=

DSH:DCP_NODE_OPTS=-q

DSH:DCP_NODE_RCP=/usr/bin/scp

DSH:DSH_CONTEXT=DSH

DSH:DSH_DEVICE_LIST=

DSH:DSH_DEVICE_OPTS=

DSH:DSH_DEVICE_RCP=

DSH:DSH_DEVICE_RSH=

DSH:DSH_ENVIRONMENT=

DSH:DSH_FANOUT=

DSH:DSH_LOG=

DSH:DSH_NODEGROUP_PATH=

DSH:DSH_NODE_LIST=/usr/local/etc/csmnodes.list

DSH:DSH_NODE_OPTS=

DSH:DSH_NODE_RCP=

DSH:DSH_NODE_RSH=/usr/bin/ssh

DSH:DSH_OUTPUT=

DSH:DSH_PATH=

DSH:DSH_REPORT=

DSH:DSH_SYNTAX=

DSH:DSH_TIMEOUT=

DSH:RSYNC_RSH=

Here’s another dsh tip I picked up. By default dsh will use the default port for ssh connections to nodes. For example, by default sshd listens on port 22 on an AIX node. I recently came across a customer environment where they had configured sshd to listen on port 6666 (not the real port number!). They wanted to use dsh from a NIM master which would connect to all the defined nodes in their custom list. When they ran it they got the following error message:

# dsh date

aixlpar1: ssh: connect to host aixlpar1 port 22: Connection refused

dsh: 2617-009 aixlpar1 remote shell had exit code 255

On the AIX node, we could see that sshd was listening on port 6666:

# netstat -a | grep 6666 | grep LIST

tcp6 0 0 *.6666 *.* LISTEN

tcp4 0 0 *.6666 *.* LISTEN

We needed to find a way to force dsh to use a different port number when starting the ssh connection. This was accomplished by setting the DSH_REMOTE_OPTS variable, as shown below.

[root@nim1]/ # export DSH_REMOTE_OPTS=-p6666

[root@nim1]/ # dsh date

aixlpar1: Tue Aug 5 17:37:16 2014

[root@nim1]/ # env | grep DSH

DSH_REMOTE_CMD=/usr/bin/ssh

DSH_NODE_LIST=/etc/ibm/sysmgt/dsm/nodelist

DSH_REMOTE_OPTS=-p6666

DSH_NODE_RSH=/usr/bin/ssh

DSH CONTEXT

The DSH CONTEXT is the in-built context for all the DSH Utilities commands. It permits a user-defined node group database contained in the local file system. The DSH_NODEGROUP_PATH environment variable specifies the path to the node group database. Each file in this directory represents a node group, and contains one host name or TCP/IP address for each node that is a group member. Blank lines and comment lines beginning with a # symbol are ignored. If all nodes are requested for the DSH CONTEXT, a full node list is built from all groups in the DSH_NODEGROUP_PATH directory, and cached in /var/ibm/sysmgt/dsm/dsh/$DSH_NODEGROUP_PATH/AllNodes. This file is recreated each time a group file is modified or added to the DSH_NODEGROUP_PATH directory. Device targets are not supported in the DSH context.

http://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.cmds2/dsh.htm?lang=en

DSH_REMOTE_OPTS

Includes the options specified in the remote command when the command is forwarded to the remote nodes.

http://ps-2.kev009.com/wisclibrary/aix52/usr/share/man/info/en_US/a_doc_lib/cmds/aixcmds3/mgmtsvr.htm

Installing PowerVP Service Pack 1

cggibbo 8 Comments 7,266 Views

I finally got PowerVP installed in my lab environment today. As part of the server agent installation process I needed to install the latest service pack (SP1) for PowerVP. After I downloaded the service pack from IBM Fix Central, I tried to run the install script from the command line on my AIX partition. It failed everytime I ran it, no matter what options I supplied the installer script!

[root@gibbo]/tmp/cg # ./PowerVP.bin.SP1 –i Silent

Preparing to install...

Extracting the installation resources from the installer archive...

Configuring the installer for this system's environment...

Launching installer...

Graphical installers are not supported by the VM. The console mode will be used instead...

===============================================================================

Choose Locale...

----------------

1- Bahasa Indonesia

2- Català

3- Dansk

4- Deutsch

->5- English

6- Español

7- Euskara

8- Français

9- Français (Canada)

10- Italiano

11- Nederlands

12- Norsk

13- Português

14- Português (Brasil)

15- Suomi

16- Svenska

CHOOSE LOCALE BY NUMBER:

=======================================================

Installer User Interface Mode Not Supported

The installer cannot run in this UI mode. To specify the interface mode, use the -i command-line option, followed by the UI mode identifier. The valid UI modes identifiers are GUI, Console, and Silent.

The only way I could get the SP installed was to use the GUI installer. I installed VNC server on my AIX partition, connected to the VNC server and then ran the installer script (as shown in the following screenshots). This worked fine.

The update fileset was extracted to the following location:

[root@gibbo]/IBM/PowerVP/PowerVP_Installation/PowerVP_Agent_Installation_Instructions/AIX # ls -ltr

total 60328

-rwxrwxr-x 1 root system 30171648 Nov 6 11:28 powervp.1.1.0.0.bff

-rwxrwxr-x 1 root system 699392 Dec 6 08:52 powervp.1.1.0.1.bff

-rwxrwxr-x 1 root system 1185 Dec 9 08:53 README_Fixpack.html

-rwxrwxr-x 1 root system 2001 Feb 17 14:19 README.html

-rw-r--r-- 1 root system 849 Feb 17 15:03 .toc

As instructed in the fixpack readme, I stopped the PowerVP processes before installing the service pack. I also unloaded the associated kernel extension, just in case.

# ps -ef | grep -i powervp

root 4128956 6291676 0 14:56:23 - 0:00 /opt/ibm/powervp/syslet

root 6291676 1 0 14:56:23 - 0:00 /bin/ksh /opt/ibm/powervp/PowerVP.sh

[root@gibbo]/ # kill 4128956 6291676

[root@gibbo]/ # cat /opt/ibm/powervp/PowerVP.sh

#!/bin/ksh

/opt/ibm/powervp/loadkex

/opt/ibm/powervp/syslet

/opt/ibm/powervp/unloadkex

Exit

[root@gibbo]/ # /opt/ibm/powervp/unloadkex

Then I installed the update (with smitty update_all).

COMMAND STATUS

Command: OK stdout: yes stderr: no

Before command completion, additional instructions may appear below.

[MORE...40]

powervp.rte 1.1.0.1

Finished processing all filesets. (Total time: 1 secs).

+-----------------------------------------------------------------------------+

Pre-commit Verification...

+-----------------------------------------------------------------------------+

Verifying requisites...done

Results...

SUCCESSES

---------

Filesets listed in this section passed pre-commit verification

and will be committed.

Selected Filesets

-----------------

powervp.rte 1.1.0.1 # PowerVP Fixpack for AIX

<< End of Success Section >>

+-----------------------------------------------------------------------------+

Committing Software...

+-----------------------------------------------------------------------------+

installp: COMMITTING software for:

powervp.rte 1.1.0.1

Finished processing all filesets. (Total time: 1 secs).

+-----------------------------------------------------------------------------+

Summaries:

+-----------------------------------------------------------------------------+

Installation Summary

--------------------

Name Level Part Event Result

-------------------------------------------------------------------------------

powervp.rte 1.1.0.1 USR APPLY SUCCESS

powervp.rte 1.1.0.1 USR COMMIT SUCCESS

[root@gibbo]/ # lslpp -l powervp.rte

Fileset Level State Description

----------------------------------------------------------------------------

Path: /usr/lib/objrepos

powervp.rte 1.1.0.1 COMMITTED PowerVP Fixpack for AIX

Path: /etc/objrepos

powervp.rte 1.1.0.0 COMMITTED PowerVP for AIX

Next, I restarted the PowerVP agent.

[root@gibbo]/ # ps ax |grep powervp

[root@gibbo]/ # /etc/rc.d/rc2.d/SPowerVP

[root@gibbo]/ # ps ax |grep powervp

4128966 pts/0 A 0:00 /opt/ibm/powervp/syslet

9109542 pts/0 A 0:00 /bin/ksh /opt/ibm/powervp/PowerVP.sh

At the same time, I also updated the PowerVP GUI on my laptop to the latest service pack as well.

After that, I was good to go and PowerVP started up fine.

The PowerVP log file will tell you if the agent has started.

[root@gibbo]/var/log # cat powervp.log
2014-02-17 17:02:46.750 MSG0080: 10.2.55.99 - Running as the system level agent.
2014-02-17 17:02:47.860 MSG0082: Agent running, waiting for clients to request data.

By the way, if you try to start the PowerVP server agent on a system that isn't at the correct fimrware level, then you'll find the following message in /var/log/powervp.log:

# cat powervp.log
2014-02-14 15:28:01.844 Error - MSG0047: MSG0205: System does not support the required interfaces. Firmware version 7.7 or greater is required.. [syslet.C:6090]

Modified by cggibbo

Multibos saved my bacon!

cggibbo Tags: bacon max_xfer_size multibos chris_gibson aix 8 Comments 9,942 Views

A colleague of mine was planning to modify the max_xfer_size attribute on a couple of FC adapters in one of his AIX LPARs. As he was describing his plan to me, I asked him how he intended to back out of the change should the LPAR fail to boot after the modifications. “But, what could possibly go wrong?” he fired back. I advised him to use multibos to create a standby (backup) instance of the AIX OS, just in case. He begrudgingly did so, just to keep me happy.

The next day he told me the following tale.

He had modified the FC adapters max_xfer_size attribute as planned. First, checking the current values, for the attribute on both adapters.

aixlpar1 : / # lsattr -El fcs0 -a max_xfer_size

max_xfer_size 0x100000 Maximum Transfer Size True

aixlpar1 : / # lsattr -El fcs1 -a max_xfer_size

max_xfer_size 0x100000 Maximum Transfer Size True

He’d created a standby AIX instance before making changes to the adapters. He also prevented multibos from changing the bootlist to the standby boot logical volume (BLV).

aixlpar1 : / # multibos -sXt

Initializing multibos methods ...

Initializing log /etc/multibos/logs/op.alog ...

Gathering system information ...

+-----------------------------------------------------------------------------+

Setup Operation

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

Creating image.data file ...

He modified the FC adapters as planned.

aixlpar1 : / # chdev -l fcs0 -a max_xfer_size=0x200000 -P

fcs0 changed

aixlpar1 : / # chdev -l fcs1 -a max_xfer_size=0x200000 -P

fcs1 changed

aixlpar1 : / # lsattr -El fcs0 -a max_xfer_size

max_xfer_size 0x200000 Maximum Transfer Size True

aixlpar1 : / # lsattr -El fcs1 -a max_xfer_size

max_xfer_size 0x200000 Maximum Transfer Size True

He verified that the standby instance still held the original values for both FC adapters.

aixlpar1 : / # multibos -S

Initializing multibos methods ...

Initializing log /etc/multibos/logs/op.alog ...

Gathering system information ...

+-----------------------------------------------------------------------------+

Multibos Shell Operation

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Mounting all standby BOS file systems ...

Mounting /bos_inst

Mounting /bos_inst/usr

Mounting /bos_inst/var

Mounting /bos_inst/opt

+-----------------------------------------------------------------------------+

Multibos Root Shell

+-----------------------------------------------------------------------------+

Starting multibos root shell ...

Active boot logical volume is hd5.

Script command is started. The file is /etc/multibos/logs/scriptlog.120713124518.txt.

aixlpar1 : / # lsattr -El fcs0 -a max_xfer_size

max_xfer_size 0x100000 Maximum Transfer Size True

aixlpar1 : / # lsattr -El fcs1 -a max_xfer_size

max_xfer_size 0x100000 Maximum Transfer Size True

aixlpar1 : / # exit

Script command is complete. The file is /etc/multibos/logs/scriptlog.120713124518.txt.

Stopping multibos root shell ...

Compressing script log file ...

Compressed script log file is /etc/multibos/logs/scriptlog.120713124518.txt.Z

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Unmounting all standby BOS file systems ...

Unmounting /bos_inst/opt

Unmounting /bos_inst/var

Unmounting /bos_inst/usr

Unmounting /bos_inst

Log file is /etc/multibos/logs/op.alog

Return Status = SUCCESS

Then he manually changed the LPARs boot list to include the standby BLV.

aixlpar1 : / # bootlist -m normal hdisk2 blv=hd5 hdisk2 blv=bos_hd5

aixlpar1 : / # bootlist -m normal -o

hdisk2 blv=hd5 pathid=0

hdisk2 blv=hd5 pathid=1

hdisk2 blv=bos_hd5 pathid=0

hdisk2 blv=bos_hd5 pathid=1

He carefully recorded the bootlist output, just in case the boot failed with new max_xfer_size values. He could use the vdevice name and location to manually select the standby BLV to start the system in an emergency.

aixlpar1 : / # bootlist -m normal -ov

'ibm,max-boot-devices' = 0x5

NVRAM variable: (boot-device=/vdevice/vfc-client@30000014/disk@50060e8006d0206a,1000000000000:2 /vdevice/vfc-client@3000001e/disk@50060e8006d0207a,1000000000000:2 /vdevice/vfc-client@30000014/disk@50060e8006d0206a,1000000000000:4 /vdevice/vfc-client@3000001e/disk@50060e8006d0207a,1000000000000:4)

Path name: (/vdevice/vfc-client@30000014/disk@50060e8006d0206a,1000000000000:2)

match_specific_info: ut=disk/fcp/htcvspmpio

hdisk2 blv=hd5 pathid=0

Path name: (/vdevice/vfc-client@3000001e/disk@50060e8006d0207a,1000000000000:2)

match_specific_info: ut=disk/fcp/htcvspmpio

hdisk2 blv=hd5 pathid=1

Path name: (/vdevice/vfc-client@30000014/disk@50060e8006d0206a,1000000000000:4)

match_specific_info: ut=disk/fcp/htcvspmpio

hdisk2 blv=bos_hd5 pathid=0

Path name: (/vdevice/vfc-client@3000001e/disk@50060e8006d0207a,1000000000000:4)

match_specific_info: ut=disk/fcp/htcvspmpio

hdisk2 blv=bos_hd5 pathid=1

He restarted the LPAR using the primary BLV, with the modified FC attributes.

The system hung at LED 554.

hscroot@hmc1:~> lsrefcode -m 795-1 -r lpar --filter "lpar_names=aixlpar1" -F lpar_name:refcode

aixlpar1:0554

He restarted the LPAR in Open Firmware mode.

At the Open Firmware prompt, he entered the following to boot the LPAR from the standby instance BLV:

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

1 = SMS Menu 5 = Default Boot List

8 = Open Firmware Prompt 6 = Stored Boot List

Memory Keyboard Network SCSI Speaker ok

0 > boot /vdevice/vfc-client@30000014/disk@50060e8006d0206a,1000000000000:4 |

The system booted OK on the standby AIX instance.

Elapsed time since release of system processors: 28858 mins 55 secs

-------------------------------------------------------------------------------

Welcome to AIX.

boot image timestamp: 02:39:13 07/13/2012

The current time and date: 02:54:02 07/13/2012

processor count: 1; memory size: 4096MB; kernel size: 35062697

boot device: /vdevice/vfc-client@30000014/disk@50060e8006d0206a,1000000000000:4

-------------------------------------------------------------------------------

The FC adapters were running on the previous values, stored in the standby instance of AIX.

aixlpar1 : / # lsattr -El fcs0 -a max_xfer_size

max_xfer_size 0x100000 Maximum Transfer Size True

aixlpar1 : / # lsattr -El fcs1 -a max_xfer_size

max_xfer_size 0x100000 Maximum Transfer Size True

The original AIX instance still held the modified values for the FC adapters.

aixlpar1 : / # multibos -S

Initializing multibos methods ...

Initializing log /etc/multibos/logs/op.alog ...

Gathering system information ...

+-----------------------------------------------------------------------------+

Multibos Shell Operation

+-----------------------------------------------------------------------------+

Verifying operation parameters ...

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Mounting all standby BOS file systems ...

Mounting /bos_inst

Mounting /bos_inst/usr

Mounting /bos_inst/var

Mounting /bos_inst/opt

+-----------------------------------------------------------------------------+

Multibos Root Shell

+-----------------------------------------------------------------------------+

Starting multibos root shell ...

Active boot logical volume is bos_hd5.

Script command is started. The file is /etc/multibos/logs/scriptlog.120713125542.txt.

aixlpar1 : / # lsattr -El fcs0 -a max_xfer_size

max_xfer_size 0x200000 Maximum Transfer Size True

aixlpar1 : / # lsattr -El fcs1 -a max_xfer_size

max_xfer_size 0x200000 Maximum Transfer Size True

aixlpar1 : / # exit

Script command is complete. The file is /etc/multibos/logs/scriptlog.120713125542.txt.

Stopping multibos root shell ...

Compressing script log file ...

Compressed script log file is /etc/multibos/logs/scriptlog.120713125542.txt.Z

+-----------------------------------------------------------------------------+

Mount Processing

+-----------------------------------------------------------------------------+

Unmounting all standby BOS file systems ...

Unmounting /bos_inst/opt

Unmounting /bos_inst/var

Unmounting /bos_inst/usr

Unmounting /bos_inst

Log file is /etc/multibos/logs/op.alog

Return Status = SUCCESS

The cause of the 554 hang appeared to be related to the fact that the VIOS physical adapters needed their max_xfer_size value changed to the new value before the client LPAR virtual fibre channel adapters were modified.

My colleague was glad he used multibos. It saved his bacon.

Feed for Blog Entries ▼ Feed for Blog Comments ▼

Blogs

Chris's AIX Blog

▼ Tags

▼ Similar Blogs

AIX Down Unde...

Asset Managem...

AIXpert Blog

Application P...

Application I...

▼ Similar Ideation Blogs

IBM PureData-...

IBM Forms Exp...

Time recordin...

IBM Platform ...

Rational Soft...

▼ Archive

▼ Blog Authors

Chris's AIX Blog

Tuning Virtual Ethernet adapters for (even) better backup performance.

Migrating to AIX 7.1 with nimadm.

NIM from A to Z in AIX 5L Redbook.

Renaming devices in AIX 7.1

DLPAR not doing it for you anymore?

AIX 7.1 – DSM dgetmacs.

Replicated rootvg, ghostdev and NPIV

Cluster Aware AIX

fatal: PRNG is not seeded.

How much memory or CPU will that CoD code activate anyway?

In a perfect world....rootvg would always reside on hdisk0.

nimadm - multibos error!?

IVM and VLAN Tagging

Tips and tricks with dsh on AIX

Installing PowerVP Service Pack 1

Multibos saved my bacon!