Chris's AIX Blog

On more than one occasion I’ve written articles and blogs on how to migrate AIX systems to a new version of the AIX operating system. You can find a list these articles (and others) here. Each of the NIM articles focuses on migrating AIX systems (which are configured as NIM clients). The approach is always the same; use a NIM master to migrate the NIM clients to the next release of AIX. What they don’t discuss is how to migrate your NIM master to a new version of the OS. This is the very first step you should take before you consider migrating your clients. The NIM master should always be at the same or higher AIX level than the level you are migrating to on a client. So, in this latest article, I’ll discuss, step by step, how to migrate your NIM master from AIX 7.1 to AIX 7.2 (the latest release of the AIX OS).

 

https://www.ibm.com/developerworks/community/blogs/cgaix/resource/Migrating_NIMMaster_AIX72_VML.pdf

When troubleshooting Fibre Channel adapter issues on AIX (or VIOS), I often use the fcstat command to assist me in the process. The command may display error numbers similar to the following:

 

$ fcstat fcs0

 

Error opening device: /dev/fscsi0

errno: 00000045

 

The error number (errno) displayed can (in some cases) be used to identify the root cause of a problem. In the example above, the error number is 45. This number is in hex. If we convert this to decimal, the number is 69. Now, if we look for 69 in the /usr/include/errno.h file (on AIX), we discover that this error number relates to a "Network is down" event.

 

# fcstat fcs0

Error opening device: /dev/fscsi0

errno: 00000045

 

# echo "ibase=16; 45"|bc

69

 

# grep 69 /usr/include/errno.h

#define ENETDOWN        69      /* Network is down */

 

The AIX error report (errpt or errlog on VIOS) also tells me that there's some type of link error. This helps me focus my investigation toward the most likely problem area. In this case, I suspect either a physical link or cable problem between my FC adapter and the SAN switch.

 

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION

7BFEEA1F   1116121816 T H fcs0           LINK ERROR

 

# errpt -aN fcs0 | grep -p Desc

...

Description

LINK ERROR

 

We can use the same process for other errors displayed by the fcstat command. For example:

 

# fcstat fcs0

Error opening device: /dev/fscsi0

errno: 00000046

 

# echo "ibase=16; 46"|bc

70

 

# grep 70 /usr/include/errno.h

#define ENETUNREACH     70      /* Network is unreachable */

 

This particular error may direct us to the solution outlined in the following tech note:

 

Fibre channel workspace takes over 2 minutes to load but returns no data

http://www-01.ibm.com/support/docview.wss?uid=swg21600064

 

 

fcstat Command (AIX 6.1)

http://www.ibm.com/support/knowledgecenter/ssw_aix_61/com.ibm.aix.cmds2/fcstat.htm

 

fcstat Command (AIX 7.1)

http://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.cmds2/fcstat.htm

 

fcstat Command (AIX 7.2)

http://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.cmds2/fcstat.htm

 

 

 

I’ve been working with a customer recently that required a special kind of disaster recovery capability for their shared storage pool (SSP) environment. The customer had implemented a private cloud solution in their existing POWER environment. The solution consisted of an IBM SmartCloud deployment with IBM Systems Director and VMControl managing the PowerVM deployment of virtual machines (AIX partitions) across the landscape.

 

The decision to use shared storage pools was driven by the fact that the customer was using SAN storage that could not be managed by VMControl. However, shared storage pool support was available with VMControl. This meant that the customer could continue using their non-IBM disk in their private cloud. The customer quickly noticed the many advantages that SSPs bring to a POWER cloud environment, such as very fast virtual machine deployment using snapshots and linked-clones.

 

Traditionally, the customer’s disaster (data) recovery process relied on SAN disk replication between its production and disaster recovery sites. In the event of a disaster the customer could recover a production system using a replicated copy of the SAN disks at the disaster recovery site. This also allowed the customer to perform regular disaster recovery tests. In these tests, the disaster recovery site is “fenced” from the real production network and SAN. The customer is then able to “bring up” a complete copy of any or all (replicated) production systems. Because these “copies” of production are fenced off from the real production environment the customer can undertake lengthy disaster recovery test programs without impacting the running production systems.

 

The customer had used this process for many, many years. They were happy with it and the flexibility it gave them. They wanted to continue using this process for the production SSP based systems.

 

The customer wanted the ability to recover a (storage) replicated copy of their entire shared storage pool at their disaster recovery (DR) site. We discussed the use of SSP failure group mirroring to cater for their DR requirements of their production AIX partitions. The following diagram shows our initial proposal of SSP mirroring across the production and DR sites.

 

 

 

However, this did not meet their needs. They needed a way to recover their AIX production partitions in isolation to their real production environment. SSP mirroring would not provide them with this flexibility. Essentially the customer could not start a “copy” of any or all production partitions at the DR site. In a SSP failure group setup, the SSP spans both the production and DR sites (as shown in the previous diagram). Meaning the SSP is in production across both sites. SSP mirroring would protect the customer in the event that the production site was to fail; they could start their production systems at the DR site, however they could not start isolated copies of the production systems in the same SSP. This meant SSP mirroring would break their current DR processes and provide them with reduced capability when compared to their traditional DR method.

 

Prior to SSP mirroring, one of the major drawbacks of SSPs was the lack of resilience at the storage layer. We were not able to mirror the disks in the pool from one storage subsystem to another (either local or remote). Having a single disk storage subsystem in a shared storage pool was a rather large single point of failure (SPOF). Starting with VIOS 2.2.3.0, we were given the capability to mirror the disks in a SSP from one storage system to another. All of the mirroring was configured and managed from the VIOS in the SSP cluster. The VIO client partitions are not aware that their disks are ‘mirror protected’ in the SSP. There are no changes required in the client partitions as the SSP mirroring feature is completely transparent to them. This feature enabled SSPs to be considered ready for production use.

 

Some other considerations that we discussed with the customer regarding SSP mirroring were the fact that the disks in the SSP failure group need to be shared across all the SSP nodes at both the production and DR sites. This is not a viable option in the case of long distances between sites (more than 100KM apart). SSP mirroring uses synchronous writes. This means that any write that happens at the production site has to be written synchronously on the failure group disks at the DR site first then production. This will introduce a delay in completing I/O which will have an impact on performance. Fortunately the customer’s two sites were far less than 100KM apart, so this consideration did not apply to them, but I mention it simply to make others aware of this point.

 

After much discussion with our SSP development team at IBM, we were able to provide the customer with an acceptable method of recovering their SSP environment at their DR site using a replica of the storage. What follows is a preview of this new recovery mechanism. This feature is not currently available as part of the standard SSP capability with VIOS 2.2.3.X. It is hoped that we will see this feature officially offered and supported at some stage in 2015. Of course, this is not an announcement and plans are always subject to change.

 

The following diagram provides a pictorial view of what the customer needed for DR with their SSP systems. As you can see in the diagram, there is a production and DR site. At the DR site there is a single SSP (with two VIOS in the cluster). Both the production and DR VIOS are in two separate shared storage pools.

 

 

 

 

 

The SSP development team provided a special ifix for the customer to install on their DR virtual I/O servers (VIOS). They also provided a script and procedure for the customer to follow. We successfully tested this procedure at the customers DR site. This was a huge success from the customer’s perspective as they were able to recover their entire production SSP in a few minutes. They could then selectively start/stop clones of the production AIX partitions at their DR site, without impacting the “real” production partitions at the production site. They were also able to continue using their existing storage replication method (as they had always done for all of their systems) in a DR situation. Not having to dramatically change their existing DR process and procedures was very important to the customer.

 

I’ll walk through the procedure now and share some insights (from my lab tests) as we go. It goes without saying, that before we started, the customer ensured that all the production SSP disks (LUNs) were being replicated to their DR site using their standard storage replication facility. The SSP repository disk was not replicated to the DR site. A new disk was used as the repos disk at the DR site.

 

The process (described in detail below) consists of a) taking a backup of the production SSP cluster configuration, b) running a “customise” script to modify the saved backup file to replace the production VIOS and hdisk names with the DR VIOS and hdisk names, c) restoring the modified VIOS backup on the DR VIOS at the DR site and finally d) verifying that the cluster has been recovered using the replicated storage at DR.

 

--

1. The first step was to take a backup of the production SSP VIOS cluster using the viosbr command. At the primary (production) site we used viosbr to create a backup of the cluster configuration. This generated a backup file named /home/padmin/cfgbackups/sspbackup.mycluster.tar.gz.

 

$ viosbr -backup -clustername mycluster -file sspbackup

 

2. The viosbr backup file was then transferred to one of the DR VIOS. On the selected DR VIOS we copy the backup file (sspbackup.mycluster.tar.gz) to the /home/padmin/backups directory.

 

3. We start an oem_setup_env shell and verify the file is intact.

 

$ oem_setup_env

# cd /home/padmin/backups

#ls

sspbackup.mycluster.tar.gz

 

We create a new file called nodelist in the backups directory. This file contains a list of the DR VIOS hostnames. These hostnames are used to create the replicated cluster at the remote site.

 

sspnode1.abc.com
sspnode2.abc.com
 

4. Also in the backups directory, we create another new file called disklist. This file contains a list of each of the replicated SSP disks with their unique_ids.  These disks are the replicated copies of the SSP pool disks from the primary site. All these disks must be accessible on all DR VIOS nodes.

 

$ pv -list -fmt , | awk -F, '{print $7}'

3E213600A0B80006E24E60000AB8F4F570A440F1818      FAStT03IBMfcp
3E213600A0B80006E250600001CD94F570E730F1818      FAStT03IBMfcp
 

5. The next step is to copy the necessary ifix file to the backups directory on the selected DR VIOS. We untar this file, which extracts two files. One file is the customize script and the other is the actual ifix we need to install on both VIOS.

  

./customize_ssp_backup
./ifix/sspDR_1.140627.epkg.Z
 

6. We install the ifix on all the DR VIOS which are going to be part of SSP setup.

 

$ updateios -install -dev /home/padmin/backups/ifix -accept

 

7. We run the script to customize the backup file for new setup at the DR site.

 

# ./customize_ssp_backup

 

Usage: customize_ssp_backup <backup_filename> <filename_with_hostnames> <filename_with_PoolDisks>

 

# ./customize_ssp_backup sspbackup.mycluster.tar.gz nodelist disklist

New configuration file created successfully: sspbackup_NEW.mycluster.tar.gz

 

8. We locate a free disk to be used as the new cluster repository disk on the selected DR VIOS.

 

$ lspv –free

 

9. We restore the cluster configuration using viosbr, specifying the customized viosbr backup file. We check the exit status is zero to confirm the restore process completed successfully.

 

$ viosbr -restore -clustername mycluster -repopvs hdisk5 -file /home/padmin/backups/sspbackup_NEW.mycluster.tar.gz -type cluster_remote

 

Backedup Devices that are unable to restore/change

==================================================

 

 

DEPLOYED or CHANGED devices:

============================

Dev name during BACKUP                  Dev name after RESTORE

----------------------                  ----------------------

$ echo $?

0

 

We can now verify the new cluster has been configured and recovered correctly. We use the cluster and lu commands to check the cluster is up and that the logical units are available in the shared storage pool.

 

$ cluster -status -clustername mycluster

$ lu –list

 

The SSP is now ready for use at the DR site.

--

 

Before we tested this procedure in the customer’s environment, I tried it in my lab first. My lab had the following configuration:

 

• 1 x POWER7 750 with a single VIOS and a Shared Storage Pool. I considered this my “production” SSP, which I wanted to recover on another POWER system in my lab.
• 1 x POWER7 PS701 Blade with a single VIOS. No SSP. I treated this system as my “DR” environment on which I wanted to recover a complete copy of my “production” SSP.

 

My “production” SSP was configured with V7000 storage.  I asked my storage administrator to create a copy (via flashcopy) of the nominated SSP disks from my 750 VIOS cluster. He then presented these new (flashcopied) LUNs to the “DR” VIOS on the PS701.

 

I then ran through the recovery steps (as outlined above). I started by taking a backup of the VIOS on the 750 using the viosbr command.

 

 

750vio1 (“production”)

-------------------------------

 

$ viosbr -backup -clustername VIOCLUSTER -file sspbackup

Backup of this node (750vio1) successful

 

I discovered that a SSP was already configured on my blade VIOS. I would need to delete the SSP cluster before I could recover the “production” cluster backup on this system. I obtained a list of the current disks in the SSP.

 

ps701-vios (“DR”)

-----------------------

 

$ pv -list

 

POOL_NAME: VIOSSP

TIER_NAME: SYSTEM

FG_NAME: Default

PV_NAME          SIZE(MB)    STATE            UDID

hdisk2           51200       ONLINE           3321360050768019E027CC000000000000~

hdisk3           51200       ONLINE           3321360050768019E027CC000000000000~

 

I also took note of the existing repository disk.

 

$ lspv | grep caa

hdisk1           000a366a031ffa37                     caavg_private    active

 

I removed any existing client VSCSI mappings for SSP disk so that I could delete the logical units from the SSP and delete the cluster.

 

$ lu -remove -lu sspdisk1_lpar1

Logical unit sspdisk1_lpar1 with udid "27cf8eb804710aa5d5995c414b2fb329" is removed.

 

$ lu -list

$

 

Then I deleted the cluster and took note of the existing disks again.

 

$ cluster -delete -clustername VIOCLUSTER

Cluster VIOCLUSTER has been removed successfully.

 

$

$ lspv

NAME             PVID                                 VG               STATUS

hdisk0           000a366a68640e58                     rootvg           active

hdisk1           000a366a031ffa37                     None

hdisk2           none                                 None

hdisk3           none                                 None

hdisk4           none                                 None

hdisk5           none                                 None

hdisk6           none                                 None

hdisk7           none                                 None

hdisk8           none                                 None

hdisk9           none                                 None

hdisk10          none                                 None

hdisk11          000a366a6b0d0d56                     None

hdisk12          000a366a6b0d0d56                     None

hdisk13          000a366a6b0d0d56                     None

$

 

I also chose to “scrub” the existing repository disk using the cleandisk command. This would allow me to re-use this disk for my SSP recovery procedure.

 

$ cleandisk -r hdisk1

0967-112 Subsystem not configured.

rmcluster: This operation will scrub hdisk1, removing any volume groups and clearing cluster identifiers.

If another cluster is using this disk, that cluster will be destroyed.

Are you sure? (y/[n]) y

WARNING: Force continue.

ERROR: import caavg_private failed.

remove_cluster_repository: Force continue.

ERROR: Cannot varyonvg caavg_private. It does not exist.

ERROR: rc=1, remove caavg_private failed.

remove_cluster_repository: Force continue.

rmcluster: Successfully removed hdisk1.

 

I entered the oem_setup_env shell and ran cfgmgr to bring in the new (flashed) disks on the “DR” VIOS. Two new disks were configured. I would use these disks with the SSP recovery script and disklist file.

 

$ oem_setup_env

# cfgmgr

#

 

# lspv

hdisk0          000a366a68640e58                    rootvg          active

hdisk1          000a366a031ffa37                    None

hdisk2          none                                None

hdisk3          none                                None

hdisk4          none                                None

hdisk5          none                                None

hdisk6          none                                None

hdisk7          none                                None

hdisk8          none                                None

hdisk9          none                                None

hdisk10         none                                None

hdisk11         000a366a6b0d0d56                    None

hdisk12         000a366a6b0d0d56                    None

hdisk13         000a366a6b0d0d56                    None

hdisk14         none                                None      <<<<<<<<

hdisk15         00f603cde9a7b15a                    None      <<<<<<<<

#

 

I took note of the hdisks and the unique_ids and placed this information in a new file named disklist.

 

$ oem_setup_env

# odmget -qname=hdisk14 CuAt |grep -p unique_id |grep value |cut -d= -f2 | sed -e

's/"//g' | sed -e 's/ //g'

hdisk14 3321360050768019E027CC00000000000009D04214503IBMfcp

# odmget -qname=hdisk15 CuAt |grep -p unique_id |grep value |cut -d= -f2 | sed -e

's/"//g' | sed -e 's/ //g'

hdisk15 3321360050768019E027CC00000000000009E04214503IBMfcp

 

# cd backups

# cat disklist

3321360050768019E027CC00000000000009D04214503IBMfcp

3321360050768019E027CC00000000000009E04214503IBMfcp

#

 

I created the nodelist file, also in the backups directory.

 

# pwd

/home/padmin/backups

# cat nodelist

ps701-vios

 

I ensured that the necessary ifix was installed on my “DR” blade VIOS partition.

 

# emgr -l

 

ID  STATE LABEL      INSTALL TIME      UPDATED BY ABSTRACT

=== ===== ========== ================= ========== ======================================

1    S    sspDR_1    07/07/14 10:38:10            Fix to restore cluster on remote site

 

Then I ran the customise script to prepare the viosbr backup file for restoration on this “DR” VIOS.

 

# ./customize_ssp_backup sspbackup.tar.gz nodelist disklist

backupfile=sspbackup.tar.gz, hostfile=/home/padmin/backups/nodelist, pvfile=/home/padmin/backups/disklist

MTM=8406-71Y0310A366A, ID=1, nodefile=VIOCLUSTERMTM8406-71Y0310A366AP1.xml, clust=VIOCLUSTER, localnode=VIOCLUSTERMTM8406-71Y0310A366AP1.xml, newbackup=sspbackup_NEW.VIOCLUSTER.tar

cp: 0653-435  VIOCLUSTERMTM8406-71Y0310A366AP1.xml and VIOCLUSTERMTM8406-71Y0310A366AP1.xml are identical (not copied).

VIOCLUSTERDB/9fa498e4896611e38d72f603cdcd9c55/VIOCLUST.xml

/home/padmin/backups/temp/VIOCLUSTERDB/9fa498e4896611e38d72f603cdcd9c55/VIOCLUST.xml

/home/padmin/backups/temp/VIOCLUSTERDB/9fa498e4896611e38d72f603cdcd9c55/VIOCLUST.xml

CLUSTER CONF FILE=/home/padmin/backups/temp/VIOCLUSTERDB/9fa498e4896611e38d72f603cdcd9c55/VIOCLUST.xml

CLUSTER CONF FILE NEW=/home/padmin/backups/temp/VIOCLUSTERDB/9fa498e4896611e38d72f603cdcd9c55/VIOCLUST2.xml

New configuration file created successfully: sspbackup_NEW.VIOCLUSTER.tar.gz

 

#

 

I then restored the SSP from the customised viosbr backup file. I specified hdisk1 as the repos disk. The restore was successful (with an exit status of zero).

 

$ viosbr -restore -clustername VIOCLUSTER -repopvs hdisk1 -file /home/padmin/backups/sspbackup_NEW.VIOCLUSTER.tar.gz -type cluster_remote

"CLUSTER restore successful.

 

"Backedup Devices that are unable to restore/change

==================================================

 

 

DEPLOYED or CHANGED devices:

============================

Dev name during BACKUP                  Dev name after RESTORE

----------------------                  ----------------------

$ echo $?

0

$

 

After the successful restore, I was able to display the cluster status and list the logical units in the recovered SSP.

 

$ cluster -list

CLUSTER_NAME:    VIOCLUSTER

CLUSTER_ID:      9fa498e4896611e38d72f603cdcd9c55

 

$ lu -list

POOL_NAME: VIOSSP

TIER_NAME: SYSTEM

LU_NAME                 SIZE(MB)    UNUSED(MB)  UDID

sspdisk1_750lpar11      10240       7251        4f42b83c0ed7e826a4784b204b1c81ea

SNAPSHOTS

sspdisk1_750lpar11_snap1

 

$ cluster -status -clustername VIOCLUSTER

Cluster Name         State

VIOCLUSTER           OK

 

    Node Name        MTM           Partition Num  State  Pool State

    ps701-vios       8406-71Y0310A366A         1  OK     OK

$

 

As expected, both of the “flashed” disks were now assigned to the recovered disk pool.

 

$ pv -list

 

POOL_NAME: VIOSSP

TIER_NAME: SYSTEM

FG_NAME: Default

PV_NAME          SIZE(MB)    STATE            UDID

hdisk15          51200       ONLINE           3321360050768019E027CC000000000000~

hdisk14          51200       ONLINE           3321360050768019E027CC000000000000~

 

The existing hdisk1 had been successfully re-used for the repository disk of the “new” cluster.

 

$ lspv | grep caa

hdisk1           000a366a031ffa37                     caavg_private    active

$

 

The VIOS error report on the “DR” VIOS also confirmed that the SSP cluster was recovered and the services started correctly.

 

$ errlog

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION

228C1EBE   0707110114 I S POOL           Cluster state change has occurred

228C1EBE   0707110014 I S POOL           Cluster state change has occurred

EDFF8E9B   0707110014 I O StorageRM      IBM.StorageRM daemon has started.

3B16518D   0707110014 I S ConfigRM       The node is online in the domain indicat

4BDDFBCC   0707110014 I S ConfigRM       The operational quorum state of the acti

AFA89905   0707110014 I O cthags         Group Services daemon started

$

 

At this point I was able to map the LU to my VIO client partition and boot it on the PS701. AIX started without an issue and I now had a complete clone of my existing AIX partition (from the 750) running on my POWER blade.

 

$ lu –map –lu sspdisk1_750lpar11 –vadapter vhost4

Assigning logical unit 'sspdisk1_750lpar11' as a backing device.

 

VTD:vtscsi4

 

$ lsmap -vadapter vhost4

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost4          U8406.71Y.10A366A-V1-C19                     0x00000006

 

VTD                   vtopt1

Status                Available

LUN                   0x8200000000000000

Backing device

Physloc

Mirrored              N/A

 

VTD                   vtscsi4

Status                Available

LUN                   0x8100000000000000

Backing device        sspdisk1_750lpar11.4f42b83c0ed7e826a4784b204b1c81ea

Physloc

Mirrored              N/A

 

Shared storage pools already provide some fantastic capabilities that are attractive to most POWER customers. Features such as SSP mirroring, snapshots and linked-clones provide some very powerful and flexible configurations that will accelerate virtual machine deployment and management on IBM Power Systems.

 

I believe the enhancement I’ve discussed here will prove very popular with many of our enterprise AIX and POWER customers. It will provide one more compelling reason for customers to consider shared storage pools in their environments, particularly for disaster recovery situations. There may even be the capability (in the future) to completely automate the entire procedure that I’ve outlined in this article. I hope that I can provide further updates on this capability in early 2015.

 

Reference material:

 

Mirroring a shared storage pool

https://www-304.ibm.com/support/knowledgecenter/POWER7/p7hb1/iphb1_mirroring_storage_pool.htm?cp=POWER7%2F1-8-3-5-7-1-2-3

 

I came across this strange LPM issue recently. Thought I’d share it with you.

 

All the customers VIOS were configured with the viosecure level set to high. 

When the VIOS is configured with a security profile set to high, PCI or DoD, a new feature is enabled during LPM. This new feature is called “Secure LPM”. As a result when you initiate an LPM operation, the “Secure LPM” feature automatically enables the VIOS firewall and configures a secure (ipsec) tunnel for all LPM traffic over the network.

Before we started LPM, the VIOS firewall is off:

 

$ viosecure -firewall view

IPv4 Firewall OFF

 

                          ALLOWED   PORTS

           Local   Remote

Interface  Port    Port    Service      IPAddress       Expiration

Time(seconds)

---------  ----    ----    -------      ---------       ---------------

 

And it's activated after we start LPM:

 

$ viosecure -firewall view

IPv4 Firewall ON

 

                          ALLOWED   PORTS

           Local   Remote

Interface  Port    Port    Service      IPAddress       Expiration

Time(seconds)

---------  ----    ----    -------      ---------       ---------------

In the customer’s case, we moved a partition from a 795 running a pair of VIOS at 2.2.1.3 to another 795 running another pair of VIOS at 2.2.2.1. This worked OK. When we attempted to move the partition back, we received an error stating that the source MSP had rejected the request.

HSCLA230 

The mover service partition on the source managed system has rejected the request to stop the migration. Verify that the migration state of the partition is Migration Starting, and try the operation again.

After a while we discovered that when the LPM operation started, the source VIOS would enable its firewall and we would immediately loose connectivity (i.e. our SSH session to the VIOS would hang). We also found that RMC connectivity between the source VIOS and HMC also dropped. Somehow the VIOS firewall was blocking network connectivity to pretty much everything. 

Eventually, after much digging we found that the source VIOS was the victim of an errant firewall rule (configured on the source VIOS itself). 

# smitty ips4_conf_filter

List IP Security Filter Rules

1 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 no udp eq 4001 eq 4001 both both no all packets 0 all 0 none  Default Rule

2 *** Dynamic filter placement rule for IKE tunnels *** no

3 deny 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 y all any 0 any 0 all both inbound no yes 0 no 0 patt_none

0 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 yes all any 0 any 0 both both no all packets 0 all 0 none  Default Rule

We removed this rule (after some discussion with IBM support).

# smitty ips4_conf_filter

Delete IP Security Filter Rules

3 deny 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 y all any 0 any 0 all both inbound no yes 0 no 0 patt_none

# smitty ips4_conf_filter

List IP Security Filter Rules

1 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 no udp eq 4001 eq 4001 both both no all packets 0 all 0 none  Default Rule

2 *** Dynamic filter placement rule for IKE tunnels *** no

0 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 yes all any 0 any 0 both both no all packets 0 all 0 none  Default Rule

We were then able to move the partition back to the original frame. We are still searching for an answer to how this happened. The IBM support team believe the rule must have been created by an administrator at some point. Of course, the administrator claims that is untrue.

The first time I ran the ’viostat  –adapter’ command I expected to find non-zero values for kbps, tps, etc, for each vfchost adapter. However, the values were always zero, no matter how much traffic traversed the adapters.                                 

                                                                       

$ viostat -adapter 1 10                                                 

...                                                                     

vadapter:                  Kbps      tps     bkread    bkwrtn          

vfchost0                    0.0      0.0        0.0       0.0          

...                                                                    

vadapter:                  Kbps      tps     bkread    bkwrtn          

vfchost1                    0.0      0.0        0.0       0.0          

                                                                       

I wondered if this was expected behaviour. Was the output supposed to report the amount of pass-thru traffic per vfchost? In 2011, I posed this question on the IBM developerWorks PowerVM forum regarding this observation. One of the replies stated:

 

"viostat does not give statistics for NPIV devices. The vfchost adapter is just a passthru, it doesn't know what the commands it gets are."    

                                                                        

https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14697753&#14697753

 

I appreciated someone taking the time to answer my question but I was still curious. I tested the same command again (in 2013) on a recent VIOS level (2.2.2.1), but I received the same result. It was time to source an official answer on this behaviour.

 

Here is the official response I received from IBM:

 

1.  FC adapter stats in viostat/iostat do not include NPIV.           

                                                                       

2.  viostat & iostat are an aggregate of all the stats from the underlying disks, which of course NPIV doesn't have.

 

There's really no way for the vfchost adapter to monitor I/O, since it doesn't know what the commands it gets are. He's just a passthru, passing the commands he gets from the client directly to the physical FC adapter.                         

                                                                        

3.  You can run fcstat on the VIOS but that has the same issues/limitations mentioned above.

 

Intent here was that customers would use tools on the client to monitor this sort of thing.

 

To summarize the comments from Development:                            

                                                                       

viostat does NOT give statistics for NPIV devices.”

 

This made sense but I wondered why the tool hadn’t been changed to exclude vfchost adapters from the output (to avoid customer confusion). There's obviously no valid reason to ever display any information for this type of adapter. I also understood that it was expected that I/O would be monitored at the client LPAR level. But I must say that an option for monitoring VFC I/O from a VIO server would be advantageous i.e. a single source view of all I/O activity for all VFC clients; particularly when there are several hundred partitions on a frame.  The response was:

 

“…the way the vfchost driver currently works is that it calls iostadd to register a dkstat structure, resulting in the adapter being listed when viostat is called.  This is misleading, however, since the vfchost driver does not actually track I/O.  The commands coming from the client partition are simply passed as-is to the physical FC adapter, and we don't know if a particular command is an I/O command or not.  The iostadd call is left over from porting the code from the vscsi driver, and Development agrees it should probably have been removed before shipping the code.                            

                                                                       

There has also been mention of a DCR #MR0413117456 (Title: FC adapter stats in viostat/iostat does not include NPIV) which you can follow-up with Marketing to register your interest/track progress if that is something you're interested in pursuing.”

I updated my lab VIOS to the latest fix pack (V2.2.2.1) this week and thought I’d try the new VIOS part command. This new command is an improved version of the existing vios_advisor tool. The major difference between the two is the fact that the new tool is now included with VIOS code and will be updated via new VIOS fix packs. The following link has some information on using the command:

 

http://pic.dhe.ibm.com/infocenter/powersys/v3r1m5/index.jsp?topic=/p7hcg/part.htm

 

This new tool “Provides performance reports with suggestions for making configurational changes to the environment, and helps to identify areas for further investigation. The reports are based on the key performance metrics of various partition resources that are collected from the Virtual I/O Server (VIOS) environment.” Just like the old VIOS advisor.

 

I ran the tool for 10 minutes on my idle VIOS, just to see how the new XML report looked.

 

$ whoami

padmin

 

$ ioslevel

2.2.2.1

 

$ part -?

usage: part {-i INTERVAL | -f FILENAME} [-t LEVEL] [--help|-?]

       -i <minutes>    interval can range between 10-60

       -f <file>       any nmon recording

       -t <level>      1 - Basic logging, 2 - Detailed logging

       -?              usage message

 

 

$ part -i 10

part: Reports are successfully generated in vioslab2_130118_11_26_49.tar

 

When the command finished, I found the following tar file in padmin’s home directory. The tar file contained a number of files but the most interesting files were the XML and nmon files.

 

$ ls -ltr

total 1352

drwxrwxr--    2 padmin   staff           256 Aug 07 2010  config

drwxr-xr-x    3 padmin   staff           256 Aug 11 2010  tivoli

drwxr-xr-x    2 538804364 staff           256 Oct 08 16:29 mnt

-rwxr-xr-x    1 padmin   staff          3141 Jan 16 15:05 ck_sum.bff

drwxr-x---    2 padmin   staff         32768 Jan 16 15:10 update

-rw-r--r--    1 root     staff        409139 Jan 16 15:56 install.log

-rw-r--r--    1 root     staff          3828 Jan 18 11:26 ioscli.log

-rw-r--r--    1 padmin   staff        204800 Jan 18 11:37 vioslab2_130118_11_26_49.tar

 

$ tar -tvf vioslab2_130118_11_26_49.tar

drwxr-xr-x 10  1        0 Jan 18 11:37:07 2013 ./vioslab2_130118_11_26_49/

drwxr-xr-x 10  1        0 Jan 18 11:37:07 2013 ./vioslab2_130118_11_26_49/images/

-r--r--r-- 10  1      902 Jan 18 11:37:07 2013 ./vioslab2_130118_11_26_49/images/Warning_icon.png

-r--r--r-- 10  1      125 Jan 18 11:37:07 2013 ./vioslab2_130118_11_26_49/images/bg.png

-r--r--r-- 10  1      614 Jan 18 11:37:07 2013 ./vioslab2_130118_11_26_49/images/close.jpg

-r--r--r-- 10  1     1336 Jan 18 11:37:07 2013 ./vioslab2_130118_11_26_49/images/correct.png

-r--r--r-- 10  1      316 Jan 18 11:37:07 2013 ./vioslab2_130118_11_26_49/images/headerLogo.png

-r--r--r-- 10  1     1241 Jan 18 11:37:07 2013 ./vioslab2_130118_11_26_49/images/investigate.png

-r--r--r-- 10  1      853 Jan 18 11:37:07 2013 ./vioslab2_130118_11_26_49/images/readonly.png

-r--r--r-- 10  1     1019 Jan 18 11:37:07 2013 ./vioslab2_130118_11_26_49/images/red-error.png

-r--r--r-- 10  1     5692 Jan 18 11:37:07 2013 ./vioslab2_130118_11_26_49/popup.js

-r--r--r-- 10  1     2869 Jan 18 11:37:07 2013 ./vioslab2_130118_11_26_49/style.css

-r--r--r-- 10  1    16869 Jan 18 11:37:07 2013 ./vioslab2_130118_11_26_49/vios_advisor.xsl

-rw-r--r-- 10  1    21158 Jan 18 11:37:07 2013 ./vioslab2_130118_11_26_49/vios_advisor_report.xml

-rw-r--r-- 10  1   136969 Jan 18 11:36:57 2013 ./vioslab2_130118_11_26_49/vioslab2_130118_1126.nmon

 

I then scp’ed the tar file to my laptop, extracted it and opened the vios_advisor_report.xml file. This is what the report looked like:

 

 

 

 

 

 

I was also able to open the nmon file using the nmon analyser tool. It produced typical nmon performance graphs as you’d expect.

 

 

So, not only does the new part tool run the VIOS advisor it also captures nmon performance data at the same time.

 

This is rather impressive and a great move by IBM. The original VIOS advisor tool was free and of course not officially supported by IBM (although the development team were very responsive to requests from users of the tool). The new tool is fully supported by the IBM team and as a result will only get better and better as time goes by. I’m not a fan of the new command name, part (good luck trying to google it!), I still prefer vios_advisor but hey, what’s in a name, right?

 

I look forward to seeing what comes next!

 

 

 http://pic.dhe.ibm.com/infocenter/powersys/v3r1m5/index.jsp?topic=/p7hb1/iphb1_vios_perf_adv.htm

 

 

 

It’s possible to search through seastat command output on the VIOS using the –s flag. You specify the following attributes with this flag: searchtype=value.

Valid search values include:

 

mac, priority, vlan, ip, host,

greaterbytessent (gbs), greaterbytesrecv (gbr),

greaterpktssent (gps), greaterpktsrecv (gpr),

smallerbytessent (sbs), smallerbytesrecv (sbr),

smallerpktssent (sps), smallerpktsrecv (spr)

 

For example, to search for a particular host:

 

$ seastat -d ent19 -s host=aixvioc1

 

================================================================================

 

Advanced Statistics for SEA

Device Name: ent19

 

================================================================================

MAC: 42:BA:1D:00:B9:0C

----------------------

 

VLAN: None

VLAN Priority: None

Hostname: aixvioc1.xyz.com

IP: 10.1.1.7

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 11630504                      Packets: 16040250

Bytes: 12106445229                     Bytes: 3629657696

 

================================================================================

 

To search for a particular IP address:

 

$ seastat -d ent19 -n -s ip=10.1.1.9

 

================================================================================

 

Advanced Statistics for SEA

Device Name: ent19

 

================================================================================

MAC: 2A:8C:8B:7C:D0:0C

----------------------

 

VLAN: None

VLAN Priority: None

IP: 10.1.1.9

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 30517                         Packets: 14900

Bytes: 36280372                        Bytes: 1751127

 

================================================================================

 

Until recently, if you were configuring a new LPAR with virtual FC adapters you couldn’t force it to log into the SAN before an operating system (such as AIX) was installed. I’ve written about this before (see link below). I also offered a way to work around this issue.

 

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/entry/new_hmc_commands_chnportlogin_and_lsnportlogin_login_in_virtual_fibre_channel_adapters_for_npiv_client_lpars1?lang=en

 

As mentioned in my post, there are new HMC commands to force a new LPAR to connect to the SAN, so that SAN administrators can zone storage to the new WWPN(s).

 

There’s also another method available in the Power Systems SMS menu.

 

In recent levels of POWER systems firmware, there is a new option in the SMS menu called SAN Zoning Support. You’ll find it mentioned here:

 

http://pic.dhe.ibm.com/infocenter/powersys/v3r1m5/index.jsp?topic=/p7hb6l/usingsmsmenus_sanzoning.htm

 

I’ve successfully used this method on both POWER6 (595) and POWER7 (795) systems. After configuring a new LPAR profile with a single VFC adapter, the VIOS reported that the client was not logged into the SAN:

 

$ lsmap -vadapter vfchost41 -npiv

Name          Physloc                            ClntID ClntName       ClntOS

------------- ---------------------------------- ------ -------------- -------

vfchost41     U9119.FHA.99D99C1-V2-C199              99

 

Status:NOT_LOGGED_IN

FC name:fcs4                    FC loc code:U5803.001.9912345-P2-C5-T1

Ports logged in:0

Flags:4<NOT_LOGGED>

VFC client name:                VFC client DRC:

 

I forced this event by booting into SMS and selecting 5 (Boot Options).

 

PowerPC Firmware

 Version EH350_120

 SMS 1.7 (c) Copyright IBM Corp. 2000,2008 All rights reserved.

-------------------------------------------------------------------------------

 Main Menu

 1.   Select Language

 2.   Setup Remote IPL (Initial Program Load)

 3.   Change SCSI Settings

 4.   Select Console

 5.   Select Boot Options

 

 

 -------------------------------------------------------------------------------

 Navigation Keys:

 

                                             X = eXit System Management Services

 -------------------------------------------------------------------------------

 Type menu item number and press Enter or select Navigation key:5

 

Then I selected 4 (SAN Zoning Support).

 

 PowerPC Firmware

 Version EH350_120

 SMS 1.7 (c) Copyright IBM Corp. 2000,2008 All rights reserved.

-------------------------------------------------------------------------------

 Multiboot

 1.   Select Install/Boot Device

 2.   Configure Boot Device Order

 3.   Multiboot Startup <OFF>

 4.   SAN Zoning Support

 

 

 

 

 -------------------------------------------------------------------------------

 Navigation keys:

 M = return to Main Menu

 ESC key = return to previous screen         X = eXit System Management Services

 -------------------------------------------------------------------------------

 Type menu item number and press Enter or select Navigation key:4

 

Next I choose the appropriate VFC adapter I’d like to connect on the SAN at this time.

 

 PowerPC Firmware

 Version EH350_120

 SMS 1.7 (c) Copyright IBM Corp. 2000,2008 All rights reserved.

-------------------------------------------------------------------------------

 Select Media Adapter

 1.  U9119.FHA.83D99C3-V99-C20-T1    /vdevice/vfc-client@30000014

 

 

 -------------------------------------------------------------------------------

 Navigation keys:

 M = return to Main Menu

 ESC key = return to previous screen         X = eXit System Management Services

 -------------------------------------------------------------------------------

 Type menu item number and press Enter or select Navigation key:1

 

 

A message appears stating that it may now be possible to perform SAN zoning to the adapters WWPN.

 

   .-----------------------------------------------------------------------.

   |  The selected adapter has been opened.                                |

   |  Zoning of attached disks may now be possible.                        |

   |  Press any key to close the adapter and return to the previous menu.  |

   `-----------------------------------------------------------------------'

 

On the VIOS I observed the client vfchost adapter status changed to LOGGED_IN.

 

$ lsmap -vadapter vfchost41 -npiv

Name          Physloc                            ClntID ClntName       ClntOS

------------- ---------------------------------- ------ -------------- -------

vfchost41     U9119.FHA.99D99C1-V2-C199              99 aixlpar1     FIRMWARE

 

Status:LOGGED_IN

FC name:fcs4                    FC loc code:U5803.001.9912345-P2-C5-T1

Ports logged in:1

Flags:a<LOGGED_IN,STRIP_MERGE>

VFC client name:Virtual Fibre Channel - Firmware driverVFC client DRC:U9119.FHA.83D99C3-V99-C20

 

It appears that the new SMS option is simply an interface to the Open Firmware Prompt and the IOINFO command. This was how we performed the task without the SMS menu in front:

 

https://www.ibm.com/developerworks/wikis/display/virtualization/vfclogin

 

So far I’ve found this new SMS option on POWER7 795 systems running firmware level AH730_078 and POWER6 595 systems running EH350_120.

Make sure you are monitoring file system space on your VIOS.

 

Why?

 

If you run out of space in the root file system, odd things can happen when you try to map virtual devices to virtual adapters with mkvdev.

 

For example, a colleague of mine was attempting to map a new hdisk to a vhost adapter on a pair of VIOS. The VIOS was running a recent version of code. He received the following error message (see below). It wasn’t a very helpful message. At first I thought it was due to the fact that he had not set the reserve_policy attribute for the new disk to no_reserve on both VIOS. Changing the value for that attribute did not help.

 

$ ioslevel

2.2.1.3

 

$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1

*******************************************************************************

The command's response was not recognized.  This may or may not indicate a problem.

*******************************************************************************

 

I enabled debug output from the padmin command line and discovered that the mkdev command was failing for some reason.

 

$ export CLI_DEBUG=33

 

$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1

AIX: "lspv -l hdisk1 2>&1 | grep 0516-320"

AIX: "mkdev  -V hdisk1  -p vhost0  -l vhdisk1 "

*******************************************************************************

The command's response was not recognized.  This may or may not indicate a problem.

*******************************************************************************

 

I entered oem_setup_env and ran the mkdev command manually. I was presented with a very clear error message describing the problem. The root file system (/) was full!

 

$ oem_setup_env

 

# mkdev  -V hdisk1  -p vhost0  -l vhdisk1

mkdev: 0514-548 Cannot perform the requested function because

                the / filesystem is full or is out of inodes.

 

# df -g /

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4           0.50      0.00  100%     7728    75% /

 

I found a 45MB core file (generated by cimserver) in the root file system. I moved the file to another file system (with plenty of free space) for further investigation.

 

# mv core /tmp/core.cg

# df -m .

Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4         512.00     45.93   92%     7727    38% /

 

Then I ran the mkvdev command successfully.

 

$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1

AIX: "lspv -l hdisk1 2>&1 | grep 0516-320"

AIX: "mkdev  -V hdisk1  -p vhost0  -l vhdisk1 "

vhdisk1 Available

AIX: "lsattr -El hdisk1 -a max_transfer |cut -d" " -f2"

 

$ lsmap -all

AIX: "lsdev -c adapter -t IBM,v-scsi-host -s vdevice -F "name" | wc -l -c"

AIX: "lsdev -c adapter -t IBM,v-scsi-host -s vdevice -F "name""

AIX: "lsdev -C -l vhost0 -F "physloc""

AIX: "lsdev -p vhost0 -F "name" | wc -l -c"

AIX: "lsdev -p vhost0 -F "name""

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost0          U9119.FHA.87654B1-V6-C21                   0x0000002e

 

VTD                   vhdisk1

Status                Available

LUN                   0x8100000000000000

Backing device        hdisk1

Physloc               U5873.001.9SS005Y-P1-C9-T1-W50060E8006D0206A-L1000000000000

Mirrored              false

 

AIX: "lsdev -C -l vhost1 -F "physloc""

AIX: "lsdev -p vhost1 -F "name" | wc -l -c"

AIX: "lsdev -p vhost1 -F "name""

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost1          U9119.FHA.87654B1-V6-C22                   0x0000002g

 

VTD                   NO VIRTUAL TARGET DEVICE FOUND

 

I found the same issue on the second VIOS i.e. a full root file system due to a core file (from cimserver). I also found no trace of a full file system event in the error report. Perhaps someone had taken it upon themselves to “clean house” at some point and had removed entries from the VIOS error log.

 

Make sure you monitor file system space on your VIOS. Who knows what else might fail if you run out of space in a critical file system.

When attempting to change the max_transfer attribute on a VSCSI hdisk, the following error was displayed:

 

# chdev -l hdisk0 -a max_transfer=0x100000 -P

Method error (/etc/methods/chgdisk):

        0514-018 The values specified for the following attributes

                 are not valid:

     max_transfer     Maximum TRANSFER Size

 

The desired value, 0x100000, was an allowed value for this type of device (as shown in the lsattr output below):

 

# lsattr -El hdisk0

PCM             PCM/friend/vscsi                 Path Control Module        False

algorithm       fail_over                        Algorithm                  True

hcheck_cmd      test_unit_rdy                    Health Check Command       True

hcheck_interval 60                               Health Check Interval      True

hcheck_mode     nonactive                        Health Check Mode          True

max_transfer    0x40000                          Maximum TRANSFER Size      True

pvid            00f6a83aeb1948950000000000000000 Physical volume identifier False

queue_depth     256                              Queue DEPTH                True

reserve_policy  no_reserve                       Reserve Policy             True

 

# lsattr -Rl hdisk0 -a max_transfer

0x20000

0x40000

0x80000

0x100000

0x200000

0x400000

0x800000

0x1000000

 

After some discussion with IBM support, it was discovered that the virtual optical device, assigned to this LPAR, was causing the problem. They advised that we should perform the following steps:

 

1. Remove the cd0 device from the LPAR i.e.

   # rmdev –dl cd0

 

2. Remove the vtopt0 device assigned to the LPAR on the VIOS i.e

   $ rmvdev –vtd vtopt0

 

3. Shutdown the LPAR.

 

4. Reboot the VIOS.

 

5. Restart the LPAR.

 

6. Change the max_transfer value for the hdisk on the LPAR.

 

Sure enough, after following these steps, we were able to change max_transfer to the desired value!

 

# chdev -l hdisk0 -a max_transfer=0x100000

hdisk0 changed

 

# lsattr -El hdisk0

PCM             PCM/friend/vscsi                 Path Control Module        False

algorithm       fail_over                        Algorithm                  True

hcheck_cmd      test_unit_rdy                    Health Check Command       True

hcheck_interval 0                                Health Check Interval      True

hcheck_mode     nonactive                        Health Check Mode          True

max_transfer    0x100000                         Maximum TRANSFER Size      True

pvid            00f6a83aeedf5ffb0000000000000000 Physical volume identifier False

queue_depth     3                                Queue DEPTH                True

reserve_policy  no_reserve                       Reserve Policy             True

 

The following explanation was given by IBM support:

 

“The virtual optical device seems to use max_transfer size as 0x40000 (256 KB) internally, which is not allowed to be modified. Virtual optical CD's max_transfer seems to limit vhost adapter's max_transfer size. So if you want to increase max_transfer size for virtual SCSI disks, you should separate them from virtual optical SCSI CD by configuring those two groups under different vhost adapters”

I was working with a customer recently on a Power Blade that was running the Integrated Virtualisation Manager (IVM). They’d installed a VIO partition onto the Blade and had hoped to install a couple of AIX LPARs on the system. However they didn’t get very far.

 

As soon as they attempted to NIM install the LPARs, they would get stuck at trying to ping the NIM master from the client. Basically, the Shared Ethernet Adapter (SEA) was not working properly and none of the LPARs could communicate with the external network. So they asked for some assistance.

 

The Blade server name was Server-8406-71Y-SN06BF99Z. The SEA was configured as ent7.

 

# lsdev -Cc adapter

ent0    Available       Logical Host Ethernet Port (lp-hea)

ent1    Available       Logical Host Ethernet Port (lp-hea)

ent2    Available       Virtual I/O Ethernet Adapter (l-lan)

ent3    Available       Virtual I/O Ethernet Adapter (l-lan)

ent4    Available       Virtual I/O Ethernet Adapter (l-lan)

ent5    Available       Virtual I/O Ethernet Adapter (l-lan)

ent6    Available       EtherChannel / IEEE 802.3ad Link Aggregation

ent7    Available       Shared Ethernet Adapter

ent9    Available       Virtual I/O Ethernet Adapter (l-lan)

fcs0    Available 02-00 8Gb PCIe FC Blade Expansion Card (7710322577107601)

fcs1    Available 02-01 8Gb PCIe FC Blade Expansion Card (7710322577107601)

ibmvmc0 Available       Virtual Management Channel

lhea0   Available       Logical Host Ethernet Adapter (l-hea)

sissas0 Available 01-08 PCI-X266 Planar 3Gb SAS Adapter

usbhc0  Available 00-08 USB Host Controller (33103500)

usbhc1  Available 00-09 USB Host Controller (33103500)

usbhc2  Available 00-0a USB Enhanced Host Controller (3310e000)

vhost0  Available       Virtual SCSI Server Adapter

vsa0    Available       LPAR Virtual Serial Adapter

vts0    Available       Virtual TTY Server Adapter

 

The SEA was configured with Port-VLAN ID (PVID) of 68 without any VLAN tags. This was the root cause of the problem.

 

# lsattr -El ent7

accounting    enabled  Enable per-client accounting of network statistics                 True

ctl_chan               Control Channel adapter for SEA failover                           True

gvrp          no       Enable GARP VLAN Registration Protocol (GVRP)                      True

ha_mode       disabled High Availability Mode                                             True

jumbo_frames  no       Enable Gigabit Ethernet Jumbo Frames                               True

large_receive no       Enable receive TCP segment aggregation                             True

largesend     0        Enable Hardware Transmit TCP Resegmentation                        True

netaddr       0        Address to ping                                                    True

pvid          68       PVID to use for the SEA device                                     True

pvid_adapter  ent9     Default virtual adapter to use for non-VLAN-tagged packets         True

qos_mode      disabled N/A                                                                True

real_adapter  ent6     Physical adapter associated with the SEA                           True

thread        1        Thread mode enabled (1) or disabled (0)                            True

virt_adapters ent9     List of virtual adapters associated with the SEA (comma separated) True

 

$ entstat -all ent8 | grep -i vlan

VLAN Ids :

    VLAN Extract: False

    VLAN tagged filtering mode: Filter according to VLAN permit array

Max number of VLAN IDs per HEA port: 20

    VLAN Extract: False

    VLAN tagged filtering mode: Filter according to VLAN permit array

Max number of VLAN IDs per HEA port: 20

Invalid VLAN ID Packets: 5388

Port VLAN ID:    68

VLAN Tag IDs:    None

 

On the network switch port, the native VLAN (PVID), was configured as 11, with VLAN tag 68 added as an allowed VLAN. If the client LPARs tried to access the network using a PVID of 68, instead of a VLAN TAG of 68, they would get stuck at the switch port i.e. the un-tagged packets for 10.1.68.X via PVID 11 would fail. The packets for 10.1.68.X needed to be tagged with VLAN id 68 in order for the switch to pass the traffic.

 

So the question was, how do we add VLAN tags in the IVM environment? If we’d been using a HMC, then this would be simple to fix. Just add the VLAN tags into the Virtual Ethernet Adapter used by the SEA and we’d be done.

 

We had to use the lshwres and chhwres commands to resolve this one. First we listed the virtual adapters known to the VIO server (IVM). At slot 12, we found our SEA adapter with port_vlan_id set to 68 and addl_vlan_ids set to none.

 

$ lshwres -r virtualio --rsubtype eth --level lpar

lpar_name=06-BF99Z,lpar_id=1,slot_num=3,state=1,ieee_virtual_eth=0,port_vlan_id=1,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62803

lpar_name=06-BF99Z,lpar_id=1,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=2,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62804

lpar_name=06-BF99Z,lpar_id=1,slot_num=5,state=1,ieee_virtual_eth=0,port_vlan_id=3,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62805

lpar_name=06-BF99Z,lpar_id=1,slot_num=6,state=1,ieee_virtual_eth=0,port_vlan_id=4,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62806

lpar_name=06-BF99Z,lpar_id=1,slot_num=12,state=1,ieee_virtual_eth=0,port_vlan_id=68,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA6280C

lpar_name=aixlpar1,lpar_id=2,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=68,addl_vlan_ids=none,is_trunk=0,trunk_priority=0,is_required=0,mac_addr=F67D5345AD04

 

We needed to change port_vlan_id to 11 and addl_vlan_ids to 68. We also required the ieee_virtual_eth value set to 1.

 

First we removed the existing SEA adapter, as we would not be able to make changes to it while it was “active”. We then removed the adapter from slot 12 and then re-added it, again at slot 12, with port_vlan_id and addl_vlan_ids set to the desired values.

 

$ chhwres -m Server-8406-71Y-SN06BF99Z -p 06-BF99Z -r virtualio --rsubtype eth -s 12 -o r 

$ chhwres -m Server-8406-71Y-SN06BF99Z -p 06-BF99Z -r virtualio --rsubtype eth -s 12 -o a -a "ieee_virtual_eth=1,port_vlan_id=11,addl_vlan_ids=68,is_trunk=1,trunk_priority=1" -d 5

 

$ lshwres -r virtualio --rsubtype eth --level lpar

lpar_name=06-BF99Z,lpar_id=1,slot_num=3,state=1,ieee_virtual_eth=0,port_vlan_id=1,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62803

lpar_name=06-BF99Z,lpar_id=1,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=2,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62804

lpar_name=06-BF99Z,lpar_id=1,slot_num=5,state=1,ieee_virtual_eth=0,port_vlan_id=3,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62805

lpar_name=06-BF99Z,lpar_id=1,slot_num=6,state=1,ieee_virtual_eth=0,port_vlan_id=4,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62806

lpar_name=06-BF99Z,lpar_id=1,slot_num=12,state=1,ieee_virtual_eth=1,port_vlan_id=11,addl_vlan_ids=68,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA6280C

lpar_name=aixlpar1,lpar_id=2,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=68,addl_vlan_ids=none,is_trunk=0,trunk_priority=0,is_required=0,mac_addr=F67D5345AD04

 

 

Using the mkvdev command we created the SEA again. Then using the entstat command we found that the PVID and VLAN tags had been configured correctly.

 

$ mkvdev -sea ent6 -vadapter ent2 -default ent2 -defaultid 11

ent7 Available

en7

et7

 

$ entstat -all ent7 | grep -i vlan

VLAN Ids :

    VLAN Extract: False

    VLAN tagged filtering mode: Filter according to VLAN permit array

Max number of VLAN IDs per HEA port: 20

    VLAN Extract: False

    VLAN tagged filtering mode: Filter according to VLAN permit array

Max number of VLAN IDs per HEA port: 20

Invalid VLAN ID Packets: 5388

Port VLAN ID:    11

VLAN Tag IDs:    68

 

Once this was done, the client LPARs were able to ping the NIM master. The customer happily started installing AIX onto each of the blades Partitions.

 

A related link: http://www.aixmind.com/?p=2133

 

Here’s a question I received recently, that relates to virtual SCSI hdisk path priority.

 

Hi, Chris,

I use MPIO for our virtualization environment.

I'd like to check which path is currently used.

How can I get the result?

For example:

 

[nim:/]# lspath
Enabled hdisk0 vscsi0
Enabled hdisk0 vscsi1
Enabled hdisk2 vscsi2
Enabled hdisk1 vscsi1
Enabled hdisk1 vscsi0
Enabled hdisk3 vscsi2
Enabled hdisk2 vscsi3
Enabled hdisk3 vscsi3

 

How can I determine vscsi0 or vscsi1 is currently carrying data through VIOS and VIOC.

Thanks a lot!

 

By default the Virtual I/O Client (VIOC) will use the first VIOS for all VSCSI traffic. You can confirm which VSCSI adapter is being used by starting nmon (on the VIOC) and pressing the ‘a’ key. This will show which vscsi adapter is currently in use. e.g.

 

┌─topas_nmon──a=disk-Adapters────Host=lpar42p682e_pubRefresh=2 secs───18:38.04─┐

│ Disk-Adapter-I/O ────────────────────────────────────────────────────────────│

│Name          %busy     read    write        xfers Disks Adapter-Type         │

│vscsi3          0.0      0.0      0.0 KB/s     0.0 1     Virtual SCSI Client A│

│vscsi2        100.0      0.0 218560.0 KB/s  1707.5 1     Virtual SCSI Client A│

│TOTALS  2 adapters       0.0 218560.0 KB/s  1707.5 2     TOTAL(MB/s)=213.4    │

│──────────────────────────────────────────────────────────────────────────────│

 

Use the lspath command to determine the current path priority for a hdisk. By default it will be set to 1 for both adapters, meaning that traffic will go via the first path. Priority 1 is the highest priority, and you can define a priority from 1 to 255.

 

# lspath -AE -l hdisk2 -p vscsi2

priority 1 Priority True

 

# lspath -AE -l hdisk2 -p vscsi3

priority 1 Priority True

 

You can change the path priority of a hdisk using the chpath command. For example, we change the priority for vscsi2 to 255, so that vscsi3 now has the highest priority:

 

# chpath -l hdisk2 -a priority=255 -p vscsi2

# chpath -l hdisk2 -a priority=1 -p vscsi3

 

This will change the path priority for hdisk2 so that its traffic will use vscsi3 instead vscsi2.

 

┌─topas_nmon──b=Black&White──────Host=lpar42p682e_pubRefresh=2 secs───18:40.56─┐

│ Disk-Adapter-I/O ────────────────────────────────────────────────────────────│

│Name          %busy     read    write        xfers Disks Adapter-Type         │

│vscsi3        100.0      0.0 224384.5 KB/s  1753.0 1     Virtual SCSI Client A│

│vscsi2          0.0      0.0      0.0 KB/s     0.0 1     Virtual SCSI Client A│

│TOTALS  2 adapters       0.0 224384.5 KB/s  1753.0 2     TOTAL(MB/s)=219.1    │

│──────────────────────────────────────────────────────────────────────────────│

 

Now all the VSCSI traffic (for hdisk2) is traversing the vscsi3 adapter instead of vscsi2.

Recently I had the pleasure of configuring a couple of POWER7 720s for a customer. Each 720 was to host roughly 12 LPARs each. There were two VIO servers and a NIM server per 720.

 

Everything went along nicely and according to plan. In a few days we had both systems built. My steps for building the systems were almost identical to those described by Rob McNelly in a recent post.

All four VIO servers were running the latest VIOS code i.e. 2.2.0.10-FP-24 SP-01. All the client LPARs were running AIX 6.1 TL6 SP3. Each VIOS was configured with two FC paths to the SAN and the SAN storage device was an IBM DS5020.

 

Native AIX MPIO was in use on the VIOS and the AIX LPARs. I did not deploy SDDPCM on the VIOS as this is currently unsupported with the DS5020.

Once the LPAR builds were complete we performed a number of “integration tests”. These typically involve disconnecting network and SAN cables from each VIOS and observing how the VIOS and LPARs respond to and recover from these types of conditions.

 

One of the integration tests required that BOTH fibre cables be disconnected from the first VIOS and confirm that the client LPARs were not impacted i.e. that all I/O travelled via the second VIOS.

During the test we notice the following:

 

• I/O on the client LPAR would hang for approximately 5 minutes. Our test was simple enough, create a file in a file system. Eventually the I/O would continue after the 5 minute delay.

 

• The VIOS became sluggish and took some time to respond. The lspath command would take a very long time to return (or in some cases it would never return). During one test, the VIOS actually hung and had to be restarted (however we were not able to reproduce this issue again).

 

• When we reconnected both the cables, the paths would recover on the first VIOS after another 5 minutes. On the client however, it took roughly 20 minutes for the paths to recover. And yes, the hcheck_interval was set to 60 on all disks.

What's even more puzzling was that if we simply rebooted the first VIOS, everything worked as expected i.e. the client LPARs were not impacted, I/O continued as normal and when the first VIOS was back up, the paths on the client LPARs recovered quickly.

After doing some research, we discovered the following post on the IBM developerWorks AIX forum:

 

http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14472352&tstart=0

 

This post highlighted a number of things we needed to check and also confirmed several decisions we’d made during the design process, such as SDDPCM not being supported with DS5020 storage and VIOS (this was good as some people were starting to believe we should have installed SDDPCM to resolve this problem. I’d only be happy to do this if it was a supported combination and it’s not).

 

Finally we found the following IBM tech note that related directly to our issue.

 
IZ66020: ACTIVE/PASSIVE PCM CONTROLLER HCHECK SUPPORT
https://www-304.ibm.com/support/docview.wss?uid=isg1IZ66020

 

The following statement seemed to match our exact problem.

 

"For active/passive storage device, such as DS3K, DS4K, or DS5K if complete access is lost to the storage device, then it may take greater than 5 minutes to fail I/O. This feature is for Active/Passive storage devices, which are running with the AIX Default A/P PCM. This includes DS3K, DS4K, and DS5K family of devices.”

The new feature was described as follows.

 

“Added feature which health checks controllers, when an enabled path becomes unavailable, due to transport problems.  By default this feature is DISABLED.  To enable this feature set the following ODM attributes for the active/passive storage device.  Enabling this feature, results in faster I/O failure times.                                                      

cntl_delay_time:                                                        
Is the amount of time in seconds the storage device's controller(s) will be health checked after a transport failure.  At the end of this period, if no paths are detected as good, then all pending and subsequent I/O to the device will be failed, until the device health checker detects a failed path has returned.                                             
                                                                        
cntl_hcheck_int:                                                        
The first controller health check will only be issued after a storage fabric transport failure had been detected. cntl_hcheck_int is the amount of time in seconds, which the next controller health check command will be issued. This value must be less than the cntl_delay_time  (unless set to "0", disabled).                                                  

If you wish to allow the storage device 30 seconds to come back on the fabric (after leaving the fabric),  then you can set cntl_delay_time=30 and cntl_hcheck_int=2.     

         
The device, /dev/hdisk#, must not be in use, when setting the ODM values (or the chdev "-P" option must be used,  which requires a reboot).                                               
                                                                        
CAUTION: There are cases where the storage device may  reboot both of the controllers and become inaccessible for a period of time. If the controller health check sequence is enabled, then this may result in an I/O failure.  It is recommended to to make sure you have an mirrored volume           
to failover to, if you are running with controller health check enabled (especially with under 60 second cntl_delay_time). “

                     

And as I suspected the issue was related to the type of storage we were using. It appears the I/O delay was attributed to the following attributes on the DS5020 hdisks on the VIOS:

 

cntl_delay_time 0  Controller Delay Time            True
cntl_hcheck_int 0  Controller Health Check Interval True

 

Based on the tech note, I attempted several tests with various values for both parameters e.g.

 

$ chdev -dev hdiskX -attr cntl_delay_time=30                        
$ chdev -dev hdiskX -attr cntl_hcheck_int=2  

 

After making the changes to the hdisks on all VIOS, I performed the same test i.e. disconnected BOTH fibre cables from the first VIOS and continued to write a file to a file system on the client LPARs. By modifying these values on all the DS5020 disks, on all the VIO servers, the I/O delay was reduced to seconds rather than five minutes!

 

The following attributes were used for the hdisks and adapters in the final configuration.

 

On the VIO servers:

 

$ lsdev -type disk | grep DS5020

hdisk3           Available   MPIO DS5020 Disk

hdisk4           Available   MPIO DS5020 Disk

hdisk5           Available   MPIO DS5020 Disk

hdisk6           Available   MPIO DS5020 Disk

hdisk7           Available   MPIO DS5020 Disk

hdisk8           Available   MPIO DS5020 Disk

hdisk9           Available   MPIO DS5020 Disk

hdisk10          Available   MPIO DS5020 Disk

hdisk11          Available   MPIO DS5020 Disk

hdisk12          Available   MPIO DS5020 Disk

hdisk13          Available   MPIO DS5020 Disk

hdisk14          Available   MPIO DS5020 Disk

hdisk15          Available   MPIO DS5020 Disk

hdisk16          Available   MPIO DS5020 Disk

hdisk17          Available   MPIO DS5020 Disk

hdisk18          Available   MPIO DS5020 Disk

hdisk19          Available   MPIO DS5020 Disk

hdisk20          Available   MPIO DS5020 Disk

hdisk21          Available   MPIO DS5020 Disk

hdisk22          Available   MPIO DS5020 Disk

hdisk23          Available   MPIO DS5020 Disk

hdisk24          Available   MPIO DS5020 Disk

hdisk25          Available   MPIO DS5020 Disk

hdisk26          Available   MPIO DS5020 Disk

hdisk27          Available   MPIO DS5020 Disk

hdisk28          Available   MPIO DS5020 Disk

hdisk29          Available   MPIO DS5020 Disk

hdisk30          Available   MPIO DS5020 Disk

hdisk31          Available   MPIO DS5020 Disk

hdisk32          Available   MPIO DS5020 Disk

 

 

$ lsdev -dev hdisk3 -attr

attribute       value                                                          description                      user_settable

 

PCM             PCM/friend/otherapdisk                                         Path Control Module              False

PR_key_value    none                                                           Persistant Reserve Key Value     True

algorithm       fail_over                                                      Algorithm                        True

autorecovery    no                                                             Path/Ownership Autorecovery      True

clr_q           no                                                             Device CLEARS its Queue on error True

cntl_delay_time 15                                                             Controller Delay Time            True

cntl_hcheck_int 2                                                              Controller Health Check Interval True

dist_err_pcnt   0                                                              Distributed Error Percentage     True

dist_tw_width   50                                                             Distributed Error Sample Time    True

hcheck_cmd      inquiry                                                        Health Check Command             True

hcheck_interval 60                                                             Health Check Interval            True

hcheck_mode     nonactive                                                      Health Check Mode                True

location                                                                       Location Label                   True

lun_id          0x11000000000000                                               Logical Unit Number ID           False

lun_reset_spt   yes                                                            LUN Reset Supported              True

max_retry_delay 60                                                             Maximum Quiesce Time             True

max_transfer    0x40000                                                        Maximum TRANSFER Size            True

node_name       0x20040080e5187564                                             FC Node Name                     False

pvid            00f6482f7869e92d0000000000000000                               Physical volume identifier       False

q_err           yes                                                            Use QERR bit                     True

q_type          simple                                                         Queuing TYPE                     True

queue_depth     24                                                             Queue DEPTH                      True

reassign_to     120                                                            REASSIGN time out value          True

reserve_policy  no_reserve                                                     Reserve Policy                   True

rw_timeout      30                                                             READ/WRITE time out value        True

scsi_id         0x11600                                                        SCSI ID                          False

start_timeout   60                                                             START unit time out value        True

unique_id       3E21360080E50001875DE000005224D2CD63B0F1814      FAStT03IBMfcp Unique device identifier         False

ww_name         0x20150080e5187564                                             FC World Wide Name               False

 

 

$ lsdev -dev fscsi0 -attr

attribute    value     description                           user_settable

 

attach       switch    How this adapter is CONNECTED         False

dyntrk       yes       Dynamic Tracking of FC Devices        True

fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True

scsi_id      0x11400   Adapter SCSI ID                       False

sw_fc_class  3         FC Class for Fabric                   True

 

 

$ r oem

oem_setup_env

# manage_disk_drivers

Device           Present Driver     Driver Options

2810XIV          AIX_AAPCM          AIX_AAPCM,AIX_non_MPIO

DS4100           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4200           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4300           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4500           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4700           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4800           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS3950           AIX_APPCM          AIX_APPCM

DS5020           AIX_APPCM          AIX_APPCM

DS5100/DS5300AIX_APPCM AIX_APPCM          AIX_APPCM

DS3500           AIX_APPCM          AIX_APPCM

Usage :

        manage_disk_drivers [-l]

        manage_disk_drivers -d device -o driver_option

        For entries with multiple model names use the first one listed.

        Ex. DS5100/DS5300 use DS5100.

        manage_disk_drivers –h

 

# mpio_get_config -Av

Frame id 0:

    Storage Subsystem worldwide name: 609e50018345de00004da7998

    Controller count: 2

    Partition count: 1

    Partition 0:

    Storage Subsystem Name = 'MyApp-DS5020'

        hdisk      LUN #   Ownership          User Label

        hdisk3        17   B (preferred)      LPAR1

        hdisk4        18   A (preferred)      LPAR1datavg

        hdisk5        19   A (preferred)      LPAR1appvg

        hdisk6        16   B (preferred)      LPAR2

        hdisk7        20   A (preferred)      LPAR2datavg

        hdisk8        21   B (preferred)      LPAR2data3vg

        hdisk9        15   A (preferred)      LPAR3

        hdisk10       14   B (preferred)      LPAR4

        hdisk11       22   B (preferred)      LPAR5

        hdisk12       23   A (preferred)      LPAR6

        hdisk13       13   B (preferred)      LPAR6datavg

        hdisk14       12   A (preferred)      LPAR7

        hdisk15       24   B (preferred)      LPAR7datavg

        hdisk16       25   A (preferred)      LPAR8

        hdisk17       11   B (preferred)      LPAR8datavg

        hdisk18       10   A (preferred)      LPAR8datavg

        hdisk19        9   B (preferred)      LPAR8datavg

        hdisk20        8   A (preferred)      LPAR8datavg

        hdisk21       26   A (preferred)      LPAR9

        hdisk22        7   A (preferred)      LPAR9datavg

        hdisk23        6   B (preferred)      LPAR9datavg

        hdisk24       27   B (preferred)      LPAR9appvg

        hdisk25       28   A (preferred)      LPAR9binvg

        hdisk26        5   A (preferred)      LPAR10

        hdisk27        4   B (preferred)      LPAR10datavg

        hdisk28       29   B (preferred)      LPAR10datavg

        hdisk29       30   A (preferred)      LPAR11

        hdisk30        3   A (preferred)      LPAR11datavg

        hdisk31       32   B (preferred)      LPAR12

        hdisk32       50   B (preferred)      LPAR12datavg

 

On the VIO clients:

 

- Changed the following attributes for all virtual SCSI disks to:

 

# lsattr -El hdisk0

PCM             PCM/friend/vscsi                 Path Control Module        False

algorithm       fail_over                        Algorithm                  True

hcheck_cmd      test_unit_rdy                    Health Check Command       True

hcheck_interval 60                               Health Check Interval      True

hcheck_mode     nonactive                        Health Check Mode          True

max_transfer    0x40000                          Maximum TRANSFER Size      True

pvid            00f6482f7869e92d0000000000000000 Physical volume identifier False

queue_depth     24                               Queue DEPTH                True

reserve_policy  no_reserve                       Reserve Policy             True

 

- Changed the following attributes for all virtual SCSI adapters to:

 

# lsattr -El vscsi0

vscsi_err_recov fast_fail N/A                       True

vscsi_path_to   30        Virtual SCSI Path Timeout True

 

Note: With regard to SDDPCM support for DS5020 with VIOS, I’ve been referring to the following IBM website:

 

http://www-01.ibm.com/support/docview.wss?uid=ssg1S4000201

 

The site has a link to the SDDPCM readme file for DS5020 storage (under SDDPCM Package for DS5000):

 

ftp://ftp.software.ibm.com/storage/subsystem/aix/2.5.2.0/sddpcm.readme.2.5.2.0.txt

 

The readme states:

 

Note: VIOS is not supported with SDDPCM on DS4000/DS5000/DS5020/DS3950 subsystem devices.

 

During our research we had this confirmed by IBM support that, at this time, SDDPCM is not supported
on VIOS with DS5000.

 

I hope this helps others that may be about to implement this type of storage with a VIOS.

Here’s a nice little matrix of current VIOS levels and the corresponding AIX level.

 

 

The following link also has some fantastic information relating to NIM installation and backup of a Virtual I/O Server (VIOS).

https://www-304.ibm.com/support/docview.wss?uid=isg3T1011386

It has an official answer for one of the questions I’m asked the most:

Q: Do I need to use an lpp_source to install the VIO server using NIM?

A: No. All of the device drivers and software packages needed to install the VIO server are in the VIO server mksysb.

I also recommend Rob McNelly’s latest blog on installing VIO servers.

http://ibmsystemsmag.blogs.com/aixchange/2011/01/an-unusual-vios-install.html