Chris's AIX Blog

The first time I ran the ’viostat  –adapter’ command I expected to find non-zero values for kbps, tps, etc, for each vfchost adapter. However, the values were always zero, no matter how much traffic traversed the adapters.                                 

                                                                       

$ viostat -adapter 1 10                                                 

...                                                                     

vadapter:                  Kbps      tps     bkread    bkwrtn          

vfchost0                    0.0      0.0        0.0       0.0          

...                                                                    

vadapter:                  Kbps      tps     bkread    bkwrtn          

vfchost1                    0.0      0.0        0.0       0.0          

                                                                       

I wondered if this was expected behaviour. Was the output supposed to report the amount of pass-thru traffic per vfchost? In 2011, I posed this question on the IBM developerWorks PowerVM forum regarding this observation. One of the replies stated:

 

"viostat does not give statistics for NPIV devices. The vfchost adapter is just a passthru, it doesn't know what the commands it gets are."    

                                                                        

https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14697753&#14697753

 

I appreciated someone taking the time to answer my question but I was still curious. I tested the same command again (in 2013) on a recent VIOS level (2.2.2.1), but I received the same result. It was time to source an official answer on this behaviour.

 

Here is the official response I received from IBM:

 

1.  FC adapter stats in viostat/iostat do not include NPIV.           

                                                                       

2.  viostat & iostat are an aggregate of all the stats from the underlying disks, which of course NPIV doesn't have.

 

There's really no way for the vfchost adapter to monitor I/O, since it doesn't know what the commands it gets are. He's just a passthru, passing the commands he gets from the client directly to the physical FC adapter.                         

                                                                        

3.  You can run fcstat on the VIOS but that has the same issues/limitations mentioned above.

 

Intent here was that customers would use tools on the client to monitor this sort of thing.

 

To summarize the comments from Development:                            

                                                                       

viostat does NOT give statistics for NPIV devices.”

 

This made sense but I wondered why the tool hadn’t been changed to exclude vfchost adapters from the output (to avoid customer confusion). There's obviously no valid reason to ever display any information for this type of adapter. I also understood that it was expected that I/O would be monitored at the client LPAR level. But I must say that an option for monitoring VFC I/O from a VIO server would be advantageous i.e. a single source view of all I/O activity for all VFC clients; particularly when there are several hundred partitions on a frame.  The response was:

 

“…the way the vfchost driver currently works is that it calls iostadd to register a dkstat structure, resulting in the adapter being listed when viostat is called.  This is misleading, however, since the vfchost driver does not actually track I/O.  The commands coming from the client partition are simply passed as-is to the physical FC adapter, and we don't know if a particular command is an I/O command or not.  The iostadd call is left over from porting the code from the vscsi driver, and Development agrees it should probably have been removed before shipping the code.                            

                                                                       

There has also been mention of a DCR #MR0413117456 (Title: FC adapter stats in viostat/iostat does not include NPIV) which you can follow-up with Marketing to register your interest/track progress if that is something you're interested in pursuing.”

Here’s a question I received recently, that relates to virtual SCSI hdisk path priority.

 

Hi, Chris,

I use MPIO for our virtualization environment.

I'd like to check which path is currently used.

How can I get the result?

For example:

 

[nim:/]# lspath
Enabled hdisk0 vscsi0
Enabled hdisk0 vscsi1
Enabled hdisk2 vscsi2
Enabled hdisk1 vscsi1
Enabled hdisk1 vscsi0
Enabled hdisk3 vscsi2
Enabled hdisk2 vscsi3
Enabled hdisk3 vscsi3

 

How can I determine vscsi0 or vscsi1 is currently carrying data through VIOS and VIOC.

Thanks a lot!

 

By default the Virtual I/O Client (VIOC) will use the first VIOS for all VSCSI traffic. You can confirm which VSCSI adapter is being used by starting nmon (on the VIOC) and pressing the ‘a’ key. This will show which vscsi adapter is currently in use. e.g.

 

┌─topas_nmon──a=disk-Adapters────Host=lpar42p682e_pubRefresh=2 secs───18:38.04─┐

│ Disk-Adapter-I/O ────────────────────────────────────────────────────────────│

│Name          %busy     read    write        xfers Disks Adapter-Type         │

│vscsi3          0.0      0.0      0.0 KB/s     0.0 1     Virtual SCSI Client A│

│vscsi2        100.0      0.0 218560.0 KB/s  1707.5 1     Virtual SCSI Client A│

│TOTALS  2 adapters       0.0 218560.0 KB/s  1707.5 2     TOTAL(MB/s)=213.4    │

│──────────────────────────────────────────────────────────────────────────────│

 

Use the lspath command to determine the current path priority for a hdisk. By default it will be set to 1 for both adapters, meaning that traffic will go via the first path. Priority 1 is the highest priority, and you can define a priority from 1 to 255.

 

# lspath -AE -l hdisk2 -p vscsi2

priority 1 Priority True

 

# lspath -AE -l hdisk2 -p vscsi3

priority 1 Priority True

 

You can change the path priority of a hdisk using the chpath command. For example, we change the priority for vscsi2 to 255, so that vscsi3 now has the highest priority:

 

# chpath -l hdisk2 -a priority=255 -p vscsi2

# chpath -l hdisk2 -a priority=1 -p vscsi3

 

This will change the path priority for hdisk2 so that its traffic will use vscsi3 instead vscsi2.

 

┌─topas_nmon──b=Black&White──────Host=lpar42p682e_pubRefresh=2 secs───18:40.56─┐

│ Disk-Adapter-I/O ────────────────────────────────────────────────────────────│

│Name          %busy     read    write        xfers Disks Adapter-Type         │

│vscsi3        100.0      0.0 224384.5 KB/s  1753.0 1     Virtual SCSI Client A│

│vscsi2          0.0      0.0      0.0 KB/s     0.0 1     Virtual SCSI Client A│

│TOTALS  2 adapters       0.0 224384.5 KB/s  1753.0 2     TOTAL(MB/s)=219.1    │

│──────────────────────────────────────────────────────────────────────────────│

 

Now all the VSCSI traffic (for hdisk2) is traversing the vscsi3 adapter instead of vscsi2.

Make sure you are monitoring file system space on your VIOS.

 

Why?

 

If you run out of space in the root file system, odd things can happen when you try to map virtual devices to virtual adapters with mkvdev.

 

For example, a colleague of mine was attempting to map a new hdisk to a vhost adapter on a pair of VIOS. The VIOS was running a recent version of code. He received the following error message (see below). It wasn’t a very helpful message. At first I thought it was due to the fact that he had not set the reserve_policy attribute for the new disk to no_reserve on both VIOS. Changing the value for that attribute did not help.

 

$ ioslevel

2.2.1.3

 

$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1

*******************************************************************************

The command's response was not recognized.  This may or may not indicate a problem.

*******************************************************************************

 

I enabled debug output from the padmin command line and discovered that the mkdev command was failing for some reason.

 

$ export CLI_DEBUG=33

 

$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1

AIX: "lspv -l hdisk1 2>&1 | grep 0516-320"

AIX: "mkdev  -V hdisk1  -p vhost0  -l vhdisk1 "

*******************************************************************************

The command's response was not recognized.  This may or may not indicate a problem.

*******************************************************************************

 

I entered oem_setup_env and ran the mkdev command manually. I was presented with a very clear error message describing the problem. The root file system (/) was full!

 

$ oem_setup_env

 

# mkdev  -V hdisk1  -p vhost0  -l vhdisk1

mkdev: 0514-548 Cannot perform the requested function because

                the / filesystem is full or is out of inodes.

 

# df -g /

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4           0.50      0.00  100%     7728    75% /

 

I found a 45MB core file (generated by cimserver) in the root file system. I moved the file to another file system (with plenty of free space) for further investigation.

 

# mv core /tmp/core.cg

# df -m .

Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4         512.00     45.93   92%     7727    38% /

 

Then I ran the mkvdev command successfully.

 

$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1

AIX: "lspv -l hdisk1 2>&1 | grep 0516-320"

AIX: "mkdev  -V hdisk1  -p vhost0  -l vhdisk1 "

vhdisk1 Available

AIX: "lsattr -El hdisk1 -a max_transfer |cut -d" " -f2"

 

$ lsmap -all

AIX: "lsdev -c adapter -t IBM,v-scsi-host -s vdevice -F "name" | wc -l -c"

AIX: "lsdev -c adapter -t IBM,v-scsi-host -s vdevice -F "name""

AIX: "lsdev -C -l vhost0 -F "physloc""

AIX: "lsdev -p vhost0 -F "name" | wc -l -c"

AIX: "lsdev -p vhost0 -F "name""

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost0          U9119.FHA.87654B1-V6-C21                   0x0000002e

 

VTD                   vhdisk1

Status                Available

LUN                   0x8100000000000000

Backing device        hdisk1

Physloc               U5873.001.9SS005Y-P1-C9-T1-W50060E8006D0206A-L1000000000000

Mirrored              false

 

AIX: "lsdev -C -l vhost1 -F "physloc""

AIX: "lsdev -p vhost1 -F "name" | wc -l -c"

AIX: "lsdev -p vhost1 -F "name""

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost1          U9119.FHA.87654B1-V6-C22                   0x0000002g

 

VTD                   NO VIRTUAL TARGET DEVICE FOUND

 

I found the same issue on the second VIOS i.e. a full root file system due to a core file (from cimserver). I also found no trace of a full file system event in the error report. Perhaps someone had taken it upon themselves to “clean house” at some point and had removed entries from the VIOS error log.

 

Make sure you monitor file system space on your VIOS. Who knows what else might fail if you run out of space in a critical file system.

Recently I had the pleasure of configuring a couple of POWER7 720s for a customer. Each 720 was to host roughly 12 LPARs each. There were two VIO servers and a NIM server per 720.

 

Everything went along nicely and according to plan. In a few days we had both systems built. My steps for building the systems were almost identical to those described by Rob McNelly in a recent post.

All four VIO servers were running the latest VIOS code i.e. 2.2.0.10-FP-24 SP-01. All the client LPARs were running AIX 6.1 TL6 SP3. Each VIOS was configured with two FC paths to the SAN and the SAN storage device was an IBM DS5020.

 

Native AIX MPIO was in use on the VIOS and the AIX LPARs. I did not deploy SDDPCM on the VIOS as this is currently unsupported with the DS5020.

Once the LPAR builds were complete we performed a number of “integration tests”. These typically involve disconnecting network and SAN cables from each VIOS and observing how the VIOS and LPARs respond to and recover from these types of conditions.

 

One of the integration tests required that BOTH fibre cables be disconnected from the first VIOS and confirm that the client LPARs were not impacted i.e. that all I/O travelled via the second VIOS.

During the test we notice the following:

 

• I/O on the client LPAR would hang for approximately 5 minutes. Our test was simple enough, create a file in a file system. Eventually the I/O would continue after the 5 minute delay.

 

• The VIOS became sluggish and took some time to respond. The lspath command would take a very long time to return (or in some cases it would never return). During one test, the VIOS actually hung and had to be restarted (however we were not able to reproduce this issue again).

 

• When we reconnected both the cables, the paths would recover on the first VIOS after another 5 minutes. On the client however, it took roughly 20 minutes for the paths to recover. And yes, the hcheck_interval was set to 60 on all disks.

What's even more puzzling was that if we simply rebooted the first VIOS, everything worked as expected i.e. the client LPARs were not impacted, I/O continued as normal and when the first VIOS was back up, the paths on the client LPARs recovered quickly.

After doing some research, we discovered the following post on the IBM developerWorks AIX forum:

 

http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14472352&tstart=0

 

This post highlighted a number of things we needed to check and also confirmed several decisions we’d made during the design process, such as SDDPCM not being supported with DS5020 storage and VIOS (this was good as some people were starting to believe we should have installed SDDPCM to resolve this problem. I’d only be happy to do this if it was a supported combination and it’s not).

 

Finally we found the following IBM tech note that related directly to our issue.

 
IZ66020: ACTIVE/PASSIVE PCM CONTROLLER HCHECK SUPPORT
https://www-304.ibm.com/support/docview.wss?uid=isg1IZ66020

 

The following statement seemed to match our exact problem.

 

"For active/passive storage device, such as DS3K, DS4K, or DS5K if complete access is lost to the storage device, then it may take greater than 5 minutes to fail I/O. This feature is for Active/Passive storage devices, which are running with the AIX Default A/P PCM. This includes DS3K, DS4K, and DS5K family of devices.”

The new feature was described as follows.

 

“Added feature which health checks controllers, when an enabled path becomes unavailable, due to transport problems.  By default this feature is DISABLED.  To enable this feature set the following ODM attributes for the active/passive storage device.  Enabling this feature, results in faster I/O failure times.                                                      

cntl_delay_time:                                                        
Is the amount of time in seconds the storage device's controller(s) will be health checked after a transport failure.  At the end of this period, if no paths are detected as good, then all pending and subsequent I/O to the device will be failed, until the device health checker detects a failed path has returned.                                             
                                                                        
cntl_hcheck_int:                                                        
The first controller health check will only be issued after a storage fabric transport failure had been detected. cntl_hcheck_int is the amount of time in seconds, which the next controller health check command will be issued. This value must be less than the cntl_delay_time  (unless set to "0", disabled).                                                  

If you wish to allow the storage device 30 seconds to come back on the fabric (after leaving the fabric),  then you can set cntl_delay_time=30 and cntl_hcheck_int=2.     

         
The device, /dev/hdisk#, must not be in use, when setting the ODM values (or the chdev "-P" option must be used,  which requires a reboot).                                               
                                                                        
CAUTION: There are cases where the storage device may  reboot both of the controllers and become inaccessible for a period of time. If the controller health check sequence is enabled, then this may result in an I/O failure.  It is recommended to to make sure you have an mirrored volume           
to failover to, if you are running with controller health check enabled (especially with under 60 second cntl_delay_time). “

                     

And as I suspected the issue was related to the type of storage we were using. It appears the I/O delay was attributed to the following attributes on the DS5020 hdisks on the VIOS:

 

cntl_delay_time 0  Controller Delay Time            True
cntl_hcheck_int 0  Controller Health Check Interval True

 

Based on the tech note, I attempted several tests with various values for both parameters e.g.

 

$ chdev -dev hdiskX -attr cntl_delay_time=30                        
$ chdev -dev hdiskX -attr cntl_hcheck_int=2  

 

After making the changes to the hdisks on all VIOS, I performed the same test i.e. disconnected BOTH fibre cables from the first VIOS and continued to write a file to a file system on the client LPARs. By modifying these values on all the DS5020 disks, on all the VIO servers, the I/O delay was reduced to seconds rather than five minutes!

 

The following attributes were used for the hdisks and adapters in the final configuration.

 

On the VIO servers:

 

$ lsdev -type disk | grep DS5020

hdisk3           Available   MPIO DS5020 Disk

hdisk4           Available   MPIO DS5020 Disk

hdisk5           Available   MPIO DS5020 Disk

hdisk6           Available   MPIO DS5020 Disk

hdisk7           Available   MPIO DS5020 Disk

hdisk8           Available   MPIO DS5020 Disk

hdisk9           Available   MPIO DS5020 Disk

hdisk10          Available   MPIO DS5020 Disk

hdisk11          Available   MPIO DS5020 Disk

hdisk12          Available   MPIO DS5020 Disk

hdisk13          Available   MPIO DS5020 Disk

hdisk14          Available   MPIO DS5020 Disk

hdisk15          Available   MPIO DS5020 Disk

hdisk16          Available   MPIO DS5020 Disk

hdisk17          Available   MPIO DS5020 Disk

hdisk18          Available   MPIO DS5020 Disk

hdisk19          Available   MPIO DS5020 Disk

hdisk20          Available   MPIO DS5020 Disk

hdisk21          Available   MPIO DS5020 Disk

hdisk22          Available   MPIO DS5020 Disk

hdisk23          Available   MPIO DS5020 Disk

hdisk24          Available   MPIO DS5020 Disk

hdisk25          Available   MPIO DS5020 Disk

hdisk26          Available   MPIO DS5020 Disk

hdisk27          Available   MPIO DS5020 Disk

hdisk28          Available   MPIO DS5020 Disk

hdisk29          Available   MPIO DS5020 Disk

hdisk30          Available   MPIO DS5020 Disk

hdisk31          Available   MPIO DS5020 Disk

hdisk32          Available   MPIO DS5020 Disk

 

 

$ lsdev -dev hdisk3 -attr

attribute       value                                                          description                      user_settable

 

PCM             PCM/friend/otherapdisk                                         Path Control Module              False

PR_key_value    none                                                           Persistant Reserve Key Value     True

algorithm       fail_over                                                      Algorithm                        True

autorecovery    no                                                             Path/Ownership Autorecovery      True

clr_q           no                                                             Device CLEARS its Queue on error True

cntl_delay_time 15                                                             Controller Delay Time            True

cntl_hcheck_int 2                                                              Controller Health Check Interval True

dist_err_pcnt   0                                                              Distributed Error Percentage     True

dist_tw_width   50                                                             Distributed Error Sample Time    True

hcheck_cmd      inquiry                                                        Health Check Command             True

hcheck_interval 60                                                             Health Check Interval            True

hcheck_mode     nonactive                                                      Health Check Mode                True

location                                                                       Location Label                   True

lun_id          0x11000000000000                                               Logical Unit Number ID           False

lun_reset_spt   yes                                                            LUN Reset Supported              True

max_retry_delay 60                                                             Maximum Quiesce Time             True

max_transfer    0x40000                                                        Maximum TRANSFER Size            True

node_name       0x20040080e5187564                                             FC Node Name                     False

pvid            00f6482f7869e92d0000000000000000                               Physical volume identifier       False

q_err           yes                                                            Use QERR bit                     True

q_type          simple                                                         Queuing TYPE                     True

queue_depth     24                                                             Queue DEPTH                      True

reassign_to     120                                                            REASSIGN time out value          True

reserve_policy  no_reserve                                                     Reserve Policy                   True

rw_timeout      30                                                             READ/WRITE time out value        True

scsi_id         0x11600                                                        SCSI ID                          False

start_timeout   60                                                             START unit time out value        True

unique_id       3E21360080E50001875DE000005224D2CD63B0F1814      FAStT03IBMfcp Unique device identifier         False

ww_name         0x20150080e5187564                                             FC World Wide Name               False

 

 

$ lsdev -dev fscsi0 -attr

attribute    value     description                           user_settable

 

attach       switch    How this adapter is CONNECTED         False

dyntrk       yes       Dynamic Tracking of FC Devices        True

fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True

scsi_id      0x11400   Adapter SCSI ID                       False

sw_fc_class  3         FC Class for Fabric                   True

 

 

$ r oem

oem_setup_env

# manage_disk_drivers

Device           Present Driver     Driver Options

2810XIV          AIX_AAPCM          AIX_AAPCM,AIX_non_MPIO

DS4100           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4200           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4300           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4500           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4700           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4800           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS3950           AIX_APPCM          AIX_APPCM

DS5020           AIX_APPCM          AIX_APPCM

DS5100/DS5300AIX_APPCM AIX_APPCM          AIX_APPCM

DS3500           AIX_APPCM          AIX_APPCM

Usage :

        manage_disk_drivers [-l]

        manage_disk_drivers -d device -o driver_option

        For entries with multiple model names use the first one listed.

        Ex. DS5100/DS5300 use DS5100.

        manage_disk_drivers –h

 

# mpio_get_config -Av

Frame id 0:

    Storage Subsystem worldwide name: 609e50018345de00004da7998

    Controller count: 2

    Partition count: 1

    Partition 0:

    Storage Subsystem Name = 'MyApp-DS5020'

        hdisk      LUN #   Ownership          User Label

        hdisk3        17   B (preferred)      LPAR1

        hdisk4        18   A (preferred)      LPAR1datavg

        hdisk5        19   A (preferred)      LPAR1appvg

        hdisk6        16   B (preferred)      LPAR2

        hdisk7        20   A (preferred)      LPAR2datavg

        hdisk8        21   B (preferred)      LPAR2data3vg

        hdisk9        15   A (preferred)      LPAR3

        hdisk10       14   B (preferred)      LPAR4

        hdisk11       22   B (preferred)      LPAR5

        hdisk12       23   A (preferred)      LPAR6

        hdisk13       13   B (preferred)      LPAR6datavg

        hdisk14       12   A (preferred)      LPAR7

        hdisk15       24   B (preferred)      LPAR7datavg

        hdisk16       25   A (preferred)      LPAR8

        hdisk17       11   B (preferred)      LPAR8datavg

        hdisk18       10   A (preferred)      LPAR8datavg

        hdisk19        9   B (preferred)      LPAR8datavg

        hdisk20        8   A (preferred)      LPAR8datavg

        hdisk21       26   A (preferred)      LPAR9

        hdisk22        7   A (preferred)      LPAR9datavg

        hdisk23        6   B (preferred)      LPAR9datavg

        hdisk24       27   B (preferred)      LPAR9appvg

        hdisk25       28   A (preferred)      LPAR9binvg

        hdisk26        5   A (preferred)      LPAR10

        hdisk27        4   B (preferred)      LPAR10datavg

        hdisk28       29   B (preferred)      LPAR10datavg

        hdisk29       30   A (preferred)      LPAR11

        hdisk30        3   A (preferred)      LPAR11datavg

        hdisk31       32   B (preferred)      LPAR12

        hdisk32       50   B (preferred)      LPAR12datavg

 

On the VIO clients:

 

- Changed the following attributes for all virtual SCSI disks to:

 

# lsattr -El hdisk0

PCM             PCM/friend/vscsi                 Path Control Module        False

algorithm       fail_over                        Algorithm                  True

hcheck_cmd      test_unit_rdy                    Health Check Command       True

hcheck_interval 60                               Health Check Interval      True

hcheck_mode     nonactive                        Health Check Mode          True

max_transfer    0x40000                          Maximum TRANSFER Size      True

pvid            00f6482f7869e92d0000000000000000 Physical volume identifier False

queue_depth     24                               Queue DEPTH                True

reserve_policy  no_reserve                       Reserve Policy             True

 

- Changed the following attributes for all virtual SCSI adapters to:

 

# lsattr -El vscsi0

vscsi_err_recov fast_fail N/A                       True

vscsi_path_to   30        Virtual SCSI Path Timeout True

 

Note: With regard to SDDPCM support for DS5020 with VIOS, I’ve been referring to the following IBM website:

 

http://www-01.ibm.com/support/docview.wss?uid=ssg1S4000201

 

The site has a link to the SDDPCM readme file for DS5020 storage (under SDDPCM Package for DS5000):

 

ftp://ftp.software.ibm.com/storage/subsystem/aix/2.5.2.0/sddpcm.readme.2.5.2.0.txt

 

The readme states:

 

Note: VIOS is not supported with SDDPCM on DS4000/DS5000/DS5020/DS3950 subsystem devices.

 

During our research we had this confirmed by IBM support that, at this time, SDDPCM is not supported
on VIOS with DS5000.

 

I hope this helps others that may be about to implement this type of storage with a VIOS.

I came across this strange LPM issue recently. Thought I’d share it with you.

 

All the customers VIOS were configured with the viosecure level set to high. 

When the VIOS is configured with a security profile set to high, PCI or DoD, a new feature is enabled during LPM. This new feature is called “Secure LPM”. As a result when you initiate an LPM operation, the “Secure LPM” feature automatically enables the VIOS firewall and configures a secure (ipsec) tunnel for all LPM traffic over the network.

Before we started LPM, the VIOS firewall is off:

 

$ viosecure -firewall view

IPv4 Firewall OFF

 

                          ALLOWED   PORTS

           Local   Remote

Interface  Port    Port    Service      IPAddress       Expiration

Time(seconds)

---------  ----    ----    -------      ---------       ---------------

 

And it's activated after we start LPM:

 

$ viosecure -firewall view

IPv4 Firewall ON

 

                          ALLOWED   PORTS

           Local   Remote

Interface  Port    Port    Service      IPAddress       Expiration

Time(seconds)

---------  ----    ----    -------      ---------       ---------------

In the customer’s case, we moved a partition from a 795 running a pair of VIOS at 2.2.1.3 to another 795 running another pair of VIOS at 2.2.2.1. This worked OK. When we attempted to move the partition back, we received an error stating that the source MSP had rejected the request.

HSCLA230 

The mover service partition on the source managed system has rejected the request to stop the migration. Verify that the migration state of the partition is Migration Starting, and try the operation again.

After a while we discovered that when the LPM operation started, the source VIOS would enable its firewall and we would immediately loose connectivity (i.e. our SSH session to the VIOS would hang). We also found that RMC connectivity between the source VIOS and HMC also dropped. Somehow the VIOS firewall was blocking network connectivity to pretty much everything. 

Eventually, after much digging we found that the source VIOS was the victim of an errant firewall rule (configured on the source VIOS itself). 

# smitty ips4_conf_filter

List IP Security Filter Rules

1 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 no udp eq 4001 eq 4001 both both no all packets 0 all 0 none  Default Rule

2 *** Dynamic filter placement rule for IKE tunnels *** no

3 deny 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 y all any 0 any 0 all both inbound no yes 0 no 0 patt_none

0 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 yes all any 0 any 0 both both no all packets 0 all 0 none  Default Rule

We removed this rule (after some discussion with IBM support).

# smitty ips4_conf_filter

Delete IP Security Filter Rules

3 deny 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 y all any 0 any 0 all both inbound no yes 0 no 0 patt_none

# smitty ips4_conf_filter

List IP Security Filter Rules

1 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 no udp eq 4001 eq 4001 both both no all packets 0 all 0 none  Default Rule

2 *** Dynamic filter placement rule for IKE tunnels *** no

0 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 yes all any 0 any 0 both both no all packets 0 all 0 none  Default Rule

We were then able to move the partition back to the original frame. We are still searching for an answer to how this happened. The IBM support team believe the rule must have been created by an administrator at some point. Of course, the administrator claims that is untrue.

Until recently, if you were configuring a new LPAR with virtual FC adapters you couldn’t force it to log into the SAN before an operating system (such as AIX) was installed. I’ve written about this before (see link below). I also offered a way to work around this issue.

 

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/entry/new_hmc_commands_chnportlogin_and_lsnportlogin_login_in_virtual_fibre_channel_adapters_for_npiv_client_lpars1?lang=en

 

As mentioned in my post, there are new HMC commands to force a new LPAR to connect to the SAN, so that SAN administrators can zone storage to the new WWPN(s).

 

There’s also another method available in the Power Systems SMS menu.

 

In recent levels of POWER systems firmware, there is a new option in the SMS menu called SAN Zoning Support. You’ll find it mentioned here:

 

http://pic.dhe.ibm.com/infocenter/powersys/v3r1m5/index.jsp?topic=/p7hb6l/usingsmsmenus_sanzoning.htm

 

I’ve successfully used this method on both POWER6 (595) and POWER7 (795) systems. After configuring a new LPAR profile with a single VFC adapter, the VIOS reported that the client was not logged into the SAN:

 

$ lsmap -vadapter vfchost41 -npiv

Name          Physloc                            ClntID ClntName       ClntOS

------------- ---------------------------------- ------ -------------- -------

vfchost41     U9119.FHA.99D99C1-V2-C199              99

 

Status:NOT_LOGGED_IN

FC name:fcs4                    FC loc code:U5803.001.9912345-P2-C5-T1

Ports logged in:0

Flags:4<NOT_LOGGED>

VFC client name:                VFC client DRC:

 

I forced this event by booting into SMS and selecting 5 (Boot Options).

 

PowerPC Firmware

 Version EH350_120

 SMS 1.7 (c) Copyright IBM Corp. 2000,2008 All rights reserved.

-------------------------------------------------------------------------------

 Main Menu

 1.   Select Language

 2.   Setup Remote IPL (Initial Program Load)

 3.   Change SCSI Settings

 4.   Select Console

 5.   Select Boot Options

 

 

 -------------------------------------------------------------------------------

 Navigation Keys:

 

                                             X = eXit System Management Services

 -------------------------------------------------------------------------------

 Type menu item number and press Enter or select Navigation key:5

 

Then I selected 4 (SAN Zoning Support).

 

 PowerPC Firmware

 Version EH350_120

 SMS 1.7 (c) Copyright IBM Corp. 2000,2008 All rights reserved.

-------------------------------------------------------------------------------

 Multiboot

 1.   Select Install/Boot Device

 2.   Configure Boot Device Order

 3.   Multiboot Startup <OFF>

 4.   SAN Zoning Support

 

 

 

 

 -------------------------------------------------------------------------------

 Navigation keys:

 M = return to Main Menu

 ESC key = return to previous screen         X = eXit System Management Services

 -------------------------------------------------------------------------------

 Type menu item number and press Enter or select Navigation key:4

 

Next I choose the appropriate VFC adapter I’d like to connect on the SAN at this time.

 

 PowerPC Firmware

 Version EH350_120

 SMS 1.7 (c) Copyright IBM Corp. 2000,2008 All rights reserved.

-------------------------------------------------------------------------------

 Select Media Adapter

 1.  U9119.FHA.83D99C3-V99-C20-T1    /vdevice/vfc-client@30000014

 

 

 -------------------------------------------------------------------------------

 Navigation keys:

 M = return to Main Menu

 ESC key = return to previous screen         X = eXit System Management Services

 -------------------------------------------------------------------------------

 Type menu item number and press Enter or select Navigation key:1

 

 

A message appears stating that it may now be possible to perform SAN zoning to the adapters WWPN.

 

   .-----------------------------------------------------------------------.

   |  The selected adapter has been opened.                                |

   |  Zoning of attached disks may now be possible.                        |

   |  Press any key to close the adapter and return to the previous menu.  |

   `-----------------------------------------------------------------------'

 

On the VIOS I observed the client vfchost adapter status changed to LOGGED_IN.

 

$ lsmap -vadapter vfchost41 -npiv

Name          Physloc                            ClntID ClntName       ClntOS

------------- ---------------------------------- ------ -------------- -------

vfchost41     U9119.FHA.99D99C1-V2-C199              99 aixlpar1     FIRMWARE

 

Status:LOGGED_IN

FC name:fcs4                    FC loc code:U5803.001.9912345-P2-C5-T1

Ports logged in:1

Flags:a<LOGGED_IN,STRIP_MERGE>

VFC client name:Virtual Fibre Channel - Firmware driverVFC client DRC:U9119.FHA.83D99C3-V99-C20

 

It appears that the new SMS option is simply an interface to the Open Firmware Prompt and the IOINFO command. This was how we performed the task without the SMS menu in front:

 

https://www.ibm.com/developerworks/wikis/display/virtualization/vfclogin

 

So far I’ve found this new SMS option on POWER7 795 systems running firmware level AH730_078 and POWER6 595 systems running EH350_120.

When attempting to change the max_transfer attribute on a VSCSI hdisk, the following error was displayed:

 

# chdev -l hdisk0 -a max_transfer=0x100000 -P

Method error (/etc/methods/chgdisk):

        0514-018 The values specified for the following attributes

                 are not valid:

     max_transfer     Maximum TRANSFER Size

 

The desired value, 0x100000, was an allowed value for this type of device (as shown in the lsattr output below):

 

# lsattr -El hdisk0

PCM             PCM/friend/vscsi                 Path Control Module        False

algorithm       fail_over                        Algorithm                  True

hcheck_cmd      test_unit_rdy                    Health Check Command       True

hcheck_interval 60                               Health Check Interval      True

hcheck_mode     nonactive                        Health Check Mode          True

max_transfer    0x40000                          Maximum TRANSFER Size      True

pvid            00f6a83aeb1948950000000000000000 Physical volume identifier False

queue_depth     256                              Queue DEPTH                True

reserve_policy  no_reserve                       Reserve Policy             True

 

# lsattr -Rl hdisk0 -a max_transfer

0x20000

0x40000

0x80000

0x100000

0x200000

0x400000

0x800000

0x1000000

 

After some discussion with IBM support, it was discovered that the virtual optical device, assigned to this LPAR, was causing the problem. They advised that we should perform the following steps:

 

1. Remove the cd0 device from the LPAR i.e.

   # rmdev –dl cd0

 

2. Remove the vtopt0 device assigned to the LPAR on the VIOS i.e

   $ rmvdev –vtd vtopt0

 

3. Shutdown the LPAR.

 

4. Reboot the VIOS.

 

5. Restart the LPAR.

 

6. Change the max_transfer value for the hdisk on the LPAR.

 

Sure enough, after following these steps, we were able to change max_transfer to the desired value!

 

# chdev -l hdisk0 -a max_transfer=0x100000

hdisk0 changed

 

# lsattr -El hdisk0

PCM             PCM/friend/vscsi                 Path Control Module        False

algorithm       fail_over                        Algorithm                  True

hcheck_cmd      test_unit_rdy                    Health Check Command       True

hcheck_interval 0                                Health Check Interval      True

hcheck_mode     nonactive                        Health Check Mode          True

max_transfer    0x100000                         Maximum TRANSFER Size      True

pvid            00f6a83aeedf5ffb0000000000000000 Physical volume identifier False

queue_depth     3                                Queue DEPTH                True

reserve_policy  no_reserve                       Reserve Policy             True

 

The following explanation was given by IBM support:

 

“The virtual optical device seems to use max_transfer size as 0x40000 (256 KB) internally, which is not allowed to be modified. Virtual optical CD's max_transfer seems to limit vhost adapter's max_transfer size. So if you want to increase max_transfer size for virtual SCSI disks, you should separate them from virtual optical SCSI CD by configuring those two groups under different vhost adapters”

Here’s a nice little matrix of current VIOS levels and the corresponding AIX level.

 

 

The following link also has some fantastic information relating to NIM installation and backup of a Virtual I/O Server (VIOS).

https://www-304.ibm.com/support/docview.wss?uid=isg3T1011386

It has an official answer for one of the questions I’m asked the most:

Q: Do I need to use an lpp_source to install the VIO server using NIM?

A: No. All of the device drivers and software packages needed to install the VIO server are in the VIO server mksysb.

I also recommend Rob McNelly’s latest blog on installing VIO servers.

http://ibmsystemsmag.blogs.com/aixchange/2011/01/an-unusual-vios-install.html

 

It’s possible to search through seastat command output on the VIOS using the –s flag. You specify the following attributes with this flag: searchtype=value.

Valid search values include:

 

mac, priority, vlan, ip, host,

greaterbytessent (gbs), greaterbytesrecv (gbr),

greaterpktssent (gps), greaterpktsrecv (gpr),

smallerbytessent (sbs), smallerbytesrecv (sbr),

smallerpktssent (sps), smallerpktsrecv (spr)

 

For example, to search for a particular host:

 

$ seastat -d ent19 -s host=aixvioc1

 

================================================================================

 

Advanced Statistics for SEA

Device Name: ent19

 

================================================================================

MAC: 42:BA:1D:00:B9:0C

----------------------

 

VLAN: None

VLAN Priority: None

Hostname: aixvioc1.xyz.com

IP: 10.1.1.7

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 11630504                      Packets: 16040250

Bytes: 12106445229                     Bytes: 3629657696

 

================================================================================

 

To search for a particular IP address:

 

$ seastat -d ent19 -n -s ip=10.1.1.9

 

================================================================================

 

Advanced Statistics for SEA

Device Name: ent19

 

================================================================================

MAC: 2A:8C:8B:7C:D0:0C

----------------------

 

VLAN: None

VLAN Priority: None

IP: 10.1.1.9

 

Transmit Statistics:                   Receive Statistics:

--------------------                   -------------------

Packets: 30517                         Packets: 14900

Bytes: 36280372                        Bytes: 1751127

 

================================================================================