Chris's AIX Blog

Recently I had the pleasure of configuring a couple of POWER7 720s for a customer. Each 720 was to host roughly 12 LPARs each. There were two VIO servers and a NIM server per 720.

 

Everything went along nicely and according to plan. In a few days we had both systems built. My steps for building the systems were almost identical to those described by Rob McNelly in a recent post.

All four VIO servers were running the latest VIOS code i.e. 2.2.0.10-FP-24 SP-01. All the client LPARs were running AIX 6.1 TL6 SP3. Each VIOS was configured with two FC paths to the SAN and the SAN storage device was an IBM DS5020.

 

Native AIX MPIO was in use on the VIOS and the AIX LPARs. I did not deploy SDDPCM on the VIOS as this is currently unsupported with the DS5020.

Once the LPAR builds were complete we performed a number of “integration tests”. These typically involve disconnecting network and SAN cables from each VIOS and observing how the VIOS and LPARs respond to and recover from these types of conditions.

 

One of the integration tests required that BOTH fibre cables be disconnected from the first VIOS and confirm that the client LPARs were not impacted i.e. that all I/O travelled via the second VIOS.

During the test we notice the following:

 

• I/O on the client LPAR would hang for approximately 5 minutes. Our test was simple enough, create a file in a file system. Eventually the I/O would continue after the 5 minute delay.

 

• The VIOS became sluggish and took some time to respond. The lspath command would take a very long time to return (or in some cases it would never return). During one test, the VIOS actually hung and had to be restarted (however we were not able to reproduce this issue again).

 

• When we reconnected both the cables, the paths would recover on the first VIOS after another 5 minutes. On the client however, it took roughly 20 minutes for the paths to recover. And yes, the hcheck_interval was set to 60 on all disks.

What's even more puzzling was that if we simply rebooted the first VIOS, everything worked as expected i.e. the client LPARs were not impacted, I/O continued as normal and when the first VIOS was back up, the paths on the client LPARs recovered quickly.

After doing some research, we discovered the following post on the IBM developerWorks AIX forum:

 

http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14472352&tstart=0

 

This post highlighted a number of things we needed to check and also confirmed several decisions we’d made during the design process, such as SDDPCM not being supported with DS5020 storage and VIOS (this was good as some people were starting to believe we should have installed SDDPCM to resolve this problem. I’d only be happy to do this if it was a supported combination and it’s not).

 

Finally we found the following IBM tech note that related directly to our issue.

 
IZ66020: ACTIVE/PASSIVE PCM CONTROLLER HCHECK SUPPORT
https://www-304.ibm.com/support/docview.wss?uid=isg1IZ66020

 

The following statement seemed to match our exact problem.

 

"For active/passive storage device, such as DS3K, DS4K, or DS5K if complete access is lost to the storage device, then it may take greater than 5 minutes to fail I/O. This feature is for Active/Passive storage devices, which are running with the AIX Default A/P PCM. This includes DS3K, DS4K, and DS5K family of devices.”

The new feature was described as follows.

 

“Added feature which health checks controllers, when an enabled path becomes unavailable, due to transport problems.  By default this feature is DISABLED.  To enable this feature set the following ODM attributes for the active/passive storage device.  Enabling this feature, results in faster I/O failure times.                                                      

cntl_delay_time:                                                        
Is the amount of time in seconds the storage device's controller(s) will be health checked after a transport failure.  At the end of this period, if no paths are detected as good, then all pending and subsequent I/O to the device will be failed, until the device health checker detects a failed path has returned.                                             
                                                                        
cntl_hcheck_int:                                                        
The first controller health check will only be issued after a storage fabric transport failure had been detected. cntl_hcheck_int is the amount of time in seconds, which the next controller health check command will be issued. This value must be less than the cntl_delay_time  (unless set to "0", disabled).                                                  

If you wish to allow the storage device 30 seconds to come back on the fabric (after leaving the fabric),  then you can set cntl_delay_time=30 and cntl_hcheck_int=2.     

         
The device, /dev/hdisk#, must not be in use, when setting the ODM values (or the chdev "-P" option must be used,  which requires a reboot).                                               
                                                                        
CAUTION: There are cases where the storage device may  reboot both of the controllers and become inaccessible for a period of time. If the controller health check sequence is enabled, then this may result in an I/O failure.  It is recommended to to make sure you have an mirrored volume           
to failover to, if you are running with controller health check enabled (especially with under 60 second cntl_delay_time). “

                     

And as I suspected the issue was related to the type of storage we were using. It appears the I/O delay was attributed to the following attributes on the DS5020 hdisks on the VIOS:

 

cntl_delay_time 0  Controller Delay Time            True
cntl_hcheck_int 0  Controller Health Check Interval True

 

Based on the tech note, I attempted several tests with various values for both parameters e.g.

 

$ chdev -dev hdiskX -attr cntl_delay_time=30                        
$ chdev -dev hdiskX -attr cntl_hcheck_int=2  

 

After making the changes to the hdisks on all VIOS, I performed the same test i.e. disconnected BOTH fibre cables from the first VIOS and continued to write a file to a file system on the client LPARs. By modifying these values on all the DS5020 disks, on all the VIO servers, the I/O delay was reduced to seconds rather than five minutes!

 

The following attributes were used for the hdisks and adapters in the final configuration.

 

On the VIO servers:

 

$ lsdev -type disk | grep DS5020

hdisk3           Available   MPIO DS5020 Disk

hdisk4           Available   MPIO DS5020 Disk

hdisk5           Available   MPIO DS5020 Disk

hdisk6           Available   MPIO DS5020 Disk

hdisk7           Available   MPIO DS5020 Disk

hdisk8           Available   MPIO DS5020 Disk

hdisk9           Available   MPIO DS5020 Disk

hdisk10          Available   MPIO DS5020 Disk

hdisk11          Available   MPIO DS5020 Disk

hdisk12          Available   MPIO DS5020 Disk

hdisk13          Available   MPIO DS5020 Disk

hdisk14          Available   MPIO DS5020 Disk

hdisk15          Available   MPIO DS5020 Disk

hdisk16          Available   MPIO DS5020 Disk

hdisk17          Available   MPIO DS5020 Disk

hdisk18          Available   MPIO DS5020 Disk

hdisk19          Available   MPIO DS5020 Disk

hdisk20          Available   MPIO DS5020 Disk

hdisk21          Available   MPIO DS5020 Disk

hdisk22          Available   MPIO DS5020 Disk

hdisk23          Available   MPIO DS5020 Disk

hdisk24          Available   MPIO DS5020 Disk

hdisk25          Available   MPIO DS5020 Disk

hdisk26          Available   MPIO DS5020 Disk

hdisk27          Available   MPIO DS5020 Disk

hdisk28          Available   MPIO DS5020 Disk

hdisk29          Available   MPIO DS5020 Disk

hdisk30          Available   MPIO DS5020 Disk

hdisk31          Available   MPIO DS5020 Disk

hdisk32          Available   MPIO DS5020 Disk

 

 

$ lsdev -dev hdisk3 -attr

attribute       value                                                          description                      user_settable

 

PCM             PCM/friend/otherapdisk                                         Path Control Module              False

PR_key_value    none                                                           Persistant Reserve Key Value     True

algorithm       fail_over                                                      Algorithm                        True

autorecovery    no                                                             Path/Ownership Autorecovery      True

clr_q           no                                                             Device CLEARS its Queue on error True

cntl_delay_time 15                                                             Controller Delay Time            True

cntl_hcheck_int 2                                                              Controller Health Check Interval True

dist_err_pcnt   0                                                              Distributed Error Percentage     True

dist_tw_width   50                                                             Distributed Error Sample Time    True

hcheck_cmd      inquiry                                                        Health Check Command             True

hcheck_interval 60                                                             Health Check Interval            True

hcheck_mode     nonactive                                                      Health Check Mode                True

location                                                                       Location Label                   True

lun_id          0x11000000000000                                               Logical Unit Number ID           False

lun_reset_spt   yes                                                            LUN Reset Supported              True

max_retry_delay 60                                                             Maximum Quiesce Time             True

max_transfer    0x40000                                                        Maximum TRANSFER Size            True

node_name       0x20040080e5187564                                             FC Node Name                     False

pvid            00f6482f7869e92d0000000000000000                               Physical volume identifier       False

q_err           yes                                                            Use QERR bit                     True

q_type          simple                                                         Queuing TYPE                     True

queue_depth     24                                                             Queue DEPTH                      True

reassign_to     120                                                            REASSIGN time out value          True

reserve_policy  no_reserve                                                     Reserve Policy                   True

rw_timeout      30                                                             READ/WRITE time out value        True

scsi_id         0x11600                                                        SCSI ID                          False

start_timeout   60                                                             START unit time out value        True

unique_id       3E21360080E50001875DE000005224D2CD63B0F1814      FAStT03IBMfcp Unique device identifier         False

ww_name         0x20150080e5187564                                             FC World Wide Name               False

 

 

$ lsdev -dev fscsi0 -attr

attribute    value     description                           user_settable

 

attach       switch    How this adapter is CONNECTED         False

dyntrk       yes       Dynamic Tracking of FC Devices        True

fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True

scsi_id      0x11400   Adapter SCSI ID                       False

sw_fc_class  3         FC Class for Fabric                   True

 

 

$ r oem

oem_setup_env

# manage_disk_drivers

Device           Present Driver     Driver Options

2810XIV          AIX_AAPCM          AIX_AAPCM,AIX_non_MPIO

DS4100           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4200           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4300           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4500           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4700           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4800           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS3950           AIX_APPCM          AIX_APPCM

DS5020           AIX_APPCM          AIX_APPCM

DS5100/DS5300AIX_APPCM AIX_APPCM          AIX_APPCM

DS3500           AIX_APPCM          AIX_APPCM

Usage :

        manage_disk_drivers [-l]

        manage_disk_drivers -d device -o driver_option

        For entries with multiple model names use the first one listed.

        Ex. DS5100/DS5300 use DS5100.

        manage_disk_drivers –h

 

# mpio_get_config -Av

Frame id 0:

    Storage Subsystem worldwide name: 609e50018345de00004da7998

    Controller count: 2

    Partition count: 1

    Partition 0:

    Storage Subsystem Name = 'MyApp-DS5020'

        hdisk      LUN #   Ownership          User Label

        hdisk3        17   B (preferred)      LPAR1

        hdisk4        18   A (preferred)      LPAR1datavg

        hdisk5        19   A (preferred)      LPAR1appvg

        hdisk6        16   B (preferred)      LPAR2

        hdisk7        20   A (preferred)      LPAR2datavg

        hdisk8        21   B (preferred)      LPAR2data3vg

        hdisk9        15   A (preferred)      LPAR3

        hdisk10       14   B (preferred)      LPAR4

        hdisk11       22   B (preferred)      LPAR5

        hdisk12       23   A (preferred)      LPAR6

        hdisk13       13   B (preferred)      LPAR6datavg

        hdisk14       12   A (preferred)      LPAR7

        hdisk15       24   B (preferred)      LPAR7datavg

        hdisk16       25   A (preferred)      LPAR8

        hdisk17       11   B (preferred)      LPAR8datavg

        hdisk18       10   A (preferred)      LPAR8datavg

        hdisk19        9   B (preferred)      LPAR8datavg

        hdisk20        8   A (preferred)      LPAR8datavg

        hdisk21       26   A (preferred)      LPAR9

        hdisk22        7   A (preferred)      LPAR9datavg

        hdisk23        6   B (preferred)      LPAR9datavg

        hdisk24       27   B (preferred)      LPAR9appvg

        hdisk25       28   A (preferred)      LPAR9binvg

        hdisk26        5   A (preferred)      LPAR10

        hdisk27        4   B (preferred)      LPAR10datavg

        hdisk28       29   B (preferred)      LPAR10datavg

        hdisk29       30   A (preferred)      LPAR11

        hdisk30        3   A (preferred)      LPAR11datavg

        hdisk31       32   B (preferred)      LPAR12

        hdisk32       50   B (preferred)      LPAR12datavg

 

On the VIO clients:

 

- Changed the following attributes for all virtual SCSI disks to:

 

# lsattr -El hdisk0

PCM             PCM/friend/vscsi                 Path Control Module        False

algorithm       fail_over                        Algorithm                  True

hcheck_cmd      test_unit_rdy                    Health Check Command       True

hcheck_interval 60                               Health Check Interval      True

hcheck_mode     nonactive                        Health Check Mode          True

max_transfer    0x40000                          Maximum TRANSFER Size      True

pvid            00f6482f7869e92d0000000000000000 Physical volume identifier False

queue_depth     24                               Queue DEPTH                True

reserve_policy  no_reserve                       Reserve Policy             True

 

- Changed the following attributes for all virtual SCSI adapters to:

 

# lsattr -El vscsi0

vscsi_err_recov fast_fail N/A                       True

vscsi_path_to   30        Virtual SCSI Path Timeout True

 

Note: With regard to SDDPCM support for DS5020 with VIOS, I’ve been referring to the following IBM website:

 

http://www-01.ibm.com/support/docview.wss?uid=ssg1S4000201

 

The site has a link to the SDDPCM readme file for DS5020 storage (under SDDPCM Package for DS5000):

 

ftp://ftp.software.ibm.com/storage/subsystem/aix/2.5.2.0/sddpcm.readme.2.5.2.0.txt

 

The readme states:

 

Note: VIOS is not supported with SDDPCM on DS4000/DS5000/DS5020/DS3950 subsystem devices.

 

During our research we had this confirmed by IBM support that, at this time, SDDPCM is not supported
on VIOS with DS5000.

 

I hope this helps others that may be about to implement this type of storage with a VIOS.