Chris's AIX Blog

I received an email from one of my customers recently that simply said:

“mate …on lpar30, for some reason the IBM.DRM and others are failing to start .. um .. any chance you could have a quick look at that ?”

So I asked, “OK, so this use to work right?”. To which I received a relatively confused reply, ”.....yep it did....actually no....it’s never worked....or has it???...I’m not sure...”.

Based on my experience, the most common issue that prevents DLPAR operations from working are network problems. Before diving into the deep end and trying to debug RSCT, it’s always best to start with the basics. For example, can you ping the HMC from the LPAR? Can you ping the LPAR from the HMC? If either of these tests fails, check the network configuration on both components before doing anything else.

On the HMC check the network settings first e.g.

·         Click on HMC Configuration and then Customize Network Settings.

– Verify the IP address, netmask, default gateway, network routes, DNS server are all set correctly.

– Check the LPAR communications box in HMC configuration screen for LAN adapter that is used for HMC-to-LPAR communication.

– By the way, unlike POWER4 systems, LPARs on POWER5 and POWER6 systems do not depend on host name resolution for DLPAR operations.

·         Check routing on the LPAR and the HMC.

– Use ping and the HMC’s Test Network Connectivity task to verify the LPAR and the HMC can communicate with each other.

If you check the network and you are happy that the LPAR and the HMC can communicate, then perhaps you need to re-initialise the RMC subsystems on the AIX LPAR. Run the following commands:

# /usr/sbin/rsct/bin/rmcctrl –z

# /usr/sbin/rsct/bin/rmcctrl –A

# /usr/sbin/rsct/bin/rmcctrl –p

Wait up to 5 minutes before trying DLPAR again. If DLPAR still doesn’t work i.e. the HMC is still reporting no values for DCaps, and the IBM.DRM subsystem still won’t start, try using the recfgct command.

WARNING: The recfgct command referenced above is *not* supported for use by customers without direct IBM support instructions. It erases all RSCT configuration info and makes it look like the node was just installed. This may be fine for DLPAR recycling, but if you have any other products dependent on RSCT on the partition in question, you will be *broken*. In particular, PowerHA 7 will crash, and Tivoli SAMP will have all its cluster info destroyed, partitioning it from the rest of the domain until it can be manually re-added (and it may also crash, depending on the presence of resources). If you find that DLPAR is not working, and all other network checks and even the RMC recycling (-z/-A/-p) does not work, it is strongly recommended that you use the ctsnap command to gather data and contact IBM support. (Capturing iptrace for a few minutes would not be a bad idea either. A complementary tcpdump on the HMC would also be good, but this may not be possible for most customers given HMC's access restrictions.) Then, if you wish to proceed with recfgct and find that it does resolve whatever the problem was, it would be equally wise to gather another ctsnap after the partition is once again connected to the HMC, to compare to the previous one.

CAUTION: Running the recfgct command on a node in a RSCT peer domain or in a Cluster Aware AIX (CAA) environment should NOT be done before taking other precautions first. This note is not designed to cover all CAA or other RSCT cluster considerations so if you have an application that is RSCT aware such as PowerHA, VIOS Storage Pools and several others do not proceed until you have contacted support. If you need to determine if your system is a member of a CAA cluster then please refer to the Reliable Scalable Cluster Technology document titled, "Troubleshooting the resource monitoring and control (RMC) subsystem". http://www-01.ibm.com/support/knowledgecenter/SGVKBA_3.1.5/com.ibm.rsct315.trouble/bl507_diagrmc.htm

hscroot@hmc1:~> lspartition -dlpar

.....

<#5> LPAR:<24*9117-MMB*10284FP, , 192.168.1.15>

       Active:<1>, OS:<AIX, 6.1, 6100-05-01-1016>, DCaps:<0x0>, CmdCaps:<0x0, 0x0>, PinnedMem:<768>

.....

# /usr/sbin/rsct/install/bin/recfgct

Wait 5 minutes. This should resolve your DLPAR issue. The IBM.DRM subsystem should now be active and there should be good (non-zero) values for DCaps:

# lssrc -g rsct_rm

Subsystem         Group            PID          Status

 IBM.DRM          rsct_rm          6881300      active

 IBM.CSMAgentRM   rsct_rm          7274530      active

 IBM.ServiceRM    rsct_rm          6029480      active

 IBM.AuditRM      rsct_rm          6357058      active

 IBM.ERRM         rsct_rm          4456566      active

 IBM.LPRM         rsct_rm          6946986      active

hscroot@hmc1:~> lspartition -dlpar

....

<#5> LPAR:<24*9117-MMB*10284FP, , 192.168.1.15>

       Active:<1>, OS:<AIX, 6.1, 6100-05-01-1016>, DCaps:<0xc5f>, CmdCaps:<0x1b, 0x1b>, PinnedMem:<994>

....

Only run the rmcctrl and recfgct commands if you believe something has become corrupt in the RMC configuration of the LPAR. The fastest way to fix a broken configuration or to clear out the RMC ACL files after cloning (via alt_disk migration) is to use the recfgct command.

These daemons should work “out of the box” and are not typically the cause of DLPAR issues. However, you can try stopping and starting the daemons when troubleshooting DLPAR issues.

The rmcctrl -z command just stops the daemons. The rmcctrl -A command ensures that the subsystem group (rsct) and the subsystem (ctrmc) objects are added to the SRC, and an appropriate entry added to the end of /etc/inittab and it starts the daemons.

The rmcctrl –p command enables the daemons for remote client connections i.e. from the HMC to the LPAR and vice versa.

If you are familiar with the System Resource Controller (SRC) you might be tempted to use stopsrc and startsrc commands to stop and start these daemons.

Do not do it; use the rmcctrl commands instead.

If /var is 100% full, use chfs to expand it. If there is no more space available, examine subdirectories and remove unnecessary files (for example, trace.*, core, and so forth). If /var is full, RMC subsystems may fail to function correctly.

The polling interval for the RMC daemons on the LPAR to check with the HMC daemons is 5-7 minutes; so you need to wait long enough for the daemons to start up and synchronize.

The Resource Monitoring and Control (RMC) daemons are part of the Reliable, Scalable Cluster Technology (RSCT) and are controlled by the System Resource Controller (SRC). These daemons run in all LPARs and communicate with equivalent RMC daemons running on the HMC. The daemons start automatically when the operating system starts and synchronize with the HMC RMC daemons.

The daemons in the LPARs and the daemons on the HMC must be able to communicate over the network for DLPAR operations to succeed. This is not the network connection between the managed system (FSP) and the HMC; it is the network connection between the operating system (AIX) in each LPAR and the HMC.

Note: Apart from rebooting, there is no way to stop and start the RMC daemons on the HMC.

The following links also contain some (out dated) information relating to DLPAR verification and troubleshooting. Even though it is quite old some of it is still relevant today and is good a place to start.

The most common reasons for failures with Dynamic Logical LPARing

http://www-01.ibm.com/support/docview.wss?rs=111&context=SWG10&q1=dcaps&uid=isg3T1010615&loc=en_US&cs=utf-8&lang=en

Dynamic LPAR tips and checklists for RMC authentication and authorization

http://www.ibm.com/developerworks/systems/articles/DLPARchecklist.html

Setting up the HMC/LPARs hostname and network (old but interesting)

http://computerstec.com/aixhowto/aixh18.htm

lsLPAR -dlpar, DCAPs values (old but still applies)

http://community.livejournal.com/eserver/18517.html

The previous link (above) provides some information relating to the values for DCaps and what they mean (also out dated):

0 - DR CPU capable  (can move CPUs)

1 - DR MEM capable  (can move memory)

2 - DR I/O capable  (can move I/O resources)

3 - DR PCI Bridge   (can move PCI bridges)

4 - DR Entitlement  (POWER 5 can change shared entitlement)

5 - Multiple DR CPU (AIX 5.3 can move 2+ CPUs at once)

0x3f = max, and 0xf is common for AIX 5.2

If you are interested in how HMC and LPAR authentication works with DLPAR, then read on. Otherwise, happy DLPARing!

HMC and LPAR authentication (RSCT authentication)

The diagram below outlines how the HMC and an LPAR authenticate with each other in order for DLPAR operations to work. RSCT authentication is used to ensure the HMC is communicating with the correct LPAR.

Authentication is the process of ensuring that another party is who it claims to be. Authorization is the process by which a cluster software component grants or denies resources based on certain criteria. The RSCT component that implements authorization is RMC. It uses access control list (ACL) files to control user access to resources.

The RMC component subsystem uses cluster security services to map the operating system user identifiers, specified in the ACL file, to network security identifiers to determine if the user has the correct permissions. This is performed by the identity mapping service, which uses information stored in the identity mapping files ctsec_map.global and ctsec_map.local.

The RSCT authorization process in detail:

1.    On the HMC: DMSRM pushes down the secret key and HMC IP address to NVRAM when it detects a new CEC; this process is repeated every five minutes. Each time an HMC is rebooted or DMSRM is restarted, a new key is used.

2.       On the AIX LPAR: CSMAgentRM, through RTAS (Run-time Abstraction Services), reads the key and HMC IP address out from NVRAM. It will then authenticate the HMC. This process is repeated every five minutes on a LPAR to detect a new HMCs and if the key has changed. An HMC with a new key is treated as a new HMC and will go though the authentication and authorization processes again.

3.       On the AIX LPAR: After authenticating the HMC, CSMAgentRM will contact the DMSRM on the HMC to create a ManagedNode resource in order to identify itself as a LPAR of this HMC. CSMAgentRM then creates a compatible ManagementServer resource on AIX. This can be displayed on AIX with the lssrsrc command. e.g.

root@aix6 / # lsrsrc "IBM.ManagementServer"

Resource Persistent Attributes for IBM.ManagementServer

resource 1:

        Name             = "192.168.1.244"

        Hostname         = "192.168.1.244"

        ManagerType      = "HMC"

        LocalHostname    = "10.153.3.133"

        ClusterTM        = "9078-160"

        ClusterSNum      = ""

        ActivePeerDomain = ""

        NodeNameList     = {"aix6"}

4.       On the AIX LPAR: After the creation of the ManagedNode and ManagementServer resources on the HMC and AIX respectively, CSMAgentRM grants HMC permission to access necessary resource classes on the LPAR. After granting the HMC permission, CSMAgentRM will change its ManagedNode, on the HMC, Status to 1. (It should be noted that without proper permission on AIX, the HMC would be able to establish a session with the LPAR but will not be able to query for OS information, DLPAR capabilities, or execute DLPAR commands afterwards.)

5.       On the HMC: After the ManagedNode Status is changed to 1, LparCmdRM establishes a session with the LPAR, queries for operating system information and DLPAR capabilities, notifies CIMOM about the DLPAR capabilities of the LPAR, and then waits for the DLPAR commands from users.

IBM recently announced some significant changes to their ‘AIX Release and Service Delivery’ strategy. Starting February 2011, the following changes will apply for both AIX 6 and AIX 7.

•          Three years of support for each Technology Level (TL).

•          A single Technology Level per AIX version, per year.

•          Service Packs will be released approximately 4 times per year per Technology Level.

Some of the benefits of the new strategy are:

• Longer support for Technology Levels.

–      More closely matches client deployment requirements.

·         Single Technology Level per year, per version, and about one Service Pack per quarter.

–      Fewer updates reduce administrative workload.

·         New hardware is supported on prior Technology Levels.

–      Easier integration of new hardware into existing environments.

You can review the official presentation and details here. I particularly like this FAQ from the presentation.

Q: Why doesn’t the new strategy include AIX V5.3?

A: AIX V5.3 is nearing end of life with the End of Marketing for AIX V5.3 set for April 2011

So if you are on AIX 5.3, start planning a migration to AIX 7.1 (or at least AIX 6.1), NOW! I’ve provided some further information from the presentation below.

Longer Support for each Technology Level

The 2011 Release Strategy extends the support for each Technology Level for up to three years from the introduction of the Technology Level. This means that clients with a Software Maintenance Agreement for the AIX OS will receive IBM defect support during that two year period without having to move up to the latest Technology Level update.

For example, a Technology Level introduced in the second half of 2010 will be supported though the second half of 2013. Since each Technology Level will be supported for three years and IBM plans to release one Technology Level per year, IBM eventually will support up to six Technology Levels for each AIX release.  Please note that due to minor variations in release dates, some Technology Levels will be supported for slightly more that three years and some will be supported for slightly less than three years. A three year service life for each Technology Level is an objective, not an absolute limit.

Fewer Technology Levels and Fewer Service Packs

Historically, new Technology Levels were the only way to deliver new hardware support to clients. Since 2007, we have supported most new hardware on previous Technology Levels via a service pack. Since this new approach to supporting new hardware has been well proven with the introduction of POWER7 in 2010, we decided to eliminate the “Spring” Technology Level which reduces the number of updates clients have to deal with. The number of Service Packs has been about 5 per year and we want to reduce this number to about once per quarter.

Between Technology Level releases, clients maintain their AIX operating systems by installing Service Packs (SP) or Interim Fixes for the entire support life of the Technology Level update. Service Packs are also used to provide support for newly released hardware.

The release dates and frequency of Service Packs are variable due to many factors. including new Technology Level releases, new hardware introduction and the need to deliver software fixes.

In the past IBM delivered new Service Packs about every eight to twelve weeks or about five to eight times per year per Technology Level. With the 2011 release strategy, IBM intends to reduce the number of Service Packs released per year, per AIX release to about four. As stated earlier, there are many factors that drive the release of a Service Pack, so releasing four Service Packs per year, per Technology Level represents a goal and objective, but not an absolute limit.

Note that new Technology Levels are almost always accompanied very closely by the first Service Pack for that new Technology Level. This first Service Pack includes fixes to problems that are discovered between the time the new Technology Level is released to manufacturing for the media to be replicated and the time the new Technology Level is actually available to clients.

Summary

The providers of all software, including platform software such as the AIX operating system, must balance the client’s need for stability against the need to enhance the software to provide new functionality. The changes introduced with the AIX OS Release and Service Strategy of 2011 balances those two conflicting goals to help provide our clients with significant improvements in the manageability of the AIX operating system.

The increased service life, better manageability and support for new hardware on older fix levels provided by the new strategy are in direct response to our client’s requirements.

As stated earlier, this new strategy represents goals and objectives of IBM and is subject to change. Our clients understand that substantial changes such as this new strategy are an indication of the commitment that IBM has to improving the market leading capabilities of the AIX operating system.

For more information and best practices please refer to:

https://www14.software.ibm.com/webapp/set2/sas/f/best/home.html

Recently I had the pleasure of configuring a couple of POWER7 720s for a customer. Each 720 was to host roughly 12 LPARs each. There were two VIO servers and a NIM server per 720.

Everything went along nicely and according to plan. In a few days we had both systems built. My steps for building the systems were almost identical to those described by Rob McNelly in a recent post.

All four VIO servers were running the latest VIOS code i.e. 2.2.0.10-FP-24 SP-01. All the client LPARs were running AIX 6.1 TL6 SP3. Each VIOS was configured with two FC paths to the SAN and the SAN storage device was an IBM DS5020.

Native AIX MPIO was in use on the VIOS and the AIX LPARs. I did not deploy SDDPCM on the VIOS as this is currently unsupported with the DS5020.

Once the LPAR builds were complete we performed a number of “integration tests”. These typically involve disconnecting network and SAN cables from each VIOS and observing how the VIOS and LPARs respond to and recover from these types of conditions.

One of the integration tests required that BOTH fibre cables be disconnected from the first VIOS and confirm that the client LPARs were not impacted i.e. that all I/O travelled via the second VIOS.

During the test we notice the following:

• I/O on the client LPAR would hang for approximately 5 minutes. Our test was simple enough, create a file in a file system. Eventually the I/O would continue after the 5 minute delay.

• The VIOS became sluggish and took some time to respond. The lspath command would take a very long time to return (or in some cases it would never return). During one test, the VIOS actually hung and had to be restarted (however we were not able to reproduce this issue again).

• When we reconnected both the cables, the paths would recover on the first VIOS after another 5 minutes. On the client however, it took roughly 20 minutes for the paths to recover. And yes, the hcheck_interval was set to 60 on all disks.

What's even more puzzling was that if we simply rebooted the first VIOS, everything worked as expected i.e. the client LPARs were not impacted, I/O continued as normal and when the first VIOS was back up, the paths on the client LPARs recovered quickly.

After doing some research, we discovered the following post on the IBM developerWorks AIX forum:

http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14472352&tstart=0

This post highlighted a number of things we needed to check and also confirmed several decisions we’d made during the design process, such as SDDPCM not being supported with DS5020 storage and VIOS (this was good as some people were starting to believe we should have installed SDDPCM to resolve this problem. I’d only be happy to do this if it was a supported combination and it’s not).

Finally we found the following IBM tech note that related directly to our issue.

 
IZ66020: ACTIVE/PASSIVE PCM CONTROLLER HCHECK SUPPORT
https://www-304.ibm.com/support/docview.wss?uid=isg1IZ66020

The following statement seemed to match our exact problem.

"For active/passive storage device, such as DS3K, DS4K, or DS5K if complete access is lost to the storage device, then it may take greater than 5 minutes to fail I/O. This feature is for Active/Passive storage devices, which are running with the AIX Default A/P PCM. This includes DS3K, DS4K, and DS5K family of devices.”

The new feature was described as follows.

“Added feature which health checks controllers, when an enabled path becomes unavailable, due to transport problems.  By default this feature is DISABLED.  To enable this feature set the following ODM attributes for the active/passive storage device.  Enabling this feature, results in faster I/O failure times.                                                      

cntl_delay_time:                                                        
Is the amount of time in seconds the storage device's controller(s) will be health checked after a transport failure.  At the end of this period, if no paths are detected as good, then all pending and subsequent I/O to the device will be failed, until the device health checker detects a failed path has returned.                                             
                                                                        
cntl_hcheck_int:                                                        
The first controller health check will only be issued after a storage fabric transport failure had been detected. cntl_hcheck_int is the amount of time in seconds, which the next controller health check command will be issued. This value must be less than the cntl_delay_time  (unless set to "0", disabled).                                                  

If you wish to allow the storage device 30 seconds to come back on the fabric (after leaving the fabric),  then you can set cntl_delay_time=30 and cntl_hcheck_int=2.     

         
The device, /dev/hdisk#, must not be in use, when setting the ODM values (or the chdev "-P" option must be used,  which requires a reboot).                                               
                                                                        
CAUTION: There are cases where the storage device may  reboot both of the controllers and become inaccessible for a period of time. If the controller health check sequence is enabled, then this may result in an I/O failure.  It is recommended to to make sure you have an mirrored volume           
to failover to, if you are running with controller health check enabled (especially with under 60 second cntl_delay_time). “

And as I suspected the issue was related to the type of storage we were using. It appears the I/O delay was attributed to the following attributes on the DS5020 hdisks on the VIOS:

cntl_delay_time 0  Controller Delay Time            True
cntl_hcheck_int 0  Controller Health Check Interval True

Based on the tech note, I attempted several tests with various values for both parameters e.g.

$ chdev -dev hdiskX -attr cntl_delay_time=30                        
$ chdev -dev hdiskX -attr cntl_hcheck_int=2  

After making the changes to the hdisks on all VIOS, I performed the same test i.e. disconnected BOTH fibre cables from the first VIOS and continued to write a file to a file system on the client LPARs. By modifying these values on all the DS5020 disks, on all the VIO servers, the I/O delay was reduced to seconds rather than five minutes!

The following attributes were used for the hdisks and adapters in the final configuration.

On the VIO servers:

$ lsdev -type disk | grep DS5020

hdisk3           Available   MPIO DS5020 Disk

hdisk4           Available   MPIO DS5020 Disk

hdisk5           Available   MPIO DS5020 Disk

hdisk6           Available   MPIO DS5020 Disk

hdisk7           Available   MPIO DS5020 Disk

hdisk8           Available   MPIO DS5020 Disk

hdisk9           Available   MPIO DS5020 Disk

hdisk10          Available   MPIO DS5020 Disk

hdisk11          Available   MPIO DS5020 Disk

hdisk12          Available   MPIO DS5020 Disk

hdisk13          Available   MPIO DS5020 Disk

hdisk14          Available   MPIO DS5020 Disk

hdisk15          Available   MPIO DS5020 Disk

hdisk16          Available   MPIO DS5020 Disk

hdisk17          Available   MPIO DS5020 Disk

hdisk18          Available   MPIO DS5020 Disk

hdisk19          Available   MPIO DS5020 Disk

hdisk20          Available   MPIO DS5020 Disk

hdisk21          Available   MPIO DS5020 Disk

hdisk22          Available   MPIO DS5020 Disk

hdisk23          Available   MPIO DS5020 Disk

hdisk24          Available   MPIO DS5020 Disk

hdisk25          Available   MPIO DS5020 Disk

hdisk26          Available   MPIO DS5020 Disk

hdisk27          Available   MPIO DS5020 Disk

hdisk28          Available   MPIO DS5020 Disk

hdisk29          Available   MPIO DS5020 Disk

hdisk30          Available   MPIO DS5020 Disk

hdisk31          Available   MPIO DS5020 Disk

hdisk32          Available   MPIO DS5020 Disk

$ lsdev -dev hdisk3 -attr

attribute       value                                                          description                      user_settable

PCM             PCM/friend/otherapdisk                                         Path Control Module              False

PR_key_value    none                                                           Persistant Reserve Key Value     True

algorithm       fail_over                                                      Algorithm                        True

autorecovery    no                                                             Path/Ownership Autorecovery      True

clr_q           no                                                             Device CLEARS its Queue on error True

cntl_delay_time 15                                                             Controller Delay Time            True

cntl_hcheck_int 2                                                              Controller Health Check Interval True

dist_err_pcnt   0                                                              Distributed Error Percentage     True

dist_tw_width   50                                                             Distributed Error Sample Time    True

hcheck_cmd      inquiry                                                        Health Check Command             True

hcheck_interval 60                                                             Health Check Interval            True

hcheck_mode     nonactive                                                      Health Check Mode                True

location                                                                       Location Label                   True

lun_id          0x11000000000000                                               Logical Unit Number ID           False

lun_reset_spt   yes                                                            LUN Reset Supported              True

max_retry_delay 60                                                             Maximum Quiesce Time             True

max_transfer    0x40000                                                        Maximum TRANSFER Size            True

node_name       0x20040080e5187564                                             FC Node Name                     False

pvid            00f6482f7869e92d0000000000000000                               Physical volume identifier       False

q_err           yes                                                            Use QERR bit                     True

q_type          simple                                                         Queuing TYPE                     True

queue_depth     24                                                             Queue DEPTH                      True

reassign_to     120                                                            REASSIGN time out value          True

reserve_policy  no_reserve                                                     Reserve Policy                   True

rw_timeout      30                                                             READ/WRITE time out value        True

scsi_id         0x11600                                                        SCSI ID                          False

start_timeout   60                                                             START unit time out value        True

unique_id       3E21360080E50001875DE000005224D2CD63B0F1814      FAStT03IBMfcp Unique device identifier         False

ww_name         0x20150080e5187564                                             FC World Wide Name               False

$ lsdev -dev fscsi0 -attr

attribute    value     description                           user_settable

attach       switch    How this adapter is CONNECTED         False

dyntrk       yes       Dynamic Tracking of FC Devices        True

fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True

scsi_id      0x11400   Adapter SCSI ID                       False

sw_fc_class  3         FC Class for Fabric                   True

$ r oem

oem_setup_env

# manage_disk_drivers

Device           Present Driver     Driver Options

2810XIV          AIX_AAPCM          AIX_AAPCM,AIX_non_MPIO

DS4100           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4200           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4300           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4500           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4700           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4800           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS3950           AIX_APPCM          AIX_APPCM

DS5020           AIX_APPCM          AIX_APPCM

DS5100/DS5300AIX_APPCM AIX_APPCM          AIX_APPCM

DS3500           AIX_APPCM          AIX_APPCM

Usage :

        manage_disk_drivers [-l]

        manage_disk_drivers -d device -o driver_option

        For entries with multiple model names use the first one listed.

        Ex. DS5100/DS5300 use DS5100.

        manage_disk_drivers –h

# mpio_get_config -Av

Frame id 0:

    Storage Subsystem worldwide name: 609e50018345de00004da7998

    Controller count: 2

    Partition count: 1

    Partition 0:

    Storage Subsystem Name = 'MyApp-DS5020'

        hdisk      LUN #   Ownership          User Label

        hdisk3        17   B (preferred)      LPAR1

        hdisk4        18   A (preferred)      LPAR1datavg

        hdisk5        19   A (preferred)      LPAR1appvg

        hdisk6        16   B (preferred)      LPAR2

        hdisk7        20   A (preferred)      LPAR2datavg

        hdisk8        21   B (preferred)      LPAR2data3vg

        hdisk9        15   A (preferred)      LPAR3

        hdisk10       14   B (preferred)      LPAR4

        hdisk11       22   B (preferred)      LPAR5

        hdisk12       23   A (preferred)      LPAR6

        hdisk13       13   B (preferred)      LPAR6datavg

        hdisk14       12   A (preferred)      LPAR7

        hdisk15       24   B (preferred)      LPAR7datavg

        hdisk16       25   A (preferred)      LPAR8

        hdisk17       11   B (preferred)      LPAR8datavg

        hdisk18       10   A (preferred)      LPAR8datavg

        hdisk19        9   B (preferred)      LPAR8datavg

        hdisk20        8   A (preferred)      LPAR8datavg

        hdisk21       26   A (preferred)      LPAR9

        hdisk22        7   A (preferred)      LPAR9datavg

        hdisk23        6   B (preferred)      LPAR9datavg

        hdisk24       27   B (preferred)      LPAR9appvg

        hdisk25       28   A (preferred)      LPAR9binvg

        hdisk26        5   A (preferred)      LPAR10

        hdisk27        4   B (preferred)      LPAR10datavg

        hdisk28       29   B (preferred)      LPAR10datavg

        hdisk29       30   A (preferred)      LPAR11

        hdisk30        3   A (preferred)      LPAR11datavg

        hdisk31       32   B (preferred)      LPAR12

        hdisk32       50   B (preferred)      LPAR12datavg

On the VIO clients:

- Changed the following attributes for all virtual SCSI disks to:

# lsattr -El hdisk0

PCM             PCM/friend/vscsi                 Path Control Module        False

algorithm       fail_over                        Algorithm                  True

hcheck_cmd      test_unit_rdy                    Health Check Command       True

hcheck_interval 60                               Health Check Interval      True

hcheck_mode     nonactive                        Health Check Mode          True

max_transfer    0x40000                          Maximum TRANSFER Size      True

pvid            00f6482f7869e92d0000000000000000 Physical volume identifier False

queue_depth     24                               Queue DEPTH                True

reserve_policy  no_reserve                       Reserve Policy             True

- Changed the following attributes for all virtual SCSI adapters to:

# lsattr -El vscsi0

vscsi_err_recov fast_fail N/A                       True

vscsi_path_to   30        Virtual SCSI Path Timeout True

Note: With regard to SDDPCM support for DS5020 with VIOS, I’ve been referring to the following IBM website:

http://www-01.ibm.com/support/docview.wss?uid=ssg1S4000201

The site has a link to the SDDPCM readme file for DS5020 storage (under SDDPCM Package for DS5000):

ftp://ftp.software.ibm.com/storage/subsystem/aix/2.5.2.0/sddpcm.readme.2.5.2.0.txt

The readme states:

Note: VIOS is not supported with SDDPCM on DS4000/DS5000/DS5020/DS3950 subsystem devices.

During our research we had this confirmed by IBM support that, at this time, SDDPCM is not supported
on VIOS with DS5000.

I hope this helps others that may be about to implement this type of storage with a VIOS.