Chris's AIX Blog

My team and I have recently been trying to stream line our AIX disaster recovery process. We’ve been looking for ways to reduce our overall recovery time. Several ideas were tossed around such as a) using a standby DR LPAR with AIX already installed and using rsync/scp to keep the Prod & DR LPARs in sync and b) using alt_disk_copy (with the –O flag for a device reset) to clone rootvg to an alternate disk which is then replicated to DR. These methods may work but are cumbersome to administer and (in the case of alt_disk_copy) require additional (permanent) resources on every production system. With over 120 production instances of AIX, the disk space requirements start to add up.

 

So far we’ve concluded that the best way to achieve our goal is by using SAN replicated rootvg volumes at our DR site.

 

Our current DR process relies on recovery of AIX systems from mksysb images from a NIM master. All our data (non-rootvg) LUNs are already replicated to our DR site. The aim was to change the process and ‘recover’ our AIX images using replicated rootvg LUNs. This will reduce our overall recovery time at DR (which is crucial if we are to meet the proposed recovery time objectives set by our business). Based on current IBM documentation we were relatively comfortable with the proposed approach. The following IBM developerWorks article (originally published in 2009 and updated in late 2010) describes “scenarios in which remapping, copying, and reuse of SAN disks is allowed and supported. More easily switch AIX environments from one system to another and help achieve higher availability and reduced down time. These scenarios also allow for fast deployment of new systems using cloning.”

 

The document focuses on fully virtualised environments that utilise shared processors and VIO servers. One area where this document is currently lacking in information is the use of NPIV and virtual fibre channel adapters in a DR scenario. We reached out to our contacts in the AIX development space and asked the following question:

 

“Hoping you can help us find some statements regarding support for a replicated rootvg environment using NPIV/Virtual Fibre Channel adapters?
 
The following IBM developerWorks article discusses VSCSI and we are looking for something similar for NPIV.
 
http://www.ibm.com/developerworks/aix/library/au-AIX_HA_SAN/index.html
 
My guess is that restrictions similar to those for physical FC adapters will apply here? But I'm hoping that given the adapters are virtual the limitations may be relaxed.
 
Are you aware of any statement regarding support (or not) for booting from another system using a disk subsystem image of rootvg replicated to another disk subsystem when using NPIV? And what, if any, additional requirements/restrictions may apply when using NPIV?”

We received the following responses:

“There are some additional considerations when using NPIV for booting from a replicated rootvg.  With NPIV the client partitions has virtual Fibre Channel adapter ports, but has physical access to the actual (physical) disk devices.  There may be an increased chance of needing to update the boot list via the Open Firmware SMS menu.  Since the clients have access to the actual disks, you have the possibility of running multipathing software besides AIX MPIO.   If you are using a multipathing software to manage the NPIV attached disks besides AIX MPIO, then you should contact the vendor that provided the software to check their support statement, 

Since one or more of the physical devices will change when booting from an NPIV replicated rootvg, it is recommend to set the ghostdev attribute.  The ghostdev attribute will trigger when it detects the AIX image is booting from either a different partition or server.  Ghostdev attribute should not trigger during LPM operations (Live Partition Mobility).  Once triggered, ghostdev will clear the customized ODM database.  This will cause detected devices to be discovered as new devices (with default settings), and avoid the issue with missing/stale device entries in ODM.  Since ghostdev does clear the entire customized ODM database, this will require you import your data (non-rootvg) volume groups again, and perform any (device) attribute customization. To set ghostdev, run "chdev -l sys0 -a ghostdev=1".  Ghostdev must be set before the rootvg is replicated. 

Similar with virtual devices, the client partition is booting an existing rootvg where the hardware may be different.  It's possible some applications have dependency on tracking the actual physical devices (instead of data on the disks).  For example PowerHA, may keep track of a disk for a cluster health checks.  If  you do have applications that have a dependency on tracking physical devices, then additional setup (of those applications) may be required after the first boot from the replicated rootvg. 

We do have multiple customers using NPIV for such scenarios.  I believe most of them worked with IBM Lab Based Services to assist with implementing such a configuration, and some of the customers required some custom scripts to further customize their system after booting from the replicated rootvg.  Those customers set the ghostdev attribute, and had custom scripts to import their data (non-rootvg) volume groups, and update PowerHA to point to the new health check disk. 

You should get support for such an NPIV setup with IBM as long as you follow the considerations listed in the WhitePaper.”

 

“Development has approved using NPIV to do this for one customer. Below are more detailed requirements for this DR strategy using NPIV. If a Disaster Recovery (DR) environment not using PowerHA Enterprise Edition  is used, then we believe the white paper located http://www.ibm.com/developerworks/aix/library/au-AIX_HA_SAN/index.html provides the guidelines in regards to setup, prerequisites, and as well as the limitations of such a DR deployment.  Deployments as detailed in the white paper are supported by IBM.  However note that such a deployment has many manually instituted responsibilities on the customer to setup and maintain such an environment. IBM expects that the customer carefully manage these manual steps without any mistakes.  The White paper does not currently cover using NPIV in such a DR scenario.

 

We have the following guidelines regarding the configuration, which includes NPIV as an option: All of the system configuration should be virtualized, with the possible exception of disk devices when using NPIV. If NPIV is used then AIX MPIO must be used as the multi-pathing solution.  If multi-pathing software is used besides AIX MPIO, then the vendor of that software must be contacted regarding a support statement. Install AIX (at least minimum required TLs/SPs for desired AIX version) and software stack (Middleware and applications) on primary systems, which is compatible with systems at both sites. Primary and secondary sites should be using systems with similar hardware, same microcode levels, and same VIOS levels.

 

Many manual steps are needed to setup the virtual and physical devices accurately on secondary site VIOS. If Virtual SCSI Disks are being used, then discover the unique identification on the primary site and map the disks to the corresponding replication disks. Map the same appropriately on VIOS on secondary site. Level of VIOS should support attribute to open the secondary devices passively. This setting needs to be setup correctly on the VIOS on secondary site. Operating environment should not have subnet dependencies. Manage the replication relationships accurately. Manually may need to switch the secondary disks to primary node. Raw disk usage may cause problems. Some middle ware products may bypass Operating system and use the disk directly. They might have their own restrictions for this environment.  (e.g. anything that is device location code or storage LUN unique ID dependant may have issues when the cloned image is restarted on the secondary system with replicated storage).

 

Set the "ghostdev" attribute using chdev command (must be done on the primary). This attribute can be set using the command "chdev -l sys0 –a ghostdev=1". The ghostdev attribute will delete customized ODM database on rootvg when AIX detects it has booted from a different LPAR or system. If the "ghostdev" attribute is not set, then the booting from the alternate site will result in devices in ODM showing up in "Defined" or "Missing" state.

 

After a failover to secondary site, may need to reset the boot device list for each LPAR before the boot of the LPAR using SMS menus of the firmware Note that this is not an exhaustive list of issues. Refer to the white paper and study the same as it applies to the environment. So as long as they are using MPIO, you're OK.  If not using MPIO and some OEM storage, then the storage vendor must also support it.”

While these responses indicate that this form of recovery is supported by IBM, we were still looking to IBM for clarity on the support position. It has been noted that other IBM customers have had mixed responses when contacting AIX support for feedback and assistance with this type of DR procedure. And it’s not hard to see why when you read statements like this from the “Supported Methods of Duplicating an AIX System” document:

 

“Unsupported Methods

1. Using a bitwise copy of a rootvg disk to another disk.

This bitwise copy can be a one-time snapshot copy such as flashcopy, from one disk to another, or a continuously-updating copy method, such as Metro Mirror.

While these methods will give you an exact duplicate of the installed AIX operating system, the copy of the OS may not be bootable. A typical scenario where this is tried is when one system is a production host and there is a desire to create a duplicate system at a disaster recovery site in a remote location.


2. Removing the rootvg disks from one system and inserting into another.

This also applies to re-zoning SAN disks that contain the rootvg so another host can see them and attempt to boot from them.


Why don't these methods work?

The reason for this is there are many objects in an AIX system that are unique to it; Hardware location codes, World-Wide Port Names, partition identifiers, and Vital Product Data (VPD) to name a few. Most of these objects or identifiers are stored in the ODM and used by AIX commands.

If a disk containing the AIX rootvg in one system is copied bit-for-bit (or removed), then inserted in another system, the firmware in the second system will describe an entirely different device tree than the AIX ODM expects to find, because it is operating on different hardware. Devices that were previously seen will show missing or removed, and usually the system will typically fail to boot with LED 554 (unknown boot disk).”

 

So, as a secondary objective, we have been working closely with our local IBM representatives to obtain some surety from IBM that our proposed DR strategy for AIX is fully supported by both the AIX development and support teams.

 

With that in mind I’ll provide an overview of our new DR approach and hope that it offers others insight to alternative method for recovery and also to assist IBM in further understanding what some of the “larger” AIX customers are looking for in terms of simplified AIX disaster recovery.

 

What follows is a detailed description of our IBM AIX, PowerVM/Power Systems environment, the proposed recovery steps and other items for consideration.

 

Our Environment:

 

·        -  AIX 5.3*, 6.1 & 7.1 - 5300-12-04-1119, 6100-06-05-1115 and 7100-01-04-1216.

·         - VIOS 2.2.1.3 – both production and DR.

·        -  VIOS physical FC adapters – both production and DR - 8Gb PCI Express Dual Port FC Adapter (5735) – Firmware level: 2.00X7.

·         - All client LPARs utilise Virtual FC adapters (NPIV) for disk storage.

·         - Production: POWER7 795 (9119-FHB). Firmware level: AH730_078.

·         - DR: POWER6 595 (9119-FHA). Firmware level: EH350_120.

·         - HDS VSP storage – at both production and DR sites.

·         - AIX MPIO only – HDS ODM driver is installed (devices.fcp.disk.Hitachi.array.mpio.rte) – HDLM is not installed.

·         - Production: Dual VIOS.

·         - DR: Single VIOS.

·         - Production: LUNs are mapped directly to client LPARs via NPIV, virtual FC adapter pass thru via dual VIOS. MPIO is in use i.e. one path per VIOS/Physical FC.

·         - DR: LUNs are mapped directly to client LPARs via NPIV, virtual FC adapter pass thru via a single VIOS. i.e. single path only.

·         - Please refer to the following table for a summary of the environment details.

 

Our Recovery Procedure:

 

1.       - Change the sys0 ghostdev attribute value to 1 on the source production AIX system. Set the "ghostdev" attribute using chdev command (must be done on the primary). This attribute can be set using the command "chdev -l sys0 –a ghostdev=1". The ghostdev attribute will delete customized ODM database on rootvg when AIX detects it has booted from a different LPAR or system.

2.       - Take note of the source systems rootvg hdisk PVID.

3.       - Select the source production rootvg LUN for replication on the HDS VSP.

4.       - Replicate the LUN from the production site to the DR HDS VSP.

5.       - In a DR test, suspend HDS replication from production to DR.

6.       - Assign the replicated LUN to the target LPAR on the DR POWER6 595 i.e. map the LUN to the WWPN of the virtual FC adapter on the DR LPAR.

7.       - Attempt to boot the DR LPAR using the replicated rootvg LUN. If necessary, enter SMS menu to update the boot list i.e. select the correct boot disk, check for the same PVID as the source host.

8.       - Once the LPAR has successfully booted, the AIX administrator would configure the necessary devices i.e. import data volume groups, configure network interfaces, etc. This may also be scripted for execution during the first boot process.

9.       - Please refer to the following diagrams for a visual representation of the proposed process.

 

Some Notes/Caveats:

 

The following is a list of items that we understand are possible limitations and issues with our new DR process.

 

·         - Booting from replicated rootvg disks may fail for several reasons, such as, a) there is unexpected corruption in the replicated LUN image due to rootvg not being quiesced during replication or b) there is a unidentified issue with the AIX system that is only apparent the next time the system is booted; this could be mis-configuration by the administrator or some other unforseen problem.

 

·         - In the event that an LPAR fails to boot via a replicated rootvg LUN, a backup method is available for recovery. Switching back to manual NIM mksysb restore provides a sufficient backup should the replicated rootvg be unusable.

 

·         - If the "ghostdev" attribute is not set, then booting from the DR site will result in devices in the ODM showing up in a "Defined" or "Missing" state.

 

·        -  Once a DR test is completed, the DR LPAR should be de-activated immediately so that SAN disk replication can be restarted between production and DR. Failure to perform this step may result in the DR LPAR failing as a result of file system corruption.

 

·         - At present we are using AIX MPIO only. There is discussion of using HDLM in the future. We will contact HDS for a support statement regarding booting from replicated rootvg LUNs with HDLM installed.

 

·         - The ghostdev attribute is not implemented in AIX 5.3. AIX 5.3 is no longer supported*.

 

So far all of our testing has been successful. We verified that we could replicate an SOE rootvg image of AIX 6.1 and 7.1 to DR and successfully boot an LPAR using the replicated disk. Based on these tests there doesn’t appear to be anything stopping us from using this method for DR purposes. The following table outlines the different versions of AIX we tested and the results.

 

Once the system was booted we needed to perform some post boot configuration tasks. These tasks were handled by two scripts that were called from /etc/inittab. On the source system we installed the new scripts (in /etc) and added new entries to the /etc/inittab file. These scripts only run if the systemid matches that of the DR systemid.  Note: Only partial contents of each script are shown below…but you get the idea.

 

# mkitab -i srcmstr "AIXDRimportvg:2:once:/etc/AIX_DRimportvg.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot."

# mkitab -i srcmstr "AIXDRconfig:2:once:/etc/AIX_DRconfig.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot."

# lsitab -a | grep DR

AIXDRconfig:2:once:/etc/AIX_DRconfig.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot.

AIXDRimportvg:2:once:/etc/AIX_DRimportvg.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot.

 

AIX_DRconfig.ksh:

 

#!/bin/ksh

#set -xv

#################################################################

#

# This script will configure the AIX environment for DR purposes.

# This script will only run if the systemid matches the systemid of the DR system.

# aixdr : / # lsattr -EOl sys0 -a systemid | grep -v systemid | awk -F, '{print $2}'

#

# 0211D11C1 = DR POWER System

#

#################################################################

 

MYNAME=$(basename $0)

DR_SYSTEMID=0211D11C1

LSATTR_SYSTEMID_DR=`lsattr -EOl sys0 -a systemid | grep -v systemid | awk -F, '{print $2}'`

 

if [ "$LSATTR_SYSTEMID_DR" = "$DR_SYSTEMID"  ]

then

   #Set the DR hostname, IP address, netmask and gateway.

   #mktcpip -h <hostname> -a <IP to use in DR> -m <network mask> -i en0 -g <gateway IP>

   #en0:10.1.5.99:255.255.252.0:10.1.1.10,255.255.255.0 10.2.2.10,255.255.255.0

 

   dr_hostname=`cat /usr/local/dr/dr_hostname.txt`

   dr_defgw=`cat /usr/local/dr/dr_defgw.txt`

   dr_intf=`cat /usr/local/dr/dr_en0.txt | awk -F: 'print $1}'`

   dr_ip=`cat /usr/local/dr/dr_en0.txt | awk -F: 'print $2}'`

   dr_netm=`cat /usr/local/dr/dr_en0.txt | awk -F: 'print $3}'`

 

   echo "$MYNAME: Setting DR hostname and IP address."

   mktcpip -h $dr_hostname -a $dr_ip -m $dr_netm -i $dr_intf -g $dr_defgw

 

   #Configure IP aliases

   #chdev -l en0 -a alias4=10.1.1.10,255.255.255.0

   #Remove IP aliases

   #chdev -l en0 -a delalias4=10.1.1.10,255.255.255.0

 

   for i in `cat /usr/local/dr/dr_en0.txt | awk -F: '{print $4}'`

   do

     chdev -l $dr_intf -a alias4=$i

   done

 

   for inet in `cat dr_inet.txt | grep -v en0`

   do

       for i in `cat /usr/local/dr/dr_$inet.txt | awk -F: '{print $4}'`

       do

         chdev -l $inet -a alias4=$i

       done

   done

 

   #Configure the AIX environment for DR.

   echo "$MYNAME: Configure the AIX environment for DR."

 

   echo

   #Delete the MDC DNS entries

   echo "$MYNAME: Configuring DNS for DR."

   namerslv -d -i 10.1.6.21

   namerslv -d -i 10.1.4.21

   namerslv -d -i 10.1.5.21

 

   #Add the DR DNS entry

   namerslv -a -i 10.1.7.38

 

   echo

   echo "$MYNAME: Configuring bootlist for DR."

   bootlist -m normal -o

   bootlist -m normal hdisk0

   bootlist -m normal -o

   echo

   echo "$MYNAME: Configuring /etc/inittab for DR."

   lsitab -a | grep nim

   chitab "nimclient:2:off:/usr/sbin/nimclient -S running > /dev/console 2>&1 #   inform nim we're running"

   lsitab -a | grep nim

   echo

 

else

   MYNAME=$(basename $0)

   echo "$MYNAME: The systemid $LSATTR_SYSTEMID_DR does not match the expected DR systemid of $DR_SYSTEMID."

   echo "$MYNAME: This script should only be executed at DR."

   echo "$MYNAME: If you are not booting the system at the DR site, then you can ignore this message."

   echo "$MYNAME: No changes have been perfomed. Script is exiting."

fi

 

 

 

 

 

The ghostdev attribute essentially provides us with a clean ODM and allows the system to discover new devices and build the ODM from scratch. If you attempt to boot from a replicated rootvg disk without first setting the ghostdev attribute, your system may fail to boot (hang at LED 554) because of a new device tree and/or missing devices.  You might be able to recover from this situation (without restoring from mksysb) by performing the steps outlined on pages 16-20 of the following document (thanks to Dominic Lancaster at IBM for the presentation).

 

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/resource/IBMSTGEducationSymposium_AX61_POWERVM_P2V.pdf

There are two new HMC (V7.7.3.0) commands that can force a client Virtual Fibre Channel adapter to log into a SAN. This should make the life of the AIX and SAN administrator easier, as they will no longer need to install AIX in order for the new VFC adapters to log into the SAN. Although there was an unsupported method* for doing this already (see links below). Nor will the SAN admins need to “blind” zone the WWPNs.

 

There was some indication of this in the latest VIOS FP readme:

 

https://www-304.ibm.com/support/docview.wss?rs=0&uid=isg400000693

 

·         Enabled SAN login from VIOS

 

And there was this:

 

IZ95569: ENABLE SAN LOGIN FROM VIOS FOR IMPROVED UABILITY

https://www-304.ibm.com/support/docview.wss?uid=isg1IZ95569

 

Problem summary

·    Customer will be able to instruct the VIOS to login to the SAN

·    for a given WWPN on a given virtual adapter to allow the

·    customer to see the WWPN in SAN management tools and easing

·    the task of configuring new NPIV client

·    partitions.

·          

Problem conclusion

·    add commands to VIOS manager interface to allow the HMC to

·    login, logout and query virtual adapters.

 

And finally this:

 

http://www-304.ibm.com/webapp/set2/sas/f/hmcl/power7/v7730.Readme.html#enhance7730

The new lsnportlogin and chnportlogin commands on the 7.730 HMC provide the ability to utilize the new function in the VIOS. From the Readme:

Added the chnportlogin and lsnportlogin commands.

 

1.  The chnportlogin command allows you to perform N Port login and logout operations for virtual fibre channel client adapters that are configured in a partition or a partition profile. Use this command to help you in zoning WWPNs on a Storage Area Network (SAN). A login operation activates all inactive WWPNs, including the second WWPN in the pair assigned to each virtual fibre channel client adapter. This feature is particularly useful for Logical Partition Migration. A logout operation deactivates all WWPNs not in use. A successful login of a virtual fibre channel adapter requires that the corresponding virtual fiber channel server adapter must exist and that it must be mapped.

 

2.  The lsnportlogin command also allows you to list WWPN login status information for virtual fibre channel client adapters configured in partitions or partition profiles.

 

I found a lot of this information on the IBM developerWorks PowerVM forum:

 

http://www.ibm.com/developerworks/forums/thread.jspa?threadID=388796

 

Thanks to http://twitter.com/Kristijan for pointing me in the direction of the PowerVM forum.

 

Here’s an example of using the lsnportlogin command on one of my systems:

 

hscroot@HMC1:~>  lsnportlogin -m  770-1-Rack_D191 --filter "profile_names=normal"

lpar_name=nim1,lpar_id=3,profile_name=normal,slot_num=32,wwpn=c050760405f0000c,wwpn_status=0

lpar_name=nim1,lpar_id=3,profile_name=normal,slot_num=32,wwpn=c050760405f0000d,wwpn_status=0

lpar_name=nim1,lpar_id=3,profile_name=normal,slot_num=33,wwpn=c050760405f0000e,wwpn_status=0

lpar_name=nim1,lpar_id=3,profile_name=normal,slot_num=33,wwpn=c050760405f0000f,wwpn_status=0

lpar_name=nim1,lpar_id=3,profile_name=normal,slot_num=30,wwpn=c050760405f00000,wwpn_status=0

lpar_name=nim1,lpar_id=3,profile_name=normal,slot_num=30,wwpn=c050760405f00001,wwpn_status=0

lpar_name=nim1,lpar_id=3,profile_name=normal,slot_num=31,wwpn=c050760405f00002,wwpn_status=0

lpar_name=nim1,lpar_id=3,profile_name=normal,slot_num=31,wwpn=c050760405f00003,wwpn_status=0

lpar_name=aix01adm,lpar_id=4,profile_name=normal,slot_num=32,wwpn=c050760405f00004,wwpn_status=0

lpar_name=aix01adm,lpar_id=4,profile_name=normal,slot_num=32,wwpn=c050760405f00005,wwpn_status=0

lpar_name=aix01adm,lpar_id=4,profile_name=normal,slot_num=33,wwpn=c050760405f00006,wwpn_status=0

lpar_name=aix01adm,lpar_id=4,profile_name=normal,slot_num=33,wwpn=c050760405f00007,wwpn_status=0

lpar_name=aix01adm,lpar_id=4,profile_name=normal,slot_num=30,wwpn=c050760405f00008,wwpn_status=0

lpar_name=aix01adm,lpar_id=4,profile_name=normal,slot_num=30,wwpn=c050760405f00009,wwpn_status=0

lpar_name=aix01adm,lpar_id=4,profile_name=normal,slot_num=31,wwpn=c050760405f0000a,wwpn_status=0

lpar_name=aix01adm,lpar_id=4,profile_name=normal,slot_num=31,wwpn=c050760405f0000b,wwpn_status=0

 

 

              Descriptions of selected command attributes:

                   wwpn_status

                        The WWPN status.  Possible values are:

                        0 - WWPN is not activated

                        1 - WWPN is activated

                        2 - WWPN status is unknown

 

Some related links:

 

lsnportlogin

http://publib.boulder.ibm.com/infocenter/powersys/v3r1m5/topic/p7edm/lsnportlogin.html

 

chnportlogin

http://publib.boulder.ibm.com/infocenter/powersys/v3r1m5/topic/p7edm/chnportlogin.html

 

*How to force a vfc-client device to log in to the SAN (The OLD way)!

https://www.ibm.com/developerworks/wikis/display/virtualization/vfclogin

 

How to Capture SAN Boot Debug for Virtual I/O Server and AIX on P6 Systems

https://www-304.ibm.com/support/docview.wss?uid=isg3T1012199

http://unixadmin.free.fr/?p=245

 

Disk path design for AIX including SAN zoning

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101914

 

 

The latest versions of AIX now allow largesend on virtual adapters to remain persistent across reboots. I can’t find any information for this attribute in the AIX 7.1 Information center yet.

 

The new attrribute is called mtu_bypass and can be changed with the chdev command. e.g

 

; AIX 7.1 TL1 SP2.

# oslevel -s

7100-01-02-1150

 

; LARGESEND is off.

# ifconfig en1

en1: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>

        inet 10.153.20.46 netmask 0xffff0000 broadcast 10.153.255.255

         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

 

; mtu_bypass is disabled.

# lsattr -El en1 -a mtu_bypass

mtu_bypass off Enable/Disable largesend for virtual Ethernet True

 

; mtu_bypass is now enabled.

# chdev -l en1 -a mtu_bypass=on

en1 changed

 

# lsattr -El en1 -a mtu_bypass

mtu_bypass on Enable/Disable largesend for virtual Ethernet True

 

; LARGESEND is on.

# ifconfig en1

en1: flags=1e080862,4c0<BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>

        inet 10.153.20.46 netmask 0xffff0000 broadcast 10.153.255.255

         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

 

 

I was collecting a list of WWPNs for a bunch of new AIX LPARs I was installing from scratch. There were two ways I could find the WWPN for a virtual Fibre Channel adapter on a new LPAR i.e. one that did not yet have an operating system installed.

 

I started by checking the LPAR properties from the HMC (as shown below).

 

 

 

To speed things up I moved to the HMC command line tool, lssyscfg, to display the WWPNs (as shown below).

 

hscroot@hmc1:~> lssyscfg -r prof -m 750-2 -F virtual_fc_adapters --filter lpar_names=lpar1

"""32/client/2/vio2/30/c0507603a2410000,c0507603a2410001/0"",""33/client/2/vio2/31/c0507603a2410002,c0507603a2410003/0"",""30/client/1/vio1/30/c0507603a2410004,c0507603a2410005/0"",""31/client/1/vio1/31/c0507603a2410006,c0507603a2410007/0""" """30/client/1/vio1/30/c0507603a2410004,c0507603a2410005/0"",""31/client/1/vio1/31/c0507603a2410006,c0507603a2410007/0"""

 

 

This was good. I now had a list of WWPNs for each LPAR.

 

# cat lpar1_wwpns.txt

c0507603a292007c

c0507603a292007e

c0507603a2920078

c0507603a292007a

 

I gave these WWPNS to my SAN administrator so that he could manually “zone in” the LPARs on the SAN switches and allocate storage to each. He then, half-jokingly said, “Gee, it would be nice if you could insert the colons into the WWPNS for me! J”. Of course, this got me thinking and after a few minutes of playing with sed, I came up with a way to do this quickly.

 

# cat lpar1_wwpns.txt | sed 's/../&:/g;s/:$//'

c0:50:76:03:a2:92:00:7c

c0:50:76:03:a2:92:00:7e

c0:50:76:03:a2:92:00:78

c0:50:76:03:a2:92:00:7a

 

Now my list of WWPNs was ready to be cut’n’paste by my SAN admin. He was happy.

The first time I ran the ’viostat  –adapter’ command I expected to find non-zero values for kbps, tps, etc, for each vfchost adapter. However, the values were always zero, no matter how much traffic traversed the adapters.                                 

                                                                       

$ viostat -adapter 1 10                                                 

...                                                                     

vadapter:                  Kbps      tps     bkread    bkwrtn          

vfchost0                    0.0      0.0        0.0       0.0          

...                                                                    

vadapter:                  Kbps      tps     bkread    bkwrtn          

vfchost1                    0.0      0.0        0.0       0.0          

                                                                       

I wondered if this was expected behaviour. Was the output supposed to report the amount of pass-thru traffic per vfchost? In 2011, I posed this question on the IBM developerWorks PowerVM forum regarding this observation. One of the replies stated:

 

"viostat does not give statistics for NPIV devices. The vfchost adapter is just a passthru, it doesn't know what the commands it gets are."    

                                                                        

https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14697753&#14697753

 

I appreciated someone taking the time to answer my question but I was still curious. I tested the same command again (in 2013) on a recent VIOS level (2.2.2.1), but I received the same result. It was time to source an official answer on this behaviour.

 

Here is the official response I received from IBM:

 

1.  FC adapter stats in viostat/iostat do not include NPIV.           

                                                                       

2.  viostat & iostat are an aggregate of all the stats from the underlying disks, which of course NPIV doesn't have.

 

There's really no way for the vfchost adapter to monitor I/O, since it doesn't know what the commands it gets are. He's just a passthru, passing the commands he gets from the client directly to the physical FC adapter.                         

                                                                        

3.  You can run fcstat on the VIOS but that has the same issues/limitations mentioned above.

 

Intent here was that customers would use tools on the client to monitor this sort of thing.

 

To summarize the comments from Development:                            

                                                                       

viostat does NOT give statistics for NPIV devices.”

 

This made sense but I wondered why the tool hadn’t been changed to exclude vfchost adapters from the output (to avoid customer confusion). There's obviously no valid reason to ever display any information for this type of adapter. I also understood that it was expected that I/O would be monitored at the client LPAR level. But I must say that an option for monitoring VFC I/O from a VIO server would be advantageous i.e. a single source view of all I/O activity for all VFC clients; particularly when there are several hundred partitions on a frame.  The response was:

 

“…the way the vfchost driver currently works is that it calls iostadd to register a dkstat structure, resulting in the adapter being listed when viostat is called.  This is misleading, however, since the vfchost driver does not actually track I/O.  The commands coming from the client partition are simply passed as-is to the physical FC adapter, and we don't know if a particular command is an I/O command or not.  The iostadd call is left over from porting the code from the vscsi driver, and Development agrees it should probably have been removed before shipping the code.                            

                                                                       

There has also been mention of a DCR #MR0413117456 (Title: FC adapter stats in viostat/iostat does not include NPIV) which you can follow-up with Marketing to register your interest/track progress if that is something you're interested in pursuing.”

It’s possible, with PowerVC, to manage existing AIX and Linux partitions in your environment. This may be useful for customers that have recently installed PowerVC in their environment and have existing AIX partitions. The question that is often asked is “Can I manage these existing partitions with PowerVC?”. These existing partitions were deployed outside of PowerVC, often long before PowerVC was even available as product from IBM. Fortunately the answer to this question is most likely yes!

 

Before you add an existing virtual machine to PowerVC, ensure that the following requirements are met:

 

http://www-01.ibm.com/support/knowledgecenter/SSXK2N_1.2.1/com.ibm.powervc.standard.help.doc/powervc_addingpre-existingvms_hmc.html

 

Here’s an example of managing an existing partition with PowerVC. Click on the Hosts icon from the PowerVC home screen then click on the “Manage Existing Virtual Machines” icon.

 

 

You are presented with a drop-down list of all the (Power Systems) Hosts known to PowerVC. Select the appropriate host where the existing partition currently resides.

 

 

Next you’ll be asked if you want to manage any supported virtual machines that are not currently being managed by PowerVC. “Supported” (with PowerVC Standard Edition v1.2.1) means any virtual machine that is either using virtual fibre channel adapters attached to supported SAN and storage devices (typically brocade switches and v7000) and/or virtual SCSI adapters attached to disk in a VIOS shared storage pool, that is currently managed by PowerVC.

 

Or you can select a specific virtual machine (or machines) to manage. I’ve selected one specific partition I wish to manage with PowerVC. It is an AIX partition that is connected to a shared storage pool (SSP). The SSP is already managed by PowerVC but the AIX partition is not.

 

 

 

A list of virtual machines (partitions) will be displayed next. Select the partition you’d like to manage with PowerVC and click on Manage.

 

 

Under the Virtual Machines view you’ll see the partition that you selected. It will appear in an Active and Pending state.

 

 

After a minute or two, the partition will change to an Active and OK state. The virtual machine is now under the management of PowerVC. You can now stop, start, restart, resize, migrate, attach volumes to, capture and/or delete the virtual machine through PowerVC.

 

 

Just as a point of interest, you can also access the Manage Existing option from the Virtual Machines view in PowerVC.

 

 

If you need to remove a partition (virtual machine) from PowerVC management (without deleting the partition) you can do so using the following procedure. From the Hosts view, select the desired Host and under Virtual Machines, select the partition you want to un-manage.

 

 

PowerVC will prompt you to confirm the removal of the virtual machine from PowerVC management. This does not delete the virtual machine; it will continue to run without disruption.

 

 

 

The virtual machine is removed from PowerVC management.

 

 

My colleague in Lab Services, Ben Swinney, recently configured the Virtual HMC on his Apple Macbook. He was kind enough to write up his experience, so that others could also run the vHMC on their Mac. Me included! 

You can find Ben’s instructions here:

https://www.ibm.com/developerworks/community/blogs/cgaix/resource/vHMC_running_on_Apple_hardware.pdf

Thanks Ben!