Are you backing up your AIX systems over Virtual Ethernet adapters? Of course you are, who isn’t right? Are your backup server and clients on the same physical POWER system? You are most likely backing up over Virtual Ethernet to another AIX LPAR that is running your enterprise backup software, such as TSM or Legato Networker for example. And you probably have a dedicated private virtual network (and adapters) on both the clients and the server to handle the traffic for the nightly backups. The next question is, have you tuned your Virtual Ethernet adapters?

There are several tips available for tuning your Virtual Ethernet adapters for better performance on AIX. These tips include changing settings such as MTU size, TCP window sizes, enabling largesend, etc. I highly recommend the following blog posts from Anthony English and Nigel Griffiths on this subject:

https://www.ibm.com/developerworks/mydeveloperworks/blogs/AIXDownUnder/entry/turbo_charge_inter_lpar_communications_using_jumbo_frames1?lang=en

https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/10gbit_ethernet_bad_assumption_and_best_practice_part_137?lang=en

https://www.ibm.com/developerworks/mydeveloperworks/blogs/aixpert/entry/powervm_virtual_ethernet_speed_is_often_confused_with_vios_sea_ive_hea_speed?lang=en

OK, so you got everything humming along nicely, your backups are flying over the virtual network (across the POWER hypervisor) and everybody is happy. After a period of time, you notice that the backups have started to “slow down”. They are taking longer to finish. The overall throughput of a backup drops. Some backups start in the evening around 9pm and are still running the next morning at 7am! In some cases you need to kill the backups or even reboot the backup server LPAR for things to return to normal.

“What is going on!?” You cry.

Well, there are a number of reasons why this could be happening. For example, your shared processor pool may be overwhelmed during the backup window. As we know, Virtual Ethernet adapters require CPU to do their work. If the CPU pool is running low on available CPU resources, this could contribute to the problem. And of course there could be tuning issues with the Virtual Ethernet adapters or the AIX OS in general. Or there may be issues with other pieces of the infrastructure, like network and SAN switches, adapters, etc. Perhaps there’s an issue with the applications and/or databases on the AIX systems? They often have their own mechanisms/tools for backing up their data to your enterprise backup software. Is the backup server sized to cope with the load i.e. CPU, memory, disk layout and I/O, sufficient tape drives, disk storage pools, etc?

So assuming you’ve checked all of the above (and more), then perhaps you’ve hit a problem that I encountered recently. In my particular case, backups “over the hypervisor” were slowing down, without any discernible cause. Initially the backups would be “very fast” but after a month or so, things would start to slow down dramatically.

We noticed that there were very large (and increasing) values for “Packets Dropped”, “Hypervisor Send/Receive Failures” and “No Resource Errors” in the output from the netstat –v command.                              

ETHERNET STATISTICS (ent1) :                                           

Device Type: Virtual I/O Ethernet Adapter (l-lan)                      

Hardware Address: 41:ba:13:e7:25:0b                                    

Elapsed Time: 42 days 4 hours 3 minutes 34 seconds                    

Transmit Statistics:       Receive Statistics:                         

--------------------       -------------------                         

Packets: 5978589961        Packets: 26139832411                        

Bytes: 779465989202        Bytes: 711051516630458                      

Interrupts: 0              Interrupts: 6804561727                      

Transmit Errors: 0         Receive Errors: 0                           

Packets Dropped: 0         Packets Dropped: 86012309               

...                                                                                             

Max Collision Errors: 0    No Resource Errors: 46113807           

...                                                                       

Hypervisor Send Failures: 0                                            

  Receiver Failures: 0                                                  

  Send Errors: 0                                                       

Hypervisor Receive Failures: 46113807                             

After some discussion with IBM AIX support, we discovered that would should increase some of the buffer sizes for our Virtual Ethernet adapter (the entX device). This would alleviate the no resource issues we’d been experiencing. Looking at the output from the netstat -v command, we also noticed that the Medium, Large and Huge buffers had all reached their maximum values in the past.                                                           

...

Receive Information                                                    

  Receive Buffers                                                      

    Buffer Type              Tiny    Small   Medium    Large     Huge  

    Min Buffers               512      512      128       24       24  

    Max Buffers              2048     2048      256       64       64  

    Allocated                 513      535      148       28       64  

    Registered                512      510      127       24       13  

    History                                                            

      Max Allocated           576      951      256       64       64  

      Lowest Registered       502      502       64       12       11  

...

The advice from IBM support was to increase these buffers using the chdev command (they also advised that we should reboot for the changes to take effect): 

# chdev -l ent1 -a min_buf_medium=512 -a max_buf_medium=1024 –a min_buf_large=96 -a max_buf_large=256 -a min_buf_huge=96 –a max_buf_huge=128 -P             

# shutdown -Fr

Since implementing this tuning change (to the adapter on the backup server), we have not had a repeat of the problem. We will continue to monitor the performance and I’ll be sure to let everyone know if we have further issues.

I remember reading this interesting document about some challenges renaming AIX devices.

http://gibsonnet.net/aix/ibm/The_Fun_of_Oracle_RAC_ASM_and_Device_Names.doc

This poor guy had to jump through some hoops to rename some disk and network devices in a clustered AIX (Oracle RAC) environment.

This comment stayed with me:

“The easy way to get network adapters and disk names to match between systems would be to run something like

# chdev –l old_name –n new_name

Except, this command option does NOT exist.”

Since then I’ve always been on the “look out” for an easy way to rename devices in AIX.

In particular in large clustered environments where it is sometimes very important to have the same disk and network device names in sync across all nodes in a cluster. And besides, it’s a lot easier  to verify a cluster configuration if the hdisk names are all the same. Matching PVIDs works but it requires a lot more effort! For example, knowing that hdisk123 is the same device on all nodes makes life easier than scanning lspv output for a PVID like 00f6048868b4gead.  Of course you can script things to make this easier but it would be great if you didn’t need to do this and that there was a way to rename devices as needed, without resorting to unsupported methods.

Well, this is no longer an issue for AIX.

Starting with AIX 7.1, you can now easily rename devices. A new command called rendev was introduced to allow AIX administrators to rename devices as required.

From the man page:

The rendev command enables devices to be renamed. The device to be renamed, is specified with the -l flag, and the new desired name is specified with the -n flag.

The new desired name must not exceed 15 characters in length. If the name has already been used or is present in the /dev directory, the operation fails. If the name formed by appending  the new name after the character r is already used as a device name, or appears in the /dev directory, the operation fails.

 If the device is in the Available state, the rendev command must unconfigure the device before renaming it. This is similar to the operation performed by the rmdev -l Name command. If the unconfigure operation fails, the renaming will also fail. If the unconfigure succeeds, the rendev command will configure the device, after renaming it, to restore it to the Available state. The -u flag may be used to prevent the device from being configured again after it is renamed.

 Some devices may have special requirements on their names in order for other devices or applications to use them. Using the rendev command to rename such a device may result in the device being unusable. Note: To protect the configuration database, the rendev command cannot be interrupted once it has started. Trying to stop this command before completion, could result in a corrupted database.

This functionality was required to help simplify cluster device configuration with the new Cluster Aware AIX (CAA) feature in AIX 7.1.

I’ll be blogging about CAA in a future post.

Here are some examples of using the rendev command on AIX 7.1 (GA) system. In the first example I will rename hdisk3 to hdisk300. Note: hdisk3 is not in use (busy).

If the disk had been allocated to a volume group, I would have needed to varyoff the volume group first.

# lspv

hdisk0          00f61ab2f73e46e2                    rootvg          active

hdisk1          00f61ab20bf28ac6                    None

hdisk2          00f61ab2202f7c0b                    None

hdisk4          00f61ab20b97190d                    None

hdisk3          00f61ab2202f93ab                    None

# rendev -l hdisk3 -n hdisk300

# lspv

hdisk0          00f61ab2f73e46e2                    rootvg          active

hdisk1          00f61ab20bf28ac6                    None

hdisk2          00f61ab2202f7c0b                    None

hdisk4          00f61ab20b97190d                    None

hdisk300        00f61ab2202f93ab                    None

Easy!

Next, I’ll rename a virtual SCSI adapter. I renamed vscsi0 to vscsi2. Note: I placed the adapter, vscsi0, in a Defined state before renaming the device.

# rmdev -Rl vscsi0

# lsdev -Cc adapter

ent0   Available  Virtual I/O Ethernet Adapter (l-lan)

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi0 Defined    Virtual SCSI Client Adapter

vscsi1 Available  Virtual SCSI Client Adapter

# rendev -l vscsi0 -n vscsi2

# lsdev -Cc adapter

ent0   Available  Virtual I/O Ethernet Adapter (l-lan)

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi1 Available  Virtual SCSI Client Adapter

vscsi2 Defined    Virtual SCSI Client Adapter

Now I’ll rename a network adapter from ent0 to ent10. I bring down the interface before changing the device name

# lsdev -Cc adapter

ent0   Available  Virtual I/O Ethernet Adapter (l-lan)

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi1 Available  Virtual SCSI Client Adapter

vscsi2 Defined    Virtual SCSI Client Adapter

# ifconfig en0

en0: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>

        inet 10.1.20.19 netmask 0xffff0000 broadcast 10.153.255.255

         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

# ifconfig en0 down detach

# rendev -l ent0 -n ent10

# lsdev -Cc adapter

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

ent10  Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi1 Available  Virtual SCSI Client Adapter

vscsi2 Defined    Virtual SCSI Client Adapter

# rendev -l en0 -n en10

# chdev -l en10 -a state=up

en10 changed

# mkdev -l inet0

inet0 Available

# ifconfig en10

en10: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>

        inet 10.1.20.19 netmask 0xffff0000 broadcast 10.153.255.255

         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

If you want to be creative you can rename devices to anything you like (as long as it’s not more than 15 characters). For example I’ll rename vscsi2 to myvscsiadapter.

# rendev -l vscsi2 -n myvscsiadapter

# lsdev -Cc adapter

ent1           Available  Virtual I/O Ethernet Adapter (l-lan)

myadapter      Available  Virtual I/O Ethernet Adapter (l-lan)

myvscsiadapter Defined    Virtual SCSI Client Adapter

vsa0           Available  LPAR Virtual Serial Adapter

vscsi1         Available  Virtual SCSI Client Adapter

And in the last example I’ll demonstrate changing virtual SCSI adapter device names on a live system.

This is single disk system (hdisk0), with two vscsi adapters.

# lspv

hdisk0          00f6048868b4deee                    rootvg          active

# lspath

Enabled hdisk0 vscsi0

Enabled hdisk0 vscsi1

# lsdev -Cc adapter

ent0   Available  Virtual I/O Ethernet Adapter (l-lan)

ent1   Available  Virtual I/O Ethernet Adapter (l-lan)

vsa0   Available  LPAR Virtual Serial Adapter

vscsi0 Available  Virtual SCSI Client Adapter

vscsi1 Available  Virtual SCSI Client Adapter

We ensure the adapter is in a Defined state before renaming it. This will fail otherwise.

# rmdev -Rl vscsi1

vscsi1 Defined

# lsdev -Cc adapter | grep vscsi

vscsi0 Available  Virtual SCSI Client Adapter

vscsi1 Defined    Virtual SCSI Client Adapter

Now we rename the adapter vscsi1 to vscsi3.

# rendev -l vscsi1 -n vscsi3

# lsdev -Cc adapter | grep vscsi

vscsi0 Available  Virtual SCSI Client Adapter

vscsi3 Defined    Virtual SCSI Client Adapter

That was easy enough. Now I need to bring the adapter and path online with cfgmgr. The lspath output displays an additional path to vscsi3.

# lspath

Enabled hdisk0 vscsi0

Defined hdisk0 vscsi1

# cfgmgr

Method error (/etc/methods/cfgscsidisk -l hdisk0 ):

        0514-082 The requested function could only be performed for some

                 of the specified paths.

# lspath

Enabled hdisk0 vscsi0

Defined hdisk0 vscsi1

Enabled hdisk0 vscsi3

Now I need to remove the old path to vscsi1. The path to vscsi3 is now Enabled. The adapter, vscsi3, is in an Available state. All is good.

# rmpath -l hdisk0 -p vscsi1 -d

path Deleted

# lspath

Enabled hdisk0 vscsi0

Enabled hdisk0 vscsi3

# lsdev -Cc adapter | grep vscsi

vscsi0 Available  Virtual SCSI Client Adapter

vscsi3 Available  Virtual SCSI Client Adapter

The same steps need to be repeated for the vscsi0 adapter. This is renamed to vscsi2.

# rmdev -Rl vscsi0

vscsi0 Defined

# lsdev -Cc adapter | grep vscsi

vscsi0 Defined    Virtual SCSI Client Adapter

vscsi3 Available  Virtual SCSI Client Adapter

# rendev -l vscsi0 -n vscsi2

# lsdev -Cc adapter | grep vscsi

vscsi2 Defined    Virtual SCSI Client Adapter

vscsi3 Available  Virtual SCSI Client Adapter

# lspath

Defined hdisk0 vscsi0

Enabled hdisk0 vscsi3

# cfgmgr

Method error (/etc/methods/cfgscsidisk -l hdisk0 ):

        0514-082 The requested function could only be performed for some

                 of the specified paths.

# lspath

Defined hdisk0 vscsi0

Enabled hdisk0 vscsi2

Enabled hdisk0 vscsi3

# rmpath -l hdisk0 -p vscsi0 -d

path Deleted

# cfgmgr

# lspath

Enabled hdisk0 vscsi2

Enabled hdisk0 vscsi3

That’s it. Both adapters have been renamed while the system was in use. No downtime required.

# lsdev -Cc adapter | grep vscsi

vscsi2 Available  Virtual SCSI Client Adapter

vscsi3 Available  Virtual SCSI Client Adapter

# lspath

Enabled hdisk0 vscsi2

Enabled hdisk0 vscsi3

Please refer to the AIX 7.1 command reference for more information on this new command:

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.cmds/doc/aixcmds4/rendev.htm

And by the way, here’s a draft of what I’ve been working on for the last 5 weeks:

IBM AIX Version 7.1 Differences Guide:

http://www.redbooks.ibm.com/Redbooks.nsf/RedpieceAbstracts/sg247910.html?Open

Follow me on Twitter:

http://twitter.com/cgibbo

I received an email from one of my customers recently that simply said:

“mate …on lpar30, for some reason the IBM.DRM and others are failing to start .. um .. any chance you could have a quick look at that ?”

So I asked, “OK, so this use to work right?”. To which I received a relatively confused reply, ”.....yep it did....actually no....it’s never worked....or has it???...I’m not sure...”.

Based on my experience, the most common issue that prevents DLPAR operations from working are network problems. Before diving into the deep end and trying to debug RSCT, it’s always best to start with the basics. For example, can you ping the HMC from the LPAR? Can you ping the LPAR from the HMC? If either of these tests fails, check the network configuration on both components before doing anything else.

On the HMC check the network settings first e.g.

·         Click on HMC Configuration and then Customize Network Settings.

– Verify the IP address, netmask, default gateway, network routes, DNS server are all set correctly.

– Check the LPAR communications box in HMC configuration screen for LAN adapter that is used for HMC-to-LPAR communication.

– By the way, unlike POWER4 systems, LPARs on POWER5 and POWER6 systems do not depend on host name resolution for DLPAR operations.

·         Check routing on the LPAR and the HMC.

– Use ping and the HMC’s Test Network Connectivity task to verify the LPAR and the HMC can communicate with each other.

If you check the network and you are happy that the LPAR and the HMC can communicate, then perhaps you need to re-initialise the RMC subsystems on the AIX LPAR. Run the following commands:

# /usr/sbin/rsct/bin/rmcctrl –z

# /usr/sbin/rsct/bin/rmcctrl –A

# /usr/sbin/rsct/bin/rmcctrl –p

Wait up to 5 minutes before trying DLPAR again. If DLPAR still doesn’t work i.e. the HMC is still reporting no values for DCaps, and the IBM.DRM subsystem still won’t start, try using the recfgct command.

hscroot@hmc1:~> lspartition -dlpar

.....

<#5> LPAR:<24*9117-MMB*10284FP, , 192.168.1.15>

       Active:<1>, OS:<AIX, 6.1, 6100-05-01-1016>, DCaps:<0x0>, CmdCaps:<0x0, 0x0>, PinnedMem:<768>

.....

# /usr/sbin/rsct/install/bin/recfgct

Wait 5 minutes. This should resolve your DLPAR issue. The IBM.DRM subsystem should now be active and there should be good (non-zero) values for DCaps:

# lssrc -g rsct_rm

Subsystem         Group            PID          Status

 IBM.DRM          rsct_rm          6881300      active

 IBM.CSMAgentRM   rsct_rm          7274530      active

 IBM.ServiceRM    rsct_rm          6029480      active

 IBM.AuditRM      rsct_rm          6357058      active

 IBM.ERRM         rsct_rm          4456566      active

 IBM.LPRM         rsct_rm          6946986      active

hscroot@hmc1:~> lspartition -dlpar

....

<#5> LPAR:<24*9117-MMB*10284FP, , 192.168.1.15>

       Active:<1>, OS:<AIX, 6.1, 6100-05-01-1016>, DCaps:<0xc5f>, CmdCaps:<0x1b, 0x1b>, PinnedMem:<994>

....

Only run the rmcctrl and recfgct commands if you believe something has become corrupt in the RMC configuration of the LPAR. The fastest way to fix a broken configuration or to clear out the RMC ACL files after cloning (via alt_disk migration) is to use the recfgct command.

These daemons should work “out of the box” and are not typically the cause of DLPAR issues. However, you can try stopping and starting the daemons when troubleshooting DLPAR issues.

The rmcctrl -z command just stops the daemons. The rmcctrl -A command ensures that the subsystem group (rsct) and the subsystem (ctrmc) objects are added to the SRC, and an appropriate entry added to the end of /etc/inittab and it starts the daemons.

The rmcctrl –p command enables the daemons for remote client connections i.e. from the HMC to the LPAR and vice versa.

If you are familiar with the System Resource Controller (SRC) you might be tempted to use stopsrc and startsrc commands to stop and start these daemons.

Do not do it; use the rmcctrl commands instead.

If /var is 100% full, use chfs to expand it. If there is no more space available, examine subdirectories and remove unnecessary files (for example, trace.*, core, and so forth). If /var is full, RMC subsystems may fail to function correctly.

The polling interval for the RMC daemons on the LPAR to check with the HMC daemons is 5-7 minutes; so you need to wait long enough for the daemons to start up and synchronize.

The Resource Monitoring and Control (RMC) daemons are part of the Reliable, Scalable Cluster Technology (RSCT) and are controlled by the System Resource Controller (SRC). These daemons run in all LPARs and communicate with equivalent RMC daemons running on the HMC. The daemons start automatically when the operating system starts and synchronize with the HMC RMC daemons.

The daemons in the LPARs and the daemons on the HMC must be able to communicate over the network for DLPAR operations to succeed. This is not the network connection between the managed system (FSP) and the HMC; it is the network connection between the operating system (AIX) in each LPAR and the HMC.

Note: Apart from rebooting, there is no way to stop and start the RMC daemons on the HMC.

The following links also contain some (out dated) information relating to DLPAR verification and troubleshooting. Even though it is quite old some of it is still relevant today and is good a place to start.

The most common reasons for failures with Dynamic Logical LPARing

http://www-01.ibm.com/support/docview.wss?rs=111&context=SWG10&q1=dcaps&uid=isg3T1010615&loc=en_US&cs=utf-8&lang=en

Dynamic LPAR tips and checklists for RMC authentication and authorization

http://www.ibm.com/developerworks/systems/articles/DLPARchecklist.html

Setting up the HMC/LPARs hostname and network (old but interesting)

http://computerstec.com/aixhowto/aixh18.htm

lsLPAR -dlpar, DCAPs values (old but still applies)

http://community.livejournal.com/eserver/18517.html

The previous link (above) provides some information relating to the values for DCaps and what they mean (also out dated):

0 - DR CPU capable  (can move CPUs)

1 - DR MEM capable  (can move memory)

2 - DR I/O capable  (can move I/O resources)

3 - DR PCI Bridge   (can move PCI bridges)

4 - DR Entitlement  (POWER 5 can change shared entitlement)

5 - Multiple DR CPU (AIX 5.3 can move 2+ CPUs at once)

0x3f = max, and 0xf is common for AIX 5.2

If you are interested in how HMC and LPAR authentication works with DLPAR, then read on. Otherwise, happy DLPARing!

HMC and LPAR authentication (RSCT authentication)

The diagram below outlines how the HMC and an LPAR authenticate with each other in order for DLPAR operations to work. RSCT authentication is used to ensure the HMC is communicating with the correct LPAR.

Authentication is the process of ensuring that another party is who it claims to be. Authorization is the process by which a cluster software component grants or denies resources based on certain criteria. The RSCT component that implements authorization is RMC. It uses access control list (ACL) files to control user access to resources.

The RMC component subsystem uses cluster security services to map the operating system user identifiers, specified in the ACL file, to network security identifiers to determine if the user has the correct permissions. This is performed by the identity mapping service, which uses information stored in the identity mapping files ctsec_map.global and ctsec_map.local.

The RSCT authorization process in detail:

1.    On the HMC: DMSRM pushes down the secret key and HMC IP address to NVRAM when it detects a new CEC; this process is repeated every five minutes. Each time an HMC is rebooted or DMSRM is restarted, a new key is used.

2.       On the AIX LPAR: CSMAgentRM, through RTAS (Run-time Abstraction Services), reads the key and HMC IP address out from NVRAM. It will then authenticate the HMC. This process is repeated every five minutes on a LPAR to detect a new HMCs and if the key has changed. An HMC with a new key is treated as a new HMC and will go though the authentication and authorization processes again.

3.       On the AIX LPAR: After authenticating the HMC, CSMAgentRM will contact the DMSRM on the HMC to create a ManagedNode resource in order to identify itself as a LPAR of this HMC. CSMAgentRM then creates a compatible ManagementServer resource on AIX. This can be displayed on AIX with the lssrsrc command. e.g.

root@aix6 / # lsrsrc "IBM.ManagementServer"

Resource Persistent Attributes for IBM.ManagementServer

resource 1:

        Name             = "192.168.1.244"

        Hostname         = "192.168.1.244"

        ManagerType      = "HMC"

        LocalHostname    = "10.153.3.133"

        ClusterTM        = "9078-160"

        ClusterSNum      = ""

        ActivePeerDomain = ""

        NodeNameList     = {"aix6"}

4.       On the AIX LPAR: After the creation of the ManagedNode and ManagementServer resources on the HMC and AIX respectively, CSMAgentRM grants HMC permission to access necessary resource classes on the LPAR. After granting the HMC permission, CSMAgentRM will change its ManagedNode, on the HMC, Status to 1. (It should be noted that without proper permission on AIX, the HMC would be able to establish a session with the LPAR but will not be able to query for OS information, DLPAR capabilities, or execute DLPAR commands afterwards.)

5.       On the HMC: After the ManagedNode Status is changed to 1, LparCmdRM establishes a session with the LPAR, queries for operating system information and DLPAR capabilities, notifies CIMOM about the DLPAR capabilities of the LPAR, and then waits for the DLPAR commands from users.

My team and I have recently been trying to stream line our AIX disaster recovery process. We’ve been looking for ways to reduce our overall recovery time. Several ideas were tossed around such as a) using a standby DR LPAR with AIX already installed and using rsync/scp to keep the Prod & DR LPARs in sync and b) using alt_disk_copy (with the –O flag for a device reset) to clone rootvg to an alternate disk which is then replicated to DR. These methods may work but are cumbersome to administer and (in the case of alt_disk_copy) require additional (permanent) resources on every production system. With over 120 production instances of AIX, the disk space requirements start to add up.

So far we’ve concluded that the best way to achieve our goal is by using SAN replicated rootvg volumes at our DR site.

Our current DR process relies on recovery of AIX systems from mksysb images from a NIM master. All our data (non-rootvg) LUNs are already replicated to our DR site. The aim was to change the process and ‘recover’ our AIX images using replicated rootvg LUNs. This will reduce our overall recovery time at DR (which is crucial if we are to meet the proposed recovery time objectives set by our business). Based on current IBM documentation we were relatively comfortable with the proposed approach. The following IBM developerWorks article (originally published in 2009 and updated in late 2010) describes “scenarios in which remapping, copying, and reuse of SAN disks is allowed and supported. More easily switch AIX environments from one system to another and help achieve higher availability and reduced down time. These scenarios also allow for fast deployment of new systems using cloning.”

The document focuses on fully virtualised environments that utilise shared processors and VIO servers. One area where this document is currently lacking in information is the use of NPIV and virtual fibre channel adapters in a DR scenario. We reached out to our contacts in the AIX development space and asked the following question:

“Hoping you can help us find some statements regarding support for a replicated rootvg environment using NPIV/Virtual Fibre Channel adapters?
 
The following IBM developerWorks article discusses VSCSI and we are looking for something similar for NPIV.
 
http://www.ibm.com/developerworks/aix/library/au-AIX_HA_SAN/index.html
 
My guess is that restrictions similar to those for physical FC adapters will apply here? But I'm hoping that given the adapters are virtual the limitations may be relaxed.
 
Are you aware of any statement regarding support (or not) for booting from another system using a disk subsystem image of rootvg replicated to another disk subsystem when using NPIV? And what, if any, additional requirements/restrictions may apply when using NPIV?”

We received the following responses:

“There are some additional considerations when using NPIV for booting from a replicated rootvg.  With NPIV the client partitions has virtual Fibre Channel adapter ports, but has physical access to the actual (physical) disk devices.  There may be an increased chance of needing to update the boot list via the Open Firmware SMS menu.  Since the clients have access to the actual disks, you have the possibility of running multipathing software besides AIX MPIO.   If you are using a multipathing software to manage the NPIV attached disks besides AIX MPIO, then you should contact the vendor that provided the software to check their support statement, 

Since one or more of the physical devices will change when booting from an NPIV replicated rootvg, it is recommend to set the ghostdev attribute.  The ghostdev attribute will trigger when it detects the AIX image is booting from either a different partition or server.  Ghostdev attribute should not trigger during LPM operations (Live Partition Mobility).  Once triggered, ghostdev will clear the customized ODM database.  This will cause detected devices to be discovered as new devices (with default settings), and avoid the issue with missing/stale device entries in ODM.  Since ghostdev does clear the entire customized ODM database, this will require you import your data (non-rootvg) volume groups again, and perform any (device) attribute customization. To set ghostdev, run "chdev -l sys0 -a ghostdev=1".  Ghostdev must be set before the rootvg is replicated. 

Similar with virtual devices, the client partition is booting an existing rootvg where the hardware may be different.  It's possible some applications have dependency on tracking the actual physical devices (instead of data on the disks).  For example PowerHA, may keep track of a disk for a cluster health checks.  If  you do have applications that have a dependency on tracking physical devices, then additional setup (of those applications) may be required after the first boot from the replicated rootvg. 

We do have multiple customers using NPIV for such scenarios.  I believe most of them worked with IBM Lab Based Services to assist with implementing such a configuration, and some of the customers required some custom scripts to further customize their system after booting from the replicated rootvg.  Those customers set the ghostdev attribute, and had custom scripts to import their data (non-rootvg) volume groups, and update PowerHA to point to the new health check disk. 

You should get support for such an NPIV setup with IBM as long as you follow the considerations listed in the WhitePaper.”

“Development has approved using NPIV to do this for one customer. Below are more detailed requirements for this DR strategy using NPIV. If a Disaster Recovery (DR) environment not using PowerHA Enterprise Edition  is used, then we believe the white paper located http://www.ibm.com/developerworks/aix/library/au-AIX_HA_SAN/index.html provides the guidelines in regards to setup, prerequisites, and as well as the limitations of such a DR deployment.  Deployments as detailed in the white paper are supported by IBM.  However note that such a deployment has many manually instituted responsibilities on the customer to setup and maintain such an environment. IBM expects that the customer carefully manage these manual steps without any mistakes.  The White paper does not currently cover using NPIV in such a DR scenario.

We have the following guidelines regarding the configuration, which includes NPIV as an option: All of the system configuration should be virtualized, with the possible exception of disk devices when using NPIV. If NPIV is used then AIX MPIO must be used as the multi-pathing solution.  If multi-pathing software is used besides AIX MPIO, then the vendor of that software must be contacted regarding a support statement. Install AIX (at least minimum required TLs/SPs for desired AIX version) and software stack (Middleware and applications) on primary systems, which is compatible with systems at both sites. Primary and secondary sites should be using systems with similar hardware, same microcode levels, and same VIOS levels.

Many manual steps are needed to setup the virtual and physical devices accurately on secondary site VIOS. If Virtual SCSI Disks are being used, then discover the unique identification on the primary site and map the disks to the corresponding replication disks. Map the same appropriately on VIOS on secondary site. Level of VIOS should support attribute to open the secondary devices passively. This setting needs to be setup correctly on the VIOS on secondary site. Operating environment should not have subnet dependencies. Manage the replication relationships accurately. Manually may need to switch the secondary disks to primary node. Raw disk usage may cause problems. Some middle ware products may bypass Operating system and use the disk directly. They might have their own restrictions for this environment.  (e.g. anything that is device location code or storage LUN unique ID dependant may have issues when the cloned image is restarted on the secondary system with replicated storage).

Set the "ghostdev" attribute using chdev command (must be done on the primary). This attribute can be set using the command "chdev -l sys0 –a ghostdev=1". The ghostdev attribute will delete customized ODM database on rootvg when AIX detects it has booted from a different LPAR or system. If the "ghostdev" attribute is not set, then the booting from the alternate site will result in devices in ODM showing up in "Defined" or "Missing" state.

After a failover to secondary site, may need to reset the boot device list for each LPAR before the boot of the LPAR using SMS menus of the firmware Note that this is not an exhaustive list of issues. Refer to the white paper and study the same as it applies to the environment. So as long as they are using MPIO, you're OK.  If not using MPIO and some OEM storage, then the storage vendor must also support it.”

While these responses indicate that this form of recovery is supported by IBM, we were still looking to IBM for clarity on the support position. It has been noted that other IBM customers have had mixed responses when contacting AIX support for feedback and assistance with this type of DR procedure. And it’s not hard to see why when you read statements like this from the “Supported Methods of Duplicating an AIX System” document:

“Unsupported Methods

1. Using a bitwise copy of a rootvg disk to another disk.

This bitwise copy can be a one-time snapshot copy such as flashcopy, from one disk to another, or a continuously-updating copy method, such as Metro Mirror.

While these methods will give you an exact duplicate of the installed AIX operating system, the copy of the OS may not be bootable. A typical scenario where this is tried is when one system is a production host and there is a desire to create a duplicate system at a disaster recovery site in a remote location.


2. Removing the rootvg disks from one system and inserting into another.

This also applies to re-zoning SAN disks that contain the rootvg so another host can see them and attempt to boot from them.


Why don't these methods work?

The reason for this is there are many objects in an AIX system that are unique to it; Hardware location codes, World-Wide Port Names, partition identifiers, and Vital Product Data (VPD) to name a few. Most of these objects or identifiers are stored in the ODM and used by AIX commands.

If a disk containing the AIX rootvg in one system is copied bit-for-bit (or removed), then inserted in another system, the firmware in the second system will describe an entirely different device tree than the AIX ODM expects to find, because it is operating on different hardware. Devices that were previously seen will show missing or removed, and usually the system will typically fail to boot with LED 554 (unknown boot disk).”

So, as a secondary objective, we have been working closely with our local IBM representatives to obtain some surety from IBM that our proposed DR strategy for AIX is fully supported by both the AIX development and support teams.

With that in mind I’ll provide an overview of our new DR approach and hope that it offers others insight to alternative method for recovery and also to assist IBM in further understanding what some of the “larger” AIX customers are looking for in terms of simplified AIX disaster recovery.

What follows is a detailed description of our IBM AIX, PowerVM/Power Systems environment, the proposed recovery steps and other items for consideration.

Our Environment:

·        -  AIX 5.3*, 6.1 & 7.1 - 5300-12-04-1119, 6100-06-05-1115 and 7100-01-04-1216.

·         - VIOS 2.2.1.3 – both production and DR.

·        -  VIOS physical FC adapters – both production and DR - 8Gb PCI Express Dual Port FC Adapter (5735) – Firmware level: 2.00X7.

·         - All client LPARs utilise Virtual FC adapters (NPIV) for disk storage.

·         - Production: POWER7 795 (9119-FHB). Firmware level: AH730_078.

·         - DR: POWER6 595 (9119-FHA). Firmware level: EH350_120.

·         - HDS VSP storage – at both production and DR sites.

·         - AIX MPIO only – HDS ODM driver is installed (devices.fcp.disk.Hitachi.array.mpio.rte) – HDLM is not installed.

·         - Production: Dual VIOS.

·         - DR: Single VIOS.

·         - Production: LUNs are mapped directly to client LPARs via NPIV, virtual FC adapter pass thru via dual VIOS. MPIO is in use i.e. one path per VIOS/Physical FC.

·         - DR: LUNs are mapped directly to client LPARs via NPIV, virtual FC adapter pass thru via a single VIOS. i.e. single path only.

·         - Please refer to the following table for a summary of the environment details.

Our Recovery Procedure:

1.       - Change the sys0 ghostdev attribute value to 1 on the source production AIX system. Set the "ghostdev" attribute using chdev command (must be done on the primary). This attribute can be set using the command "chdev -l sys0 –a ghostdev=1". The ghostdev attribute will delete customized ODM database on rootvg when AIX detects it has booted from a different LPAR or system.

2.       - Take note of the source systems rootvg hdisk PVID.

3.       - Select the source production rootvg LUN for replication on the HDS VSP.

4.       - Replicate the LUN from the production site to the DR HDS VSP.

5.       - In a DR test, suspend HDS replication from production to DR.

6.       - Assign the replicated LUN to the target LPAR on the DR POWER6 595 i.e. map the LUN to the WWPN of the virtual FC adapter on the DR LPAR.

7.       - Attempt to boot the DR LPAR using the replicated rootvg LUN. If necessary, enter SMS menu to update the boot list i.e. select the correct boot disk, check for the same PVID as the source host.

8.       - Once the LPAR has successfully booted, the AIX administrator would configure the necessary devices i.e. import data volume groups, configure network interfaces, etc. This may also be scripted for execution during the first boot process.

9.       - Please refer to the following diagrams for a visual representation of the proposed process.

 

Some Notes/Caveats:

The following is a list of items that we understand are possible limitations and issues with our new DR process.

·         - Booting from replicated rootvg disks may fail for several reasons, such as, a) there is unexpected corruption in the replicated LUN image due to rootvg not being quiesced during replication or b) there is a unidentified issue with the AIX system that is only apparent the next time the system is booted; this could be mis-configuration by the administrator or some other unforseen problem.

·         - In the event that an LPAR fails to boot via a replicated rootvg LUN, a backup method is available for recovery. Switching back to manual NIM mksysb restore provides a sufficient backup should the replicated rootvg be unusable.

·         - If the "ghostdev" attribute is not set, then booting from the DR site will result in devices in the ODM showing up in a "Defined" or "Missing" state.

·        -  Once a DR test is completed, the DR LPAR should be de-activated immediately so that SAN disk replication can be restarted between production and DR. Failure to perform this step may result in the DR LPAR failing as a result of file system corruption.

·         - At present we are using AIX MPIO only. There is discussion of using HDLM in the future. We will contact HDS for a support statement regarding booting from replicated rootvg LUNs with HDLM installed.

·         - The ghostdev attribute is not implemented in AIX 5.3. AIX 5.3 is no longer supported*.

So far all of our testing has been successful. We verified that we could replicate an SOE rootvg image of AIX 6.1 and 7.1 to DR and successfully boot an LPAR using the replicated disk. Based on these tests there doesn’t appear to be anything stopping us from using this method for DR purposes. The following table outlines the different versions of AIX we tested and the results.

Once the system was booted we needed to perform some post boot configuration tasks. These tasks were handled by two scripts that were called from /etc/inittab. On the source system we installed the new scripts (in /etc) and added new entries to the /etc/inittab file. These scripts only run if the systemid matches that of the DR systemid.  Note: Only partial contents of each script are shown below…but you get the idea.

# mkitab -i srcmstr "AIXDRimportvg:2:once:/etc/AIX_DRimportvg.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot."

# mkitab -i srcmstr "AIXDRconfig:2:once:/etc/AIX_DRconfig.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot."

# lsitab -a | grep DR

AIXDRconfig:2:once:/etc/AIX_DRconfig.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot.

AIXDRimportvg:2:once:/etc/AIX_DRimportvg.ksh > /dev/console 2>&1 #Reconfigure the system for DR on boot.

AIX_DRconfig.ksh:

#!/bin/ksh

#set -xv

#################################################################

#

# This script will configure the AIX environment for DR purposes.

# This script will only run if the systemid matches the systemid of the DR system.

# aixdr : / # lsattr -EOl sys0 -a systemid | grep -v systemid | awk -F, '{print $2}'

#

# 0211D11C1 = DR POWER System

#

#################################################################

MYNAME=$(basename $0)

DR_SYSTEMID=0211D11C1

LSATTR_SYSTEMID_DR=`lsattr -EOl sys0 -a systemid | grep -v systemid | awk -F, '{print $2}'`

if [ "$LSATTR_SYSTEMID_DR" = "$DR_SYSTEMID"  ]

then

   #Set the DR hostname, IP address, netmask and gateway.

   #mktcpip -h <hostname> -a <IP to use in DR> -m <network mask> -i en0 -g <gateway IP>

   #en0:10.1.5.99:255.255.252.0:10.1.1.10,255.255.255.0 10.2.2.10,255.255.255.0

   dr_hostname=`cat /usr/local/dr/dr_hostname.txt`

   dr_defgw=`cat /usr/local/dr/dr_defgw.txt`

   dr_intf=`cat /usr/local/dr/dr_en0.txt | awk -F: 'print $1}'`

   dr_ip=`cat /usr/local/dr/dr_en0.txt | awk -F: 'print $2}'`

   dr_netm=`cat /usr/local/dr/dr_en0.txt | awk -F: 'print $3}'`

   echo "$MYNAME: Setting DR hostname and IP address."

   mktcpip -h $dr_hostname -a $dr_ip -m $dr_netm -i $dr_intf -g $dr_defgw

   #Configure IP aliases

   #chdev -l en0 -a alias4=10.1.1.10,255.255.255.0

   #Remove IP aliases

   #chdev -l en0 -a delalias4=10.1.1.10,255.255.255.0

   for i in `cat /usr/local/dr/dr_en0.txt | awk -F: '{print $4}'`

   do

     chdev -l $dr_intf -a alias4=$i

   done

   for inet in `cat dr_inet.txt | grep -v en0`

   do

       for i in `cat /usr/local/dr/dr_$inet.txt | awk -F: '{print $4}'`

       do

         chdev -l $inet -a alias4=$i

       done

   done

   #Configure the AIX environment for DR.

   echo "$MYNAME: Configure the AIX environment for DR."

   echo

   #Delete the MDC DNS entries

   echo "$MYNAME: Configuring DNS for DR."

   namerslv -d -i 10.1.6.21

   namerslv -d -i 10.1.4.21

   namerslv -d -i 10.1.5.21

   #Add the DR DNS entry

   namerslv -a -i 10.1.7.38

   echo

   echo "$MYNAME: Configuring bootlist for DR."

   bootlist -m normal -o

   bootlist -m normal hdisk0

   bootlist -m normal -o

   echo

   echo "$MYNAME: Configuring /etc/inittab for DR."

   lsitab -a | grep nim

   chitab "nimclient:2:off:/usr/sbin/nimclient -S running > /dev/console 2>&1 #   inform nim we're running"

   lsitab -a | grep nim

   echo

else

   MYNAME=$(basename $0)

   echo "$MYNAME: The systemid $LSATTR_SYSTEMID_DR does not match the expected DR systemid of $DR_SYSTEMID."

   echo "$MYNAME: This script should only be executed at DR."

   echo "$MYNAME: If you are not booting the system at the DR site, then you can ignore this message."

   echo "$MYNAME: No changes have been perfomed. Script is exiting."

fi

The ghostdev attribute essentially provides us with a clean ODM and allows the system to discover new devices and build the ODM from scratch. If you attempt to boot from a replicated rootvg disk without first setting the ghostdev attribute, your system may fail to boot (hang at LED 554) because of a new device tree and/or missing devices.  You might be able to recover from this situation (without restoring from mksysb) by performing the steps outlined on pages 16-20 of the following document (thanks to Dominic Lancaster at IBM for the presentation).

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/resource/IBMSTGEducationSymposium_AX61_POWERVM_P2V.pdf

I received an email this week from a colleague that worked with me on the NIM Redbook back in 2006. He was experiencing an issue with DSM and NIM. He was attempting to use the dgetmacs command to obtain the MAC address of the network adapters on an LPAR. The command was failing to return the right information.

This was the output from his attempt with dgetmacs:

# dgetmacs -n lpar1

Using an adapter type of "ent".

Could not dsh to node lpar1.

Attempting to use openfirmware method to collect MAC addresses.

Acquiring adapter information from Open Firmware for node lpar1.

dgetmacs: 2760-185 Issuing the command "/opt/ibm/sysmgt/dsm/dsmbin/lpar_netboot -M -n -t ent -J hmc1 -j hmc -F /etc/ibm/sysmgt/dsm/config/my_password_file –A 4 10582FP 8202-E4B" gave a return code of 1.  The routine will continue.

No adapters found for node lpar1.

Could not get mac address for this aix node "lpar1"

I experienced this very issue during the writing of the AIX 7.1 Differences Guide Redbook. And given that I was in Austin, sitting in the same building as the AIX development team, I was able to speak with the developers directly about the issue. At that time they provided me with the following workaround.

First they asked me to check the size of the /usr/lib/nls/msg/en_US/IBMhsc.netboot.cat message catalog file.

# ls –l /usr/lib/nls/msg/en_US/IBMhsc.netboot.cat

-rw-r--r--    1 bin      bin            3905 Aug 08 09:54

They were surprised to find that the file appeared to be “too small”. They promptly sent me the catalog file from one of their development AIX 7.1 systems.  I replaced the file as follows:

# cd  /usr/lib/nls/msg/en_US/

# ls -ltr IBMhsc*

-rw-r--r--    1 bin      bin            3905 Aug 08 09:54 IBMhsc.netboot.cat

# cp -p IBMhsc.netboot.cat IBMhsc.netboot.cat.old

# cp /tmp/lpar1/IBMhsc.netboot.cat.new IBMhsc.netboot.cat

# ls -ltr IBMhsc*

-rw-r--r--    1 bin      bin         3905 Aug 08 09:54 IBMhsc.netboot.cat.old

-rw-r--r--    1 bin      bin        26374 Dec 23 11:24 IBMhsc.netboot.cat

This fixed the problem for me during the residency.

So I asked my friend to do the same (after I sent him the message catalog file). He ran the dgetmacs command again and this time it returned the MAC address for all the network adapters in his LPAR. Success!

# dgetmacs -n lpar1

Using an adapter type of "ent".

Could not dsh to node lpar1.

Attempting to use openfirmware method to collect MAC addresses.

Acquiring adapter information from Open Firmware for node lpar1.

# Node::adapter_type::interface_name::MAC_address::location::media_speed::adapter_duplex::UNUSED::install_gateway::ping_status::machine_type::netaddr::subnet_mask

lpar1::ent_v::::4A7009608816::U8202.E4B.10582FP-V4-C22-T1::auto::auto::::::n/a::secondary::::

lpar1::ent_v::::4A7009608818::U8202.E4B.10582FP-V4-C24-T1::auto::auto::::::n/a::secondary::::

This then allowed him to network boot his LPAR and install AIX 7.1 via NIM.

# nim -o bos_inst -a bosinst_data=noprompt -a source=rte -a installp_flags=agX -a accept_licenses=yes -a spot=spotaix7tl0sp2 -a lpp_source=aix7tl0sp2 lpar1

dnetboot Status: Invoking /opt/ibm/sysmgt/dsm/dsmbin/lpar_netboot lpar1

dnetboot Status: Was successful network booting node lpar1.

I thought this might have been resolved in the latest service pack for AIX 7.1. However, both of us have the latest service pack installed (i.e. TL0 SP2 for AIX 7.1).

# oslevel -s

7100-00-02-1041

If anyone encounters this problem, feel free to contact me (aix@gibsonnet.net).

For more information, please refer to section 5.2.7 Using DSM and NIM, in the AIX 7.1 Differences Guide Redbook:

http://www.redbooks.ibm.com/Redbooks.nsf/RedpieceAbstracts/sg247910.html?Open

This little tip was passed on to me by a friendly IBM hardware engineer many years ago.

When entering a capacity on demand (CoD) code into a Power system, you can tell how many processors and how much memory will be activated, just by looking at the code you’ve given by IBM.

For example, the following codes, when entered for the appropriate Power system, will enable 4 processors (POD) and 64GB of memory (MOD). I can also tell* that once the VET code is entered, this system will be licensed for PowerVM Enterprise Edition (2C28).

0000 = Reset to base defaults (No PowerVM)
1xxx = Express
2c00 = Standard
2c20 = Enterprise

*Refer to the following link for more information on “How to tell which PowerVM Editions feature was ordered”: https://www-304.ibm.com/support/docview.wss?uid=isg3T1010860

You can obtain your CoD codes from the following website: http://www-912.ibm.com/pod/pod

======

UPDATE 12/9/2011: Script from Chris-R that collects CoD information for a system from the IBM CoD Website. Thanks Chris-R, nice job!

Done!

$ ./activations.pl 9117 10 06ebp
02 processors activated on 05/23/2011
024 GB memory activated on 05/23/2011
02 processors activated on 03/11/2011
016 GB memory activated on 03/11/2011
055 GB memory activated on 03/02/2011
06 processors activated on 02/22/2011
096 GB memory activated on 02/22/2011
01 processors activated on 01/12/2011
016 GB memory activated on 01/12/2011
06 processors activated on 03/17/2010
072 GB memory activated on 03/17/2010
19 processors activated on 03/16/2010
228 GB memory activated on 03/16/2010
================
TOTAL CPU=36 RAM=519168MB (507GB)

$cat activations.pl
#!/usr/bin/perl
#
# PoC Script to go out on the net to the IBM POD site and tally up activations for a given i or p Series machine
# Chris-R rutherfc@gmail.com
# v1.0 9/9/2011
#
# todo - add logic for processor deactivation - who uses that? IBM dont even publish the code on the pod site ;)
#
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$ua->agent("mozilla 8.0");
# $ua->proxy(['http'], 'http://proxy:8080/');
$ua->timeout(10);
use HTTP::Request::Common qw(POST);
if ($#ARGV != 2) {
print "usage: $0 MODEL XX XXXXX e.g $0 9119 83 9f6bf\n";
exit;
}
($model, $serial1, $serial2) = @ARGV;

##### main #####
get('http://www-912.ibm.com/pod/pod',"$serial2.htm");
html2txt("$serial2.htm","$serial2.txt");
total("$serial2.txt");
exit;

sub get # fakes a mozilla browser, fills in the CGI form and snags the returned page to a local html file
{
my $req = (POST $_[0],
["system_type" => $model,
"system_serial_number1" => $serial1,
"system_serial_number2" => $serial2 ]);
$request = $ua->request($req);
$activations = $request->content;
open(POD,">$_[1]");
print POD $activations;
close(POD);
}

sub html2txt # strips out the crap and converts the table to a local txt file to parse
{
open(HTML,"<$_[0]");
open(TXT,">$_[1]");
while (<HTML>) {
if (/<\/table>/) {$f = 0;};
if (/<th>Posted Date \(MM/) {$f = 1;};
if ($f == 1) {
# poor mans HTML::TableExtract - excuse my sed like perl....
s/<tr align="center">/:/g;
s/<[^>][^>]*>//g;
s/ //g;
s/\n//g;
s/:/\n/g;
print TXT $_;
};
};
close(TXT);
close(HTML);
}
sub total # totals up the de/activations to get totals
{
open(TXT,"<$_[0]");
$[ = 1; $\ = "\n";# set array base & output record separator
while (<TXT>) {
($code,$hex,$date) = split(' ', $_, -1);
if (/POD/) {
$p = substr($hex, 27, 2);
print $p . ' processors activated on ' . $date;
$pt = $pt + $p;
};
if (/MOD/) {
$r = substr($hex, 26, 3);
print $r . ' GB memory activated on ' . $date;
$rt = $rt + $r;
};
if (/RMEM/) {
$r = substr($hex, 27, 2);
print $r . ' GB memory activated on ' . $date;
$rt = $rt - $r;
};
};
print '================';
print 'TOTAL CPU=' . $pt . ' RAM=' . $rt*1024 . 'MB (' . $rt . 'GB)';
close(TXT);
}
</TXT></tr></th></HTML>

Using Cluster Aware AIX with AIX 7.1 you can create a cluster of AIX nodes. This interconnected cluster of nodes can immediately provide several capabilities, such as:

·         Clusterwide event management.

·         Communication and storage events:

o   Node UP and node DOWN

o   Network adapter UP and DOWN

o   Network address change

o   Point-of-contact UP and DOWN

o   Disk UP and DOWN

o   Predefined and user-defined events

·         Clusterwide storage naming service

·         Clusterwide command distribution

·         Clusterwide communication making use of networking and storage connectivity

All of the capabilities of CAA are built into the AIX Version 7.1 operating system. CAA is essentially a set of tools and services embedded into AIX that help to manage a cluster of AIX nodes and assist in running cluster software on AIX. Cluster products, such as PowerHA and RSCT from IBM, can utilise CAA to simplify the configuration, management and monitoring of an AIX cluster.

CAA does not form a cluster by itself. It is a tool set. There is no notion of quorum. (If 20 nodes of a 21 node cluster are down, CAA still runs on the remaining node). CAA does not eject nodes from a cluster. CAA provides tools to fence a node but never fences a node and will continue to run on a fenced node.

As you can see in the following diagram, cluster products, like PowerHA, will integrate with CAA to help form and manage highly available clusters.

So you will still need some form of cluster product, either from IBM or another vendor, in order to build a cluster that provides high availability capabilities like node failover/takeover.

What follows are some of the most important “snippets” from the CAA documentation (IMHO), available in the AIX 7.1 Information Centre (see the link at the bottom of the page).

Just like any cluster, each node that is added to a cluster by using CAA must have common storage devices available, for example via SAN storage device. These storage devices are used for the cluster repository disk and for any clustered shared data disks.

The Storage Naming Service provides a global device view across all the nodes in the cluster. The Storage Naming Service also provides a single global device name for a disk from any node in the cluster. The global device name, for example, cldisk1, refers to the same physical disk from any node in the cluster.

The cluster repository disk is used as the central repository for the cluster configuration data. The cluster repository disk must be accessible from all nodes in the cluster and is a minimum of 10 GB in size. Given the importance of the cluster configuration data, the cluster repository disk should be backed up by a redundant and highly available storage configuration. Even though the cluster repository disk is visible as a disk device to AIX, it should be treated as a special device for the cluster. The use of LVM commands is not supported when used on a cluster repository disk. The AIX LVM commands are designed as single node administrative commands, and are not applicable in a clustered configuration.

The cluster repository disk is renamed to a private device name (caa_private0). A raw section of the disk and a section of the disk that contains a special volume group and special logical volumes are used during cluster operations.

A multicast address is used for cluster communications between the nodes in the cluster.

These are configured automatically during the creation of the cluster. These nodes support cluster monitoring of events and cluster configuration attributes.

Scalable reliable multicasting is implemented in the cluster with a special gossip protocol over the multicast address. The gossip protocol determines the node configuration and then transmits the gossip packets over all available networking and storage communication interfaces (either Fibre Channel and/or SAS adapters). If no storage communication interfaces are configured, only the traditional networking interfaces are used.

When you first configure CAA, the following actions are performed:

•             The cluster is created using the mkcluster command.

•             The cluster configuration is written to the raw section of the cluster repository disk.

•             Primary and secondary database nodes are selected from the list of candidate nodes in the mkcluster command.

•             Special volume groups and logical volumes are created on the cluster repository disk.

•             Cluster file systems are created on the special volume group.

•             The cluster repository database is created on both primary and secondary nodes.

•             The cluster repository database is started.

•             Cluster services are made available to other functions in the operating system, such as Reliable Scalable Cluster Technology (RSCT) and PowerHA.

•             Storage framework register lists are created on the cluster repository disk.

•             A global device namespace is created and interaction with LVM starts for handling associated volume group events.

•             A clusterwide multicast address is established.

•             The node discovers all of the available communication interfaces.

•             The cluster interface monitoring starts.

•             The cluster interacts with Autonomic Health Advisory File System (AHAFS) for clusterwide event distribution.

•             The cluster exports cluster messaging and cluster socket services to other functions in the operating system, such as Reliable Scalable Cluster Technology (RSCT) and PowerHA.

In the following example I created a two node cluster using CAA tools. First of all, I had to use the mkcluster command to define the cluster nodes, shared storage and repository disk. The node names are 7502lp01 and 7502lp02. The shared data storage disks are hdisk2, hdisk3 and hdisk4. The repository disk, used to house the cluster configuration data, is hdisk1.

# mkcluster -n mycluster -r hdisk1 -d hdisk2,hdisk3,hdisk4 -m 7502lp01,7502lp02

mkcluster: Cluster shared disks are automatically renamed to names such as

        cldisk1, [cldisk2, ...] on all cluster nodes.  However, this cannot

        take place while a disk is busy or on a node which is down or not

        reachable.  If any disks cannot be renamed now, they will be renamed

        later by the clconfd daemon, when the node is available and the disks

        are not busy.

You’ll notice that the mkcluster command informed me that the cluster shared disks are automatically renamed to cluster disk names like cldisk1. After I’d run the command I noticed something quite interesting and impressive. Prior to configuring the cluster the shared disks on both nodes had names like hdiskX. Afterwards the shared disks had been renamed across both nodes. Now these disks had the same name!

This is going to simplify cluster configuration and management. No more will I need to remove and recreate disks in order to resolve disk naming inconsistencies in a cluster. The lspv output (shown below) from both nodes shows that I have three shared disks (cldisk1, cldisk2 and cldisk3). These would be used for shared data. The disk named caa_private0 is my cluster repository disk. This is used to store and share the cluster configuration data.

7502lp01:

# lspv | sort

caa_private0    00f61ab20b97190d                    caavg_private   active

cldisk1         00f61ab20bf28ac6                    None

cldisk2         none                                None

cldisk3         none                                None

hdisk0          00f61ab2f73e46e2                    rootvg          active

7502lp02:

# lspv | sort

caa_private0    00f61ab20b97190d                    caavg_private   active

cldisk1         00f61ab20bf28ac6                    None

cldisk2         none                                None

cldisk3         none                                None

hdisk0          00f61ab2895e4cbe                    rootvg          active

Cluster Aware AIX tells you what nodes are in the cluster plus information on those nodes, including state. A special “gossip” protocol is used over the multicast address to determine node information and implement scalable reliable multicast. No traditional heartbeat mechanism is employed. Gossip packets travel over all interfaces, including storage. Immediately after running mkcluster, I was able to query the status of the nodes in my cluster without any further configuration!

# lscluster -m

Calling node query for all nodes

Node query number of nodes examined: 2

        Node name: 7502lp01

        Cluster shorthand id for node: 1

        uuid for node: 3cd9cb00-bf55-11df-b015-6e8dd0af6304

        State of node:  UP

        Smoothed rtt to node: 7

        Mean Deviation in network rtt to node: 3

        Number of zones this node is a member in: 0

        Number of clusters node is a member in: 1

        CLUSTER NAME       TYPE  SHID   UUID

        mycluster          local        267ce7fc-bf55-11df-a3b9-6e8dd877b814

        Number of points_of_contact for node: 1

        Point-of-contact interface & contact state

         en0  UP

------------------------------

        Node name: 7502lp02

        Cluster shorthand id for node: 2

        uuid for node: d1a46164-bf46-11df-94b3-6e8dd877b814

        State of node:  UP  NODE_LOCAL

        Smoothed rtt to node: 0

        Mean Deviation in network rtt to node: 0

        Number of zones this node is a member in: 0

        Number of clusters node is a member in: 1

        CLUSTER NAME       TYPE  SHID   UUID

        mycluster          local        267ce7fc-bf55-11df-a3b9-6e8dd877b814

        Number of points_of_contact for node: 0

        Point-of-contact interface & contact state

         n/a

CAA tells you what interfaces have been discovered on a node plus information on those interfaces, including state.

# lscluster –i –n

Node 7502lp01

Node uuid = 110b2422-7efc-11df-aed7-1612a0003002

Number of interfaces discovered = 1

        Interface number 1 en0

                ifnet type = 6 ndd type = 7

                Mac address length = 6

                Mac address =  16.12.a0.0.30.2

                Smoothed rrt across interface = 7

                Mean Deviation in network rrt across interface = 3

                Probe interval for interface = 100 ms

                ifnet flags for interface = 0x1e080863

                ndd flags for interface = 0x21081b

                Interface state  UP

                Number of regular addresses configured on interface = 1

                IPV4 ADDRESS: 9.3.28.136  broadcast 9.3.28.159  netmask 255.255.255.224

                Number of cluster multicast addresses configured on interface = 1

                IPV4 MULTICAST ADDRESS: 228.8.8.8  broadcast 0.0.0.0  netmask 0.0.0.0

CAA tells you what disks are in the cluster plus information on those disks, including the state and type.

# lscluster –d

Storage Interface Query

Cluster Name:  mycluster

Cluster uuid: a1bde416-83b9-11df-b438-1612a0004002

Number of nodes reporting = 2

Number of nodes expected = 2

Node oscar-dev8.austin.ibm.com

Node uuid = 9807465b-3194-f1e5-8a04-1f044bc82593

Number of disk discovered = 2

        cldisk1

          state : UP

          uDid  : 533E3E213600A0B8000475C200000E7114BC825930F1818      FAStT03IB

Mfcp05VDASD03AIXvscsi

             type  : CLUSDISK

          uUid  : 9807465b-3194-f1e5-8a04-1f044bc82593

        hdisk1

          state : UP

          uDid  :

          uUid  : 600a0b80-0047-5d0a-0000-e6094bc82690

          type  : REPDISK

You can run cluster wide commands on all nodes immediately after configuring your cluster using the clcmd command. No further configuration required.

# clcmd ps -ef

-------------------------------

NODE 7502lp01

-------------------------------

     UID     PID         PPID      C    STIME    TTY  TIME CMD

    root       1           0       0   Aug 25      -  0:00 /etc/init

    root  655526           1       0   Aug 25      -  0:18 [cimserve]

    root 1310840           1       0   Aug 25      -  0:00 /usr/ccs/bin/shlap64

    root 2490482           1       0   Aug 25      -  0:42 /usr/sbin/srcmstr

    root 3145874         3801262   0 11:17:13      -  0:00 telnetd -a

    root 3276994         2490482   0   Aug 25      -  0:10 /usr/sbin/snmpd

-------------------------------

NODE 7502lp02

-------------------------------

     UID     PID         PPID   C    STIME    TTY  TIME CMD

    root       1           0   1   Aug 28      -  0:00 /etc/init

    root 1441934           1   0   Aug 28      -  0:39 /usr/sbin/syncd 60

    root 1638496           1   0   Aug 28      -  0:00 /usr/ccs/bin/shlap64

    root 2228252     2556054   0   Aug 28      -  0:00 /usr/sbin/hostmibd

I believe that PowerHA SystemMirror 7.1 will be the first cluster product from IBM to provide high availability on AIX using the new Cluster Aware AIX features. An IBM Redbook Residency recently covered the integration of CAA and PowerHA. I’m eagerly awaiting the first draft of this book so that I can learn how these two components will “play” together.

Refer to the AIX 7.1 Information Centre for more information on Cluster Aware AIX:

http://publib.boulder.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.clusteraware/claware_main.htm

Refer to the AIX 7.1 Information Centre for more information on AHAFS and the new AIX Event Infrastructure:

http://publib.boulder.ibm.com/infocenter/aix/v7r1/index.jsp?topic=/com.ibm.aix.baseadmn/doc/baseadmndita/aix_ev.htm

In a perfect world, 99.9% of AIX administrators would prefer their systems to look like this:

# lspv | grep rootvg

hdisk0         00c342c68dfcbdfb                    rootvg          active

However, in reality, 99.9% of AIX administrators live with systems that look something like this:

# lspv | grep rootvg

hdisk39         00c342c68dfcbdfb                    rootvg          active

And 99.9% of them don’t have time to tidy up their systems so that rootvg resides on hdisk0.

Most of them have much bigger fish to fry, such as performance, virtualisation, automation, security, project delivery, TPS reports, etc!

If they did have time, they could use the mirrorvg and rendev commands to ‘bring order to the Universe’.

1. Add a new disk to the system.

# lspv

hdisk39         00c342c68dfcbdfb                    rootvg          active

hdisk40         00c342c6161c6b47                    None

2. Rename the new disk to hdisk0.

# rendev -l hdisk40 -n hdisk0

hdisk0

3. Add the disk to rootvg and mirror to it.

# extendvg rootvg hdisk0

# mirrorvg rootvg hdisk0

0516-1804 chvg: The quorum change takes effect immediately.

0516-1126 mirrorvg: rootvg successfully mirrored, user should perform

        bosboot of system to initialize boot records.  Then, user must modify

        bootlist to include:  hdisk0 hdisk39.

4. Create a boot image on hdisk0.

# bosboot –a –d /dev/hdisk0

bosboot: Boot image is 49180 512 byte blocks.

# ipl_varyon -i

[S 3670248 14942228 07/24/12-10:18:11:104 ipl_varyon.c 1270] ipl_varyon -i

PVNAME          BOOT DEVICE     PVID                    VOLUME GROUP ID

hdisk39         YES             00c342c68dfcbdfb0000000000000000        00c342c600004c00

hdisk0          YES             00c342c6161c6b470000000000000000        00c342c600004c00

[E 3670248 0:087 ipl_varyon.c 1410] ipl_varyon: exited with rc=0

5. Check and update the bootlist. Ensure hdisk0 is in the bootlist.

# bootlist –m normal hdisk0

# bootlist –m normal –o

hdisk0 blv=hd5 pathid=0

hdisk0 blv=hd5 pathid=1

6. Unmirror rootvg on hdisk39. Remove hdisk39 from rootvg.

# unmirrorvg rootvg hdisk39

0516-1246 rmlvcopy: If hd5 is the boot logical volume, please run 'chpv -c <diskname>'

        as root user to clear the boot record and avoid a potential boot

        off an old boot image that may reside on the disk from which this

        logical volume is moved/removed.

0516-1804 chvg: The quorum change takes effect immediately.

0516-1144 unmirrorvg: rootvg successfully unmirrored, user should perform

        bosboot of system to reinitialize boot records.  Then, user must modify

        bootlist to just include:  hdisk0.

# lspv –l hdisk39

#

# reducevg rootvg hdisk39

#

7. Order in the Universe has been restored.

# lspv | grep rootvg

hdisk0          00c342c6161c6b47                    rootvg          active

Of course, all of this assumes that the name, hdisk0, is not already in use by a hdisk in another volume group on the system.

Again, if you have the time, then you could perform the following to rectify the situation.

1. datavg resides on hdisk0.

# lspv | grep datavg

hdisk0          00c342c6161c6b47                    datavg          active

2. Unmount all data file systems in this volume group. Varyoff the VG.

# varyoffvg datavg

3. Rename hdisk0 to hdisk99 (or something other than hdisk0).

# rendev -l hdisk0 -n hdisk99

hdisk99

4. Varyon the volume group, datavg. Mount the data file systems in datavg.

# varyonvg datavg

# mount /datafs

5. datavg now resides on (the newly renamed) hdisk99.

# lspv | grep datavg

hdisk99         00c342c6161c6b47                    datavg

Now that the name, hdisk0, is free, you can perform the steps, outlined above, to restore ‘Order in the Universe’ and put rootvg back on hdisk0!

Q: Does alt_disk_copy convert JFS filesystems to JFS2?
A: Yes! Starting at 6100-04, a new flag was introduced that would convert JFS filesystems to JFS2. We now have the -T which does the conversion.

http://www-01.ibm.com/support/docview.wss?uid=isg3T1011055

Here’s an example of converting rootvg file systems from JFS to JFS2 using alt_disk_copy.

My lab system was migrated from AIX 5.3 to 7.1 via nimadm. Unfortunately, nimadm does not convert JFS file systems to JFS2 during the migration. So, in this case, even though I’ve migrated to AIX 7.1 (which is a good thing) I’m still left with legacy JFS file systems in rootvg.

And because the AIX 5.3 version of the alt_disk_copy command does not have the –T option, I can’t convert my JFS file systems to JFS2 before I migrate to AIX 7.1. So my best option is to migrate to AIX 7.1 then convert rootvg to JFS2 file systems. A few hops in the process but it’s good enough.

aixlpar1 : / # lsvg -l rootvg

rootvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

hd5                 boot       1       1       1    closed/syncd  N/A

hd6                 paging     32      32      1    open/syncd    N/A

hd8                 jfslog     1       1       1    open/syncd    N/A

hd4                 jfs        4       4       1    open/syncd    /

hd2                 jfs        29      29      1    open/syncd    /usr

hd9var              jfs        20      20      1    open/syncd    /var

hd3                 jfs        164     164     1    open/syncd    /tmp

hd1                 jfs        4       4       1    open/syncd    /home

hd10opt             jfs        4       4       1    open/syncd    /opt

local               jfs        4       4       1    open/syncd    /usr/local

loglv               jfs        4       4       1    open/syncd    /var/log

hd7                 sysdump    9       9       1    open/syncd    N/A

hd71                sysdump    9       9       1    open/syncd    N/A

hd11admin           jfs        2       2       1    open/syncd    /admin

aixlpar1 : / # oslevel -s

7100-01-01-1141

I clone rootvg to a spare disk using alt_disk_copy and the –T flag (which will convert the file systems to JFS2). The process converts the file systems to JFS2, as shown in the section below (highlighted in green).

aixlpar1 : / # alt_disk_copy -d hdisk1 -T

Source boot disk is: hdisk2

jfs2j2: Current data file /image.data moved to /image.data.acct.save.3735802.

jfs2j2: Creating new data file (/image.data).

jfs2j2: Converting filesystems in data file (/image.data) from jfs to jfs2.

jfs2j2: Converted data file (/image.data) moved to /tmp/image.data.conv.3407932.

jfs2j2: Cleaning up.

jfs2j2: Moving previous data file /image.data.acct.save.3735802 to IDFILE.

jfs2j2: Successfully completed converting image.data to jfs2.

Checking disk sizes.

Creating cloned rootvg volume group and associated logical volumes.

Creating logical volume alt_hd5

Creating logical volume alt_hd6

Creating logical volume alt_hd8

Creating logical volume alt_hd4

Creating logical volume alt_hd2

Creating logical volume alt_hd9var

Creating logical volume alt_hd3

Creating logical volume alt_hd1

Creating logical volume alt_hd10opt

Creating logical volume alt_local

Creating logical volume alt_loglv

Creating logical volume alt_hd7

Creating logical volume alt_hd71

Creating logical volume alt_hd11admin

Creating /alt_inst/ file system.

/alt_inst filesystem not converted.

        Small inode extents are already enabled.

Creating /alt_inst/admin file system.

/alt_inst/admin filesystem not converted.

        Small inode extents are already enabled.

Creating /alt_inst/home file system.

/alt_inst/home filesystem not converted.

        Small inode extents are already enabled.

Creating /alt_inst/opt file system.

/alt_inst/opt filesystem not converted.

        Small inode extents are already enabled.

Creating /alt_inst/tmp file system.

/alt_inst/tmp filesystem not converted.

        Small inode extents are already enabled.

Creating /alt_inst/usr file system.

/alt_inst/usr filesystem not converted.

        Small inode extents are already enabled.

Creating /alt_inst/usr/local file system.

/alt_inst/usr/local filesystem not converted.

        Small inode extents are already enabled.

Creating /alt_inst/var file system.

/alt_inst/var filesystem not converted.

        Small inode extents are already enabled.

Creating /alt_inst/var/log file system.

/alt_inst/var/log filesystem not converted.

        Small inode extents are already enabled.

Generating a list of files

for backup and restore into the alternate file system...

Backing-up the rootvg files and restoring them to the

alternate file system...

Running user specified customization script /tmp/cust.script.conv.3407932...

Modifying ODM on cloned disk.

Building boot image on cloned disk.

forced unmount of /alt_inst/var/log

forced unmount of /alt_inst/var

forced unmount of /alt_inst/var

forced unmount of /alt_inst/usr/local/chksys

forced unmount of /alt_inst/usr/local

forced unmount of /alt_inst/usr

forced unmount of /alt_inst/tmp

forced unmount of /alt_inst/opt

forced unmount of /alt_inst/home

forced unmount of /alt_inst/admin

forced unmount of /alt_inst

Changing logical volume names in volume group descriptor area.

Fixing LV control blocks...

Fixing file system superblocks...

Bootlist is set to the boot disk: hdisk1 blv=hd5

Before I reboot the system on the alternate rootvg I verify that the cloned volume group now contains JFS2 file systems only. I “wake up” the altinst_rootvg and run the lsvg command to confirm the file system is correct. I then put the altinst_rootvg to “sleep”, reboot the system and verify all rootvg file systems are mounted as jfs2.

aixlpar1 : / # alt_rootvg_op -W -d hdisk1

Waking up altinst_rootvg volume group ...

aixlpar1 : / # lsvg -l altinst_rootvg

altinst_rootvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

alt_hd5             boot       1       1       1    closed/syncd  N/A

alt_hd6             paging     32      32      1    closed/syncd  N/A

alt_hd8             jfs2log    1       1       1    open/syncd    N/A

alt_hd4             jfs2       4       4       1    open/syncd    /alt_inst

alt_hd2             jfs2       29      29      1    open/syncd    /alt_inst/usr

alt_hd9var          jfs2       20      20      1    open/syncd    /alt_inst/var

alt_hd3             jfs2       164     164     1    open/syncd    /alt_inst/tmp

alt_hd1             jfs2       4       4       1    open/syncd    /alt_inst/home

alt_hd10opt         jfs2       4       4       1    open/syncd    /alt_inst/opt

alt_local           jfs2       4       4       1    open/syncd    /alt_inst/usr/local

alt_loglv           jfs2       4       4       1    open/syncd    /alt_inst/var/log

alt_hd7             sysdump    9       9       1    closed/syncd  N/A

alt_hd71            sysdump    9       9       1    closed/syncd  N/A

alt_hd11admin       jfs2       2       2       1    open/syncd    /alt_inst/admin

aixlpar1 : / # alt_rootvg_op -S altinst_rootvg

Putting volume group altinst_rootvg to sleep ...

forced unmount of /alt_inst/var/log

forced unmount of /alt_inst/var

forced unmount of /alt_inst/usr/local

forced unmount of /alt_inst/usr

forced unmount of /alt_inst/tmp

forced unmount of /alt_inst/opt

forced unmount of /alt_inst/home

forced unmount of /alt_inst/admin

forced unmount of /alt_inst

Fixing LV control blocks...

Fixing file system superblocks...

aixlpar1 : / #

; Reboot on the alternate rootvg hdisk

aixlpar1 : / # uptime

  10:18AM   up 1 min,  1 user,  load average: 0.32, 0.09, 0.03

aixlpar1 : / # lspv

hdisk1          00c342c637f21a59                    rootvg          active

hdisk2          00c342c6161c6b47                    old_rootvg

aixlpar1 : / # df

Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4          524288    411568   22%     3040     7% /

/dev/hd2         3801088    537176   86%    34089    35% /usr

/dev/hd9var      2621440   2413520    8%     3554     2% /var

/dev/hd3        21495808  21476312    1%      110     1% /tmp

/dev/hd1          524288    522456    1%       90     1% /home

/proc                  -         -    -         -     -  /proc

/dev/hd10opt      524288    266560   50%     5133    15% /opt

/dev/local        524288    495320    6%      249     1% /usr/local

/dev/loglv        524288    522040    1%       49     1% /var/log

/dev/hd11admin     262144    261384    1%        7     1% /admin

aixlpar1 : / # lsvg -l rootvg

rootvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

hd5                 boot       1       1       1    closed/syncd  N/A

hd6                 paging     32      32      1    open/syncd    N/A

hd8                 jfs2log    1       1       1    open/syncd    N/A

hd4                 jfs2       4       4       1    open/syncd    /

hd2                 jfs2       29      29      1    open/syncd    /usr

hd9var              jfs2       20      20      1    open/syncd    /var

hd3                 jfs2       164     164     1    open/syncd    /tmp

hd1                 jfs2       4       4       1    open/syncd    /home

hd10opt             jfs2       4       4       1    open/syncd    /opt

local               jfs2       4       4       1    open/syncd    /usr/local

loglv               jfs2       4       4       1    open/syncd    /var/log

hd7                 sysdump    9       9       1    open/syncd    N/A

hd71                sysdump    9       9       1    open/syncd    N/A

hd11admin           jfs2       2       2       1    open/syncd    /admin

===

The message “filesystem not converted” (below) is not related to the JFS to JFS2 conversion. This messge refers to whether or not the file system needs to be changed to use Variable Inode Extents (VIX). This is the default setting for JFS2 file systems.

Creating /alt_inst/ file system.

/alt_inst filesystem not converted.

        Small inode extents are already enabled.

aixlpar1 : / # lsfs -q /

Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting

/dev/hd4        --         /                      jfs2  524288  --         yes  no

  (lv size: 524288, fs size: 524288, block size: 4096, sparse files: yes, inline log: no, inline log size: 0, EAformat: v1, Quota: no, DMAPI: no, VIX: yes, EFS: no, ISNAPSHOT: no, MAXEXT: 0, MountGuard: no)

aixlpar1 : / #

Please refer to the following link for more information on VIX in AIX.

Use of Variable Inode Extents (VIX) in AIX

http://www-01.ibm.com/support/docview.wss?uid=isg3T1010368

===

Why would I want to convert rootvg to JFS2 anyway? Well for starters, it’s generally considered best practice to use JFS2 as it offers several performance & scalability enhancements over JFS. For example, you cannot create files greater than 2GB on JFS, unless the file system was created as “large (big) file” enabled; jfs file systems in rootvg were never created as large file enabled.

Another reason…..eventually JFS will be retired.

Here’s an example of a potential problem with JFS in rootvg. You try to create a file of a size greater than 2GB in /tmp (type jfs). Even though the ulimit settings are not restricting the creation of a file of this size, the JFS file system will not allow it. The file creation process fails. The bf attribute for the /tmp file system is set to false. This indicates the file system is not “large file” enabled.

aixlpar1 : /tmp # df -g .

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

/dev/hd3          10.25      6.94   33%      117     1% /tmp

aixlpar1 : /tmp # cat 1gbfile 1gbfile 1gbfile > 3gbfile

cat: 0652-054 Cannot write to output.

Error 0

aixlpar1 : /tmp # ulimit -a

time(seconds)        unlimited

file(blocks)         unlimited

data(kbytes)         unlimited

stack(kbytes)        unlimited

memory(kbytes)       unlimited

coredump(blocks)     2097151

nofiles(descriptors) unlimited

threads(per process) unlimited

processes(per user)  unlimited

aixlpar1 : /tmp # lsfs -q /tmp

Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting

/dev/hd3        --         /tmp                   jfs   21495808 --         yes  no

  (lv size: 21495808, fs size: 21495808, frag size: 4096, nbpi: 4096, compress: no, bf: false, ag: 8)

OK, so creating files greater than 2GB in rootvg is probably not a big problem for most AIX shops. But why risk it! My advice is to always use JFS2 (even in rootvg).

Keep this in mind when migrating from AIX 5.3 to 6.1 or 7.1 via nimadm.

The following link provides an overview and comparison of both JFS and JFS2 file systems.

http://pic.dhe.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.baseadmn/doc/baseadmndita/fs_jfs_jfs2.htm

I’ve been working with a customer recently on an issue with nimadm. They were attempting to migrate a system from AIX 5.3 to 7.1 using nimadm. The NIM client AIX level was 5.3 TL12 SP4 and the NIM master was running AIX 7.1 TL1 SP1.

lpar1 : / # oslevel -s

5300-12-04-1119

root@nim1 : / # oslevel -s

7100-01-01-1141

They were running the following nimadm command:

# nimadm -j nimadmvg -c lpar1 -s spotaix710101 -l lpp_sourceaix710101 -d hdisk1 -Y

The nimadm operation would always fail at phase 11:

+-----------------------------------------------------------------------------+
Executing nimadm phase 11.
+-----------------------------------------------------------------------------+
Cloning altinst_rootvg on client, Phase 3.
Client alt_disk_install command: alt_disk_copy -j -M 7.1 -P3 -d "hdisk1"
## Phase 3 ###################
Verifying altinst_rootvg...
alt_disk_copy: 0505-218 ATTENTION: init_multibos() returned an unexpected result.
Cleaning up.
forced unmount of /alt_inst/var/log
forced unmount of /alt_inst/var/log
forced unmount of /alt_inst/var
forced unmount of /alt_inst/var
forced unmount of /alt_inst/usr/local
forced unmount of /alt_inst/usr/local
forced unmount of /alt_inst/usr
forced unmount of /alt_inst/usr
forced unmount of /alt_inst/tmp
forced unmount of /alt_inst/tmp
forced unmount of /alt_inst/opt
forced unmount of /alt_inst/opt
forced unmount of /alt_inst/home
forced unmount of /alt_inst/home
forced unmount of /alt_inst/admin
forced unmount of /alt_inst/admin
forced unmount of /alt_inst
forced unmount of /alt_inst
0505-187 nimadm: Error cloning altinst_rootvg on client.
Cleaning up alt_disk_migration on the NIM master.
Cleaning up alt_disk_migration on client lpar1.
Client alt_disk_install command: alt_disk_install -M 7.1 -X
Bootlist is set to the boot disk: hdisk0 blv=hd5

We also found the init_multibos error in the /var/adm/ras/alt_disk_inst.log file on the NIM client:

Tue Nov 22 14:48:12 EETDT 2011

cmd: /ALT_MIG_SPOT/sbin/alt_disk_copy -j -M 7.1 -P3 -d hdisk1

Verifying altinst_rootvg...

alt_disk_copy: 0505-218 ATTENTION: init_multibos() returned an unexpected result.

Cleaning up.

Given that the error appeared to be related to init_multibos, we assumed the failure was due to some multibos checks being performed by alt_disk_copy on the client. The client system did not have an existing multibos standby instance. So, we tried two things: First we created a standby instance on the client (multibos –s –X) and re-tried the nimadm operation. This failed. Next we removed the standby instance (multibos –R) and re-tried the nimadm operation. This worked and the client then migrated to AIX 7.1 successfully. We re-tried the same operations (i.e. create standby instance, remove standby instance & nimadm) several times and each worked as expected.

So it appeared that the unofficial work around to this problem would be to create and then remove a standby multibos instance prior to the nimadm migrate. However, the customer has over 200 LPARs that they need to migrate to AIX 7.1. If possible they would really rather avoid this extra step in the AIX 7.1 migration plan. We’ve made contact with IBM support and are hoping they can assist us in identifying the root cause of the issue and provide us with an official solution to the problem.

And just yesterday we hit the same problem when migrating from AIX 6.1 to 7.1 using nimadm. I’ll update my blog with any progress we make with this problem. In the meantime, our unofficial work around will get us “out of hot water”!

UPDATE (14/12/2011): The simple fix is to remove the /bos_inst directory before attempting the AIX migration. i.e.

# rm –r /bos_inst

In my nimadm article I wrote:

• If you have a multibos image in rootvg, remove it. AIX migrations are not supported with multibos enabled systems. Ensure all rootvg LVs are renamed to their legacy names. If necessary, create a new instance of rootvg and reboot the LPAR. For example:

Confirm the legacy LV names are now in use that is, not bos_.

Remove the old multibos instance.

Unfortunately, it appears that ‘multibos –R’ may not clean up the /bos_inst directory. If this directory exists the nimadm operation will most likely fail.

I was working with a customer recently on a Power Blade that was running the Integrated Virtualisation Manager (IVM). They’d installed a VIO partition onto the Blade and had hoped to install a couple of AIX LPARs on the system. However they didn’t get very far.

As soon as they attempted to NIM install the LPARs, they would get stuck at trying to ping the NIM master from the client. Basically, the Shared Ethernet Adapter (SEA) was not working properly and none of the LPARs could communicate with the external network. So they asked for some assistance.

The Blade server name was Server-8406-71Y-SN06BF99Z. The SEA was configured as ent7.

# lsdev -Cc adapter

ent0    Available       Logical Host Ethernet Port (lp-hea)

ent1    Available       Logical Host Ethernet Port (lp-hea)

ent2    Available       Virtual I/O Ethernet Adapter (l-lan)

ent3    Available       Virtual I/O Ethernet Adapter (l-lan)

ent4    Available       Virtual I/O Ethernet Adapter (l-lan)

ent5    Available       Virtual I/O Ethernet Adapter (l-lan)

ent6    Available       EtherChannel / IEEE 802.3ad Link Aggregation

ent7    Available       Shared Ethernet Adapter

ent9    Available       Virtual I/O Ethernet Adapter (l-lan)

fcs0    Available 02-00 8Gb PCIe FC Blade Expansion Card (7710322577107601)

fcs1    Available 02-01 8Gb PCIe FC Blade Expansion Card (7710322577107601)

ibmvmc0 Available       Virtual Management Channel

lhea0   Available       Logical Host Ethernet Adapter (l-hea)

sissas0 Available 01-08 PCI-X266 Planar 3Gb SAS Adapter

usbhc0  Available 00-08 USB Host Controller (33103500)

usbhc1  Available 00-09 USB Host Controller (33103500)

usbhc2  Available 00-0a USB Enhanced Host Controller (3310e000)

vhost0  Available       Virtual SCSI Server Adapter

vsa0    Available       LPAR Virtual Serial Adapter

vts0    Available       Virtual TTY Server Adapter

The SEA was configured with Port-VLAN ID (PVID) of 68 without any VLAN tags. This was the root cause of the problem.

# lsattr -El ent7

accounting    enabled  Enable per-client accounting of network statistics                 True

ctl_chan               Control Channel adapter for SEA failover                           True

gvrp          no       Enable GARP VLAN Registration Protocol (GVRP)                      True

ha_mode       disabled High Availability Mode                                             True

jumbo_frames  no       Enable Gigabit Ethernet Jumbo Frames                               True

large_receive no       Enable receive TCP segment aggregation                             True

largesend     0        Enable Hardware Transmit TCP Resegmentation                        True

netaddr       0        Address to ping                                                    True

pvid          68       PVID to use for the SEA device                                     True

pvid_adapter  ent9     Default virtual adapter to use for non-VLAN-tagged packets         True

qos_mode      disabled N/A                                                                True

real_adapter  ent6     Physical adapter associated with the SEA                           True

thread        1        Thread mode enabled (1) or disabled (0)                            True

virt_adapters ent9     List of virtual adapters associated with the SEA (comma separated) True

$ entstat -all ent8 | grep -i vlan

VLAN Ids :

    VLAN Extract: False

    VLAN tagged filtering mode: Filter according to VLAN permit array

Max number of VLAN IDs per HEA port: 20

    VLAN Extract: False

    VLAN tagged filtering mode: Filter according to VLAN permit array

Max number of VLAN IDs per HEA port: 20

Invalid VLAN ID Packets: 5388

Port VLAN ID:    68

VLAN Tag IDs:    None

On the network switch port, the native VLAN (PVID), was configured as 11, with VLAN tag 68 added as an allowed VLAN. If the client LPARs tried to access the network using a PVID of 68, instead of a VLAN TAG of 68, they would get stuck at the switch port i.e. the un-tagged packets for 10.1.68.X via PVID 11 would fail. The packets for 10.1.68.X needed to be tagged with VLAN id 68 in order for the switch to pass the traffic.

So the question was, how do we add VLAN tags in the IVM environment? If we’d been using a HMC, then this would be simple to fix. Just add the VLAN tags into the Virtual Ethernet Adapter used by the SEA and we’d be done.

We had to use the lshwres and chhwres commands to resolve this one. First we listed the virtual adapters known to the VIO server (IVM). At slot 12, we found our SEA adapter with port_vlan_id set to 68 and addl_vlan_ids set to none.

$ lshwres -r virtualio --rsubtype eth --level lpar

lpar_name=06-BF99Z,lpar_id=1,slot_num=3,state=1,ieee_virtual_eth=0,port_vlan_id=1,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62803

lpar_name=06-BF99Z,lpar_id=1,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=2,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62804

lpar_name=06-BF99Z,lpar_id=1,slot_num=5,state=1,ieee_virtual_eth=0,port_vlan_id=3,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62805

lpar_name=06-BF99Z,lpar_id=1,slot_num=6,state=1,ieee_virtual_eth=0,port_vlan_id=4,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62806

lpar_name=06-BF99Z,lpar_id=1,slot_num=12,state=1,ieee_virtual_eth=0,port_vlan_id=68,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA6280C

lpar_name=aixlpar1,lpar_id=2,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=68,addl_vlan_ids=none,is_trunk=0,trunk_priority=0,is_required=0,mac_addr=F67D5345AD04

We needed to change port_vlan_id to 11 and addl_vlan_ids to 68. We also required the ieee_virtual_eth value set to 1.

First we removed the existing SEA adapter, as we would not be able to make changes to it while it was “active”. We then removed the adapter from slot 12 and then re-added it, again at slot 12, with port_vlan_id and addl_vlan_ids set to the desired values.

$ chhwres -m Server-8406-71Y-SN06BF99Z -p 06-BF99Z -r virtualio --rsubtype eth -s 12 -o r 

$ chhwres -m Server-8406-71Y-SN06BF99Z -p 06-BF99Z -r virtualio --rsubtype eth -s 12 -o a -a "ieee_virtual_eth=1,port_vlan_id=11,addl_vlan_ids=68,is_trunk=1,trunk_priority=1" -d 5

$ lshwres -r virtualio --rsubtype eth --level lpar

lpar_name=06-BF99Z,lpar_id=1,slot_num=3,state=1,ieee_virtual_eth=0,port_vlan_id=1,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62803

lpar_name=06-BF99Z,lpar_id=1,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=2,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62804

lpar_name=06-BF99Z,lpar_id=1,slot_num=5,state=1,ieee_virtual_eth=0,port_vlan_id=3,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62805

lpar_name=06-BF99Z,lpar_id=1,slot_num=6,state=1,ieee_virtual_eth=0,port_vlan_id=4,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62806

lpar_name=06-BF99Z,lpar_id=1,slot_num=12,state=1,ieee_virtual_eth=1,port_vlan_id=11,addl_vlan_ids=68,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA6280C

lpar_name=aixlpar1,lpar_id=2,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=68,addl_vlan_ids=none,is_trunk=0,trunk_priority=0,is_required=0,mac_addr=F67D5345AD04

Using the mkvdev command we created the SEA again. Then using the entstat command we found that the PVID and VLAN tags had been configured correctly.

$ mkvdev -sea ent6 -vadapter ent2 -default ent2 -defaultid 11

ent7 Available

en7

et7

$ entstat -all ent7 | grep -i vlan

VLAN Ids :

    VLAN Extract: False

    VLAN tagged filtering mode: Filter according to VLAN permit array

Max number of VLAN IDs per HEA port: 20

    VLAN Extract: False

    VLAN tagged filtering mode: Filter according to VLAN permit array

Max number of VLAN IDs per HEA port: 20

Invalid VLAN ID Packets: 5388

Port VLAN ID:    11

VLAN Tag IDs:    68

Once this was done, the client LPARs were able to ping the NIM master. The customer happily started installing AIX onto each of the blades Partitions.

A related link: http://www.aixmind.com/?p=2133

I received the following errors whilst running dsh on a NIM master recently.

root@nim1 : / # dsh -waixlpar1 date

0042-053 lsnim: there is no NIM object named "aixlpar1"

The node aixlpar1 is not defined in NIM database.

aixlpar1: Mon Aug  4 14:01:57 EET 2014

I had to set the following environment variable, shown below. By setting DSH_CONTEXT to DSH this prevented the dsh command from referring to the NIM database and instead forced it to query a user-defined node list.

root@nim1 : / # export DSH_CONTEXT=DSH

root@nim1 : / # dsh -waixlpar1 date

aixlpar1: Mon Aug  4 14:02:22 EET 2014

root@nim1 : / # env | grep -i dsh

DSH_CONTEXT=DSH

DSH_NODE_RSH=/usr/bin/ssh

root@nim1 : / # dsh -q

DSH:DCP_DEVICE_OPTS=

DSH:DCP_DEVICE_RCP=

DSH:DCP_NODE_OPTS=-q

DSH:DCP_NODE_RCP=/usr/bin/scp

DSH:DSH_CONTEXT=DSH

DSH:DSH_DEVICE_LIST=

DSH:DSH_DEVICE_OPTS=

DSH:DSH_DEVICE_RCP=

DSH:DSH_DEVICE_RSH=

DSH:DSH_ENVIRONMENT=

DSH:DSH_FANOUT=

DSH:DSH_LOG=

DSH:DSH_NODEGROUP_PATH=

DSH:DSH_NODE_LIST=/usr/local/etc/csmnodes.list

DSH:DSH_NODE_OPTS=

DSH:DSH_NODE_RCP=

DSH:DSH_NODE_RSH=/usr/bin/ssh

DSH:DSH_OUTPUT=

DSH:DSH_PATH=

DSH:DSH_REPORT=

DSH:DSH_SYNTAX=

DSH:DSH_TIMEOUT=

DSH:RSYNC_RSH=

Here’s another dsh tip I picked up. By default dsh will use the default port for ssh connections to nodes. For example, by default sshd listens on port 22 on an AIX node. I recently came across a customer environment where they had configured sshd to listen on port 6666 (not the real port number!). They wanted to use dsh from a NIM master which would connect to all the defined nodes in their custom list. When they ran it they got the following error message:

# dsh date

aixlpar1: ssh: connect to host aixlpar1 port 22: Connection refused

dsh:  2617-009 aixlpar1 remote shell had exit code 255

On the AIX node, we could see that sshd was listening on port 6666:

# netstat -a | grep 6666 | grep LIST

tcp6       0      0  *.6666                 *.*                    LISTEN

tcp4       0      0  *.6666                 *.*                    LISTEN

We needed to find a way to force dsh to use a different port number when starting the ssh connection. This was accomplished by setting the DSH_REMOTE_OPTS variable, as shown below.

[root@nim1]/ # export DSH_REMOTE_OPTS=-p6666

[root@nim1]/ # dsh date

aixlpar1: Tue Aug  5 17:37:16  2014

[root@nim1]/ # env | grep DSH

DSH_REMOTE_CMD=/usr/bin/ssh

DSH_NODE_LIST=/etc/ibm/sysmgt/dsm/nodelist

DSH_REMOTE_OPTS=-p6666

DSH_NODE_RSH=/usr/bin/ssh

DSH CONTEXT    

The DSH CONTEXT is the in-built context for all the DSH Utilities commands. It permits a user-defined node group database contained in the local file system. The DSH_NODEGROUP_PATH environment variable specifies the path to the node group database. Each file in this directory represents a node group, and contains one host name or TCP/IP address for each node that is a group member. Blank lines and comment lines beginning with a # symbol are ignored. If all nodes are requested for the DSH CONTEXT, a full node list is built from all groups in the DSH_NODEGROUP_PATH directory, and cached in /var/ibm/sysmgt/dsm/dsh/$DSH_NODEGROUP_PATH/AllNodes. This file is recreated each time a group file is modified or added to the DSH_NODEGROUP_PATH directory. Device targets are not supported in the DSH context.

http://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.cmds2/dsh.htm?lang=en

DSH_REMOTE_OPTS

Includes the options specified in the remote command when the command is forwarded to the remote nodes.

http://ps-2.kev009.com/wisclibrary/aix52/usr/share/man/info/en_US/a_doc_lib/cmds/aixcmds3/mgmtsvr.htm

I finally got PowerVP installed in my lab environment today. As part of the server agent installation process I needed to install the latest service pack (SP1) for PowerVP. After I downloaded the service pack from IBM Fix Central, I tried to run the install script from the command line on my AIX partition. It failed everytime I ran it, no matter what options I supplied the installer script!

[root@gibbo]/tmp/cg # ./PowerVP.bin.SP1 –i Silent

Preparing to install...

Extracting the installation resources from the installer archive...

Configuring the installer for this system's environment...

Launching installer...

Graphical installers are not supported by the VM. The console mode will be used instead...

===============================================================================

Choose Locale...

----------------

    1- Bahasa Indonesia

    2- Català

    3- Dansk

    4- Deutsch

  ->5- English

    6- Español

    7- Euskara

    8- Français

    9- Français (Canada)

   10- Italiano

   11- Nederlands

   12- Norsk

   13- Português

   14- Português  (Brasil)

   15- Suomi

   16- Svenska

CHOOSE LOCALE BY NUMBER:

=======================================================

Installer User Interface Mode Not Supported

The installer cannot run in this UI mode. To specify the interface mode, use the -i command-line option, followed by the UI mode identifier. The valid UI modes identifiers are GUI, Console, and Silent.

The only way I could get the SP installed was to use the GUI installer. I installed VNC server on my AIX partition, connected to the VNC server and then ran the installer script (as shown in the following screenshots). This worked fine.

The update fileset was extracted to the following location:                                                                                    

[root@gibbo]/IBM/PowerVP/PowerVP_Installation/PowerVP_Agent_Installation_Instructions/AIX # ls -ltr

total 60328

-rwxrwxr-x    1 root     system     30171648 Nov  6 11:28 powervp.1.1.0.0.bff

-rwxrwxr-x    1 root     system       699392 Dec  6 08:52 powervp.1.1.0.1.bff

-rwxrwxr-x    1 root     system         1185 Dec  9 08:53 README_Fixpack.html

-rwxrwxr-x    1 root     system         2001 Feb 17 14:19 README.html

-rw-r--r--    1 root     system          849 Feb 17 15:03 .toc

As instructed in the fixpack readme, I stopped the PowerVP processes before installing the service pack. I also unloaded the associated kernel extension, just in case.

# ps -ef | grep -i powervp

    root 4128956 6291676   0 14:56:23      -  0:00 /opt/ibm/powervp/syslet

    root 6291676       1   0 14:56:23      -  0:00 /bin/ksh /opt/ibm/powervp/PowerVP.sh

[root@gibbo]/ # kill 4128956 6291676

[root@gibbo]/ # cat /opt/ibm/powervp/PowerVP.sh

#!/bin/ksh

/opt/ibm/powervp/loadkex

/opt/ibm/powervp/syslet

/opt/ibm/powervp/unloadkex

Exit

[root@gibbo]/ # /opt/ibm/powervp/unloadkex

Then I installed the update (with smitty update_all).

                                                                                           COMMAND STATUS

Command: OK            stdout: yes           stderr: no

Before command completion, additional instructions may appear below.

[MORE...40]

        powervp.rte 1.1.0.1

Finished processing all filesets.  (Total time:  1 secs).

+-----------------------------------------------------------------------------+

                        Pre-commit Verification...

+-----------------------------------------------------------------------------+

Verifying requisites...done

Results...

SUCCESSES

---------

  Filesets listed in this section passed pre-commit verification

  and will be committed.

  Selected Filesets

  -----------------

  powervp.rte 1.1.0.1                         # PowerVP Fixpack for AIX

  << End of Success Section >>

+-----------------------------------------------------------------------------+

                          Committing Software...

+-----------------------------------------------------------------------------+

installp: COMMITTING software for:

        powervp.rte 1.1.0.1

Finished processing all filesets.  (Total time:  1 secs).

+-----------------------------------------------------------------------------+

                                Summaries:

+-----------------------------------------------------------------------------+

Installation Summary

--------------------

Name                        Level           Part        Event       Result

-------------------------------------------------------------------------------

powervp.rte                 1.1.0.1         USR         APPLY       SUCCESS

powervp.rte                 1.1.0.1         USR         COMMIT      SUCCESS

[root@gibbo]/ # lslpp -l powervp.rte

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  powervp.rte                1.1.0.1  COMMITTED  PowerVP Fixpack for AIX

Path: /etc/objrepos

  powervp.rte                1.1.0.0  COMMITTED  PowerVP for AIX

Next, I restarted the PowerVP agent.

[root@gibbo]/ # ps ax |grep powervp

[root@gibbo]/ # /etc/rc.d/rc2.d/SPowerVP

[root@gibbo]/ # ps ax |grep powervp

 4128966  pts/0 A     0:00 /opt/ibm/powervp/syslet

 9109542  pts/0 A     0:00 /bin/ksh /opt/ibm/powervp/PowerVP.sh

At the same time, I also updated the PowerVP GUI on my laptop to the latest service pack as well.

After that, I was good to go and PowerVP started up fine.

The PowerVP log file will tell you if the agent has started.

[root@gibbo]/var/log # cat powervp.log
2014-02-17 17:02:46.750 MSG0080: 10.2.55.99 - Running as the system level agent.
2014-02-17 17:02:47.860 MSG0082: Agent running, waiting for clients to request data.
 

By the way, if you try to start the PowerVP server agent on a system that isn't at the correct fimrware level, then you'll find the following message in /var/log/powervp.log:

# cat powervp.log
2014-02-14 15:28:01.844 Error - MSG0047: MSG0205: System does not support the required interfaces. Firmware version 7.7 or greater is required.. [syslet.C:6090]