Chris's AIX Blog

Following on from my previous post, on working with scalable VGs.

 

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/entry/working_with_aix_scalable_volume_groups1?lang=en

 

You may find yourself in a situation where you’d like to convert a VG to scalable.

 

For example, you might start off with a normal VG i.e. one that can support a maximum of 256 LVs and 32 PVs.

 

# mkvg -s 512M -y myvg hdisk5

myvg

 

# lsvg myvg

VOLUME GROUP:       myvg                     VG IDENTIFIER:  00f6027300004c0000000133aa24d1f6

VG STATE:           active                   PP SIZE:        512 megabyte(s)

VG PERMISSION:      read/write               TOTAL PPs:      99 (50688 megabytes)

MAX LVs:            256                      FREE PPs:       0 (0 megabytes)

LVs:                1                        USED PPs:       99 (50688 megabytes)

OPEN LVs:           1                        QUORUM:         2 (Enabled)

TOTAL PVs:          1                        VG DESCRIPTORS: 2

STALE PVs:          0                        STALE PPs:      0

ACTIVE PVs:         1                        AUTO ON:        yes

MAX PPs per VG:     32512

MAX PPs per PV:     1016                     MAX PVs:        32

LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no

HOT SPARE:          no                       BB POLICY:      relocatable

PV RESTRICTION:     none

 

Sometime later you discover you need to grow the volume group and add more hdisks. But you can’t because you’ve reached the maximum PV limit of 32.

 

 You examine the chvg man page and discover the –G flag! Of course you take note of the special considerations when converting to scalable.

 

Changes the volume group to Scalable VG format. This can accommodate up to 1024 physical volumes and 4096 logical volumes. Notes:

 

1    The -G flag cannot be used if there are any stale physical partitions.

 

2    Once the volume group is converted, it cannot be imported into AIX(R) 5.2 or lower versions.

 

3    The -G flag cannot be used if the volume group is varied on.

 

4    There must be enough free partitions available on each physical volume for the VGDA expansion for this operation to be successful.

 

5    Since the VGDA resides on the edge of the disk and it requires contiguous space for expansion, the free partitions are required on the edge of the disk. If those partitions are allocated for user usage, they will be migrated to other free partitions on the same disk. The rest of the physical partitions will be renumbered to reflect the loss of the partitions for VGDA usage. This will change the mappings of the logical to physical partitions in all the PVs of this VG. If you have saved the mappings of the LVs for a potential recovery operation, you should generate the maps again after the completion of the conversion operation. Also, if the backup of the VG is taken with the map option and you plan to restore using those maps, the restore operation may fail since the partition number may no longer exist (due to reduction). It is recommended that backup is taken before the conversion, and right after the conversion if the map option is utilized.

 

6    Because the VGDA space has been increased substantially, every VGDA update operation (creating a logical volume, changing a log ical volume, adding a physical volume, and so on) may take considerably longer to run.

 

Excellent, I can convert my VG to scalable.

 

The volume group must be varied off before it can be converted.

 

# chvg -G myvg

0516-1707 chvg: The volume group must be varied off during conversion to

        scalable volume group format.

0516-732 chvg: Unable to change volume group myvg.

 

# umount /myfs

 

# lsvg -l myvg

myvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

mylv                jfs2       99      99      1    closed/syncd  /myfs

 

# varyoffvg myvg

 

There must be enough free PPs in the volume group (on each hdisk) in order for the conversion to work. If there aren’t then you’ll see this error:

 

# chvg -G myvg

0516-1214 chvg: Not enough free physical partitions exist on hdisk5 for the

expansion of the volume group descriptor area. Migrate/reorganize to free up

1 partitions and run chvg again.

0516-732 chvg: Unable to change volume group myvg.

 

In this example, we vary on the VG again, and shrink the file system by 1 PP, giving us 1 PP free in the volume group. Remember, “There must be enough free partitions available on each physical volume for the VGDA expansion for this operation to be successful.”

 

# varyonvg myvg

 

# chfs -a size=-512M /myfs

Filesystem size changed to 102760448

Inlinelog size changed to 196 MB.

 

# lsvg myvg

VOLUME GROUP:       myvg                     VG IDENTIFIER:  00f6027300004c0000000133aa24d1f6

VG STATE:           active                   PP SIZE:        512 megabyte(s)

VG PERMISSION:      read/write               TOTAL PPs:      99 (50688 megabytes)

MAX LVs:            256                      FREE PPs:       1 (512 megabytes)

LVs:                1                        USED PPs:       98 (50176 megabytes)

OPEN LVs:           0                        QUORUM:         2 (Enabled)

TOTAL PVs:          1                        VG DESCRIPTORS: 2

STALE PVs:          0                        STALE PPs:      0

ACTIVE PVs:         1                        AUTO ON:        yes

MAX PPs per VG:     32512

MAX PPs per PV:     1016                     MAX PVs:        32

LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no

HOT SPARE:          no                       BB POLICY:      relocatable

PV RESTRICTION:     none

 

The VG is varied off again.

 

# varyoffvg myvg

# lsvg -o

rootvg

 

We convert the VG to scalable.

 

# chvg -G myvg

0516-1224 chvg: WARNING, once this operation is completed, volume group myvg

cannot be imported into AIX 5.2 or lower versions. Continue (y/n) ? y

0516-1216 chvg: Physical partitions are being migrated for volume group

descriptor area expansion.  Please wait.

0516-1712 chvg: Volume group myvg changed.  myvg can include up to 1024 physical volumes with 2097152 total physical partitions in the volume group.

 

We vary on the volume group and check that it is now scalable. You’ll notice that we’ve now lost that free PP in the VG as it has been used in the conversion process.

 

# varyonvg myvg

 

# lsvg myvg

VOLUME GROUP:       myvg                     VG IDENTIFIER:  00f6027300004c0000000133aa24d1f6

VG STATE:           active                   PP SIZE:        512 megabyte(s)

VG PERMISSION:      read/write               TOTAL PPs:      98 (50176 megabytes)

MAX LVs:            256                      FREE PPs:       0 (0 megabytes)

LVs:                1                        USED PPs:       98 (50176 megabytes)

OPEN LVs:           0                        QUORUM:         2 (Enabled)

TOTAL PVs:          1                        VG DESCRIPTORS: 2

STALE PVs:          0                        STALE PPs:      0

ACTIVE PVs:         1                        AUTO ON:        yes

MAX PPs per VG:     32768                    MAX PVs:        1024

LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no

HOT SPARE:          no                       BB POLICY:      relocatable

PV RESTRICTION:     none

 

 

If you are unable to free up a physical partition on a your hdisk (or hdisks) then you may need to use a tool like migratelp to move logical partitions around in order to free up some space. From the migratelp man page:

 

migratelp Command

 

Purpose

 

Moves allocated logical partition from one physical partition to another physical partition on a different physical volume.

 

 

 

Ever wondered what process just core dumped?

The AIX errpt command will tell you where the core file has been dumped and in most cases the program name:

# errpt -a | head -45
---------------------------------------------------------------------------
LABEL:          CORE_DUMP
IDENTIFIER:     C69F5C9B

Date/Time:       Wed Jul 22 13:41:52 EET 2009
Sequence Number: 2694
Machine Id:      00C01C704C00
Node Id:         hxaix14
Class:           S
Type:            PERM
Resource Name:   SYSPROC

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SOFTWARE PROGRAM

User Causes
USER GENERATED SIGNAL

        Recommended Actions
        CORRECT THEN RETRY

Failure Causes
SOFTWARE PROGRAM

        Recommended Actions
        RERUN THE APPLICATION PROGRAM
        IF PROBLEM PERSISTS THEN DO THE FOLLOWING
        CONTACT APPROPRIATE SERVICE REPRESENTATIVE

Detail Data
SIGNAL NUMBER
          11
USER'S PROCESS ID:
               1646842
FILE SYSTEM SERIAL NUMBER
          19
INODE NUMBER
       18736
CORE FILE NAME
/usr/sap/SMD/J98/SMDAgent/core
PROGRAM NAME
jlaunch

But what if the AIX error report has been cleared out and all you have is the core file. How do you know who generated the core file?

Once you've found the core file, run the following command to determine what command and/or process failed:

# cd /usr/sap/SMD/J98/SMDAgent/

# lquerypv -h core 6b0 64
000006B0   7FFFFFFF FFFFFFFF 7FFFFFFF FFFFFFFF  |................|
000006C0   00000000 000007D0 7FFFFFFF FFFFFFFF  |................|
000006D0   00120000 1A7743F8 00000000 00000003  |................|
000006E0   736C6565 70000000 00000000 00000000  |jlaunch.........|
000006F0   00000000 00000000 00000000 00000000  |................|
00000700   00000000 00000000 00000000 00000027  |...............'|
00000710   00000000 00000001 00000000 00000027  |...............'|

 

 

Starting with HMC V8R8.1.0.1 there’s a new, enhanced HMC interface available. You are given the choice of using either the Classic or Enhanced interface when you login to the HMC.

 

 

If you select ‘Enhanced’ you may find that the traditional DLPAR menu (Dynamic partitioning) has seemingly disappeared!  You may be expecting the “classic” DLPAR menu, as shown in the following image.

 

CLASSIC DLPAR

 

 

You will not find the ‘Dynamic partitioning’ option or interface.  Instead, in the Enhanced interface, you must select ‘Manage’ (as shown in the following image).

 

ENHANCED DLPAR

 

You are presented with a very different DLPAR interface. However, you can perform normal DLPAR operations as you have in the past. In the example (shown below) we can immediately see that the RMC connection to the partition is active and the partition is running. Here we could change the processor settings for the partition. For example, I could change the partition from Uncapped to Capped and then simply select either OK and/or Apply to make this change dynamically.

 

 

Learn about the differences between the Classic and Enhanced graphical user interface (GUI) in the Hardware Management Console (HMC).

 

http://www-01.ibm.com/support/knowledgecenter/POWER8/p8ha1/p8ha1_classicvsenhanced.htm?cp=POWER8%2F1-5-1-1-4

When troubleshooting Fibre Channel adapter issues on AIX (or VIOS), I often use the fcstat command to assist me in the process. The command may display error numbers similar to the following:

 

$ fcstat fcs0

 

Error opening device: /dev/fscsi0

errno: 00000045

 

The error number (errno) displayed can (in some cases) be used to identify the root cause of a problem. In the example above, the error number is 45. This number is in hex. If we convert this to decimal, the number is 69. Now, if we look for 69 in the /usr/include/errno.h file (on AIX), we discover that this error number relates to a "Network is down" event.

 

# fcstat fcs0

Error opening device: /dev/fscsi0

errno: 00000045

 

# echo "ibase=16; 45"|bc

69

 

# grep 69 /usr/include/errno.h

#define ENETDOWN        69      /* Network is down */

 

The AIX error report (errpt or errlog on VIOS) also tells me that there's some type of link error. This helps me focus my investigation toward the most likely problem area. In this case, I suspect either a physical link or cable problem between my FC adapter and the SAN switch.

 

IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION

7BFEEA1F   1116121816 T H fcs0           LINK ERROR

 

# errpt -aN fcs0 | grep -p Desc

...

Description

LINK ERROR

 

We can use the same process for other errors displayed by the fcstat command. For example:

 

# fcstat fcs0

Error opening device: /dev/fscsi0

errno: 00000046

 

# echo "ibase=16; 46"|bc

70

 

# grep 70 /usr/include/errno.h

#define ENETUNREACH     70      /* Network is unreachable */

 

This particular error may direct us to the solution outlined in the following tech note:

 

Fibre channel workspace takes over 2 minutes to load but returns no data

http://www-01.ibm.com/support/docview.wss?uid=swg21600064

 

 

fcstat Command (AIX 6.1)

http://www.ibm.com/support/knowledgecenter/ssw_aix_61/com.ibm.aix.cmds2/fcstat.htm

 

fcstat Command (AIX 7.1)

http://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.cmds2/fcstat.htm

 

fcstat Command (AIX 7.2)

http://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.cmds2/fcstat.htm

 

 

 

Here’s a question I received recently, that relates to virtual SCSI hdisk path priority.

 

Hi, Chris,

I use MPIO for our virtualization environment.

I'd like to check which path is currently used.

How can I get the result?

For example:

 

[nim:/]# lspath
Enabled hdisk0 vscsi0
Enabled hdisk0 vscsi1
Enabled hdisk2 vscsi2
Enabled hdisk1 vscsi1
Enabled hdisk1 vscsi0
Enabled hdisk3 vscsi2
Enabled hdisk2 vscsi3
Enabled hdisk3 vscsi3

 

How can I determine vscsi0 or vscsi1 is currently carrying data through VIOS and VIOC.

Thanks a lot!

 

By default the Virtual I/O Client (VIOC) will use the first VIOS for all VSCSI traffic. You can confirm which VSCSI adapter is being used by starting nmon (on the VIOC) and pressing the ‘a’ key. This will show which vscsi adapter is currently in use. e.g.

 

┌─topas_nmon──a=disk-Adapters────Host=lpar42p682e_pubRefresh=2 secs───18:38.04─┐

│ Disk-Adapter-I/O ────────────────────────────────────────────────────────────│

│Name          %busy     read    write        xfers Disks Adapter-Type         │

│vscsi3          0.0      0.0      0.0 KB/s     0.0 1     Virtual SCSI Client A│

│vscsi2        100.0      0.0 218560.0 KB/s  1707.5 1     Virtual SCSI Client A│

│TOTALS  2 adapters       0.0 218560.0 KB/s  1707.5 2     TOTAL(MB/s)=213.4    │

│──────────────────────────────────────────────────────────────────────────────│

 

Use the lspath command to determine the current path priority for a hdisk. By default it will be set to 1 for both adapters, meaning that traffic will go via the first path. Priority 1 is the highest priority, and you can define a priority from 1 to 255.

 

# lspath -AE -l hdisk2 -p vscsi2

priority 1 Priority True

 

# lspath -AE -l hdisk2 -p vscsi3

priority 1 Priority True

 

You can change the path priority of a hdisk using the chpath command. For example, we change the priority for vscsi2 to 255, so that vscsi3 now has the highest priority:

 

# chpath -l hdisk2 -a priority=255 -p vscsi2

# chpath -l hdisk2 -a priority=1 -p vscsi3

 

This will change the path priority for hdisk2 so that its traffic will use vscsi3 instead vscsi2.

 

┌─topas_nmon──b=Black&White──────Host=lpar42p682e_pubRefresh=2 secs───18:40.56─┐

│ Disk-Adapter-I/O ────────────────────────────────────────────────────────────│

│Name          %busy     read    write        xfers Disks Adapter-Type         │

│vscsi3        100.0      0.0 224384.5 KB/s  1753.0 1     Virtual SCSI Client A│

│vscsi2          0.0      0.0      0.0 KB/s     0.0 1     Virtual SCSI Client A│

│TOTALS  2 adapters       0.0 224384.5 KB/s  1753.0 2     TOTAL(MB/s)=219.1    │

│──────────────────────────────────────────────────────────────────────────────│

 

Now all the VSCSI traffic (for hdisk2) is traversing the vscsi3 adapter instead of vscsi2.

Here are some questions I received recently regarding VLAN tagging on the VIO server. My answers are shown in green.

 

“Hi Chris,

 

Q: I’m trying to understand when, where and why there would be the need to use ‘mkvdev –vlan (etc.) on the VIOS, and I’m wondering whether you would be able to clarify this for me, please.

 

Is it necessary to add the VLAN tag devices to the SEA, or is it suffice to just have them defined within the Virtual Ethernet itself which is part of the SEA?”

 

A: It is suffice to simply define the VLAN ids assigned to the Virtual Ethernet adapters associated with the SEA.

 

“Q: For completeness, on the rare occasions I have done this, I have added the VLAN’s to the Virtual Ethernet and also as VLAN devices on the VIOS (mkvdev –vlan etc.)”

 

A:  mkvdev –vlan is not necessary, unless the VIOS needs to communicate with hosts on different VLANs i.e. you need an IP address on the VIOS for each VLAN. This does not mean the SEA will bridge this VLAN traffic for VIOCs.

 

“Q: The reason I started thinking of this is, is because one of our customers wants to add new VLAN’s to their SEA, but they’re not running Power7 hardware. Therefore, the online method would be to add a new Virtual Adapter which contains the new VLAN ID’s to the VIOS using DLPAR, then use chdev –dev (etc.) on the SEA to include the new Virtual Ethernet.”

 

A: Agreed. The “IBM PowerVM Virtualization Managing and Monitoring” Redbook states: “If your system doesn’t support dynamic VLAN modifications and you are modifying the VLAN list of a virtual Ethernet adapter that is configured in a SEA with ha_mode enabled, the HMC will not allow you to reconfigure the list of VLANs on that interface. You will need to add an additional virtual Ethernet adapter and modify the virt_adapters list of the SEA, or modify the profile of both Virtual I/O Servers and re-activate both Virtual I/O Servers at the same time.”

 

“Q: From the phone call I had, it would appear that the VLAN tags are included on the Virtual Ethernet device, but have not been added to the SEA by running mkvdev –vlan (etc. ) on the VIOS’s.  This leads me to assume that the ‘mkvdev –vlan’ is only required if there is a requirement to access the VIOS itself from a particular VLAN. Am I right, or is there something I’m not understanding? I’m unable to find documentation that explains the answer. Do you happen to know?”

 

A: That is also my understanding (based on my experience). On page 483 of the “IBM PowerVM Virtualization Introduction and Configuration” Redbook , it states: “The addition of VLAN interfaces to the SEA adapter is only necessary if the VIO Server itself needs to communicate on these VLANs”.

 

“Q: Hi Chris,

We are trying to associate a new entX Virtual Ethernet Trunk Device to an existing SEA. The new device must be configured for VLAN tagging. The existing virtual Ethernet adapter that (is already associated with the SEA) is not configured for VLAN tagging. This device will remain associated to the SEA and continue to pass untagged packets to the already configured network.

 

Ultimately  the configuration we want would be two entX devices associated with the existing SEA. One entX device is configured for notagged packets and the other entX device is configured for tagging.

 

How do we configured this?”

 

A: What you are trying to do should work:

 

Existing setup:

 

VIOC1 boot1 PVID 20 ---> VIOS1 ent3 PVID 20 ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X

VIOC1 boot2 PVID 21 ---> VIOS2 ent3 PVID 21 ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X

 

Proposed setup:

 

VIOC1 boot1 PVID 20 ---> VIOS1 ent3 PVID 20 ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X (NO CHANGE)

VIOC1 boot2 PVID 21 ---> VIOS2 ent3 PVID 21 ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X (NO CHANGE)

 

VIOC2 boot1 PVID 22 ---> VIOS1 ent6 PVID 92*, VID 22, 802.1Q   ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X, VID=22 (NEW)

VIOC2 boot2 PVID 23 ---> VIOS2 ent6 PVID 93*,  VID 23, 802.1Q  ---> SEA ent5 ---> Physical Ethernet adapter connected to Network Switch Port PVID=X, VID=23  (NEW)

 

*The PVID can be any unused/throw away number. The new ent6 adapter would be configured similar to this:

 

 

Then you would run this command to associate the new ent6 with the existing SEA, ent5:

 

$ chdev -dev ent5 -attr virt_adapters=ent3,ent6

 

Have a look at the following sections of this Redbook:

 

3.2.3 Virtual I/O Server: Dynamically add or modify an existing adapter such that the required VLAN is listed in the 802.1q Additional VLAN fields (on POWER7 only).

 

3.2.2 Hardware Management Console: Dynamic VLAN modification in the GUI (on POWER7 only).

 

http://www.redbooks.ibm.com/redpieces/abstracts/sg247590.html?Open

 

This document (by Anthony English) is also a very useful document relating to VLANs, SEAs and VIOS:

 

http://www.ibm.com/developerworks/aix/library/au-managevlans/index.html?ca=drs-

 

 

Reply: “hmm ok I see what you are saying, I will give it a go and tell you how it turns out...thanks. ok finally got around to testing using a VIOS at DR site. Created new virtual adapter PVID 55 and VID 888 (ent9) then added it to the existing SEA as shown below:

 

 

accounting    enabled   Enable per-client accounting of network statistics                 True
ctl_chan      ent6      Control Channel adapter for SEA failover                           True
gvrp          no        Enable GARP VLAN Registration Protocol (GVRP)                      True
ha_mode       auto      High Availability Mode                                             True
jumbo_frames  no        Enable Gigabit Ethernet Jumbo Frames                               True
large_receive no        Enable receive TCP segment aggregation                             True
largesend     0         Enable Hardware Transmit TCP Resegmentation                        True
lldpsvc       no        Enable IEEE 802.1qbg services                                      True
netaddr       0         Address to ping                                                    True
pvid          40        PVID to use for the SEA device                                     True
pvid_adapter  ent4      Default virtual adapter to use for non-VLAN-tagged packets         True
qos_mode      disabled  N/A                                                                True
real_adapter  ent2      Physical adapter associated with the SEA                           True
thread        1         Thread mode enabled (1) or disabled (0)                            True
virt_adapters ent4,ent9 List of virtual adapters associated with the SEA (comma separated) True

 

VLAN Ids :
Invalid VLAN ID Packets: 0
Port VLAN ID:    40
VLAN Tag IDs:  None
Invalid VLAN ID Packets: 0
Port VLAN ID:    55
VLAN Tag IDs:   888
Invalid VLAN ID Packets: 0
Port VLAN ID:    91
VLAN Tag IDs:  None

 

“So looks ok. Thanks for your help”.

 

 

 

 

Recently I had the pleasure of configuring a couple of POWER7 720s for a customer. Each 720 was to host roughly 12 LPARs each. There were two VIO servers and a NIM server per 720.

 

Everything went along nicely and according to plan. In a few days we had both systems built. My steps for building the systems were almost identical to those described by Rob McNelly in a recent post.

All four VIO servers were running the latest VIOS code i.e. 2.2.0.10-FP-24 SP-01. All the client LPARs were running AIX 6.1 TL6 SP3. Each VIOS was configured with two FC paths to the SAN and the SAN storage device was an IBM DS5020.

 

Native AIX MPIO was in use on the VIOS and the AIX LPARs. I did not deploy SDDPCM on the VIOS as this is currently unsupported with the DS5020.

Once the LPAR builds were complete we performed a number of “integration tests”. These typically involve disconnecting network and SAN cables from each VIOS and observing how the VIOS and LPARs respond to and recover from these types of conditions.

 

One of the integration tests required that BOTH fibre cables be disconnected from the first VIOS and confirm that the client LPARs were not impacted i.e. that all I/O travelled via the second VIOS.

During the test we notice the following:

 

• I/O on the client LPAR would hang for approximately 5 minutes. Our test was simple enough, create a file in a file system. Eventually the I/O would continue after the 5 minute delay.

 

• The VIOS became sluggish and took some time to respond. The lspath command would take a very long time to return (or in some cases it would never return). During one test, the VIOS actually hung and had to be restarted (however we were not able to reproduce this issue again).

 

• When we reconnected both the cables, the paths would recover on the first VIOS after another 5 minutes. On the client however, it took roughly 20 minutes for the paths to recover. And yes, the hcheck_interval was set to 60 on all disks.

What's even more puzzling was that if we simply rebooted the first VIOS, everything worked as expected i.e. the client LPARs were not impacted, I/O continued as normal and when the first VIOS was back up, the paths on the client LPARs recovered quickly.

After doing some research, we discovered the following post on the IBM developerWorks AIX forum:

 

http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14472352&tstart=0

 

This post highlighted a number of things we needed to check and also confirmed several decisions we’d made during the design process, such as SDDPCM not being supported with DS5020 storage and VIOS (this was good as some people were starting to believe we should have installed SDDPCM to resolve this problem. I’d only be happy to do this if it was a supported combination and it’s not).

 

Finally we found the following IBM tech note that related directly to our issue.

 
IZ66020: ACTIVE/PASSIVE PCM CONTROLLER HCHECK SUPPORT
https://www-304.ibm.com/support/docview.wss?uid=isg1IZ66020

 

The following statement seemed to match our exact problem.

 

"For active/passive storage device, such as DS3K, DS4K, or DS5K if complete access is lost to the storage device, then it may take greater than 5 minutes to fail I/O. This feature is for Active/Passive storage devices, which are running with the AIX Default A/P PCM. This includes DS3K, DS4K, and DS5K family of devices.”

The new feature was described as follows.

 

“Added feature which health checks controllers, when an enabled path becomes unavailable, due to transport problems.  By default this feature is DISABLED.  To enable this feature set the following ODM attributes for the active/passive storage device.  Enabling this feature, results in faster I/O failure times.                                                      

cntl_delay_time:                                                        
Is the amount of time in seconds the storage device's controller(s) will be health checked after a transport failure.  At the end of this period, if no paths are detected as good, then all pending and subsequent I/O to the device will be failed, until the device health checker detects a failed path has returned.                                             
                                                                        
cntl_hcheck_int:                                                        
The first controller health check will only be issued after a storage fabric transport failure had been detected. cntl_hcheck_int is the amount of time in seconds, which the next controller health check command will be issued. This value must be less than the cntl_delay_time  (unless set to "0", disabled).                                                  

If you wish to allow the storage device 30 seconds to come back on the fabric (after leaving the fabric),  then you can set cntl_delay_time=30 and cntl_hcheck_int=2.     

         
The device, /dev/hdisk#, must not be in use, when setting the ODM values (or the chdev "-P" option must be used,  which requires a reboot).                                               
                                                                        
CAUTION: There are cases where the storage device may  reboot both of the controllers and become inaccessible for a period of time. If the controller health check sequence is enabled, then this may result in an I/O failure.  It is recommended to to make sure you have an mirrored volume           
to failover to, if you are running with controller health check enabled (especially with under 60 second cntl_delay_time). “

                     

And as I suspected the issue was related to the type of storage we were using. It appears the I/O delay was attributed to the following attributes on the DS5020 hdisks on the VIOS:

 

cntl_delay_time 0  Controller Delay Time            True
cntl_hcheck_int 0  Controller Health Check Interval True

 

Based on the tech note, I attempted several tests with various values for both parameters e.g.

 

$ chdev -dev hdiskX -attr cntl_delay_time=30                        
$ chdev -dev hdiskX -attr cntl_hcheck_int=2  

 

After making the changes to the hdisks on all VIOS, I performed the same test i.e. disconnected BOTH fibre cables from the first VIOS and continued to write a file to a file system on the client LPARs. By modifying these values on all the DS5020 disks, on all the VIO servers, the I/O delay was reduced to seconds rather than five minutes!

 

The following attributes were used for the hdisks and adapters in the final configuration.

 

On the VIO servers:

 

$ lsdev -type disk | grep DS5020

hdisk3           Available   MPIO DS5020 Disk

hdisk4           Available   MPIO DS5020 Disk

hdisk5           Available   MPIO DS5020 Disk

hdisk6           Available   MPIO DS5020 Disk

hdisk7           Available   MPIO DS5020 Disk

hdisk8           Available   MPIO DS5020 Disk

hdisk9           Available   MPIO DS5020 Disk

hdisk10          Available   MPIO DS5020 Disk

hdisk11          Available   MPIO DS5020 Disk

hdisk12          Available   MPIO DS5020 Disk

hdisk13          Available   MPIO DS5020 Disk

hdisk14          Available   MPIO DS5020 Disk

hdisk15          Available   MPIO DS5020 Disk

hdisk16          Available   MPIO DS5020 Disk

hdisk17          Available   MPIO DS5020 Disk

hdisk18          Available   MPIO DS5020 Disk

hdisk19          Available   MPIO DS5020 Disk

hdisk20          Available   MPIO DS5020 Disk

hdisk21          Available   MPIO DS5020 Disk

hdisk22          Available   MPIO DS5020 Disk

hdisk23          Available   MPIO DS5020 Disk

hdisk24          Available   MPIO DS5020 Disk

hdisk25          Available   MPIO DS5020 Disk

hdisk26          Available   MPIO DS5020 Disk

hdisk27          Available   MPIO DS5020 Disk

hdisk28          Available   MPIO DS5020 Disk

hdisk29          Available   MPIO DS5020 Disk

hdisk30          Available   MPIO DS5020 Disk

hdisk31          Available   MPIO DS5020 Disk

hdisk32          Available   MPIO DS5020 Disk

 

 

$ lsdev -dev hdisk3 -attr

attribute       value                                                          description                      user_settable

 

PCM             PCM/friend/otherapdisk                                         Path Control Module              False

PR_key_value    none                                                           Persistant Reserve Key Value     True

algorithm       fail_over                                                      Algorithm                        True

autorecovery    no                                                             Path/Ownership Autorecovery      True

clr_q           no                                                             Device CLEARS its Queue on error True

cntl_delay_time 15                                                             Controller Delay Time            True

cntl_hcheck_int 2                                                              Controller Health Check Interval True

dist_err_pcnt   0                                                              Distributed Error Percentage     True

dist_tw_width   50                                                             Distributed Error Sample Time    True

hcheck_cmd      inquiry                                                        Health Check Command             True

hcheck_interval 60                                                             Health Check Interval            True

hcheck_mode     nonactive                                                      Health Check Mode                True

location                                                                       Location Label                   True

lun_id          0x11000000000000                                               Logical Unit Number ID           False

lun_reset_spt   yes                                                            LUN Reset Supported              True

max_retry_delay 60                                                             Maximum Quiesce Time             True

max_transfer    0x40000                                                        Maximum TRANSFER Size            True

node_name       0x20040080e5187564                                             FC Node Name                     False

pvid            00f6482f7869e92d0000000000000000                               Physical volume identifier       False

q_err           yes                                                            Use QERR bit                     True

q_type          simple                                                         Queuing TYPE                     True

queue_depth     24                                                             Queue DEPTH                      True

reassign_to     120                                                            REASSIGN time out value          True

reserve_policy  no_reserve                                                     Reserve Policy                   True

rw_timeout      30                                                             READ/WRITE time out value        True

scsi_id         0x11600                                                        SCSI ID                          False

start_timeout   60                                                             START unit time out value        True

unique_id       3E21360080E50001875DE000005224D2CD63B0F1814      FAStT03IBMfcp Unique device identifier         False

ww_name         0x20150080e5187564                                             FC World Wide Name               False

 

 

$ lsdev -dev fscsi0 -attr

attribute    value     description                           user_settable

 

attach       switch    How this adapter is CONNECTED         False

dyntrk       yes       Dynamic Tracking of FC Devices        True

fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True

scsi_id      0x11400   Adapter SCSI ID                       False

sw_fc_class  3         FC Class for Fabric                   True

 

 

$ r oem

oem_setup_env

# manage_disk_drivers

Device           Present Driver     Driver Options

2810XIV          AIX_AAPCM          AIX_AAPCM,AIX_non_MPIO

DS4100           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4200           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4300           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4500           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4700           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS4800           AIX_APPCM          AIX_APPCM,AIX_fcparray

DS3950           AIX_APPCM          AIX_APPCM

DS5020           AIX_APPCM          AIX_APPCM

DS5100/DS5300AIX_APPCM AIX_APPCM          AIX_APPCM

DS3500           AIX_APPCM          AIX_APPCM

Usage :

        manage_disk_drivers [-l]

        manage_disk_drivers -d device -o driver_option

        For entries with multiple model names use the first one listed.

        Ex. DS5100/DS5300 use DS5100.

        manage_disk_drivers –h

 

# mpio_get_config -Av

Frame id 0:

    Storage Subsystem worldwide name: 609e50018345de00004da7998

    Controller count: 2

    Partition count: 1

    Partition 0:

    Storage Subsystem Name = 'MyApp-DS5020'

        hdisk      LUN #   Ownership          User Label

        hdisk3        17   B (preferred)      LPAR1

        hdisk4        18   A (preferred)      LPAR1datavg

        hdisk5        19   A (preferred)      LPAR1appvg

        hdisk6        16   B (preferred)      LPAR2

        hdisk7        20   A (preferred)      LPAR2datavg

        hdisk8        21   B (preferred)      LPAR2data3vg

        hdisk9        15   A (preferred)      LPAR3

        hdisk10       14   B (preferred)      LPAR4

        hdisk11       22   B (preferred)      LPAR5

        hdisk12       23   A (preferred)      LPAR6

        hdisk13       13   B (preferred)      LPAR6datavg

        hdisk14       12   A (preferred)      LPAR7

        hdisk15       24   B (preferred)      LPAR7datavg

        hdisk16       25   A (preferred)      LPAR8

        hdisk17       11   B (preferred)      LPAR8datavg

        hdisk18       10   A (preferred)      LPAR8datavg

        hdisk19        9   B (preferred)      LPAR8datavg

        hdisk20        8   A (preferred)      LPAR8datavg

        hdisk21       26   A (preferred)      LPAR9

        hdisk22        7   A (preferred)      LPAR9datavg

        hdisk23        6   B (preferred)      LPAR9datavg

        hdisk24       27   B (preferred)      LPAR9appvg

        hdisk25       28   A (preferred)      LPAR9binvg

        hdisk26        5   A (preferred)      LPAR10

        hdisk27        4   B (preferred)      LPAR10datavg

        hdisk28       29   B (preferred)      LPAR10datavg

        hdisk29       30   A (preferred)      LPAR11

        hdisk30        3   A (preferred)      LPAR11datavg

        hdisk31       32   B (preferred)      LPAR12

        hdisk32       50   B (preferred)      LPAR12datavg

 

On the VIO clients:

 

- Changed the following attributes for all virtual SCSI disks to:

 

# lsattr -El hdisk0

PCM             PCM/friend/vscsi                 Path Control Module        False

algorithm       fail_over                        Algorithm                  True

hcheck_cmd      test_unit_rdy                    Health Check Command       True

hcheck_interval 60                               Health Check Interval      True

hcheck_mode     nonactive                        Health Check Mode          True

max_transfer    0x40000                          Maximum TRANSFER Size      True

pvid            00f6482f7869e92d0000000000000000 Physical volume identifier False

queue_depth     24                               Queue DEPTH                True

reserve_policy  no_reserve                       Reserve Policy             True

 

- Changed the following attributes for all virtual SCSI adapters to:

 

# lsattr -El vscsi0

vscsi_err_recov fast_fail N/A                       True

vscsi_path_to   30        Virtual SCSI Path Timeout True

 

Note: With regard to SDDPCM support for DS5020 with VIOS, I’ve been referring to the following IBM website:

 

http://www-01.ibm.com/support/docview.wss?uid=ssg1S4000201

 

The site has a link to the SDDPCM readme file for DS5020 storage (under SDDPCM Package for DS5000):

 

ftp://ftp.software.ibm.com/storage/subsystem/aix/2.5.2.0/sddpcm.readme.2.5.2.0.txt

 

The readme states:

 

Note: VIOS is not supported with SDDPCM on DS4000/DS5000/DS5020/DS3950 subsystem devices.

 

During our research we had this confirmed by IBM support that, at this time, SDDPCM is not supported
on VIOS with DS5000.

 

I hope this helps others that may be about to implement this type of storage with a VIOS.

Here’s a nice little matrix of current VIOS levels and the corresponding AIX level.

 

 

The following link also has some fantastic information relating to NIM installation and backup of a Virtual I/O Server (VIOS).

https://www-304.ibm.com/support/docview.wss?uid=isg3T1011386

It has an official answer for one of the questions I’m asked the most:

Q: Do I need to use an lpp_source to install the VIO server using NIM?

A: No. All of the device drivers and software packages needed to install the VIO server are in the VIO server mksysb.

I also recommend Rob McNelly’s latest blog on installing VIO servers.

http://ibmsystemsmag.blogs.com/aixchange/2011/01/an-unusual-vios-install.html

 

Everyone knows about the VIOS advisor (the vios_advisor utility), right?

 

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power%20Systems/page/VIOS%20Advisor

 

And most of us know that this functionality was officially included in a recent release of the VIOS (with the part command), right?

 

https://www.ibm.com/developerworks/community/blogs/cgaix/entry/the_new_vios_part_command?lang=en

 

But did you know that the VIOS advisor provides an extra panel of information that is not currently included with the new part command?

 

The extra panel provides an overview your Shared Ethernet Adapter (SEA) configuration.

 

 

This “new” panel provides advice (and in this case) warnings related to the current SEA configuration on a VIOS. In this example, both large send and receive are not enabled on the SEA. Clicking on the warning icon, I see the following message:

 

 

I believe that a future release of the VIOS will include an update to the part command which will provide a similar report. But for now, if you’d like to generate this report, you’ll need to download the original VIOS advisor and run it on your VIOS.

Last month I applied the latest fix pack for VIOS 2.1 i.e. FP22, to one of our lab VIOS.

I came across the following error while doing so: 

 

Renaming Media Repository Logical Volume:

 

VMLibrary_LV --> VMLibrary

umount: error unmounting /dev/VMLibrary_LV: Device busy

 

Media Repository failed to unmount. We cannot continue

until it can be successfully unmounted. Please address

and retry.

 

update:  Failed while executing the ios.cli.rte.post_u script.

 

installp:  The installation has FAILED for the "root" part

        of the following filesets:

        ios.cli.rte 6.1.4.1

 

 

Despite the error, the ioslevel command reported that the fix pack had been applied. This was misleading, as a critical fileset had not been updated i.e. ios.cli.rte.

 

$ ioslevel

2.1.2.10-FP-22

 

To workaround this problem I removed the Virtual Media Repository on the VIOS.

 

$ rmrep

 

 

As the ios.cli.rte fileset had not installed properly, I had to reinstall the base fileset and then apply the update (fix pack) again.

 

  ----

    1  Total to be installed

 

+-----------------------------------------------------------------------------+

                         Installing Software...

+-----------------------------------------------------------------------------+

 

installp: APPLYING software for:

        ios.cli.rte 6.1.2.0

 

 

. . . . . << Copyright notice for ios.cli >> . . . . . . .

 Licensed Materials - Property of IBM

 

 5765G3400

   Copyright International Business Machines Corp. 2004, 2008.

 

 All rights reserved.

 US Government Users Restricted Rights - Use, duplication or disclosure

 restricted by GSA ADP Schedule Contract with IBM Corp.

. . . . . << End of copyright notice for ios.cli >>. . . .

 

0 * * * * /usr/ios/cli/cron_mail_check.sh 2>/dev/null

0 * * * * /usr/ios/cli/cron_mail_check.sh 2>/dev/null

0 * * * * /usr/ios/cli/cron_mail_check.sh 2>/dev/null

root

padmin

Finished processing all filesets.  (Total time:  18 secs).

 

+-----------------------------------------------------------------------------+

                                Summaries:

+-----------------------------------------------------------------------------+

 

Installation Summary

--------------------

Name                        Level           Part        Event       Result

-------------------------------------------------------------------------------

ios.cli.rte                 6.1.2.0         USR         APPLY       SUCCESS

ios.cli.rte                 6.1.2.0         ROOT        APPLY       SUCCESS

 

 

# lslpp -l ios.cli.rte

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  ios.cli.rte                6.1.4.1  COMMITTED  Virtual I/O Technologies

 

Path: /etc/objrepos

  ios.cli.rte                6.1.4.1  COMMITTED  Virtual I/O Technologies

 

 

Then I recreated the media repository with mkrep.

 

$ mkrep -sp rootvg -size 10G

 

At the time I thought perhaps I had done something wrong. I forgot all about it, until now. Today (a month after my experience), IBM updated the VIOS support page with the following information:

 

http://www14.software.ibm.com/webapp/set2/sas/f/vios/download/fbmr.html

 

Known issues and limitations

File Backed Optical Media Repository: If you use the File Backed Optical Media Repository, you must unload any media images before you apply Fix Pack 22.1. If you have any media images loaded, the fix pack installation fails for the

ios.cli.rte fileset. The failure may not be apparent, because the ioslevel will be updated. Refer to this page for information to help you resolve this problem.

 

And they  re-released FP22 as FP22.1.

 

http://www14.software.ibm.com/webapp/set2/sas/f/vios/download/fixpack.html

 

Fixes in Fix Pack 22.1 (VIOS V2.1.2.10 FP-22.1) include:

            Fixed problem with Electronic Service Agent in IVM

            Fixed problem in bus device driver for NPIV

            Fixed problem in device driver pertaining to reading SSD disks (MPIO fixes)

            Fixed problem in xmtopas (install problem)

            Fixed packaging problem with the Media Repository

            Fixed problem with backupios ? fails to give adequate details on stdout errors

 

 

Just the other day, I needed to use the AIX splitvg command in order to copy some data from one system to another.

I thought I’d share the experience here.

The splitvg command can split a single mirror copy of a fully mirrored volume group into a separate “snapshot” volume group.

From the man page:

The original volume group VGname will stop using the disks that are now part of the snapshot volume group SnapVGname. Both volume groups will keep track of the writes within the volume group so that when the snapshot volume group is rejoined with the original volume group consistent data is maintained across the rejoined mirrors copies. Notes:

1         To split a volume group, all logical volumes in the volume group must have the target mirror copy and the mirror must exist on a disk or set of disks. Only the target mirror copy must exist on the target disk or disks.

 

2         The splitvg command will fail if any of the disks to be split are not active within the original volume group.

 

3         In the unlikely event of a system crash or loss of quorum while running this command, the joinvg command must be run to rejoin the disks back to the original volume group.

 

4         There is no concurrent or enhanced concurrent mode support for creating snapshot volume groups.

 

5         New logical volumes and file system mount points will be created in the snapshot volume group.

 

6         The splitvg command is not supported for the rootvg.

 

7         The splitvg command is not supported for a volume group that has an active paging space.

 

8         When the splitvg command targets a concurrent-capable volume group which is varied on in non-concurrent mode, the new volume group that is created will not be varied on when the splitvg command completes. The new volume group must be varied on manually.

 

So, looking at point 4, above, if you are using enhanced concurrent volume groups (for example with PowerHA), then you will not be able to use the splitvg command. This is disappointing as this would have been very handy in some of the large PowerHA systems I have worked with…..perhaps this will be supported in the future?

Anyway, back to my example. I had wanted to break-off one of the mirrors of a mirrored volume group and then assign the “split” volume group to another host to copy some data off.

The volume group datavg contained two disks, hdisk0 and hdisk3, as shown in the lspv output below.

 

# lspv
hdisk1          00c01c705bdc6136                    old_rootvg
hdisk0          00c01c70c050810f                    datavg              active
hdisk2          00c01c7018a47201                    rootvg              active
hdisk3          00c01c70fed9e41a                    datavg              active

 

There were only two logical volumes (loglvl00, jfs2 log and fslv00, data) and a single file system (/data) in this volume group.

 

# lsvg -l datavg
datavg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
loglv00             jfs2log    1       2       2    open/syncd    N/A
fslv00              jfs2       16      32      2    open/syncd    /data

 

The volume group datavg, was mirrored across hdisk0 and hdisk3, as shown in the lsvg output below.

 

# lsvg -p datavg
datavg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            583         566         117..100..116..116..117
hdisk3            active            583         566         117..100..116..116..117

 

The logical volumes for the /data file system and the JFS2 log were both mirrored, as shown in the lslv/lspv output.

 

# lslv fslv00
LOGICAL VOLUME:     fslv00                 VOLUME GROUP:   datavg
LV IDENTIFIER:      00c01c7000004c0000000122e9038480.2 PERMISSION:     read/write
VG STATE:           active/complete        LV STATE:       opened/syncd
TYPE:               jfs2                   WRITE VERIFY:   off
MAX LPs:            512                    PP SIZE:        256 megabyte(s)
COPIES:             2                      SCHED POLICY:   parallel
LPs:                16                     PPs:            32
STALE PPs:          0                      BB POLICY:      relocatable
INTER-POLICY:       minimum                RELOCATABLE:    yes
INTRA-POLICY:       middle                 UPPER BOUND:    32
MOUNT POINT:        /data                    LABEL:          /data
MIRROR WRITE CONSISTENCY: on/ACTIVE
EACH LP COPY ON A SEPARATE PV ?: yes
Serialize IO ?:     NO

 

# lslv loglv00
LOGICAL VOLUME:     loglv00                VOLUME GROUP:   datavg
LV IDENTIFIER:      00c01c7000004c0000000122e9038480.1 PERMISSION:     read/write
VG STATE:           active/complete        LV STATE:       opened/syncd
TYPE:               jfs2log                WRITE VERIFY:   off
MAX LPs:            512                    PP SIZE:        256 megabyte(s)
COPIES:             2                      SCHED POLICY:   parallel
LPs:                1                      PPs:            2
STALE PPs:          0                      BB POLICY:      relocatable
INTER-POLICY:       minimum                RELOCATABLE:    yes
INTRA-POLICY:       middle                 UPPER BOUND:    32
MOUNT POINT:        N/A                    LABEL:          None
MIRROR WRITE CONSISTENCY: on/ACTIVE
EACH LP COPY ON A SEPARATE PV ?: yes
Serialize IO ?:     NO

 

# lspv -l hdisk0
hdisk0:
LV NAME               LPs     PPs     DISTRIBUTION          MOUNT POINT
fslv00                16      16      00..16..00..00..00    /data
loglv00               1       1       00..01..00..00..00    N/A

 

# lspv -l hdisk3
hdisk3:
LV NAME               LPs     PPs     DISTRIBUTION          MOUNT POINT
fslv00                16      16      00..16..00..00..00    /data
loglv00               1       1       00..01..00..00..00    N/A

 

Using the splitvg command I was able to break-off one of the disks from the mirrored pair. This created a new volume group on hdisk3 called vg00.

 

# splitvg -c1 datavg
# lspv
hdisk1          00c01c705bdc6136                    old_rootvg
hdisk0          00c01c70c050810f                    datavg            active
hdisk2          00c01c7018a47201                    rootvg          active
hdisk3          00c01c70fed9e41a                    vg00            active

 

The new volume group contains a new logical volume (pre-fixed with fs i.e. fsfslv00) and a file system (pre-fixed with /fs i.e. /fs/data). I can mount this file system and access the data in the file system and create and/or modify files (as shown below).

 

# mount /fs/data
Replaying log for /dev/fsfslv00.

# cd /fs/data
# ls -ltr
total 112
drwxrwxrwx    2 root     system        53248 Jan 30 2009  AIX61TL2SP2
drwxr-xr-x    2 root     system          256 Aug 05 15:24 lost+found
-rw-r--r--    1 root     system            0 Sep 15 10:29 2

# touch 3
# ls -ltr
total 112
drwxrwxrwx    2 root     system        53248 Jan 30 2009  AIX61TL2SP2
drwxr-xr-x    2 root     system          256 Aug 05 15:24 lost+found
-rw-r--r--    1 root     system            0 Sep 15 10:29 2
-rw-r--r--    1 root     system            0 Sep 15 10:46 3

 

At this point I was able to export the volume group and import it on another system. I had to re-map the Virtual SCSI disk on my VIOS first.  

 

# cd
# umount /fs/data

# varyoffvg vg00
# exportvg vg00

# rmdev –dl hdisk3

; Re-map the disk at the VIOS layer to the other AIX LPAR.

; Configure the disk on the other LPAR (with cfgmgr) and import the volume group (with importvg).

 

Once I was finished with vg00 on the other LPAR, I re-mapped the disk to the original AIX LPAR, configured the disk and then re-joined the disk to the datavg volume group, with the joinvg command.

 

; Export the VG and remove the disk on the other AIX LPAR.

; Re-map the disk at the VIOS layer to the original AIX LPAR.

# cfgmgr

# lsdev –Cc | grep hdisk3

hdisk3 Available  Virtual SCSI Disk Drive

 

# joinvg datavg
# lspv
hdisk1          00c01c705bdc6136                    old_rootvg
hdisk0          00c01c70c050810f                    datavg            active
hdisk2          00c01c7018a47201                    rootvg          active
hdisk3          00c01c70fed9e41a                    datavg            active

 

The volume group is now fully mirrored again. However the partitions are stale and need to be sync’ed. I manually re-sync the volume group with the syncvg command.

 

# lsvg -l datavg
datavg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
loglv00             jfs2log    1       2       2    open/stale    N/A
fslv00              jfs2       16      32      2    open/stale    /data

# syncvg –v datavg

# lsvg -l datavg
datavg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
loglv00             jfs2log    1       2       2    open/syncd    N/A
fslv00              jfs2       16      32      2    open/syncd    /data

 

More information can be found here: http://www-01.ibm.com/support/docview.wss?uid=isg3T1010934

 

 

                                                                       

We all know that nimadm and multibos don’t play well together (at least that’s the case right now). If you're planning on migrating a system to a newer version of AIX (6.1 to 7.1 for example) you must ensure that your BOS LVs are reset to their traditional names and that all standby BOS instances have been removed. It’s clear that if you plan on using nimadm, to migrate a NIM client to a new version of AIX, you should first remove any and all instances/remnants of multibos, on the client system, before you start.

 

What’s not immediately clear is that even the NIM master should be cleansed of all evidence of multibos activity.

 

For example, let’s say you use multibos to install a new TL and SP on your NIM master e.g. AIX 7.1 TL1 SP4 to AIX 7.1 TL1 SP5. You reboot the system and run from the standby BOS instance (leaving the original instance in rootvg, just in case you need to back out).

 

You then attempt to migrate a NIM client, using nimadm, to a new version of AIX. The operation fails at phase 8, with the following error message:

 

+---------------------------------------------------------------+

Executing nimadm phase 8.                                              

+---------------------------------------------------------------+

Creating client boot image.

ls: 0653-341 The file /dev/hd5 does not exist.                         

 

At first you might think that this error relates to the NIM client. You check the client system…..there’s absolutely no evidence of multibos on the system. Good.

 

The issue lies with the fact that the NIM master is enabled for multibos and running from the standby instance.  Restarting the NIM master, on the original instance, booting from hd5, rather than bos_hd5 and then removing the standby instance (multibos –R), will resolve the issue. Now, nimadm will run without error.

 

AIX Updates With Multibos

http://www.ibmsystemsmag.com/aix/trends/aix/AIX-Updates-With-Multibos/

 

Working With Multibos

http://www.ibmsystemsmag.com/aix/administrator/systemsmanagement/Working-With-multibos/

 

Migrating to AIX 7.1 with nimadm

http://www.ibm.com/developerworks/aix/library/au-aix-nimadm/index.html

 

Migrating to AIX 6.1 with nimadm

http://www.ibm.com/developerworks/aix/library/au-migrate_nimadm/?S_TACT=105AGY20&S_CMP=HPU

 

 

At the IBM Technical Symposium in Sydney last week, a person approached me to discuss NIM and some of its capabilities. During the conversation we discussed how NIM could be used to copy files from the NIM master to its NIM clients. I promised to send them some information on how to achieve this ASAP. They were smart and followed up with an email the next day! They’d even tried to configure this on their systems but hit a small problem.

 

"Hello Chris,
 
So trying to push out a new netbackup tar file to one of our nim clients (usually we do all sort of stuff via our SSH deployment server). But decided to use NIM.
 
So I created the resource file_res called netbackup. Where the tar file is placed on the master

# nim -o define -t file_res -a location=/tmp/netbackup -a dest_dir=/tmp/netbackup -a server=master netbackup
 
# lsnim -l netbackup
netbackup:
   class       = resources
   type        = file_res
   Rstate      = ready for use
   prev_state  = unavailable for use
   location    = /tmp/netbackup
   alloc_count = 0
   server      = master
   dest_dir    = /tmp/netbackup

But for the life of me I cannot find anywhere under NIM to allocate the resource to the client i.e. to push the file out..
I though it would be under install software, but the resource is not listed only lpp_sources.

Do you have any ideas, this one I am stuck on. 

Can I actually push files out by themselves? Thanks"

 

The answer is yes! You can define a file_res resource on the NIM master. From the AIX 7.2 Knowledge Centre:

 

"A file_res resource is where NIM allows for resource files to be stored on the server. When the resource is allocated to a client, a copy of the directory contents is placed on the client at a location that is specified by the dest_dir attribute."

"-a location=Value     Specifies the full path name of the directory on the NIM server. This path is used as a source directory among clients.
-a dest_dir=Value     Specifies the full path name of the directory on the NIM client. This path is where the source directory is recursively copied into.

 

Notes: If the target directory does not exist on the destination machine, the entire source directory contents are copied (including the hidden files in the top-level directory). If the target directory exists on the destination machine, the source directory contents are copied (excluding the hidden files in the top-level directory)."

 

http://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.install/defining_file_res_resource.htm
 
Essentially, you can place the files you need to distribute to your NIM clients, into a directory on the NIM master. Then you can create a file_res NIM resource that points to this directory. After that, you can allocate this resource to the NIM client or NIM machine group, and run a NIM customisation operation against the client (or machine group). This will copy the directory and all its files to the NIM client. Pretty cool really! 

 

Here's an example. I want to distribute (copy) all of the files located in /usr/local/etc, to a NIM client. The original (source) directory and files reside on the NIM master.

 

root@MASTER / # ls –l /usr/local/etc/
total 16
-rw-r--r--    1 root     system          542 Aug 22 09:53 logger.conf
-rw-r--r--    1 root     system         3219 Aug 22 10:41 fugazi.cf

 

First, I'll create a new file_res NIM resource, called soe_conf.

 

root@MASTER / # nim -o define -t file_res -a location=/usr/local/etc -a dest_dir=/usr/local/etc -a server=master soe_conf

root@MASTER / # lsnim -t file_res
soe_conf     resources       file_res

# lsnim -l soe_conf
soe_conf:
   class       = resources
   type        = file_res
   dest_dir    = /usr/local/etc
   Rstate      = ready for use
   prev_state  = unavailable for use
   location    = /usr/local/etc
   alloc_count = 0
   server      = master

 

Now I can allocate the new resource to the NIM client named AIXmig.

root@MASTER / # nim -o allocate -a file_res=soe_conf AIXmig

root@MASTER / # lsnim -l AIXmig
AIXmig:
   class          = machines
   type           = standalone
   connect        = nimsh
   file_res       = soe_conf
   platform       = chrp
   netboot_kernel = 64
   if1            = VLAN50 AIXmig 0
   cable_type1    = N/A
   Cstate         = ready for a NIM operation
   prev_state     = not running
   Mstate         = currently running
   cpuid          = 00F94F584C00
   control        = master

 

Before I customise the client, the /usr/local/etc directory does NOT exist on the NIM client.

 

root@AIXmig / # ls -ltr /usr/local/etc
ls: 0653-341 The file /usr/local/etc does not exist.

 

On the NIM master, I issue the "nim -o cust" operation against the NIM client AIXmig.

 

root@MASTER /# nim -o cust AIXmig

 

Looking again on the NIM client, I now find the directory and all of its files have been copied to the correct location on the host.

root@AIXmig / # ls –l /usr/local/etc
total 16
-rw-r--r--    1 root     system          542 Aug 22 09:53 logger.conf
-rw-r--r--    1 root     system         3219 Aug 22 10:41 fugazi.cf

 

The file_res resource, is automatically deallocated from the NIM client.

 

root@MASTER / # lsnim -l AIXmig
AIXmig:
   class          = machines
   type           = standalone
   connect        = nimsh
   platform       = chrp
   netboot_kernel = 64
   if1            = VLAN50 AIXmig 0
   cable_type1    = N/A
   Cstate         = ready for a NIM operation
   prev_state     = customization is being performed
   Mstate         = currently running
   cpuid          = 00F94F584C00
   Cstate_result  = success
     
If you’re looking for a fast and cheap way of copying a bunch of files from a central location to one or more servers, then this is a very good option for AIX administrators. It’s like a “poor mans” file collections; similar to what we once had with Cluster System Management (CSM, which is now discontinued) and PowerHA File Collections (but not as powerful or configurable). 
   
  
   

You can use the chhwres command, on the HMC, to “free up” allocated processor and memory resources from your logical partitions.

 

I may want to do this if I need to improve the affinity placement of a partition on a POWER7 system or if I need to free up resources from trial capacity on demand in order to re-use the resources with On/Off capacity instead.

 

Here’s an example of “freeing up” some processor resources from a partition.

 

I’d like to remove the processor allocation from my AIX partition named 750lpar6. This partition is currently active and has 0.5 processing units assigned to it.

 

hscroot@hmc1:~> lparls | grep 750lpar6

   750lpar6             Running

 

hscroot@hmc1:~> lparlscpu | grep 750lpar6

   750lpar6             0.5

 

 



 

I’ll stop the partition (shutdown –F) so that I may free up its processor resources. The partition is now inactive.

 

hscroot@hmc1:~> lparls | grep 750lpar6

   750lpar6             Not Activated

 

Even though the partition is no longer running, the processor allocation of 0.5 cores remains assigned to this partition.

 

 

The number of processors ‘assigned to partitions’ is 6.5 units; as shown on the HMC GUI, under the properties of the managed system.

 

 

Now I’ll use the chhwres command to “free” the currently allocated cores from the partition.

 

hscroot@hmc1:~> chhwres -r proc -m 750-1 -o r --id 8 --procunits 0.5

 

The partition is now allocated a total of 0.0 cores and is no longer using any processor resources.

 

hscroot@hmc1:~> lparlscpu | grep 750lpar6

   750lpar6             0.0

 

Looking at the managed system properties again, the number of processors ‘assigned to partitions’ has dropped to 6.0 units.

 

 

The partitions profile is NOT changed.

 

hscroot@hmc1:~> lparinfo 750lpar6

750lpar6

  On frame 750-1

  Current State:  Not Activated

  OS version: AIX 6.1 6100-05-01-1016

 

  Profile properties:

  Profile properties:
name=Default,lpar_name=750lpar6,lpar_id=8,lpar_env=aixlinux,all_resources=0,
min_mem=256,desired_mem=10240,max_mem=24576,min_num_huge_pages=0,desired_num_huge_pages=0,
max_num_huge_pages=0,mem_mode=ded,hpt_ratio=1:64,proc_mode=shared,min_proc_units=0.1,
desired_proc_units=0.5,max_proc_units=2.0,min_procs=1,desired_procs=4,max_procs=8,
sharing_mode=uncap,uncap_weight=128,shared

The partition is started again (activated) and it re-acquires 0.5 cores as it did prior to shut down.

 

hscroot@hmc1:~> lparpoweron 750lpar6

 

hscroot@hmc1:~> lparls | grep 750lpar6

   750lpar6             Running

 

hscroot@hmc1:~> lparlscpu | grep 750lpar6

   750lpar6             0.5

 

I’ve received a couple of requests for an example of using a post migration script with nimadm. What follows is a simple example of using such a resource with NIM. If you are not familiar with the nimadm tool then perhaps you’d like to start first by reading my article on using nimadm to migrate to AIX 6.1.

 

The nimadm utility can perform both pre and post migration tasks. This is accomplished by running NIM scripts either before or after a migration. The tool accepts the following flags for pre and post migration script resources:

 

-a PreMigrationScript Specifies the pre-migration NIM script resource.

-z PostMigrationScript Specifies the post-migration NIM script resource.

 

pre-migration

This script resource that is run on the NIM master, but in the environment of the client's alt_inst file system that is mounted on the master (this is done by using the chroot command). This script is run before the migration begins.

 

post-migration

This script resource is similar to the pre-migration script, but it is executed after the migration is complete.

 

We are going to focus on post-migration only, although the configuration is the same for both.

 

In this example I need to uninstall and install a 3^rd party device fileset for a storage device. I need to perform this task as part of the migration process. To protect the innocent, I have not named the storage vendor in this post. But I will say that it was not IBM storage we are dealing with in this case.

 

Before we start, first we collect all the necessary device filesets that provide support for this type of storage on AIX. We place them into a local directory on my NIM master. Along with the software, I also place a copy of my NIM script in the same directory on the NIM master. The script name is XYZpost.ksh.

 

root@nim1 : /usr/local/XYZ # ls -ltr

total 544

-r-xr-xr-x    1 root     system        51200 Mar 11 2011  MPIO_1001I

-r-xr-xr-x    1 root     system        51200 Mar 11 2011  MPIO_1002U

-r-xr-xr-x    1 root     system        51200 Mar 11 2011  MPIO_1003U

-r-xr-xr-x    1 root     system        51200 Mar 11 2011  MPIO_1004U

-r-xr-xr-x    1 root     system        51200 May 18 16:39 MPIO_1005U

-r-xr-xr-x    1 root     system          715 May 24 16:57 XYZpost.ksh

-rw-r--r--    1 root     system         2310 May 25 14:57 .toc

 

The contents of my script are simple. This script will de-install the old device fileset and then immediately install the latest version of the VendorXYZ’s device fileset. The script will then change the attributes for the vendor’s storage to more appropriate default values.

 

 

 

At this point I copy the same directory and all of its contents to the NIM client.

 

root@nim1 : /usr/local # scp –pr XYZ lparaix01:/usr/local/

…etc…

 

lparaix01 : /usr/local/XYZ # ls -ltr

total 0

-r-xr-xr-x    1 root     system        51200 Mar 11 2011  MPIO_1001I

-r-xr-xr-x    1 root     system        51200 Mar 11 2011  MPIO_1002U

-r-xr-xr-x    1 root     system        51200 Mar 11 2011  MPIO_1003U

-r-xr-xr-x    1 root     system        51200 Mar 11 2011  MPIO_1004U

-r-xr-xr-x    1 root     system        51200 May 18 16:39 MPIO_1005U

-r-xr-xr-x    1 root     system          715 May 24 16:57 XYZpost.ksh

-rw-r--r--    1 root     system         2310 May 25 14:57 .toc

 

Make sure that any scripts you write for use with nimadm start with an appropriate ‘hashbang’ to announce it is a shell script and the shell that must be used to execute it e.g. #!/usr/bin/ksh. If you forget to do this nimadm will fail to execute your script and will report an error message similar to the following:

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 7.

+-----------------------------------------------------------------------------+

Executing user chroot script /usr/local/XYZ/XYZpost.ksh.

/lparaix01_alt/alt_inst/tmp/.alt_mig_chroot_script.11731036: Cannot run a file that does not have a valid format.

 

The next step is to define the script as a NIM resource so that nimadm can call the resource during the migration process. I’ve decided to call this new NIM resource, XYZPOST.

 

This is easily achieved using smit nim_mkres:

 

root@nim1 : / # smit nim_mkres

 

|   script          = an executable file which is executed on a client     |

 

  

          Define a Resource

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                                                        [Entry Fields]

* Resource Name                                      [XYZPOST]

* Resource Type                                       script

* Server of Resource                                 [master]         +

* Location of Resource                [/usr/local/XYZ/XYZpost.ksh]    /

 

We can confirm that the NIM script resource is now available using the lsnim command.

 

root@nim1 : / # lsnim -t script

XYZPOST     resources       script

 

root@nim1 : / # lsnim -l XYZPOST

XYZPOST:

   class       = resources

   type        = script

   Rstate      = ready for use

   prev_state  = unavailable for use

   location    = /usr/local/XYZ/XYZpost.ksh

   alloc_count = 0

   server      = master                                                                 

 

Now that the script is in place, and defined to NIM, we are ready to test it. We will migrate the system from AIX 5.3 to AIX 6.1 using nimadm. Once the migration phase is complete (phases 1 to 6), the post-migration script will be executed in the NIM clients nimadm (chroot) environment on the NIM master. Once this is finished the NIM clients data is synced back to the NIM clients alternate disk and the boot image is created. The migration process is then complete.

 

We add the –z flag to our nimadm command line options to specify the post migration resource.

 

In normal operation we would simply let nimadm run all phases in sequence with the following command.

 

root@nim1 : / # nimadm -j nimadmvg -c lparaix01 -s spotaix610605 -l lpp_sourceaix610605 -d hdisk2 -z XYZPOST -Y

 

In our case we would like to test the post-migration script phase as a separate step.

 

We will start by running only the first 6 phases. We will stop at the completion of phase 6 and run phase 7 (post-migration script) manually, before moving on to phases 8 through 12.

 

You’ll observe that we have specified the –z option with the name of the NIM script resource and the –P flag with phases 1 to 6 only.

 

 

root@nim1 : / # nimadm -j nimadmvg -c lparaix01 -s spotaix610605 -l lpp_sourceaix610605 -d hdisk2 -z XYZPOST -P1,2,3,4,5,6 -Y

Initializing the NIM master.

Initializing NIM client lparaix01.

Verifying alt_disk_migration eligibility.

Initializing log: /var/adm/ras/alt_mig/lparaix01_alt_mig.log

Starting Alternate Disk Migration.

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 1.

+-----------------------------------------------------------------------------+

Cloning altinst_rootvg on client, Phase 1.

Client alt_disk_install command: alt_disk_copy -j -M 6.1 -P1 -d "hdisk2"

Calling mkszfile to create new /image.data file.

...etc...

Filesets processed:  1 of 2  (Total time:  3 secs).

 

installp:  COMMITTING software for:

        X11.compat.lib.X11R6_motif 6.1.6.15

 

Finished processing all filesets.  (Total time:  4 secs).

...etc...

install_all_updates: Checking for recommended maintenance level 6100-06.

install_all_updates: Executing /usr/bin/oslevel -rf, Result = 6100-06

install_all_updates: Verification completed.

install_all_updates: Log file is /var/adm/ras/install_all_updates.log

install_all_updates: Result = SUCCESS

Known Recommended Maintenance Levels

------------------------------------

Restoring device ODM database.

 

root@nim1 : / #

 

Phase 6 has completed successfully. The NIM clients rootvg data has been migrated from AIX 5.3 to 6.1 on the NIM master. The data has not yet been synced back to the NIM client.

 

At this stage we can now run phase 7 separately and ensure that it performs the required task. We expect it will de-install the device fileset, install the latest version and change the ODM default attributes for the device type. Again, you’ll notice that we specify the –P flag for phase 7 only.

 

Let’s test it.

 

root@nim1 : / # nimadm -j nimadmvg -c lparaix01 -s spotaix610605 -l lpp_sourceaix610605 -d hdisk2 -z XYZPOST -P7 -Y

Initializing the NIM master.

Initializing NIM client lparaix01.

Initializing log: /var/adm/ras/alt_mig/lparaix01_alt_mig.log

Starting Alternate Disk Migration.

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 7.

+-----------------------------------------------------------------------------+

Executing user chroot script /usr/local/XYZ/XYZpost.ksh.

Uninstalling XYZ ODM fileset: devices.fcp.disk.VendorXYZ.mpio.rte

+-----------------------------------------------------------------------------+

                    Pre-deinstall Verification...

+-----------------------------------------------------------------------------+

Verifying selections...done

Verifying requisites...done

Results...

 

FILESET STATISTICS

------------------

    1  Selected to be deinstalled, of which:

        1  Passed pre-deinstall verification

  ----

    1  Total to be deinstalled

 

SUCCESSES

---------

  Filesets listed in this section passed pre-deinstall verification

  and will be removed.

 

  Selected Filesets

  -----------------

  devices.fcp.disk.VendorXYZ.mpio.rte  1.2.0.0 # AIX MPIO Support for VendorXYZ Disk

 

  << End of Success Section >>

 

+-----------------------------------------------------------------------------+

                           Deinstalling Software...

+-----------------------------------------------------------------------------+

 

installp:  DEINSTALLING software for:

        devices.fcp.disk.VendorXYZ.mpio.rte  1.2.0.0

 

...etc...

Installation Summary

--------------------

Name                        Level           Part        Event       Result

-------------------------------------------------------------------------------

devices.fcp.disk.VendorXYZ.mpio 1.2.0.0         USR         DEINSTALL   SUCCESS

Return code is: 0

 

Installing XYZ ODM fileset: devices.fcp.disk.VendorXYZ.mpio.rte

+-----------------------------------------------------------------------------+

                    Pre-installation Verification...

+-----------------------------------------------------------------------------+

Verifying selections...done

Verifying requisites...done

Results...

 

FILESET STATISTICS

------------------

    1  Selected to be installed, of which:

        1  Passed pre-installation verification

  ----

    1  Total to be installed

 

SUCCESSES

---------

  Filesets listed in this section passed pre-installation verification

  and will be installed.

 

  Selected Filesets

  -----------------

  devices.fcp.disk.VendorXYZ.mpio.rte  1.4.0.0 # AIX MPIO Support for VendorXYZ Disk

 

  << End of Success Section >>

 

+-----------------------------------------------------------------------------+

                         Installing Software...

+-----------------------------------------------------------------------------+

 

installp:  APPLYING software for:

        devices.fcp.disk.VendorXYZ.mpio.rte  1.4.0.0

 

 

. . . . . << Copyright notice for devices.fcp.disk.VendorXYZ.mpio >> . . . . . . .

...etc...

Installation Summary

--------------------

Name                        Level           Part        Event       Result

-------------------------------------------------------------------------------

devices.fcp.disk.VendorXYZ.mpio 1.4.0.0         USR         APPLY       SUCCESS

Return code is: 0

 

Changing default ODM (PdAt) attributes for VendorXYZ storage

 

+ odmget -q uniquetype=disk/fcp/xyzmpio and attribute=queue_depth PdAt

+ sed s/deflt = "2"/deflt = "32"/g

+ 1> /usr/local/XYZ/chg_pdat_qd

+ odmchange -o PdAt -q uniquetype=disk/fcp/xyzmpio and attribute=queue_depth /usr/local/XYZ/chg_pdat_qd

+ odmget -q uniquetype=disk/fcp/xyzmpio and attribute=reserve_policy PdAt

+ sed s/deflt = "single_path"/deflt = "no_reserve"/g

+ 1> /usr/local/XYZ/chg_pdat_rp

+ odmchange -o PdAt -q uniquetype=disk/fcp/xyzmpio and attribute=reserve_policy /usr/local/XYZ/chg_pdat_rp

+ odmget -q uniquetype=PCM/friend/VendorXYZ and attribute=algorithm PdAt

+ sed s/deflt = "fail_over"/deflt = "round_robin"/g

+ 1> /usr/local/XYZ/chg_pdat_al

+ odmchange -o PdAt -q uniquetype=PCM/friend/VendorXYZ and attribute=algorithm /usr/local/XYZ/chg_pdat_al

 

root@nim1 : / #

 

Great news! Our script has worked as expected. The old fileset was de-installed, the new fileset was installed and the PdAt default attributes were changed successfully.

 

Note: You can also review the post migration script output at a later date if you wish. All nimadm activities are logged, on the NIM master, to /var/adm/ras/alt_mig/NIMclientname_alt_mig.log (where NIMclientname is the name of the NIM client being migrated by nimadm).

 

With regard to nimadm log files, please be aware that if you choose to run nimadm in phases (as I’ve shown in this example) that each run will generate a new log file. So in my case, when I ran phases 1 to 6, this created a log file named lparaix01_alt_mig.log. When I ran phase 7, the original log file was moved to lparaix01_alt_mig.log.prev. A new log file was created and used for phase 7. Then when I ran phases 8 to 12, the phase 7 log file was moved to lparaix01_alt_mig.log.prev and a new log file was used for phases 8-12. For this reason you may want to backup each log file to a unique file name as you execute each phase group, so that you do not lose any of the information logged to the .log or .log.prev files.

 

Now we can complete the rest of the migration and execute the remaining phases, 8 through 12.

 

root@nim1 : / # nimadm -j nimadmvg -c lparaix01 -s spotaix610605 -l lpp_sourceaix610605 -d hdisk2 -z XYZPOST -P8,9,10,11,12 -Y

Initializing the NIM master.

Initializing NIM client lparaix01.

Initializing log: /var/adm/ras/alt_mig/lparaix01_alt_mig.log

Starting Alternate Disk Migration.

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 8.

+-----------------------------------------------------------------------------+

Creating client boot image.

bosboot: Boot image is 46989 512 byte blocks.

Writing boot image to client's alternate boot disk hdisk2.

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 9.

+-----------------------------------------------------------------------------+

Adjusting client file system sizes ...

Adjusting size for /

Adjusting size for /admin

Adjusting size for /home

Adjusting size for /opt

Adjusting size for /tmp

Adjusting size for /usr

Expanding /alt_inst/usr client filesystem.

Filesystem size changed to 4325376

Adjusting size for /usr/local

Adjusting size for /usr/local/chksys

Adjusting size for /var

Adjusting size for /var/log

Adjusting size for /var/log/audit

Adjusting size for /var/tsm/log

Syncing cache data to client ...

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 10.

+-----------------------------------------------------------------------------+

Unmounting client mounts on the NIM master.

forced unmount of /lparaix01_alt/alt_inst/var/log/audit

forced unmount of /lparaix01_alt/alt_inst/var/log

forced unmount of /lparaix01_alt/alt_inst/var

forced unmount of /lparaix01_alt/alt_inst/usr/local

forced unmount of /lparaix01_alt/alt_inst/usr

forced unmount of /lparaix01_alt/alt_inst/tmp

forced unmount of /lparaix01_alt/alt_inst/opt

forced unmount of /lparaix01_alt/alt_inst/home

forced unmount of /lparaix01_alt/alt_inst/admin

forced unmount of /lparaix01_alt/alt_inst

Removing nimadm cache file systems.

Removing cache file system /lparaix01_alt/alt_inst

Removing cache file system /lparaix01_alt/alt_inst/admin

Removing cache file system /lparaix01_alt/alt_inst/home

Removing cache file system /lparaix01_alt/alt_inst/opt

Removing cache file system /lparaix01_alt/alt_inst/tmp

Removing cache file system /lparaix01_alt/alt_inst/usr

Removing cache file system /lparaix01_alt/alt_inst/usr/local

Removing cache file system /lparaix01_alt/alt_inst/var

Removing cache file system /lparaix01_alt/alt_inst/var/log

Removing cache file system /lparaix01_alt/alt_inst/var/log/audit

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 11.

+-----------------------------------------------------------------------------+

Cloning altinst_rootvg on client, Phase 3.

Client alt_disk_install command: alt_disk_copy -j -M 6.1 -P3 -d "hdisk2"

## Phase 3 ###################

Verifying altinst_rootvg...

Modifying ODM on cloned disk.

forced unmount of /alt_inst/var/log/audit

forced unmount of /alt_inst/var/log

forced unmount of /alt_inst/var

forced unmount of /alt_inst/usr/local

forced unmount of /alt_inst/usr

forced unmount of /alt_inst/tmp

forced unmount of /alt_inst/opt

forced unmount of /alt_inst/home

forced unmount of /alt_inst/admin

forced unmount of /alt_inst

Changing logical volume names in volume group descriptor area.

Fixing LV control blocks...

Fixing file system superblocks...

Bootlist is set to the boot disk: hdisk2 blv=hd5

 

+-----------------------------------------------------------------------------+

Executing nimadm phase 12.

+-----------------------------------------------------------------------------+

Cleaning up alt_disk_migration on the NIM master.

Cleaning up alt_disk_migration on client lparaix01.

root@nim1 : / #

 

Our nimadm operation is finished. The NIM client is rebooted. We verify it is now running AIX 6.1. And the correct version of the vendor’s device fileset is installed.

 

lparaix01 : / # oslevel -s

6100-06-05-1115

 

lparaix01 : / # lslpp -l devices.fcp.disk.VendorXYZ.mpio.rte 

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  devices.fcp.disk.VendorXYZ.mpio.rte

                                1.4.0.0  COMMITTED  AIX MPIO Support for VendorXYZ Disk

 

lparaix01 : / # lsdev -Cc disk

hdisk1 Available 20-T1-01 VendorXYZ MPIO Disk

hdisk2 Available 20-T1-01 VendorXYZ MPIO Disk

hdisk3 Available 20-T1-01 VendorXYZ MPIO Disk

hdisk4 Available 20-T1-01 VendorXYZ MPIO Disk

 

And there you have it, an example of using a post migration script with nimadm.

 

nimadm Command

http://pic.dhe.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.cmds/doc/aixcmds4/nimadm.htm

Following on from my post on March 25^th regarding AIX Virtual Ethernet Link Status. It appears that the latest TL and SP for AIX 7.1 (TL3 SP3) allows an administrator to view the (VIOS SEA) physical link status from a virtual Ethernet adapter.

 

# oslevel -s

7100-03-03-1415

# chdev -l ent0 -a poll_uplink=yes –P

# shutdown –Fr

# entstat -d ent0 | grep PHY

        PHYS_LINK_UP

 

Using sdiff I compared the output from entstat –d (before and after I applied TL3 SP3).

 

When you upgrade PowerHA (formerly HACMP) don’t forget to check/update your TSM client as well!

 

We successfully upgraded from PowerHA 5.4 to PowerHA 6.1 SP6 first.

 

# halevel -s

6.1.0 SP6

 

We found that our DB2 backups to TSM were no longer working.

 

$ db2adutl query

 

Error: Initialize environment failed with TSM return code 4602

 

In the TSM error log file we also saw the following message repeated over and over:

 

# tail -1 dsmerror_db2_ha.log

10/17/11   12:42:57 ANS1478E CLUSTERNODE is set to YES but the cluster software is not available on this system.

 

Obviously cluster services were most certainly installed and running! So we suspected the TSM client was in need of an update or two! This was no surprise to us.

 

The current installed TSM client version was 5.5.2.

 

# lslpp -L tivoli*

  Fileset                      Level  State  Type  Description (Uninstaller)

  ----------------------------------------------------------------------------

  tivoli.tivguid             1.3.2.1    C     F    IBM Tivoli GUID on AIX

  tivoli.tsm.client.api.32bit

                             5.5.2.0    C     F    TSM Client - Application

                                                   Programming Interface

  tivoli.tsm.client.api.64bit

                             5.5.2.0    C     F    TSM Client - 64bit Application

                                                   Programming Interface

  tivoli.tsm.client.ba.32bit.base

                             5.5.2.0    C     F    TSM Client - Backup/Archive

                                                   Base Files

  tivoli.tsm.client.ba.32bit.common

                             5.5.2.0    C     F    TSM Client - Backup/Archive

                                                   Common Files

  tivoli.tsm.client.ba.32bit.web

                             5.5.2.0    C     F    TSM Client - Backup/Archive

                                                   WEB Client

We upgraded the TSM client to version 6.2.

 

# lslpp -l tiv*

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  tivoli.tivguid             1.3.2.1  COMMITTED  IBM Tivoli GUID on AIX

  tivoli.tsm.client.api.32bit

                             6.2.2.3  COMMITTED  TSM Client - Application

                                                 Programming Interface

  tivoli.tsm.client.api.64bit

                             6.2.2.3  COMMITTED  TSM Client - 64bit Application

                                                 Programming Interface

  tivoli.tsm.client.ba.64bit.base

                             6.2.2.3  COMMITTED  TSM Client 64 - Backup/Archive

                                                 Base Files

  tivoli.tsm.client.ba.64bit.common

                             6.2.2.3  COMMITTED  TSM Client 64 - Backup/Archive

                                                 Common Files

  tivoli.tsm.client.ba.64bit.image

                             6.2.2.3  COMMITTED  TSM Client 64 - IMAGE Backup

                                                 Client

  tivoli.tsm.client.ba.64bit.web

                             6.2.2.3  COMMITTED  TSM Client 64 - Backup/Archive

                                                 Java GUI & WEB Client

 

Path: /etc/objrepos

  tivoli.tivguid             1.3.2.1  COMMITTED  IBM Tivoli GUID on AIX

 

 

Now our DB2 connection to the TSM server was working again.

 

$ db2adutl query

 

Query for database C01DCVBD1

 

 

Retrieving FULL DATABASE BACKUP information.

    1 Time: 20111024221807  Oldest log: S0093112.LOG  DB Partition Number: 0    Sessions: 1

    2 Time: 20111023215632  Oldest log: S0093111.LOG  DB Partition Number: 0    Sessions: 1

    3 Time: 20111022220109  Oldest log: S0093109.LOG  DB Partition Number: 0    Sessions: 1

    4 Time: 20111021221842  Oldest log: S0093108.LOG  DB Partition Number: 0    Sessions: 1

    5 Time: 20111020223428  Oldest log: S0093106.LOG  DB Partition Number: 0    Sessions: 1

    6 Time: 20111019221958  Oldest log: S0093105.LOG  DB Partition Number: 0    Sessions: 1

    7 Time: 20111018221802  Oldest log: S0093103.LOG  DB Partition Number: 0    Sessions: 1

 

 

Retrieving INCREMENTAL DATABASE BACKUP information.

  No INCREMENTAL DATABASE BACKUP images found for C01DCVBD1

 

 

Retrieving DELTA DATABASE BACKUP information.

  No DELTA DATABASE BACKUP images found for C01DCVBD1

 

 

Retrieving TABLESPACE BACKUP information.

  No TABLESPACE BACKUP images found for C01DCVBD1

 

 

Retrieving INCREMENTAL TABLESPACE BACKUP information.

  No INCREMENTAL TABLESPACE BACKUP images found for C01DCVBD1

 

 

Retrieving DELTA TABLESPACE BACKUP information.

  No DELTA TABLESPACE BACKUP images found for C01DCVBD1

 

 

Retrieving LOAD COPY information.

  No LOAD COPY images found for C01DCVBD1

 

 

Retrieving LOG ARCHIVE information.

   Log file: S0093099.LOG, Chain Num: 9, DB Partition Number: 0, Taken at: 2011-10-15-04.03.52

 

We also discovered that we needed to comment out the following entries in the TSM client dsm.sys file (note the MAILPROG variable appeared twice in our dsm.sys file):

 

*MAILPROG                /usr/bin/mailx  root

*CLUSTERNODE             YES

 

The latest TSM client no longer supports the MAILPROG option, as we discovered when we ran the TSM client from the command line:

 

# dsmc q sess

ANS1677W The MAILPROG option is ignored because it is no longer supported.

IBM Tivoli Storage Manager

Command Line Backup/Archive Client Interface

  Client Version 5, Release 5, Level 2.0

  Client date/time: 17-10-2011 12:01:34

(c) Copyright by IBM Corporation and other(s) 1990, 2009. All Rights Reserved.

 

Node Name: AIX1

Session established with server TSM1: AIX-RS/6000

  Server Version 5, Release 5, Level 5.0

  Server date/time: 17-10-2011 12:01:38  Last access: 17-10-2011 09:44:26

 

TSM Server Connection Information

 

Server Name.............: TSM1

Server Type.............: AIX-RS/6000

Archive Retain Protect..: "No"

Server Version..........: Ver. 5, Rel. 5, Lev. 5.0

Last Access Date........: 17-10-2011 09:44:26

Delete Backup Files.....: "No"

Delete Archive Files....: "Yes"

 

Node Name...............: AIX1

User Name...............: root

 

The CLUSTERNODE option was previously required when using TSM in a PowerHA (HACMP) environment. But not anymore:

 

Question

Do tsm client 6.1 and 6.2 support PowerHA cluster system?

 

Answer

TSM client supports all versions of PowerHA. In the past, TSM client used to publish information in the client books about the HACMP versions that it supported. The reason for this was, at that time TSM client was invoking the HACMP API in order to support it. From TSM version 6.1, TSM client does not use the HACMP API. As long as TSM client is configured with HACMP as documented in the client books and the HACMP documentation, TSM will be able to support any version of PowerHA (formerly known as HACMP).

 

https://www-304.ibm.com/support/docview.wss?uid=swg21468887

 

IBM Tivoli Storage Manager Requirements for the IBM® AIX® Backup-Archive and API Client

http://www-01.ibm.com/support/docview.wss?rs=663&tc=SSGSG7&uid=swg21052226

 

After updating my NIM master to AIX 7.1 TL2 SP1 (7100-02-01-1245), I noticed a problem. Whenever I installed a new AIX partition using NIM, the resources allocated to the NIM client were not being de-allocated, even though the installation was completing successfully. Also, if I tried to run my usual ‘NIM client reset’ script (below), the resources were still allocated.

 

#!/usr/bin/ksh

# Reset a NIM client.

 

if [[ "$1" = "" ]] ; then

  echo Please specify a NIM client to reset e.g. aixlpar1.

else

  if lsnim -l $1 > /dev/null 2>&1 ; then

    nim -o reset -F $1

    nim -Fo de-allocate  -a subclass=all $1

    nim -Fo change -a cpuid= $1

  else

    echo Not a valid NIM client!?

  fi

fi

                                                                     

For example, here’s my NIM client with the lpp_source, mksysb and SPOT resources assigned to it (even though the AIX install completed OK).

                                                                       

root@nim1 : / # lsnim -l aixlpar1                                

aixlpar1:                                                           

   class           = machines                                          

   type            = standalone                                        

   connect         = shell                                             

   platform        = chrp                                              

   netboot_kernel  = 64                                                

   if1             = network1 aixlpar1 0                             

   cable_type1     = N/A                                               

   Cstate          = ready for a NIM operation                         

   prev_state      = not running                                       

   Mstate          = currently running                                 

   boot            = boot                                              

   lpp_source      = lpp_sourceaix710105                               

   mksysb          = aixlpar1-71                                 

   nim_script      = nim_script                                        

   spot            = spotaix710105                                     

   cpuid           = 00C453C75C00                                      

   control         = master                                            

   Cstate_result   = success                                           

   installed_image = aixlpar1-71                  

                                                                        

My workaround was to use 'smit nim_mac_res' to manually de-allocate  resources from the client:                               

                                                                       

====                                                                    

De-allocate  Network Install Resources                                   

aixlpar1     machines       standalone                              

                                                                       

>lpp_sourceaix710105     lpp_source                                    

>spotaix710105           spot                                          

>aixlpar1-71       mksysb                                        

====                                                                    

                                                                                                                                             

It appears that others were also experiencing this problem. I found the following thread on the IBM developerWorks AIX user forum:     

                                             

NIM de-allocate  with subclass=all doesn't work:

https://www.ibm.com/developerworks/forums/thread.jspa?threadID=464710

 

Someone even provided their own version of a ‘NIM client reset’ script that offered another workaround to the problem:

 

nim -Fo reset $1

for i in bosinst_data fb_script image_data installp_bundle lpp_source mksysb nim_script script spot ; do

  ressources_to_clean=$(lsnim -l $1 | grep $i | awk '{print $NF}')

  for j in ${ressources_to_clean} ; do

    nim -Fo de-allocate  -a $i=$j $1

  done

done

                                                                   

I reported the issue to IBM AIX support and discovered that this is a known issue with 7.1 TL2.

 

“This is a known issue which recently has been documented in 2 open     

APARs:  IV32570 and IV32670:  UPDATE NIM BINARIES FOR UPDATES.         

                                                                       

Error description                                                   

One known symptom (of this issue) is that "nim -Fo               

de-allocate  -a subclass=all <client>" returns successfully but the

resource will still be allocated.                                 

                                                                       

The problem occured because the "update" pack that brings you from     

7.1.1.x to 7.1.2.0 missed a piece that effected the bos.sysmgt.7.1.2.0 

fileset.                                                               

                                                                       

Currently there is no fix available and no projected timeframe when this

issue will be fixed (i assume it will be with the next SP for AIX      

7100-02)                                                               

                                                                       

However, a workaround is allready documented in our incidents          

database - you simply need to force reinstall the bos.sysmgt.nim.master

7.1.2.0 fileset using your AIX 7.1 TL02 media.                         

                                                                        

This is not a backleveling of the fileset, you are simply force        

overwriting the full base level at the same version that you are       

currently at.  It will still be at 7100-02.                            

                                                                        

The workaround is a force overwrite of the bos.sysmgt.nim.master 7.1.2.0

from media." 

 

The AIX support team suggested that I download the necessary install fileset from the following location:

 

ftp://testcase.software.ibm.com/fromibm/aix/bos.sysmgt.7.1.2.0.I  

 

I backed up my NIM database before re-installing the fileset, just in case.

 

---------------------------------------------------------

Starting NIM Database Backup on nim1 at Wed Dec 12 12:39:43 EETDT 2012

 

a ./etc/objrepos/nim_attr 32 blocks.

a ./etc/objrepos/nim_attr.vc 112 blocks.

a ./etc/objrepos/nim_object 8 blocks.

a ./etc/objrepos/nim_object.vc 24 blocks.

a ./etc/NIM.level 1 blocks.

a ./etc/niminfo 1 blocks.

a ./etc/NIM.primary.cpuid 1 blocks.

 

Finished NIM Database Backup on nim1 at Wed Dec 12 12:39:44 EETDT 2012

---------------------------------------------------------

 

I placed the fileset into a separate directory on the NIM master and ran inutoc.

 

root@nim1 : /home/cgibson/NIMPMR # ls -ltr

total 36170

-rw-r--r--    1 cgibson  staff      18512896 Dec 12 12:21 bos.sysmgt.7.1.2.0.I

-rw-r--r--    1 root     system         4778 Dec 12 12:35 .toc

 

Then I re-installed the fileset (force OVERWRITE) on the NIM master:

 

                             Install Software

 

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

 

                                                        [Entry Fields]

* INPUT device / directory for software               .

* SOFTWARE to install                                [@ 7.1.2.0  Network Install Manager - Master Tools                      @@I:bos.sysmgt.nim.master 7>

  PREVIEW only? (install operation will NOT occur)    no

  COMMIT software updates?                            yes

  SAVE replaced files?                                no

  AUTOMATICALLY install requisite software?           no 

  EXTEND file systems if space needed?                yes

  OVERWRITE same or newer versions?                   yes

  VERIFY install and check file sizes?                no 

  Include corresponding LANGUAGE filesets?            yes 

  DETAILED output?                                    yes 

  Process multiple volumes?                           yes 

  ACCEPT new license agreements?                      yes  

  Preview new LICENSE agreements?                     no      

 

 

The fileset re-installed without a problem.

 

Installation Summary

--------------------

Name                        Level           Part        Event       Result

-------------------------------------------------------------------------------

bos.sysmgt.nim.master       7.1.2.0         USR         APPLY       SUCCESS

 

root@nim1 : /home/cgibson/NIMPMR # lslpp -l bos.sysmgt.nim.master

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  bos.sysmgt.nim.master      7.1.2.0  COMMITTED  Network Install Manager -

                                                 Master Tools

 

I re-tried installing AIX on a NIM client.

 

root@nim1 : / # lsnim -l aixlpar1

aixlpar1:

   class           = machines

   type            = standalone

   connect         = shell

   platform        = chrp

   netboot_kernel  = 64

   if1             = network1 aixlpar1 0

   cable_type1     = N/A

   Cstate          = Base Operating System installation is being performed

   prev_state      = BOS installation has been enabled

   Mstate          = in the process of booting

   info            = verifying_data_files

   boot            = boot

   lpp_source      = lpp_sourceaix710105

   mksysb          = aixlpar1-71

   nim_script      = nim_script

   spot            = spotaix710105

   cpuid           = 00C453C75C00

   control         = master

   Cstate_result   = success

   installed_image = aixlpar1-71

 

Once the install completed I found that all allocated NIM resources (lpp_source, mksysb and SPOT) were de-allocated. Problem solved.

 

root@nim1 : / # lsnim -l aixlpar1

aixlpar1:

   class           = machines

   type            = standalone

   connect         = shell

   platform        = chrp

   netboot_kernel  = 64

   if1             = network1 aixlpar1 0

   cable_type1     = N/A

   Cstate          = ready for a NIM operation

   prev_state      = in the process of booting

   Mstate          = not running

   cpuid           = 00C453C75C00

   Cstate_result   = success

   installed_image = aixlpar1-71

 

So if you are planning on updating your NIM master to 7.1 TL2, please take note and make sure you implement the workaround.

 

 

 

 

 

Need to update the running attributes of a virtual SCSI adapter on an AIX system? Try these steps.

 

I want to change value for the  vscsi_err_recov attribute to fast_fail. I can’t do this if the adapter is busy i.e. has open/available paths to a disk device.

 

# lsattr -El vscsi2

vscsi_err_recov delayed_fail N/A                       True

vscsi_path_to   0            Virtual SCSI Path Timeout True

 

# chdev -l vscsi2 -a vscsi_err_recov=fast_fail

Method error (/usr/lib/methods/chggen):

        0514-029 Cannot perform the requested function because a

                 child device of the specified device is not in a correct state.

 

 

I have two paths to my disk devices. This will allow me to take one path offline while I change the attributes of the adapter.

 

# lspath | grep vscsi2

Enabled hdisk4 vscsi2

Enabled hdisk5 vscsi2

 

# rmpath -l hdisk4 -p vscsi2

path Defined

 

# rmpath -l hdisk5 -p vscsi2

path Defined

 

# lspath | grep vscsi2

Defined hdisk4 vscsi2

Defined hdisk5 vscsi2

 

With the path offline, I can now change the attributes of the device and then bring the path online again.

 

# chdev -l vscsi2 -a vscsi_err_recov=fast_fail

vscsi2 changed

 

# lsattr -El vscsi2 -a vscsi_err_recov

vscsi_err_recov fast_fail N/A True

 

# cfgmgr                                        < Bring the path online.

 

# lspath | grep vscsi2

Enabled hdisk4 vscsi2

Enabled hdisk5 vscsi2

 

Now I can do the same for the other VSCSI adapter.

 

# lsattr -El vscsi3 -a vscsi_err_recov

vscsi_err_recov delayed_fail N/A True

 

# chdev -l vscsi3 -a vscsi_err_recov=fast_fail

Method error (/usr/lib/methods/chggen):

        0514-029 Cannot perform the requested function because a

                 child device of the specified device is not in a correct state.

 

# lspath | grep vscsi3

Enabled hdisk4 vscsi3

Enabled hdisk5 vscsi3

 

# rmpath -l hdisk4 -p vscsi3

path Defined

 

# rmpath -l hdisk5 -p vscsi3

path Defined

 

# lspath | grep vscsi3

Defined hdisk4 vscsi3

Defined hdisk5 vscsi3

 

# chdev -l vscsi3 -a vscsi_err_recov=fast_fail

vscsi3 changed

 

# lsattr -El vscsi3 -a vscsi_err_recov

vscsi_err_recov fast_fail N/A True

 

# cfgmgr

# lspath | grep vscsi3

Enabled hdisk4 vscsi3

Enabled hdisk5 vscsi3

 

All paths are available again and the adapters have been updated. All of this was done without taking an outage.

 

# lspath

Enabled hdisk4 vscsi2

Enabled hdisk4 vscsi3

Enabled hdisk5 vscsi2

Enabled hdisk5 vscsi3

 

# lsattr -El vscsi2 -a vscsi_err_recov

vscsi_err_recov fast_fail N/A True

 

# lsattr -El vscsi3 -a vscsi_err_recov

vscsi_err_recov fast_fail N/A True

 

If I’d chosen not to take the paths offline, one at a time, I would have needed to change the adapters attributes by specifying the –P flag with the chdev command. Then I would need to reboot the system.

 

 -P  

Changes the device's characteristics permanently in the Customized Devices object class without actually changing the device. This is useful for devices that cannot be made unavailable and cannot be changed while in the available state. The change is made to the database, and the changes are applied to the device when the system is rebooted. This flag cannot be used with the -T flag. Not all devices support the -P flag.