Chris's AIX Blog

Following on from my previous post, on working with scalable VGs.

 

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/entry/working_with_aix_scalable_volume_groups1?lang=en

 

You may find yourself in a situation where you’d like to convert a VG to scalable.

 

For example, you might start off with a normal VG i.e. one that can support a maximum of 256 LVs and 32 PVs.

 

# mkvg -s 512M -y myvg hdisk5

myvg

 

# lsvg myvg

VOLUME GROUP:       myvg                     VG IDENTIFIER:  00f6027300004c0000000133aa24d1f6

VG STATE:           active                   PP SIZE:        512 megabyte(s)

VG PERMISSION:      read/write               TOTAL PPs:      99 (50688 megabytes)

MAX LVs:            256                      FREE PPs:       0 (0 megabytes)

LVs:                1                        USED PPs:       99 (50688 megabytes)

OPEN LVs:           1                        QUORUM:         2 (Enabled)

TOTAL PVs:          1                        VG DESCRIPTORS: 2

STALE PVs:          0                        STALE PPs:      0

ACTIVE PVs:         1                        AUTO ON:        yes

MAX PPs per VG:     32512

MAX PPs per PV:     1016                     MAX PVs:        32

LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no

HOT SPARE:          no                       BB POLICY:      relocatable

PV RESTRICTION:     none

 

Sometime later you discover you need to grow the volume group and add more hdisks. But you can’t because you’ve reached the maximum PV limit of 32.

 

 You examine the chvg man page and discover the –G flag! Of course you take note of the special considerations when converting to scalable.

 

Changes the volume group to Scalable VG format. This can accommodate up to 1024 physical volumes and 4096 logical volumes. Notes:

 

1    The -G flag cannot be used if there are any stale physical partitions.

 

2    Once the volume group is converted, it cannot be imported into AIX(R) 5.2 or lower versions.

 

3    The -G flag cannot be used if the volume group is varied on.

 

4    There must be enough free partitions available on each physical volume for the VGDA expansion for this operation to be successful.

 

5    Since the VGDA resides on the edge of the disk and it requires contiguous space for expansion, the free partitions are required on the edge of the disk. If those partitions are allocated for user usage, they will be migrated to other free partitions on the same disk. The rest of the physical partitions will be renumbered to reflect the loss of the partitions for VGDA usage. This will change the mappings of the logical to physical partitions in all the PVs of this VG. If you have saved the mappings of the LVs for a potential recovery operation, you should generate the maps again after the completion of the conversion operation. Also, if the backup of the VG is taken with the map option and you plan to restore using those maps, the restore operation may fail since the partition number may no longer exist (due to reduction). It is recommended that backup is taken before the conversion, and right after the conversion if the map option is utilized.

 

6    Because the VGDA space has been increased substantially, every VGDA update operation (creating a logical volume, changing a log ical volume, adding a physical volume, and so on) may take considerably longer to run.

 

Excellent, I can convert my VG to scalable.

 

The volume group must be varied off before it can be converted.

 

# chvg -G myvg

0516-1707 chvg: The volume group must be varied off during conversion to

        scalable volume group format.

0516-732 chvg: Unable to change volume group myvg.

 

# umount /myfs

 

# lsvg -l myvg

myvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

mylv                jfs2       99      99      1    closed/syncd  /myfs

 

# varyoffvg myvg

 

There must be enough free PPs in the volume group (on each hdisk) in order for the conversion to work. If there aren’t then you’ll see this error:

 

# chvg -G myvg

0516-1214 chvg: Not enough free physical partitions exist on hdisk5 for the

expansion of the volume group descriptor area. Migrate/reorganize to free up

1 partitions and run chvg again.

0516-732 chvg: Unable to change volume group myvg.

 

In this example, we vary on the VG again, and shrink the file system by 1 PP, giving us 1 PP free in the volume group. Remember, “There must be enough free partitions available on each physical volume for the VGDA expansion for this operation to be successful.”

 

# varyonvg myvg

 

# chfs -a size=-512M /myfs

Filesystem size changed to 102760448

Inlinelog size changed to 196 MB.

 

# lsvg myvg

VOLUME GROUP:       myvg                     VG IDENTIFIER:  00f6027300004c0000000133aa24d1f6

VG STATE:           active                   PP SIZE:        512 megabyte(s)

VG PERMISSION:      read/write               TOTAL PPs:      99 (50688 megabytes)

MAX LVs:            256                      FREE PPs:       1 (512 megabytes)

LVs:                1                        USED PPs:       98 (50176 megabytes)

OPEN LVs:           0                        QUORUM:         2 (Enabled)

TOTAL PVs:          1                        VG DESCRIPTORS: 2

STALE PVs:          0                        STALE PPs:      0

ACTIVE PVs:         1                        AUTO ON:        yes

MAX PPs per VG:     32512

MAX PPs per PV:     1016                     MAX PVs:        32

LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no

HOT SPARE:          no                       BB POLICY:      relocatable

PV RESTRICTION:     none

 

The VG is varied off again.

 

# varyoffvg myvg

# lsvg -o

rootvg

 

We convert the VG to scalable.

 

# chvg -G myvg

0516-1224 chvg: WARNING, once this operation is completed, volume group myvg

cannot be imported into AIX 5.2 or lower versions. Continue (y/n) ? y

0516-1216 chvg: Physical partitions are being migrated for volume group

descriptor area expansion.  Please wait.

0516-1712 chvg: Volume group myvg changed.  myvg can include up to 1024 physical volumes with 2097152 total physical partitions in the volume group.

 

We vary on the volume group and check that it is now scalable. You’ll notice that we’ve now lost that free PP in the VG as it has been used in the conversion process.

 

# varyonvg myvg

 

# lsvg myvg

VOLUME GROUP:       myvg                     VG IDENTIFIER:  00f6027300004c0000000133aa24d1f6

VG STATE:           active                   PP SIZE:        512 megabyte(s)

VG PERMISSION:      read/write               TOTAL PPs:      98 (50176 megabytes)

MAX LVs:            256                      FREE PPs:       0 (0 megabytes)

LVs:                1                        USED PPs:       98 (50176 megabytes)

OPEN LVs:           0                        QUORUM:         2 (Enabled)

TOTAL PVs:          1                        VG DESCRIPTORS: 2

STALE PVs:          0                        STALE PPs:      0

ACTIVE PVs:         1                        AUTO ON:        yes

MAX PPs per VG:     32768                    MAX PVs:        1024

LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no

HOT SPARE:          no                       BB POLICY:      relocatable

PV RESTRICTION:     none

 

 

If you are unable to free up a physical partition on a your hdisk (or hdisks) then you may need to use a tool like migratelp to move logical partitions around in order to free up some space. From the migratelp man page:

 

migratelp Command

 

Purpose

 

Moves allocated logical partition from one physical partition to another physical partition on a different physical volume.

 

 

 

Scalable volume groups first appeared in AIX 5.3. They allow a maximum of 1024 physical volumes and 4096 logical volumes per volume group. This is pretty cool for large systems. In the ‘old days’ we could end up in situations where we’d reached the maximum PV count (32 or 128) for a VG and have to create a new VG to work around the issue.

 

I use scalable VGs for all my new large AIX system builds.

 

The following command will create a scalable VG (named svg) on hdisk4. You’ll notice that MAX PVs is set to 1024 and MAX LVs is set to 256.

 

# mkvg -S -ysvg hdisk4

svg

 

# lsvg svg

VOLUME GROUP:       svg                      VG IDENTIFIER:  00f6048800004c0000000133336703a0

VG STATE:           active                   PP SIZE:        32 megabyte(s)

VG PERMISSION:      read/write               TOTAL PPs:      1597 (51104 megabytes)

MAX LVs:            256                      FREE PPs:       1597 (51104 megabytes)

LVs:                0                        USED PPs:       0 (0 megabytes)

OPEN LVs:           0                        QUORUM:         2 (Enabled)

TOTAL PVs:          1                        VG DESCRIPTORS: 2

STALE PVs:          0                        STALE PPs:      0

ACTIVE PVs:         1                        AUTO ON:        yes

MAX PPs per VG:     32768                    MAX PVs:        1024

LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no

HOT SPARE:          no                       BB POLICY:      relocatable

PV RESTRICTION:     none

 

So, MAX LVs is set to 256 by default in a scalable VG. Fine, but what if I need to grow higher than that?  Well, we can use the chvg command to set MAX LVs to a higher number, such as 4096 (the maximum).

 

# chvg -v 4096 svg

 

We made this change while the VG was varied on and file systems are open.....very nice indeed!

 

# lsvg svg

VOLUME GROUP:       svg                      VG IDENTIFIER:  00f6048800004c0000000133336703a0

VG STATE:           active                   PP SIZE:        32 megabyte(s)

VG PERMISSION:      read/write               TOTAL PPs:      1597 (51104 megabytes)

MAX LVs:            4096                     FREE PPs:       1597 (51104 megabytes)

LVs:                0                        USED PPs:       0 (0 megabytes)

OPEN LVs:           0                        QUORUM:         2 (Enabled)

TOTAL PVs:          1                        VG DESCRIPTORS: 2

STALE PVs:          0                        STALE PPs:      0

ACTIVE PVs:         1                        AUTO ON:        yes

MAX PPs per VG:     32768                    MAX PVs:        1024

LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no

HOT SPARE:          no                       BB POLICY:      relocatable

PV RESTRICTION:     none

l488pp062_pub[/] >

 

From the chvg man page:

 

# man chvg

.....

 

-v LogicalVolumes

Increases the number of logical volumes that can be created. Valid values are 512, 1024, 2048 and 4096. The value should be larger than the current value or no action is taken. This option is only valid with Scalable-type volume groups. This option is not available for volume groups varied on in concurrent mode.

 

Once we’ve changed MAX LVs to a higher value, we can’t go backwards. For example, if we try to change MAX LVs to a lower value, we receive the following message:

 

# chvg -v 256 svg

0516-1617 lchangevg: 256 is an invalid value for maxlvs, the value must be greater

        than the current value, 4096, and cannot be greater than the limit, 4096.

0516-732 chvg: Unable to change volume group svg.

 

If we attempted to set MAX LVs to a higher value than what’s supported we’d receive the following message:

 

# chvg -v 8192 svg

0516-1614 lchangevg: The valid values for maxlvs include 256, 512, 1024,

        2048, and 4096. This option is only usable for scalable

        volume groups.

Usage: lchangevg -g VGid -p PVid [-r RemoveMode] [-a AllocateMode]

0516-732 chvg: Unable to change volume group svg.

Here’s a question I received recently, that relates to virtual SCSI hdisk path priority.

 

Hi, Chris,

I use MPIO for our virtualization environment.

I'd like to check which path is currently used.

How can I get the result?

For example:

 

[nim:/]# lspath
Enabled hdisk0 vscsi0
Enabled hdisk0 vscsi1
Enabled hdisk2 vscsi2
Enabled hdisk1 vscsi1
Enabled hdisk1 vscsi0
Enabled hdisk3 vscsi2
Enabled hdisk2 vscsi3
Enabled hdisk3 vscsi3

 

How can I determine vscsi0 or vscsi1 is currently carrying data through VIOS and VIOC.

Thanks a lot!

 

By default the Virtual I/O Client (VIOC) will use the first VIOS for all VSCSI traffic. You can confirm which VSCSI adapter is being used by starting nmon (on the VIOC) and pressing the ‘a’ key. This will show which vscsi adapter is currently in use. e.g.

 

┌─topas_nmon──a=disk-Adapters────Host=lpar42p682e_pubRefresh=2 secs───18:38.04─┐

│ Disk-Adapter-I/O ────────────────────────────────────────────────────────────│

│Name          %busy     read    write        xfers Disks Adapter-Type         │

│vscsi3          0.0      0.0      0.0 KB/s     0.0 1     Virtual SCSI Client A│

│vscsi2        100.0      0.0 218560.0 KB/s  1707.5 1     Virtual SCSI Client A│

│TOTALS  2 adapters       0.0 218560.0 KB/s  1707.5 2     TOTAL(MB/s)=213.4    │

│──────────────────────────────────────────────────────────────────────────────│

 

Use the lspath command to determine the current path priority for a hdisk. By default it will be set to 1 for both adapters, meaning that traffic will go via the first path. Priority 1 is the highest priority, and you can define a priority from 1 to 255.

 

# lspath -AE -l hdisk2 -p vscsi2

priority 1 Priority True

 

# lspath -AE -l hdisk2 -p vscsi3

priority 1 Priority True

 

You can change the path priority of a hdisk using the chpath command. For example, we change the priority for vscsi2 to 255, so that vscsi3 now has the highest priority:

 

# chpath -l hdisk2 -a priority=255 -p vscsi2

# chpath -l hdisk2 -a priority=1 -p vscsi3

 

This will change the path priority for hdisk2 so that its traffic will use vscsi3 instead vscsi2.

 

┌─topas_nmon──b=Black&White──────Host=lpar42p682e_pubRefresh=2 secs───18:40.56─┐

│ Disk-Adapter-I/O ────────────────────────────────────────────────────────────│

│Name          %busy     read    write        xfers Disks Adapter-Type         │

│vscsi3        100.0      0.0 224384.5 KB/s  1753.0 1     Virtual SCSI Client A│

│vscsi2          0.0      0.0      0.0 KB/s     0.0 1     Virtual SCSI Client A│

│TOTALS  2 adapters       0.0 224384.5 KB/s  1753.0 2     TOTAL(MB/s)=219.1    │

│──────────────────────────────────────────────────────────────────────────────│

 

Now all the VSCSI traffic (for hdisk2) is traversing the vscsi3 adapter instead of vscsi2.