Hot spot management in logical volumes

You can identify hot spot problems with your logical volumes and remedy those problems without interrupting the use of your system.

A hot-spot problem occurs when some of the logical partitions on your disk have so much disk I/O that your system performance noticeably suffers.

The first step toward solving the problem is to identify it. By default, the system does not collect statistics for logical volume use. After you enable the gathering of these statistics, the first time you enter the lvmstat command, the system displays the counter values since the previous system reboot. Thereafter, each time you enter the lvmstat command, the system displays the difference since the previous lvmstat command.

By interpreting the output of the lvmstat command, you can identify the logical partitions with the heaviest traffic. If you have several logical partitions with heavy usage on one physical disk and want to balance these across the available disks, you can use the migratelp command to move these logical partitions to other physical disks.

In the following example, the gathering of statistics is enabled and the lvmstat command is used repeatedly to gather a baseline of statistics:

# lvmstat -v rootvg -e
# lvmstat -v rootvg -C
# lvmstat -v rootvg
The output is similar to the following:
Logical Volume            iocnt      Kb_read     Kb_wrtn      Kbps
  hd8                         4            0          16      0.00
  paging01                    0            0           0      0.00
  lv01                        0            0           0      0.00
  hd1                         0            0           0      0.00
  hd3                         0            0           0      0.00
  hd9var                      0            0           0      0.00
  hd2                         0            0           0      0.00
  hd4                         0            0           0      0.00
  hd6                         0            0           0      0.00
  hd5                         0            0           0      0.00

The previous output shows that all counters have been reset to zero. In the following example, data is first copied from the /unix directory to the /tmp directory. The lvmstat command output reflects the activity for the rootvg:

# cp -p /unix /tmp
# lvmstat -v rootvg

Logical Volume            iocnt      Kb_read     Kb_wrtn      Kbps
  hd3                       296            0        6916      0.04
  hd8                        47            0         188      0.00
  hd4                        29            0         128      0.00
  hd2                        16            0          72      0.00
  paging01                    0            0           0      0.00
  lv01                        0            0           0      0.00
  hd1                         0            0           0      0.00
  hd9var                      0            0           0      0.00
  hd6                         0            0           0      0.00
  hd5                         0            0           0      0.00

The output shows activity on the hd3 logical volume, which is mounted in the /tmp directory, on hd8, which is the JFS log logical volume, on hd4, which is / (root), on hd2, which is the /usr directory, and on hd9var, which is the /var directory. The following output provides details for hd3 and hd2:

# lvmstat -l hd3

Log_part    mirror#    iocnt    Kb_read    Kb_wrtn     Kbps
       1         1       299          0       6896     0.04
       3         1         4          0         52     0.00
       2         1         0          0          0     0.00
       4         1         0          0          0     0.00
# lvmstat -l hd2
Log_part    mirror#    iocnt    Kb_read    Kb_wrtn     Kbps
       2         1         9          0         52     0.00
       3         1         9          0         36     0.00
       7         1         9          0         36     0.00
       4         1         4          0         16     0.00
       9         1         1          0          4     0.00
      14         1         1          0          4     0.00
       1         1         0          0          0     0.00

The output for a volume group provides a summary for all the I/O activity of a logical volume. It is separated into the number of I/O requests (iocnt), the kilobytes read and written (Kb_read and Kb_wrtn, respectively), and the transferred data in KB/s (Kbps). If you request the information for a logical volume, you receive the same information, but for each logical partition separately. If you have mirrored logical volumes, you receive statistics for each of the mirror volumes. In the previous sample output, several lines for logical partitions without any activity were omitted. The output is always sorted in decreasing order on the iocnt column.

The migratelp command uses, as parameters, the name of the logical volume, the number of the logical partition (as it is displayed in the lvmstat output), and an optional number for a specific mirror copy. If information is omitted, the first mirror copy is used. You must specify the target physical volume for the move; in addition, you can specify a target physical partition number. If successful, the output is similar to the following:

# migratelp hd3/1 hdisk1/109
  migratelp: Mirror copy 1 of logical partition 1 of logical volume
        hd3 migrated to physical partition 109 of hdisk1.

After the hot spot feature is enabled, either for a logical volume or a volume group, you can define your reporting and statistics, display your statistics, select logical partitions to migrate, specify the destination physical partition, and verify the information before committing your changes.