Multipathing with SCSI disks

This hands-on example shows how to setup multipathing.

A setup for SCSI disks looks like:

#load the zfcp module – using the FCP cards
modprobe zfcp

#setting the FCP adapter online
echo 1 >> /sys/bus/ccw/devices/0.0.<fcp-port-address>/online

#adding a Storage Server (WWPN) to the FCP adapter
echo <wwpn-address> > /sys/bus/ccw/devices/0.0.<fcp-device>/port_add

#adding a disk, which is reached over fcp-port and via wwpn-address
echo <disk> > /sys/bus/ccw/devices/0.0.<fcp-device>/<wwpn-address>/unit_add

Multipathing requires each disk to be attached over a different path:

#setting the FCP card online
echo 1 >> /sys/bus/ccw/devices/0.0.1700/online
echo 1 >> /sys/bus/ccw/devices/0.0.1800/online

#adding a Storage Server (WWPN) to the FCP card
echo 0x500507630308c481 > /sys/bus/ccw/devices/0.0.1700/port_add
echo 0x500507630308c482 > /sys/bus/ccw/devices/0.0.1800/port_add

#adding a disk, which is reached over fcp-port and via wwpn-address
echo 0x4014400300000000 > /sys/bus/ccw/devices/0.0.1700/0x500507630308c481/unit_add
echo 0x4014400300000000 > /sys/bus/ccw/devices/0.0.1800/0x500507630308c482/unit_add

Enabling multipathing:

#load the multipath module
modprobe dm-multipath

#issue the multipath command to look what was recognized
multipath -l

#for proper multipath work start the multipathd service
/etc/init.d/multipathd start

Multipathd can be started automatically after every restart enabling the /etc/init.d/multipathd script at startup.

Multipath output:

...
36005076303ffc4820000000000001403
[size=10 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active][best]
\_ 6:0:1:0 sdg 8:96 [active]
\_ round-robin 0 [enabled]
\_ 6:0:2:1 sdw 65:96 [active]
...

To see all disks, issue

multipath -F and multipath -l

for flushing and recreating the multipath configuration

To change the profile how to write to your SCSI disks, issue

multipath -p <profile multibus|failover|...>

Explanation of the above example:

Disk 36005076303ffc4820000000000001403 is accessible over the sd-devices /dev/sdg or /dev/sdw. It looks like two separate disks, but each sd-device represents one path to this disk.

The multipath tools recognize the different paths to each disk and create a device (in this example /dev/mapper/36005076303ffc4820000000000001403), which takes over the access to this disk. The multipath tools handle the utilization of the paths.

The output shows a "failover" mode usage. The first path (sdg) is for primary I/O and the second path (sdw) is only used in case the first path fails.

Another commonly used mode is "multibus". For more details read the man-page of the multipath tools.

Back to top

To use the multipathed disks, use the device created by the multipath tools (in the above example/dev/mapper/36005076303ffc4820000000000001403).

Alternatively, you can combine several multipathed disks to one logical volume. We recommend to create this logical volume using LVM2.

The example below describes the creation of such a logical volume using LVM2.

1. To access the multipathed devices, modify the file /etc/lvm/lvm.conf.

md_component_detection = 1
devices {
filter = [ "r|/dev/disk/by-name/.*|","r|/dev/disk/by-id/.*|","r| sd*|","a|/dev/dm.*|", "r|.*|" ]
types = [ "device-mapper", 253 ]
}

The string "a|/dev/dm.*|" introduces those devices to LVM2, which are created by the multipath tools. Now these devices can be accessed.

2. To check the multipath configuration, issue

   multipath -l

Now you can prepare these disks for a usage within a logical volume (LV).

3. Declare the disks as "physical volume" with pvcreate:

   pvcreate /dev/mapper/36005076303ffc4820000000000001403p1 ...

4. Merge the just created physical volumes into your volume group (sample_vg) with vgcreate:

   vgcreate sample_vg /dev/mapper/36005076303ffc4820000000000001403p1 ...

5. Create your LV. This example creates a striped LV with a size of 120GiB and 16 stripes for I/O. It uses the disks from sample_vg:

   lvcreate -i 16 -l 120G -n sample_lv sample_vg

6. Create the filesystem and mount the LV. Now you are ready for using it.

   mkfs.ext3 /dev/samlple_vg/sample_lv
mount /dev/samlple_vg/sample_lv /mnt/sample

Optional step:

If you want to reuse an LV after reboot,

• either manually
  
  
  1. set your disks online
  2. start your multipathd
  3. issue

     vgchange -a y
mount /dev/<vgname>/<lvname> <mountpoint>

• or automatically
  
  set the SCSI disks online at boot time by recreating the mkinitrd (SLES) or by using the /etc/zfcp.conf (RHEL or SLES). Then the LV init scripts bring the LV online.

Note: Due to multipath limitations, disks with device names i.e. sdaa or sdaz precede devices i.e. sda or sdk, even if the latter devices are before the former. Thus, sdaa or sdaz automatically describes the primary path and sda or sdk describe the secondary (failover) path.

Example of a multipath output:

...
36005076303ffc4820000000000001413
[size=10 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active][best]
\_ 6:0:1:0 sdaa [active]
\_ round-robin 0 [enabled]
\_ 6:0:2:1 sda [active]
...

Device path sdaa, which was anticipated as failover path slips in front of sda which was anticipated as primary path.

Nevertheless, be aware in which order disks/paths are brought online when using more than 26 disks/paths.

It is not possible to use multipathing with persistent device names created with udev. Multipath tools prefer sd-device names.