Multipath setup

The multipath daemon is relevant for multipathing with FCP disks or when PAV devices are used instead of HyperPAV.

Multipathing with FCP disks is required for failover in case the path to the disks fails or is in maintenance, or to balance the load over multiple paths. Each path defined to a FCP disk appears in the system as independent disk device, but it is still the same disk. These devices should not be used, because they do not provide failover and bear the risk of destroying the file system structure or files when multiple processes use the disks via different paths. Instead a tool is required which manages the different paths and ensures consistency by providing one unique device to the system. This is done by the multipath daemon.

Multipath devices

The multipath daemon provides the resulting disk device either as /dev/dm-[n] device or if an alias is defined in the multipath.conf, via /dev/mapper/<alias name>. The latter is highly recommended to give the multipath devices a meaningful name, for example, FCP-<FCP-LUN>.

path_grouping_policy

Another important option is the path_grouping_policy, it can be:

failover Only one path is used, when that fails, the next path which works is used
multibus All paths are used in a balanced mode
- multibus Introduces the rr_min_io parameter which defines the number of requests after which the path is changed

wwid

The multipath devices are indexed in the multipaths section via the World Wide Identifier (wwid). To identify the wwid's of a given setup:

Enable all FCP disks devices and paths
Run the multipath command, it reports the wwid and the disk type, for example,
```
multipath 
create: 36005076309ffc5ee000000000000e427 undef IBM,2107900
```
Create a suitable multipath.conf file in /etc
Call multipath -F to flush the existing map In case of a logical volume it reports
```
Jul 31 16:46:53 | 36005076309ffc5ee000000000000e027_part1: map in use
```
Then disable the volume group using vgchange -an <volumegroup> and repeat the command
Call the multipath command again to activate your multipath.conf file

Important pitfall

The multipath.conf file is structured in multiple sections, the relevant sections are:

defaults
devices
multipaths

This also introduces a hierarchy, whereby definitions in the defaults sections are overruled by the devices and multipaths section, and the multipaths section overrules the devices section. This is important because there are a lot of defaults defined in the devices section. These definitions overwrite what is defined in the defaults section, which might affect both parameters the path_grouping_policy and rr_min_io.

multipath.conf setup

The safest way is to specify all definitions in the multipaths section, but these definitions are for each multipath device, and may require many changes. The alternative is to use the lsscsi command to show the used disk devices, for example:

lsscsi 
[0:0:18:1076314336]disk  IBM      2107900          .190  /dev/sdc
[0:0:18:1076314337]disk  IBM      2107900          .190  /dev/sdd

Here the disks are of type IBM and 2107900.

As a next step, call multipath -t to show the internally used defaults, which are for our disks:

device { 
vendor “IBM”
product “2107900” 
path_checker tur
checker tur
rr_min_io 1000
}

This overwrites a definition of rr_min_io defined in the defaults section. For this reason we introduced a devices section in our multipath.conf file to overwrite defaults.

The final test to check if it works correctly is multipath -l, the output looks like this:

For a multibus setup:

FCPe527 (36005076309ffc5ee000000000000e527) dm-2 IBM,2107900 
size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw 
`-+- policy='round-robin 0' prio=0 status=active 
  |- 0:0:18:1076314341 sdb 8:16  active undef running 
  |- 1:0:1:1076314341  sdf 8:80  active undef running 
  |- 2:0:5:1076314341  sdj 8:144 active undef running 
  `- 3:0:3:1076314341  sdn 8:208 active undef running

For a failover setup:

FCPe527 (36005076309ffc5ee000000000000e527) dm-2 IBM,2107900 
size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw 
|-+- policy='round-robin 0' prio=0 status=active 
| `- 0:0:18:1076314341 sdb 8:16  active undef running 
|-+- policy='round-robin 0' prio=0 status=enabled 
| `- 1:0:1:1076314341  sdf 8:80  active undef running 
|-+- policy='round-robin 0' prio=0 status=enabled 
| `- 2:0:5:1076314341  sdj 8:144 active undef running 
`-+- policy='round-robin 0' prio=0 status=enabled 
 `- 3:0:3:1076314341  sdn 8:208 active undef running

The dmsetup table command can be used to check the rr_min_io parameter. For example for a rr_min_io value of 100 and multibus:

dmsetup table 
FCPe527: 0 20971520 multipath 1 queue_if_no_path 0 1 1 round-robin 
0 4 1 8:16 100 8:80 100 8:144 100 8:208 100 
The values 8:16 100 are showing the <disk major>:<disk minor> <rr_min_io value>

Here a sample of a multipath.conf file (resides in /etc)

defaults { 
        user_friendly_names     no 
#       path_grouping_policy    multibus 
        rr_min_io       1 
} 

blacklist { 
        devnode "*" 
} 

blacklist_exceptions { 
        devnode "^sdp+[0-9]*" 
        devnode "^sdo+[0-9]*" 
        devnode "^sdn+[0-9]*" 
        devnode "^sdm+[0-9]*" 
        devnode "^sdl+[0-9]*" 
        devnode "^sdk+[0-9]*" 
        devnode "^sdj+[0-9]*" 
        devnode "^sdi+[0-9]*" 
        devnode "^sdh+[0-9]*" 
        devnode "^sdg+[0-9]*" 
        devnode "^sdf+[0-9]*" 
        devnode "^sde+[0-9]*" 
        devnode "^sdd+[0-9]*" 
        devnode "^sdc+[0-9]*" 
        devnode "^sdb+[0-9]*" 
        devnode "^sda+[0-9]*" 
        device { 
                vendor "IBM" 
                product "S/390.*" 
        } 
} 

devices { 
        device { 
                vendor "IBM" 
                product "2107900" 
                path_grouping_policy multibus 
                path_checker tur 
                checker tur 
                rr_min_io 100 
        } 
} 

multipaths { 
        multipath { 
                wwid    "36005076309ffc5ee000000000000e527" 
                alias   "FCPe527" 
        } 
        multipath { 
                wwid    "36005076309ffc5ee000000000000e427" 
                alias   "FCPe427" 
        } 
        multipath { 
                wwid    "36005076309ffc5ee000000000000e127" 
                alias   "FCPe127" 
        } 
        multipath { 
                wwid    "36005076309ffc5ee000000000000e027" 
                alias   "FCPe027" 
        } 
}