Linux Device-Mapper Multipathing, also called device-mapper or DM-MPIO, lets you use more than one physical path to transfer data between the host and an external storage device. This capability allows you to load balance I/O over multiple paths and provide I/O failover. It is well known in the Linux world, but using multipathing in a guest VM is quite new.
It is not possible to use multipath on full x86 virtualization because there is a single kernel that virtualizes all I/O resources. Power Systems provides a distinct advantage over x86 virtualization by allowing multiple I/O virtualization servers on a single server, and as a result, supports multipathing inside the guest VM. Overall, PowerVM provides better redundancy, accessibility, and most importantly, serviceability for your virtual servers.
PowerVM with a dual VIOS setup supports multipath I/O configurations in the guest VM. However, when deploying Red Hat Enterprise Linux 7.x (RHEL 7.x)
workloads with PowerVC, extra configuration setup is required for successful deployment of multipath guest VMs.
In this article, I will explain how to set up a multipathing I/O configuration for a RHEL 7.x guest with PowerVC to be deployed in a dual VIOS environment.
Caution – Please follow these procedures to ensure that your RHEL 7.x guest VM with I/O multipathing doesn’t crash as a result of incorrectly installed and deployed I/O multipathing.
This information is presented in two articles; this first article describes the quick workaround. It will be followed by a later article that describes the root cause of the potential issue when the workaround is not used.
Before capturing an image with PowerVC, make sure that the multipath is working properly by running the below commands.
Verify that the device-mapper is installed by running the rpm command and grepping for device-mapper. The RPM versions that are returned might be different in your environment depending on your OS level:
[root@rhel7~]#rpm -qa | grep device-mapper
device-mapper-persistent-data-0.4.1-2.el7.ppc64
device-mapper-libs-1.02.93-3.el7.ppc64
device-mapper-event-1.02.93-3.el7.ppc64
device-mapper-multipath-0.4.9-77.el7.ppc64
device-mapper-1.02.93-3.el7.ppc64
device-mapper-event-libs-1.02.93-3.el7.ppc64
device-mapper-multipath-libs-0.4.9-77.el7.ppc64
Note: These steps are for RHEL7.x.
If the device-mapper is not installed, then run the following command to complete the installation.
1. Use yum to install device-mapper:
[root@rhel7~]yum install device-mapper
2. After installing multipath, enable it with the following commands:
[root@rhel7~]mpathconf --enable --with_multipathd y
[root@rhel7~]systemctl start multipathd
[root@rhel7~]systemctl enable multipathd
The default RHEL 7 multipath settings are OK for most environments.
Capture a Virtual Machine
1.) To ensure a clean capture, make sure that /etc/multipath/bindings and /etc/multipath/wwids files are empty, as shown:
[root@rhel7] > /etc/multipath/wwids
[root@rhel7] > /etc/multipath/bindings
2.) Rebuild initramfs with multipath:
[root@rhel7~] dracut --force -H --add multipath
3. Set up cloud-init if you have not already. Edit the /etc/cloud/cloud.cfg file:
Edit the following values:
disable_root: 0
ssh_pwauth: 1
ssh_deletekeys: 1
Add the following before the cloud_init_modules section:
disable_ec2_metadata: True
datasource_list: ['ConfigDrive']
For cloud-init version 7.4.5 or later, add the following options in the section cloud_init_modules
- reset-rmc
- update-bootlist
- runcmd
Add the following option in section cloud_final_modules:
- power-state-change
Multipath will not work on the first boot (RHEL 7.x), so we need to restart the server after setting up multipath. It can be done with the power-state-change module, as shown. Delay +% means that cloud-init will reboot the system after 5 minutes. Timeout is in seconds:
power_state:
delay: "+5"
mode: reboot
message: Bye Bye
timeout: 120
condition: True
Add a runcmd section at the end of the file:
runcmd:
- rm -f /etc/multipath/wwids
- ofpathname -l $(lsprop /proc/device-tree/chosen | grep bootpath | awk -F "\"" '{print $2}') > /tmp/bootdisk
- multipath -a /dev/$(cat /tmp/bootdisk)
- dracut -f -a multipath
Explanation:
The runcmd part is a little tricky. To find the active boot disk we use ofpathname and lsprop commands. lsprop lists the boothpath v-scsi address.
In my example:
[root@linuxlpar ~]# lsprop /proc/device-tree/chosen | grep bootpath
bootpath "/vdevice/v-scsi@30000003/disk@8200000000000000"
The ofpathname commands translate logical device names to their Open Firmware device path names. Both commands are PowerPC specific.
Run this command to eliminate boot issues with multiple mpath disks.
[root@linuxlpar ~]# ofpathname -l /vdevice/v-scsi@30000003/disk@8200000000000000
sda
Rebuild the initramfs with the dracut command and update the bootlist. Be sure to test all commands before deploying. Special characters can be problematic when you
copy-paste from your web browser.
4. After setting up cloud-init, shut down the machine. Cloud-init does not shut down the virtual machine automatically like Activation Engine does.
For details, see "Understanding cloud-init for PowerVC".
5. Capture the virtual server from the PowerVC user interface.
Deploy a Virtual Machine
On my tests with RHEL 7.2, multipath did not work on the first boot. Standard multipath procedures like multipath -r did not work and I got messages like multipath: error getting device.
To address this, I needed to restart the virtual server with cloud-init power-state-change module.
Don't forget to run the bootlist command after the boot. There are different methods to run the bootlist automatically. I put a script including the following code to /etc/rc.local.
ofpathname -l $(lsprop /proc/device-tree/chosen | grep bootpath | awk -F "\"" '{print $2}') > /tmp/bootdisk
multipath -l /dev/$(cat /tmp/bootdisk) |head -1 | awk '{print "bootlist -m both /dev/mapper/"$1}' |bash
After deploying a virtual machine, continue to section Check Multipath Status.
Manual Setup
Follow these steps if you want to set up multipath manually, for example if your cloud-init version is older or if the system was already deployed and the operating system crashed.
These instructions are for a running operating system. If your OS does not boot, refer to the Red Hat topic “Booting into Rescue Mode.”
Add the second disk to the configuration file. Note that the disk name might be different. Refer to the output from df and pvs to determine the inactive disk.
[root@rhel7~]multipath -a /dev/sdb
wwid 'xxxxxxx' added
Rebuild the initramfs:
[root@rhel7~] dracut --force -H --add multipath
If multipath was not working on the first boot, reboot. It should work fine on the second boot.
Update the boot list:
[root@rhel7~]bootlist -m both /dev/mapper/mpatha
You can also write a basic script and run it by using the Activation Input field when deploying an image via the PowerVC. user interface.
Check Multipath Status
Check multipath status by running multipath -l and lvm setup with pvs and bootlist. The LVM volumes must be running on mpath devices, not /dev/sdx.
[root@rhel7~] multipath -l
mpatha (360050768018107bdf800000000000158) dm-0 AIX ,VDASD
size=50G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
|- 0:0:2:0 sda 8:0 active undef running
`- 1:0:1:0 sdb 8:16 active undef running
[root@rhel7~] pvs
PV VG Fmt Attr PSize PFree
/dev/mapper/mpatha3 rhel_linuxlpar lvm2 a-- 59.50g 64.00m
[root@rhel7~]bootlist -m normal -o
sda
sdb
Watch for Part II of this series for technical explanation, failover and multibus setup. In the meantime, these steps will help you take advantage of multipath in your environment. If you have any questions, please comment below. And don’t forget to follow us on LinkedIn, Facebook, and Twitter!
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
* Crash log might be similar to the following
Nov 18 15:37:24 test2.localdomain kernel: device-mapper: table: 253:2: multipath
Nov 18 15:37:24 test2.localdomain kernel: device-mapper: ioctl: error adding tar
Nov 18 15:37:25 test2.localdomain multipathd[588]: mpatha: failed in domap for a
Nov 18 15:37:25 test2.localdomain multipathd[588]: uevent trigger error
Nov 18 15:37:25 test2.localdomain multipathd[588]: dm-2: remove map (uevent)
Nov 18 15:37:25 test2.localdomain multipathd[588]: dm-2: remove map (uevent)
Nov 18 15:37:25 test2.localdomain multipathd[588]: dm-2: remove map (uevent)
Nov 18 15:37:25 test2.localdomain multipathd[588]: dm-2: remove map (uevent)
Nov 18 15:38:53 test2.localdomain systemd[1]: Job dev-disk-by\x2duuid-b4f64cd0\x
Nov 18 15:38:53 test2.localdomain systemd[1]: Timed out waiting for device dev-d
If you have questions, please comment below. We’d love to hear from you! Don’t forget to follow us on LinkedIn, Facebook, and Twitter.
Author: Ozan Uzun
IBM Systems Lab Services
Power Systems Consultant
About the author: I have been working for IBM Systems for the last 3.5 years. My areas of interest are Linux and Open Source Solutions. I started using Linux with Slackware on the late 90's and still enjoy it.
