 | Please note...
The description below assumes that you are using a POWER5/POWER6 System with dual VIOs and install SLES10 on one ore more LPARs.
If you are planning to install SLES10 on a system with direct (SAN) attached fc-adapters it should work, too but be carefull with placing the multipath.conf file in the initrd. We've seen installations which fail to boot while the multipath.conf is in the initrd. So the recommendation for such environments w/o VIOS in between the storage and the lpar is, NOT to add the multipath.conf to the initrd!
|
| Please note...
As attachement you'll find two xml-examples for autoyast - one for the automatic setup for dm-multipath and the other for an automatic installation of sles10 on dm-mulitpath plus md-raid1 (mirroring).
Both work in the environments I've tested them but they must not in your environment ;-D Feel free to adjust them as appropriate!
|
root on dm-multipath
The following text descripes the setup of the root filesystem on a dm-multipath device using SLES10.
The following scenario includes two Virtual I/O servers (VIOS) and one logical partition (LPAR) on a POWER5 based system using the IBM Advanced POWER Virtualization.
It assumes that the disk device for the LPAR is somehow shared across the two VIOS (e.g. a SAN disk).
The reason for doing this is straightforward - you want to provide some kind of high availability to your LPARs which means you can do concurrent maintenance on your
VIOS and your LPARs stay up and running or even if you want to avoid that your LPARs crashes,too when one VIOS go down. Furthermore by using dm-multipath you can
save some storage on your disk subsystems compared to using mdadm (SW-RAID). And at last the other possibility of implementing multipath root filesystem is to use
the "old" but bullet-proof mdadm. This works quite fine but the main disadvantage is that you must monitor your md-devices somehow - with dm-multipath they will be monitored and more important than that restored automatically.
Preparation for LPARs using dual VIOS
In order to get dm-multipath working on a LPAR it is required, that the correct scsi pages (0x83) are available on the system - which means that they will be passed fr
the Virtual I/O servers to the LPAR(s). This functionality was implemented in VIOS version 1.3.0 FP8.0 but unfortunately is "hidden" or not activated per default.
Nevertheless it is there and it is fully supported by IBM.
To activate the script you must add the required lines to the ODM of the VIOS - which sounds horrible but is quite easy to accomplish.
As a first step copy or create the following script on the VIOS.
print "PdAt:" > /tmp/vpd.add
print " uniquetype = \"adapter/vdevice/IBM,v-scsi-host\"" >>/tmp/vpd.add
print " attribute = \"feature_support\"" >>/tmp/vpd.add
print " deflt = \"enable_vpd_pages\"" >>/tmp/vpd.add
print " values = \"\"" >>/tmp/vpd.add
print " width = \"\"" >>/tmp/vpd.add
print " type = \"R\"" >>/tmp/vpd.add
print " generic = \"\"" >>/tmp/vpd.add
print " rep = \"\"" >>/tmp/vpd.add
print " nls_index = 0" >>/tmp/vpd.add
print " " >>/tmp/vpd.add
odmadd /tmp/vpd.add
savebase
rm /tmp/vpd.add
Now login to the VIOS as user padmin whith the corresponding password. In the next step you must leave the restricted shell using oem_setup_env.
Run the script in the oem_setup_environment.
If you want to add the capability to just one or two adapters simply remove the adapters in the unrestricted shell using rmdev -l vhostX -R where X stands for the number of the vhost adapter (e.g. vhost10) and then run cfgmgr to redetect the adapter.
If you want to enable all adapters in the system to pass the scsi pages to client LPARs, remove all adapters and then reboot!
Finally leave the unrestricted shell or wait until the VIOS is online again.
Don't forget to repeat the steps on the second VIOS!!!
That's it. Everything else works the way you know, like setting the reserve_policy of the hdisks (e.g. chdev -dev hdisk5 -attr reserve_policy=no_reserve) on both VIOS and map map the scsi disks (like hdisk5 or whatever it is you are planning to use) to the appropriate vscsi adapters. Once again, this must be done on both VIOS!
Installation of SLES10
Beside the manual installation you can use AutoYast to automatically install and setup of SLES10 on dm-multipath. I've attached an example file - feel free to adjust it as appropriate.
 | Warning!
Due to a bug in the SLES10 and SLES10-SP1 kernel, multipathing will remount all filesystems read-only if the active path fails.
This is fixed!!!! So please please please update your installation using YOU to the latest and greatest versions of the kernel and the multipath-tools (which are kernel-ppc64-2.6.16.53-0.16 and multipath-tools-0.4.7-34.23 at this point in time).
But this will lead to another problem with lilo which is not able to update the bootloader on a system with a running dm-multipath setup.
You might get the following error:
570-lpar10:~ # lilo
running on chrp
Boot target is /dev/sda1
add note section for RS6K
MKDOSFS options: -s 1 -F 16 -n 'rootlv'
+ bootpartkernelname=sda1
++ dmsetup ls
+ [[ mpath0-part2 (253, 2)
mpath0-part1 (253, 1)
vg00-rootlv (253, 3)
mpath0 (253, 0)
vg00-swaplv (253, 4) == *sda1* ]]
+ set +x
mkfs.msdos 2.11 (12 Mar 2005)
mkfs.msdos: unable to open /dev/sda1
There's a solution available - but nevertheless it would be nice if Novell fixes this issue because this is a annoying bug...ok, after you've installed the new kernel reboot your system and start a network installation. But be sure not to use an autoyast file!!!! Instead Abort the installation and then on the "Manual setup program" select 4) Start Installation or System and then 3) Start Rescue System - answer the remaining questions as appropriate.
At the rescue login, login a user root...
Mount your original root filesystem...e.g. mount /dev/vg00/rootvg /mnt
Copy the device files for /dev/sda* and /dev/sdb* to your mounted root filesystem...e.g. cp -a /dev/sda* /mnt/dev etc.
Now chroot...e.g. chroot /mnt
Finally update your bootloader...e.g. lilo
The output should be something like this:
Rescue:/# lilo
running on chrp
Boot target is /dev/sda1
add note section for RS6K
MKDOSFS options: -s 1 -F 16 -n 'rootlv'
+ bootpartkernelname=sda1
++ dmsetup ls
+ [[ vg00-rootlv (253, 0)
vg00-swaplv (253, 1) == *sda1* ]]
+ set +x
mkfs.msdos 2.11 (12 Mar 2005)
`/lib/lilo/chrp/yaboot.chrp' -> `boot/yaboot'
`/tmp/ppc_lilo/yaboot.conf' -> `boot/yaboot.cnf'
`/boot/vmlinux-2.6.16.53-0.16-ppc64' -> `boot/vmlinux.001'
`/boot/initrd-2.6.16.53-0.16-ppc64' -> `boot/initrd.001'
Couldn't open /dev/nvram: No such file or directory
Prepending '/vdevice/v-scsi@30000003/disk@8100000000000000:1,yaboot' to open firmware variable boot-device
Couldn't open /dev/nvram: No such file or directory
Couldn't open /dev/nvram: No such file or directory
FYI: The error messages regarding /dev/nvram are okay and could be negated.
Reboot your system and you should startup with the new kernel version
|
The only thing you must be aware when installing SLES10 on multipath devices is, that you don't have to care or even think about that the device is a multipath one! This means the installer is not able to handle dm-multipath devices during installation.
So install SLES10 like you normally do using only one disk. Forget about the second one YaST shows you for the moment!
|
Don't forget to install the multipath-tools!! |
Steps required after installation
After you've successfully installed SLES10 and the system is up and running there are a few steps necessary to implement root on dm-multipath.
| Please note...
Don't get confused when trying to use multipath after the initial reboot. It won't let you add the multipath as long as they are online. So you'll have to reboot in order to see if it works. |
After the system is up and running you can check your devices with fdisk for example.
570-lpar8:~ # fdisk -l
Disk /dev/sda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 3 24066 6 FAT16
/dev/sda2 4 1305 10458315 8e Linux LVM
Disk /dev/sdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 * 1 3 24066 6 FAT16
/dev/sdb2 4 1305 10458315 8e Linux LVM
But more important than that is to see if you're getting the required scsi page 0x83 using the scsi_id command like shown below.
570-lpar8:~ # scsi_id -g -u -s /block/sda
3600a0b80000cd0490000001e45dbf9a3
570-lpar8:~ # scsi_id -g -u -s /block/sdb
3600a0b80000cd0490000001e45dbf9a3
As you can see in the example above the devices sda and sdb are identically - or in other words there are two ways to the same device but unfortunately Linux is not aware of this fact, yet.
So the first thing you might want to do is to copy one of the multipath configuration file templates available in /usr/share/doc/packages/multipath-tools.
This is not required but it allows you to assign names to the multipath devices which tends to be more meaningfull. Ok, here's an example, and please note
that I just added four lines to the multipath.conf file to fit my needs - but hey, feel free to adjust it to fit your needs!
570-lpar8:~ # cp /usr/share/doc/packages/multipath-tools/multipath.conf.annotated /etc/multipath.conf
570-lpar8:~ # vi /etc/multipath.conf
...
multipaths {
...skipping some output here...
multipath {
wwid 3600a0b80000cd0490000001e45dbf9a3
alias system
}
}
...and skipping some output there...
Note the wwid I've specified - this is the output of the scsi_id command.
Now start the multipath services as shown below.
570-lpar8:~ # /etc/init.d/boot.multipath start
570-lpar8:~ # /etc/init.d/multipathd start
You can try multipath -d -v 3 to see if it's working - but be carefull it will probably show you two multipaths - one for the first and the other of the second partition on the disk.
Now activate the services to load during boot - this mainly because of additional multipath devices you might want/need to choose. But this is optional.
570-lpar8:~ # chkconfig boot.multipath on
570-lpar8:~ # chkconfig multipathd on
Finally rebuild your initrd - and this is important - using the option -f mpath!!!
570-lpar8:~ # mkinitrd -f mpath
Root device: /dev/rootvg/rootlv (mounted on / as ext3)
Module list: ibmvscsic dm_mod dm-mod dm-snapshot dm-multipath dm-round-robin dm-emc (xennet xenblk dm-mod dm-snapshot)
Kernel image: /boot/vmlinux-2.6.16.21-0.8-ppc64
Initrd image: /boot/initrd-2.6.16.21-0.8-ppc64
Shared libs: lib/ld-2.4.so lib/libacl.so.1.1.0 lib/libattr.so.1.1.0 lib/libblkid.so.1.0 lib/libc-2.4.so lib/libcom_err.so.2.1 lib/libdevmapper.so.1.02 lib/libdl-2.4.so lib/libext2fs.so.2.4 lib/libhistory.so.5.1 lib/libncurses.so.5.5 lib/libpthread-2.4.so lib/libreadline.so.5.1 lib/librt-2.4.so lib/libsysfs.so.1.0.3 lib/libuuid.so.1.2 lib64/ld-2.4.so lib64/libc-2.4.so
Driver modules: scsi_mod sd_mod ibmvscsic dm-mod dm-snapshot dm-multipath dm-round-robin dm-emc
Filesystem modules:
Including: initramfs dm/mpath dm/lvm2 fsck.ext3
19385 blocks
Don't forget to update your bootloader using lilo.
570-lpar8:~ # lilo
running on chrp
Boot target is /dev/sda1
mkfs.msdos 2.11 (12 Mar 2005)
+ bootpartkernelname=sda1
++ dmsetup ls
+ [[ system (253, 2)
rootvg-swaplv (253, 1)
rootvg-rootlv (253, 0) == *sda1* ]]
+ set +x
`/lib/lilo/chrp/yaboot.chrp' -> `boot/yaboot'
`/tmp/ppc_lilo/yaboot.conf' -> `boot/yaboot.cnf'
`/boot/vmlinux' -> `boot/vmlinux.001'
`/boot/initrd' -> `boot/initrd.001'
Prepending '/vdevice/v-scsi@30000003/disk@8100000000000000:1,yaboot' to open firmware variable boot-device
| Please note...
During the creation of this article I was facing the situation that the creation of the bootloader failed - saying that /dev/sda1 is already mounted or busy - and this error could lead to the destruction of the PReP boot partition which means the system (LPAR) cannot find any disk devices or in most cases in an not-updated boot partition..
I had to reboot the system manually using network boot and then start the installed system. After that reboot the system has no multipath but I was able to recreate the PReP boot partition and I was able to reboot the system.
But more important that that it seems that there's a bug in the lilo script which prevents the creation of an initrd if you've updated the kernel to a new release/version/whatever! So think twice if you want to (kernel-)upgrade your system!!! As far as I know it should be fixed with Service Pack 1 . Even with SP1 the situation is the same. As descriped above you'll have to boot the system in rescue mode and update the bootloader manually.
For the first recreation of the PReP boot partition after creation of the initial ram disk for dm-multipath everything worked fine!!! |
| Catch22...
Testing w/Sles10 sp1 u1 I have found that if you have booted into the multipath root environment, running lilo
resets the boot partition from a PPC PReP Boot to a FAT16, thus forcing you to network boot, and recover the
boot partition
Hmm, the conversion from PReP Boot to FAT16 is the default behaviour but the open firmware is capable to boot either from PReP Boot or FAT16 partition... |
Finally you can check if your bootlist is set correctly using the bootlist command.
570-lpar8:~ # bootlist -m normal -r
/vdevice/v-scsi@30000003/disk@8100000000000000:1,yaboot
/vdevice/v-scsi@30000003/disk@8200000000000000:1,yaboot
And yes, that's it. No magic, no voodoo just Linux 
If you use LVM2...
...don't forget to update your lvm.conf to enable the use of dm-multipath devices.
570-lpar8:~ # vi /etc/lvm/lvm.conf
...skipping...
#filter = [ "r|/dev/.*/by-path/.*|", "r|/dev/.*/by-id/.*|", "a/.*/" ]
filter = [ "a|/dev/.*/by-id/.*", a|/dev/.*/by-name/.*|", "r|.*|" ]
...skipping...
types = [ "device-mapper", 1 ]
...skipping...
The first "filter" (the one which is commented out) is the original one. This tells LVM2 to scan every device expect devices by-path and by-id and it will lead you to some output that it found some duplicates. Not nice.
The second "filter" only scans for devices found under /dev/disk/by-name and /dev/disk/by-id and ignores the rest.
Please note that I've found it necessary to include /dev/disk/by-id because it could happen that /dev/disk/by-name was not present at boot time!
Feel free to adjust the filter as desired/required!
The second entry called device-mapper is required.
Check your settings with a simple vgscan - it should not return anything else than that it is scanning the physical devices plus the volume groups it has found during this scan.
570-lpar8:~ # vgscan
Reading all physical volumes. This may take a while...
Found volume group "rootvg" using metadata type lvm2
Ok - ready? Restart your LPAR.
Checking if it is working the way it should
Now the system has restarted login as user root and check if multipath is running.
570-lpar8:~ # multipath -ll
system (3600a0b80000cd0490000001e45dbf9a3) AIX,VDASD
[size=10G][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][active]
\_ 0:0:1:0 sda 8:0 [active][ready]
\_ round-robin 0 [prio=1][enabled]
\_ 1:0:1:0 sdb 8:16 [active][ready]
Check if your LVM2 configuration is correct:
570-lpar8:~ # pvscan
PV /dev/disk/by-name/system-part2 VG rootvg lvm2 [9.94 GB / 0 free]
Total: 1 [9.94 GB] / in use: 1 [9.94 GB] / in no VG: 0 [0 ]
And now it is getting interesting (if you'll like such things) - I am removing one path from the first VIOS and watch the output of /var/log/messages.
Jul 19 13:33:10 570-lpar8 multipathd: sda: readsector0 checker reports path is down
Jul 19 13:33:10 570-lpar8 multipathd: checker failed path 8:0 in map system
Jul 19 13:33:10 570-lpar8 kernel: device-mapper: dm-multipath: Failing path 8:0.
Jul 19 13:33:15 570-lpar8 multipathd: system: remaining active paths: 1
Jul 19 13:33:15 570-lpar8 multipathd: sda: readsector0 checker reports path is down
In addition the output of multipath -ll shows the failed path.
570-lpar8:~ # multipath -ll
sda: checker msg is "readsector0 checker reports path is down"
system (3600a0b80000cd0490000001e45dbf9a3) AIX,VDASD
[size=10G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][enabled]
\_ 0:0:1:0 sda 8:0 [failed][faulty]
\_ round-robin 0 [prio=1][active]
\_ 1:0:1:0 sdb 8:16 [active][ready]
And after the VIOS has remapped the disk the path is up and running again.
Jul 19 13:33:55 570-lpar8 multipathd: 8:0: reinstated
Jul 19 13:33:55 570-lpar8 multipathd: system: remaining active paths: 2
...
570-lpar8:~ # multipath -ll
system (3600a0b80000cd0490000001e45dbf9a3) AIX,VDASD
[size=10G][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][enabled]
\_ 0:0:1:0 sda 8:0 [active][ready]
\_ round-robin 0 [prio=1][active]
\_ 1:0:1:0 sdb 8:16 [active][ready]
So yes, it is working and it is quite easy to setup!
