Multipath storage with Xen and DS4800
Provide multipath SAN storage access to a Xen environment with IBM System Storage DS4800
Virtualization technology was part of the IBM mainframe decades before virtualization gained its current industry adoption. The recent broader adoption of virtualization is due to the fact that enterprise datacenters can no longer keep up with the demands of new applications by adding racks of servers. Datacenters are already overstretched in terms of server count, electricity consumed, and heat generated per square foot of raised floor space.
While virtualization, which reduces server count by increasing utilization efficiency and lowering operating costs, is an excellent solution to the problem of overwhelming demand, it can also add a burden to system availability since more applications are now running in each box. Any failure can now cause more disruption to business.
One way to counter this is by using a fully redundant path to access external SAN (storage area network) devices—in this way, you can reduce failures caused by storage subsystem access problems.
This article uses Red Hat Enterprise Linux 5 update 1 (RHEL 5.1) to show you how to enable multipath access in Xen host and guests with the IBM-provided RDAC (redundant disk array controller) multipath driver for accessing the IBM System Storage DS4800.
Virtualization and Xen
Much virtual ink has been put to virtual paper to explain the various types of virtualization technologies out there, such as hardware partition, logical partition, hypervisor, and so on. Xen is a paravirtualization hypervisor technology. In other words, the Xen hypervisor is a thin layer of software that sits on top of the hardware; each guest operating system's kernel and drivers have to be modified to run on it.
The advantage is that by collaborating between guest OS and hypervisor when making low-level calls, a Xen guest can achieve near native performance in most cases (which is to say that this type of virtualization is the most cost-effective implementation on commodity hardware).
The downside is that only modified kernels are supported, so you can not use it to consolidate older operating systems such as Windows® 2000. However, since virtualization is becoming such a hot selling point, chip manufacturers are rushing to provide further support to this trend so processors that have Intel-VT or AMD Pacifica feature are able to support full virtualization for running unmodified guests on the Xen hypervisor, albeit with some performance hit.
With the Xen hypervisor now at version 3 and widely used, some vendors have decided to include it as part of their Linux distribution. For instance, in Red Hat Enterprise Linux 5 and Novell SUSE Linux Enterprise Server 10, Xen is robust, supported, and last but not least, free!
When you start up a Xen host, the Xen hypervisor takes control of the system first, then it loads the first guest OS, which is called the domain 0 (or Dom0) in Xen lingo. User interacts with Xen through the Dom0. All the device drivers for accessing hardware are also loaded in Dom0. Dom0 provides virtual block device and virtual network device to the other guest OS, which is sometime referred to as domain U (or DomU). From here on, I won't consider Dom0 as a guest OS anymore, although it is essentially just a guest instance with some special capabilities.
Within each guest (DomU), there are some lightweight device drivers for accessing virtual block devices and virtual network devices; they are called frontend drivers to distinguish them from the heavy-lifting backend drivers running in Dom0. Figure 1 shows the Xen architecture.
Figure 1. Simplified architecture of Xen
Actually, starting from Xen 3.0.3, the virtual block frontend is implemented in
the module xenblk, while the virtual block backend is handled by the modules blkbk
and blktap. The virtual network frontend is handled by the module xennet and
the backend by netbk. To see which modules are loaded in your Linux system, use
modinfo for module information.
Multipath SAN storage
As people move storage from direct host attachment into a shared SAN storage infrastructure, they're adding complexity to the picture. What was once a dedicated SCSI or Fibre Channel cable between the host and storage enclosure now is a bunch of FC cables, SAN switches and director, and a storage server with its own controller, disk enclosures, cables, and so on. While the benefits of SAN storage are indisputable, there are also disadvantages due to higher probability of component failures between the hosts and the actual hard drives. Multipath data access is a remedy to this issue.
Essentially, the host has more than one independent channel to access the same logical drive (using the logical unit number, or LUN) on the storage server. A multipath driver combines all the paths to show only one block device to the operating system. Each LUN has a preferred path, and if that fails, the multipath driver routes I/O requests to the alternate paths. Multipath drivers are not made equal: some are able to handle more intelligent load balancing, while others can handle only failover. For the DS4800 storage server, IBM recommends and provides the RDAC driver to use on top of the HBA (host bust adapter) driver. Remember, the multipath driver does not replace the HBA driver: they complement each other. The multipath driver is analogous to the brain, and the HBA driver is analogous to the arms and legs. Modern operating systems also provide their own multipath driver, but make sure it has been certified by your storage vendor before using it.
Hardware and software
Enough theory; let's look at the architecture of the test environment. I have an IBM x3550 server with an HBA card that has two FC ports. Each port is connected to a different SAN switch. The DS4800 storage server has two controllers that have four FC ports on each. For simplicity, I show only two FC connections in Figure 2; each one is connected to the same set of switches as the host. So, any interruption to one of the access path does not affect storage access from the host. The RDAC driver installed in domain 0 takes care of using the preferred path in normal circumstances and switching to the alternate path in case of interruption. The guest OS doesn't need any special driver to have multipath access to the backend storage.
Figure 2. Test environment architecture
Here's the hardware and software in my test environment:
- IBM System x3550
- 2 Dual-Core Xeon, 4 GB memory
- QLogic HBA adapter QLE2462
- Red Hat Enterprise Linux 5 update 1
- IBM RDAC driver, version 09.01.C5.11
- QLogic driver, version qla2xxx 8.01.07.15
- IBM System Storage DS 4800 model 1815-84A
- 2 controllers
- 16 FC disks of 136GB
- 8 FC ports at 4Gbps
- Brocade switches, model 2109-F32
- 32 ports at 2Gbps
Setting up the storage
Because the IBM SAN storage servers were designed for multipath access, you don't need to do anything special to make a logical drive accessible from multiple paths. In the case of DS4800, when creating a logical volume, it automatically alternates controller ownership as the preferred path for load balancing. In Figure 3, I created 4 LUNs and mapped them to my host group. Remember that LUN numbers have to be contiguous for the host to be able to scan them all, and it starts at 0. If you remove a logical drive, make sure the other LUN numbers remain contiguous!
Figure 3. Logical drives created in the DS4800 mapped to my host group
At the switch fabric, I put the first WWPN of the host HBA into one zone, along with a WWPN of storage controller A. Then I put the second WWPN of the host HBA with a WWPN of controller B into another zone.
Setting up the Xen host
Setting up the Xen hypervisor and domain 0 is straightforward. The procedure goes like this:
- Install RHEL 5.1 normally on a local drive. You can install for SAN boot as well, but that is outside the scope of this article.
- After you have RHEL 5.1 up and running, install the Xen kernel from the
installation source, like this:
Listing 1. Installing the Xen kernel packages
rpm -ivh kernel-xen-2.6.18-53.el5.i686.rpm rpm -ivh kernel-xen-devel-2.6.18-53.el5.i686.rpm
- Now you should have a new entry in your boot manager, /boot/grub/grub.conf. Make sure it boots the Xen kernel by default. Reboot the system.
- Verify that you're now running the Xen kernel after reboot:
Listing 2. Verifying the Xen kernel
[root@xenhost ~]# uname -a Linux xenhost 2.6.18-53.el5xen #1 SMP Wed Oct 10 17:06:12 EDT 2007 i686 i686 i386 GNU/Linux
- Congratulation! You now have a Xen hypervisor with Dom0 running on top of it.
Installing the Xen tools
Now that Xen is running, you need to install some tools to interact with it. In the RHEL 5.1 installation source, the Xen tools are in the /VT directory. The basic command-line tools are very powerful but not pretty, and the graphical tool is easy to use but not powerful. That's life! Anyway, let's go ahead and install both of them.
These are the dependent packages that you have to install first; get them from the /Server directory on the DVD or CDs:
Listing 3. Installing the dependent packages for Xen tools
rpm -ivh bridge-utils-1.1-2.i386.rpm rpm -ivh dnsmasq-2.39-2.el5.i386.rpm rpm -ivh gnome-python2-gnomekeyring-2.16.0-1.fc6.i386.rpm rpm -ivh xen-libs-3.0.3-41.el5.i386.rpm
When installing the Xen tools packages, it seems to me that there is a circular dependency no matter what order I install them in, so I have to force the first package to install without dependency verification and then install the rest in this order:
Listing 4. Installing the Xen tools packages
rpm -ivh --nodeps libvirt-0.2.3-9.el5.i386.rpm rpm -ivh libvirt-devel-0.2.3-9.el5.i386.rpm rpm -ivh libvirt-python-0.2.3-9.el5.i386.rpm rpm -ivh python-virtinst-0.103.0-3.el5.noarch.rpm rpm -ivh xen-3.0.3-41.el5.i386.rpm rpm -ivh xen-devel-3.0.3-41.el5.i386.rpm rpm -ivh virt-manager-0.4.0-3.el5.i386.rpm rpm -ivh gnome-applet-vm-0.1.2-1.el5.i386.rpm rpm -ivh Virtualization-en-US-5.1.0-12.noarch.rpm
Alright, let's kick some tires and try some commands now! But before you do that,
you have to start the xend daemon. All of the Xen management tools communicate
with the xend daemon to extract information. To start it, run
xend start as root. Normally the xend service is set to
start at system boot, so you don't have to do this every time.
Now you can run the command
xm info to see some
information from the Xen hypervisor.
Listing 5. Displaying the Xen hypervisor information
[root@xenhost ~]# xm info host : xenhost release : 2.6.18-53.el5xen version : #1 SMP Wed Oct 10 17:06:12 EDT 2007 machine : i686 nr_cpus : 4 nr_nodes : 1 sockets_per_node : 2 cores_per_socket : 2 threads_per_core : 1 cpu_mhz : 2992 hw_caps : bfebfbff:20100000:00000000:00000140:0004e3bd:00000000:00000001 total_memory : 4095 free_memory : 20 xen_major : 3 xen_minor : 1 xen_extra : .0-53.el5 xen_caps : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p xen_pagesize : 4096 platform_params : virt_start=0xf5800000 xen_changeset : unavailable cc_compiler : gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) cc_compile_by : brewbuilder cc_compile_domain : build.redhat.com cc_compile_date : Wed Oct 10 16:30:55 EDT 2007 xend_config_format : 2
Then you can run the command
xm list to see the
running virtual machines. Naturally at this point, you'll only see the domain 0
running. You can use
xentop for this as well.
Listing 6. Listing the running Xen guests
[root@xenhost ~]# xm list Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 2928 4 r----- 61790.3
If those two commands run successfully, then you can be assured that your Xen environment is configured properly. If the commands can't connect to the xend daemon, try to reboot the machine. Alternatively, you can look at the Xen logs in the /var/log/xen directory.
If you prefer a graphical tool, log on to a Gnome session, and then go to the menu Applications > System Tools > Virtual Machine Manager. You should see the status of your virtual machines as in Figure 4.
Figure 4. Virtual Machine Manager main panel
Installing the multipath driver
Now that the Xen environment is up and running, it's time to install the drivers to access the SAN storage. The first thing to install is the HBA adapter driver, which is the QLogic driver in my case. Download the QLogic driver installation package from the vendor's Web site and install it according to its documentation. You must turn off the default failover option afterward by adding an option in /etc/modprobe.conf as follows:
Listing 7. Option to disable QLogic driver failover
options qla2xxx ql2xfailover=0
It's time to install the RDAC multipath driver now. Again, download the installation package from the IBM Web site (see Related topics for a link) and follow the documentation to install the driver. It should compile the source code, install the module, and create a new initrd image. You should manually modify your boot manager configuration file /boot/grub/grub.conf to use the new initrd image. Be careful here; do not use the example provided by the installation script, since that isn't applicable to a Xen host. Instead use the current format in grub.conf, and substitute initrd-xyz.img with mpp-xyz.img.
My example looks like this:
Listing 8. Boot options in grub.conf
title Red Hat Enterprise Linux Server with RDAC driver (2.6.18-53.el5xen) root (hd0,0) kernel /xen.gz-2.6.18-53.el5 module /vmlinuz-2.6.18-53.el5xen ro root=LABEL=/ rhgb quiet module /mpp-2.6.18-53.el5xen.img
After reboot, list all the logical drives with the command
/opt/mpp/lsvdev. If you're not seeing all the LUNs, run
Listing 9. Listing the LUNs detected by domain 0
[root@xenhost ~]# /opt/mpp/lsvdev Array Name Lun sd device ------------------------------------- ARCJMTDS48K1 0 -> /dev/sdb ARCJMTDS48K1 1 -> /dev/sdc ARCJMTDS48K1 2 -> /dev/sdd ARCJMTDS48K1 3 -> /dev/sde
Now you should have multipath access to your SAN storage from domain 0.
Setting up the Xen guests
Before installing the Xen guests, you need to do two things on the host:
- Make sure that the virtual network is set up properly.
- Make the installation media available to the Xen guests through the network, which could be HTTP, FTP, or NFS—but it doesn't take a locally mounted ISO image!
For my virtual network, I bound it to the eth0 adapter specifically. You can use the graphical Virtual Machine Manager (VMM) to do that; go to the menu Edit > Host details > Virtual Networks to modify or add a new one.
Figure 5. Virtual network configuration in VMM
For the installation media, I simply extract all the files from the DVD image
and put them under /var/www/html/rhel51 directory, then start the Apache Web
service httpd start. All the files are then
accessible within the Xen guests as http://192.168.100.1/rhel51. Make sure you
don't have iptable blocking incoming connection to port 80.
For the first guest OS, I want it to have three logical drives mapped directly to three LUNs on the DS4800 through Dom0, of course.
Figure 6. Storage devices mapping for the guest test0
- Click on the New button at the main panel of VMM.
Figure 7. VMM main panel with domain 0
- Give a name to the guest, such as test0.
- Choose the Paravirtualized mode.
- At the next screen, type:
in the Install Media URL field.
- Assign a Normal Disk Partition to the guest; the first one should be
/dev/sdb from the perspective of the host. The remaining disks can be added
Figure 8. Assigning storage space to the guest test0
- Choose the Virtual Network that you've set up previously.
- Allocate memory to the guest. In my case, I set the startup memory to 512MB
and max memory to 1GB. For virtual CPU, I set it to 2.
Figure 9. Allocating memory to the guest test0
- Here is the summary panel that I have:
Figure 10. Guest test0 creation summary panel
- Click Finish to start the creation process.
- At this point, you should see the familiar Red Hat installation program kick
in. Just follow through to complete the installation. This takes a
while, as usual.
Figure 11. Installation begins at the guest test0
- At any time during or after the installation, you may add the two remaining
logical drives to this guest. At the VMM main panel, select test0 and click
Figure 12. VMM main panel with 1 guest
- At the Hardware tab, click Add.
Figure 13. Hardware details panel of the guest test0
- Accept the default choice of Storage device.
- Choose a Normal Disk Partition and type
- Click Finish to complete the process.
- Repeat the process to add /dev/sdd.
- Now we should have three disks defined to the guest.
Figure 14. After adding 2 storage devices to guest test0
- Find an appropriate time to shut down and restart the guest to make the change effective—rebooting won't do here.
To access the guest OS:
- You can use the graphical console as you did during OS installation.
- From the host terminal session, you can run
xm console test0to access a text console.
- You can ssh to test0 if you know its DHCP-assigned IP address.
To see the new disks, you can run
fdisk -l and
fdisk to create partition on them.
The guest configuration is stored in a plain text file in the /etc/xen directory. For example, here's the content of /etc/xen/test0:
Listing 10. Configuration file of guest test0
[root@xenhost ~]# cat /etc/xen/test0 name = "test0" uuid = "dddf02f6-5f90-74a5-0098-365a51b54282" maxmem = 1000 memory = 500 vcpus = 2 bootloader = "/usr/bin/pygrub" on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" vfb = [ "type=vnc,vncunused=1,keymap=en-us" ] disk = [ "phy:/dev/sdb,xvda,w", "phy:/dev/sdc,xvdb,w", "phy:/dev/sdd,xvdc,w" ] vif = [ "mac=00:16:3e:79:2f:e1,bridge=vnet0" ]
Creating this file from scratch isn't easy, but it's fairly easy to modify this
file manually. For example, if you want to change the memory allocation or number
of virtual CPU, you just need to edit this file and shut down/restart the guest
with the command
xm shutdown test0, then
xm create test0.
Or you can use the VMM tool.
For the sake of this exercise, I modified the name associated to each disk in
the guest from
hdx like this:
Listing 11. Changing virtual device names
disk = [ "phy:/dev/sdb,hda,w", "phy:/dev/sdc,hdb,w", "phy:/dev/sdd,hdc,w" ]
Actually, it doesn't matter how you name it; you can try
sdx as well. The guest simply boots the first
block device in the list. After creating partitions and filesystems on the virtual
disks, I get this from within test0:
Listing 12. Listing of all the filesystems in test0
[root@test0 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 3.8G 1.8G 1.8G 51% / /dev/hda1 99M 13M 82M 14% /boot tmpfs 251M 0 251M 0% /dev/shm /dev/hdb1 2.0G 36M 1.9G 2% /mnt/disk1 /dev/hdc1 2.0G 36M 1.9G 2% /mnt/disk2
This completes the setup process for test0, which now has multipath access to three LUNs on the DS4800.
For the second guest, I want to map image files in Dom0 as virtual disks to the
guest. The LUN #3 on DS4800 is made available to Dom0 as /dev/sde. I create a
partition on it using
fdisk, then create a
filesystem on that partition using
mkfs. Then I mount the filesystem to the
default Xen image file location at
Figure 15. Storage devices mapping for the guest test1
Now let's proceed to create the guest test1 with the same procedure as earlier, by clicking New in the VMM main panel, giving it the System Name of test1, and choosing Paravirtualized. At the storage assignment screen, choose Simple File with the location of /var/lib/xen/images/test1-xvda.img. Give it a size and check the option Allocate entire virtual disk now.
Figure 16. Assigning storage space to the guest test1
Then, step through the other screens as we did before; the OS installation should kick in again at the end. During the installation, open the VMM test1 details panel to add two more disks using the image file name of /var/lib/xen/images/test1-xvdb.img and /var/lib/xen/images/test1-xvdc.img. Then you should have three disks as shown in Figure 17.
Figure 17. Hardware details panel of the guest test1
This is the configuration file that I have for test1:
Listing 13. Configuration file of guest test1
[root@xenhost ~]# cat /etc/xen/test1 name = "test1" uuid = "53b39c1e9edc6143a06d4011154beab9" maxmem = 1000 memory = 600 vcpus = 2 bootloader = "/usr/bin/pygrub" on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" vfb = [ "type=vnc,vncunused=1,keymap=en-us" ] disk = [ "tap:aio:/var/lib/xen/images/test1-xvda.img,xvda,w", "tap:aio:/var/lib/xen/images/test1-xvdb.img,xvdb,w", "tap:aio:/var/lib/xen/images/ test1-xvdc.img,xvdc,w" ] vif = [ "mac=00:16:3e:7b:78:63,bridge=vnet0" ]
tap:aio is used to represent a logical disk
based on the image file, while the tag
phy represents a
physical disk. After OS installation, shut down and restart test1, and you should
see the two added disks with
fdisk -l. After creating
partitions and filesystems on these disks, I have this in test1:
Listing 14. Listing of all the filesystems in test1
[root@test1 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 4.7G 1.8G 2.9G 39% / /dev/xvda1 99M 13M 82M 14% /boot tmpfs 301M 0 301M 0% /dev/shm /dev/xvdb1 2.0G 35M 1.8G 2% /mnt/disk1 /dev/xvdc1 2.0G 35M 1.8G 2% /mnt/disk2
At this point, both guests are fully installed and configured to access their respective virtual disks with multipath through domain 0. My VMM is showing this now:
Figure 18. VMM main panel with 2 guests
Pulling some cables
To make sure that multipath access is functional in both guests, I run an IBM internal I/O workload tool in each guest. It writes block of data, then reads and verifies continuously on all three virtual disks. Then I purposely interrupt one of the access paths by pulling off an FC cable between the host and switch. The I/O tool freezes for about 5 seconds, and then resumes activities as usual.
To simulate a SAN switch failure, I disable a port on the other switch where the host is connected to, and again, the I/O tool continues to run normally.
Finally, I disable one of the controllers on the DS4800 and as expected, the I/O tool continues to run flawlessly! Obviously, between each test, I restore the environment back to a functional state and wait a few minutes for the multipath driver to reestablish its connection to the broken path.
In this article, you've seen how to set up a Xen environment with multipath access to the IBM DS4800 storage server. You've learned how multipath access can be used to dramatically increase the uptime of Xen virtual machines in the event of storage path failure or storage infrastructure maintenance.
If a dual path does not give you sufficient availability and bandwidth, you can have four or up to eight paths! The high-end storage servers even have the intelligence to load-balance access paths if you have a need for that. So, if you're building a virtualized environment with SAN storage, you should definitely consider multipath access in your strategy to achieve high availability, reliability, and serviceability.
- In "Virtual Linux" (developerWorks, December 2006), take a tour of virtualization methods, architectures, and implementations.
- "Virtualization with coLinux" (developerWorks, March 2007) introduces coLinux for Linux/Windows virtualization, and gives you another path to virtualization.
- Downloads for DS4800 Midrange Disk System provides support and downloads, including firmware, storage management and host bus adapter software, and management and diagnostic tools.
- Xen.org delivers up-to-date info on the Xen hypervisor, which provides a feature set for virtualization of x86, x86_64, IA64, PowerPC™, and other CPU architectures, as well as a wide range of guest operating systems including Windows, Linux, Solaris, and various versions of the BSD operating systems.
- In the developerWorks Linux zone, find more resources for Linux developers, and scan our most popular articles and tutorials.
- See all Linux tips and Linux tutorials on developerWorks.