Introduction

In this three-part series, I’ve been introducing you to a clustering technology for Linux called openMosix. As you may know, clustering technologies allow two or more networked Linux systems (called "nodes") to combine their computing resources to solve computing challenges faster than would be possible if they were tackling the problem on their own. Choice of hardware is flexible of course, and openMosix is not limited to any single hardware platform, but clusters built on IBM xSeries systems running Intel® Xeon^TM processors will have some unique advantages. Making use of performance-enhancing technologies such as Intel's Hyper-Threading Technology, now supported under Linux, improves the performance of multi-threaded applications by allowing a single Xeon processor to appear to the operating system as two virtual processors. By taking advantage of Hyper-Threading, you benefit from having multiple physical and/or virtual processors, and also enjoy the benefits of openMosix itself.

As this series progresses, I'll guide you through the process of setting up your own openMosix cluster. By the end of the series, you'll have your openMosix mini-cluster up and running and will be ready to use it to effectively accelerate your computing tasks.

In this second installment, I'll walk you through the actual steps of setting up your openMosix cluster. To follow along, you'll need two or more Linux systems connected on a LAN via Ethernet or some other high-speed network transport.

Back to top

Getting started

The process of setting up an openMosix cluster is actually quite simple. First, we'll install openMosix-enabled Linux kernels on all our Linux systems, each of which will become a node in our cluster. Then, we'll install the openMosix userland tools on each system. Finally, we'll configure a few configuration files in /etc and create an openMosix startup script. Then, we will reboot our nodes and execute the openMosix startup script on each node in our cluster; at this point, openMosix will be enabled and processes will be able to migrate to the nodes where they can execute most efficiently.

Before we begin, a note. If you're using Debian GNU/Linux on any of your nodes, you can type "apt-get openmosix" to have apt perform much of the nitty-gritty openMosix setup for you. Also, if you're using Gentoo Linux, you can type "emerge sys-apps/openmosix-user", which will install an openMosix kernel source tree in /usr/src/linux along with the openMosix userland tools. To make things even easier, the openMosix team now has openMosix RPMs available for your use. However, while these automatic installation methods are convenient if you already know how to use openMosix, they aren't a very good educational tool. Therefore, you may want to skip the available automatic installation options and follow along as we install openMosix from scratch, particularly if you've never used openMosix before. By doing so, you'll get a better feel for what makes openMosix tick.

To get started, first head over to the openMosix main page. We'll need to download two files—the first is the openMosix kernel patch. We'll apply this patch to a specific version of the stock Linux kernel, thereby adding openMosix extensions. At the time this article was written, the most recent version of the openMosix kernel patch was 2.4.17pre3. This name is a bit confusing because this particular patch is named "openMosix-2.4.16-3.gz" and should be applied against a stock 2.4.16 kernel source tree. There's a note on the main openMosix page that helps to clarify this discrepancy.

To begin our adventure, go ahead and download the most recent version of the openMosix kernel patch, being sure to note which kernel source tree version to which it should be applied—somewhere on the openMosix main page should indicate which kernel source tree it depends upon.

You'll also need to grab the latest version of the openMosix userland tools, which will be named something like "openMosixUserland-0.1.3.tgz." You should now have two files on disk: one the openMosix kernel patch, and the other the userland tools source tarball.

Back to top

Creating an openMosix kernel

There may be one additional file that you'll need to download. If you don't have the appropriate stock kernel source tree on hand, head over to http://www.kernel.org/pub/linux/kernel/v2.4/ and download the kernel sources that the openMosix patch depends upon. Since my version of the openMosix patch is designed to work with 2.4.16 sources, I grabbed linux-2.4.16.tar.gz. Once the kernel sources have completed downloading, we'll install our new openMosix kernel source tree as follows:

# cd /usr/src
# mv linux linux.old
# tar xzvf /path/to/linux-2.4.16.tar.gz
# cd linux
# cat /path/to/openMosix-2.4.16-3.gz | gzip -d | patch -p1 -l

You'll now have openMosix-enabled sources in /usr/src/linux, and can follow the traditional kernel configuration, compilation, and installation steps to install a new openMosix-enabled kernel on each of your nodes. All you'll need to do is enable the appropriate settings under the new MOSIX configuration section, which you can access via "make menuconfig." Under the MOSIX section, you should enable at least the following options:

[*] openMOSIX process migration support
[*] Stricter security on openMOSIX portsrt
[*] Direct File-System Access
[*] openMOSIX File-System
[*] Poll/Select exceptions on pipes

Configure, compile, and install the kernel as you would normally. Once all of your nodes have an openMosix kernel installed, we'll work towards getting the userland tools installed on each node.

Back to top

Installing the userland tools

The openMosix userland tools are fairly easy to compile and install on each system manually (see the included "Installation" file). However, since it can be a pain to compile and install the userland tools on each machine individually, we're going to compile them on one node.

Then, we'll create a tarball containing all the files that need to be installed on each node in our cluster. Once the tarball is created, we can simply extract it to the root directory of every node in our cluster. This'll also make setup easier if we choose to add additional nodes to our cluster at a later date. To get our userland distribution tarball ready, first choose a node with an openMosix-enabled kernel source tree installed in /usr/src/linux, and type:

# cd /tmp
# tar xzvf /path/to/openMosixUserland-0.1.3.tgz
# cd openMosixUserland-0.1.3

Then, according to the "Installation" instructions, modify the "configuration" file so it looks identical (or at least close) to this:

OPENMOSIX=/usr/src/linux
PROCDIR=/proc/mosix
MONNAME=mmon
CC=gcc
INSTALLDIR=/usr
CFLAGS=-I/m/include -I./ -I/usr/include -I\$(OPENMOSIX)/include -O2
INSTALL=/usr/bin/install

Now compile the tools by typing:

# make clean build man

Install the tools into a temporary directory as follows:

# mkdir /tmp/openmosix-tools
# make INSTALLDIR=/tmp/openmosix-tools/usr install
# mkdir /tmp/openmosix-tools/etc

Back to top

Final installation steps

A snapshot of all the openMosix userland tools is now installed inside /tmp/openmosix-tools. While we could simply tar up this directory and extract it to / on each of our nodes, we're going to add a few more things to our distribution tarball in order to save us some time. Let's add an openMosix startup/shutdown script and configuration file to our distribution tarball so that everything can be installed in one fell swoop. First, we'll create the /etc/mosix.map file, which describes the configuration of our cluster. To create a version of /etc/mosix.map for distribution, type "vi /tmp/openmosix-tools/etc/mosix.map" (replace "vi" with the name of your favorite editor, of course), and then add a line for each node in your cluster in the following format:

node number         ip address         span

node number should be set to the official number of each node, starting at 1 and being incremented by 1 on each successive line. Follow the node number by the node's IP address, and then specify a "1" for the span argument.

My mosix.map file, consisting of two nodes with IP addresses of 192.168.1.1 and 192.168.1.9, looks like this:

1       192.168.1.1       1
2       192.168.1.9       1

If I wanted to specify a cluster of 100 nodes with consecutive IPs starting at 192.168.1.1 and going to 192.168.1.100, I could take advantage of the span argument and add a single line that looks like this:

1       192.168.1.1       100

Without the span argument, I'd need to create a mosix.map file with 100 lines in it (one for each node), which would be a pain. OK, now that our mosix.map file has been created, go ahead and save mosix.map to disk and exit the editor.

Now we'll create an openMosix rc script that is used to turn openMosix on and off.

First, type:

# install -d /tmp/openmosix-tools/etc/rc.d/init.d

Type vi /tmp/openmosix-tools/etc/rc.d/init.d/openmosix and paste in the following script:

#!/bin/bash
case $1 in
stop)
        echo "Stopping openMosix"
        echo 0 > /proc/mosix/admin/mospe
        rm -f /var/lock/subsys/mosix
        ;;
start)
	echo "Starting openMosix"
        if [ -s /etc/overheads -a -f /proc/mosix/admin/overheads ]
        then
                 grep -v '^#' /etc/overheads > /proc/mosix/admin/overheads
        fi
        if [ -s /etc/mfscosts -a -f /proc/mosix/admin/mfscosts ]
        then
                 grep -v '^#' /etc/mfscosts > /proc/mosix/admin/mfscosts
        fi
        local a1
        local a2
        a1=
        [ -s /etc/mospe ] && a1="-p `cat /etc/mospe`"
        a2=
        [ -s /etc/mosgates ] && a2="-g `cat /etc/mosgates`"
        setpe -W $a1 $a2 -f /etc/mosix.map
        touch /var/lock/subsys/mosix
  ;;
esac

Finally, we'll make the script executable by typing:

# chmod +x /tmp/openmosix-tools/etc/rc.d/init.d/openmosix

Before we create our openMosix userland distribution tarball, we'll create one more directory, /mfs. We'll use /mfs when we configure each node to mount a special openMosix networked filesystem to the /mfs mount point.

Type:

# mkdir /tmp/openmosix-tools/mfs

Now we're ready to create our userland distribution tarball by performing the following steps:

# cd /tmp/openmosix-tools
# tar czvf /tmp/openmosix-tools.tar.gz .

You should now have an "openmosix-tools.tar.gz" tarball in your /tmp directory.

On each node in your cluster (including the one you're on now), install the tarball by typing:

# tar xzpvf /path/to/openmosix-tools.tar.gz -C /

The openMosix userland tools are now installed. We're almost ready to reboot all the nodes and start our openMosix cluster. But first, log in to each node and add to the following line to each /etc/fstab file:

mymfs         /mfs         fs         dfsa=1         0         0

This new line will ensure that the openMosix networked filesystem is mounted to the /mfs mount point when our nodes are restarted. Now, take the appropriate steps to add the /etc/rc.d/init.d/openmosix script to each node's default runlevel; the openmosix script requires that your network is started first—take that into account when choosing when the script will run. Now you're ready to reboot all your nodes. When all our systems complete the boot process, they'll be part of a single openMosix cluster.

Back to top

OpenMosix up and running!

Once all the nodes are up and running, log in to one of them and type "mmon," which brings up a console-based animated bar graph of the loads on each of the nodes in your cluster.

The openMosix filesystem should also be functional on every node in your cluster. Look inside the /mfs/1 directory. You’ll find node 1's entire filesystem tree. You'll also find additional numeric directories for every node in your cluster. The openMosix filesystem is extremely handy for cluster-related tasks, but can also be used to simply copy files back and forth between the various nodes in your cluster. Inside /mfs, you'll also see various other directories such as "home." These directories provide logical (rather than direct) mappings to certain nodes in your cluster.

OK, your cluster is up and running! Before concluding, let's do a little test to see openMosix in action. Log in to a node in your cluster, and start up "mmon." We're going to run several CPU-intensive programs to demonstrate openMosix's process migration capabilities. Open up a new shell, and type in the following mini-script at the shell prompt:

# for x in 1 2 3 4
do
 awk 'BEGIN {for(i=0;i<10000;i++)for(j=0;j<10000;j++);}' &
done

Four CPU-intensive awk scripts will begin executing immediately. In your mmon window, you should see the system load on your current node rise sharply.

However, within a few seconds you should notice the system load of your current node drop as several awk processes are migrated to other nodes in your cluster. In turn, you will see the load that was once concentrated on a single node spread over up to four nodes in your cluster. Welcome to the world of openMosix!

Please join me in my next article, when we will look at various ways to take avantage of our new openMosix cluster. See you then!

Resources

• Read the other installments in this series:
  • "Advantages of OpenMosix on IBM xSeries, Part 1" (developerWorks, October 2002)
  • "Advantages of OpenMosix on IBM xSeries, Part 3" (developerWorks, December 2002)
  
  
  
• Visit the home of the openMosix project on SourceForge.
  
  
• Check out Qlusters, Inc., the developers of next-generation openMosix functionality.
  
  
• Keep up with all the stuff the project leader of openMosix is doing at Moshe Bar's home page.
  
  
• Visit the Intel Developer Web site.
  
  

About the author

Rate this article

Comments

Back to top

Table of contents

• Introduction
• Getting started
• Creating an openMosix kernel
• Installing the userland tools
• Final installation steps
• OpenMosix up and running!
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

Dig deeper into Linux on developerWorks

Discover knowledge paths

Skill-building guides for Linux, open source, cloud, Java, business analytics, and more.

Special offers