Create an ooRexx build environment on Linux KVM

Kernel Virtual Machine improves build performance

Construct an on-demand software build service using ooRexx that uses the Linux® Kernel Virtual Machine (KVM) for better performance. KVM acts as the host for the guest operating systems that build the target software for the user. The Apache Web server controls the builds and stores the results for later retrieval by the user. Learn how to set up the build server and create guests, customize build requests, and organize and access build results.

W. David Ashley, Senior IT Specialist, IBM

David is a Senior IT Specialist for IBM Lab Services. He has over 25 years of IT software development experience.



14 July 2009

Also available in Japanese

Recently, the Open Object Rexx project (ooRexx; see Resources later in this article for more info) converted its old on-demand software build system from VMware-hosted guest operating systems to Linux Kernel Virtual Machine (KVM)-hosted guests. This change provided a more efficient build environment and reduced the build times for users.

The ooRexx software build system allows developers to build ooRexx software packages for multiple x86-based platforms and operating systems. Currently, the supported guest operating systems include Windows® XP (i386), Fedora 10 (i386 and x86_64), and Ubuntu 8.04 (i386). These guest operating systems can produce ooRexx installation and documentation packages for Windows (EXE), Fedora and openSUSE (RPM), and Ubuntu (DEB). Other x86-based operating systems will come online as demanded by the developers and users of ooRexx.

This article shows how to create your own software build system, taking the ooRexx development team's setup as an example and including tips and direction as needed for developers experienced with ooRexx, Apache, and Linux. You can download the server and guest scripts at the end of this article. As designed, the system is specifically for building the ooRexx software, but the concepts can be applied to a general purpose build system for any software.

The broad requirements for this system include the following:

  • You need a Web interface for generating the build request.
  • You need a Web interface for retrieving the build results.
  • You need support for multiple guest operating systems.
  • Guest operating systems must perform fully automated builds.
  • An e-mail should be generated and sent to the requesting user at the end of the build.

To accommodate these requirements, the development team and I used a quad-core Xeon-based server. The server contains 4GB of memory and 250GB of disk space. We chose the Fedora 10 x86_64 distribution as the main operating system mainly due to the stability and up-to-date release of KVM used by the distribution. Your choice of hardware and software may be different, but the main hardware criterion is that your processor should have hardware virtualization available—it's a must in order to use KVM.

Set up the server

The first step in setting up the build server is to determine the partitioning scheme. We decided to segregate the Web store and the images for the guest operating systems into separate partitions. We set aside 50GB for the Web store and about 150GB for the /var partition where the guest images are placed. The rest was divided between the /home partition and the /root partition.

General requirements of a build system

Some of the more basic requirements of a build system include:

  • Frequent builds to catch problems as early as possible
  • Faster builds (the faster they are, the more you can do)
  • Incremental build processing (or build avoidance) to reflect small development updates
  • Support for managing (at least the low-level) source code dependencies to keep your system as flexible as possible
  • Extraction/reporting capabilities on build, compile, and link usage
  • A reporting system that can trace source to binary matching (efficiently comparing the old to the new)
  • The ability to report on build status and things like testing pass or fail
  • The ability to create release notes and system documentation

Next, we installed the main operating system using the Fedora 10 x86_64 release distribution. If you're setting up your own system, you'll save yourself a lot of trouble if you do the following:

  • Enable hardware virtualization via the machine's BIOS prior to starting the installation so that Fedora will see that KVM is available.
  • Perform a custom install of the software components so that you can select the Fedora virtualization options.

After we installed the server operating system, we configured it for access by the guest operating systems. This involved enabling Samba for the Windows guests and NFS for the Linux guests. This gives the guests access to the build results partition so they can store their build files for access by the user. Both the main Samba share and the main NFS export point to the same location for all guests.

Next, we configured the Apache Web server to provide access to the build request system, which I discuss below under The build requests, and the build results repository.

One configuration decision you will need to make concerns the networking option for the guests. The default installation is configured to supply a private internal network for all guests. A class C network is provided along with a DHCP server to provide IP addresses to the guests. The other option is to set up the system so that one of the network devices acts as a bridge to the server external network. This will need to be configured by hand. You can see an example of how to configure your server for this option at the libvirt Wiki (see Resources for a link).


Create the guests

There are two approaches to creating guests for use by KVM. In the first approach, you create just the guests you need to meet your requirements. The second way takes a more long-range approach to creating guests. This is the method we used to create guests, and we recommend it as a standard approach if the necessary resources are available.

We first determined the number and types of guests from the requirements. We needed operating systems to create software built for those environments and another operating system for creating the documentation. It turned out that in our case, the documentation and i386 RPM tasks could be combined and handled by a single guest. Here are the guests and tasks assigned to them:

  • Windows XP (i386): Build the Windows install executable.
  • Fedora 10 (i386): Build the i386 RPM file and the documentation ZIP file.
  • Fedora 10 (x86_64): Build the x86_64 RPM file.
  • Ubuntu 8.04 (i386): Build the DEB file.

The approach we used created the previously mentioned guests as images that could be cloned at a later time. So, each guest had a basic version that we kept for later cloning and a customized clone of that guest that performed the actual build task.

Cloning a KVM guest is really easy. The virt-clone script provided by Fedora 10 can completely automate the task.

Listing 1. Fedora 10's virt-clone script
$ virt-clone --original=Fedora10-i386-Base \
             --name=Fedora10-i386-Build    \
             --file=/var/lib/libvirt/images/Fedora10-i386-Build.img

The original option specifies the name of the guest operating system as it is known to the virtual machine manager. The name option specifies the name of the new guest. The file option specifies the file name of the new image file for the guest. This will completely clone an existing guest and copy it to a new version of the guest. It will also modify the MAC address of the new guest as well as the UUID. Thus, the original guest is preserved for further cloning, if necessary, and a new guest is provided for your customization.

One thing about the virt-clone script should be noted: If the original guest has an unconnected device, such as a CD-ROM, the script will fail. All unconnected devices must be removed prior to cloning the guest. If necessary, you can add them back after the cloning operation.

Once the basic build guests were created, each was then customized for the build task. The Windows XP guest was the most challenging in this area, because the Visual C++ compiler and Software Development Kit needed to be installed as part of the customization. The Linux guests required practically no customization except for installing NFS on the Ubuntu guest.


The build requests

All the build guests require one additional customization: Open Object Rexx version 3.2 must be installed on each of them. Why? Because a script written in ooRexx controls the request and build process on each guest. The script accesses a TCP/IP port on the build machine server to search for build requests. If a request is found, the build process is invoked and a set of installation files (RPM, DEB, EXE), along with a build report, is created and stored on the build machine server for access via HTTP by the user.

The ooRexx build request script is smart enough to not duplicate an already built version of the software. It also notifies the user via e-mail that the build is finished and provides a URL to the location on the build machine server where the build was stored.

Build requests are generated via a CGI script on the build machine server. The user visits an HTTP Web page and requests a build on their choice of operating system. The Apache server then invokes the CGI script, which places the request in a queue. The build guests are then responsible for accessing those requests and performing the requested operation. The results of the build are also available via the Web site for download.

As mentioned above, the code for both the server and the build guest clients is available for download below. All the scripts are written in ooRexx and should be easy to follow. The following files are included:

  • buildserver.rex: This script runs on the Web server and maintains the queues that the guest operating systems use to look for work.
  • buildd.rex: This script runs on each guest operating system and accesses the queues on the Web server looking for work. Each guest has only one queue for which it is responsible and thus will build only one kind of output file.
  • simq.cls: This file is not a script but is a class file used by the buildserver.rex script.
  • socket.cls: This class file is used by the buildd.rex script.
  • streamsocket.cls: This class file is also used by the buildd.rex script.

The build results

The results of a build are placed on the Web site by the build guests. The builds are organized such that each Subversion revision has its own subdirectory to contain all the platform guest builds for that revision. Also, each platform guest produces a build report that is also stored in the revision subdirectory.

The build report is an absolute necessity for all this to really work. If an error occurs during the build, and the final output files are not produced, the user must have a way to discover what went wrong. So, the build process on each guest always produces a build report and stores it in the build repository.

The build repository location segregates the ooRexx software builds from the documentations builds. The documentation build produces a large number of output files, so it was deemed convenient to separate those builds from the software builds.

Currently, there is no automated build cleanup process. The project does not produce enough output to make that necessary at this time.


Conclusion

By switching from VMware build guests to Linux KVM guests, build times for each guest are reduced by as much as 50 percent. This means the development cycle for ooRexx has also been reduced.

The ooRexx development team is by far the largest user group of the build machine, but during the alpha and beta cycles for major releases, a large number of ooRexx users also make use of the environment. This allows users to provide timely feedback for any bug fixes or enhancements made to ooRexx.

One of the most-used build guests is the documentation builder. All the ooRexx documentation is coded in DocBook. This makes the documentation hard to build on a Windows system without some experience with DocBook in that environment. This is a real problem for most of our developers and users. By providing a documentation build process that everyone can access, we ensure that developers and users can contribute to and maintain the documentation. This has produced documentation for ooRexx that is much better than most open source projects, a source of pride for our development team.


Download

DescriptionNameSize
Build server and guest scriptsbuildserver.zip23KB

Resources

Learn

Get products and technologies

  • SourceForge hosts Open Object Rexx downloads.
  • The IBM REXX family includes the Compiler and Library for REXX on zSeries, REXX for VSE, and REXX for CICS.
  • With IBM trial software, available for download directly from developerWorks, build your next development project on Linux.

Discuss

  • Get involved in the My developerWorks community; with your personal profile and custom home page, you can tailor developerWorks to your interests and interact with other developerWorks users.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Linux on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Linux, Open source
ArticleID=412930
ArticleTitle=Create an ooRexx build environment on Linux KVM
publish-date=07142009