Automate the application build and distribution process

Well-organized directory structures and good scripting make it easy

You have enough to consider when building an open source application for a single type of system, but what if you're building that application for distribution among a range of different, incompatible machines? There's no easy answer, but using a little discipline and some custom scripts, you can simplify the process. This article looks at how to create a structure for building and distributing applications, including heavily customized versions, and a simple way of disseminating the applications among a number of machines, manually or automatically, as easily as possible.

Share:

Martin Brown (questions@mcslp.com), Freelance Writer and Consultant

Martin C. Brown is a former IT Director with experience in cross-platform integration. A keen developer, he has produced dynamic sites for blue-chip customers including HP and Oracle and is the Technical Director of Foodware.net. Now a freelance writer and consultant, MC, as he is better known, works closely with Microsoft as an SME, is the LAMP Technologies Editor for LinuxWorld magazine, is a core member of the AnswerSquad.com team, and has written a number of books on topics as diverse as Microsoft Certification, iMacs, and open source programming. Despite his best attempts, he remains a regular and voracious programmer on many platforms and numerous environments. You can contact MC at questions@mcslp.com, or through his Web site.



14 September 2004

I always self-build applications on my systems rather than using one of the available binary or RPM distributions. This isn't because I don't trust RPMs, but because for many systems I use a customized environment. I might want debugging switched on or a reduced or specific set of extensions, modules, or other options. Sometimes, I even create a completely different directory structure for a specific project.

To complicate matters, I work with a wide range of platforms that use open source projects, including Linux™, BSD, OS X, and various commercial UNIX® distributions. For some of these, I only have one machine; for others, I have 10, 20, or even 50 machines of each type. When a new release of a piece of software becomes available, it can be a mammoth task to update each machine by hand. The actual build and install process -- assuming we ignore the customization -- is generally easy enough, as shown in Listing 1:

Listing 1. The build and install process
$ bunzip2 -c latest-release.tar.bz|tar xf -
$ cd latest-release
$ ./configure
$ make
$ make install

But without some discipline I could easily be doing that by hand on hundreds of machines. Sometimes it feels like a full-time job just trying to keep up with the latest versions, let alone working with beta and alpha releases of certain packages for testing purposes.

A few years ago I decided to simplify the process quite significantly by following a few simple rules and adding structure to the process:

  1. Create a directory for each OS
  2. Create a directory within each OS directory for each software package
  3. Create a directory for each release
  4. Build once for each OS
  5. Deploy many times on each machine

You can start by building a suitable structure for all the software that you can share across the network.

Creating a central build directory

The first step is to create a directory, which you'll share through NFS, that will be used to hold applications while they are being built. Once built, you can use the directory to hold everything from the source directories to the configuration files and other components you'll need from the system. NFS is key, though. For the system to work, you need to be able to access this central build directory from all of the machines on the network that will need access to the applications you are building and installing.

Creating a top-level structure

The first stage is to create the top-level structure for each of the operating systems that you need to support on the network. Each directory will be specific to a particular operating system, architecture, and, if the changes are significant, the version numbers. These directories should match the machines on your network. For example, I have a wide selection of machines on my network, so my top-level directory structure is quite extensive, as you can see from Listing 2:

Listing 2. Operating system-level directories in my build environment
total 58
drwxrwxrwx   2 root     other        512 Jul 25 12:01 aix/
drwxrwxrwx   2 root     other        512 Jul 25 13:14 darwin-ppc/
drwxrwxrwx   2 root     other        512 Jul 25 13:14 darwin-x86/
drwxrwxrwx   2 root     other        512 Jul 25 12:02 freebsd-x86/
drwxrwxrwx   2 root     other        512 Jul 25 13:14 freedbsd-sparc/
drwxrwxrwx   3 root     other       1024 Mar  9  2003 incoming/
drwxrwxrwx   2 root     other        512 Jul 25 12:02 linux-debian-x86/
drwxrwxrwx   2 root     other        512 Jul 25 12:02 linux-fedora-1-x86/
drwxrwxrwx   2 root     other        512 Jul 25 12:02 linux-fedora-2-x86/
drwxrwxrwx  11 root     other        512 Apr 13 16:56 linux-redhat-9.0-x86/
drwxr-xr-x   3 root     root        8192 Feb  6  2002 lost+found/
drwxrwxrwx  13 root     other        512 Jun 15 15:02 macosx/
drwxrwxrwx   2 root     other        512 Jul 25 12:02 netbsd-x86/
drwxrwxrwx   2 root     other        512 Jul 25 12:02 openbsd/
lrwxrwxrwx   1 root     other         12 Jul 25 13:13 solaris -> solaris8-x86/
drwxrwxrwx   2 root     other        512 Jul 25 13:13 solaris8-sparc/
drwxrwxrwx  32 root     other       1024 Jul 25 13:12 solaris8-x86/
drwxrwxrwx   2 root     other        512 Jul 25 13:13 solaris9-sparc/
drwxrwxrwx   2 root     other        512 Jul 25 13:13 solaris9-x86/

Don't worry too much right now about the applications. These operating system directories will be used to hold all the relevant application directories for each OS revision. Customization of application builds will be covered individually in each directory.

I use a separate file system for the build directory exported as /export/build. The file system has plenty of space; for the build directory you'll need enough space for a fully built version of each application on each operating system in each configuration. For example, Perl v5.9.0 compiled takes up about 90 MB. Apache takes up about 27 MB.

Creating an application structure

Within each operating system directory, you need to create an additional directory for each application that you expect to install. Note that these are specific to each OS and you need to create only the application directories that are actually required. For example, you might install Perl on your Fedora machines, but not on others. You can see a sample layout in Listing 3:

Listing 3. Application directories in an OS directory
drwxrwxrwx   4 mc       staff        512 May 12 12:30 apache/
drwxrwxrwx   4 root     other        512 Apr 16  2003 assassin/
drwxrwxrwx   3 501      other        512 Jun 11 10:11 bind/
drwxrwxrwx   3 root     other        512 Jul 24 15:37 ccache/
drwxrwxrwx   5 root     other        512 Jun 21 17:12 clamav/
drwxrwxrwx   6 root     other        512 Jul 25 15:28 cpan/
drwxrwxrwx   4 root     other        512 Jul 25 12:35 cpanplus/
drwxrwxrwx   4 root     other        512 Sep  6  2003 cyrus/
drwxrwxrwx   3 root     other        512 Jun  6 16:29 dhcp/
drwxrwxrwx   3 root     other        512 Apr 11 15:42 distcc/
drwxrwxrwx   3 root     other        512 May 11 10:39 eggdrop/
drwxrwxrwx   3 root     other        512 Apr  9 05:50 ethereal/
drwxrwxrwx   5 root     other        512 Jun  8 19:12 gawk/
drwxrwxrwx   2 root     other        512 May 15 08:19 gcc/
drwxrwxrwx   4 root     other        512 Jul 28  2003 ircd/
drwxrwxrwx   4 root     other        512 Apr  9 05:44 libpcap/
drwxrwxrwx   5 root     other        512 Jul 25 13:22 mysql/
drwxrwxrwx   3 root     other        512 Mar 25 13:07 nmap/
drwxrwxrwx   3 root     other        512 Nov 21  2003 ogsi-lite/
drwxrwxrwx   6 root     other        512 Jul 25 13:33 perl/
drwxrwxrwx   3 root     other        512 Aug 19  2003 php/
drwxrwxrwx   6 root     other        512 Mar  5 15:09 ppro/
drwxrwxrwx   3 root     other        512 May 15 09:02 python/
drwxrwxrwx   4 root     other        512 Apr 22 06:06 razor/
drwxrwxrwx   3 root     other        512 Aug 24  2003 razorsdk/
drwxrwxrwx   4 root     other        512 Dec  4  2003 sendmail/
drwxrwxrwx   3 root     other        512 May 23 09:37 ssh/
drwxrwxrwx   4 root     other        512 May 11 10:25 tcl/
drwxrwxrwx   3 root     other        512 Dec 22  2003 tcpwrappers/
drwxrwxrwx   6 root     other        512 Apr 24 07:13 xmltv/

Within each of these directories, you need to create one of two structures, depending on the application and how you choose to organize and configure your individual builds. Each directory you create will be used to hold a live application, that is a configured and built version of the application. You can organize this in two ways, per-version directories or direct application versions:

  • Per-version directories: one additional directory level for each version of the application. This is useful for applications where you expect to create a number of different build configurations for each version. For example, you might create one directory called perl-5.9.0/debug-build and another called perl-5.9.0/std-build>.
  • Direct version directories: each subdirectory is the directory created by the original tarball. This works if you use the same configuration across all machines for a given OS.

You can use either structure, or both, depending on how you expect to build and deploy your applications. Just make sure you are aware of the structure in each case, because you'll need to know the directory layout as you start deployment. For example, for Apache I might have three or four directories for each version to hold different configurations for debugging, and installation directories for the development, staging, and production versions. That gives a layout like the one in Listing 4:

Listing 4. An Apache layout supporting different active configurations
./httpd-2.0.46
./httpd-2.0.46/staging-build
./httpd-2.0.46/devel-build
./httpd-2.0.46/prod-build
./httpd-2.0.49
./httpd-2.0.49/staging-build
./httpd-2.0.49/devel-build

The devel-build directory contains the same directory that would be created when you extracted the original tarball. By comparison, Listing 5 shows a layout of Perl on a development machine that has just every version of Perl in a standard build installed on it. You can see from the listing that we have a simpler structure.

Listing 5. Direct version application directories
./perl/
./perl/perl-5.8.2
./perl/perl-5.8.3
./perl/perl-5.8.5
./perl/perl-5.9.0

With the directories in place, you can start to build each application once for each operating system and configuration that you want to support. With each application/configuration built once, deploying each one across operating systems should be relatively easy and straightforward. You can actually simplify this, too, with just a little bit of forethought.


Configuring individual builds

One of the additional problems with supporting many different applications and different configurations is that it's easy to forget exactly what the configuration was for a given application and situation. When building Apache, for example, I specify an alternate deployment directory and an explicit list of the modules and components that I want to support. This works fine the first time I work out what I want, but it becomes a nightmare to remember certain aspects of the configuration when I go back to the process three months later and decide to do a new build of the latest revision.

To get around this, I use the directory structure just created in combination with some simple, one-line scripts that hold the line I would normally use for configuration. For example, I use the script shown in Listing 6 for my Apache staging servers:

Listing 6. Script for Apache staging servers
./configure --enable-so --enable-mods-shared=most --prefix=/export/httpd2/staging

You can place that line into a script, give it a suitable name, and put that within the directory for the application. Again, I can do this either on a per-version basis (vital for applications that have changed their configuration system, say from Apache 1.3.x to Apache 2.x), or in the general application directory. For example, I could name this file apache/staging-config to indicate that I can use it within any version and build directory for configuring an Apache staging server.

To actually perform a configuration and build for a given operating system, application, and configuration, I would do the following:

Listing 7. Performing a configuration and build
$ cd redhat-linux-9.0/apache/httpd-2.0.46/staging-build
$ ../../staging-config
$ make

Automating installation

Once you have the first compiled and ready-to-install version of an application for an operating system, actually distributing and installing the application on the various machines in your network becomes easy. You don't have to run the configure/build process on each machine individually -- providing the machines are the same. All you need to run is make install.

I automate this on a system-by-system level by creating a file in each operating system directory that contains a list of the applications that I want to install on each machine. For some, I'll even create a number of different files to take care of all the different types of installations I have to do. The file is just a list of the application build directories, as shown in Listing 8:

Listing 8. Operating system application selections
perl/perl-5.9.0
python/Python-2.3
ccache/ccache-2.3
distcc/distcc-2.13

The files are processed by the script shown in Listing 9. There's nothing fancy here; you just go through each of the directories that listed and run make install in each one.

Listing 9. Your automatic installation script
#!/bin/sh

os=$1
src=$2
currentdir=`pwd`
basedir=/export/build

if [ -z "$os" ]
then
	echo "No operating system definition specified"
	exit 1
fi

if [ -z "$src" ]
then
	echo "No application definition specified"
	exit 1
fi

cd $basedir

if [ -r "$basedir/$os/$src" ]
then
	for dir in `cat $os/$src`
	  do
	  cd $basedir/$os/$dir
	  make install
	done
else
	echo "No application definition specified"
	exit 1
fi

Now to install a particular set of applications on a machine, all you have to do is run the script and supply the operating system folder and application selection files, like this one shown in Listing 10:

Listing 10. Installing a particular set of applications on a machine
$ appinstall.sh solaris8-x86 standard

The script, and the carefully constructed directory structure, do the rest.


Summary

Using this method for installing and distributing applications takes a while to set up and get used to, but once running, it becomes almost automatic. In fact, because you populate a file with the information about which applications you can install, you can also use the same system to automatically update machines on the network by running the script as a cron job. I run mine once a week, although you could probably get by running it just once a month. For 99 percent of the installers out there, it doesn't matter if the current version and the version being installed are the same. All the installer will do is overwrite identical copies of the files that were probably used last time the installer ran.

Best of all, if you use the automatic method, bringing all your machines up to date is simply a case of updating the application selection file with the later versions of the build directories. The latest installation will then be installed during the scheduled update by all of your machines. No fancy scripts or working with rsync, ssh, or rsh are needed. And you get to configure and customize your applications to your requirements and allow different machines to use different applications sets.

As I said at the start, all you need is a little discipline. And maybe a lot of disk space.

Resources

  • If you need help compiling software from sources, check out the Compiling and installing software from sources tutorial.
  • The greatest application in the world is only as good as your build process. Read Debugging configure (developerWorks, December 2003) to learn what to do when an automatic configuration script doesn't work.
  • Dealing with Linux config files can chew up a lot of valuable time, but Managing Linux configuration files (developerWorks, June 2004) shows how to simplify this chore using a tool you might not have considered: CVS.
  • In a developerWorks three-part series, learn how to package your software using RPM, the Red Hat package manager. Part 1 covers how to package simple software, Part 2 shows how to handle more complex packaging situations, and Part 3 explains how to incorporate scripts into your packaged software.
  • Create Debian Linux packages (developerWorks, July 2003) shows the basics of the Debian packaging system.
  • As an alternative to RPM or Debian packaging, learn how to Manage packages using Stow (developerWorks, February 2003), a Perl-based tarfile utility.
  • For industrial-strength software and systems configuration management for Linux and many other platforms, read about the Tivoli Change and Configuration Management Solution.
  • The Tivoli zone of developerWorks offers technical resources for implementing and administering Tivoli® products.
  • Find more resources for Linux developers in the developerWorks Linux zone.
  • Develop and test your Linux applications using the latest IBM tools and middleware with a developerWorks Subscription: you get IBM software from WebSphere®, DB2®, Lotus®, Rational®, and Tivoli, and a license to use the software for 12 months, all for less money than you might think.
  • Download no-charge trial versions of selected developerWorks Subscription products that run on Linux, including WebSphere Studio Application Developer, WebSphere Application Server, DB2 Universal Database, Tivoli Access Manager, and Tivoli Directory Server, and explore how-to articles and tech support, in the Speed-start your Linux app section of developerWorks.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Linux on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Linux
ArticleID=15115
ArticleTitle=Automate the application build and distribution process
publish-date=09142004