About a decade ago, when I started coding software for a living, my team and I used to write code, run some tests locally with the modified sources, and then check the code in. There were just too many tests, so it made sense that the developer ran only a required subset of the tests. Every evening at about 8:00 pm, a cron job would compile the sources and run integration testing. Every couple of days, it turned out that developer A's code and developer B's code ran fine in isolation, but integration testing would fail.

The rapid pace of software development over the past decade has meant that we need better tools for integration testing. This is where CruiseControl comes in (see Resources for a link). CruiseControl is a continuous-integration tool that meshes seamlessly with source code management systems, such as Apache Subversion. Another tool that you can learn about in this article is SCons (see Resources)—a a build tool, similar to Makefile, except that it is based on Python. So you are spared the ordeal of learning yet another build system-specific language such as make or jam.

Working with CruiseControl

CruiseControl works seamlessly with quite a few tools, including the Concurrent Versions System (CVS), Subversion, Git, Perforce, Microsoft® Visual SourceSafe, and IBM® Rational® ClearCase®. This article uses Subversion as the revision control tool of choice.

You can download CruiseControl from the project home page (see Resources for a link. The latest revision as of this writing is 2.8.4, and that is what this article refers to.

The core of the CruiseControl system is config.xml, which triggers builds and runs tests in response to code changes, and then notifies you of the results. CruiseControl defines several tags that the XML file typically uses: <listeners>, <bootstrappers>, <modificationset>, <schedule>, and <publishers>.

Creating a configuration file

The root element of the configuration is called cruisecontrol (aptly enough), and it takes no attributes. The root element typically contains one or more property elements. These property elements contain name-value pairs and are an obvious means of setting log directories, log files, and so on. Listing 1 shows a basic configuration.

	
<?xml version="1.0"?>
<cruisecontrol>
  <property name="builddir" value=""/opt/logs/project1" />
  <property name="buildlog" value="${builddir}/build.log" />
</cruisecontrol>

The next and probably the most important tag in the configuration file is the <project> tag. A configuration file can contain multiple <project> tags, each defining a separate project. If there are multiple projects and you want to run them in parallel, you need to specify the <thread> tag and make the count attribute of the <thread> tag greater than 1. This is a good option to explore if you have a build or integration server with multiple cores at your disposal. Note that the <thread> tag is part of the <system> tag hierarchy. Listing 2 shows the <thread> tag being used.

	
<?xml version="1.0"?>
<cruisecontrol>
  <property name="builddir" value="/opt/logs/project1" />
  <property name="buildlog" value="${builddir}/build.log" />
  <system>
     <configuration>
         <threads count="8" />
     </configuration>
  </system>
</cruisecontrol>

For every <project> tag, you need to define the name attribute. Listing 3 has multiple <project> tags.

	
<?xml version="1.0"?>
<cruisecontrol>
  <property name="builddir" value=""/opt/logs/project1" />
  <property name="buildlog" value="${builddir}/build.log" />
  <system>
     <configuration>
         <threads count="8" />
     </configuration>
  </system>
  <project name="project1">
    <! need to configure project child tags>
  </project>
  <project name="project2">
     <! need to configure project child tags>
  </project>
</cruisecontrol>

The <project> tag has a couple of interesting attributes. Of particular interest to us are the buildafterfailed and requireModification attributes. If buildafterfailed is set to True, then CruiseControl keeps trying to build your project, even though the last build has failed. When is this useful? It is worth a shot when your system has too many dependencies—such as sourcing files from a remote repository and connection fails or getting information from a database that may have a response timeout. By default, buildafterfailed is False and the requireModification attribute is set to True: It tells CruiseControl to rebuild sources only if there are check-ins in the code base. Set this attribute to False if you want to have builds after short intervals, irrespective of changes in the code. In the following example, these attributes are used.

  <project name="project1" buildafterfailed=true requireModification=no>
    <! need to configure project child tags>
  </project>

Next, you need to configure the hierarchy or child tags for each individual project. First, you need to see how to associate the source control with the <project> tag. Is there a tag for that? Of course—that is what the <modificationset> tag is for. Listing 4 shows how the Subversion source repository is being accessed from CruiseControl.

	
<?xml version="1.0"?>
<cruisecontrol>
  <property name="builddir" value="/opt/logs/project1" />
  <property name="buildlog" value="${builddir}/build.log" />
  <system>
     <configuration>
         <threads count="8" />
     </configuration>
  </system>
  <project name="project1">
    <modificationset>
       <svn RepositoryLocation="svn+ssh://dev/svn/project/trunk"
          username="integration" password="integration" >
    </modificationset>
  </project>
</cruisecontrol>

If you prefer building from a local copy of the repository, consider using the LocalWorkingCopy attribute, instead (see Listing 5).

	
  <project name="project1">
    <modificationset>
       <svn LocalWorkingCopy="/build/checkout/project1">
    </modificationset>
  </project>

Note that you need to provide either the RepositoryLocation or the LocalWorkingCopy attribute with the <svn> tag. The username and password attributes are clearly optional and necessary only if your authentication system demands it. For other source controls, you have similar tags, such as <cvs> and <vss>. The code in Listing 5 also brings up an interesting question: If you are specifying only the local working copy, then whose responsibility is it to keep this copy updated? The answer to this question brings us to the <bootstrappers> tag. The <bootstrappers> tag might have child tags such as <svnbootstrapper> or <clearcasebootstrapper> depending on the repository that you are using. Listing 6 shows the bootstrapping for the configuration in Listing 5.

	
  <project name="project1">
    <bootstrappers>
       <svnbootstrappers localWorkingCopy="/build/checkout/project1" 
          username="integration" password="integration" >
    </bootstrappers>
    <modificationset>
       <svn LocalWorkingCopy="/build/checkout/project1">
    </modificationset>
  </project>

The <svnbootstrapper> tag indicates to CruiseControl the local copy of the sources on which to run the update command before the build is fired.

So, you now have the source control set up. How about deciding when the build should run? Which command should you run to fire the build? From what you have seen so far, you need a new tag: Say hello to the <schedule>. Listing 7 provides an example of <schedule> in use.

	
  <project name="project1">
    <bootstrappers>
       <svnbootstrappers localWorkingCopy="/build/checkout/project1" 
          username="integration" password="integration" >
    </bootstrappers>
    <modificationset>
       <svn LocalWorkingCopy="/build/checkout/project1">
    </modificationset>
    <schedule>
       <exec command="/opt/build-tools/mybuildscript.sh"
                  workingdir="/build/checkout/project1"
                  args="link=shared variant=debug j4"
                  errorstr="build failed" >
    </schedule>
  </project>

The <exec> tag (which is a child tag of <schedule>) runs a specified command or script as part of the build on the directory specified in workingdir. If the build fails, then there is also the option of a user-provided string in the error output using the errorstr attribute. If you want to schedule the build every hour, irrespective of code changes in the repository, <schedule> has an attribute called interval that specifies the build time interval in seconds, as shown in Listing 8.

	
  <project name="project1" requireModifications=no>
    <bootstrappers>
       <svnbootstrappers localWorkingCopy="/build/checkout/project1" 
          username="integration" password="integration" >
    </bootstrappers>
    <modificationset>
       <svn LocalWorkingCopy="/build/checkout/project1">
    </modificationset>
    <schedule interval=3600>
       <exec command="/opt/build-tools/mybuildscript.sh"
                  workingdir="/build/checkout/project1"
                  args="link=shared variant=debug j4"
                  errorstr="build failed" >
    </schedule>
  </project>

After the build has started, users will likely inquire about the status of the build. Having a web-based interface is probably one of the easiest things to provide for such cases. And this brings us to the listeners child element of the <project> tag. The tag with a particular interest to us is <currentbuildstatuslistener>, which writes a build status snippet to a file. If you want to explicitly specify the folder in which build logs should be maintained, then you need the <log> tag and its optional dir attribute to provide for the relevant folder. The <log> tag has a child tag aptly called <merge> that is used to merge miscellaneous data from the integration testing process into the CruiseControl log file. Using the command:

<merge dir=dir-name pattern="*.xml">

. . . merges all XML files in the folder dir-name with the CruiseControl log file—a neat feature if you prefer having an overall picture of both the build and the testing. Listing 9 shows the configuration.

	
  <project name="project1" requireModifications=no>
    <bootstrappers>
       <svnbootstrappers localWorkingCopy="/build/checkout/project1" 
          username="integration" password="integration" >
    </bootstrappers>
    <modificationset>
       <svn LocalWorkingCopy="/build/checkout/project1">
    </modificationset>
    <schedule interval=3600>
       <exec command="/opt/build-tools/mybuildscript.sh"
                  workingdir="/build/checkout/project1"
                  args="link=shared variant=debug j4"
                  errorstr="build failed" >
    </schedule>
    <log dir="/build/logs/project1">
       <merge dir="/build/tests/project1" pattern="*.output" />
    </log>
    <listeners>
       <currentbuildstatuslistener file="/build/staus/latest" />
    </listeners>
  </project>

You are almost done here, except that you need to take a few post-build actions. For example, developers may need a message, or may be you need to build the .zip files out of the sources. This is where the <publishers> tag comes in handy. The <publishers> tag has a whole host of child tags that allows you to customize your post-build situation. For our purposes, try the most common approach: email. You can choose either the <email> or <htmlemail> child tag of <publishers>. Listing 10 shows a sample usage.

	
  <project name="project1" requireModifications=no>
    <bootstrappers>
       <svnbootstrappers localWorkingCopy="/build/checkout/project1" 
          username="integration" password="integration" >
    </bootstrappers>
    <modificationset>
       <svn LocalWorkingCopy="/build/checkout/project1">
    </modificationset>
    <schedule interval=3600>
       <exec command="/opt/build-tools/mybuildscript.sh"
                  workingdir="/build/checkout/project1"
                  args="link=shared variant=debug j4"
                  errorstr="build failed" >
    </schedule>
    <log dir="/build/logs/project1">
       <merge dir="/build/tests/project1" pattern="*.output" />
    </log>
    <listeners>
       <currentbuildstatuslistener file="/build/staus/latest" />
    </listeners>
    <publishers>
        <email mailhost="smtp.myjob.com" subject="Build/Test Stats">
            <failure address="all@myjob.com" />
            <success address="developers@myjob.com" />
        </email> 
    </publishers>
  </project>

Back to top

SCons

Yet another nifty software build management tool is SCons. SCons stands for software construction and is easily a replacement for make. Why bother to use SCons when you have make? For starters, make, automake, autoconf, and so on come with a syntax that is peculiar to the tool, while SCons is Python based. SCons syntax is also a lot easier on the head, and the SConstruct files (similar to Makefiles) are easier to maintain. SCons also comes with specialized support for C/C++ and Java™ files, and SCons accomplishes the make functionality with a lot less code.

Hello, World with SCons

Listing 11 shows a simple Hello, World program in C++.

	
#include <iostream>
using namespace std;

int main( )
{
  cout << "Hello World" >> endl;
}

Here's the SConstruct file to create the executable:

env = Environment()
env.Program(hello.cpp)

That is it: Compare this with the make-based approach, and you can realize the benefits of SCons immediately. Program is called a builder_method—a Python application programming interface (API) that tells SCons that you want to build an executable file. It is possible to have multiple executables in the same file and also have SCons generating messages during the build process. Listing 12 provides a sample.

	
env = Environment()
env.Program('hello.cpp')

print "Done with building hello, on to hello2"       
env.Program('hello2.cpp')
print "Finished building hello2"

So, what is this env business? In SCons parlance, you are creating a construction environment, and env holds a lot of information about the compiler, linker flags, and the like.

Useful things to try with SCons

Here are some common tasks you can perform with SCons:

• You want the executable to have a custom name. That means that hello2.cpp should generate an executable called newworld2. To do it, specify the name as the first argument to Program:
  
  

  env = Environment() print "Beginning building newworld2" env.Program('newworld2', 'hello2.cpp') print "Finished building newworld2"

  
  
• You want to build an executable that depends on multiple sources. Say that newworld2 depends on hello2.cpp and world3.cpp. The SConstruct file would then look like this:
  
  

  env = Environment() print "Beginning building newworld2" env.Program('newworld2', ['hello2.cpp', 'world3.cpp']) print "Finished building newworld2"

  
  
• To clean your builds, just type scons c at a shell prompt.
• It is possible that you want to build two executables that have a number of source files in common. It is prudent to have these files compiled into object files only once. Here is an example:
  
  

  env = Environment() common = ['common1.cpp', 'common2.cpp'] hello1_sources = ['hello1.cpp'] + common hello2_sources = ['hello2.cpp', 'world2.cpp'] + common env.Program('hello1', hello1_sources) env.Program('hello2', hello2_sources)

  
  
• Executables often need to link with specific libraries, such as pthread or math. Program can handle additional arguments to specify the LIBS and LIBPATH variables. Here is a sample usage to link against the pthread library:
  
  

  env = Environment() env.Program('new-exe', 'concurrent.cpp', LIBS = 'pthread', LIBPATH = ['/usr/lib', '/usr/local/lib'])

  
  
• What if all files with a .cpp extension in a folder go toward the making of an executable? Thankfully, you do not need to specify the names of all the files. The Glob function makes your life easier:
  
  

  env = Environment() Program('new-exe', Glob('*.cpp'), LIBS = 'pthread', LIBPATH = ['/usr/lib', '/usr/local/lib'])

  
  
• Building a static library is a breeze in SCons. Just use the Library function call:
  
  

  env = Environment() env.Library('arith', ['math1.cpp', 'math2.cpp', 'math3.cpp'])

  
  

  Note that internally, SCons creates math1.o, math2.o and math3.o, then follows it up with ar rc libarith.a math1.o math2.o math3.o and finally ranlib libarith.a.

• Creating a shared library in SCons is almost as easy as creating a static library:
  
  

  env = Environment() env.SharedLibrary('arith', ['math1.cpp', 'math2.cpp', 'math3.cpp'])

  
  
• Different source files may be required to be compiled with different compiler options. For example, if you want to pass DX86_64 to the sources, indicating some optimizations made for 64-bit platforms to be compiled, here is how you should tweak the SConstruct file:
  
  

  env = Environment() env.Object('hello.cpp', CCFLAGS='-DX86_64') env.Object('world.cpp', CCFLAGS='-DX86_64') env.Program('new-exe', ['hello.o', 'world.o']

  
  

  Note that you have also introduced a new builder function called Object. Program can be built using object files as well, so no need to pass in the sources as you have done earlier.

• Throughout this discussion, I have mentioned the target as the first item. If you ever want to reverse or play around with the order, here is how you would go about it:
  
  

  common = ['common1.cpp', 'common2.cpp'] hello1_sources = ['hello1.cpp'] + common hello2_sources = ['hello2.cpp', 'world2.cpp'] + common Program(source = hello1_sources, target = 'hello1') Program(target = 'hello2', source = hello2_sources)

  
  

  Note that source and target are SCons-specific keywords.

• Although Program, SharedLibrary, and the rest are all members of the env object, it is okay to use them even without prefixing them with env. So, the following SConstruct file is just fine:
  
  

  common = ['common1.cpp', 'common2.cpp'] hello1_sources = ['hello1.cpp'] + common hello2_sources = ['hello2.cpp', 'world2.cpp'] + common Program('hello1', hello1_sources) Program('hello2', hello2_sources)

  
  

Back to top

Conclusion

In this article, you have learned about CruiseControl and continuous integration, including the more important tags, such as <bootstrappers>, <modificationset>, <schedule>, and <listeners>. You have also used SCons to build a small C++ project and learned how to build static and shared libraries. Both tools have a lot more to them than what was discussed in this article, so be sure to check out the Resources section for details.

Resources

Learn

• Learn more about CruiseControl.
  
  
• Learn more about SCons.
  
  
• Compare SCons with other build tools in the SCons wiki.
  
  
• AIX and UNIX developerWorks zone: The IBM AIX® and UNIX® zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills.
  
  
• New to AIX and UNIX? Visit the New to AIX and UNIX page to learn more.
  
  
• Technology bookstore: Browse the technology bookstore for books on this and other technical topics.
  
  

Get products and technologies

• Download the latest version of CruiseControl from the project's website.
  
  
• Download SCons version 2.1 from SourceForge.
  
  
• Learn more about and download Subversion.
  
  
• Try out IBM software for free. Download a trial version, log into an online trial, work with a product in a sandbox environment, or access it through the cloud. Choose from over 100 IBM product trials.
  
  

Discuss

• Follow developerWorks on Twitter.
  
  
• Participate in developerWorks blogs and get involved in the developerWorks community.
  
  
• Get involved in the My developerWorks community.
  
  
• Participate in the AIX and UNIX® forums:
  • AIX Forum
  • AIX Forum for developers
  • Cluster Systems Management
  • Performance Tools Forum
  • Virtualization Forum
  • More AIX and UNIX Forums
  
  

About the author

Rate this article

Comments

Back to top

Table of contents

• Working with CruiseControl
• SCons
• Conclusion
• Resources
• About the author
• Comments

Dig deeper into AIX and UNIX on developerWorks

IBM SmartCloud trial. No charge.

Unleash the power of hybrid cloud computing today!

---

Public and private solutions in one trial.

Special offers