In pursuit of code quality

Use test categorization for agile builds

Run tests at different frequencies for shorter build duration


Content series:

This content is part # of # in the series: In pursuit of code quality

Stay tuned for additional content in this series.

This content is part of the series:In pursuit of code quality

Stay tuned for additional content in this series.

If it's not too painful, try to imagine you're a developer at a startup company in early 2002. Flush with cash, you and your cohorts have been tasked to build a large data-driven Web application using the latest and greatest Java™ APIs. You and your management are firm believers in what will eventually become known as agile processes. From day one, tests have been authored with JUnit and run as often as possible, as part of an Ant build process. Eventually a cron job will be set up to run the build nightly. At some later point, someone will download CruiseControl and the growing suite of tests will be run at every check-in.

Now fast-forward to today.

Over the course of the past few years, your company has developed an enormous code base with an equally enormous suite of JUnit tests. Everything was fine until about a year ago, when your test suite eclipsed 2,000 tests and people started noticing that the build process was taking more than three hours to run. Some months before that, you stopped running unit tests via Continuous Integration (CI) during code check-ins because of the strain placed upon the CI server. You switched to running tests nightly, which made the following morning stressful as developers tried to figure out what broke and why.

These days, it seems as if the suite of tests is rarely run more than once during the evening -- and why should they be? They take forever! No one has a few hours to wait around just to find out that things are working well (or not). And besides, the entire test suite is run at night, right?

Because you run the tests so infrequently, they are often fraught with errors. Consequently, you and your team have begun to question the value of unit tests: if they are so important to code quality, why are they such a pain? You all agree there is a fundamental value to unit tests, if only you could run them in a more agile manner.

Try test categorization

What you need is a strategy to transition your build to a more agile state. You need a solution that lets you run tests more than once a day and puts your test suite back where it was before the build took three hours to complete.

Before you try to come up with a strategy for getting your entire test suite back in shape, it might help to reconsider the generic term "unit tests." Just as the statements "I have an animal at home" and "I like cars" aren't very specific, so, too, unfortunately, is the phrase "we write unit tests." These days, a unit test can mean anything.

Take the earlier statements about animals and cars: they lead to more questions. For example, what kind of animal do you have at home? Is it a cat, a lizard, or a bear? "I have a bear at home" is certainly different from "I have a cat at home." Likewise, "I like cars" isn't very helpful when talking to a car salesman. What types of cars do you like: sports cars, trucks, or station wagons? Any answer could lead you down a different path.

Likewise, with developer testing, it can be helpful to categorize tests by type. Doing so enables more precise language and can enable your team to run different test types at different frequencies. Categorization is key to avoiding the dreaded three-hour build that runs all "unit tests."

Three categories

Visualize your test suite sorted into three layers, each layer representing a different type of developer test defined by how long it takes to run. As you see in Figure 1, each layer adds more length to the total build time, either to run or, as it turns out, to author.

Figure 1. Three layers of test categorization
Three layers of test categorization
Three layers of test categorization

The bottom layer is made up of the shortest running tests, which, as you can imagine, are the easiest to write. They also touch the least amount of code. The top layer is made up of more high-level tests that exercise a greater portion of the application. These tests are a bit more difficult to code and take considerably more time to execute. The middle layer is for tests that fall between the two extremes.

The three categories are as follows:

  • Unit tests
  • Component tests
  • System tests

Let's consider each one separately.

1. Unit tests

A unit test verifies an object or multiple objects in isolation. Unit tests don't deal with databases, file systems, or anything else that would tend to make tests run longer; consequently, unit tests can be written from day one. In fact, this is exactly what JUnit was designed for. The isolated notion of unit testing is behind the myriad mock-object libraries that facilitate isolating a particular object from its external dependencies. What's more, unit tests can be written before the actual code under test is written -- hence the notion of test-first development.

Unit tests are usually easy to code because they are not reliant on architectural dependencies, and they also run quickly. On the downside, individual unit tests offer somewhat limited code coverage. The value of unit tests (which is considerable) is that they enable developers to guarantee object reliability at the lowest possible level.

Because unit tests run so quickly and are so easy to write, a code base should have a good number of them and they should be run as often as possible. You should always run them when you execute a build, whether on your machine or in the context of a CI environment (meaning you should run them anytime code is checked into an SCM system).

2. Component tests

Component tests verify multiple objects interacting together, but they breach the notion of isolation. Because component tests deal with multiple layers of an architecture, they often deal with databases, file systems, network elements, etc. Also, component tests are somewhat difficult to write early, so incorporating them into a truly test-first/test-driven scenario is challenging at best.

Component tests take longer to write because they're more involved than unit tests. On the other hand, they achieve more code coverage than unit tests do because of their wide reach. They also take longer to run, so running a lot of them together can increase your overall test time dramatically.

A host of frameworks facilitate the challenges associated with testing large architectural components. DbUnit is a perfect example of this type of framework. DbUnit makes writing tests that rely on a database easier by handling the associated complexity of seeding a database between test states.

When the testing aspect of a build is prolonged, you can generally predict that a large suite of component tests is involved. Because these tests take longer to run than true unit tests, you may find that you can't run them all the time. Accordingly, it makes sense in a CI environment to run the tests at least hourly. You should always run these tests on a local developer's box before checking in any code, too.

3. System tests

System tests verify a software application end-to-end. Consequently, they introduce a high degree of architectural complexity: the entire application must be running for a system test to take place. If it's a Web application, you need access to the database, along with a Web server, container, and any associated configuration aspects to run a system test. It follows that most system tests are written in the latter cycles of a software life cycle.

System tests are challenging to author and also take a good deal of time to actually execute. On the other hand, they offer quite a bit of bang for the buck, so to speak, in terms of architectural code coverage.

System tests are very similar to functional tests. The difference is that they don't emulate a user, they mimic one. Just as in component tests, a number of frameworks have been created to help facilitate these tests. For example, jWebUnit facilitates testing Web applications by mimicking a browser.

Implementing test categorization

So, it turns out that your suite of unit tests is really a suite of unit tests, component tests, and system tests. Moreover, you now know after inspecting the tests that the reason the build takes three hours is that the majority of them are component tests. Your next question is, how does one implement test categorization with JUnit?

You have a few options, but let's stick with the two easiest ones:

  • Create custom JUnit suite files corresponding to desired categories.
  • Create custom directories for each type of test.

Creating custom suites

You can use JUnit's TestSuite class (which is of type Test) to define a collection of tests as belonging together. You start by creating an instance of TestSuite and adding corresponding test classes or test methods to it. You can then point JUnit at the TestSuite instance by defining a public static method named suite(). All tests included will then be executed as a single run. Consequently, you can implement test categorization by creating a unit TestSuite, a component TestSuite, and a system TestSuite.

For example, the class shown in Listing 1 creates a TestSuite holding all component tests in the suite() method. Note that this class isn't very JUnit-specific. It doesn't extend TestCase, nor does it have any test cases defined. But JUnit will reflectively find the suite() method and run all tests returned by it.

Listing 1. A TestSuite for component tests

import junit.framework.Test;
import junit.framework.TestSuite;

public class ComponentTestSuite {

 public static void main(String[] args) {;

 public static Test suite(){
  TestSuite suite = new TestSuite();
  return suite;

The process of defining TestSuites does require that you go through your existing tests and add them to the corresponding class (that is, all unit tests are added in a UnitTestSuite). It also means that as you author new tests in a given category, you'll have to programmatically add them to the appropriate TestSuite, and of course, recompile them.

Running individual TestSuites then becomes an exercise in creating unique Ant tasks that invoke the correct test collection. You can define a component-test task that picks up the ComponentTestSuite and so on, as I've done in Listing 2:

Listing 2. An Ant task to run only component tests
<target name="component-test" 
 <mkdir dir="${testreportdir}"/>   
 <junit dir="./" failureproperty="test.failure" 
             fork="true" haltonerror="true">
   <sysproperty key="basedir" value="."/>     
   <formatter type="xml"/>      
   <formatter usefile="false" type="plain"/>     
    <path refid="build.classpath"/>       
    <pathelement path="${testclassesdir}"/>        
    <pathelement path="${classesdir}"/>      
   <batchtest todir="${testreportdir}">
    <fileset dir="test">
     <include name="**/"/>                 

Ideally, you would also have tasks for invoking unit tests and system tests. Lastly, for the occasions where you would like to run the entire test suite, you would create a fourth task that depended on all three test categories, as I've done in Listing 3:

Listing 3. An Ant task for all tests
<target name="test-all" depends="unit-test,component-test,system-test"/>

Creating custom TestSuites is a quick solution for implementing test categorization. The downside of this method is that as you create new tests, you must add them programmatically to the appropriate TestSuite, which can be a pain. Creating a custom directory for each type of test is more extensible and allows you to add new categorized tests without having to recompile.

Creating custom directories

I find that the easiest way to implement test categorization with JUnit is to logically divide tests into specific directories corresponding to their test type. Using this technique, all unit tests will reside in a unit directory, all component tests in a component directory, and so on.

For example, in a test directory where all uncategorized tests are kept, you can create three new subdirectories like those shown in Listing 4:

Listing 4. Directory structure implementing test categories

To run these tests, you must define at least four Ant tasks: one for unit tests, another for component tests, and so on. The fourth task is a convenience task that runs all three test types (like the one shown in Listing 3).

The JUnit task is much like the one defined in Listing 2. What's different, however, is a detail in the batchtest aspect of the task. This time, the fileset points at a specific directory. In the case of Listing 5, it's pointing at the unit directory:

Listing 5. The batchtest aspect of the JUnit task to run all unit tests
<batchtest todir="${testreportdir}">
 <fileset dir="test/unit"> 
  <include name="**/**"/>       

Note that this test only runs any tests in the test/unit directory. When new unit tests (or any other tests for that matter) are created, you simply need to drop them into the directory and things are good to go! This is a bit easier than having to add a new line to the TestSuite file and recompile it.

Problem solved!

Getting back to our original scenario, let's say that you and your team decide that using specific directories is the most extensible solution to your build-time problem. The hardest aspects of the task are the inspection and distribution of test types. You refactor your Ant build file and create four new tasks (three for individual test types and one to run them all). Moreover, you modify CruiseControl to only run your true unit tests on check-ins and component tests on an hourly basis. Upon further inspection, it turns out that system tests can also be run hourly, so you create an additional task that runs component tests and system tests together.

The end result is that tests are run many times a day and your team is able to catch integration errors more quickly -- normally within a few hours.

Creating build agility isn't exactly hip, but it certainly plays a vital role in ensuring code quality. With your tests running more often, concerns about the value of developer testing are a distant memory. And, best of all, it's now 2006 and your company is wildly successful!

Downloadable resources

Related topics


Sign in or register to add and subscribe to comments.

Zone=Java development
ArticleTitle=In pursuit of code quality: Use test categorization for agile builds