 | Level: Introductory Andrew Glover (aglover@stelligent.com), President, Stelligent Incorporated
31 Oct 2006 Everyone agrees that developer testing is important,
but why is it so darn time consuming to run tests? This month, Andrew
Glover reveals the three categories of testing needed to ensure
end-to-end system soundness and then shows you how to automatically sort and
run tests by category. The result is a dramatically reduced
built time, even with today's massive test suites.
If it's not too painful, try to imagine you're a developer at a startup company in early 2002. Flush with cash, you and your cohorts have been tasked to build a large data-driven Web application using the latest and greatest Java™ APIs. You and your management are firm believers in what will eventually become known as agile processes. From day one, tests have been authored with JUnit and run as often as possible, as part of an Ant build process. Eventually a cron job will be set up to run the build nightly. At some later point, someone will download CruiseControl and the growing suite of tests will be run at every check-in.
Now fast-forward to today.
Over the course of the past few years, your company has developed an
enormous code base with an equally enormous suite of JUnit tests.
Everything was fine until about a year ago, when your test suite
eclipsed 2,000 tests and people started noticing that the build process was
taking more than three hours to run. Some months before that, you
stopped running unit tests via Continuous Integration (CI) during code
check-ins because of the strain placed upon the CI server. You switched to
running tests nightly, which made the following morning stressful as
developers tried to figure out what broke and why.
These days, it seems as if the suite of tests is rarely run more
than once during the evening -- and why should they be? They take
forever! No one has a few hours to wait around just to find out that
things are working well (or not). And besides, the entire test suite is
run at night, right?
Because you run the tests so infrequently, they are often fraught
with errors. Consequently, you and your team have begun to question the
value of unit tests: if they are so important to code quality, why are
they such a pain? You all agree there is a fundamental value to unit
tests, if only you could run them in a more agile manner.
Try test categorization
What you need is a strategy to transition your build to a more
agile state. You need a solution that lets you run tests more than once a day and puts your test suite back where it was before the build took three hours to complete.
Before you try to come up with a strategy for getting your entire
test suite back in shape, it might help to reconsider the generic term
"unit tests." Just as the statements "I have an animal at home" and "I
like cars" aren't very specific, so, too, unfortunately, is the phrase "we
write unit tests." These days, a unit test can mean anything.
Take the earlier statements about animals and cars: they lead
to more questions. For example, what kind of animal do you have at home?
Is it a cat, a lizard, or a bear? "I have a bear at home" is certainly
different from "I have a cat at home." Likewise, "I like cars" isn't
very helpful when talking to a car salesman. What types of cars do you
like: sports cars, trucks, or station wagons? Any answer could lead you
down a different path.
Likewise, with developer testing, it can be helpful to categorize
tests by type. Doing so enables more precise language and can enable
your team to run different test types at different frequencies.
Categorization is key to avoiding the dreaded three-hour build that runs
all "unit tests."
Three categories
Visualize your test suite sorted into three layers, each layer representing a different type of developer test defined by how long it takes to run. As you see in Figure 1, each layer adds more length to the total build time, either to run or, as it turns out, to author.
Figure 1. Three layers of test categorization
The bottom layer is made up of the shortest running tests, which, as you can imagine, are the easiest to write. They also touch the least amount of code. The top layer is made up of more high-level tests that exercise a greater portion of the application. These tests are a bit more difficult to code and take considerably more time to execute. The middle layer is for tests that fall between the two extremes.
The three categories are as follows:
- Unit tests
- Component tests
- System tests
Let's consider each one separately.
1. Unit tests
A unit test verifies an object or multiple objects in
isolation. Unit tests don't deal with databases, file systems, or
anything else that would tend to make tests run longer; consequently,
unit tests can be written from day one. In fact, this is exactly what JUnit was designed for. The isolated notion of unit testing is behind the myriad mock-object libraries that facilitate
isolating a particular object from its external dependencies. What's more, unit tests can be written before the actual code under test is written -- hence the notion of test-first development.
Unit tests are usually easy to code because they are not reliant
on architectural dependencies, and they also run quickly. On the downside,
individual unit tests offer somewhat limited code coverage. The value
of unit tests (which is considerable) is that they enable developers to guarantee object reliability at the lowest possible level.
Because unit tests run so quickly and are so easy to write, a code
base should have a good number of them and they should be run as
often as possible. You should always run them when you execute a
build, whether on your machine or in the context of a
CI environment (meaning you should run them anytime
code is checked into an SCM system).
2. Component tests
Component tests verify multiple objects interacting together, but
they breach the notion of isolation. Because component tests deal with
multiple layers of an architecture, they often deal with
databases, file systems, network elements, etc. Also, component tests are
somewhat difficult to write early, so incorporating them into a truly test-first/test-driven scenario is challenging at best.
Component tests take longer to write because they're more
involved than unit tests. On the other hand, they achieve more code coverage
than unit tests do because of their wide reach. They also take longer to run,
so running a lot of them together can increase your overall test time dramatically.
A host of frameworks facilitate the challenges associated with
testing large architectural components. DbUnit is a
perfect example of this type of framework. DbUnit makes writing tests
that rely on a database easier by handling the associated
complexity of seeding a database between test states.
When the testing aspect of a build is prolonged, you can generally
predict that a large suite of component tests is involved. Because these
tests take longer to run than true unit tests, you may find that you
can't run them all the time. Accordingly, it makes sense in a CI
environment to run the tests at least hourly. You should always run these tests on a local developer's box before checking in any code, too.
|
What about acceptance tests?
Acceptance tests are similar to functional tests, the difference being that, ideally, customers or end users write them. Just like functional tests, acceptance tests test as an end user would. One acceptance framework that has garnered a lot of attention is Selenium (see Resources), which uses a browser to test Web applications. Selenium can be automated via a build process, just like JUnit tests. But Selenium is a new platform: it doesn't use JUnit, nor is it similar in manner. |
|
3. System tests
System tests verify a software application end-to-end.
Consequently, they introduce a high degree of architectural complexity:
the entire application must be running for a system test to take
place. If it's a Web application, you need access to the database,
along with a Web server, container, and any associated configuration
aspects to run a system test. It follows that most system tests are
written in the latter cycles of a software life cycle.
System tests are challenging to author and also take a good deal of
time to actually execute. On the other hand, they offer quite a bit of
bang for the buck, so to speak, in terms of architectural code coverage.
System tests are very similar to functional tests. The difference is
that they don't emulate a user, they mimic one. Just as in
component tests, a number of frameworks have been created to help
facilitate these tests. For example, jWebUnit facilitates testing Web
applications by mimicking a browser.
|
Should I use jWebUnit or Selenium?
jWebUnit is a JUnit extension framework designed for system testing; consequently, it requires you to write the tests. Selenium is excellent for acceptance testing and functional testing, and unlike jWebUnit, it enables nonprogrammers to write tests. Ideally, your team could use both tools to verify application functionality. |
|
Implementing test categorization
So, it turns out that your suite of unit tests is really a suite of
unit tests, component tests, and system tests. Moreover, you now know after inspecting the tests that the reason the build takes three hours is that the majority of them are component tests. Your next question is, how does
one implement test categorization with JUnit?
You have a few options, but let's stick with the two easiest ones:
- Create custom JUnit suite files corresponding to desired categories.
- Create custom directories for each type of test.
|
Test categorization with TestNG
Implementing test categorization with TestNG is quite easy. With TestNG's group annotation, logically dividing tests by category is as easy as applying the proper group annotation to a desired test. Running a particular category then is a matter of passing in the corresponding group name to a test runner, such as Ant. |
|
Creating custom suites
You can use JUnit's TestSuite class (which is
of type Test) to define a collection of tests as belonging together. You start by creating an
instance of TestSuite and adding corresponding test classes or test methods to it. You can then point JUnit at the TestSuite instance by defining a public
static method named suite(). All
tests included will then be executed as a single run. Consequently, you can
implement test categorization by creating a unit TestSuite, a component TestSuite, and a system TestSuite.
For example, the class shown in Listing 1 creates a TestSuite holding all component tests in the suite() method. Note that this class isn't very JUnit-specific. It doesn't extend TestCase, nor
does it have any test cases defined. But JUnit will reflectively
find the suite() method and run all tests
returned by it.
Listing 1. A TestSuite for component tests
package test.org.acme.widget;
import junit.framework.Test;
import junit.framework.TestSuite;
import test.org.acme.widget.*;
public class ComponentTestSuite {
public static void main(String[] args) {
junit.textui.TestRunner.run(ComponentTestSuite.suite());
}
public static Test suite(){
TestSuite suite = new TestSuite();
suite.addTestSuite(DefaultSpringWidgetDAOImplTest.class);
suite.addTestSuite(WidgetDAOImplLoadTest.class);
...
suite.addTestSuite(WidgetReportTest.class);
return suite;
}
}
|
The process of defining TestSuites
does require that you go through your existing tests and add them
to the corresponding class (that is, all unit tests are added in a UnitTestSuite). It also means that as you author
new tests in a given category, you'll have to programmatically add them to the appropriate TestSuite, and of course, recompile them.
Running individual TestSuites then
becomes an exercise in creating unique Ant tasks that invoke the
correct test collection. You can define a component-test task that picks up the ComponentTestSuite and so on, as I've done in Listing 2:
Listing 2. An Ant task to run only component tests
<target name="component-test"
if="Junit.present"
depends="junit-present,compile-tests">
<mkdir dir="${testreportdir}"/>
<junit dir="./" failureproperty="test.failure"
printSummary="yes"
fork="true" haltonerror="true">
<sysproperty key="basedir" value="."/>
<formatter type="xml"/>
<formatter usefile="false" type="plain"/>
<classpath>
<path refid="build.classpath"/>
<pathelement path="${testclassesdir}"/>
<pathelement path="${classesdir}"/>
</classpath>
<batchtest todir="${testreportdir}">
<fileset dir="test">
<include name="**/ComponentTestSuite.java"/>
</fileset>
</batchtest>
</junit>
</target>
|
Ideally, you would also have tasks for invoking unit tests and system
tests. Lastly, for the occasions where you would like to run the entire test
suite, you would create a fourth task that depended on all three test
categories, as I've done in Listing 3:
Listing 3. An Ant task for all tests
<target name="test-all" depends="unit-test,component-test,system-test"/>
|
Creating custom TestSuites is a quick
solution for implementing test categorization. The downside of this
method is that as you create new tests, you must add them programmatically to the appropriate TestSuite,
which can be a pain. Creating a custom directory for each type of
test is more extensible and allows you to add new categorized tests
without having to recompile.
Creating custom directories
I find that the easiest way to implement test categorization with
JUnit is to logically divide tests into specific directories
corresponding to their test type. Using this technique, all unit tests
will reside in a unit directory, all
component tests in a component directory, and
so on.
For example, in a test directory where all
uncategorized tests are kept, you can create three new subdirectories
like those shown in Listing 4:
Listing 4. Directory structure implementing test categories
acme-proj/
test/
unit/
component/
system/
conf/
|
To run these tests, you must define at least four Ant tasks: one for unit tests, another for component tests, and so on. The
fourth task is a convenience task that runs all three test types (like
the one shown in Listing 3).
The JUnit task is much like the one defined in Listing 2. What's different, however, is a detail in the
batchtest aspect of the task.
This time, the fileset points at a specific
directory. In the case of Listing 5, it's pointing at the unit directory:
Listing 5. The batchtest aspect of the JUnit task to run all unit tests
<batchtest todir="${testreportdir}">
<fileset dir="test/unit">
<include name="**/**Test.java"/>
</fileset>
</batchtest>
|
Note that this test only runs any tests in the test/unit directory. When new unit tests (or any other tests for that matter) are created, you simply need to drop them into the directory and things are good to go! This is a bit easier than having to add a new line to the TestSuite file and recompile it.
Problem solved!
Getting back to our original scenario, let's say that you and your
team decide that using specific directories is the most extensible
solution to your build-time problem. The hardest aspects of the task are
the inspection and distribution of test types. You refactor your Ant
build file and create four new tasks (three for individual test types
and one to run them all). Moreover, you modify CruiseControl to only run
your true unit tests on check-ins and component tests on an hourly
basis. Upon further inspection, it turns out that system tests can also
be run hourly, so you create an additional task that runs component
tests and system tests together.
The end result is that tests are run many times a day and your team
is able to catch integration errors more quickly -- normally within a
few hours.
Creating build agility isn't exactly hip, but it certainly plays a
vital role in ensuring code quality. With your tests running more often,
concerns about the value of developer testing are a distant memory.
And, best of all, it's now 2006 and your company is wildly successful!
Resources Learn
- "Automate acceptance tests with Selenium" (Christian Hellsten, developerWorks, December 2005): Architects, developers, and testers learn how to use the Selenium testing tools to automate acceptance tests.
- "Effective Unit Testing with DbUnit" (Andrew Glover, OnJava, January 2004): Introduces database-dependent testing with DbUnit.
- "An early look at JUnit 4" (Elliotte Harold, developerWorks, September 2005): Obsessive code tester Elliotte Harold takes JUnit 4 out for a spin.
- "Repeatable system tests" (Andrew Glover, developerWorks, September 2006): Andrew Glover introduces Cargo, an open source framework that automates container management in a generic fashion.
- "Create test cases for Web applications" (Amit Tuli, developerWorks, May 2005): Software engineer Amit Tuli introduces jWebUnit.
- "In pursuit of code quality: JUnit 4 vs. TestNG" (Andrew Glover, developerWorks, April 2006): Has JUnit 4 rendered TestNG obsolete? Find out why not.
- In pursuit of code quality series (Andrew Glover, developerWorks): Learn more about code metrics, test frameworks,
and writing quality-focused code.
- developerWorks: Hundreds of articles about every aspect of Java programming.
Get products and technologies
Discuss
About the author | 
| | Andrew Glover is president of Stelligent Incorporated, which helps companies address software quality with effective developer testing strategies and continuous integration techniques that enable teams to monitor code quality early and often. Check out Andy's blog for a list of his publications. |
Rate this page
| |
