![Close [x]](http://dw1.s81c.com/i/c.gif){kind=small}
Test-driven development is the most significant innovation in programming in the last 10 years, although test-first programming and unit testing are hardly new. Some of the best programmers have been using these techniques for half a century; however, only in the last few years have they become widely recognized as critical components in developing robust, bug-free software on time and on budget. But test-driven development is only as good as the tests are. Tests improve code quality, but only for the parts of the code base that are actually being tested. You need a tool that tells you which parts of a program are not being tested so you can write tests for those parts and find more bugs.
Mark Doliner's Cobertura (cobertura is Spanish for coverage) is a free as in speech GPL tool that handles this job. Cobertura monitors tests by instrumenting the bytecode with extra statements to log which lines are and are not being reached as the test suite executes. It then produces a report in HTML or XML that shows exactly which packages, classes, methods, and individual lines of code are not being tested. You can write more tests for those specific areas to reveal any lingering bugs.
Let's begin with the generated Cobertura output. Figure 1 shows a report produced by running Cobertura on the Jaxen test suite (see Resources). You can see that coverage ranges from great (almost 100 percent in the org.jaxen.expr.iter package) to extremely poor (no coverage at all in org.jaxen.dom.html).
Figure 1. Package-level coverage statistics for Jaxen
Cobertura calculates coverage both by the number of lines tested and by the number of branches tested. For a first pass, the difference between these two is not hugely important. Cobertura also calculates the average McCabe's cyclomatic complexity for the class (see Resources).
You can drill down through the HTML report to view the coverage of a particular package or class. Figure 2 shows coverage statistics for the org.jaxen.function package. In this package, coverage ranges from 100 percent for the SumFunction class to a mere 5 percent for the IdFunction class.
Figure 2. Code coverage in the org.jaxen.function package
Drilling down further into individual classes, you can see exactly which lines aren't being tested. Figure 3 shows part of the coverage in the NameFunction class. The left-hand column shows the line number. The next column shows the number of times that line is executed during the test run. As you can see, line 112 is executed 100 times, and line 114 is executed 28 times. Lines
highlighted in red are not tested at all. This report reveals that although the method as a whole is tested, many branches are not.
Figure 3. Code coverage in the NameFunction class
Using Cobertura's reports, you can identify the untested parts of the code and write tests for them. For example, Figure 3 shows that Jaxen needs tests that apply the name() function to text nodes, comment nodes, processing instruction nodes, attribute nodes, and namespace nodes.
Adding all the missing tests is time-consuming when you have as much uncovered code as Cobertura has identified here -- but it's worth doing. You don't have to do it all at once. Begin with the least-tested code, such as any packages that have no coverage. Once you have tested all packages a little, you can write some tests for each class that shows no coverage. Once you partially test all the classes, write tests that cover any uncovered methods. Once all the methods are tested, you can begin looking at what's necessary to activate any untested statements.
Leave (almost) no code untested
Is there anything you can test but shouldn't? It depends on who you ask. In the JUnit FAQ, J. B. Rainsberger writes, "The general philosophy is this: if it can't break on its own, it's too simple to break. First example is the getX() method. Suppose the getX() method only answers the value of an instance variable. In that case, getX() cannot break unless either the compiler or the interpreter is also broken. For that reason, don't test getX(); there is no benefit. The same is true of the setX() method, although if your setX() method does any parameter validation or has any side effects, you likely need to test it."
I don't agree. I've lost count of the number of bugs I've found in code that was "too simple to break." It's true that some getters and setters are so trivial that there's no way they can fail. But I've never been able to figure out how to tell which methods really are too simple to fail and which ones just look that way. It's not hard to write tests that cover simple methods like setters and getters. The minimal time you take to do this will be more than compensated by the number of such methods in which you'll find unexpected bugs.
Generally, it's fairly easy to reach 90 percent test coverage once you start measuring. Increasing coverage to 95 percent or more can require some sneakiness. For example, you might load different versions of supporting libraries to test workarounds for bugs that don't show up in all versions of the libraries. Or you can restructure code so the tests can reach parts of the code they wouldn't normally touch. You might extend classes simply to make their protected methods public so they can be tested. These tricks might seem excessive, but they've helped me find more undiscovered bugs about half the time.
Perfect, 100 percent code coverage is not always attainable. Sometimes you find lines, methods, or even entire classes that simply cannot be reached by tests, no matter how much you contort the code. The following are some examples of challenges you might come across:
-
Code that executes only on a specific platform. For example, in a well-designed GUI application, the code that adds an Exit menu item would run on a Windows PC but not a Mac.
-
catchclauses that catch exceptions that won't happen, such asIOExceptions thrown when reading from aByteArrayInputStream. -
Methods inside nonpublic classes that are never actually invoked but must be implemented in order to satisfy the contract of an interface.
-
Code blocks that handle virtual-machine bugs, such as a failure to
recognize the UTF-8 encoding.
Given these and similar issues, I find the efforts of some extreme programmers to delete all untested code automatically to be unrealistic and perhaps satirical. The fact that you can't always have absolutely perfect test coverage doesn't mean you shouldn't have better coverage.
Nonetheless, more often than not unreachable statements and methods are vestigial code that no longer has any purpose and can be cut from the code base with no effect. It is sometimes possible to test untested code through really skanky hacks that use reflection to access private members. It is also possible to write tests for untested, package-protected code by putting the tests classes in the same package as the classes they're testing. Don't do this. Any code that cannot be reached through the published (public and protected) interfaces should be deleted instead. Unreachable code should not be part of a code base. The smaller a code base, the easier it is to understand and maintain.
Now that you know about the benefits of measuring code coverage, let's talk about the practical details of how you measure code coverage with Cobertura. Cobertura is designed to be run from Ant. No IDE plug-ins are available yet, though some might be developed over the next year or two.
First, you need to add a task definition to the build.xml file. This top-level taskdef element specifies that the cobertura.jar file is in the current working directory:
<taskdef classpath="cobertura.jar" resource="tasks.properties" /> |
Next, you need a cobertura-instrument task that adds the logging code to the already compiled class files. The todir attribute instructs where to put the instrumented classes. The fileset child element specifies which .class files to instrument:
<target name="instrument">
<cobertura-instrument todir="target/instrumented-classes">
<fileset dir="target/classes">
<include name="**/*.class"/>
</fileset>
</cobertura-instrument>
</target> |
You run the tests with the same type of Ant task that normally runs the test suite. The only differences are that the instrumented classes need to appear in the classpath before the original classes, and you need to add the Cobertura JAR file to the classpath:
<target name="cover-test" depends="instrument">
<mkdir dir="${testreportdir}" />
<junit dir="./" failureproperty="test.failure" printSummary="yes"
fork="true" haltonerror="true">
<!-- Normally you can create this task by copying your existing JUnit
target, changing its name, and adding these next two lines.
You may need to change the locations to point to wherever
you've put the cobertura.jar file and the instrumented classes. -->
<classpath location="cobertura.jar"/>
<classpath location="target/instrumented-classes"/>
<classpath>
<fileset dir="${libdir}">
<include name="*.jar" />
</fileset>
<pathelement path="${testclassesdir}" />
<pathelement path="${classesdir}" />
</classpath>
<batchtest todir="${testreportdir}">
<fileset dir="src/java/test">
<include name="**/*Test.java" />
<include name="org/jaxen/javabean/*Test.java" />
</fileset>
</batchtest>
</junit>
</target>> |
The Jaxen project uses JUnit as its test framework, but Cobertura is framework agnostic. It works equally well with TestNG, Artima SuiteRunner, HTTPUnit, or the home-brewed system you cooked up in your basement.
Finally, the cobertura-report task generates the HTML files you saw at the beginning of the article:
<target name="coverage-report" depends="cover-test"> <cobertura-report srcdir="src/java/main" destdir="cobertura"/> </target> |
The srcdir attribute specifies where the original .java source code files reside. The destdir attribute names the directory where Cobertura should place the output HTML.
Once you've added similar tasks to your own Ant build file, you generate a coverage report by typing:
% ant instrument % ant cover-test % ant coverage-report |
Of course, you can change the names of the targets or merge these three tasks into a single target if you prefer.
Cobertura is an important addition to the agile programmer's toolbox. By producing hard numbers for code coverage, Cobertura transforms unit testing from an art to a science. It finds gaps in test coverage, which leads directly to finding bugs. Measuring code coverage gives you the information you need to find and fix bugs, producing more robust software for everyone.
- Download Cobertura from SourceForge.
- Get test infected with JUnit, the de facto standard unit testing framework for the Java platform.
- Read Dave Thomas and Andy Hunt's Pragmatic Unit Testing in Java With JUnit (Pragmatic Bookshelf, 2003).
- Cenqua's Clover is a somewhat more polished, payware test-coverage tool that does essentially the same job as Cobertura with fewer rough edges.
- IBM Rational offers a comprehensive suite of testing tools designed to help you inspect your code. "Get started with automated testing: Road map to success" provides you with the foundation you need to start testing your code now.
- PureCoverage, part of the IBM Rational PurifyPlus suite of testing tools, is an excellent code coverage tool. Rational Application Developor includes PureCoverage functionality; read more about it in "Component testing with IBM Rational Application Developer" (developerWorks, March 2005).
- Cobertura is a fork of jcoverage.
- Carnegie Mellon's Software Engineering Institute has published a detailed explanation of McCabe's Cyclomatic Complexity.
- "Keeping critters out of your code: How to use WebSphere and JUnit to prevent programming bugs" (developerWorks, June 2003) by David Carew and Sandeep Desai looks at taking an XP approach to testing.
- Dennis M. Sosnoski kicks off the Classworking toolkit series
by exploring the open source Hansel and Gretel code-coverage tools.
- Read "Test your tests with Jester" to learn about an open source JUnit test tester.
- Eric Allen and Roy Miller covered unit testing frequently in their respective columns, Diagnosing Java code and Demystifying Extreme Programming.
- Explore FoCuS, a tool from IBM alphaWorks that implements the functional coverage methodology and improves testing of applications by providing detailed coverage information on the areas in which testing is lacking.
- The Jaxen project used as a guinea pig in this article is an open source XPath engine for Java programming that's adaptable to many different object models.
- You'll find articles about every aspect of Java programming in the developerWorks Java technology zone.
- Browse for books on these and other technical topics.
Elliotte Rusty Harold is originally from New Orleans, to which he returns periodically in search of a decent bowl of gumbo. However, he resides in the Prospect Heights neighborhood of Brooklyn with his wife Beth and cats Charm (named after the quark) and Marjorie (named after his mother-in-law). He's an adjunct professor of computer science at Polytechnic University, where he teaches Java technology and object-oriented programming. His Cafe au Lait Web site has become one of the most popular independent Java sites on the Internet, and his spin-off site, Cafe con Leche, has become one of the most popular XML sites. His books include Effective XML, Processing XML with Java, Java Network Programming, and The XML 1.1 Bible. He's currently working on the XOM API for processing XML and the XQuisitor GUI query tool. You can contact him at elharo@metalab.unc.edu.
