Measure test coverage with Cobertura

Find untested code where bugs lurk

Elliotte Rusty Harold (elharo@metalab.unc.edu), Adjunct Professor, Polytechnic University

Elliotte Rusty Harold is originally from New Orleans, to which he returns periodically in search of a decent bowl of gumbo. However, he resides in the Prospect Heights neighborhood of Brooklyn with his wife Beth and cats Charm (named after the quark) and Marjorie (named after his mother-in-law). He's an adjunct professor of computer science at Polytechnic University, where he teaches Java technology and object-oriented programming. His Cafe au Lait Web site has become one of the most popular independent Java sites on the Internet, and his spin-off site, Cafe con Leche, has become one of the most popular XML sites. His books include Effective XML, Processing XML with Java, Java Network Programming, and The XML 1.1 Bible. He's currently working on the XOM API for processing XML and the XQuisitor GUI query tool. You can contact him at elharo@metalab.unc.edu.

Summary: Cobertura is an open source tool that measures test coverage by instrumenting a code base and watching which lines of code are and are not executed as the test suite runs. In addition to identifying untested code and locating bugs, Cobertura can optimize code by flagging dead, unreachable code and can provide insights into how an API operates in practice. Elliotte Rusty Harold shares how you can take advantage of Cobertura using code-coverage best practices.

Date: 03 May 2005
Level: Intermediate
Activity: 5232 views
Comments:

Test-driven development is the most significant innovation in programming in the last 10 years, although test-first programming and unit testing are hardly new. Some of the best programmers have been using these techniques for half a century; however, only in the last few years have they become widely recognized as critical components in developing robust, bug-free software on time and on budget. But test-driven development is only as good as the tests are. Tests improve code quality, but only for the parts of the code base that are actually being tested. You need a tool that tells you which parts of a program are not being tested so you can write tests for those parts and find more bugs.

Mark Doliner's Cobertura (cobertura is Spanish for coverage) is a free as in speech GPL tool that handles this job. Cobertura monitors tests by instrumenting the bytecode with extra statements to log which lines are and are not being reached as the test suite executes. It then produces a report in HTML or XML that shows exactly which packages, classes, methods, and individual lines of code are not being tested. You can write more tests for those specific areas to reveal any lingering bugs.

Reading Cobertura output

Let's begin with the generated Cobertura output. Figure 1 shows a report produced by running Cobertura on the Jaxen test suite (see Resources). You can see that coverage ranges from great (almost 100 percent in the org.jaxen.expr.iter package) to extremely poor (no coverage at all in org.jaxen.dom.html).

Figure 1. Package-level coverage statistics for Jaxen

Cobertura calculates coverage both by the number of lines tested and by the number of branches tested. For a first pass, the difference between these two is not hugely important. Cobertura also calculates the average McCabe's cyclomatic complexity for the class (see Resources).

You can drill down through the HTML report to view the coverage of a particular package or class. Figure 2 shows coverage statistics for the org.jaxen.function package. In this package, coverage ranges from 100 percent for the SumFunction class to a mere 5 percent for the IdFunction class.

Figure 2. Code coverage in the org.jaxen.function package

Drilling down further into individual classes, you can see exactly which lines aren't being tested. Figure 3 shows part of the coverage in the NameFunction class. The left-hand column shows the line number. The next column shows the number of times that line is executed during the test run. As you can see, line 112 is executed 100 times, and line 114 is executed 28 times. Lines highlighted in red are not tested at all. This report reveals that although the method as a whole is tested, many branches are not.

Figure 3. Code coverage in the NameFunction class

Cobertura is a fork of jcoverage (see Resources). The GPL edition of jcoverage has not been updated in over a year and has some longstanding bugs that Cobertura repairs. Instead of continuing with open source development, the original jcoverage developers have been focusing on jcoverage commercial edition and jcoverage+, closed source products derived from the same code base. Such are the wonders of open source that a product doesn't need to die just because the original developers decide they want to be paid for their work.

Identify missing tests

Using Cobertura's reports, you can identify the untested parts of the code and write tests for them. For example, Figure 3 shows that Jaxen needs tests that apply the name() function to text nodes, comment nodes, processing instruction nodes, attribute nodes, and namespace nodes.

Adding all the missing tests is time-consuming when you have as much uncovered code as Cobertura has identified here -- but it's worth doing. You don't have to do it all at once. Begin with the least-tested code, such as any packages that have no coverage. Once you have tested all packages a little, you can write some tests for each class that shows no coverage. Once you partially test all the classes, write tests that cover any uncovered methods. Once all the methods are tested, you can begin looking at what's necessary to activate any untested statements.

Leave (almost) no code untested

Is there anything you can test but shouldn't? It depends on who you ask. In the JUnit FAQ, J. B. Rainsberger writes, "The general philosophy is this: if it can't break on its own, it's too simple to break. First example is the getX() method. Suppose the getX() method only answers the value of an instance variable. In that case, getX() cannot break unless either the compiler or the interpreter is also broken. For that reason, don't test getX(); there is no benefit. The same is true of the setX() method, although if your setX() method does any parameter validation or has any side effects, you likely need to test it."

I don't agree. I've lost count of the number of bugs I've found in code that was "too simple to break." It's true that some getters and setters are so trivial that there's no way they can fail. But I've never been able to figure out how to tell which methods really are too simple to fail and which ones just look that way. It's not hard to write tests that cover simple methods like setters and getters. The minimal time you take to do this will be more than compensated by the number of such methods in which you'll find unexpected bugs.

Generally, it's fairly easy to reach 90 percent test coverage once you start measuring. Increasing coverage to 95 percent or more can require some sneakiness. For example, you might load different versions of supporting libraries to test workarounds for bugs that don't show up in all versions of the libraries. Or you can restructure code so the tests can reach parts of the code they wouldn't normally touch. You might extend classes simply to make their protected methods public so they can be tested. These tricks might seem excessive, but they've helped me find more undiscovered bugs about half the time.

Perfect, 100 percent code coverage is not always attainable. Sometimes you find lines, methods, or even entire classes that simply cannot be reached by tests, no matter how much you contort the code. The following are some examples of challenges you might come across:

Code that executes only on a specific platform. For example, in a well-designed GUI application, the code that adds an Exit menu item would run on a Windows PC but not a Mac.
catch clauses that catch exceptions that won't happen, such as IOExceptions thrown when reading from a ByteArrayInputStream.
Methods inside nonpublic classes that are never actually invoked but must be implemented in order to satisfy the contract of an interface.
Code blocks that handle virtual-machine bugs, such as a failure to recognize the UTF-8 encoding.

Don't neglect to measure the unit test packages and classes themselves. More than once I've noticed a test method or class that isn't actually run by the test suite. Normally this signifies a bug in the naming conventions (e.g., naming a method tesSomeReallyComplexCondition instead of testSomeReallyComplexCondition) or a class that I've neglected to add to the primary suite() method. Other times, unexpected conditions meant code inside a test method was being skipped. Either way, I wrote the tests, but they weren't actually running. JUnit won't tell you that it isn't running all the tests you think it is, but Cobertura will. Once spotted, this is normally easy to fix.

Given these and similar issues, I find the efforts of some extreme programmers to delete all untested code automatically to be unrealistic and perhaps satirical. The fact that you can't always have absolutely perfect test coverage doesn't mean you shouldn't have better coverage.

Nonetheless, more often than not unreachable statements and methods are vestigial code that no longer has any purpose and can be cut from the code base with no effect. It is sometimes possible to test untested code through really skanky hacks that use reflection to access private members. It is also possible to write tests for untested, package-protected code by putting the tests classes in the same package as the classes they're testing. Don't do this. Any code that cannot be reached through the published (public and protected) interfaces should be deleted instead. Unreachable code should not be part of a code base. The smaller a code base, the easier it is to understand and maintain.

Running Cobertura

Now that you know about the benefits of measuring code coverage, let's talk about the practical details of how you measure code coverage with Cobertura. Cobertura is designed to be run from Ant. No IDE plug-ins are available yet, though some might be developed over the next year or two.

First, you need to add a task definition to the build.xml file. This top-level taskdef element specifies that the cobertura.jar file is in the current working directory:

<taskdef classpath="cobertura.jar" resource="tasks.properties" />

Next, you need a cobertura-instrument task that adds the logging code to the already compiled class files. The todir attribute instructs where to put the instrumented classes. The fileset child element specifies which .class files to instrument:

<target name="instrument">
  <cobertura-instrument todir="target/instrumented-classes">
    <fileset dir="target/classes">
      <include name="**/*.class"/>
    </fileset>
  </cobertura-instrument>
</target>

You run the tests with the same type of Ant task that normally runs the test suite. The only differences are that the instrumented classes need to appear in the classpath before the original classes, and you need to add the Cobertura JAR file to the classpath:

<target name="cover-test" depends="instrument">
  <mkdir dir="${testreportdir}" />
  <junit dir="./" failureproperty="test.failure" printSummary="yes" 
         fork="true" haltonerror="true">
    <!-- Normally you can create this task by copying your existing JUnit
         target, changing its name, and adding these next two lines.
         You may need to change the locations to point to wherever 
         you've put the cobertura.jar file and the instrumented classes. -->
    <classpath location="cobertura.jar"/>
    <classpath location="target/instrumented-classes"/>
    <classpath>
      <fileset dir="${libdir}">
        <include name="*.jar" />
      </fileset>
      <pathelement path="${testclassesdir}" />
      <pathelement path="${classesdir}" />
    </classpath>
    <batchtest todir="${testreportdir}">
      <fileset dir="src/java/test">
        <include name="**/*Test.java" />
        <include name="org/jaxen/javabean/*Test.java" />
      </fileset>
    </batchtest>
  </junit>
</target>>

The Jaxen project uses JUnit as its test framework, but Cobertura is framework agnostic. It works equally well with TestNG, Artima SuiteRunner, HTTPUnit, or the home-brewed system you cooked up in your basement.

Finally, the cobertura-report task generates the HTML files you saw at the beginning of the article:

<target name="coverage-report" depends="cover-test">
 <cobertura-report srcdir="src/java/main" destdir="cobertura"/>
</target>

The srcdir attribute specifies where the original .java source code files reside. The destdir attribute names the directory where Cobertura should place the output HTML.

Once you've added similar tasks to your own Ant build file, you generate a coverage report by typing:

% ant instrument
% ant cover-test
% ant coverage-report

Of course, you can change the names of the targets or merge these three tasks into a single target if you prefer.

Summary

Cobertura is an important addition to the agile programmer's toolbox. By producing hard numbers for code coverage, Cobertura transforms unit testing from an art to a science. It finds gaps in test coverage, which leads directly to finding bugs. Measuring code coverage gives you the information you need to find and fix bugs, producing more robust software for everyone.

Resources

Download Cobertura from SourceForge.
Get test infected with JUnit, the de facto standard unit testing framework for the Java platform.
Read Dave Thomas and Andy Hunt's Pragmatic Unit Testing in Java With JUnit (Pragmatic Bookshelf, 2003).
Cenqua's Clover is a somewhat more polished, payware test-coverage tool that does essentially the same job as Cobertura with fewer rough edges.
IBM Rational offers a comprehensive suite of testing tools designed to help you inspect your code. "Get started with automated testing: Road map to success" provides you with the foundation you need to start testing your code now.
PureCoverage, part of the IBM Rational PurifyPlus suite of testing tools, is an excellent code coverage tool. Rational Application Developor includes PureCoverage functionality; read more about it in "Component testing with IBM Rational Application Developer" (developerWorks, March 2005).
Cobertura is a fork of jcoverage.
Carnegie Mellon's Software Engineering Institute has published a detailed explanation of McCabe's Cyclomatic Complexity.
"Keeping critters out of your code: How to use WebSphere and JUnit to prevent programming bugs" (developerWorks, June 2003) by David Carew and Sandeep Desai looks at taking an XP approach to testing.
Dennis M. Sosnoski kicks off the Classworking toolkit series by exploring the open source Hansel and Gretel code-coverage tools.
Read "Test your tests with Jester" to learn about an open source JUnit test tester.
Eric Allen and Roy Miller covered unit testing frequently in their respective columns, Diagnosing Java code and Demystifying Extreme Programming.
Explore FoCuS, a tool from IBM alphaWorks that implements the functional coverage methodology and improves testing of applications by providing detailed coverage information on the areas in which testing is lacking.
The Jaxen project used as a guinea pig in this article is an open source XPath engine for Java programming that's adaptable to many different object models.
You'll find articles about every aspect of Java programming in the developerWorks Java technology zone.
Browse for books on these and other technical topics.