 | Level: Introductory Dennis Sosnoski (dms@sosnoski.com), Consultant, Sosnoski Software Solutions, Inc.
08 Feb 2005 Unit tests provide a great technique for making sure that code performs to specifications. But the quality of unit tests is up to the test writer, and the results from unit tests are only as good as the quality of the tests. How can you make sure your unit tests deliver the quality you need? In the first article of this new series dedicated to classworking tools, regular developerWorks contributor Dennis Sosnoski discusses how code coverage tools provide one important quality check for your tests.
This month kicks off my new series on Java classworking tools. For the first installment, I'm covering a pair of related tools named
Hansel and Gretel. These both address the issue of code coverage,
determining which code is actually run during an execution of your application. Even
though they're designed for very different situations, both Hansel and Gretel
have some unique and interesting features that set them apart from other tools of
this type.
As you might expect based on the names, Hansel and Gretel are
related projects. Gretel actually came first, then Hansel was developed using
portions of the Gretel code as a base. I'm going to reverse this order in the
article, using Hansel first for an easy introduction to the principles of code
coverage, then taking a quick look at Gretel to show how coverage testing is
actually implemented down at the bytecode level.
I'll be looking at code coverage mainly from the standpoint of unit
testing. Unit testing isn't the only scenario where code coverage tools are
useful, but it's certainly a major use case. If you're not executing some code
during your tests, that code is not being tested and is a potential source of
undetected problems -- exactly what unit testing is intended to prevent. Code
coverage tools let you expand the unit testing mantra of "clean and green" to
"clean, green, and covered," with real benefits to the effectiveness of your
unit tests.
Using Hansel
Hansel has one feature that makes it a great starting point for a
discussion of coverage tools -- it integrates with the JUnit framework for unit
tests, and lets you easily check the coverage of your unit test suites. JUnit is
what most people assume when you discuss unit testing in Java development, which gives
Hansel a big advantage over standalone coverage tools for use in unit testing.
Testing as normal
Listing 1 gives the source code for my StringArray
class, a wrapper for ordered arrays of Strings. Besides the constructor, which takes an
array and makes sure that it's ordered with no duplicates, the class also
provides an indexOf() method that uses a binary search to
find the index number of a string in the ordered array, along with a couple of
simple access methods.
Listing 1. StringArray class
public class StringArray
{
/** Ordered array of strings. */
private final String[] m_list;
/**
* Constructor from array of values. This checks the array
* values to make sure they're ordered and unique. If
* they're not, it sorts them and eliminates duplicates. Once
* the array has been passed to this constructor, it must
* not be modified by outside code.
*
* @param list array of values
*/
public StringArray(String[] list) {
// first make sure the array values are ordered
int dupl = 0;
if (list.length > 0) {
String last = list[0];
int index = 0;
while (++index < list.length) {
String comp = list[index];
int diff = last.compareTo(comp);
if (diff > 0) {
// disordered, sort values in array
Arrays.sort(list);
last = list[index];
// there's an error here! see "When
// coverage is not enough"
} else if (diff < 0) {
last = comp;
} else {
dupl++;
}
}
}
// eliminate duplicates if present
if (dupl > 0) {
String[] uniques = new String[list.length - dupl];
String last = uniques[0] = list[0];
int index = 0;
int fill = 1;
while (++index < list.length) {
if (!last.equals(list[index])) {
last = list[index];
uniques[fill++] = last;
}
}
m_list = uniques;
} else {
m_list = list;
}
}
/**
* Get string at a particular index in the list.
*
* @param index list index to be returned
* @return string at that index position
*/
public String get(int index) {
return m_list[index];
}
/**
* Find index of a particular string in the array. This does
* a binary search through the array values, using a pair of
* index bounds to track the subarray of possible matches at
* each iteration.
*
* @param value string to be found in list
* @return index of string in array or -1 if not present
*/
public int indexOf(String value) {
int base = 0;
int limit = m_list.length - 1;
while (base <= limit) {
int cur = (base + limit) >> 1;
int diff = value.compareTo(m_list[cur]);
if (diff < 0) {
limit = cur - 1;
} else if (diff > 0) {
base = cur + 1;
} else {
return cur;
}
}
return -1;
}
/**
* Get number of values in array
*
* @return number of values in array
*/
public int size() {
return m_list.length;
}
}
|
Listing 2 gives a simple JUnit test case for this class, and Figure
1 shows the results when I run this test in Eclipse:
Listing 2. JUnit test for StringArray class
public class StringArrayTest extends TestCase
{
private static String[] s_list1 = {
"a", "b", "ccc", "ccd", "d", "e", "f", "g"
};
private static String[] s_list2 = { // s_list1 reordered
"a", "b", "ccd", "ccc", "g", "f", "e", "d"
};
public void testStringArray() {
StringArray array1 = new StringArray(s_list1);
assertEquals(8, array1.size());
StringArray array2 = new StringArray(s_list2);
assertEquals(8, array2.size());
}
public void testIndexOf() {
StringArray array = new StringArray(s_list1);
assertEquals(-1, array.indexOf("ee"));
assertEquals(4, array.indexOf("d"));
}
}
|
Figure 1. Running original test case
Testing with coverage
So the code is "clean and green," according to my simple unit test
case -- but that's meaningful only if I'm really testing the code thoroughly.
To see just how effective my tests are at exercising the code, I can check
code coverage with Hansel.
Adding Hansel checking to your JUnit tests is simple. You just need
to add the Hansel JAR file to your test classpath and then create
a test suite with a Hansel decorator (a class that wraps another class,
modifying the behavior of the wrapped class) that references the class
or classes being tested. For my StringArrayTest
example, I can do this by adding the following simple method to the class:
public static Test suite() {
return new CoverageDecorator(StringArrayTest.class,
new Class[] { StringArray.class });
}
|
After this addition, my JUnit test results are very different, as
shown in Figure 2 -- I now have four JUnit test failures!
Figure 2. Running test with Hansel coverage
checking
Interpreting Hansel failures
In the Figure 2 test run, Hansel created a test failure for every
coverage lapse it detected. This approach is a very nice way of showing the results if
you're using an IDE that provides JUnit integration (like Eclipse, which was used for
these tests), because it allows you to click on the failure trace line and
immediately view the source code involved. In Figure 2, I've done this for the
first reported failure, a branch not completely covered. Listing 3 shows the detailed
failure message, along with a snippet of the code it's pointing at:
Listing 3. First failure
Coverage failure: Branch not completely covered. Condition
'list.length <= 0' is not fulfilled.
at com.sosnoski.demo.StringArray.<init>(StringArray.java:29)
// first make sure the array values are ordered
int dupl = 0;
if (list.length > 0) {
String last = list[0];
int index = 0;
|
This failure says that I haven't tested the code using a zero-length array.
That's easy enough to fix, so I can just add a quick test and move on to the
next failure, shown in Listing 4:
Listing 4. Second failure
Probe in "StringArray.java" line 41
Coverage failure: Branch not completely covered. Condition
'fill >= 0' is not fulfilled.
at com.sosnoski.demo.StringArray.<init>(StringArray.java:41)
if (diff > 0) {
// disordered, sort values in array
Arrays.sort(list);
last = list[index];
// there's an error here! see "When
// coverage is not enough"
} else if (diff < 0) {
last = comp;
} else {
dupl++;
}
|
This time the message is a little more difficult to relate to the
code. First off, the variable name doesn't match the source code. This result appears to
be an error in the way Hansel interprets local variable information from the
class file. The code here uses a local variable diff, which
is only defined within this block of code, while later in the constructor a
separate variable, fill, is declared at the
same level. It looks like Hansel has confused the two variable names because
they're using the same stack frame slot. That's annoying, but not too confusing once
you know what's happening, because you'll still see the correct line of
code.
The second difficulty with this failure is that the condition
doesn't seem to match the actual test case -- the greater-than-zero test actually
precedes the else statement this failure specifies. This
result is also easy to understand once you know how Hansel works. The failure message
is generated without respect to any of the preceding code in the method,
looking only at the problem in isolation. With this idea in mind, the failure can be
restated as "Condition 'diff < 0' was always true," which makes a lot more
sense; my test cases don't include any arrays with duplicate values, which is
what would be required to get to the third and final block in this set of three
alternatives. Once I understand this failure, it's also easy enough to
fix, so I'll again add another line to my tests and move on to the next
failure, as shown in Listing 5:
Listing 5. Third failure
Probe in "StringArray.java" line 50
Coverage failure: Branch not completely covered. Condition
'!(dupl <= 0)' is not fulfilled.
at com.sosnoski.demo.StringArray.<init>(StringArray.java:50)
// eliminate duplicates if present
if (dupl > 0) {
String[] uniques = new String[list.length - dupl];
|
This failure message is a little on the cryptic side, but with a
little Boolean Logic 101, I can translate the message to "Condition '(dupl <= 0)' was always true." In terms of the test cases, this failure means that
none of the tests had duplicates in the arrays I supplied to the constructor -- the
same cause for the second failure. That leaves the fourth and final
failure, shown in Listing 6:
Listing 6. Fourth failure
Probe in "StringArray.java" line 75
Coverage failure: Method not covered.
at com.sosnoski.demo.StringArray.get(StringArray.java:75)
public String get(int index) {
return m_list[index];
}
|
This failure is easy to understand, but I'm not sure I really want
to "fix" it -- the method I'm not testing is trivial, and adding tests for
trivial methods can be a waste of effort. Unfortunately, Hansel doesn't provide any
options for suppressing failure messages, so if I want my JUnit tests to run with
Hansel coverage testing and execute successfully, I need to add a test that
uses this method. I'll use this requirement to improve the overall quality of my
tests with one that verifies the lookup code, ending up with the test case shown in
Listing 7 (changes are shown in bold):
Listing 7. Modified JUnit test for full coverage
public class StringArrayTest extends TestCase
{
private static String[] s_list1 = {
"a", "b", "ccc", "ccd", "d", "e", "f", "g"
};
private static String[] s_list2 = { // s_list1 reordered
"a", "b", "ccd", "ccc", "g", "f", "e", "d"
};
private static String[] s_list3 = { // duplicates
"a", "g", "ccc", "d", "d", "e", "ccc", "g"
};
public void testStringArray() {
// added line below for first coverage failure
StringArray array0 = new StringArray(new String[0]);
StringArray array1 = new StringArray(s_list1);
assertEquals(8, array1.size());
StringArray array2 = new StringArray(s_list2);
assertEquals(8, array2.size());
// added line below for second/third coverage failure
StringArray array3 = new StringArray(s_list3);
}
// changed this to test both get and indexOf, and sorting
public void testLookup() {
StringArray array = new StringArray(s_list2);
assertEquals(-1, array.indexOf("ee"));
for (int i = 0; i < s_list1.length; i++) {
String value = array.get(i);
assertEquals(s_list1[i], value);
assertEquals(i, array.indexOf(value));
}
}
public static Test suite() {
return new CoverageDecorator(StringArrayTest.class,
new Class[] { StringArray.class });
}
}
|
This modified test case passes on all fronts, including the Hansel
code coverage checks.
The limits of coverage
Unfortunately, code coverage testing only makes sure you're
executing all the code you'd like to test. It doesn't guarantee the quality of your
tests. In the case of the code I've been using for this article, there's a major
error in the interaction between the two loops in the constructor. The first loop
checks both the order and uniqueness of values in the array passed to the
constructor, while the second loop eliminates duplicate values counted by the first loop.
Listing 8 shows that first loop, with the problem code shown in bold:
Listing 8. Problem code from constructor
// first make sure the array values are ordered
int dupl = 0;
if (list.length > 0) {
String last = list[0];
int index = 0;
while (++index < list.length) {
String comp = list[index];
int diff = last.compareTo(comp);
if (diff > 0) {
// disordered, sort values in array
Arrays.sort(list);,
last = list[index];
} else if (diff < 0) {
last = comp;
} else {
dupl++;
}
}
}
|
The problem with this code is that when the array of values is
sorted, the count of duplicates will no longer be accurate. To
demonstrate this error, I can change the second array in the unit tests to cause a duplicate
value to be counted twice (once before the list is sorted, once after):
private static String[] s_list2 = { // s_list1 reordered
"e", "e", "ccd", "ccc", "g", "f", "d", "a", "b"
};
|
This new test case gives me an ArrayIndexOutOfBoundsException in the second loop of the
constructor. The problem is easy to fix, by just restarting the
duplicate scan when the array is sorted, as shown in Listing 9:
Listing 9. Fixed constructor code
// disordered, sort values and restart scan
Arrays.sort(list);
dupl = 0;
index = 0;
last = list[index];
|
But Hansel assured me that all my code was being executed by the
original JUnit tests, so why didn't this problem show up in those tests? The
reason is that this problem is data-dependent; it'll lurk in the code until it
spots just the pattern of data it wants, and then leap out and bite me (I tend to
take bugs personally!). Data-dependent errors are a lot harder to detect than
simple logic errors, and, unfortunately, code coverage tools can't make sure that
you've tested all the combinations of data that might trigger such an error. The best
you can do to flush these errors out into the open is use a wide variety of
data in your unit tests.
Uncovering classworking
So how does code coverage relate to classworking? Test coverage
tools modify your code to leave behind a trail of "bread crumbs" during
execution (hence the names of the tools I'm targeting in this column). After the
tests have been executed, the tool can check this trail to see which portions
of the code were actually executed during the test. To demonstrate how this feature
works, I'll introduce you to the other half of our fairy tale duo, Gretel.
While Hansel is designed for class-at-a-time coverage checking
integrated into JUnit tests, Gretel is designed for application-wide coverage
checking. You start by using Gretel to instrument your code, next running one or
more tests with the Gretel instrumentation saving execution data to files, then
using Gretel again to view the saved data. In this way, Gretel is similar to several other open source or commercial
coverage checking tools.
Where Gretel goes beyond most other code coverage tools is in its support of
incremental coverage checking. Adding instrumentation to your code slows execution
slightly, and if you run a large set of tests for your application, the
performance loss due to instrumentation may be a serious concern (especially if you have
tests that are timing-sensitive). To alleviate this problem, Gretel lets you
reinstrument your code after running an initial set of tests. This
reinstrumentation step removes the instrumentation from code that's
already been covered, only leaving it in place for code that wasn't executed by
the initial tests. You can repeat this reinstrumentation step as many times
as you like, so you can run tests that gradually expand the range of code
tested while simultaneously eliminating the instrumentation from most of the code
being executed.
Gretel's user interface is a simple Swing application and doesn't
provide much in the way of user-friendly features. I've listed a couple of
other open source choices in the Resources section that you may want to look into
if you're interested in application-wide code coverage checking, and there are
also commercial alternatives I haven't investigated. But rather than go into
detail about using any of these tools in this article, I'm just going to use
Gretel to show you how code coverage tracking is actually implemented.
The theory of tracking
To demonstrate how coverage tracking works, I'm going to go back to the
constructor for the StringArray class I've used
throughout this article. Suppose you want to add code to track which lines of the
constructor actually get executed while running some tests. One way
would be to create an array of booleans, one for each line of the original
method, and add code that sets the appropriate flag as each line is executed. Then,
after running your tests, you'd just need to inspect the array to see which
lines were not executed.
This approach of separately tracking each line works, but isn't very
efficient. If you look at execution paths through the constructor code,
shown in Listing 10, you can see that there are really only a few points that
need to be checked to make sure that all the code has executed. For instance, you
can see that any time the if condition is true, the
nested while loop will execute at least one time
(because the list length will be greater than zero). Going a step further, look at
the structure of the code inside the while loop. This
is just a three-way conditional. Because these three blocks are alternatives
(parallel paths through the code), each needs to be tracked separately at the
points shown in bold. But once the tracking has been implemented for these three
blocks, it'll supply all the information needed for the entire loop.
Listing 10. Coverage code sample
public StringArray(String[] list) {
// first make sure the array values are ordered
int dupl = 0;
if (list.length > 0) {
String last = list[0];
int index = 0;
while (++index < list.length) {
String comp = list[index];
int diff = last.compareTo(comp);
if (diff > 0) {
// disordered, sort values and restart scan
Arrays.sort(list);
dupl = 0;
index = 0;
last = list[index];
} else if (diff < 0) {
last = comp;
} else {
dupl++;
}
}
}
...
|
Gretel bread crumbs
Most coverage tools perform flow analysis of the Java bytecode along
the lines discussed in the last section, and only insert tracking where
needed. As an example of the actual tracking implementation, Listing 11 shows the
source equivalent of the Listing 10 code after processing by Gretel
("source equivalent" because Gretel works at the bytecode level, modifying the compiled
class files). Gretel uses calls to a static method to track executed blocks, adding
these into the bytecode only where needed.
Listing 11. Code modified by Gretel
public StringArray(String[] list) {
// first make sure the array values are ordered
int dupl = 0;
if (list.length > 0) {
String last = list[0];
int index = 0;
while (++index < list.length) {
String comp = list[index];
int diff = last.compareTo(comp);
if (diff > 0) {
// disordered, sort values and restart scan
Arrays.sort(list);
dupl = 0;
index = 0;
last = list[index];
residue.runtime.Monitor.hit(0);
} else if (diff < 0) {
last = comp;
residue.runtime.Monitor.hit(1);
} else {
residue.runtime.Monitor.hit(2);
dupl++;
}
}
}
...
|
Conclusions
Coverage tests let you see which code is actually being executed in
a class or application. In this article, I've looked at coverage from the
testing standpoint, but there are obviously other ways the information can be
used. For instance, most large projects grow by accretion as different developers
extend or maintain the project over a period of years. That makes it easy to
end up with orphan code in the source tree. If you run a series of tests for
your application and find that some code is never executed, this won't
always be due to incomplete tests -- sometimes you'll find that the code has just been
cut off from the execution paths by changes in the program logic or supported
configurations. This gives you the chance to prune dead code from your
source tree, which is always a worthwhile activity.
From the testing standpoint, any unit tests are always
better than no unit tests, but some unit tests are better than others.
The completeness of unit tests determines what proportion of errors will be
caught by the tests (rather than by those embarrassing crashes during
management demonstrations or in customer deployments). Code coverage tools give
you one way to measure test completeness. They can't check that your tests
include all the appropriate data patterns, but they can let you at least make sure
that all your code is being executed during the tests. Just by itself, that's a major
advance over tests that never execute some code at all!
In my earlier articles on Java programming dynamics (see Resources), I
showed how you can use classworking techniques to implement systematic changes to
program behavior. This type of approach is the basis for most of the work being
done with aspect-oriented programming (AOP) on the Java platform. Next month I'll dig into
the increasingly popular AspectWerkz framework, which provides flexible AOP
extensions for Java programming, and how you can apply it to one of the classic AOP use cases.
Check back with Classworking toolkit to see how AspectWerkz
measures up to the job.
Download | Name | Size | Download method |
|---|
| j-cwt02095-source.zip | 3 KB | HTTP |
Resources - Be sure to read all the installments in the Classworking toolkit series by Dennis Sosnoski
- "Keeping
critters out of your code: How to use WebSphere and JUnit to prevent
programming bugs" (developerWorks, June 2003) by David Carew and Sandeep Desai looks at taking an extreme programming approach to testing.
- "Incremental
development with Ant and JUnit" (developerWorks, November 2000) by Malcolm Davis discusses the benefits of unit testing.
- "Testing,
fun? Really?" (developerWorks, March 2001) by Jeff Canna looks at testing as a necessary evil.
- It's 10 pm. Do you know where your program is executing? Get
the open source Hansel and
Gretel
tools to help you track your code.
- Other code coverage tools for Java programming don't offer the advantage of using
names out of Mother Goose, but nonetheless provide some very nice
features. Check out Quilt or
JBlanket for
other approaches that integrate with JUnit and Ant,
Emma for large-scale
enterprise applications support, and
Jester for a source-based approach that mutates your code to see what
breaks, along with many other open source or commercial tools. And let me know
what you think -- if there's interest, I may cover one or more of these in
future columns.
- Not using unit tests? Quick,
get JUnit now and start
making unit tests a part of your development cycle before you're trapped
forever in the maintenance ward.
- Collect the whole Java
programming dynamics series by author Dennis Sosnoski, which takes you on a tour
of the Java class structure, reflection, and classworking.
- Find hundreds more Java technology resources on the developerWorks Java technology zone
- Browse for books on these and other technical topics.
About the author | 
| | Dennis Sosnoski is the founder and lead consultant of Seattle-area
Java technology consulting company Sosnoski Software Solutions, Inc., specialists in
XML and Web services training and
consulting. His professional software development experience spans over 30 years, with
the last several years focused on server-side XML and Java technologies. Dennis
is a frequent speaker at conferences nationwide. He's also the lead
developer of the open source JiBX XML Data Binding
framework built around Java classworking technology. |
Rate this page
| |
