Classworking toolkit: Cover your code with Hansel and Gretel

This month kicks off my new series on Java classworking tools. For the first installment, I'm covering a pair of related tools named Hansel and Gretel. These both address the issue of code coverage, determining which code is actually run during an execution of your application. Even though they're designed for very different situations, both Hansel and Gretel have some unique and interesting features that set them apart from other tools of this type.

As you might expect based on the names, Hansel and Gretel are related projects. Gretel actually came first, then Hansel was developed using portions of the Gretel code as a base. I'm going to reverse this order in the article, using Hansel first for an easy introduction to the principles of code coverage, then taking a quick look at Gretel to show how coverage testing is actually implemented down at the bytecode level.

I'll be looking at code coverage mainly from the standpoint of unit testing. Unit testing isn't the only scenario where code coverage tools are useful, but it's certainly a major use case. If you're not executing some code during your tests, that code is not being tested and is a potential source of undetected problems -- exactly what unit testing is intended to prevent. Code coverage tools let you expand the unit testing mantra of "clean and green" to "clean, green, and covered," with real benefits to the effectiveness of your unit tests.

Using Hansel

Hansel has one feature that makes it a great starting point for a discussion of coverage tools -- it integrates with the JUnit framework for unit tests, and lets you easily check the coverage of your unit test suites. JUnit is what most people assume when you discuss unit testing in Java development, which gives Hansel a big advantage over standalone coverage tools for use in unit testing.

Testing as normal

Listing 1 gives the source code for my StringArray class, a wrapper for ordered arrays of Strings. Besides the constructor, which takes an array and makes sure that it's ordered with no duplicates, the class also provides an indexOf() method that uses a binary search to find the index number of a string in the ordered array, along with a couple of simple access methods.

public class StringArray
{
    /** Ordered array of strings. */
    private final String[] m_list;
    
    /**
     * Constructor from array of values. This checks the array
     * values to make sure they're ordered and unique. If
     * they're not, it sorts them and eliminates duplicates. Once
     * the array has been passed to this constructor, it must
     * not be modified by outside code.
     * 
     * @param list array of values
     */
    
    public StringArray(String[] list) {
        
        // first make sure the array values are ordered
        int dupl = 0;
        if (list.length > 0) {
            String last = list[0];
            int index = 0;
            while (++index < list.length) {
                String comp = list[index];
                int diff = last.compareTo(comp);
                if (diff > 0) {
                    
                    // disordered, sort values in array
                    Arrays.sort(list);
                    last = list[index];
                    // there's an error here! see "When
                    // coverage is not enough"
                    
                } else if (diff < 0) {
                    last = comp;
                } else {
                    dupl++;
                }
            }
        }
        
        // eliminate duplicates if present
        if (dupl > 0) {
            String[] uniques = new String[list.length - dupl];
            String last = uniques[0] = list[0];
            int index = 0;
            int fill = 1;
            while (++index < list.length) {
                if (!last.equals(list[index])) {
                    last = list[index];
                    uniques[fill++] = last;
                }
            }
            m_list = uniques;
        } else {
            m_list = list;
        }
    }
    
    /**
     * Get string at a particular index in the list.
     *
     * @param index list index to be returned
     * @return string at that index position
     */
     
    public String get(int index) {
        return m_list[index];
    }
    
    /**
     * Find index of a particular string in the array. This does
     * a binary search through the array values, using a pair of
     * index bounds to track the subarray of possible matches at
     * each iteration.
     *
     * @param value string to be found in list
     * @return index of string in array or -1 if not present
     */
     
    public int indexOf(String value) {
        int base = 0;
        int limit = m_list.length - 1;
        while (base <= limit) {
            int cur = (base + limit) >> 1;
            int diff = value.compareTo(m_list[cur]);
            if (diff < 0) {
                limit = cur - 1;
            } else if (diff > 0) {
                base = cur + 1;
            } else {
                return cur;
            }
        }
        return -1;
    }
    
    /**
     * Get number of values in array
     *
     * @return number of values in array
     */
     
    public int size() {
        return m_list.length;
    }
}

Listing 2 gives a simple JUnit test case for this class, and Figure 1 shows the results when I run this test in Eclipse:

public class StringArrayTest extends TestCase
{
    private static String[] s_list1 = {
        "a", "b", "ccc", "ccd", "d", "e", "f", "g"
    };
    
    private static String[] s_list2 = { // s_list1 reordered
        "a", "b", "ccd", "ccc", "g", "f", "e", "d"
    };
    
    public void testStringArray() {
        StringArray array1 = new StringArray(s_list1);
        assertEquals(8, array1.size());
        StringArray array2 = new StringArray(s_list2);
        assertEquals(8, array2.size());
    }

    public void testIndexOf() {
        StringArray array = new StringArray(s_list1);
        assertEquals(-1, array.indexOf("ee"));
        assertEquals(4, array.indexOf("d"));
    }
}

Testing with coverage

So the code is "clean and green," according to my simple unit test case -- but that's meaningful only if I'm really testing the code thoroughly. To see just how effective my tests are at exercising the code, I can check code coverage with Hansel.

Adding Hansel checking to your JUnit tests is simple. You just need to add the Hansel JAR file to your test classpath and then create a test suite with a Hansel decorator (a class that wraps another class, modifying the behavior of the wrapped class) that references the class or classes being tested. For my StringArrayTest example, I can do this by adding the following simple method to the class:

public static Test suite() {
    return new CoverageDecorator(StringArrayTest.class,
        new Class[] { StringArray.class });
}

After this addition, my JUnit test results are very different, as shown in Figure 2 -- I now have four JUnit test failures!

Interpreting Hansel failures

In the Figure 2 test run, Hansel created a test failure for every coverage lapse it detected. This approach is a very nice way of showing the results if you're using an IDE that provides JUnit integration (like Eclipse, which was used for these tests), because it allows you to click on the failure trace line and immediately view the source code involved. In Figure 2, I've done this for the first reported failure, a branch not completely covered. Listing 3 shows the detailed failure message, along with a snippet of the code it's pointing at:

Coverage failure: Branch not completely covered. Condition
 'list.length <= 0' is not fulfilled.
 at com.sosnoski.demo.StringArray.<init>(StringArray.java:29)

       // first make sure the array values are ordered
       int dupl = 0;
       if (list.length > 0) {
           String last = list[0];
           int index = 0;

This failure says that I haven't tested the code using a zero-length array. That's easy enough to fix, so I can just add a quick test and move on to the next failure, shown in Listing 4:

Probe in "StringArray.java" line 41
 Coverage failure: Branch not completely covered. Condition
  'fill >= 0' is not fulfilled.
  at com.sosnoski.demo.StringArray.<init>(StringArray.java:41)
  
                if (diff > 0) {
                    
                    // disordered, sort values in array
                    Arrays.sort(list);
                    last = list[index];
                    // there's an error here! see "When
                    // coverage is not enough"
                    
                } else if (diff < 0) {
                    last = comp;
                } else {
                    dupl++;
                }

This time the message is a little more difficult to relate to the code. First off, the variable name doesn't match the source code. This result appears to be an error in the way Hansel interprets local variable information from the class file. The code here uses a local variable diff, which is only defined within this block of code, while later in the constructor a separate variable, fill, is declared at the same level. It looks like Hansel has confused the two variable names because they're using the same stack frame slot. That's annoying, but not too confusing once you know what's happening, because you'll still see the correct line of code.

The second difficulty with this failure is that the condition doesn't seem to match the actual test case -- the greater-than-zero test actually precedes the else statement this failure specifies. This result is also easy to understand once you know how Hansel works. The failure message is generated without respect to any of the preceding code in the method, looking only at the problem in isolation. With this idea in mind, the failure can be restated as "Condition 'diff < 0' was always true," which makes a lot more sense; my test cases don't include any arrays with duplicate values, which is what would be required to get to the third and final block in this set of three alternatives. Once I understand this failure, it's also easy enough to fix, so I'll again add another line to my tests and move on to the next failure, as shown in Listing 5:

Probe in "StringArray.java" line 50
 Coverage failure: Branch not completely covered. Condition
  '!(dupl <= 0)' is not fulfilled.
  at com.sosnoski.demo.StringArray.<init>(StringArray.java:50)
  
        // eliminate duplicates if present
        if (dupl > 0) {
            String[] uniques = new String[list.length - dupl];

This failure message is a little on the cryptic side, but with a little Boolean Logic 101, I can translate the message to "Condition '(dupl <= 0)' was always true." In terms of the test cases, this failure means that none of the tests had duplicates in the arrays I supplied to the constructor -- the same cause for the second failure. That leaves the fourth and final failure, shown in Listing 6:

Probe in "StringArray.java" line 75
 Coverage failure: Method not covered.
  at com.sosnoski.demo.StringArray.get(StringArray.java:75)
  
    public String get(int index) {
        return m_list[index];
    }

This failure is easy to understand, but I'm not sure I really want to "fix" it -- the method I'm not testing is trivial, and adding tests for trivial methods can be a waste of effort. Unfortunately, Hansel doesn't provide any options for suppressing failure messages, so if I want my JUnit tests to run with Hansel coverage testing and execute successfully, I need to add a test that uses this method. I'll use this requirement to improve the overall quality of my tests with one that verifies the lookup code, ending up with the test case shown in Listing 7 (changes are shown in bold):

public class StringArrayTest extends TestCase
{
    private static String[] s_list1 = {
        "a", "b", "ccc", "ccd", "d", "e", "f", "g"
    };
    
    private static String[] s_list2 = { // s_list1 reordered
        "a", "b", "ccd", "ccc", "g", "f", "e", "d"
    };
    
    private static String[] s_list3 = { // duplicates
        "a", "g", "ccc", "d", "d", "e", "ccc", "g"
    };
    
    public void testStringArray() {
        // added line below for first coverage failure
        StringArray array0 = new StringArray(new String[0]);
        StringArray array1 = new StringArray(s_list1);
        assertEquals(8, array1.size());
        StringArray array2 = new StringArray(s_list2);
        assertEquals(8, array2.size());
        // added line below for second/third coverage failure
        StringArray array3 = new StringArray(s_list3);
    }

    // changed this to test both get and indexOf, and sorting
    public void testLookup() {
        StringArray array = new StringArray(s_list2);
        assertEquals(-1, array.indexOf("ee"));
        for (int i = 0; i < s_list1.length; i++) {
            String value = array.get(i);
            assertEquals(s_list1[i], value);
            assertEquals(i, array.indexOf(value));
        }
    }
    
    public static Test suite() {
        return new CoverageDecorator(StringArrayTest.class,
            new Class[] { StringArray.class });
    }
}

This modified test case passes on all fronts, including the Hansel code coverage checks.

Back to top

The limits of coverage

Unfortunately, code coverage testing only makes sure you're executing all the code you'd like to test. It doesn't guarantee the quality of your tests. In the case of the code I've been using for this article, there's a major error in the interaction between the two loops in the constructor. The first loop checks both the order and uniqueness of values in the array passed to the constructor, while the second loop eliminates duplicate values counted by the first loop. Listing 8 shows that first loop, with the problem code shown in bold:

// first make sure the array values are ordered
int dupl = 0;
if (list.length > 0) {
    String last = list[0];
    int index = 0;
    while (++index < list.length) {
        String comp = list[index];
        int diff = last.compareTo(comp);
        if (diff > 0) {
                    
            // disordered, sort values in array
            Arrays.sort(list);,
            last = list[index];
                    
        } else if (diff < 0) {
            last = comp;
        } else {
            dupl++;
        }
    }
}

The problem with this code is that when the array of values is sorted, the count of duplicates will no longer be accurate. To demonstrate this error, I can change the second array in the unit tests to cause a duplicate value to be counted twice (once before the list is sorted, once after):

private static String[] s_list2 = { // s_list1 reordered
    "e", "e", "ccd", "ccc", "g", "f", "d", "a", "b"
};

This new test case gives me an ArrayIndexOutOfBoundsException in the second loop of the constructor. The problem is easy to fix, by just restarting the duplicate scan when the array is sorted, as shown in Listing 9:

// disordered, sort values and restart scan
Arrays.sort(list);
dupl = 0;
index = 0;
  last = list[index];

But Hansel assured me that all my code was being executed by the original JUnit tests, so why didn't this problem show up in those tests? The reason is that this problem is data-dependent; it'll lurk in the code until it spots just the pattern of data it wants, and then leap out and bite me (I tend to take bugs personally!). Data-dependent errors are a lot harder to detect than simple logic errors, and, unfortunately, code coverage tools can't make sure that you've tested all the combinations of data that might trigger such an error. The best you can do to flush these errors out into the open is use a wide variety of data in your unit tests.

Back to top

Uncovering classworking

So how does code coverage relate to classworking? Test coverage tools modify your code to leave behind a trail of "bread crumbs" during execution (hence the names of the tools I'm targeting in this column). After the tests have been executed, the tool can check this trail to see which portions of the code were actually executed during the test. To demonstrate how this feature works, I'll introduce you to the other half of our fairy tale duo, Gretel.

While Hansel is designed for class-at-a-time coverage checking integrated into JUnit tests, Gretel is designed for application-wide coverage checking. You start by using Gretel to instrument your code, next running one or more tests with the Gretel instrumentation saving execution data to files, then using Gretel again to view the saved data. In this way, Gretel is similar to several other open source or commercial coverage checking tools.

Where Gretel goes beyond most other code coverage tools is in its support of incremental coverage checking. Adding instrumentation to your code slows execution slightly, and if you run a large set of tests for your application, the performance loss due to instrumentation may be a serious concern (especially if you have tests that are timing-sensitive). To alleviate this problem, Gretel lets you reinstrument your code after running an initial set of tests. This reinstrumentation step removes the instrumentation from code that's already been covered, only leaving it in place for code that wasn't executed by the initial tests. You can repeat this reinstrumentation step as many times as you like, so you can run tests that gradually expand the range of code tested while simultaneously eliminating the instrumentation from most of the code being executed.

Gretel's user interface is a simple Swing application and doesn't provide much in the way of user-friendly features. I've listed a couple of other open source choices in the Resources section that you may want to look into if you're interested in application-wide code coverage checking, and there are also commercial alternatives I haven't investigated. But rather than go into detail about using any of these tools in this article, I'm just going to use Gretel to show you how code coverage tracking is actually implemented.

The theory of tracking

To demonstrate how coverage tracking works, I'm going to go back to the constructor for the StringArray class I've used throughout this article. Suppose you want to add code to track which lines of the constructor actually get executed while running some tests. One way would be to create an array of booleans, one for each line of the original method, and add code that sets the appropriate flag as each line is executed. Then, after running your tests, you'd just need to inspect the array to see which lines were not executed.

This approach of separately tracking each line works, but isn't very efficient. If you look at execution paths through the constructor code, shown in Listing 10, you can see that there are really only a few points that need to be checked to make sure that all the code has executed. For instance, you can see that any time the if condition is true, the nested while loop will execute at least one time (because the list length will be greater than zero). Going a step further, look at the structure of the code inside the while loop. This is just a three-way conditional. Because these three blocks are alternatives (parallel paths through the code), each needs to be tracked separately at the points shown in bold. But once the tracking has been implemented for these three blocks, it'll supply all the information needed for the entire loop.

public StringArray(String[] list) {
        
    // first make sure the array values are ordered
    int dupl = 0;
    if (list.length > 0) {
        String last = list[0];
        int index = 0;
        while (++index < list.length) {
            String comp = list[index];
            int diff = last.compareTo(comp);
            if (diff > 0) {
                    
                // disordered, sort values and restart scan
                Arrays.sort(list);
                dupl = 0;
                index = 0;
                last = list[index];
                    
            } else if (diff < 0) {
                last = comp;
            } else {
                dupl++;
            }
        }
    }
        ...

Gretel bread crumbs

Most coverage tools perform flow analysis of the Java bytecode along the lines discussed in the last section, and only insert tracking where needed. As an example of the actual tracking implementation, Listing 11 shows the source equivalent of the Listing 10 code after processing by Gretel ("source equivalent" because Gretel works at the bytecode level, modifying the compiled class files). Gretel uses calls to a static method to track executed blocks, adding these into the bytecode only where needed.

public StringArray(String[] list) {
        
    // first make sure the array values are ordered
    int dupl = 0;
    if (list.length > 0) {
        String last = list[0];
        int index = 0;
        while (++index < list.length) {
            String comp = list[index];
            int diff = last.compareTo(comp);
            if (diff > 0) {
                    
                // disordered, sort values and restart scan
                Arrays.sort(list);
                dupl = 0;
                index = 0;
                last = list[index];
                residue.runtime.Monitor.hit(0);
                    
            } else if (diff < 0) {
                last = comp;
                residue.runtime.Monitor.hit(1);
            } else {
                residue.runtime.Monitor.hit(2);
                dupl++;
            }
        }
    }
    ...

Back to top

Conclusions

Coverage tests let you see which code is actually being executed in a class or application. In this article, I've looked at coverage from the testing standpoint, but there are obviously other ways the information can be used. For instance, most large projects grow by accretion as different developers extend or maintain the project over a period of years. That makes it easy to end up with orphan code in the source tree. If you run a series of tests for your application and find that some code is never executed, this won't always be due to incomplete tests -- sometimes you'll find that the code has just been cut off from the execution paths by changes in the program logic or supported configurations. This gives you the chance to prune dead code from your source tree, which is always a worthwhile activity.

From the testing standpoint, any unit tests are always better than no unit tests, but some unit tests are better than others. The completeness of unit tests determines what proportion of errors will be caught by the tests (rather than by those embarrassing crashes during management demonstrations or in customer deployments). Code coverage tools give you one way to measure test completeness. They can't check that your tests include all the appropriate data patterns, but they can let you at least make sure that all your code is being executed during the tests. Just by itself, that's a major advance over tests that never execute some code at all!

In my earlier articles on Java programming dynamics (see Resources), I showed how you can use classworking techniques to implement systematic changes to program behavior. This type of approach is the basis for most of the work being done with aspect-oriented programming (AOP) on the Java platform. Next month I'll dig into the increasingly popular AspectWerkz framework, which provides flexible AOP extensions for Java programming, and how you can apply it to one of the classic AOP use cases. Check back with Classworking toolkit to see how AspectWerkz measures up to the job.

Back to top

Download

Information about download methods

Resources

• Be sure to read all the installments in the Classworking toolkit series by Dennis Sosnoski
  
  
• "Keeping critters out of your code: How to use WebSphere and JUnit to prevent programming bugs" (developerWorks, June 2003) by David Carew and Sandeep Desai looks at taking an extreme programming approach to testing.
  
  
• "Testing, fun? Really?" (developerWorks, March 2001) by Jeff Canna looks at testing as a necessary evil.
  
  
• It's 10 pm. Do you know where your program is executing? Get the open source Hansel and Gretel tools to help you track your code.
  
  
• Other code coverage tools for Java programming don't offer the advantage of using names out of Mother Goose, but nonetheless provide some very nice features. Check out Quilt for other approaches that integrate with JUnit and Ant, Emma for large-scale enterprise applications support, and Jester for a source-based approach that mutates your code to see what breaks, along with many other open source or commercial tools. And let me know what you think -- if there's interest, I may cover one or more of these in future columns.
  
  
• Not using unit tests? Quick, get JUnit now and start making unit tests a part of your development cycle before you're trapped forever in the maintenance ward.
  
  
• Collect the whole Java programming dynamics series by author Dennis Sosnoski, which takes you on a tour of the Java class structure, reflection, and classworking.
  
  
• Find hundreds more Java technology resources on the developerWorks Java technology zone
  
  
• Browse for books on these and other technical topics.
  
  

About the author

Dennis Sosnoski is the founder and lead consultant of Seattle-area Java technology consulting company Sosnoski Software Solutions, Inc., specialists in XML and Web services training and consulting. His professional software development experience spans over 30 years, with the last several years focused on server-side XML and Java technologies. Dennis is a frequent speaker at conferences nationwide. He's also the lead developer of the open source JiBX XML Data Binding framework built around Java classworking technology.

Comments

Back to top

Trademarks

Table of contents

• Using Hansel
• The limits of coverage
• Uncovering classworking
• Conclusions
• Download
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

My developerWorks community

Dig deeper into Java on developerWorks

You tell us!

How does the developerWorks community use social media?

---

Take the survey

Special offers