 | Level: Introductory Elliotte Harold (elharo@metalab.unc.edu), Adjunct Professor, Polytechnic University
04 Apr 2006 Test-first programming is the most effective coding
practice since object-oriented programming, but it assumes you're
starting from a blank screen. What do you do when the code already
exists? Using a popular open source Java™ tool as his example, author
Elliotte Rusty Harold shows you how to develop a test suite for legacy
code that's never been tested.
/* I have no idea how this works but it seems to. Whatever you
do, don't touch this function, and don't break this code! */ |
You're not alone if you've ever come across a chunk of code with a
comment like this one. Entire systems are sometimes ruled
off-limits because no one understands them; yet these systems still need
to be maintained. Even if a system were perfectly bug free (and what
system is?), changing external conditions necessitates code changes. The
Y2K turnover was simply the largest and most obvious example. The
introduction of the Euro was equally traumatic for some financial
systems. Sarbanes-Oxley introduced new reporting requirements that
didn't exist before, and legacy software had to be retrofitted to
support these new regulations. The world is not static, and software
can't be either. It has to evolve or be replaced.
The good news is that test-driven development isn't just for new
code. Even programmers maintaining old systems can profitably write,
run, and pass tests. Indeed tests are even more important for
legacy systems already in production. Only by testing can you be
confident that changes you make to one part of a system will not break
another part somewhere else. Sure, you might not have the time or the
budget to achieve 100 percent test coverage for a large legacy code base, but
even less-than-perfect coverage reduces the risk of failure, speeds
up development, and produces more robust code.
This article shows you how to develop a unit test suite for
legacy code that's never been tested, using jEdit as an example.
jEdit is a popular open-source text editor that has
absolutely no test suite -- zip, zero, zilch, nada! But I'm going
to fix that, starting right now. In this article, I begin developing a
test suite that aims to make future development of jEdit more
productive, more efficient, and more fun.
The first test
As the old Chinese saying goes, a journey of a thousand miles begins
with a single step. A test suite for legacy code begins with a single
test. Focus on what you can do and work from there. Don't fall into the
trap of believing that because you can't test every line of code, you
must test nothing. Just open up your IDE and start writing tests. With
JUnit (or NUnit, or CppUnit, or whichever framework you prefer) and a
half-decent IDE, you can usually write your first test in under 20
minutes. Writing tests is easier than writing model code. Tests are very
small, self-contained chunks of code. They don't take a lot of setup,
thought, or understanding. You don't have to be "in the zone" to churn
out quality tests.
The first thing the test suite should do is go right down the
center of the road. Look for the biggest, most sweeping test you
can make. For a stand-alone application, this is likely the main() method. For example, here's my first jEdit
test case. All it does is run the application's main() method and verify that it puts the right
kind of window on the screen:
Listing 1. Testing the jEdit main() method
import java.awt.Frame;
import junit.framework.TestCase;
public class MainTest extends TestCase {
public void testMain() {
org.gjt.sp.jedit.jEdit.main(new String[0]);
// make sure there's a window on the screen
Frame[] allFrames = Frame.getFrames();
for (int i = 0; i < allFrames.length; i++) {
Frame f = allFrames[i];
if (f.isFocused()) {
assertTrue(f instanceof org.gjt.sp.jedit.View);
}
}
}
} |
The goal of the first test is not to tug at the edge conditions and
see what unravels. The first test is a smoke test, giving you a broad
idea of what might be wrong. Even the most basic test can uncover
problems in the build system, the runtime environment, the installed
software, and other major problems that break essentially everything.
Indeed, my first test case found exactly such a problem with the jEdit
code base: I didn't have all the necessary directories in my
classpath.
I didn't set out to test the classpath setup, but the problem I found
is important because it could make the code base harder to debug. A
broad test like this one cuts a large swath across the application. Lots
of different things could break and cause this test to fail. In this
sense, it's not very unitary. In test-first programming that's a
problem, but when testing legacy code, you don't have the time or budget
to write separate tests for each individual method and branch. You have
to write each test to cover as much as it possibly can. It's better to
have most code tested by something than not tested at all.
Troubleshooting main()
Testing the main() method does not work with all
applications. For instance, libraries do not have main() methods. Some applications that do have a
main() method may not expect that method to
be called more than once. Static initializers in particular can
get very confused if you do that. Some objects and classes may not be
cleaned up because they assume that once the program exits, the virtual
machine will exit too, and everything will be cleaned up automatically.
If this is the case, you may need to look deeper into the application to
find your first test points.
Don't go deeper than you have to, though. One problem you're likely
to encounter in test-last as opposed to test-first development,
especially with legacy code, is that there are often unadvertised
dependencies and prerequisites. Some methods assume other objects
exist and have been set up before they run. For instance, most menu bars
won't work in isolation from their parent frame.
In fact, jEdit gets very confused if you try to call the main() method more than once. I'd like to not call
it at all. However, a lot of other code depends on the jEdit.initSystemProperties() method having been
called, and this method is private. The only way to execute it is to
call main(). The solution I took was to call
main() only if it hadn't already been called
once, as shown below:
private static boolean hasMain = false;
protected void setUp() {
if (!hasMain) {
jEdit.main(new String[0]);
hasMain = true;
}
View view = jEdit.getFirstView();
while (view == null) {
// First window may take a little while to appear
view = jEdit.getFirstView();
}
menubar = view.getJMenuBar();
} |
| | It's important that the hasMain field be static here. JUnit constructs a
new test-case object for each test method, so only static fields can
hold per-suite state. |
|
Your job is easier if you have the freedom to refactor the code
you're testing. In particular, being able to change a few private methods
to public would have made this code a lot easier to write. None of this
is likely to be an issue in test-first development because there you
tend to design the code so it's easily testable. However, testability has rarely
been a consideration for legacy code, and consequently, you have to pull
stunts like this one.
Introducing
fixtures
Once you've written the first test, you can often develop more
tests very quickly from the same framework. This is where fixtures come
in handy. Pull the initialization and cleanup code out into the setUp() and tearDown()
methods and see how many tests you can write really fast from there.
For example, I wrote some basic tests to ensure the jEdit menu bar shows
up and has the right menus in the right places, as shown in Listing 2:
Listing 2. Testing the jEdit
menus
package org.jedit.test;
import javax.swing.*;
import org.gjt.sp.jedit.*;
import junit.framework.TestCase;
public class MenuTest extends TestCase {
private JMenuBar menubar;
private static boolean hasMain = false;
protected void setUp() {
if (!hasMain) {
jEdit.main(new String[0]);
hasMain = true;
}
View view = jEdit.getFirstView();
while (view == null) {
// First window may take a little while to appear
view = jEdit.getFirstView();
}
menubar = view.getJMenuBar();
}
public void testFileIsFirstMenu() {
JMenu file = menubar.getMenu(0);
assertEquals("File", file.getText());
}
public void testEditIsSecondMenu() {
JMenu edit = menubar.getMenu(1);
assertEquals("Edit", edit.getText());
}
public void testHelpIsLastMenu() {
JMenu help = menubar.getMenu(menubar.getMenuCount()-1);
assertEquals("Help", help.getText());
}
// Tests for other menus...
} |
Unlike classic test-first programming, I'm writing a lot of tests
here without necessarily writing any model code. Normal TDD writes only
enough code to make a test fail. Then it switches into model coding
until the test passes. Then it switches back. I'm not saying that the
TDD doctrine is wrong, but it's not really an option when two
million lines of legacy model code already exist. In that case, the goal
is to get as much coverage as possible, as quickly as possible.
| | You might have noticed another problem with
my tests. jEdit is internationalized, so the tests should be too. That
is, the strings like "File" and "Edit" should be pulled out into
resource bundles so the tests will pass on localized systems where they
might have other names such as "Fichier," "Edition," and "Aide."
That's not hard to do, but it's a tangential point, so I've left it for
another time. |
|
Test or debug?
Hopefully, if the legacy system is a good one, most of your tests
will pass. Nonetheless you will find bugs. When testing a previously
untested code base, that's likely to happen sooner rather than later.
The normal TDD approach at this point is to stop testing and start
fixing until the test passes. However, this assumes that you have tests
for everything else in the model and can be fairly confident that
you'll find out immediately if your fix breaks some other piece of the
system. In legacy testing, this isn't such a safe approach. You may well
introduce a new bug in untested code while fixing an old bug, and if so,
you probably won't immediately notice the new bug. Consequently, I'm
sorely tempted to write more tests first and leave the bug fixing for
later. Ultimately this is a judgment call based on a few factors:
- Does the bug seem simple, obvious, and local?
- Do you understand the code where the bug appears?
- Do you understand the fix?
If the answer to all the above questions is yes, then go ahead
and fix the bug. If the answer is no, then you might try to expand the
test suite first before fixing the code.
Testing by functional division
The fastest gains in code coverage come by starting your tests at the
highest level. For legacy code, rather than looking at individual
methods, look at what the application is doing. Try to write tests for
each thing it does. For a GUI application like jEdit, the menu items
offer a good cross-section of application functionality. Activate each
one and make sure it does what it's supposed to. For example, Listing 3
shows a test that puts some text in a window, activates the "select all"
menu item, cuts the selection, and then verifies that the text is on the
clipboard and not in the window:
Listing 3. Testing the jEdit menus
public void testCut() {
JEditTextArea ta = view.getTextArea();
ta.setText("Testing 1 2 3");
JMenu edit = menubar.getMenu(1);
JMenuItem selectAll = edit.getItem(8);
selectAll.doClick();
JMenuItem cut = edit.getItem(3);
cut.doClick();
assertEquals("", ta.getText());
assertEquals("Testing 1 2 3", getClipboardText());
}
private static String getClipboardText() {
Clipboard clipboard = Toolkit.getDefaultToolkit().getSystemClipboard();
Transferable contents = clipboard.getContents(null);
if (contents != null
&& contents.isDataFlavorSupported(DataFlavor.stringFlavor) ) {
try {
return (String) contents.getTransferData(DataFlavor.stringFlavor);
}
catch (Exception ex){
return null;
}
}
return null;
}
|
It's possible that I tested a little too much in this method.
It might have been better to move some of the test out into fixtures.
However, by all means make the tests.
|
Uh oh ...
I'm still investigating, but the test in Listing 3 may have found a
real bug in jEdit. The first time I ran it, the test failed. When I set
a breakpoint and ran it in the debugger, it passed. It's a
Heisenbug! I suspected a timing problem involving multiple
threads, so I threw in a 2.5-second delay before the call to setText(), and the test passed. Without the delay, it
fails consistently. With the delay, it passes. The next step is to
figure out why the delay is necessary and determine if that's a fixable
bug. No one said writing tests for GUI code was easy.
|
|
Testing by code
structure
Once you've tested the basic functionality of an application, it
becomes important to consider alternate pathways through the code. In
most languages, you can break up your tests as follows:
- Write one test for every package or module.
- Once every package has at least one test, write one test for every
class.
- Once every class has at least one test, write one test for every
method.
- Once every method has at least one test, use a code coverage tool,
such as Cobertura, to write one test for every branch until each line of
code is tested.
You could use a code coverage tool before Step 4, but I prefer to do
the preceding steps manually. Although many classes, packages, and
methods are tested by the functionality tests, you often find
different problems when looking at the program from a programmer's view
rather than a user's view.
In fact, most of the time, you're never going to get to Step 4. You
simply won't have the time or the budget to write every possible test.
That's OK: the difference between some tests and no tests is much larger
and more significant than the difference between all tests and some
tests.
Automated testing
It is possible to use reflection to generate a test skeleton. This
makes it easier to find all the public methods you need to test.
Each test starts life looking like this:
public void testMethodName() {
fail("Test Code Not Written Yet");
} |
The downside to this approach is that you'll immediately have
hundreds to thousands of failing tests. An alternative is to add a TODO
comment in each test rather than an outright failure. Then as time
permits, you can go through and fill in tests.
public void testMethodName() {
// TODO fill in test code
} |
If you're using JUnit 4, you could simply annotate the tests as
@Ignore until you fill them in, like so:
@Ignore public void testMethodName() {
// TODO fill in test code
} |
In this case, the test runner reminds you that it skipped the
tests, but it won't actually fail you. In essence, it grades you
Pass-Fail-Incomplete.
You can find a few tools that automatically write tests for you.
However, the tests such tools come up with tend to be fairly trivial and
basic stuff, like whether a method can handle being passed a null
argument. Such tools don't really understand what each class or method
is supposed to be doing. For that, you need human intelligence.
Handling dead code
One thing that's likely to surprise you at this stage is how much of
your code you don't really need. Legacy code bases tend to have lots of
vestigial code; code that was necessary at one point in
time but is no longer. The older and crustier the legacy code is, the
more vestigial code you'll find. Sometimes the code is very obviously
unreachable (uncalled private methods, unread local variables, and so
forth). This sort of vestigial code can also be found by static code
analysis tools like PMD and FindBugs. Sometimes the code isn't so
obviously vestigial, and only the attempt to reach it for testing can
reveal just how vestigial it really is.
However you find such vestigial code, or whyever it was put
there in the first place, take it out. The less code you have to
maintain, the better.
Exploratory testing
Going into a legacy system, you'll often have a good idea of where
you need to look: You're having problems with a particular module or
package or set of circumstances, and that problem drives your testing.
In this case, by all means focus your tests on that one area.
Sometimes you have a very clear and obvious bug. Before you start
fixing it, write a test. Then run the test to make sure the test fails.
Surprisingly often, however, your first instinct about the cause of a
bug is not correct, in which case the test passes. Whether a test fails
or not, don't throw it away. It's still valuable for future development.
Leave it in your test suite and write another test. Repeat until you
find a test that does fail and thus discover the true cause of the bug.
In conclusion
Don't let perfection be the enemy of the good. Even if you have a
large, untested legacy code base, start writing tests for it now. Don't
worry about getting to 100 percent coverage. Every test you write will increase
your confidence in the code, squeeze out bugs, and provide more
flexibility for future development. Need to add a feature? Write a test.
Find a bug? Write another test. Legacy programmers can be agile too.
Resources Learn
- "Incremental development with Ant and JUnit" (Malcolm Davis, developerWorks, November 2000): Introduces unit testing on the Java platform.
- "Demystifying Extreme Programming: Test-driven programming" (Roy Miller, developerWorks, April 2003): Explains what test-driven programming is all about, and more importantly, what it isn't about.
- "TestNG makes Java unit testing a breeze" (Filippo Diotalevi, developerWorks, January 2005): TestNG is less focused on unit testing and test-first programming than JUnit is, making it in some ways a superior product for
testing legacy code.
- "Keeping critters out of your code" (David Carew, Sandeep Desai, Anthony Young-Garner; developerWorks, June 2003): Introduces unit testing a server-side application server environment.
- "Measure test coverage with Cobertura" (Elliotte Rusty Harold, developerWorks, May 2005): Shows how to use an open-source tool to identify untested code.
- "An early look at JUnit 4" (Elliotte Rusty Harold, developerWorks, September 2005): Introduces the new annotation-based architecture of JUnit 4. This requires Java 5 or later.
- Pragmatic Unit Testing in Java (Dave Thomas and Andy Hunt, Pragmatic Programmers, September 2003): Read Dave Thomas and Andy Hunt's book.
- The Java technology zone: Hundreds of articles about every aspect of Java programming.
Get products and technologies
- jEdit: The open source programmer's
editor used as a guinea pig in this article.
- JUnit: Get Test Infected.
- Cobertura: Download it from
SourceForge.
Discuss
About the author | 
| | Elliotte Rusty Harold is originally from New Orleans, to which he returns periodically in search of a decent bowl of gumbo. However, he resides in the Prospect Heights neighborhood of Brooklyn with his wife Beth and cats Charm (named after the quark) and Marjorie (named after his mother-in-law). He's an adjunct professor of computer science at Polytechnic University, where he teaches Java technology and object-oriented programming. His Cafe au Lait Web site has become one of the most popular independent Java sites on the Internet, and his spin-off site, Cafe con Leche, has become one of the most popular XML sites. His books include Effective XML, Processing XML with Java, Java Network Programming, and The XML 1.1 Bible. He's currently working on the XOM API for processing XML, the Jaxen XPath engine, and the Jester test coverage tool.
|
Rate this page
| |
