Testing legacy code

Extreme maintenance programming for legacy systems

Elliotte Rusty Harold (elharo@metalab.unc.edu), Adjunct Professor, Polytechnic University

Elliotte Rusty Harold is originally from New Orleans, to which he returns periodically in search of a decent bowl of gumbo. However, he resides in the Prospect Heights neighborhood of Brooklyn with his wife Beth and cats Charm (named after the quark) and Marjorie (named after his mother-in-law). He's an adjunct professor of computer science at Polytechnic University, where he teaches Java technology and object-oriented programming. His Cafe au Lait Web site has become one of the most popular independent Java sites on the Internet, and his spin-off site, Cafe con Leche, has become one of the most popular XML sites. His books include Effective XML, Processing XML with Java, Java Network Programming, and The XML 1.1 Bible. He's currently working on the XOM API for processing XML, the Jaxen XPath engine, and the Jester test coverage tool.

Summary: Test-first programming is the most effective coding practice since object-oriented programming, but it assumes you're starting from a blank screen. What do you do when the code already exists? Using a popular open source Java™ tool as his example, author Elliotte Rusty Harold shows you how to develop a test suite for legacy code that's never been tested.

Date: 04 Apr 2006
Level: Introductory
Activity: 1070 views
Comments:

/* I have no idea how this works but it seems to.  Whatever you  
   do, don't touch this function, and don't break this code! */

You're not alone if you've ever come across a chunk of code with a comment like this one. Entire systems are sometimes ruled off-limits because no one understands them; yet these systems still need to be maintained. Even if a system were perfectly bug free (and what system is?), changing external conditions necessitates code changes. The Y2K turnover was simply the largest and most obvious example. The introduction of the Euro was equally traumatic for some financial systems. Sarbanes-Oxley introduced new reporting requirements that didn't exist before, and legacy software had to be retrofitted to support these new regulations. The world is not static, and software can't be either. It has to evolve or be replaced.

The good news is that test-driven development isn't just for new code. Even programmers maintaining old systems can profitably write, run, and pass tests. Indeed tests are even more important for legacy systems already in production. Only by testing can you be confident that changes you make to one part of a system will not break another part somewhere else. Sure, you might not have the time or the budget to achieve 100 percent test coverage for a large legacy code base, but even less-than-perfect coverage reduces the risk of failure, speeds up development, and produces more robust code.

This article shows you how to develop a unit test suite for legacy code that's never been tested, using jEdit as an example. jEdit is a popular open-source text editor that has absolutely no test suite -- zip, zero, zilch, nada! But I'm going to fix that, starting right now. In this article, I begin developing a test suite that aims to make future development of jEdit more productive, more efficient, and more fun.

The first test

As the old Chinese saying goes, a journey of a thousand miles begins with a single step. A test suite for legacy code begins with a single test. Focus on what you can do and work from there. Don't fall into the trap of believing that because you can't test every line of code, you must test nothing. Just open up your IDE and start writing tests. With JUnit (or NUnit, or CppUnit, or whichever framework you prefer) and a half-decent IDE, you can usually write your first test in under 20 minutes. Writing tests is easier than writing model code. Tests are very small, self-contained chunks of code. They don't take a lot of setup, thought, or understanding. You don't have to be "in the zone" to churn out quality tests.

The first thing the test suite should do is go right down the center of the road. Look for the biggest, most sweeping test you can make. For a stand-alone application, this is likely the main() method. For example, here's my first jEdit test case. All it does is run the application's main() method and verify that it puts the right kind of window on the screen:

Listing 1. Testing the jEdit main() method

import java.awt.Frame;
import junit.framework.TestCase;

public class MainTest extends TestCase {

    public void testMain() {
        org.gjt.sp.jedit.jEdit.main(new String[0]);
        // make sure there's a window on the screen
        Frame[] allFrames = Frame.getFrames();
        for (int i = 0; i < allFrames.length; i++) {
           Frame f = allFrames[i];
           if (f.isFocused()) {
              assertTrue(f instanceof org.gjt.sp.jedit.View);
           }
        }
    }

}

The goal of the first test is not to tug at the edge conditions and see what unravels. The first test is a smoke test, giving you a broad idea of what might be wrong. Even the most basic test can uncover problems in the build system, the runtime environment, the installed software, and other major problems that break essentially everything. Indeed, my first test case found exactly such a problem with the jEdit code base: I didn't have all the necessary directories in my classpath.

I didn't set out to test the classpath setup, but the problem I found is important because it could make the code base harder to debug. A broad test like this one cuts a large swath across the application. Lots of different things could break and cause this test to fail. In this sense, it's not very unitary. In test-first programming that's a problem, but when testing legacy code, you don't have the time or budget to write separate tests for each individual method and branch. You have to write each test to cover as much as it possibly can. It's better to have most code tested by something than not tested at all.

Troubleshooting main()

Testing the main() method does not work with all applications. For instance, libraries do not have main() methods. Some applications that do have a main() method may not expect that method to be called more than once. Static initializers in particular can get very confused if you do that. Some objects and classes may not be cleaned up because they assume that once the program exits, the virtual machine will exit too, and everything will be cleaned up automatically. If this is the case, you may need to look deeper into the application to find your first test points.

Don't go deeper than you have to, though. One problem you're likely to encounter in test-last as opposed to test-first development, especially with legacy code, is that there are often unadvertised dependencies and prerequisites. Some methods assume other objects exist and have been set up before they run. For instance, most menu bars won't work in isolation from their parent frame.

In fact, jEdit gets very confused if you try to call the main() method more than once. I'd like to not call it at all. However, a lot of other code depends on the jEdit.initSystemProperties() method having been called, and this method is private. The only way to execute it is to call main(). The solution I took was to call main() only if it hadn't already been called once, as shown below:

private static boolean hasMain = false;

 protected void setUp() {
    if (!hasMain) {
        jEdit.main(new String[0]);
        hasMain = true;
    }
    View view = jEdit.getFirstView();
    while (view == null) {
        // First window may take a little while to appear
        view = jEdit.getFirstView();
    }
    menubar = view.getJMenuBar();
   }

Your job is easier if you have the freedom to refactor the code you're testing. In particular, being able to change a few private methods to public would have made this code a lot easier to write. None of this is likely to be an issue in test-first development because there you tend to design the code so it's easily testable. However, testability has rarely been a consideration for legacy code, and consequently, you have to pull stunts like this one.

Introducing fixtures

Once you've written the first test, you can often develop more tests very quickly from the same framework. This is where fixtures come in handy. Pull the initialization and cleanup code out into the setUp() and tearDown() methods and see how many tests you can write really fast from there. For example, I wrote some basic tests to ensure the jEdit menu bar shows up and has the right menus in the right places, as shown in Listing 2:

Listing 2. Testing the jEdit menus


package org.jedit.test;

import javax.swing.*;
import org.gjt.sp.jedit.*;
import junit.framework.TestCase;

public class MenuTest extends TestCase {

    private JMenuBar menubar;
    private static boolean hasMain = false;

    protected void setUp() {
        if (!hasMain) {
            jEdit.main(new String[0]);
            hasMain = true;
        }
        View view = jEdit.getFirstView();
        while (view == null) {
            // First window may take a little while to appear
            view = jEdit.getFirstView();
        }
        menubar = view.getJMenuBar();
    }

    public void testFileIsFirstMenu() {
        JMenu file = menubar.getMenu(0);
        assertEquals("File", file.getText());
    }

    public void testEditIsSecondMenu() {
        JMenu edit = menubar.getMenu(1);
        assertEquals("Edit", edit.getText());
    }

    public void testHelpIsLastMenu() {
        JMenu help = menubar.getMenu(menubar.getMenuCount()-1);
        assertEquals("Help", help.getText());
    }

    // Tests for other menus...

}

Unlike classic test-first programming, I'm writing a lot of tests here without necessarily writing any model code. Normal TDD writes only enough code to make a test fail. Then it switches into model coding until the test passes. Then it switches back. I'm not saying that the TDD doctrine is wrong, but it's not really an option when two million lines of legacy model code already exist. In that case, the goal is to get as much coverage as possible, as quickly as possible.

Test or debug?

Hopefully, if the legacy system is a good one, most of your tests will pass. Nonetheless you will find bugs. When testing a previously untested code base, that's likely to happen sooner rather than later. The normal TDD approach at this point is to stop testing and start fixing until the test passes. However, this assumes that you have tests for everything else in the model and can be fairly confident that you'll find out immediately if your fix breaks some other piece of the system. In legacy testing, this isn't such a safe approach. You may well introduce a new bug in untested code while fixing an old bug, and if so, you probably won't immediately notice the new bug. Consequently, I'm sorely tempted to write more tests first and leave the bug fixing for later. Ultimately this is a judgment call based on a few factors:

Does the bug seem simple, obvious, and local?
Do you understand the code where the bug appears?
Do you understand the fix?

If the answer to all the above questions is yes, then go ahead and fix the bug. If the answer is no, then you might try to expand the test suite first before fixing the code.

Testing by functional division

The fastest gains in code coverage come by starting your tests at the highest level. For legacy code, rather than looking at individual methods, look at what the application is doing. Try to write tests for each thing it does. For a GUI application like jEdit, the menu items offer a good cross-section of application functionality. Activate each one and make sure it does what it's supposed to. For example, Listing 3 shows a test that puts some text in a window, activates the "select all" menu item, cuts the selection, and then verifies that the text is on the clipboard and not in the window:

Listing 3. Testing the jEdit menus


public void testCut() {
      JEditTextArea ta = view.getTextArea();
      ta.setText("Testing 1 2 3");
      JMenu edit = menubar.getMenu(1);
      JMenuItem selectAll = edit.getItem(8);
      selectAll.doClick();
      JMenuItem cut = edit.getItem(3);
      cut.doClick();
      assertEquals("", ta.getText());
      assertEquals("Testing 1 2 3", getClipboardText());
}

private static String getClipboardText() {
      Clipboard clipboard = Toolkit.getDefaultToolkit().getSystemClipboard();
      Transferable contents = clipboard.getContents(null);
      if (contents != null 
            && contents.isDataFlavorSupported(DataFlavor.stringFlavor) ) {
       try {
           return (String) contents.getTransferData(DataFlavor.stringFlavor);
       }
       catch (Exception ex){
           return null;
       }
      }
      return null;
}

It's possible that I tested a little too much in this method. It might have been better to move some of the test out into fixtures. However, by all means make the tests.

Uh oh ...

I'm still investigating, but the test in Listing 3 may have found a real bug in jEdit. The first time I ran it, the test failed. When I set a breakpoint and ran it in the debugger, it passed. It's a Heisenbug! I suspected a timing problem involving multiple threads, so I threw in a 2.5-second delay before the call to setText(), and the test passed. Without the delay, it fails consistently. With the delay, it passes. The next step is to figure out why the delay is necessary and determine if that's a fixable bug. No one said writing tests for GUI code was easy.

Testing by code structure

Once you've tested the basic functionality of an application, it becomes important to consider alternate pathways through the code. In most languages, you can break up your tests as follows:

Write one test for every package or module.
Once every package has at least one test, write one test for every class.
Once every class has at least one test, write one test for every method.
Once every method has at least one test, use a code coverage tool, such as Cobertura, to write one test for every branch until each line of code is tested.

You could use a code coverage tool before Step 4, but I prefer to do the preceding steps manually. Although many classes, packages, and methods are tested by the functionality tests, you often find different problems when looking at the program from a programmer's view rather than a user's view.

In fact, most of the time, you're never going to get to Step 4. You simply won't have the time or the budget to write every possible test. That's OK: the difference between some tests and no tests is much larger and more significant than the difference between all tests and some tests.

Automated testing

It is possible to use reflection to generate a test skeleton. This makes it easier to find all the public methods you need to test. Each test starts life looking like this:

public void testMethodName() {
   fail("Test Code Not Written Yet");
}

The downside to this approach is that you'll immediately have hundreds to thousands of failing tests. An alternative is to add a TODO comment in each test rather than an outright failure. Then as time permits, you can go through and fill in tests.

public void testMethodName() {
   // TODO fill in test code
}

If you're using JUnit 4, you could simply annotate the tests as @Ignore until you fill them in, like so:

@Ignore public void testMethodName() {
   // TODO fill in test code
}

In this case, the test runner reminds you that it skipped the tests, but it won't actually fail you. In essence, it grades you Pass-Fail-Incomplete.

You can find a few tools that automatically write tests for you. However, the tests such tools come up with tend to be fairly trivial and basic stuff, like whether a method can handle being passed a null argument. Such tools don't really understand what each class or method is supposed to be doing. For that, you need human intelligence.

Handling dead code

One thing that's likely to surprise you at this stage is how much of your code you don't really need. Legacy code bases tend to have lots of vestigial code; code that was necessary at one point in time but is no longer. The older and crustier the legacy code is, the more vestigial code you'll find. Sometimes the code is very obviously unreachable (uncalled private methods, unread local variables, and so forth). This sort of vestigial code can also be found by static code analysis tools like PMD and FindBugs. Sometimes the code isn't so obviously vestigial, and only the attempt to reach it for testing can reveal just how vestigial it really is.

However you find such vestigial code, or whyever it was put there in the first place, take it out. The less code you have to maintain, the better.

Exploratory testing

Going into a legacy system, you'll often have a good idea of where you need to look: You're having problems with a particular module or package or set of circumstances, and that problem drives your testing. In this case, by all means focus your tests on that one area.

Sometimes you have a very clear and obvious bug. Before you start fixing it, write a test. Then run the test to make sure the test fails. Surprisingly often, however, your first instinct about the cause of a bug is not correct, in which case the test passes. Whether a test fails or not, don't throw it away. It's still valuable for future development. Leave it in your test suite and write another test. Repeat until you find a test that does fail and thus discover the true cause of the bug.

In conclusion

Don't let perfection be the enemy of the good. Even if you have a large, untested legacy code base, start writing tests for it now. Don't worry about getting to 100 percent coverage. Every test you write will increase your confidence in the code, squeeze out bugs, and provide more flexibility for future development. Need to add a feature? Write a test. Find a bug? Write another test. Legacy programmers can be agile too.

Resources

Learn

"Demystifying Extreme Programming: Test-driven programming" (Roy Miller, developerWorks, April 2003): Explains what test-driven programming is all about, and more importantly, what it isn't about.
"TestNG makes Java unit testing a breeze" (Filippo Diotalevi, developerWorks, January 2005): TestNG is less focused on unit testing and test-first programming than JUnit is, making it in some ways a superior product for testing legacy code.
"Keeping critters out of your code" (David Carew, Sandeep Desai, Anthony Young-Garner; developerWorks, June 2003): Introduces unit testing a server-side application server environment.
"Measure test coverage with Cobertura" (Elliotte Rusty Harold, developerWorks, May 2005): Shows how to use an open-source tool to identify untested code.
"An early look at JUnit 4" (Elliotte Rusty Harold, developerWorks, September 2005): Introduces the new annotation-based architecture of JUnit 4. This requires Java 5 or later.
Pragmatic Unit Testing in Java (Dave Thomas and Andy Hunt, Pragmatic Programmers, September 2003): Read Dave Thomas and Andy Hunt's book.
The Java technology zone: Hundreds of articles about every aspect of Java programming.