Verifying application performance is almost always a secondary concern during application development. Note that I'm stressing the verification of application performance. An application's performance is always a chief concern, but verification is rarely part of the development cycle.

Performance testing is usually put off until later cycles for a variety of reasons. In my experience, businesses don't include performance testing in the development process because they don't know what to expect from the application in progress. Numbers are thrown out but they're guesses based on expected load.

Performance testing usually becomes a primary concern after one of two things happens:

• A noticeable performance problem surfaces in production.
• A client or prospective customer asks about performance numbers before agreeing to spend money.

This month, I'll introduce two easy techniques for testing performance before either of these issues crops up.

Testing with JUnitPerf

It is easy to ascertain basic low-level performance numbers using JUnit in the early stages of software development. The JUnitPerf framework enables you to quickly turn tests into simple load tests and even stress tests.

You can create two types of tests using JUnitPerf: TimedTests and LoadTests. Both are based on the Decorator design pattern and utilize JUnit's suite mechanism. TimedTests create a top-level bound for a test case -- if this time is surpassed, then the test fails. LoadTests work in cooperation with timers and create an artificial load on a particular test case by running it a desired number of times separated by the configured timer.

Back to top

Felicitously timed tests

JUnitPerf TimedTests enable you to write tests with an associated time threshold -- if the threshold is exceeded, the test is considered a failure (even if the test logic itself is actually successful). Timed tests are helpful in ascertaining and monitoring performance numbers for business-critical methods, among other things. You can even make tests a bit more granular and test a series of methods to ensure they meet a certain time threshold.

For example, imagine a Widget application where certain business-critical methods, such as the aptly named createWidget(), are the target of stringent performance thresholds. Presumably, you will need to performance test the functional aspect of executing the create() method. This is typically ascertained in later development cycles by different teams using different tools, which are usually unable to pinpoint precise methods. But let's say you've decided to try the test early and often methodology instead.

Creating a TimedTest starts with a normal JUnit test. In other words, you will extend TestCase or some derivative of it and write a method starting with test, as shown in Listing 1:

public class WidgetDAOImplTest extends TestCase {	
 private WidgetDAO dao;

 public void testCreate() throws Exception{
  IWidget wdgt = new Widget();
  wdgt.setWidgetId(1000);
  wdgt.setPartNumber("12-34-BBD");  
  try{
   this.dao.createWidget(wdgt);
  }catch(CreateException e){
   TestCase.fail("CreateException thrown creating a Widget");
  }	    
 }

 protected void setUp() throws Exception {       
  ApplicationContext context = 
   new ClassPathXmlApplicationContext("spring-config.xml");      
  this.dao = (WidgetDAO) context.getBean("widgetDAO");      
 }   	
}

Because JUnitPerf is a decorator-based framework, in order to actually harness it, you must provide a suite() method and decorate your existing test with a TimedTest. A TimedTest takes as parameters a Test and a maximum amount of time for the test to execute.

You also have the option to pass in a boolean flag as the third argument (false), which will cause the test to fail fast -- meaning if the maximum time is exceeded, JUnitPerf will force a failure immediately. Otherwise, the test case is run to completion and then it is failed. The difference is subtle: running the test without the optional flag helps you understand the total time, even in the case of a failure. Passing in false, however, means you won't be provided a total run time.

For example, in Listing 2, I've run testCreate() with a two second ceiling. If during execution the total time exceeds this time, the test case will fail. Because I didn't pass in the optional boolean parameter, the test will run to completion, however long that takes.

public static Test suite() {
 long maxElapsedTime = 2000; //2 seconds 
 Test timedTest = new TimedTest(
   new WidgetDAOImplTest("testCreate"), maxElapsedTime);
 return timedTest;    			
}

This test is run normally within the JUnit framework -- your existing Ant tasks, Eclipse runners, and so on will run this test as they would any other JUnit test. The only difference is that it will happen in the context of a timer.

Back to top

A surfeit of load tests

As opposed to verifying a time threshold on a method (or series of methods) in a test scenario, JUnitPerf also facilitates load testing. Just like with TimedTests, JUnitPerf's LoadTests work as decorators by wrapping a JUnit Test with additional threading information to simulate load.

With a LoadTest, you can specify a number of users to simulate (that is, threads) and even provide a timing mechanism for the launching of threads. Two types of Timer are provided by JUnitPerf: a ConstantTimer and a RandomTimer. By providing these to a LoadTest, you can more realistically simulate user load. Without a Timer, all threads are launched simultaneously.

Listing 3 is a load test for 10 simulated users, implemented with a ConstantTimer:

public static Test suite() {
 int users = 10;
 Timer timer = new ConstantTimer(100);		
 return new LoadTest(
  new WidgetDAOImplTest("testCreate"), 
    users, timer);		
}

Note that the testCreate() method is run 10 times with each thread launched 100 milliseconds apart. No threshold is set -- these methods are simply run to completion and if any fails to execute, JUnit will accordingly report a failure.

Back to top

Decorate with style

Decorators aren't limited to a single decoration. For example, in Java™ I/O, it is possible to decorate a FileInputStream with an InputStreamReader with a BufferedReader (just remember this: BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream("infilename"), "UTF8"))).

Decoration can happen on multiple levels, and so it is with JUnitPerf's TimedTests and LoadTests. When these two classes decorate each other, it leads to some compelling test scenarios, such as one where a load is placed on a business case and a time threshold is also applied. Or we could just combine the previous two test scenarios as follows:

• Place a load on the testCreate() method.
• Specify that every thread must finish within the time threshold.

Listing 4 shows what happens when I apply the above specifications by decorating a normal Test with a LoadTest, which is decorated by a TimedTest:

public static Test suite() {
 int users = 10;
 Timer timer = new ConstantTimer(100);
 long maxElapsedTime = 2000; 	 
 return new TimedTest(new LoadTest(
   new WidgetDAOImplTest("testCreate"), users, timer), 
     maxElapsedTime);  		
}

As you see, the testCreate() method is run 10 times (each thread launching 100 milliseconds apart), and each thread must complete within two seconds or the entire test scenario will fail.

Back to top

Caveat emptor

While JUnitPerf is a performance testing framework, the numbers you place around it should be considered rough estimates. Because all tests decorated by JUnitPerf are run through the JUnit framework, there is an added overhead, especially if you utilize fixtures. Because JUnit itself decorates all test cases with a setUp and a tearDown() method, execution times must be considered within the overall context of a test scenario.

Accordingly, I often create tests that utilize my desired fixture logic but also run a blank test case to determine the baseline number. This is a rough estimate, but it serves as a baseline that must be added to any desired threshold for testing.

For example, if running a blank test decorated by fixture logic that uses DbUnit takes 2.5 seconds, then all desired JUnitPerf test thresholds should consider this additional time -- which can be learned from a benchmark test like the one shown in Listing 5:

public class DBUnitSetUpBenchmarkTest extends DatabaseTestCase {
 private WidgetDAO dao = null;

 public void testNothing(){
  //should be about 2.5 seconds
 }

 protected IDatabaseConnection getConnection() throws Exception {        
  Class driverClass = Class.forName("org.hsqldb.jdbcDriver");
  Connection jdbcConnection = 
   DriverManager.getConnection(
     "jdbc:hsqldb:hsql://127.0.0.1", "sa", "");
  return new DatabaseConnection(jdbcConnection);
 }

 protected IDataSet getDataSet() throws Exception {        
  return new FlatXmlDataSet(new File("test/conf/seed.xml"));
 }

 protected void setUp() throws Exception {
  super.setUp();      
  final ApplicationContext context = 
   new ClassPathXmlApplicationContext("spring-config.xml");      
  this.dao = (WidgetDAO) context.getBean("widgetDAO");      
 }
}

Notice that the Listing 5 test case testNothing() does nothing. Its only purpose is to ascertain the total time to run the setUp() method (which of course also sets up a database via DbUnit).

Also remember that test times will vary depending on machine configurations and what's running at a given point in time during JUnitPerf test execution. I often find that placing JUnitPerf tests in their own category helps segregate them from normal tests. This means they aren't run every time during a test run, such as in a code check-in within a CI environment. I also end up creating specific Ant tasks to run these tests only during choreographed scenarios or environments where performance testing is taken into account.

Back to top

Just try it!

Performance testing with JUnitPerf is by no means an exact science, but it is an excellent way to ascertain and monitor low-level performance of application code early in a development life cycle. Plus, because it is a decorator-based JUnit extension framework, you can easily use JUnitPerf to decorate existing JUnit tests.

Just think about all the time you've spent worrying about how your application will perform under load. Performance testing with JUnitPerf is one way to carve out time for the better things in your life, while also ensuring the quality of your application code.

Resources

Learn

• "Java theory and practice: Decorating with dynamic proxies" (Brian Goetz, developerWorks, August 2005): Dynamic proxies are a convenient tool for building decorators and adapters.
  
  
• "Repeatable system tests" (Andrew Glover, developerWorks, September 2006): Andrew Glover introduces Cargo, an open source framework that automates container management in a generic fashion.
  
  
• "Continuous Performance Testing With JUnitPerf" (Mike Clark, devnewz, July 2003): Mike Clark shows you how to tune with more confidence and less stress by writing automated performance tests that tell no lies.
  
  
• "Effective Unit Testing with DbUnit" (Andrew Glover, OnJava, January 2004): Introduces database-dependent testing with DbUnit.
  
  
• Groovy performance testing (thediscoblog.com): Learn how to utilize JUnitPerf with your tests written in Groovy.
  
  
• In pursuit of code quality series (Andrew Glover, developerWorks): Learn more about code metrics, test frameworks, and writing quality-focused code.
  
  
• The Java technology zone: Hundreds of articles about every aspect of Java programming.
  
  

Get products and technologies

• Download JUnit: Find out what's new with JUnit 4.
  
  
• Download JUnitPerf: A collection of JUnit test decorators used to measure performance and scalability.
  
  

Discuss

• Discussion forum: Improve your code quality: Andrew shares his expertise as a consultant focused on improving code quality.
  
  

About the author

Andrew Glover is president of Stelligent Incorporated, which helps companies address software quality with effective developer testing strategies and continuous integration techniques that enable teams to monitor code quality early and often. Check out Andy's blog for a list of his publications.

Rate this article

Comments

Back to top

Table of contents

• Testing with JUnitPerf
• Felicitously timed tests
• A surfeit of load tests
• Decorate with style
• Caveat emptor
• Just try it!
• Resources
• About the author
• Comments