By their very nature, test frameworks like JUnit and TestNG facilitate the creation of repeatable tests. Because these frameworks leverage the reliability of simple Boolean logic (in the form of assert methods), it's possible to run tests without human intervention. In fact, automation is one of the primary benefits of test frameworks -- I can write a fairly complex test asserting specific behaviors, and if those behaviors ever change, the framework reports an error that anyone can interpret.

Utilizing a mature test framework gives you the benefit of framework repeatability, right out of the box. But logical repeatability is up to you. For example, consider the challenge of creating repeatable tests that verify Web applications. A few JUnit extension frameworks (such as JWebUnit and HttpUnit) excel at facilitating automated Web testing. But it's the developer's job to make the plumbing of the test repeatable, and that's hard to do when it comes to deploying Web application resources.

The actual construction of a JWebUnit test is fairly simple, as demonstrated by Listing 1:

package test.come.acme.widget.Web;

import net.sourceforge.jwebunit.WebTester;
import junit.framework.TestCase;

public class WidgetCreationTest extends TestCase {
 private WebTester tester;

 protected void setUp() throws Exception {
  this.tester = new WebTester();
  this.tester.getTestContext().
   setBaseUrl("http://localhost:8080/widget/");	
 }

 public void testWidgetCreation() {
  this.tester.beginAt("/CreateWidget.html");		
  this.tester.setFormElement("widget-id", "893-44");
  this.tester.setFormElement("part-num", "rt45-3");

  this.tester.submit();		
  this.tester.assertTextPresent("893-44");
  this.tester.assertTextPresent("successfully created.");
 }
}  

This test communicates with a Web application and attempts to create a widget based on the interaction. The test then verifies that the widget was successfully created. Readers of previous articles in this series may notice a subtle repeatability issue with the test, however. Do you see it? What do you think will happen if this test case is run twice consecutively?

Judging by the identification aspect of the widget instance (that is, widget-id), it's safe to assume that database constraints in the application will likely prohibit the creation of an additional widget that already exists. Without a process for removing the test case's intended widget before another test is run, there's a high probability that the test case will fail if run twice in a row.

Fortunately, as I've discussed in a previous article, there is a mechanism that facilitates database-dependent test case repeatability -- namely DbUnit.

DbUnit goes to work

Enhancing the test case from Listing 1 to employ DbUnit is fairly simple. All DbUnit needs is some data to insert into a database and a corresponding database connection, as shown in Listing 2:

package test.come.acme.widget.Web;

import java.io.File;
import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;

import org.dbunit.database.DatabaseConnection;
import org.dbunit.database.IDatabaseConnection;
import org.dbunit.dataset.DataSetException;
import org.dbunit.dataset.IDataSet;
import org.dbunit.dataset.xml.FlatXmlDataSet;
import org.dbunit.operation.DatabaseOperation;

import net.sourceforge.jwebunit.WebTester;
import junit.framework.TestCase;

public class RepeatableWidgetCreationTest extends TestCase {
 private WebTester tester;

 protected void setUp() throws Exception {
  this.handleSetUpOperation();
  this.tester = new WebTester();
  this.tester.getTestContext().
   setBaseUrl("http://localhost:8080/widget/");	
 }

 public void testWidgetCreation() {
  this.tester.beginAt("/CreateWord.html");		
  this.tester.setFormElement("widget-id", "893-44");
  this.tester.setFormElement("part-num", "rt45-3");

  this.tester.submit();		
  this.tester.assertTextPresent("893-44");
  this.tester.assertTextPresent("successfully created.");
 }

 private void handleSetUpOperation() throws Exception{
  final IDatabaseConnection conn = this.getConnection();
  final IDataSet data = this.getDataSet();        
  try{
   DatabaseOperation.CLEAN_INSERT.execute(conn, data);
  }finally{
   conn.close();
  }
 }

 private IDataSet getDataSet() throws IOException, DataSetException {
  return new FlatXmlDataSet(new File("test/conf/seed.xml"));
 }

 private IDatabaseConnection getConnection() throws 
   ClassNotFoundException, SQLException {
    Class.forName("org.hsqldb.jdbcDriver");
    final Connection jdbcConnection = 
     DriverManager.getConnection("jdbc:hsqldb:hsql://127.0.0.1", 
	  "sa", "");
    return new DatabaseConnection(jdbcConnection);
 }
}

With the addition of DbUnit, the test case is truly repeatable. In the handleSetUpOperation() method, DbUnit executes a CLEAN_INSERT on the data every time a test case is run. This operation essentially cleanses a database of data and inserts a fresh set, thereby removing any previously created widgets.

But that doesn't mean I'm done with the topic of test case repeatability. In fact, I'm just getting started.

Repeating system tests

The test cases defined in Listing 1 and Listing 2 are what I like to call system tests. Because system tests exercise a fully installed application, such as a Web application, they usually incorporate a servlet container and an associated database. These tests tend to verify that external interfaces (such as Web pages in the case of a Web application) operate end-to-end as designed.

Because they're designed to test fully functional applications, system tests tend toward lengthy run times, not the least of which involves the total time to set up a test. For example, the logical test shown in Listing 1 and Listing 2 requires the following steps before it is even run:

1. Create a war file containing all associated Web artifacts, such as JSP files, servlets, third-party jar files, images, etc.
   
   
2. Deploy the war file to the targeted Web container. (If the container isn't started, start it.)
   
   
3. Start any associated databases. (If a database needs its schema updated, update that before starting.)

Now, that's a lot of leg work for one measly test! If this process proves time-consuming, how often do you think the test will be run? Faced with the requirement to make system tests logically repeatable (say, in a continuous integration environment) this list of steps is certifiably daunting.

Back to top

Introducing Cargo

The good news is that you can automate all the major setup steps in the previous list. In fact, if you happen to be doing Java Web development, you're probably already automating Step 1 with Ant, Maven, or some other build tool.

Step 2 presents an interesting hurdle, however. Automating a Web container can be a bit tricky. For instance, some containers have custom Ant tasks that facilitate their auto-deployment and running, but the tasks are container specific. What's more, these tasks make some assumptions, such as about where the container is installed and, more importantly, that the container is installed.

Cargo is an innovative open source project that aims to automate container management in a generic fashion, such that the same API used to deploy a WAR file to JBoss can also start and stop Jetty. Cargo also can download and install a container automatically. You can utilize Cargo's API in a few different fashions, ranging from Java code to Ant tasks to Maven goals.

Using a tool like Cargo takes care of one of the major challenges to writing logically repeatable test cases. Furthermore, you can construct a build that harnesses the power of Cargo to automatically do the following:

1. Download a desired container.
2. Install the container.
3. Start the container.
4. Deploy a selected WAR or EAR file to the container.

Pretty easy, huh? At a later point, you can also have Cargo stop a selected container.

'Speaking' Cargo

It's a good idea to get the basics of Cargo down before diving in head first. Namely, you want to be sure you understand the concept of containers and container management as Cargo relates to them.

For starters, there is obviously the notion of a container. Containers are servers that host applications. Those applications can be Web-based, EJB-based, or both, which is why we have Web containers and EJB containers. Tomcat is a Web container and JBoss could be considered an EJB container. Consequently, Cargo supports quite a few containers, but for my example, I'll use Tomcat version 5.0.28. (Cargo would call this a "tomcat5x" container.)

Next, if a container isn't already installed, you can use Cargo to download and install a particular one. To do this, you need to supply Cargo with a download URL. Once you've installed a container, Cargo also allows you to use configuration options to configure it. These options take the form of name-value pairs.

Lastly, there is the notion of a deployable resource, which in the case of my example is a WAR file. Note that it could just as easily be an EAR file.

Keep these concepts in mind and let's see what we can do with Cargo.

Back to top

Cargo in action

The example for this article involves using Cargo in Ant, which entails wrapping the system test I defined earlier with Cargo Ant tasks. These tasks then install, start, deploy, and stop the container. We'll do the setup first, run the test, and then stop the container.

The first step required to use Cargo in an Ant build is to provide a task definition for all the Cargo tasks. This step allows you to reference Cargo's tasks later in your build file. There are a number of ways to handle this step. Listing 3 simply loads the tasks from a properties file found in Cargo's JAR files:

<taskdef resource="cargo.tasks">
 <classpath>
  <pathelement location="${libdir}/${cargo-jar}"/>
  <pathelement location="${libdir}/${cargo-ant-jar}"/>
 </classpath>
</taskdef>	

Once you've defined Cargo's tasks, the real action begins. Listing 4 defines Cargo tasks to download, install, and start a Tomcat container. The zipurlinstaller task downloads and installs Tomcat from http://www.apache.org/dist/tomcat/tomcat-5/v5.0.28/bin/ jakarta-tomcat-5.0.28.zip to a local temporary directory.

<cargo containerId="tomcat5x" action="start" 
       wait="false" id="${tomcat-refid}">
	   
 <zipurlinstaller installurl="${tomcat-installer-url}"/>
 
 <configuration type="standalone" home="${tomcatdir}">
  <property name="cargo.remote.username" value="admin"/>
  <property name="cargo.remote.password" value=""/>

  <deployable type="war" file="${wardir}/${warfile}"/>

 </configuration>		
 
</cargo>

Note that to start and stop a container from within different tasks, as you'd like to do, you must associate the container with a unique id, which is the id="${tomcat-refid}" aspect of the cargo task.

Also note that configuration aspects of Tomcat are handled inside the cargo task. In Tomcat's case, you must set the username and password properties. Finally, you define a pointer to a WAR file using the deployable element.

Properties of Cargo

All the properties used in your Cargo tasks are shown in Listing 5. For example, tomcatdir defines one of two locations where Tomcat will be installed. This particular location is a mirrored structure, which will be referenced by the actual downloaded and installed Tomcat instance (which is found in a temporary directory). The property tomcat-refid helps associate a unique instance of a container as well.

<property name="tomcat-installer-url" 
  value="http://www.apache.org/dist/tomcat/tomcat-5/v5.0.28/bin/
    jakarta-tomcat-5.0.28.zip"/>
<property name="tomcatdir" value="target/tomcat"/>	
<property name="tomcat.username" value="admin"/>	
<property name="tomcat.passwrd" value=""/>	
<property name="wardir" value="target/war"/>
<property name="warfile" value="words.war"/>	
<property name="tomcat-refid" value="tmptmct01"/>	

To stop a container, you can define a task that references the tomcat-refid property, as shown in Listing 6:

<cargo containerId="tomcat5x" action="stop" 
       refid="${tomcat-refid}"/>	

Wrapping with Cargo

Listing 7 combines the code in Listing 4 and Listing 6, wrapping a test target with two Cargo tasks: one for starting and one for stopping Tomcat. The task antcall calls the target named _run-system-tests, which is defined in Listing 8.

<target name="system-test" if="Junit.present" 
        depends="init,junit-present,compile-tests,war">

 <cargo containerId="tomcat5x" action="start" 
       wait="false" id="${tomcat-refid}">			
  <zipurlinstaller installurl="${tomcat-installer-url}"/>
  <configuration type="standalone" home="${tomcatdir}">
   <property name="cargo.remote.username" value="admin"/>
   <property name="cargo.remote.password" value=""/>
   <deployable type="war" file="${wardir}/${warfile}"/>
  </configuration>
 </cargo>

 <antcall target="_run-system-tests"/>

 <cargo containerId="tomcat5x" action="stop" 
       refid="${tomcat-refid}"/>			

</target>	

Listing 8 defines your test target, called _run-system-tests. Note that this task only runs system tests, which are placed in the test/system directory. For example, the test case defined in Listing 2 lives in this directory.

<target name="_run-system-tests">
 <mkdir dir="${testreportdir}"/>   
 <junit dir="./" failureproperty="test.failure" 
        printSummary="yes" fork="true" 
		haltonerror="true">
  <sysproperty key="basedir" value="."/>     
  <formatter type="xml"/>      
  <formatter usefile="false" type="plain"/>     
  <classpath>
   <path refid="build.classpath"/>       
   <pathelement path="${testclassesdir}"/>
   <pathelement path="${classesdir}"/>      
  </classpath>
  <batchtest todir="${testreportdir}">
   <fileset dir="test/system">
    <include name="**/**Test.java"/> 
   </fileset>
  </batchtest>
 </junit>
</target>	

In Listing 7, you fully configured your Ant build file to wrap your system tests with Cargo's deployment voodoo. The code in Listing 7 ensures that all the system tests in the test/system directory from Listing 8 are logically repeatable. You can run these system tests on any machine at any time, which is perfect for a Continuous Integration environment. The tests make no assumptions about the container -- not about its location or even if it's running! (Of course, these tests still make one assumption I haven't addressed, namely that the underlying database is properly configured and running. But that's a subject for another day.)

Back to top

Repeatable results

In Listing 9, you can see the results of your efforts. When you issue the system-test command to an Ant build, your system tests are executed. Cargo handles all the details of managing the container of your choice, without making repeatability-crushing assumptions about the test environment.

war:
 [war] Building war: C:\dev\projects\acme\target\widget.war

system-test:

_run-system-tests:
 [mkdir] Created dir: C:\dev\projects\acme\target\test-reports
 [junit] Running test.come.acme.widget.Web.RepeatableWordCreationTest
 [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.53 sec
 [junit] Testcase: testWordCreation took 4.436 sec

BUILD SUCCESSFUL
Total time: 1 minute 2 seconds

Keep in mind that Cargo also works in Maven builds. Furthermore, the Cargo Java API facilitates programmatic management of containers in anything from normal applications to test cases. And Cargo isn't only for JUnit (even though the code examples are written in JUnit). TestNG users will be happy to know that Cargo works just as well for their test suites. In fact, it doesn't matter what your tests are written in: just wrap them with Cargo and your container management issues will be automated out of the box!

Back to top

In conclusion

It's up to you to ensure that your tests are logically repeatable, but you've seen this month that Cargo can really help. Cargo manages the container environment so you don't have to. Incorporate Cargo into your testing routine -- it will definitely lighten the load of crafting repeatable tests that verify Web applications.

Resources

Learn

• "Effective Unit Testing with DbUnit" (Andrew Glover, OnJava, January 2004): Introduces database-dependent testing with DbUnit.
  
  
• Hip system tests with Cargo (Andrew Glover, thediscoblog.com, April 2006): Learn more about Cargo's Java API.
  
  
• "Make Ant easy with Eclipse" (Prashant Deva, developerWorks, April 2006): Discover the Ant integration features in the Eclipse IDE.
  
  
• "Practically Groovy: Ant scripting with Groovy" (Andrew Glover, developerWorks, December 2004): Combine Groovy with Ant and Maven for more expressive and controllable builds.
  
  
• In pursuit of code quality: JUnit 4 vs. TestNG (Andrew Glover, developerWorks, April 2006): Has JUnit 4 rendered TestNG obsolete? Find out why not.
  
  
• In pursuit of code quality series (Andrew Glover, developerWorks): Learn more about code metrics, test frameworks, and writing quality-focused code.
  
  
• developerWorks: Hundreds of articles about every aspect of Java programming.
  
  

Get products and technologies

• Download Cargo: Make repeatable Web tests easier.
  
  
• Download JUnit: Find out what's new with JUnit 4.
  
  
• Download TestNG: Another powerful testing framework.
  
  

Discuss

• Participate in the discussion forum.
  
  

About the author

Andrew Glover is president of Stelligent Incorporated, which helps companies address software quality with effective developer testing strategies and continuous integration techniques that enable teams to monitor code quality early and often. Check out Andy's blog for a list of his publications.

Rate this article

Comments

Back to top

Table of contents

• DbUnit goes to work
• Repeating system tests
• Introducing Cargo
• Cargo in action
• Repeatable results
• In conclusion
• Resources
• About the author
• Comments