In pursuit of code quality: Repeatable system tests

Automate container management with Cargo

Writing logically repeatable tests is especially tricky when testing Web applications that incorporate a servlet container. In his continued quest to improve code quality, Andrew Glover introduces Cargo, an open source framework that automates container management in a generic fashion, so you can write logically repeatable system tests every time.

Share:

Andrew Glover, President, Stelligent Incorporated

Andrew GloverAndrew Glover is president of Stelligent Incorporated, which helps companies address software quality with effective developer testing strategies and continuous integration techniques that enable teams to monitor code quality early and often. Check out Andy's blog for a list of his publications.



26 September 2006

Also available in Chinese Russian Japanese

By their very nature, test frameworks like JUnit and TestNG facilitate the creation of repeatable tests. Because these frameworks leverage the reliability of simple Boolean logic (in the form of assert methods), it's possible to run tests without human intervention. In fact, automation is one of the primary benefits of test frameworks -- I can write a fairly complex test asserting specific behaviors, and if those behaviors ever change, the framework reports an error that anyone can interpret.

Utilizing a mature test framework gives you the benefit of framework repeatability, right out of the box. But logical repeatability is up to you. For example, consider the challenge of creating repeatable tests that verify Web applications. A few JUnit extension frameworks (such as JWebUnit and HttpUnit) excel at facilitating automated Web testing. But it's the developer's job to make the plumbing of the test repeatable, and that's hard to do when it comes to deploying Web application resources.

The actual construction of a JWebUnit test is fairly simple, as demonstrated by Listing 1:

Listing 1. A simple JWebUnit test
package test.come.acme.widget.Web;

import net.sourceforge.jwebunit.WebTester;
import junit.framework.TestCase;

public class WidgetCreationTest extends TestCase {
 private WebTester tester;

 protected void setUp() throws Exception {
  this.tester = new WebTester();
  this.tester.getTestContext().
   setBaseUrl("http://localhost:8080/widget/");	
 }

 public void testWidgetCreation() {
  this.tester.beginAt("/CreateWidget.html");		
  this.tester.setFormElement("widget-id", "893-44");
  this.tester.setFormElement("part-num", "rt45-3");

  this.tester.submit();		
  this.tester.assertTextPresent("893-44");
  this.tester.assertTextPresent("successfully created.");
 }
}

This test communicates with a Web application and attempts to create a widget based on the interaction. The test then verifies that the widget was successfully created. Readers of previous articles in this series may notice a subtle repeatability issue with the test, however. Do you see it? What do you think will happen if this test case is run twice consecutively?

Improve your code quality

Don't miss Andrew's accompanying discussion forum for assistance with code metrics, test frameworks, and writing quality-focused code.

Judging by the identification aspect of the widget instance (that is, widget-id), it's safe to assume that database constraints in the application will likely prohibit the creation of an additional widget that already exists. Without a process for removing the test case's intended widget before another test is run, there's a high probability that the test case will fail if run twice in a row.

Fortunately, as I've discussed in a previous article, there is a mechanism that facilitates database-dependent test case repeatability -- namely DbUnit.

DbUnit goes to work

Enhancing the test case from Listing 1 to employ DbUnit is fairly simple. All DbUnit needs is some data to insert into a database and a corresponding database connection, as shown in Listing 2:

Listing 2. Database-dependent testing with DbUnit
package test.come.acme.widget.Web;

import java.io.File;
import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;

import org.dbunit.database.DatabaseConnection;
import org.dbunit.database.IDatabaseConnection;
import org.dbunit.dataset.DataSetException;
import org.dbunit.dataset.IDataSet;
import org.dbunit.dataset.xml.FlatXmlDataSet;
import org.dbunit.operation.DatabaseOperation;

import net.sourceforge.jwebunit.WebTester;
import junit.framework.TestCase;

public class RepeatableWidgetCreationTest extends TestCase {
 private WebTester tester;

 protected void setUp() throws Exception {
  this.handleSetUpOperation();
  this.tester = new WebTester();
  this.tester.getTestContext().
   setBaseUrl("http://localhost:8080/widget/");	
 }

 public void testWidgetCreation() {
  this.tester.beginAt("/CreateWord.html");		
  this.tester.setFormElement("widget-id", "893-44");
  this.tester.setFormElement("part-num", "rt45-3");

  this.tester.submit();		
  this.tester.assertTextPresent("893-44");
  this.tester.assertTextPresent("successfully created.");
 }

 private void handleSetUpOperation() throws Exception{
  final IDatabaseConnection conn = this.getConnection();
  final IDataSet data = this.getDataSet();        
  try{
   DatabaseOperation.CLEAN_INSERT.execute(conn, data);
  }finally{
   conn.close();
  }
 }

 private IDataSet getDataSet() throws IOException, DataSetException {
  return new FlatXmlDataSet(new File("test/conf/seed.xml"));
 }

 private IDatabaseConnection getConnection() throws 
   ClassNotFoundException, SQLException {
    Class.forName("org.hsqldb.jdbcDriver");
    final Connection jdbcConnection = 
     DriverManager.getConnection("jdbc:hsqldb:hsql://127.0.0.1", 
	  "sa", "");
    return new DatabaseConnection(jdbcConnection);
 }
}

With the addition of DbUnit, the test case is truly repeatable. In the handleSetUpOperation() method, DbUnit executes a CLEAN_INSERT on the data every time a test case is run. This operation essentially cleanses a database of data and inserts a fresh set, thereby removing any previously created widgets.

DbUnit is what again?

DbUnit is a JUnit extension that facilitates putting a database into a known state between test runs. Developers use XML seed files to insert specific data into a database, which a test case can then rely on. Thus, DbUnit facilitates the repeatability of test cases that rely on one or more databases.

But that doesn't mean I'm done with the topic of test case repeatability. In fact, I'm just getting started.

Repeating system tests

The test cases defined in Listing 1 and Listing 2 are what I like to call system tests. Because system tests exercise a fully installed application, such as a Web application, they usually incorporate a servlet container and an associated database. These tests tend to verify that external interfaces (such as Web pages in the case of a Web application) operate end-to-end as designed.

Prioritizing flexibility

As a general rule of thumb, avoid test case inheritance whenever possible. Many JUnit extension frameworks offer specialized test cases that can be inherited from to facilitate testing a particular architecture. Test cases that inherit classes from a framework suffer from inflexibility, however, because of the Java™ platform's single-inherence paradigm. More often than not, these same JUnit extension frameworks offer a delegation API, which makes it easy to combine various frameworks without taking on a rigid inheritance structure.

Because they're designed to test fully functional applications, system tests tend toward lengthy run times, not the least of which involves the total time to set up a test. For example, the logical test shown in Listing 1 and Listing 2 requires the following steps before it is even run:

  1. Create a war file containing all associated Web artifacts, such as JSP files, servlets, third-party jar files, images, etc.
  2. Deploy the war file to the targeted Web container. (If the container isn't started, start it.)
  3. Start any associated databases. (If a database needs its schema updated, update that before starting.)

Now, that's a lot of leg work for one measly test! If this process proves time-consuming, how often do you think the test will be run? Faced with the requirement to make system tests logically repeatable (say, in a continuous integration environment) this list of steps is certifiably daunting.


Introducing Cargo

The good news is that you can automate all the major setup steps in the previous list. In fact, if you happen to be doing Java Web development, you're probably already automating Step 1 with Ant, Maven, or some other build tool.

Step 2 presents an interesting hurdle, however. Automating a Web container can be a bit tricky. For instance, some containers have custom Ant tasks that facilitate their auto-deployment and running, but the tasks are container specific. What's more, these tasks make some assumptions, such as about where the container is installed and, more importantly, that the container is installed.

Cargo is an innovative open source project that aims to automate container management in a generic fashion, such that the same API used to deploy a WAR file to JBoss can also start and stop Jetty. Cargo also can download and install a container automatically. You can utilize Cargo's API in a few different fashions, ranging from Java code to Ant tasks to Maven goals.

Using a tool like Cargo takes care of one of the major challenges to writing logically repeatable test cases. Furthermore, you can construct a build that harnesses the power of Cargo to automatically do the following:

  1. Download a desired container.
  2. Install the container.
  3. Start the container.
  4. Deploy a selected WAR or EAR file to the container.

Pretty easy, huh? At a later point, you can also have Cargo stop a selected container.

'Speaking' Cargo

It's a good idea to get the basics of Cargo down before diving in head first. Namely, you want to be sure you understand the concept of containers and container management as Cargo relates to them.

For starters, there is obviously the notion of a container. Containers are servers that host applications. Those applications can be Web-based, EJB-based, or both, which is why we have Web containers and EJB containers. Tomcat is a Web container and JBoss could be considered an EJB container. Consequently, Cargo supports quite a few containers, but for my example, I'll use Tomcat version 5.0.28. (Cargo would call this a "tomcat5x" container.)

Next, if a container isn't already installed, you can use Cargo to download and install a particular one. To do this, you need to supply Cargo with a download URL. Once you've installed a container, Cargo also allows you to use configuration options to configure it. These options take the form of name-value pairs.

Lastly, there is the notion of a deployable resource, which in the case of my example is a WAR file. Note that it could just as easily be an EAR file.

Keep these concepts in mind and let's see what we can do with Cargo.


Cargo in action

The example for this article involves using Cargo in Ant, which entails wrapping the system test I defined earlier with Cargo Ant tasks. These tasks then install, start, deploy, and stop the container. We'll do the setup first, run the test, and then stop the container.

The first step required to use Cargo in an Ant build is to provide a task definition for all the Cargo tasks. This step allows you to reference Cargo's tasks later in your build file. There are a number of ways to handle this step. Listing 3 simply loads the tasks from a properties file found in Cargo's JAR files:

Listing 3. Loading all Cargo tasks in Ant
<taskdef resource="cargo.tasks">
 <classpath>
  <pathelement location="${libdir}/${cargo-jar}"/>
  <pathelement location="${libdir}/${cargo-ant-jar}"/>
 </classpath>
</taskdef>

Once you've defined Cargo's tasks, the real action begins. Listing 4 defines Cargo tasks to download, install, and start a Tomcat container. The zipurlinstaller task downloads and installs Tomcat from http://www.apache.org/dist/tomcat/tomcat-5/v5.0.28/bin/ jakarta-tomcat-5.0.28.zip to a local temporary directory.

Listing 4. Downloading and starting Tomcat 5.0.28
<cargo containerId="tomcat5x" action="start" 
       wait="false" id="${tomcat-refid}">
	   
 <zipurlinstaller installurl="${tomcat-installer-url}"/>
 
 <configuration type="standalone" home="${tomcatdir}">
  <property name="cargo.remote.username" value="admin"/>
  <property name="cargo.remote.password" value=""/>

  <deployable type="war" file="${wardir}/${warfile}"/>

 </configuration>		
 
</cargo>

Note that to start and stop a container from within different tasks, as you'd like to do, you must associate the container with a unique id, which is the id="${tomcat-refid}" aspect of the cargo task.

Also note that configuration aspects of Tomcat are handled inside the cargo task. In Tomcat's case, you must set the username and password properties. Finally, you define a pointer to a WAR file using the deployable element.

Properties of Cargo

All the properties used in your Cargo tasks are shown in Listing 5. For example, tomcatdir defines one of two locations where Tomcat will be installed. This particular location is a mirrored structure, which will be referenced by the actual downloaded and installed Tomcat instance (which is found in a temporary directory). The property tomcat-refid helps associate a unique instance of a container as well.

Listing 5. Cargo properties
<property name="tomcat-installer-url" 
  value="http://www.apache.org/dist/tomcat/tomcat-5/v5.0.28/bin/
    jakarta-tomcat-5.0.28.zip"/>
<property name="tomcatdir" value="target/tomcat"/>	
<property name="tomcat.username" value="admin"/>	
<property name="tomcat.passwrd" value=""/>	
<property name="wardir" value="target/war"/>
<property name="warfile" value="words.war"/>	
<property name="tomcat-refid" value="tmptmct01"/>

To stop a container, you can define a task that references the tomcat-refid property, as shown in Listing 6:

Listing 6. Stopping a container a la Cargo
<cargo containerId="tomcat5x" action="stop" 
       refid="${tomcat-refid}"/>

Wrapping with Cargo

Listing 7 combines the code in Listing 4 and Listing 6, wrapping a test target with two Cargo tasks: one for starting and one for stopping Tomcat. The task antcall calls the target named _run-system-tests, which is defined in Listing 8.

Listing 7. Wrapping a test target with Cargo
<target name="system-test" if="Junit.present" 
        depends="init,junit-present,compile-tests,war">

 <cargo containerId="tomcat5x" action="start" 
       wait="false" id="${tomcat-refid}">			
  <zipurlinstaller installurl="${tomcat-installer-url}"/>
  <configuration type="standalone" home="${tomcatdir}">
   <property name="cargo.remote.username" value="admin"/>
   <property name="cargo.remote.password" value=""/>
   <deployable type="war" file="${wardir}/${warfile}"/>
  </configuration>
 </cargo>

 <antcall target="_run-system-tests"/>

 <cargo containerId="tomcat5x" action="stop" 
       refid="${tomcat-refid}"/>			

</target>

Listing 8 defines your test target, called _run-system-tests. Note that this task only runs system tests, which are placed in the test/system directory. For example, the test case defined in Listing 2 lives in this directory.

Listing 8. Running JUnit via Ant
<target name="_run-system-tests">
 <mkdir dir="${testreportdir}"/>   
 <junit dir="./" failureproperty="test.failure" 
        printSummary="yes" fork="true" 
		haltonerror="true">
  <sysproperty key="basedir" value="."/>     
  <formatter type="xml"/>      
  <formatter usefile="false" type="plain"/>     
  <classpath>
   <path refid="build.classpath"/>       
   <pathelement path="${testclassesdir}"/>
   <pathelement path="${classesdir}"/>      
  </classpath>
  <batchtest todir="${testreportdir}">
   <fileset dir="test/system">
    <include name="**/**Test.java"/> 
   </fileset>
  </batchtest>
 </junit>
</target>

In Listing 7, you fully configured your Ant build file to wrap your system tests with Cargo's deployment voodoo. The code in Listing 7 ensures that all the system tests in the test/system directory from Listing 8 are logically repeatable. You can run these system tests on any machine at any time, which is perfect for a Continuous Integration environment. The tests make no assumptions about the container -- not about its location or even if it's running! (Of course, these tests still make one assumption I haven't addressed, namely that the underlying database is properly configured and running. But that's a subject for another day.)


Repeatable results

In Listing 9, you can see the results of your efforts. When you issue the system-test command to an Ant build, your system tests are executed. Cargo handles all the details of managing the container of your choice, without making repeatability-crushing assumptions about the test environment.

Listing 9. The enhanced build
war:
 [war] Building war: C:\dev\projects\acme\target\widget.war

system-test:

_run-system-tests:
 [mkdir] Created dir: C:\dev\projects\acme\target\test-reports
 [junit] Running test.come.acme.widget.Web.RepeatableWordCreationTest
 [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.53 sec
 [junit] Testcase: testWordCreation took 4.436 sec

BUILD SUCCESSFUL
Total time: 1 minute 2 seconds

Keep in mind that Cargo also works in Maven builds. Furthermore, the Cargo Java API facilitates programmatic management of containers in anything from normal applications to test cases. And Cargo isn't only for JUnit (even though the code examples are written in JUnit). TestNG users will be happy to know that Cargo works just as well for their test suites. In fact, it doesn't matter what your tests are written in: just wrap them with Cargo and your container management issues will be automated out of the box!


In conclusion

It's up to you to ensure that your tests are logically repeatable, but you've seen this month that Cargo can really help. Cargo manages the container environment so you don't have to. Incorporate Cargo into your testing routine -- it will definitely lighten the load of crafting repeatable tests that verify Web applications.

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Java technology on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Java technology
ArticleID=162528
ArticleTitle=In pursuit of code quality: Repeatable system tests
publish-date=09262006