Selenium is a Web testing framework that establishes a new approach to validating Web applications. Unlike most Web testing tools, which attempt to simulate HTTP requests, Selenium approaches Web testing as if it were the browser itself. When you run an automated Selenium test, the framework fires up a browser and actually drives the browser through the steps delineated in the test, just the way a user would do when interacting with your application.

Selenium also distinguishes itself from many application testing frameworks by making it easy for both developers and non-developers to write tests. In Selenium, you can author tests programmatically or using Fit-style tables, and once you've authored a test you can fully automate it. It is easy to use an Ant build (for example) to run your entire Selenium test suite and it's also possible to run Selenium tests in a Continuous Integration (CI) environment.

This month, I'll introduce Selenium and walk you through the features that make it a standout Web application testing framework -- especially when combined with the likes of TestNG, DbUnit, and Cargo.

Programmatic testing with Selenium

In Selenium, you can author tests programmatically using the language of your choice or via Fit-style tables. From a testing standpoint, the process and results aren't much different regardless of which one you choose. I'm interested in exploring Selenium's programmatic approach here because it offers some interesting possibilities when combined with TestNG.

One advantage of using Selenium programmatically with a framework like TestNG is that it allows you to create intelligent fixtures, which is challenging to do with Fit-style tables. TestNG is an especially good match for Selenium because it enables you to do some things that aren't possible using other frameworks, such as test using dependencies, rerun failed tests, and set up parametric tests with the parameters defined in separate files. Having all of these features together would be notable in any Web application testing framework, but being able to use them in a fully automated acceptance test is outstanding, as you'll see.

Configuring your first test

The Selenium architecture essentially consists of two logical entities: the test code you write and the Selenium server, which facilitates interactions with the application under test. For your tests to successfully execute, both the Selenium server instance and the application under test must be up and running. (The outcome of the test depends on how well you've written the application, of course!)

Fortunately, the Selenium server is a lightweight process that can be programmatically started and stopped within the confines of an actual test. Starting and stopping a Selenium server (which is embodied by the Selenium object) is the responsibility of a fixture.

To programmatically start a Selenium server, you must create a new Selenium object and tell it which compatible browser to use -- I use Firefox for the following examples. You must also provide a location where the server instance should run (localhost is typical, but not required) and then the base URL for the application under test.

In Listing 1, I configure a local instance of Selenium to drive Firefox at a locally installed Web application (http://localhost:8080/gt15/). As you can probably infer from the arguments, the Selenium object acts as a proxy to the application under test, and accordingly facilitates testing it.

                
Selenium driver = 
  new DefaultSelenium("localhost", SeleniumServer.getDefaultPort(), 
   "*firefox", "http://localhost:8080/gt15/");

driver.start();
//go to web pages and do stuff...
driver.stop();

Once you've created an instance of Selenium, you can start and stop it while it's running. This means you can interact with the Selenium server programmatically and have it drive the browser through a test procedure.

Back to top

Driving the application

Programmatically interacting with Web pages is an exercise in using logical ids. (Some readers will be familiar with this concept from February's article about TestNG-Abbot.) The first step to interacting with a page element is to find it, which you can usually do using HTML element IDs. Selenium also allows you to use XPath, regular expressions, or even JavaScript to locate specific elements, if you so desire.

The HTML in Listing 2 is part of a simple Web application that uses Groovlets. The code defines a form that contains one input and a submit button. If I want Selenium to interact with this form, I must provide an ID for the input and a corresponding value. I also need to provide an ID for the submit button, so that Selenium can "click" it. Once clicked, the form will be submitted to a Groovlet -- in this case FindWidget.groovy.

                
<form method=post action="./FindWidget.groovy">
 <table border="0" style="border-style: dotted">
  <tr>
   <td  class="heading">Widget:</td>
   <td class="value"><input type="text" name="widget"></td>
  </tr>
  <tr>
   <td></td>
   <td class="value"><input type="submit" value="Find Description" name="submit"></td>
  </tr>
 </table>
</form>

I can now programmatically interact with the HTML form by using the IDs widget (to enter a value) and submit (to click the button), as shown in Listing 3:

                
driver.type("widget", "pg98-01");		
driver.click("submit");
driver.waitForPageToLoad("10000");
//assert some return value...

Selenium's API for interacting with Web page elements is quite intuitive. For input fields, I can use the type() method to associate a value with an ID. I can then programmatically click buttons if I want to. In Listing 3, I make Selenium wait 10 seconds -- long enough to allow the form-submit request to finish processing. Once the code in FindWidget.groovy runs its course and returns a response, I can use it to find specific page elements and verify that everything worked properly.

Back to top

Selenium and TestNG

TestNG's flexible and parametric fixtures make it an excellent workhorse for defining Selenium's driven acceptance tests. Add in TestNG's ability to define test dependencies and rerun failed tests, and it's a no-brainer that Selenium-plus-TestNG is a winning combination.

Let's start with a Web application that allows users to create, find, update, or delete widgets. Creating a widget requires three attributes: a name, a type, and a definition. The form to create a widget is shown in Figure 1:

Note that the form element Type is a drop-down list with three different choices, as shown in Figure 2:

Clicking Create Widget forces a Groovlet to process the request. If everything is all right (that is, the name and definition are not blank and the instance isn't already in the database), the Groovlet creates a new widget instance and returns a status page like the one shown in Figure 3:

Using Selenium along with TestNG to verify the simple Create Widget use case is a manageable exercise:

1. Configure and start an instance of the Selenium server.
2. Interact with the Create Widget Web form and submit it.
3. Verify that the resulting page contains a success message with the widget's name.
4. Stop the Selenium server instance.

Note that each step in the use case is done via Selenium -- TestNG just helps with the plumbing, so to speak. Now let's give it a try.

Back to top

The Create Widget test case

I'd like the configuration of the Selenium server to be flexible, so I'll write a parameterized fixture (TestNG-Selenium style) that I can then use to generically create Selenium servers for different browsers, different locations, and even assorted Web application addresses (like localhost and production, to name a few). Listing 4 defines my flexible Selenium server fixture:

                
 @Parameters({"selen-svr-addr","brwsr-path","aut-addr"})
 @BeforeClass
 private void init(String selenSrvrAddr, String bpath, 
   String appPath) throws Exception {
  driver = new DefaultSelenium(selenSrvrAddr, 
    SeleniumServer.getDefaultPort(), bpath, appPath);
  driver.start();
 }
 //....
 @AfterClass
 private void stop() throws Exception {
  driver.stop();
 }

I have to link the parameter names with values in TestNG's testng.xml file; consequently, I define three parameter elements as shown in Listing 5. (I default the brwsr-path parameter to Firefox, but I could just as easily define a new set of tests using Internet Explorer.)

                
 <parameter name="selen-svr-addr" value="localhost"/> 
 <parameter name="aut-addr" value="http://localhost:8080/gt15/"/> 
 <parameter name="brwsr-path" value="*firefox"/>

Next, I define the test case shown in Listing 6, which also takes a parameter for the base URL of the application under test. This test forces the browser to open a specific page within the Web application and manipulates the form shown in Figure 1.

                
 @Parameters({"aut-addr"})
 @Test
 public void verifyCreate(String appPath) throws Exception {
  driver.open(appPath + "/CreateWidget.html");
  driver.type("widget", "book-01");
  driver.select("type", "book");
  driver.type("definition", "book widget type book");
  driver.click("submit");

  driver.waitForPageToLoad("10000");		
  assertEquals(driver.getText("success"), 
    "The widget book-01 was successfully created.", 
    "test didn't return expected message");
 }

Once the form is submitted via the driver.click("submit") call, I force Selenium to wait for the response to load and then I assert the presence of a successful creation message. (Note that the response Web page has an element whose ID is success.)

Putting this all together yields a flexible test class that verifies two scenarios: a sunny-day path and an edge case where no definition is provided, as shown in Listing 7:

                
public class CreateWidgetUATest {
 private Selenium driver;

 @Parameters({"selen-svr-addr","brwsr-path","aut-addr"})
 @BeforeClass
 private void init(String selenSrvrAddr, String bpath, 
   String appPath) throws Exception {
  driver = new DefaultSelenium(selenSrvrAddr, 
    SeleniumServer.getDefaultPort(), bpath, appPath);
  driver.start();
 }

 @Parameters({"aut-addr"})
 @Test
 public void verifyCreate(String appPath) throws Exception {
  driver.open(appPath + "/CreateWidget.html");
  driver.type("widget", "book-01");
  driver.select("type", "book");
  driver.type("definition", "book widget type book");
  driver.click("submit");

  driver.waitForPageToLoad("10000");		
  assertEquals(driver.getText("success"), 
    "The widget book-01 was successfully created.", 
    "test didn't return expected message");
 }

 @Parameters({"aut-addr"})
 @Test
 public void verifyCreationError(String appPath) throws Exception {
  driver.open(appPath + "/CreateWidget.html");
  driver.type("widget", "book-02");
  driver.select("type", "book");
  //definition explicitly set to blank
  driver.type("definition", "");
  driver.click("submit");

  driver.waitForPageToLoad("10000");		
  assertEquals(driver.getText("failure"), 
    "There was an error in creating the widget.", 
    "test didn't return expected message");
 }

 @AfterClass
 private void stop() throws Exception {
  driver.stop();
 }
}

So far, I've defined two Selenium tests that are flexible enough to facilitate testing with multiple browsers and against multiple locations, which is pretty good for a beginner. Getting a little more advanced, though, I want to start thinking about whether the logic in my tests is repeatable. For instance, what will happen if I run the CreateWidgetUATest test class twice in a row? How can I be sure that my Web application is running with the latest version of the code on my local machine, or on any machine, for that matter?

Back to top

Repeatable acceptance tests

Both the Selenium server and the Web application to be verified must be running when executing Selenium tests. By implication, all the associated architectural dependencies of the application must also be running -- which in the case of most Java™ Web applications means a Servlet container and an associated database.

As I explained in my article about repeatable system tests, DbUnit and Cargo are two of my favorite technologies for implementing logical repeatability in database-dependent Web applications. DbUnit manages the data in a database and Cargo automates container management in a generic fashion. The following sections show you how the two work together with Selenium and TestNG to ensure logically repeatable acceptance tests.

Back to top

DbUnit rides again and again

As you might recall, DbUnit facilitates working with databases by effectively managing their data within the context of a test scenario. With DbUnit, you can load a known set of data into a database before a test, which means you can rely on that data being present during testing. What's more, you can also remove from the database any data created as a result of a test, once the test is done. DbUnit facilitates all this by acting as a convenient fixture -- whether it be in JUnit or TestNG -- that reads seed files containing test data and logically inserts, removes, or updates that data into corresponding database tables.

Because I'm using TestNG to drive Selenium, I'm going to create a DbUnit fixture, which will run at the test level. TestNG supports running fixtures at five levels of granularity. The lowest two, method and class, are the most obvious -- a fixture for each test method or one for the entire class. After these, TestNG defines a fixture for a collection of tests (which are defined in TestNG configuration files and designated by the test element), a fixture for an entire suite (which is a collection of a collection of tests and designated by the suite element), and a fixture for a group of tests (which are defined in TestNG's Test annotation).

Test details

Creating a DbUnit fixture to run at the test level means that a collection of test classes will share the same logic to properly seed a database before you run any tests. In my case, I'd like the database to have a clean set of data before each logical test collection is run. Using DbUnit's CLEAN_INSERT command ensures that any rows created in previous test runs will be removed -- hence, I can rerun a test that creates data over and over again and not worry about database constraints.

What's more, I want my fixture to rely on parametric data, which will give me the flexibility to switch seed files, and even the location of a particular database, before a specific test run. Associating parameters with TestNG couldn't be easier: all I have to do is decorate my fixture with the Parameters annotation, declare corresponding parameters in my method signature, and provide values in TestNG's configuration file.

Listing 8 defines a simple DbUnit fixture that seeds a database with a desired seed file. Note that the fixture is defined to take five parameters. (That's probably too many, but isn't it cool having parameters in a fixture?)

                
public class DatabaseFixture {

 @Parameters({"seed-path","db-driver","db-url","db-user","db-psswrd"})
 @BeforeTest
 public void seedDatabase(String seedpath, String driver, 
   String url, String user, String pssword) throws Exception {

  IDatabaseConnection conn = this.getConnection(driver, url, user, pssword);
  IDataSet data = this.getDataSet(seedpath);

  try {
   DatabaseOperation.CLEAN_INSERT.execute(conn, data);
  }finally {
   conn.close();
  }
 }

 private IDataSet getDataSet(String path) throws IOException, DataSetException {
  return new FlatXmlDataSet(new File(path));
 }

 private IDatabaseConnection getConnection(String driver, 
   String url, String user, String pssword ) throws ClassNotFoundException,
    SQLException {
  Class.forName(driver);
  Connection jdbcConnection = 
    DriverManager.getConnection(url, user, pssword);
  return new DatabaseConnection(jdbcConnection);
 }
}

To associate real values with the parameters found in Listing 8, I have to define them in TestNG's testng.xml file, as shown in Listing 9:

                
 <parameter name="seed-path" value="test/conf/gt15-seed.xml"/> 
 <parameter name="db-driver" value="org.hsqldb.jdbcDriver"/>
 <parameter name="db-url" value="jdbc:hsqldb:hsql://127.0.0.1"/>
 <parameter name="db-user" value="sa"/>
 <parameter name="db-psswrd" value=""/>

Generic parameter values

Now that I've defined a flexible fixture that handles database state and some corresponding tests, I'm ready to use TestNG to wire everything together. As usual, knowing what I want is the first step to achieving it. In this case, I'd like to achieve the following:

• I want my DbUnit fixture to do its thing before any logical collection of tests is run.
• I want to run the same collection of tests twice: once for Firefox and once for Internet Explorer.

It's a good thing for me that TestNG's parameter elements are locally scoped. As a result, I can easily define generic parameter values in the TestNG configuration file and then override them if necessary in TestNG's test grouping elements.

For example, to run two sets of tests, I simply create two test elements. I can then include my fixture and associated tests via TestNG's package element, which facilitates finding all tests (or fixtures) in a package structure. Next, I can associate the brwsr-path parameter for Firefox and Internet Explorer in the two defined test groupings. All of this is shown in the testng.xml in Listing 10:

                
<suite name="User Acceptance Tests" verbose="1" >
 
 <!-- required for DbUnit fixture   -->
 <parameter name="seed-path" value="test/conf/gt15-seed.xml"/> 
 <parameter name="db-driver" value="org.hsqldb.jdbcDriver"/>
 <parameter name="db-url" value="jdbc:hsqldb:hsql://127.0.0.1"/>
 <parameter name="db-user" value="sa"/>
 <parameter name="db-psswrd" value=""/>
 
 <!-- required for Selenium fixture -->
 <parameter name="selen-svr-addr" value="localhost"/> 
 <parameter name="aut-addr" value="http://localhost:8080/gt15/"/> 	

 <test name="GT15 CRUDs- Firefox" > 
 
  <parameter name="brwsr-path" value="*firefox"/>

  <packages>
   <package name="test.com.acme.gt15.Web.selenium" />
   <package name="test.com.acme.gt15.Web.selenium.fixtures" />
  </packages>
 </test>

 <test name="GT15 CRUDs- IE" > 
  
  <parameter name="brwsr-path" value="*iexplore"/>

  <packages>
   <package name="test.com.acme.gt15.Web.selenium" />
   <package name="test.com.acme.gt15.Web.selenium.fixtures" />
  </packages> 
 </test>
</suite>

I'm pleased to announce that I have done almost everything necessary to create a suite of repeatable acceptance tests. All I have left is to handle the Web application container itself. Lucky for me, I have Cargo on my side.

Back to top

Cargo carries the load

Cargo is an innovative open source project that automates container management in a generic fashion, such that the same API used to deploy a WAR file to JBoss can also start and stop Tomcat. Cargo also can download and install a container automatically -- you can utilize the Cargo API a number of ways, ranging from Java code to Ant tasks and even Maven.

Using a tool like Cargo takes care of one of the major challenges to writing logically repeatable test cases, which is avoiding the underlying assumption that a running container has the latest and greatest application code. Furthermore, you can construct a build process (like within Ant) that harnesses the power of Cargo to automatically do the following:

1. Download a desired container.
2. Install the container.
3. Start the container.
4. Deploy a selected WAR or EAR file to the container.

At a later point, you can also have Cargo stop a selected container. (Oh, and there's no need to be alarmed about downloading and installing a container, either -- if the correct version of one is already on the local machine, Cargo skips Steps 1 and 2.)

I'd like to use Cargo to ensure the latest and greatest version of my Web application is up and running. What's more, I don't want to worry about where I need to deploy a WAR file, nor do I want to have to ensure the most recent WAR is being used. What I really want to do is make user acceptance testing a non-event -- I just want to issue one command, like ua-test, and sit back and wait for the results. Even better, in a CI environment, I won't be waiting for anything; I'll simply get a notification at some point after the tests complete!

Test container management

To set up Cargo within Ant, I need to define a task that downloads a specific version of Tomcat and installs it into a temporary directory on the local machine. Next, the latest version of the code (bundled in a WAR file) is deployed to Tomcat as shown in Listing 11:

                
<target name="ua-test" depends="compile-tests,war">

 <taskdef resource="cargo.tasks">
  <classpath>
  <pathelement location="${libdir}/${cargo-jar}" />
  <pathelement location="${libdir}/${cargo-ant-jar}" />
  </classpath>
 </taskdef>
 
 <cargo containerId="tomcat5x" action="start" wait="false" id="${tomcat-refid}">
  <zipurlinstaller installurl="${tomcat-installer-url}" />
  <configuration type="standalone" home="${tomcatdir}">
   <property name="cargo.remote.username" value="admin" />
   <property name="cargo.remote.password" value="" />
   <deployable type="war" file="${wardir}/${warfile}" />
  </configuration>
 </cargo>

 <antcall target="_start-selenium" />

 <cargo containerId="tomcat5x" action="stop" refid="${tomcat-refid}" />
</target>

The target in Listing 11 uses antcall to invoke another target. In essence, the last cargo task in Listing 11 wraps the _start-selenium target and ensures that Tomcat is stopped after a test run.

In the _start-selenium target, defined in Listing 12, I need to start (and later stop) the Selenium server. My tests will also connect to this server instance in their Selenium fixtures, by the way. Note how this target then refers to another target -- _run-ua-tests (defined in Listing 13).

                
<target name="_start-selenium">
 <java jar="${libdir}/${selenium-srvr-jar}" fork="true" spawn="true" />
 <antcall target="_run-ua-tests" />
 <get dest="${testreportdir}/results.txt" 
        src="${selenium-srvr-loc}/selenium-server/driver/?cmd=shutDown" />
</target>

Finally, the last target in the group is the one that actually runs my programmatic Selenium tests via TestNG. Note how I force TestNG to use my testng.xml file by using an xmlfileset element in the _run-ua-tests target in Listing 13:

<target name="_run-ua-tests">
 <taskdef classpathref="build.classpath" resource="testngtasks" />
 <testng outputDir="${testreportdir}" 
         classpath="${testclassesdir};${classesdir}" haltonfailure="true">
  <xmlfileset dir="./test/conf" includes="testng.xml" />
  <classpath>
   <path refid="build.classpath" />
  </classpath>
 </testng>
</target>

Back to top

In conclusion

As you have seen, Selenium facilitates user acceptance testing quite nicely, especially when driven by TestNG. While programmatic testing isn't for everyone (non-developers will likely prefer Selenium's Fit-style tables), it does give you access to TestNG's exceptional flexibility. Programmatic testing also allows you to build out your test framework with DbUnit and Cargo, thus ensuring the logical repeatability of your tests.

The evolution of open source Web testing frameworks is by no means finished, which is good news for code quality perfectionists. Selenium is one of the first of a new breed of open source, browser-driving Web testing frameworks that automate user acceptance testing -- and as such, it is outstanding. Combine Selenium with TestNG, as I've done in this article, and you've got yourself a great test driver, along with the considerable benefits of dependency testing and parametric testing. Give Selenium with TestNG a try -- your users will thank you for it.

Resources

Learn

• "TestNG makes Java unit testing a breeze" (Filippo Diotalevi, developerWorks, January 2005): An introduction to unit testing with TestNG.
  
  
• " In pursuit of code quality: Repeatable system tests" (Andrew Glover, developerWorks, September 2006): Find out why you need logically repeatable tests.
  
  
• " In pursuit of code quality: Automate GUI testing with TestNG-Abbot" (Andrew Glover, developerWorks, February 2007): Discover a testing framework that breathes new life into testing GUI components.
  
  
• " In pursuit of code quality: Resolve to get FIT" (Andrew Glover, developerWorks, February 2006): The Framework for Integrated Tests makes it easier for non-developers to write tests.
  
  
• "Control your test-environment with DbUnit and Anthill" (Philippe Girolami, developerWorks, April 2004): Use DbUnit with JUnit to control even a Continuous Integration test environment.
  
  
• "Practically Groovy: Go server-side up, with Groovy" (Andrew Glover, developerWorks, March 2005): Groovy offers a simplified alternative for developing server-side applications quickly and easily.
  
  
• "Effective Unit Testing with DbUnit" (Andrew Glover, OnJava, January 2004): Introduces database-dependent testing with DbUnit.
  
  
• An interview with Cargo's Vincent Massol (Andrew Glover, thediscoblog.com, February 2006): Andrew interviews Cargo founder Vincent Massol.
  
  
• Hip system tests with Cargo (Andrew Glover, thediscoblog.com, April 2006): An introduction to Cargo's Java API.
  
  
• developerWorks Java technology zone: Hundreds of articles about every aspect of Java programming.
  
  

Get products and technologies

• Download Selenium: Run user acceptance tests directly in IE or Firefox.
  
  
• Download TestNG: A flexible testing framework that can also be used to drive Selenium's tests.
  
  
• Download Cargo: Make repeatable tests of Web applications even easier.
  
  
• Download DbUnit: Puts your database into a known state between test runs.
  
  

Discuss

• Participate in the discussion forum.
  
  

About the author

Andrew Glover is president of Stelligent Incorporated, which helps companies address software quality with effective developer testing strategies and continuous integration techniques that enable teams to monitor code quality early and often. Check out Andy's blog for a list of his publications.

Rate this article

Comments

Back to top

Table of contents

• Programmatic testing with Selenium
• Driving the application
• Selenium and TestNG
• The Create Widget test case
• Repeatable acceptance tests
• DbUnit rides again and again
• Cargo carries the load
• In conclusion
• Resources
• About the author
• Comments

Dig deeper into Java on developerWorks

IBM SmartCloud trial. No charge.

Unleash the power of hybrid cloud computing today!

---

Public and private solutions in one trial.

Special offers