Automate unit and integration testing using cloud technologies

Explore unit and integration testing, and how to perform these tests using cloud technologies such as Amazon EC2. This article describes the Jenkins continuous integration build tool and offers basic steps for getting started with Selenium Grid.

Nathan A. Good, Senior Consultant and Freelance Developer

Nathan GoodNathan A. Good lives in the Twin Cities area of Minnesota. Professionally, he does software development, software architecture, and systems administration. When he's not writing software, he enjoys building PCs and servers, reading about and working with new technologies, and trying to get his friends to make the move to open source software. He's written and co-written many books and articles, including Professional Red Hat Enterprise Linux 3, Regular Expression Recipes: A Problem-Solution Approach, and Foundations of PEAR: Rapid PHP Development.



06 March 2012

Also available in Chinese

This article describes unit and integration testing and using cloud technologies that enable you to simulate real-world usage of your application. The first few sections provide an overview of both unit and integration testing as well as the benefits of automating the tests. The following sections describe tools for testing and includes script examples.

The tools used in this article are JUnit for unit testing, Jenkins for running continuous integration builds, Selenium for testing your web application from the browser, and Amazon Elastic Compute Cloud (Amazon EC2) for scaling up to many agents running an integration test. This article provides a high-level overview of installing, configuring, and running these tools. See Resources for links to more in-depth tutorials.

If you are new to unit and integration tests, the difference between the two can be unclear. The next sections describe the differences between the two types of tests and the correct scenarios under which each type of test is useful.

Unit tests

Unit tests are runnable methods that verify the functionality of your code. Properly written, unit tests should be fairly granular in what they test. A well-written unit test will verify the functionality of a single method in a class.

Among developers who write unit tests, a debatable topic is whether to test accessor methods—that is, public getters and setters. Modern tooling that generates those accessors, such as those available in Eclipse's Source > Generate Getters and Setters menu, certainly helps to eliminate mistakes while creating the accessors. Without automatically generating the accessors, a common mistake comes from using a cut and paste technique to create different accessors that incorrectly set the wrong private variable.

A good case for writing unit tests can be made for testing any public method, including accessors. A solid bed of regression tests that offer the most complete coverage possible will only ensure that the entire class operates as intended.

Code coverage tools allow you to verify that as much of the code has been exercised as possible during the execution of the unit tests. Ideally, all of the code should be covered. Realistically, 100% coverage is often not easily achieved and can be cost-prohibitive in your project.

Integration tests test larger pieces of functionality than unit tests.

Integration tests

Whereas unit tests verify the functionality of a granular piece of code, integration tests verify higher-level functionality. An example of an integration test is calling a method on a servlet. This method is in turn likely to call one or more of the methods.

An important benefit of using code coverage tools during integration testing is that doing so helps to identify dead branches of code that unit test coverage hides. If you exercise every public method during the unit tests, some methods are likely to be executed that may never be executed during real-life uses of your application.


Benefits of automated testing

Most integrated development environments (IDEs) now include features that allow developers to easily execute unit tests while they are writing their code. This level of integration offers quick feedback to make sure that no recent code changes have broken any code.

In most projects, a source control management (SCM) tool is used to maintain the code for the entire team. Even if a developer executes tests locally, it's a good idea to run the tests in an automated fashion using build tools such as Jenkins. Like many other build tools, you can configure Jenkins to monitor the SCM to automatically compile the entire project and run tests whenever it finds changes to the source code. This technique, called continuous integration builds, provides feedback to the entire team upon each commit of changes to the source code.

The immediate feedback allows teams to respond to failed tests as quickly as possible. The main goal of doing continuous integration builds is to ensure that at any point in time the code that is in the SCM could be deployed to production so long as it's feature-complete. At the very least, anyone on the team should be able to pull code from the source's repository and compile and run it locally.

Skill sets

An added benefit of running automated integration tests with a tool like Jenkins is the reduced skill set burden on the team. When the initial configuration is complete, little ongoing maintenance of the automated process is necessary. For the most part, everyone on the team can almost forget about the tests until one of them fails. Rarely does each person on the team need to be an expert in this configuration.

Often, the integration tests provide an example of how the code under test is intended to be called. Tests can be used as examples when using the classes and methods in other code.


Integration testing with Jenkins

This section provides a high-level overview of compiling the code and executing the tests with Jenkins. See Resources for links to more information.

To install and run Jenkins, download the jenkins.war file from the project site (see Resources). Alternatively, you can download the binary package for the platform on which you intend to run Jenkins. Jenkins comes with its own embedded web server, so once you've downloaded the web archive (WAR) file, you can start Jenkins by simply using the following Java™ command:

$ java -jar jenkins.war

When Jenkins has started, use your browser to navigate to http://server_name:8080, where server_name is the name of the server on which Jenkins is running. From there, you can write an Apache Ant script to execute the tests during each build.


Testing during building

As mentioned previously, you can configure Jenkins to run both your unit tests and integration tests for the team whenever code changes in the SCM's repository. Jenkins offers many different ways to run the tests. This article focuses on executing the different tests using an Ant script to keep the examples as straightforward as possible. See Resources for more information about Ant.

Listing 1 provides an example of a simple script that compiles Java code using the Ant javac task in the compile target. Then, in the unit-test target, the Ant script uses the junit task to execute the JUnit tests for unit tests.

Listing 1. An example Ant script that compiles code and executes unit tests with JUnit
<?xml version="1.0" encoding="UTF-8"?>
<project name="mywebapp" default="main" basedir="."> 

<target name="compile" depends="resolve">
<mkdir dir="${build.dir}"/>
<javac srcdir="${src.dir}" destdir="${build.dir}" debug="true">
<classpath refid="base.classpath" />
</javac>
</target>

<target name="compile-tests" depends="compile">
<mkdir dir="${test.build.dir}" />
<javac srcdir="${test.dir}" destdir="${test.build.dir}" debug="on">
<classpath refid="project.test.classpath" />
</javac>
</target>

<target name="run-tests" depends="compile-tests">

<junit haltonfailure="false" fork="yes">
<sysproperty key="net.sourceforge.cobertura.datafile" 
    file="${workspace.dir}/cobertura.ser" />
<classpath location="${instrumented.dir}" />
<classpath refid="project.test.classpath" />
<formatter type="xml" />
<batchtest todir="${reports.dir}/junit">
<fileset dir="${test.build.dir}">
<include name="**/*Test.class" />
</fileset>
</batchtest>
</junit>

<junitreport todir="${reports.dir}/junit">
<fileset dir="${reports.dir}/junit">
<include name="TEST-*.xml" />
</fileset>
<report format="noframes" todir="${reports.dir}/junit" />
</junitreport>
</target>
</project>

The Selenium IDE allows you to record tests and export them as JUnit or TestNG-based tests that you can execute from Jenkins. An example of the integration test exported from the Selenium IDE is shown in Listing 2.

Listing 2. The exported test code from the Selenium IDE
package com.example.tests;

import com.thoughtworks.selenium.Selenium;
import com.thoughtworks.selenium.SeleneseTestCase;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebDriverBackedSelenium;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import java.util.regex.Pattern;

public class MyWebTest extends SeleneseTestCase {
  @Before
  public void setUp() throws Exception {
    WebDriver driver = new FirefoxDriver();
    String baseUrl = "http://mywebapp.example.com";
    selenium = new WebDriverBackedSelenium(driver, baseUrl);
  }

  @Test
  public void testMytest() throws Exception {
  }

  @After
  public void tearDown() throws Exception {
    selenium.stop();
  }
}

The integration tests have been separated from the unit tests, because they represent different steps that can be taken in the build. Sometimes, integration tests can take longer to execute. Without using parallelism techniques, common workarounds are to run certain unit tests in the continuous integration builds and defer integration builds to run at a less-frequent schedule, such as once daily. Using Selenium Grid helps run these longer-running tests in a shorter amount of time so that quick feedback is still possible.

When you are successfully executing the tests in Jenkins, you can configure Jenkins to view the results of the tests on your build page by installing the JUnit plug-in for Jenkins.

Now that you are successfully executing integration tests during your builds, you are ready to execute the integration tests in the cloud to ensure that your code scales to multiple users.


Taking testing to the cloud

Fortunately, recent cloud offerings provide increasingly easier methods for executing tests using off-premises infrastructure. Using a cloud infrastructure for your testing offers several advantages over building out and using the infrastructure internally. Two of those advantages are relatively simple parallelism and scale as well as reduced infrastructure costs.

Parallelism and scale

Executing the integration tests using multiple cloud clients gives you the ability to test your application quickly by using the advantage of parallelism. The tests, when run in parallel, provide quick feedback that all is right with your application after changes have been made. Because you get feedback faster, you don't have to worry about sacrificing the integrity of the builds by scheduling longer-running tests to occur less frequently (such as only once or twice daily).

Reduced infrastructure

Reduced infrastructure costs come in the form of using someone else's equipment for executing the tests that require several machines. Using the cloud for the temporary machines saves you more than simply the cost of the hardware. You also save yourself the cost of licensing the operating system as well as the cost of labor for building and maintaining the server.

Most cloud offerings have plans that only charge you when a virtual machine is running, so you don't have to pay continually for power or floor space for a machine that might sit idle for a great deal of the time unless tests are running.

Running the tests

Now that you have Jenkins up and running and have some Selenium tests running through Jenkins, you can use Selenium Grid to run many tests simultaneously from your build. Running the Selenium Grid on Amazon EC2 requires a few steps. First, you must download and configure the Selenium Grid binary distribution to your local machine. The binary distribution requires Java and Ant. You can find links to both of these tools in Resources.

To run automated tests on Amazon EC2 using Selenium Grid:

  1. Create an Amazon account. (See Resources for a link to information on setting up your Amazon account.)
  2. Create your access keys and X.509 certificate. Click the Security Credentials link under My Account/Console; then, under X.509 Certificates, click the Create a new Certificate link.

    Make sure to download your certificates and the private keys to a location such as a directory called /home/myusername/.ec2 on Linux® or Apple Mac. On Windows®, use a directory such as C:\Users\MyUserName\ec2.

  3. Under Key Pairs, download your public key into the .ec2 directory. Create a new one if needed. When you are finished downloading the certificate, private key, and public key, the contents of your ec2 directory should look like this:
    -r-------- 1 user user 916 Sep  8 18:54 cert-7KWYWUSFL63NXEJAWPEOYFO7FWGKJZQT.pem
    -r-------- 1 user user 932 Sep  8 18:54 pk-7KWYWUSFL63NXEJAWPEOYFO7FWGKJZQT.pem
    -r-------- 1 user user 272 Sep  8 18:58 rsa-APKAIAXDHYVMKMZMCFPA.pem

    For security reasons, these files should be Read-only by your user account.

  4. Now that you have your Amazon account created and the security files downloaded, download the Amazon EC2 application programming interface (API) command-line tools. Extract them into a directory on your computer and take note of it (referred to as EC2_HOME during the rest of this article).
  5. You will need to set up the environment variables as shown in Listing 3. For ease of ongoing use, consider writing a script or .bat file, as shown.
    Listing 3. Using a script to set the environment variables
    #!/bin/bash
    export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-i386/jre
    
    export EC2_HOME=/home/user/opt/ec2-api-tools-1.6.1.5
    export EC2_PRIVATE_KEY=/home/user/.ec2/pk-7KWYWUSFL63NXEJAWPEOYFO7FWGKJZQT.pem
    export EC2_CERT=/home/user/.ec2/cert-7KWYWUSFL63NXEJAWPEOYFO7FWGKJZQT.pem
    export EC2_KEYPAIR=/home/user/.ec2/rsa-APKAIAXDHYVMKMZMCFPA.pem
    export EC2_KEYPAIR_NAME=APKAIAXDHYVMKMZMCFPA 
    
    export PATH=${EC2_HOME}/bin:${PATH}
  6. You can verify your Amazon EC2 API command-line tools by typing the command ec2-version. To verify that you have access to the images with selenium-grid installed on them, type the following command:
    $ ec2-describe-images -a | grep selenium grid
  7. After you've verified the Amazon EC2 API command-line tools, you need to install Capistrano (see Resources for a link). Capistrano is required for the Selenium Grid distribution.
  8. Next, download the Selenium Grid distribution, and run the following command to verify that your setup is complete:
    $ cap ec2:check_settings
  9. A Capfile inside the distribution contains configuration values. You can set the number of Amazon EC2 instances that you want to launch by editing the line in the file as shown:
    $ set :remote_control_farms_count, 3
  10. After you have set the number of instances, boot the instances by typing cap grid:boot. You may have to wait some time before the grid is ready to use.
  11. To run the tests inside Jenkins, use an Ant script based on the one that comes with the Selenium Grid demo, as shown in Listing 4.
    Listing 4. Using an Ant script to launch integration tests using Selenium Grid
      <!-- snipped... -->
      <target name="run-tests-grid" description="Run integration tests in parallel">
        <java classpathref="test.classpath"
            classname="org.testng.TestNG"
            failoner
    
            >
          <sysproperty key="java.security.policy" file="${basedir}/lib/testng.policy"/>
          <sysproperty key="webSite" value="${my.web.site}" />
          <sysproperty key="seleniumHost" value="${selenium.host}" />
          <sysproperty key="seleniumPort" value="${selenium.port}" />
          <sysproperty key="browser" value="${browser}" />
    
          <arg value="-d" />
          <arg value="${basedir}/target/reports" />
          <arg value="-suitename" />
          <arg value="Selenium Grid Demo In Parallel" />
          <arg value="-parallel"/>
          <arg value="methods"/>
          <arg value="-threadcount"/>
          <arg value="10"/>
          <arg value="-testclass"/>
          <arg value="com.example.tests.MyWebTest"/>
        </java>
      </target>
      <!-- snipped... -->

Notice in Listing 4 that the classname has been updated to the class that you created earlier in this article.


Test automation and continuous deployment

Automating your test execution during every build is the first step toward continuous deployment. Continuous deployment describes the development methodology of using automated processes to compile code every time the code changes, test the code, and push the code out to environments.

The goal behind continuous integration building and continuous deployment is to create a development environment in which deployment is frictionless. Deployment should be easy, predictable, and require little manual intervention.


Conclusion

Unit and integration testing techniques, combined with tools such as Jenkins for building and Selenium Grid for testing in the cloud, allow you to create applications that are predictable and consistent. Using Selenium Grid for integration testing allows you to verify functionality quickly by running tests in parallel.

Resources

Learn

Get products and technologies

  • Read about and get JUnit for unit testing your Java code.
  • Download Jenkins from the project's website.
  • Download the Amazon EC2 command-line tools.
  • Download Capistrano from the official site.
  • Evaluate IBM products in the way that suits you best: Download a product trial, try a product online, use a product in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement service-oriented architecture efficiently.

Discuss

  • Get involved in the My developerWorks community. Connect with other developerWorks users while exploring the developer-driven blogs, forums, groups, and wikis.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Cloud computing on developerWorks


  • Bluemix Developers Community

    Get samples, articles, product docs, and community resources to help build, deploy, and manage your cloud apps.

  • Cloud digest

    Complete cloud software, infrastructure, and platform knowledge.

  • DevOps Services

    Software development in the cloud. Register today to create a project.

  • Try SoftLayer Cloud

    Deploy public cloud instances in as few as 5 minutes. Try the SoftLayer public cloud instance for one month.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Cloud computing
ArticleID=860637
ArticleTitle=Automate unit and integration testing using cloud technologies
publish-date=03062012