Automated testing with Selenium and Cucumber

Write, batch, and run automated tests on your RIAs

Take the pain out of test automation with Selenium and Cucumber, by learning to write tests in simple feature files and drive them through your browser with the flip of a switch. This article is a hands-on introduction to setting up a test automation framework with Selenium and Cucumber, creating a test suite for single-page web applications, and running tests across multiple web and mobile browsers via Selenium Grid.

Alan Bowers (alan.bowers@uk.ibm.com), Web and Mobile Consultant, IBM

Alan BowersAlan Bowers is a Web 2.0 and mobile consultant in the Software Services for Mobile Team, part of IBM Software Group. He has been in his current role since April 2011. During this time, Alan has been exposed to a wide range of web technologies, especially JavaScript, Dojo, jQuery, and Worklight, as well as the complete development lifecycle and agile development methodologies. Prior to his current role, Alan was a Java developer working in a small agile team developing diagnostic tooling for Java-based applications.



James Bell (james.bell@uk.ibm.com), Managing Consultant, IBM

James BellJames Bell is a consultant for IBM Interactive within Global Business Services, with experience leading, managing, and developing technical solutions. James has worked on both large and small projects, leading local and global delivery teams.



06 August 2013

Ensuring that a web application works and will continue to work across a set of supported browsers is an essential part of the web application development lifecycle. Traditionally, web developers have used extensive regression testing to ensure cross-browser compatibility, but this approach is expensive in terms of both time and resources. An alternative approach is to create multiple suits of fully automated tests that can be run against a range of browsers.

You can build and run suits of fully automated tests as part of a scheduled build process, or as part of a build pipeline. In either case, you have the benefit of detecting problems early and getting rapid feedback. By the time a build is promoted, you will already be confident that the application code works, which can greatly reduce or even completely eliminate the need for regression testing.

In this article, we demonstrate the advantages of using Selenium with Cucumber for automated testing. Using this framework, you write tests in feature files in a form understandable by any business manager or other non-technical stakeholder. The tests are then translated into the Java language by Cucumber, which supports multiple scripting and programming languages. Selenium is used to drive the browser.

We walk through a complete process of setting up an automated, Java-based testing framework with Cucumber and Selenium. We start with an introduction to writing feature files in Cucumber, then show you how to use Selenium, along with a modified Page Object pattern implementation, to test a rich Internet application (RIA) in the supported browser of your choice. We also quickly demonstrate the setup to test application code on an additional browser such as Chrome, or on a mobile browser. We conclude with a look at a Maven build setup to automate a test suite. Throughout the article, we focus on how to plan and implement your test code for a maintainable solution.

You can download the mock RIA application used for the demonstration at any time.

Testing with Cucumber

Cucumber is a testing framework that helps to bridge the gap between software developers and business managers. Tests are written in plain language based on the behavior-driven development (BDD) style of Given, When, Then, which any layperson can understand. Test cases are then placed into feature files that cover one or more test scenarios. Cucumber interprets the tests into the specified programming language and uses Selenium to drive the test cases in a browser. Our tests are translated into Java code.

URL for the demo app

Examples in this article assume that you are serving the demo application locally under the following URL: http://localhost/MockApplication/html/MockApplication.html. If you change the application URL, then you will need to update your test code accordingly.

The BDD Given, When, Then syntax is designed to be intuitive. Consider the syntax elements:

  • Given provides context for the test scenario about to be executed, such as the point in your application that the test occurs as well as any prerequisite data.
  • When specifies the set of actions that triggers the test, such as user or subsystem actions.
  • Then specifies the expected result of the test.

Listing 1 shows a login test for a simple website. The user logs on to the website and should be advanced to the logged-in page.

Listing 1. Initial login test
Feature: Test login

Scenario Outline: Login Success and Failure
	Given I navigate to the mock application
	When I try to login with valid credentials
	Then I should see that I logged in successfully

If we wanted to make this test reuseable we could, because Cucumber allows us to substitute values from a table. It is very easy to refactor the test in Listing 1 to test for success and failure, as shown in Listing 2:

Listing 2. Login testing for success or failure
@run
Feature: Test Login

Scenario Outline: Login Success and Failure
	Given I navigate to the mock application
	When I try to login with '<type>' credentials
	Then I should see that I logged in '<status>'
	
Examples:
	| type		| status		|
	| valid		| successfully		|
	| invalid	| unsuccessfully	|

Cucumber reads the feature files in a specified location and runs the tests with the specified tags. So, for example, if we placed Listing 2 in a directory called MockApp in the test resources folder, a Java test class like the one shown in Listing 3 would read it:

Listing 3. An example Java test class
@RunWith(Cucumber.class)
@Cucumber.Options(
        features = "MockApp",//path to the features
        format = {"json:target/integration_cucumber.json"},//what formatters to use
        tags = {"@run"})//what tags to include(@)/exclude(@~)

public class RunTests {
}

If you were using Maven, you would also need to add the dependencies in Listing 4 to the POM:

Listing 4. A Maven POM with Cucumber's dependencies
<dependency>
	<groupId>info.cukes</groupId>
	<artifactId>cucumber-core</artifactId>
	<version>${cucumber.version}</version>
</dependency>
<dependency>
	<groupId>info.cukes</groupId>
	<artifactId>cucumber-java</artifactId>
	<version>${cucumber.version}</version>
</dependency>
<dependency>
	<groupId>info.cukes</groupId>
	<artifactId>cucumber-junit</artifactId>
	<version>${cucumber.version}</version>
</dependency>

Interpreting feature files

Note that we haven't yet told the test what the lines in the feature files mean. To do this, we need to create another class in the same package as the class we created in Listing 3. For instance, the class in Listing 5 would interpret the feature files we created in Listing 2 and write out statements to the log as it goes through each step:

Listing 5. LoginSteps
public class LoginSteps {
	private static final Logger LOGGER = Logger.getLogger(LoginSteps.class.getName());
	
	@Given("^I navigate to the mock application$")
	public void given_I_navigate_to_the_mock_application(){
		LOGGER.info("Entering: I navigate to the mock application");
	}
	
	@When("^I try to login with '(.+)' credentials$")
	public void when_I_try_to_login(String credentialsType){
		LOGGER.info("Entering: I try to login with " + 
			credentialsType + " credentials");
	}
	
	@Then("^I should see that I logged in '(.+)'$")
	public void then_I_login(String outcome){
		LOGGER.info("Entering: I should see that I logged in " + outcome);
	}
}

Each method is annotated with a Given, When, or Then, which contains a regular expression matching the lines in the feature file.

If you try running the Cucumber test, you should see that it passes. This is because we haven't yet written the test code, which we do in the next section.


Driving the browser with Selenium

Now we have some tests written, our next step is to drive them through a web browser. For this, we use Selenium, a web browser automation framework that pairs well with Cucumber. There are two ways to drive a browser with Selenium:

  1. Selenium-RC uses JavaScript to drive the web page and runs inside the JavaScript sandbox.
  2. WebDriver uses native automation, which is faster and less prone to errors but is supported by fewer browsers.

We use WebDriver to test our demo app.

Opening a browser instance

If you are using Maven to build your application, start by adding the following dependency to your POM:

<dependency>
	<groupId>org.seleniumhq.selenium</groupId>
	<artifactId>selenium-java</artifactId>
	<version>${selenium.version}</version>
</dependency>

Next, we create a class called BrowserDriver, along with methods to get and close the browser, as shown in Listing 6:

Listing 6. BrowserDriver
private static WebDriver mDriver;
	
public synchronized static WebDriver getCurrentDriver() {
	if (mDriver==null) {
		try {
                	mDriver = new FirefoxDriver(new FirefoxProfile());
	        } finally{
	        	Runtime.getRuntime().addShutdownHook(
					new Thread(new BrowserCleanup()));
	        }
	}
	return mDriver;
}

private static class BrowserCleanup implements Runnable {
	public void run() {
		LOGGER.info("Closing the browser");
		close();
	}
}

public static void close() {
	try {
		getCurrentDriver().quit();
		mDriver = null;
		LOGGER.info("closing the browser");
	} catch (UnreachableBrowserException e) {
		LOGGER.info("cannot close browser: unreachable browser");
	}
}

This example creates a driver for the Firefox web browser, but you could replace it with a driver for any browser supported by Selenium. We also added a shutdown hook to the driver. This ensures that the browser will automatically close when the test has finished, regardless of whether it has passed or failed.

Next, we need methods for the browser to do something useful. Listing 7 is a method to load a page, which we add to the BrowserDriver class from Listing 6:

Listing 7. Loading a page
public static void loadPage(String url){;
	LOGGER.info("Directing browser to:" + url);
	getCurrentDriver().get(url);
}

Connecting test code to the browser

Now we want to hook BrowserDriver's methods into our test. We create a new class called Navigation to drive the tests. The Navigation class also delegates tasks to page containers, which we discuss in the next section.

To drive a test, the Navigation class needs to have methods that correspond to the methods in that test. For instance, in Listing 8 it uses the following method from LoginSteps (see Listing 5) to load a web page:

Listing 8. LoginSteps methods in BrowserDriver
public void given_I_navigate_to_the_mock_application(){
	BrowserDriver.loadPage(
		"http://localhost/MockApplication/html/MockApplication.html");
}

We are now using the method in Listing 8 to load the web page. Next, we need to extend the LoginSteps class to call the this method. Replace the existing method (from Listing 5) with the one in Listing 9:

Listing 9. new LoginStep
	@Given("^I navigate to the mock application$")
	public void given_I_navigate_to_the_mock_application(){
		LOGGER.info("Entering: I navigate to the mock application");
		navigation.given_I_navigate_to_the_mock_application();
	}

When you run this code, your browser should load the mock application.


Using page objects for RIA

The next step in this process is to confirm that the web page loads successfully. For this, we introduce a variation on the traditional Page Object pattern (see Resources), which we use to carry out page-specific interactions.

Page objects are used to abstract the interactions between a user and a web page into a single object. You could say that this object represents the services that a web page offers. Under the hood, these objects need to know about the DOM elements on a page, and how to interact with them. In our implementation, we call a number of page object "view" classes and separate out the DOM references into "container" classes.

The big difference between our implementation and the usual way of doing page objects is in what a page actually represents. In a traditional web application, after every server request you would get a new page back. Thus, a page object would be the entire web page. In modern, single-page web applications, multiple pages are returned upfront, with sections being hidden and shown. In single-page applications, it doesn't make sense for a page object to represent the whole web page; instead, we make the page object represent a self-contained section of the page.

The sections in our example application are a Login View and a Home View. Although our application only shows one section at a time, a larger application could show multiple sections simultaneously. Furthermore, because a new page is not returned every time, it is challenging to know when a server request has finished. For example, when a user logs in with invalid credentials, you will probably display an error on the screen. To know the request has finished, and test accordingly, you will have to poll for the error display, or until a defined maximum time period has elapsed.

DOM lookups in a view

Let's look at some implementation code. The first thing we do is to create a container class called LoginContainer. This class is used to find the DOM elements of the Login View. For the DOM lookup, we use the @FindBy annotation, which takes parameters specifying what to find. In Listing 10, we use the How parameter for the lookup and using as the lookup variable. How options include CSS, ID, and XPATH.

Listing 10. LoginContainer with DOM lookups
	@FindBy(how = How.ID, using = "LoginPage")
	public WebElement loginPageDiv;
	
	@FindBy(how = How.CSS, using = "#LoginPage input[name=username]")
	public WebElement usernameInput;
	
	@FindBy(how = How.CSS, using = "#LoginPage input[name=password]")
	public WebElement passwordInput;
	
	@FindBy(how = How.CSS, using = "#LoginPage span[role='button']")
	public WebElement submitButton;

Next, we create a LoginPage class, which we use for specific interactions with the login page of our test application.

When the browser is directed to our site, we want to check that the login page is displayed, so we simply edit Navigation's given_I_navigate_to_the_mock_application() method to call isDisplayedCheck() method in a new LoginView class, shown in Listing 11:

Listing 11. Check that the login page is displayed
	public static void isDisplayedCheck(){
		LOGGER.info("Checking login page is displayed");
		BrowserDriver.waitForElement(loginContainer.loginPageDiv);
		loginContainer.loginPageDiv.isDisplayed();
	}

That's it — we've tested whether the login page is displayed when a browser navigates to our site, which was the first scenario in our feature file. Next, we test whether a user can successfully log in.


Logging in with credentials

Looking back at the feature file in Listing 2, we see that the next phase of our test is to see what happens when a user tries to log in to the web page with "<type>" credentials. Type in this case means one of potentially many options, which we've reduced to "valid" or "invalid."

In Cucumber, it is possible to go from a given string type to an enum, but there are limitations to this approach. For instance, if you wanted "correct" and "valid" to mean the same thing, you would need to use separate enum instances. Another option is to do a lookup with an array of Strings as the argument for your enum. Placing such a lookup method in our enum in Listing 12 enables us to have the same meaning for multiple strings:

Listing 12. Credentials type enum
public enum CredentialsType {

   VALID(new String[]{"valid", "correct"}),
   INVALID(new String[]{"invalid"});
	
   private String[] aliases;
	
   private CredentialsType(String[] aliases){
        this.aliases = aliases;
   }

   public static CredentialsType credentialsTypeForName(String credentialsType) 
		throws IllegalArgumentException{
	for(CredentialsType ct: values()){
		for(String alias: ct.aliases){
			if(alias.equalsIgnoreCase(credentialsType)){
				return ct;
			}
		}
	}
	throw credentialsTypeNotFound(credentialsType);
    }

    private static IllegalArgumentException credentialsTypeNotFound(String ct) {
        return new IllegalArgumentException(("Invalid credentials type [" + ct + "]"));
    }
}

Next, we create a user of the specified type. We implement this user as the test's state machine (see Resources) as we're likely to reuse the user credentials later on in the test. For example, we might want to check whether the username is displayed after a user has logged in. Implementing a user factory is more efficient than hard-coding individual user data into the test, which would mean having a user's details in several places. The test user and user factory are shown in Listing 13:

Listing 13. Test user and user factory
public class User {
	private String username;
	private String password;
	
	public User withUserName(String username){
		this.username = username;
		return this;
	}
	
	public String getUsername(){
		return username;
	}
	
	public User withPassword(String password){
		this.password = password;
		return this;
	}
	
	public String getPassword(){
		return password;
	}
	
}

public class Users {
	public static User createValidUser(){
		User user = new User();
		user.withUserName("username").withPassword("password");
		return user;
	}
	
	public static User createInvalidUser(){
		User user = new User();
		user.withUserName("").withPassword("");
		return user;
	}
}

Next, we add the method in Listing 14 to our Navigation class and change the appropriate method in LoginSteps to call it:

Listing 14. Login method for the Navigation class
private User user;

public void when_I_try_to_login(String credentialsType) {
	CredentialsType ct = CredentialsType.credentialsTypeForName(credentialsType);
	switch(ct){
		case VALID:
			//create a valid user
			user = Users.createValidUser();
		break;
		case INVALID:
			//create an invalid user
			user = Users.createInvalidUser();
		break;
	}
	//try to login
	LoginView.login(user.getUsername(), user.getPassword());
}

Our final step is to create the method login into the LoginView class, shown in Listing 15. This method uses the previously created loginContainer (from Listing 10), which tells us how to find the relevant DOM elements. Note that the Selenium wrappers around these elements have several useful methods to allow us to interact with them. The two methods we need are sendKeys, which lets us type a username and password into an input box, and click, which lets us submit login details.

Listing 15. login method for the LoginView class
public static void login(String username, String password){
	LOGGER.info("Logging in with username:"+username+" password:"+password);
	loginContainer.usernameInput.sendKeys(username);
	loginContainer.passwordInput.sendKeys(password);
	loginContainer.submitButton.click();
	LOGGER.info("Login submitted");
}

If you run the test now, you should see the username and password being typed into the appropriate input elements, after which the Submit button is clicked. See the source download included with this article for the complete implementation code.


Testing against different browsers

So far, we have only run our test cases against a Firefox web browser, which Selenium WebDriver supports out-of-the-box, along with Opera, Safari, and Internet Explorer. With a little more work, we can also run our test cases against Chrome, iOS, Android, and Opera Mobile browsers.

We have found it relatively easy to switch browsers by passing in a system property at runtime, such as -Dbrowser=safari. This system property could then be read by the tests using the code in Listing 16:

Listing 16. Specify the test browser
Browsers browser;
WebDriver driver;
		
if(System.getProperty(BROWSER_PROP_KEY)==null){
	browser = Browsers.FIREFOX;
}else{
	browser=Browsers.browserForName(System.getProperty(BROWSER_PROP_KEY));
}
switch(browser){
	case CHROME:
		System.setProperty("webdriver.chrome.driver", 
"src/main/resources/chromedriver");
		driver = new ChromeDriver();
		break;
	case SAFARI:
		driver = new SafariDriver();
		break;
	case FIREFOX:
		default:
		driver = new FirefoxDriver();
		break;
}

Dynamic test data

Maintaining test data is one of the biggest issues we have found with automated testing. When moving a test to a new environment, you don't want to have to set up all of your test data by hand, and if that data changes, you don't want to have to update it for every test case. Dynamically creating the data at the start of the test by getting the test to talk to the relevant database or web services is one way to work around this.

Note that "Browsers" is a stand-in for your own enum for the browser you want to run against.

To run test cases against Chrome, you need to manually download a Chrome driver and set the system property "webdriver.chrome.driver" to the location of the driver, as shown in Listing 16.


Testing and the build environment

It's all very well being able to run tests locally, but that's not much use unless you can connect them to your build. We conclude by showing you how to use Selenium Grid (see Resources) to run multiple tests at once from your build machine.

Start by setting up Selenium Grid on your test machine. Next, pass in a system property such as -Dremote=true, which gives you the option to switch tests from running a single local browser to running against the grid.

Now you only need to update the browser driver. Previously, we created a single local Firefox web driver using the code in Listing 17:

Listing 17. Specify Firefox as your test browser
case FIREFOX:
	driver = new FirefoxDriver();
	break;

Changing the code like that in Listing 18 lets you switch between a local driver and the grid:

Listing 18. Specify a remote test browser
case FIREFOX:
	String isRemote = System.getProperty("remote");
	if("true".equalsIgnoreCase(isRemote);{
		DesiredCapabilities firefox = DesiredCapabilities.firefox();
           firefox.setVersion("17");
           firefox.setPlatform(Platform.ANY);
           driver = new RemoteWebDriver(new URL("http://localhost:4444/wd/hub"), firefox);
	else{
		driver = new FirefoxDriver();
	}
	break;

The URL in Listing 18 should be the address of your Selenium Grid hub.

Running test suites

Once you run and deploy your build, you want to run your tests against it. One option is to run all of your tests as part of a single, large test suite. As you start accumulating tests, however, you might find that this setup consumes too much time. An alternative would be to batch tests in small suites from the start. You can use different annotation tags at the top of your feature files to group tests. For instance, you could use the @login tag to group all of your login tests into a single suite. You would then need an associated Java class like the one in Listing 19 for each suite:

Listing 19. Running a test batch
@RunWith(Cucumber.class)
@Cucumber.Options(
        features = "MockApp",//path to the features
        format = {"json:target/integration_cucumber.json"},//what formatters to use
        tags = {"@login"})//what tags to include(@)/exclude(@~)
public class RunTests {
}

Conclusion

Setting up an automated framework for web application testing is a challenge, and also demonstrably worthwhile in the long run. We found that using Selenium and Cucumber makes the task much easier and results in more maintainable test suites. In this article, we focused on using Selenium and Cucumber to set up an automated test suite for a rich Internet application, including a brief introduction to writing and using feature files and tips for easily testing your application code across different browsers. Advanced planning in the following additional areas can reward your effort over time:

  • Maintaining test data across multiple environments is a predictable pain point of test automation. Connecting your test automation framework to a database or web services infrastructure enables your test cases to dynamically set up required data prior to running.
  • Your feature file interpreter (see Listing 5) can quickly become a very large and messy file. Properly document the steps as you create new tests, so that you don't have to trawl through thousands of lines of test code when something goes awry.
  • Running the tests can take quite some time, especially if you want to run against many different browsers. Map out how many browsers (and thus possibly machines — although multiple browsers can run on the same machine) you need to hook up to your Selenium grid in order for to complete your tests in a reasonable time period.

Download

DescriptionNameSize
Sample codea-automating-ria-src.zip10MB

Resources

Learn

Get products and technologies

  • Cucumber: Write tests in plain text and run them in Java code.
  • Selenium: A test automation suite compatible with most web and mobile browsers.
  • Selenium Grid: Distributes automated tests across multiple browsers.

Discuss

  • Get involved in the My developerWorks community. Connect with other developerWorks users while exploring the developer-driven blogs, forums, groups, and wikis.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Select information in your profile (name, country/region, and company) is displayed to the public and will accompany any content you post. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into DevOps on developerWorks


  • developerWorks Labs

    Experiment with new directions in software development.

  • DevOps digest

    Updates on continuous delivery of software-driven innovation.

  • JazzHub

    Software development in the cloud. Register today and get free private projects through 2014.

  • IBM evaluation software

    Evaluate IBM software and solutions, and transform challenges into opportunities.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=DevOps, Open source, Mobile development
ArticleID=939418
ArticleTitle=Automated testing with Selenium and Cucumber
publish-date=08062013