Functional testing for Web applications
Using Selenium, Windmill, and twill to test Google App Engine applications
As applications move further away from an individually hosted model into the cloud, reliability and predictability become even more important. The cloud moves many factors outside of our control, so having solid, tested code is more important than ever before.
Most developers, whether they test their code or not, have at least gotten a lecture about testing code at some point. Web developers—more so than most developers—need to deliver applications quickly, so unit testing often takes a backseat to a deadline. In some circles, it is never all right to skip unit testing any code, as a unit test tests actual components of the application and provides a way of explaining the internal workings of the code to other developers. Functionally testing your Web application is quite a different story though, and for some reason hasn't gotten as much of a buzz.
Functional testing with twill
I'll start the discussion of functional testing with the lightweight command-line Web browser and scripting tool, twill, and a default Google App Engine project.
The first thing you do is establish a connection to your application. To do
that, use the
go command, as shown in Listing 1.
Note, if you enter
it then shows the actual output.
Listing 1. Sample show output
# twill-sh -= Welcome to twill! =- >> go localhost:8087 ==> at http://localhost:8087 current page: http://localhost:8087 >> show Hello World! current page: http://localhost:8087
Another interesting feature of twill is its ability to check
Listing 2. Sample twill output of http status codes.
> go http://localhost:8087 ==> at http://localhost:8087 current page: http://localhost:8087 >> code 200 current page: http://localhost:8087 >> code 400 ERROR: code is 200 != 400 current page: http://localhost:8087
As you can see from the output of the command, it only returns an error if it
gets a status code that doesn't match what is expected. Twill also supports
the ability to run these actions as a script. You can name the file anything
you want and pass it to
twill-sh. If you place those commands in a file called
test_twill.script, you will see something like Listing 3.
Listing 3. More sample twill output
# twill-sh test_twill.script >> EXECUTING FILE test_twill.script AT LINE: test_twill.script:0 ==> at http://localhost:8087 AT LINE: test_twill.script:1 -- 1 of 1 files SUCCEEDED.
Functional testing with Selenium
Unfortunately, a unit test does not catch this type of bug. In fact, these bugs often make a Web developer skeptical of doing any testing in the first place. They figure that testing is yet another hoop to jump through, that it gets in the way of their already tight deadline, and it doesn't even reliably work. So why bother?
Selenium (as well as other browser testing tools) is an answer to this problem. You can write functional tests that run in each browser, and then implement some form of continuous integration system that runs the functional tests upon each check-in of the source code. This allows potential bugs in the browser to quickly get caught, and has an immediate payoff.
One of the most basic things Selenium does is record browser actions so they can be replayed as a test. Looking at the sample in Figure 1, you see a window where you place a base url. From there you simply record the actions of the Web site you are testing. In this case, I am testing a Google App Engine site: http://shell.appspot.com, which is a demo Ajax Python interpreter. After the session has been recorded, you can then export the tests in Python and run them using Selenium.
Figure 1. Selenium IDE window
To run the tests, simply save your tests as Python code (or whatever language you choose), download and run the Selenium RC test server, and then run your tests. Listing 4 shows an example of what that test might look like.
Listing 4. Example Selenium test
from selenium import selenium import unittest, time, re class NewTest(unittest.TestCase): def setUp(self): self.verificationErrors =  self.selenium = selenium("localhost", 4444, "*chrome", "http://shell.appspot.com") self.selenium.start() def test_new(self): sel = self.selenium sel.open("/") sel.click("link=source") sel.wait_for_page_to_load("30000") def tearDown(self): self.selenium.stop() self.assertEqual(, self.verificationErrors) if __name__ == "__main__": unittest.main()
You can launch Selenium RC, which acts as a proxy for testing, on multiple browsers, and then run your functional test. Listing 5 shows what the output of Selenium RC looks like.
Listing 5. Sample of Selenium RC output
# java -jar selenium-server.jar 01:18:47.909 INFO - Java: Apple Inc. 1.5.0_16-133 01:18:47.910 INFO - OS: Mac OS X 10.5.6 i386 01:18:47.915 INFO - v1.0-beta-1 ,  01:18:48.044 INFO - Version Jetty/5.1.x 01:18:48.045 INFO - Started HttpContext[/,/] 01:18:48.047 INFO - Started HttpContext[/selenium-server] 01:18:48.047 INFO - Started HttpContext[/selenium-server/driver] 01:18:48.055 INFO - Started SocketListener on 0.0.0.0:4444 [output suppressed for space]
I encourage you to read the full Selenium RC FAQ to understand how it interacts with multiple browsers (see Related topics). As you can see, using Selenium to automate cross-platform functional tests is quite easy, and there is support for many languages including plain HTML, Java™ code, C#, Perl, PHP, Python, and Ruby.
To get started using Windmill, simply run the command:
easy_install windmill. This installs the windmill testing framework. Next,
if you enter
windmill firefox, the Windmill IDE
opens (see Figure 2). Then the testing page opens, as shown in Figure 3. In Figure 2 you can see that the IDE records
actions much like Selenium. You can then save the test file, which looks
like the output in Listing 6.
Figure 2. Windmill IDE window
Figure 3. Windmill tutorial
Listing 6. Sample test file output
# Generated by the windmill services transformer from windmill.authoring import WindmillTestClient def test_recordingSuite0(): client = WindmillTestClient(__name__) client.click(id=u'recordedClickId') client.click(id=u'textFieldOne') client.type(text=u'foo bar', id=u'textFieldOne') client.click(id=u'btnSub')
There is an option to save a test as JSON or Python, and in this case I save it as Python. Next, to actually run the test, you simply run the test file from the command line with the test option. Listing 7 shows how this looks.
Listing 7. Running the test
# windmill firefox test=windmill_test.py http://tutorial.getwindmill.com/ Started ['/Applications/Firefox.app/Contents/MacOS/firefox-bin', '-profile', '/var/folders/1K/ 1KgyCzqJHButzT6vq8vwHU+++TI/-Tmp-/tmp_ovtnN.mozrunner', 'http://tutorial.getwindmill.com/windmill-serv/start.html'] Server running...
You can easily adapt this example to your Google App Engine application or any other Web application, for that matter. Windmill also has great documentation that explains how to do more advanced tasks such as extending windmill with plug-ins.
Functional testing is essential to the process of Web development. Without functional testing, Web development becomes a guessing game that can be filled with frantic, error-prone deployments and refactoring.
So, should functional testing be mandatory for all Web developers? I would argue, yes, all Web applications should be tested, especially if they are destined for a cloud environment. Lack of functional testing for Web applications is a definite red flag, considering how easy it is to doâat least a minimum of testingâwith Selenium, Windmill, or twill. To borrow a tag line from the author of twill, Dr. Titus Brown, "if you don't test your code, how you can say it works?".
- Python For Unix and Linux System Administration: Use this book to develop your own set of command-line utilities with Python to tackle a wide range of problems.
- "Getting Started With Google App Engine:" This article gives you a good introduction to the Google App Engine.
- Google App Engine in Action: Use this book to create applications with a custom feel to them using Python.
- Google App Engine: Read the official documentation.
- Hello World Google App Engine: Read a more detailed article on getting a "Hello World" application running from scratch.
- Python Tutorial: Take this tutorial to learn the basic concepts and features of the Python language and system.
- Selenium RC Server: Get more information on the Selenium RC Server.
- twill: Get more information on the twill scripting language.
- Windmill- Test Automation : Find out more about the Windmill testing framework.
- An Introduction to Testing Web Applications With twill and Selenium: Use this resource to quickly start writing your own functional tests.
- developerWorks Cloud Computing space: Discover why cloud computing is important, how to get started, and where to learn more about it.