 | Level: Intermediate Jeff Feldstein (jbf@sherpas.com), Manager, Software Development, Cisco Systems
07 Oct 2005
Updated 28 Nov 2005 The classical method of test automation takes an application through the same sequence of test cases until all the bugs have been fixed. The problem with this method is that a customer who uses the functions of the program in a different order may run into bugs the test automation never encountered. Model-based testing, on the other hand, generates tests at random from a description of the application's behavior and thus has the advantage of testing every possible sequence a customer might try. If you implement model-based testing with IBM® Rational® Functional Tester for Java® and Web, you have the power of Java programming at your fingertips to apply to your automation effort.
In this article we'll first explore the case for model-based testing (not to be confused with the development approaches of model-driven development and model-driven architecture, which use UML) and then look at the basic data structures, logic flow, and navigation algorithms required for model-based testing of Java and Web applications using Rational Functional Tester. A working version of a model-based test system and all of its source code is available from my model-based testing Web page. I'll be showing you parts of that here to illustrate how model-based testing works. I'd suggest that you download and run the model after reading the article. You can also download an evaluation version of Rational Functional Tester for Java and Web if you want to take it for a test drive.
related developerWorks content.
USE LINK EXAMPLE FOR LINKS IN extended abstract
The case for model-based testing
Most software testing begins with a test engineer sitting down at the computer, starting the application, randomly navigating its features, searching for bugs, and tripping across crashes. Since the test engineer knows (or learns while testing) what the application is supposed to do, he can file a bug when the application doesn't get it right. In addition, he'll file bugs for all the crashes he's able to reproduce. Eventually, the test engineer generates enough test cases that he wants to remember to rerun on the next build or release that he's able to write a comprehensive test plan. The core of the test plan is the description of all the test cases to run on each build.
After running through the test plan a few times, the good test engineer decides that the computer can do a faster, more accurate job of running some of these test cases, so he automates as many as he can. He might then logically sequence these test cases and run through the entire use case, checking for errors and crashes. This automation forms the basis of his regression test. The good engineer might even data-drive the test cases so that the same logic can be reused against large sets of input and expected-result data. I'll refer to this type of automation as "classic test automation."
One problem with this evolution is that the random sequencing that was implicit (and probably unconscious) in the manual testing is gone. As soon as the application can run through the sequence of test cases with no apparent errors or crashes, it's shipped. Now the customer, having just purchased and installed the application, uses functions in an order that differs from the automation and finds some nasty bugs. The customer calls support, support reports the sequence to development, and development fixes the bug. The test engineer, in reviewing the bug, sees what he missed in his automation, adds this new sequence to his tests, and is happy -- at least until the next customer tries the same functions in yet a different order and breaks the application in yet a new way. Right about now, management is probably thinking that maybe an army of manual testers wasn't such a bad idea.
There is, however, another approach. Imagine that the test automation knew where the current feature (or screen) being tested lived in the application relative to the other features (or screens). Imagine further that it knew all of the possible ways to transit out of the current feature and where in the application it's supposed to be after this transition. Based on this behavioral model, the automation could execute the current feature, check for the correct results, pick a transition at random, end up in a new feature, test this new feature, and randomly pick yet another transition. This process could then be repeated indefinitely, bringing back the random testing that was lost when the human was removed from running the tests. The term commonly used in the software test industry for this type of testing is model-based testing.
Model-based testing is a form of automated testing that implements random behavior in your test automation software. Each application feature (or screen) becomes a "state," and each button, link, menu selection, or command leading to the next function becomes a "transition." Your application's model is a memory-based representation of each state, all of the possible transitions, and the destination state that should result when a particular transition is selected. In a Web or graphical Java application, a screen or message box usually corresponds to a state. All the buttons, links, tabs, and other objects that take the user to another state or screen are the transitions. Validation of the business or application logic at each state is achieved in the same way as in classic test automation.
By applying various navigation algorithms to the model (for example, randomly choose an unvisited state, visit all states in order), the test exercises the application in new ways or more complete ways than in classic automation. In addition, with model-based testing, adding computers increases the test coverage over a given time by generating even more paths through the code. In classic automation, no matter how many times you run through the automation it's always the same. In model-based testing, since the navigation can be random, computers running in parallel to each other will be executing different paths through the code. In Java or Web applications where many clients run against a single server, this method is a valuable tool to test multiuser access as well.
Model-based testing for Java or Web applications
Model-based testing can be used for many types of software or application testing. The concepts remain the same whether you're testing a cell phone's embedded system or a browser-based Internet application. There's a problem, however, when you try to implement model-based testing for Java or Web applications. Most of the legacy test tools are designed primarily for script recording and playback. Although the logic can be hand-manipulated and tuned by directly editing the source code, the native languages of these tools don't allow for easy declaration and manipulation of large-scale data structures, and object-oriented design is nearly impossible.
These issues are solved with Rational Functional Tester for Java and Web, because Java itself is the test script language. This isn't some subset of Java or any "Javalike" language -- it's the real thing. It runs on a variety of JVMs and is implemented as a plug-in of the Eclipse® 3.0 IDE and runtime environment. The test engineer, therefore, has all of the programming power afforded to application programmers. This environment lends itself to highly engineered, highly scalable, flexible, and reliable test automation software.
To take advantage of the power of model-based testing, you need a level of skill beyond that required of the normal manual tester or automation script writer. You need a good foundation in software engineering as well as a sound basis in Java programming. The implementation described below, although written entirely in Rational Functional Tester for Java and Web, makes heavy use of native Java concepts. You can grasp the value of model-based testing without detailed knowledge of Java, but to adapt the implementation to a real-world application, you'll need to have a high level of comfort with Java programming techniques.
Implementation details
Now let's consider the details of how to implement model-based testing with Rational Functional Tester. I'll just touch on the highlights here. For a good tutorial on model-based testing, see "Intelligent Test Automation" by Harry Robinson.
Take a look at Figure 1. It shows a simple Web portal to illustrate a browser-based application consisting of six tabs. Each tab brings you to a new screen. Each screen is a state. Since all six tabs are available from each other, you can navigate from any state to any other by clicking its associated tab. Each tab becomes a transition to a new state.
Figure 1: A simple Web portal with six tabs = six states
Figure 2 illustrates a three-state model using just three of the tabs -- Home, Images, and News. Each screen is a state. From the Home state, three transitions are shown -- Home, News, and Images. A transition from a state is shown as an arrow leading out of a state. Each arrow points to its corresponding destination state. This model shows that when you click the News tab, you should transition to the News screen; the Images tab should take you to the Images screen, and the Home tab should return you to the Home state. Similar information is stored for the Images and News states. These states, transitions, and destinations form the model of the application that we can test against.
Figure 2: A three-state model of the Web portal
Table 5. Table using a caption tag, alternating color rows
| From State | Transition | Target State |
|---|
| Home screen | Home screen | Home screen | | News tab | Images tab | Home tab | | News screen | Images screen | Home screen |
- List item 1
- List item 2
- List item 3
Use monospace code font for names of commands,
such as the make install command, as well as for
classes, objects, methods, code snippets, and any text the user types in.
You can also use bold and italics for emphasis or clarity in text.
Another major heading
Code listings are common in developerWorks content.
Listing 1. Caption goes here
<!-- Please limit code lines to 90 characters max. -->
<!-- The following line is the max length: -->
|-------10--------20--------30--------40--------50--------60--------70--------80--------90
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Line of code
Line of code
Line of code
Line of code
Line of indented code
Line of indented code
Line of code with <angle brackets>
Line of code with an ampersand (&)
Line of code with something in bold
|
Minor heading
|
When to use a sidebar
Use a sidebar for a content detour or reinforcement. For example, you might
define terms or provide a bit of historical background in a sidebar. Or you
might summarize the key points of a lengthy discussion. For a sidebar in action,
see the article "Open
source development using C99".
|
|
Technical graphics such as screen captures, diagrams, and photographs can enhance your
content. See "Illustrating
your article or tutorial for developerWorks" to learn how to create and deliver effective
graphics.
To keep readers engaged with your content, and not wandering away mid-article to pursue
non-developerWorks content, try to minimize off-site links in the body of your article.
Instead, in the body, link to the Resources section and include
related content (articles, tutorials, developerWorks views, technical briefings, Webcasts,
demos, books, Web sites, or offers) and urls there. Likewise, for code (downloads, demos,
etc.), don't include the urls in the body; rather, link to the Downloads
section and include the urls there.
Table examples
Use any of the following styles for tables in your article. Table 1 uses a <caption> tag
and left-aligned columns, and is the recommended style.
Download
Resources Learn
Get products and technologies
Discuss
About the author | | | Jeff Feldstein has been a software developer, tester, and manager for 24 years. Jeff's current position is at Cisco Systems, managing a team of 38 test engineers who test an internally developed operating system used by Cisco's network management products. A frequent speaker at test and user conferences throughout the United States, Jeff is a strong advocate of applying the principles of software engineering employed in the best applications to the test automation of those systems. |
Rate this page
| |
