The case for model-based testing

Most software testing begins with a test engineer sitting down at the computer, starting the application, randomly navigating its features, searching for bugs, and tripping across crashes. Since the test engineer knows (or learns while testing) what the application is supposed to do, he can file a bug when the application doesn't get it right. In addition, he'll file bugs for all the crashes he's able to reproduce. Eventually, the test engineer generates enough test cases that he wants to remember to rerun on the next build or release that he's able to write a comprehensive test plan. The core of the test plan is the description of all the test cases to run on each build.

After running through the test plan a few times, the good test engineer decides that the computer can do a faster, more accurate job of running some of these test cases, so he automates as many as he can. He might then logically sequence these test cases and run through the entire use case, checking for errors and crashes. This automation forms the basis of his regression test. The good engineer might even data-drive the test cases so that the same logic can be reused against large sets of input and expected-result data. I'll refer to this type of automation as "classic test automation."

One problem with this evolution is that the random sequencing that was implicit (and probably unconscious) in the manual testing is gone. As soon as the application can run through the sequence of test cases with no apparent errors or crashes, it's shipped. Now the customer, having just purchased and installed the application, uses functions in an order that differs from the automation and finds some nasty bugs. The customer calls support, support reports the sequence to development, and development fixes the bug. The test engineer, in reviewing the bug, sees what he missed in his automation, adds this new sequence to his tests, and is happy -- at least until the next customer tries the same functions in yet a different order and breaks the application in yet a new way. Right about now, management is probably thinking that maybe an army of manual testers wasn't such a bad idea.

There is, however, another approach. Imagine that the test automation knew where the current feature (or screen) being tested lived in the application relative to the other features (or screens). Imagine further that it knew all of the possible ways to transit out of the current feature and where in the application it's supposed to be after this transition. Based on this behavioral model, the automation could execute the current feature, check for the correct results, pick a transition at random, end up in a new feature, test this new feature, and randomly pick yet another transition. This process could then be repeated indefinitely, bringing back the random testing that was lost when the human was removed from running the tests. The term commonly used in the software test industry for this type of testing is model-based testing.

Model-based testing is a form of automated testing that implements random behavior in your test automation software. Each application feature (or screen) becomes a "state," and each button, link, menu selection, or command leading to the next function becomes a "transition." Your application's model is a memory-based representation of each state, all of the possible transitions, and the destination state that should result when a particular transition is selected. In a Web or graphical Java application, a screen or message box usually corresponds to a state. All the buttons, links, tabs, and other objects that take the user to another state or screen are the transitions. Validation of the business or application logic at each state is achieved in the same way as in classic test automation.

By applying various navigation algorithms to the model (for example, randomly choose an unvisited state, visit all states in order), the test exercises the application in new ways or more complete ways than in classic automation. In addition, with model-based testing, adding computers increases the test coverage over a given time by generating even more paths through the code. In classic automation, no matter how many times you run through the automation it's always the same. In model-based testing, since the navigation can be random, computers running in parallel to each other will be executing different paths through the code. In Java or Web applications where many clients run against a single server, this method is a valuable tool to test multiuser access as well.

Back to top

Model-based testing for Java or Web applications

Model-based testing can be used for many types of software or application testing. The concepts remain the same whether you're testing a cell phone's embedded system or a browser-based Internet application. There's a problem, however, when you try to implement model-based testing for Java or Web applications. Most of the legacy test tools are designed primarily for script recording and playback. Although the logic can be hand-manipulated and tuned by directly editing the source code, the native languages of these tools don't allow for easy declaration and manipulation of large-scale data structures, and object-oriented design is nearly impossible.

These issues are solved with Rational Functional Tester for Java and Web, because Java itself is the test script language. This isn't some subset of Java or any "Javalike" language -- it's the real thing. It runs on a variety of JVMs and is implemented as a plug-in of the Eclipse® 3.0 IDE and runtime environment. The test engineer, therefore, has all of the programming power afforded to application programmers. This environment lends itself to highly engineered, highly scalable, flexible, and reliable test automation software.

To take advantage of the power of model-based testing, you need a level of skill beyond that required of the normal manual tester or automation script writer. You need a good foundation in software engineering as well as a sound basis in Java programming. The implementation described below, although written entirely in Rational Functional Tester for Java and Web, makes heavy use of native Java concepts. You can grasp the value of model-based testing without detailed knowledge of Java, but to adapt the implementation to a real-world application, you'll need to have a high level of comfort with Java programming techniques.

Back to top

Implementation details

Now let's consider the details of how to implement model-based testing with Rational Functional Tester. I'll just touch on the highlights here. For a good tutorial on model-based testing, see "Intelligent Test Automation" by Harry Robinson. Take a look at Figure 1. It shows a simple Web portal to illustrate a browser-based application consisting of six tabs. Each tab brings you to a new screen. Each screen is a state. Since all six tabs are available from each other, you can navigate from any state to any other by clicking its associated tab. Each tab becomes a transition to a new state.

Figure 2 illustrates a three-state model using just three of the tabs -- Home, Images, and News. Each screen is a state. From the Home state, three transitions are shown -- Home, News, and Images. A transition from a state is shown as an arrow leading out of a state. Each arrow points to its corresponding destination state. This model shows that when you click the News tab, you should transition to the News screen; the Images tab should take you to the Images screen, and the Home tab should return you to the Home state. Similar information is stored for the Images and News states. These states, transitions, and destinations form the model of the application that we can test against.

Resources

Learn

• High Availability Disaster Recovery (HADR): Learn about the feature in DB2 UDB V8.2 that provides a high-availability solution for protection against both partial and complete site failures.
  
  
• Pyrex Web site: Find Pyrex manuals and a tutorial to help get you started.
  
  
• " Charming Python: Beat spam using hashcash" (developerWorks, 2004): Learn more about David's Mertz' pure-Python implementation of hashcash.
  
  
• All Charming Python articles: For more on Python, browse all installments of David's column on developerWorks.
  
  
• developerWorks Linux zone: Find more resources for Linux developers.
  
  

Get products and technologies

• IBM DB2 Universal Database Enterprise Server Edition V8.2 for Linux: Download a trial version from developerWorks.
  
  
• IBM trial products for download: Build your next development project on Linux with IBM trial software, available for download directly from developerWorks.
  
  

Discuss

• Participate in the discussion forum.
  
  
• KernelNewbies.org: If you are new to hacking the kernel, find lots of resources including an IRC channel, a mailing list, and a wiki.
  
  
• developerWorks blogs: Get involved in the developerWorks community.
  
  

About the author

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• The case for model-based testing
• Model-based testing for Java or Web applications
• Implementation details
• Resources
• About the author
• Comments

Next steps from IBM

• Try: Rational Functional Tester
• Webcast: IT performance management: Measure, monitor and modify for operational excellence
• Briefing: Jazz and Rational Team Concert
• Buy: Rational Functional Tester

My developerWorks community

Dig deeper into Rational on developerWorks

Try Rational System Architect

See how this fully integrated collection of models and documents can help you better align your IT systems and infrastructure to simplify business processes

Special offers