Many times, I start automating without thinking of the big picture. It's just so easy to do. You have the tools, the reusable source code from the last project, and you know what needs to be automated. It seems so simple. "All I need to do is what I did last time (assuming last time was successful)." Well, that may be true, or it may not. Depending on the differences between the two projects and the software under test, past success may not be an accurate indicator. In addition, the risks for this project are most likely not the same as they were in the previous project(s). These differences should indicate to me that I need to slow down and re-examine the context in light of what I know.

One of the primary concerns when I start looking at the big picture for a project is whether I want to implement my scripts independent of one another, or dependent on one another. The issue of dependence can be quite large. It will affect how many scripts I have, how complicated (maintainable) they are, and how robust they are. Moreover, the strategy will also most likely influence what types of tests I automate (this can be a direct influence or an indirect influence). These issues are non-trivial, and the decision for independence or dependence often precedes other implementation issues (like test frameworks).

This article we will closely examine the topic, and will look at some different ways that IBM® Rational® tools play into the different paradigms. This article is targeted at readers with past automation experience, who may be able to draw on experience as we talk about some of the issues. This is not to say a beginner won't get value out of the article, just that it will be difficult to relate to some of the concerns unless you have dealt with them in the past.

A look at test script dependence and independence

Test script dependence refers to ways in which one test script relies on another test script to do some essential task. Commonly, this dependence falls into one of two categories: dependence on another test script for application data, or dependence on another test script for application state.

• Scripts that are application data dependent rely on some other script to create or maintain the data they depend on. That is, they use data that was not created or modified within the script itself.
• Scripts that are application state dependent rely on some other script to navigate them to a certain point in the application before turning control. That is, they do not start at the beginning of the application, but instead somewhere in the middle of the application. A different script got them to that point.

Typically, with script dependence, order is important. Because any given script depends on the data or application state changes of the script executed before them, they have to be run in the order in which they were intended. If you run the scripts out of order, your scripts will most likely fail due to missing or incorrect data, or invalid application states.

Test script independence is not exactly "just the opposite" of script dependence, but it's close. Script independence is where all of your scripts are completely self-sufficient. No test script relies on another test script to do some essential task. Typically, with script independence, order is not important, and any given script can be executed at any given time. There are a couple of additional issues to look at with independent test scripts, which this article will discuss later.

Let's take a quick look at each type of dependence. Let's pretend you are testing a typical e-commerce website (search, add to cart, checkout, check order status, and so on). For an application such as this, you might create test cases in the following two ways:

An example of data dependence:

• Test case one tests placing an order.
• Test case two tests checking an order's status (using the order placed in test case one).
• Test case three fulfills the order placed in test case one.
• Test case four tests checking an order's status again (using the order placed in test case one).
• And so on.

In this example, we have three (maybe four) distinct tests that share the same data. The tester here reasons that since one must place an order before checking an order's status, "Why shouldn't I use an order from a previous test?" In this example, the data is the only thing that's shared between the scripts. All scripts start with a new browser, and close the browser when they are done.

An example of application state dependence:

• Test case one opens the website and verifies that the home page loads with all the correct elements.
• Test case two performs a search and verifies that the correct results are returned.
• Test case three adds an item to the cart and verifies that the cart updates correctly.
• Test case four submits the order and verifies that the order is received correctly, and that the customer is billed correctly.
• And so on.

This example takes the overall scenario of placing an order and breaks it down into bite-size steps. Each test script looks for specific functional requirements as it executes, ignoring the larger picture of the test scenario that's being run. The tester here reasons, "Each set of requirements needs to be verified in its own script, and then using those scripts in conjunction will allow me to create more complex test scenarios." In this example, the state is the only thing that's shared between the scripts. The data is independent of the tests taking place.

An example of independent test scripts:

• Test case one tests three search scenarios.
• Test case two places two orders (one large, one small).
• Test case three places an order and then checks the order status.
• Test case four adds and removes items from the shopping cart, checking for correct amount calculations.
• And so on.

In this example, some test scripts look for specific functional requirements and others look at the larger picture of test scenarios. Nothing is shared between the scripts. The data is independent of the tests taking place (with the exception of the inventory on the website), and all scripts start with a new browser and close the browser when they are done.

Back to top

Advantages and disadvantages of both methods

At this point, independent test script might seem like the clear choice, so why create dependent scripts? Typically, script dependence results in fewer test scripts, increased modularity, and shorter test scripts (less code due to reuse). This often leads to a decreased cost of code maintenance, as there is less code and more reuse. For example, suppose you only have one script that processes login, and all of the other scripts are application state dependent on the first script to log them in to the application. In this case, if the login changes or breaks, you only need to update that one script (not all your scripts that log in).

This brings up the classic argument against record and playback technology. The more independent scripts you have, the more they are affected by application changes, increasing your costs associated with script maintenance. Script dependence attempts to reduce this cost. By sharing application data, you typically get fewer test cases. By sharing application state, you typically get more robust test cases due to modularity. That is, with script dependence, you typically have fewer scripts and a lower cost of code maintenance.

In addition, some applications are designed to function like an application wizard (certain screens or fields must be completely processed before advancing to other screens in the wizard). It can be argued that in this context, it makes sense to modularize screens (or portions of the application) as they will always be processed in the same order. Furthermore, using other techniques, such as data-driven testing (check out "Choosing a test automation framework"), you can make your modularization even more powerful.

On the other hand, script dependence is not without its problems. Often, dependent scripts can result in blocking bugs. A blocking bug is a defect or issue that is preventing you from doing something else until it gets fixed. With dependent test scripts, script bottlenecks can occur. What happens if a script fails due to a screen change, but the majority of the scripts you have use that script? For example, if the script you use to process the shopping cart fails because the developers changed the way items in the shopping cart are displayed, why should your check order status test case care about that? Can't it just click the order button in the shopping cart without validating the table format? That's not what it's testing. With dependence, if the first script fails, all scripts following that script will likely fail.

With dependent scripts, application state and data sharing can become an issue. If a script fails, what does that mean for all the scripts that rely on it? How do you check order status if an order was never placed? How do you add an item to the shopping cart if you never performed a search? The complexity of sharing data and state can make script debugging more difficult, and it often takes longer to figure out if a script failure is a problem with the test script or the application under test. These problems make reliability an issue. In general, independent test scripts are more reliable than dependent test scripts.

These comparative advantages are illustrated in Figure 1.

In most cases, this is the most basic tradeoff you make when you choose between script dependence and independence. This is not the only factor in this equation, but fundamentally, it’s an important one. Regardless of what method you choose, there are many ways to make dependent scripts more reliable (automating "behind" the GUI for example) and independent scripts more maintainable (having better object recognition, like IBM® Rational® ScriptAssure™ technology).

However, your choice regarding whether you use dependent or independent test scripts will affect more than just your script reliability and your maintenance costs. It will also affect what types of test scripts you create. In their book Lessons Learned in Software Testing, Kaner, Bach, and Pettichord talk about bias and how it affects our testing (see the Resources section). The method of implementation you choose will most likely form a bias, causing you to create some types of tests instead of others. The research of the authors would indicate that when testing, you tend to favor the tests that readily come to mind and are easiest for you to implement. This is not because you are lazy; most of us are not even aware we are doing it.

For example, you may or may not be familiar with high-volume random regression testing. If not, take some time to get familiar with the concept. I recommend "High Volume Test Automation" by Kaner, Bond, and McGee. Read it. It's really worth your time. If you use dependent scripts, you can no longer do this type of testing (or at the very least it's much harder). With dependent test scripts order is often important, and in high-volume random regression, as the word random implies, order is fairly dynamic. Thus the decision to use dependent test scripts has limited your ability to do a specific type of testing.

Another example of your choice influencing your testing might be performance testing. How does performance testing influence your functional test scripts, you ask? Well, take a look at this example where I used everyday automated scripts to gather performance information. If you have independent test scripts, you are much less likely to incorporate this type of testing, as the costs of implementation are much higher than if you already have a high level of reuse in your scripts. In this case, the decision to use independent scripts has affected your ability to make cascading changes to your suite of test scripts.

I hope by this point I have adequately shown that in most cases, the choice you make between script dependence and independence is not trivial. In some cases, the choice is clear based on your context and the software you are testing, but most of the time it's not. Many times, it's easiest to just look at your context, think about how you want to use your scripts in the future, and then create the scripts using the model that makes the most sense for the testing you are trying to accomplish at the time with the future in mind.

Back to top

How do IBM Rational tools fit into all of this?

Let's look at some specific examples of using Rational tools to either take advantage of dependent and independent scripts, or to mitigate some of the challenges of using dependent and independent scripts.

ScriptAssure reduces the cost of script independence

One of the biggest implementation models for independent scripts is record and playback. Regardless of the revulsion against record and playback, it has a place in the test automation toolkit (see "Using Cost-Benefit Analysis to Compare Different Test Structures for Rational Robot" for more on when to use record and playback). In my experience, the largest reason why people don't like record and playback is that the scripts are just unmaintainable for any significant length of time (which is the same reason I don't like record and playback). We identified this same problem for independent test scripts. The more scripts you have, the higher the cost of maintenance.

ScriptAssure (which is a feature in IBM® Rational® Functional Tester) is a set of advanced capabilities for functional and regression testing of Java and Web applications, as well as .NET applications. It can reduce the time and effort you spend on script maintenance tasks. Typically, even minor changes to user interface objects break the scripts used to test an application. To make test scripts resilient, ScriptAssure separates the functionality of interface components from the user interface. As the script designer, you can assign weights to the characteristics of the interface objects -- such as color, position, or logical name -- and the weight tells ScriptAssure whether or not to ignore those types of changes. When you give design characteristics of the user interface low weights, you can test how well the application works without the user interface changes breaking your scripts. This may reduce the cost of script maintenance.

While this technology alone will not make record and playback a long-term solution, it does go a long way to minimize the maintenance requirements for independence test scripts in Rational Functional Tester. When used in conjunction with other good scripting practices, it can be a powerhouse of a feature. If you can lower the cost of maintenance for your suite of scripts, you can shift the maintainability and reliability curve in Figure 1 to the right, giving you greater maintainability for any given level of reliability (sorry for the economic digression).

Datapools add independence to your dependent scripts

Datapools allow for data-driven testing, which adds some independence to your dependent scripts. Data-driven testing is a technique where a single test script is used repeatedly with varying input and response data that comes from a predefined data set. In both IBM® Rational® Robot and Rational Functional Tester, data-driven testing can be implemented via datapools. A datapool is a collection of related data records that supply values for the variables in a test script during script playback. When you use datapools, the script receives a different set of test data each time you play it back. I go into detail about using datapools in "Using IBM Rational Functional Tester to test financial services applications." If you have not used datapools, I would recommend you look there for a detailed example and discussion.

Datapools add independence in that they separate the test case data from the test script code. This can allow you to use the same code for many tests. This can potentially reduce the state dependence of your application. For example, if I'm testing a calculator program, I may want to run the following tests:

• Test case one tests addition.
• Test case two tests subtraction.
• Test case three tests multiplication.
• And so on.

More than likely this will result in a very large number of small tests. Using script independence, You could have each test case script login, set the mode to standard (if it has both standard and scientific modes), execute a test, and then close the calculator. If you use script dependence, you might have a single script open the calculator, another single script set the mode to standard (if it has both standard and scientific modes), and then have each test case script execute back to back (calling a single script to reset the application state between each test case) and then close the calculator when all tests are complete. The dependence here is application state.

With datapools, you can remove the dependence on application state and still have less code to maintain. You create one script that opens the calculator, sets the mode, reads the test from a datapool, executes the test, and closes the calculator. It repeats this cycle for every dataset in the datapool. You only have one script to maintain, and you have no data dependence or application state dependence.

Model-based testing and high-volume random regression testing

Finally, let's look at a couple of really cool examples of script independence and dependence hard at work. Rational Functional Tester is great for model-based testing (model-based testing can also be implemented in Rational Robot -- it's just not as easy as with Rational Functional Tester). In his article "Model-based testing using IBM Rational Functional Tester", Jeff Feldstein defines model-based testing as a test technique that "generates tests at random from a description of the application's behavior and thus has the advantage of testing every possible sequence a customer might try. " The concept is simple, the implementation is where it can get complex (I'll let you read Jeff's article for that). The end goal of model-based testing is a test framework that -- when you execute it -- will randomly generate tests based on the model you supply. In the context of this discussion, these tests are extremely application state and data specific (almost by definition).

On the other hand, if all your scripts are independent (really independent), IBM® Rational® TestManager can be used for high-volume random regression testing. I don't want to steal too much from Kaner, Bond, and McGee, but I should mention the basics of high-volume random regression. The idea is this: you develop all of these regression scripts, each of which is (probably) designed to find specific functional bugs. Once you execute these tests once, their value goes down. If you execute them again, their value goes down again. Every time you execute a regression script, its value decreases. This is because it's not providing any new information about the software you're testing.

One way to give that old script new value is to use it in a new and interesting way. This is where high-volume random regression comes in. If you take all of your not-interesting regression scripts and run them in random order for very long periods (hours, days, perhaps weeks), you might be able to learn new information about the application you are testing. For example, you might discover timing problems, memory corruption and memory leaks, or other very hard-to-find, hard-to-troubleshoot, and hard-to-fix problems. When executed randomly, the test scripts will run in an order most likely different then the order you run them when you execute your regression tests. In addition, if the application is being exercised for a very long period, runtime problems start to become more visible.

Using Rational TestManager, you can create a suite for random regression testing. Simply create a new suite, any number of computer groups (depending on how many scripts you want to run simultaneously), add a Selector set to Random with replacement, and insert your test scripts. An example is shown in Figure 2.

Rational TestManager is great for stuff like this, because it allows you to distribute the testing on multiple machines (which other test management tools don't readily do for you). In addition, it only takes about five minutes to create the suite, meaning you can make custom suites as often as you like.

Back to top

Going forward

Well this should get you thinking about how you organize your scripts, as well as what the implications of that are for your project. There is no right or wrong answer to this equation. It really depends on what you need to do at the time, and what you think you will need to do in the future. Check out some of the resources below for more ideas, and be sure to try both methods before you decide to go with one or the other. Write me and let me know what your findings are!

Resources

• Participate in the discussion forum.
  
  
• In the article, I made reference to test frameworks and better scripting. Check out Choosing a test automation framework or Using Cost-Benefit Analysis to Compare Different Test Structures for Rational Robot by Michael Kelly and Improving the Maintainability of Automated Test Suites by Cem Kaner for more on this topic.
  
  
• For ideas on how to leverage your test scripts for other types of testing, look at High Volume Test Automation by Kaner, Bond, and McGee, Gathering Performance Information While Executing Everyday Automated Tests by Michael Kelly, Model-based testing using IBM Rational Functional Tester by Jeff Feldstein, and Using IBM Rational Functional Tester to test financial services applications by Michael Kelly for more on datapools.
  
  
• For more on recognizing and managing your biases, read chapter two in Lessons Learned in Software Testing by Kaner, Bach, and Pettichord and Maps of Bounded Rationality: A Perspective on Intuitive Judgment and Choice by Daniel Kahneman.
  
  
• Try out the Rational testing products by visiting the trial downloads area.
  
  
• To learn more, visit the Functional Tester product area and the TestManager product area on developerWorks Rational. You'll find technical documentation, how-to articles, education, downloads, product information, and more.
  
  
• Get involved in the developerWorks community by participating in developerWorks blogs.
  
  

About the author

Comments

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• A look at test script dependence and independence
• Advantages and disadvantages of both methods
• How do IBM Rational tools fit into all of this?
• Going forward
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

My developerWorks community

Dig deeper into Rational on developerWorks

Rate a product. Write a review.

Write a review for Rational products today!

Special offers