One of the hardest things to decide at the beginning of any software testing process is what level of testing the project will receive. Testing even a simple project can be taken to extremes and end up costing you more in time and resources than development did. On some systems this is necessary -- but only rarely. In this article I'll discuss using cost-benefit analysis to determine what level of testing your application needs, specifically related to automated testing with IBM Rational Robot.
The standard justification for choosing one level of testing over another is that the benefits exceed the costs over the life cycle of the project. Doing an actual analysis of the benefits and costs of using various testing structures can encourage better decision making and ensure that resources are allocated effectively to support the maximum level of testing necessary for a project at the lowest cost. At least three different structures (including the one currently in use, if there is one), plus the option of not automating at all, should be considered in performing a cost-benefit analysis, and the alternative chosen should be the most cost-effective within the context of budgetary and timeline considerations.
The term "timeline considerations" can be misleading at first. The timeline for automation is often much different from the timeline for the project. You'll need to take into consideration the following:
- what functionality you'll be testing
- development time for this project (or iteration of the project)
- how many future iterations your tests will be used on
- how many patches, updates, and so on will be released
- resources available
- reusability of previous test structures
- baseline for future test structures
Cost-benefit analysis must include comprehensive estimates of the projected benefits and costs of all potential test structures. Along with tangible benefits and costs, benefits to which a dollar value can't be assigned (intangible benefits) should be included. Evaluating and comparing both kinds of benefits requires extra effort, but it allows subjective judgment to be a factor in the analysis.
You should start your cost-benefit analysis during the elaboration phase for any project (or iteration of a project). As your requirements develop, you should be able to determine what types of testing you'll be doing and you should be starting the development of your test plans. Using the early outlines of the test plans, you'll be able to roughly define what you'll be trying to automate.
While a test manager should be responsible for ensuring that the cost-benefit analysis is done, that person should be assisted in the analysis by a team with expertise in automated test development. Because a team brings different perspectives to the analysis and the process of estimating costs and benefits, this should ensure more realistic estimates than those of just one person. Additionally, one person rarely has expertise in all of the areas required for a cost-benefit analysis plus knowledge of the testing that's being automated.
How Do I Perform a Cost-Benefit Analysis?
The first step in performing a cost-benefit analysis is to define what you're testing and what you plan to automate. This will invariably come from the test plans and requirements as well as previous automation efforts. Then proceed as follows:
- Identify potential structures.
- Document assumptions.
- Estimate costs.
- Estimate benefits.
- Convert the costs and benefits so you can compare tangible and intangible values.
- Compare the alternatives.
As I said earlier, at least three different structures (including the one currently in use, if there is one), plus the option of not automating at all, should be considered. Potential structures could include these:
- a simple recording-and-playback structure
- a framework structure
- defining features of the application
- defining commands or features of the tool's programming language
- defining small, conceptually unified tasks that are done frequently
- defining larger, complex chunks of test cases that are used in several test cases
- defining utility functions
- a data-driven structure
- spreadsheet driven
- datapool driven
- database driven
- a hybrid structure
Data must be collected for each potential structure for estimating the costs and benefits. Some possible sources of data are experience, past projects, current automation costs, online whitepapers, and Rational Developer Network(sm). When comparing structures, be sure you're looking at the same levels of test coverage so that you're comparing apples to apples. At this point in the process, some structures can be rejected because they aren't feasible for reasons other than costs and benefits.
As you're looking at structures, you'll be making a lot of assumptions. It's important to document all of them, and, if possible, justify them on the basis of prior experiences or actual data. For example, you may assume that there will be three patches to this version of the application-under-test. This could be justified by averaging the number of patches for each past version. Knowing your track record for releasing patches helps you decide whether maintainability is going to be an issue.
This can also be an opportunity to explain why some structures weren't included in the analysis. If you did consider and eliminate some structures early on because of a conclusion that they wouldn't be feasible, the assumptions behind that conclusion should be clearly explained and justified.
Next you'll want to estimate the costs. The following factors, plus any others you know to be relevant, should be addressed for each alternative:
- How much time do you have for development, testing, errors, rework, and so forth?
- Have you ever worked with this structure before?
- How many automated testers will be developing the scripts/structure?
- How much will personnel cost?
- How much hardware will this require?
- How much will hardware resources cost?
In some cases, data may not be available to provide an adequate cost estimate. In that situation, the best alternative is to use the judgment and experience of team members to estimate costs. The more team members you can reasonably involve, the better your estimate will be.
The costs of automation are normally easy to figure out. It's the benefits that are harder to assign dollar amounts to.
First identify the benefits for both the project and the automated testing team. Benefits to the project might include:
- the reduction in manual testing time for this iteration and for future iterations,
- automation of testing that would otherwise be impossible (for example, load testing), and
- reusability if the project is large.
Benefits for the testing team might include
- the ability to reuse artifacts from past structures
- the ability to draw on the current knowledge base of automated test developers
- development of artifacts and knowledge base for future structures
- reduction of the level of risk of implementing the project
- reduction of maintenance requirements, and
- increase in reliability.
After the benefits have been identified, establish performance measures for each benefit and estimate the value of the benefit. If a benefit can't reasonably be assigned a monetary value, it should be valued using a more subjective, qualitative rating system (which assigns relative numerical values). In the past I've used the value scale shown in Table 1.
| ||||||||||||
| Table 1: A scale for assigning relative numerical values to benefits |
Convert the Costs and Benefits to Comparable Values
After the costs and benefits for each structure alternative have been estimated, you'll have a mixture of dollar values and subjective "level of benefit" numerical values. If there's no realistic way to relate the value of the intangible benefits to the tangible ones, they can't be considered significant for the cost analysis. In this case, they can be used as tie breakers if the comparison of structures doesn't show that one structure is a clear winner.
When intangible benefits are significant factors in the analysis, one of two options can be exercised in converting them.
- If it's possible, the relative values can be converted to dollar values. This is a very difficult thing to do and may be impossible to defend. There's no proven basis for assigning a dollar value to a benefit such as lower technical risk, and the amount of the dollar value could be used to influence the selection of the best alternative.
- The dollar values of the tangible benefits can be converted to the same rating scale as used to arrive at the relative values of the intangible benefits. (This is the option I would go for.) This can be done by dividing all dollar values by some arbitrary number, such as 1,000, that will give you values lower than the highest value on your "Level of Benefit" scale.
At this point, the analysis can proceed by comparing the net benefit (benefits minus costs) for each structure to determine which structure is best for your current project. The alternative with the greatest net benefit should be selected.
For this example of the process I just described, I'll use my favorite application in the whole world -- the Windows Calculator. Let's assume I have to automate the testing of all of the buttons and functions. I know that's a bad requirement, but hey -- we're talking about structures, not use cases! (Disclaimer: All numbers, amounts, and assumptions used in this example are fictitious.)
The potential structures I identified are shown in Table 2.
| |||||||||||||||||||||
| Table 2: Potential structures for our example case |
Here are my assumptions:
- There will be a very large number of permutations for the testing of each function within the application. This leads me to believe that there will be a very large number of similar test cases for each function. This leads me to favor framework structures, which eliminate repetitive record and playback, as well as datapool-driven structures, which reduce the number of scripts necessary.
- There will be changes to the Scientific view. At some point in the project, a graphing function will be added. This will change the locations, indexes, and groupings of some of the buttons on that page. This leads me to favor framework structures, as they're more maintainable over multiple iterations than a recording-and-playback structure.
- There will be three weeks to develop the scripts for this iteration of the project. This leads me to favor datapool-driven structures, as they reduce the number of scripts necessary.
- I'll have inexperienced personnel on my team. This leads me to favor a recording-and-playback structure, as there's very little learning curve involved.
- Structure 2 will take 122.5 hours to develop, structure 3 will take 110 hours to develop, structure 4 will take 120 hours to develop, structure 5 will take 100 hours to develop, and structure 6 will take 130 hours to develop (including research and prototype time).
My cost estimates are shown in Table 3.
| ||||||||||||||||||||||||||||||||||||||||||
| Table 3: Cost estimates for our example case |
My benefit estimates are shown in Table 4.
| |||||||||||||||||||||||||||||||||||||||||||||||||
| Table 4: Benefit estimates for our example case |
Table 5 shows how I converted the costs.
| ||||||||||||||||||||||||||||
| Table 5: How I converted the costs for the example case |
Table 6 shows how the structures compare in terms of costs and benefits.
| |||||||||||||||||||||||||||||||||||
| Table 6: How the costs and benefits of the potential structures compare |
It's only beneficial to not automate if all of the frameworks yield a negative difference. In this case, given my assumptions and "off the top of my head" time estimates, structure 5, the datapool-driven structure, is the best option because it has the greatest net benefit.
I've just shown you a simple process for analyzing the costs and benefits of using various structures for your automated testing effort in Robot. Going through this process may seem a bit tedious, but believe me -- it's more than worth it if the final comparison helps you eliminate a structure or structures that would have been more or less than what you need for testing your application.
Mike Kelly is currently a software testing consultant for the Computer Horizons Corporation in Indianapolis. He's had experience managing a software automation testing team and has been working with the Rational tools since 1999. His primary areas of interest are software development lifecycles, software test automation, and project management. Mike can be reached by e-mail.
Comments (Undergoing maintenance)
Table of contents
Local resources
My developerWorks community
Tags
Use the slider bar to see more or fewer tags.
Popular tags shows the top tags for this particular content zone (for example, Java technology, Linux, WebSphere).
My tags shows your tags for this particular content zone (for example, Java technology, Linux, WebSphere).
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
