Effective testing: A combinatorial approach for improving test efficiency
ABDULRAUF 270002CXRA Visits (11093)
Abstract— In software industry, test efficiency becomes an increasingly critical and strategic necessity. Assuming the level of testing in the past was sufficient (which is rarely the case), how do we possibly keep up with this new explosive pace of project deployment while retaining satisfactory test coverage and reducing risk? The answer is either more people for manual testing, or introduce smart approaches like combinatorial testing that will help for reduction in project cycle times with out reducing the scope of the quality. In this paper we are discussing how we introduced a combinatorial approach for improving test efficiency in an embedded environment using simulation testing.
Suppose you got an assignment for doing the simulation analysis of embedded software using simulation testing (Eg: Electronic power assisted steering system). The requirement we need to test is that to run multiple releases of this software on a simulator (Micro controller simulator) for specified number of kilometers (eg: 2, 50,000KM) and make sure that the system running with out any issues. For simulating the situation the test team need to identify various inputs like sensing options (Like Angular range, angular resolution, angular accuracy, response time, rotation speed, speed accuracy, torque range etc), power consideration and other control aspects. Each parameter could have numerous values. As a result, there were tens of thousands of potential input variations, each producing different results, and making exhaustive testing of the simulation nearly impossible. In such a situation test team will face difficulties in selecting the suitable test combination for getting test completeness and coverage. Manual process of selecting the input combination is mind numbing and error prone.
Random selection and exploratory approaches do not provide an easy way of assessing the effectiveness of our test coverage and usually lead to redundant tests. In this situation, we need some way to systematically produce a subset of all possible tests that effectively tests important combinations; has a high probability of exposing functional, multi-modal defects and risks and provides confidence in the test coverage. Testing combinations of input variables that affect a common output is important because several industry and academic studies conclude that variable interactions are often problematic and a common cause of multi-modal bugs in software. But, it is usually not feasible to test exhaustively all possible combinations of input variables. So how do we derive a suitable set of test cases when it’s just not feasible to use all the possible combinations and variations of test parameters? The article addresses one solution through combinatorial test approach.
II. The combinatorial approach
Combinatorial analysis is an effective functional test technique that can help us systematically evaluate complex relationships of interdependent input variables that directly or indirectly affect some predetermined output state or condition. In order to apply this technique effectively, we must understand how to decompose the feature under test. Feature decomposition for combinatorial testing generally involves identifying the output conditions we wish to evaluate, identifying the parameters that directly or indirectly affect that output condition, and understanding how the variables for each interdependent parameter affect the output condition. The benefits of moving from commonly practiced "intuition-based" methods of test case selection to adopting combinatorial testing approaches are powerful, repeatable, and easier to achieve than ever before, given the availability of user-friendly tools. Despite this, awareness among many testing professionals remains relatively low. The benefits of combinatorial testing include:
Below are the some of the approaches that we followed in our case study of simulation testing.
A. Decision table preparation
Combinational test models uses decision tables to represent the implementation under test. Decision tables are widely used in software industry and are ideal for representing test model for applications like embedded real – time, data processing etc… Following are some of the steps used in our reliability analysis case study.
Decision table has two parts:
Decision table does not cover a detailed specification for the process, algorithm or implementations of the actions. But the testability and error prevention are easy when a testable specification for each action exists. One important aspect of the combinational testing is that each combination of conditions must be unique but action may appear in more than once. In our simulation testing for reduced the number of test inputs with the help of the following actions
B. Decision table validation
It is common practice that a decision table to validate before it is used to design or generate test suite. Need to check the characteristics like testability, completeness, consistency and also need to ensure that the table is free from technical errors. The model capability also should be reviewed. This kind of checking will reveal errors and omissions before a test suite is developed and run, increasing the effectiveness and efficiency of combinational test suites. In our case study we done the validation based on the following things
C. Test Generation
We started test case generation after validating the decision table/truth table and logic functions that we derived. For deriving the suites we employed mixed approaches of the following strategies
III. Test setup
The Test server is the place were we start the test execution for all the test machines and also used to store the execution logs of the test runs (This part is fully automated). Client machines are loaded with software under test and other hardware parts (Hardware parts are modeled with .dll files) and Keil simulator. Every day we used to execute the test runs till it reaches the target time/kilometers. One of the observations we noticed in this process is that more than half of the errors is coming due to the single parameter value and the remaining part is due to the combinational parameters. In combinational failure, the major chunk is from the 2 –way combination. Very less defect identified using 4-way and above combinations. This observation helped us to plan our test execution accordingly by concluding that the degree of interaction involved in faults is relatively low.
Exhaustive testing in most cases is not possible, feasible, or cost effective. The effort to include all possible combinations and variations of test input parameters is generally an impossible undertaking for complex and an attempt to do so can result in scope creep. In this situation, with proper planning it will be good to try the combinatorial testing approach that makes an excellent trade -off between test effort and test effectiveness. Employing pair wise testing will also help to reduce the number of tests dramatically with out any reduction in faults