Combinatorial Test Design: Who needs it and what’s it good for?

Share this post:

Eitan Farchi
Editor’s note: This article is by Eitan Farchi, a senior technical staff member at IBM Research – Haifa, who specializes in software testing analysis and reviews.
With existing networks heading towards the limit of their capacity, the need for a new Internet architecture – sometimes referred to as the Future Internet – is becoming critical. Future Internet is the motivation driving breakthrough infrastructures, applications, and services that will make it possible to continue growing the Internet and lay the groundwork for new models we need to help business move forward – whether in the cloud, through more social applications, or via mobile.

But how do we develop applications for the Future Internet, and more importantly, how do we make sure these applications are reliable, secure, and work as intended?

FITTEST, a European project (2010-2013) aimed at developing an integrated environment for the automated and continuous testing of Future Internet applications was created to help solve this challenge. As partners in the project, IBM researchers in Haifa, Israel are contributing unique testing tools based on combinatorial test design (CTD), and using the project’s tools and techniques to test a simulation system for cloud management. 

What is CTD?

Testing applications or services is all about examining how the system will react under different scenarios. The difficult part is making sure you take into account all the different situations that could possibly arise. If you do more intuitive test planning, there’s always a chance you’ll forget a good number of scenarios due to the massive number of combinations that need to be considered. CTD helps testers systematically model the gamut of possible scenarios to be tested, and then works to optimize the number of tests so they cover these situations.
Say, for example, you want to test a banking application that allows users to carry out financial transactions on the Internet. These transactions need to be examined to make sure they transfer the right amount of money from the correct account, at the right time, to the right place. Beyond that, you would need to verify that the application works correctly even if the web site falls under cyber-attack, in which massive amounts of requests are sent at the same time and the system crashes because of the overload. This is a case where CTD can help make sure all possible scenarios are covered.
Aside from validating that testing is effective and of the highest quality, it’s essential that the effort involved does not become impractical. Another great benefit of CTD is its ability to dramatically reduce the amount of testing required. IBM researchers found that CTD has the potential to cut testing from an order of 10,000 hours to 100! In our work with a large European insurance company to test their Internet applications, it would have taken months to do the thousands of tests defined in their testing plan. Using our CTD methods and tool, we managed to reduce the plan to 30 tests – which were carried out in a matter of hours. 

How does it work?

A few years ago, our team decided it was time to tackle the challenges that plague the testing process. There were too many combinations to deal with and it was taking too long to test all of them. We wanted to control the risks we took by leaving out certain tests, yet be sure we covered all the important scenarios. The solution was to find a method that maximizes the value of each tested scenario so we could reduce the number of tests, while controlling the risk of leaving out important scenarios.

To balance the cost and risks, we first define a systematic testing plan that outlines all possible tests and then choose a subset of tests that cover all the interactions of variables at some level of interaction (pairs, three-way, etc.). We developed a CTD algorithm that identifies a small test plan that covers 100 percent of a given interaction level. The end result is what we call IBM FOCUS CTD – the IBM Functional Coverage Unified Solution.
As part of the FITTEST initiative, we contributed the IBM FOCUS CTD tool to help generate high quality, optimized test plans. IBM FOCUS is independent of an application’s domain and can be applied at different levels of testing. It can also read existing tests, analyze their functional coverage, select a subset of the tests that maintains the same coverage, and generate new tests to close the coverage gaps.
What’s it good for?

CTD can help businesses carry out testing in all kinds of industries, whether banking, health insurance, retail — essentially, any business that involves a web service. CTD is especially effective in domains where the testing efforts require huge amounts of resources and are expected to cover large areas of complex business processes.

“I believe we’re seeing a new revolution in the area of testing. CTD bridges between formal methods for advanced software specification … and the current state-of-the-practice in testing.”  
— Eitan Farchi
CTD bridges between formal methods for advanced software specification—which is difficult for most people to use—and the current state-of-the-practice in testing. Our workshops and training sessions have shown that anyone can use th
is method, even people who are not programmers. We can teach people to use our tools within a few hours and start to model in just one business day.
The Future Internet will be a complex interconnection of services, applications, content, and media, offering a rich user experience beyond current hyperlink-based navigation. With society becoming increasingly dependent on these services for business, entertainment, government, and education, it’s vital that the applications running on top of the Future Internet meet strict standards for quality and reliability. IBM FOCUS CTD is one tool that is helping make this a reality.
If you’re interested in finding out more about IBM FOCUS CTD, contact the author at
More stories

A new supercomputing-powered weather model may ready us for Exascale

In the U.S. alone, extreme weather caused some 297 deaths and $53.5 billion in economic damage in 2016. Globally, natural disasters caused $175 billion in damage. It’s essential for governments, business and people to receive advance warning of wild weather in order to minimize its impact, yet today the information we get is limited. Current […]

Continue reading

DREAM Challenge results: Can machine learning help improve accuracy in breast cancer screening?

        Breast Cancer is the most common cancer in women. It is estimated that one out of eight women will be diagnosed with breast cancer in their lifetime. The good news is that 99 percent of women whose breast cancer was detected early (stage 1 or 0) survive beyond five years after […]

Continue reading

Computational Neuroscience

New Issue of the IBM Journal of Research and Development   Understanding the brain’s dynamics is of central importance to neuroscience. Our ability to observe, model, and infer from neuroscientific data the principles and mechanisms of brain dynamics determines our ability to understand the brain’s unusual cognitive and behavioral capabilities. Our guest editors, James Kozloski, […]

Continue reading