I was fortunate enough to get married last year. I say "fortunate," as I rarely discover software testing professionals burning up the "Most Wanted Bachelor" charts. It was a wonderful event, followed by a wonderful honeymoon in Fiji. Upon our return, my wife and I went to the bank to convert our remaining Fijian dollars, and as we waited in line, a commotion erupted. The security guard locked the door. The bank had just been robbed.

Within minutes, officers and detectives were on the scene. The investigation began. To hear the descriptions given by the staff and customers, you would think no two people saw the same event. Nevertheless, within 30 minutes the detectives were able to collate the divergent accounts of the event into a single, accurate portrayal of what had transpired and gather a precise description of the perpetrator. Within an hour a suspect had been apprehended.

What's the lesson in this tale? That no one single perspective can fully depict a given event or situation. This is not news. In fact, in 1995, Rational's own Philippe Kruchten described this same problem in an IEEE article entitled "Architectural Blueprints: The '4+1' View Model of Software Architecture¹." In this article Kruchten describes how five concurrent views are necessary to fully describe a software system. The same is true with testing. No single type of test provides enough information to quantify the quality of a system. Instead, multiple types of testing must be undertaken to fully ascertain a software application's quality.

Black box and white box tests represent two broad categories of test types. Neither in isolation can accurately depict the quality of the system. Together, however, they give testers a much clearer perspective on system quality.

I Thought We Were Testing Software, Not Boxes

The "box" in black box and white box testing refers to the system under test; the color refers to the visibility that the tester has into the inner workings of the system. With black box testing, the tester has no visibility into those inner workings. The tester sees only the interfaces exposed by the system. By contrast, white box testing offers the tester full visibility into how the system works. Think of a soda vending machine. A black box test would involve inserting the money into the machine and verifying that a soda drops out and that correct change is given. A white box test might involve opening the back panel on the machine and manually triggering the switch that drops the soda.

Black box testing is sometimes referred to as functional or behavioral testing, and it offers numerous benefits. In the first place, a black box test validates whether or not a given system conforms to its software specification. In implementation, black box tests introduce a series of inputs to a system and compare the outputs to a pre-defined test specification. They test not only individual system components, but also the integration between them. The tests are architecture independent. They do not concern themselves with how a given output is produced, only with whether that output is the desired and expected output. Finally, as they require no knowledge of the underlying system, one need not be a software engineer to design black box tests.

White box testing is sometimes referred to as structural testing. Because white box tests involve the individual components of a system, they require an implicit knowledge of the system's inner workings. In implementation, white box tests introduce a given set of inputs to a component or individual function of a system and compare the outputs to an expected result. Testing is generally not done through a user interface, but by using the debugging features of the given development environment.

How to Choose a Black Box or White Box Test

Black box testing is generally performed by QA analysts who are concerned about the predictability of the end-user experience. Their job is to make sure that the application meets customer requirements and that the system does what it's designed to do. But black box tests offer no guarantee that every line of code has been tested. Being architecture independent, black box testing cannot determine the efficiency of the code. Finally, black box testing will not find any errors, such as memory leaks, that are not explicitly and instantly exposed by the application. This stands in sharp contrast to white box testing, in which, given an infinite amount of time, all lines of code can be tested and clues as to the code's relative efficiency can be ascertained. Generally, developers whose time is relatively more expensive than that of QA analysts perform white box testing. White box testing proves insufficient, however, for situations in which testing isolated components may not reveal integration errors with other components.

The Tools

Rational Software has been producing tools for automated black box and automated white box testing for several years. Rational's functional regression testing tools capture the results of black box tests in a script format. Once captured, these scripts can be executed against future builds of an application to verify that new functionality hasn't disabled previous functionality. Rational also offers white box testing tools that 1) provide run-time error and memory leak detection; 2) record the exact amount of time the application spends in any given block of code for the purpose of finding inefficient code bottlenecks; and 3) pinpoint areas of the application that have and have not been executed.

For as long as these tools have been available, the black box testing tools have resided almost exclusively on the QA analyst's machine, while the white box tools have been purchased primarily by developers. There are several reasons for this divide.

In the first place, QA analysts traditionally do not have a coding and debugging environment at their disposal. And even if they did, the argument goes, most analysts would not understand the information output by the tools. Rational believes that this is an artificial barrier, and that QA work can be improved via white box testing tools that do not require access to source code or a development environment. Now, Rational uses a patented Object Code Insertion (OCI) technology to instrument an application's executable files. No source code is required. This approach to the problem also enables Rational tools, Purify Quantify and PureCoverage, to perform white box testing on third party code and controls, for which source code may not otherwise be available.

With the introduction of Rational Test Studio in early 1999, white box testing became integrated with black box testing; since then, QA analysts have been able to perform white box tests at the same time as black box tests. With Rational's OCI technology eliminating the need for source code or a development environment, QA analysts now have visibility into the "black" box.

While QA analysts are running their functional black box tests, structural white box tests for memory leaks, code bottlenecks, or measuring code coverage can also occur, that development teams as the intended audience. In essence, the QA group can now undertake a job which previously had to be completed by an engineer. Given the relative average salary levels for these two populations, this is clearly an efficient optimization.

Great Minds Don't Think Alike, They Think Together

I don't think it's too much of a stretch to claim that the technological advancement of Rational's tool set is like a marriage of the white box and black box testing roles. That's why I began this article with the anecdote about my own recent marriage. We're still different people, my wife and I, but our goals are more focused now, and we're working together toward some common objectives. Similarly, the more testing professionals can share their detection tools and information base, the more quickly and accurately they can ascertain the overall quality of a software application.

Testers performing traditional black box tests can leverage these same scripts to reap information from their testing that was previously only available to the white box tester on the development team. In so doing, QA analysts have the opportunity to lighten the load on the development team at very little expense. Earlier, I said that "given an infinite amount of time, all lines of code can be tested." Obviously, no software team has the luxury of an infinite amount of time. However, if one reduces the goal of white box testing to finding memory leaks and application crashes throughout the code base, then the time needed for white box testing becomes much more manageable. By putting a common tool set on both the developer's and the QA analyst's desktops, Rational has not only brought black box testing and white box testing a step closer together, but it has also brought developers and QA analysts closer together. This more unified team may be the greatest benefit of all.

About the author

Brian Bryson joined Rational Software in 1995 with a strong background in Quality Assurance. Currently, he is a member of the product team that plans and delivers the IBM Rational automated toolset. A former instructor at Carleton University in Ottawa, Canada, Mr. Bryson has taught the Rational approach to quality automation to college students as well as employees of organizations such as Microsoft, Sun Microsystems, and Oracle.

Rate this article

Comments

Back to top

Table of contents

• About the author
• Comments

Next steps from IBM

• Try: Rational PurifyPlus
• Buy: Rational PurifyPlus

Dig deeper into Rational on developerWorks

Discover knowledge paths

Skill-building guides for Rational, Linux, open source, cloud, Java, business analytics, and more.

Special offers