Level: Introductory Allan Pratt, Senior Software Engineer, IBM
15 Dec 2007 
Is IBM Rational PurifyPlus a tool for testers or for developers? The answer is: both. It helps support both these roles in their shared quest to maximize product quality by eliminating defects and optimizing reliability and performance. This article explains how PurifyPlus works and how both developers and testers can employ it effectively.
From The Rational Edge.
Developers and testers play different roles in an organization, but both share the goal of maximizing overall project quality. Whether it means adding features and capabilities that customers want or making sure those features work properly, your project team needs to sustain a high level of quality to have any chance of success.
IBM® Rational® PurifyPlus™ can help both developers and testers achieve their common goal of project quality. Some people see PurifyPlus as a testing tool, and others see it as a developer tool. The truth is it's both, because PurifyPlus is a quality tool. Both developers and testers benefit from PurifyPlus as they strive for high levels of quality in software projects.
In this article I present an overview of PurifyPlus, how it works, and why it's so valuable for both developers and testers not only to boost product quality but also to increase individual and team efficiency and effectiveness across many important quality assessment activities.
The costs of low quality
For your project to be successful, it is imperative to find and fix bugs early. Defects cost much less to fix when they are found early, either by the developer or in early-phase testing. Bugs that get through testing and ship to customers are much more expensive to find and fix, and they are also costly to your reputation and ultimately to your business.
By using PurifyPlus during both development and testing, you can drive quality into the project early and avoid the higher costs associated with insufficient testing coverage, addressing performance problems, and/or finding and fixing memory usage errors later on.
The components of PurifyPlus
The Rational PurifyPlus product has three components: Purify for memory error checking, Quantify for performance data collection, and PureCoverage for code-coverage analysis. I'll discuss in turn how each of these components works.
The Purify component
Purify analyzes memory usage in C/C++ programs. It issues an error report when it detects things like using more memory than was allocated, or using memory before it was initialized or after it was freed. Purify also reports memory leaks -- i.e., when a program fails to release one or more blocks of memory before losing track of the memory. In Java™ and Microsoft .NET programs specifically, Purify helps you identify areas of your program that use memory and don't release it, which is another kind of memory leak.
For C and C++ programs (not Java or .NET), Purify works by inserting extra instructions around every memory access the program performs. This is called instrumentation. When you run the instrumented program, it will run normally, doing all the things it usually does. But, in addition, Purify is there inside it, checking for problems. If the program reads from memory that hasn't been initialized, or reads or writes memory that has been freed or lies beyond the ends of allocated blocks, you will see a Purify error report.
Some types of memory access errors (like using a NULL pointer) cause a program to crash outright. That's an easy kind of error to spot -- the kind you can find with a regular debugger. Purify lets you see the more insidious kinds of memory errors that can occur even when the program seems to be acting normally. The individual functions in the program can produce the right answers and your tests can pass, but there can still be memory errors.
For example, a flaw in the logic can cause a program to use 100 entries in an array when only 99 were allocated. That kind of off-by-one error might not cause your program to crash or your test to fail, but there is still memory corruption going on. Some day after you ship the program that memory corruption bug is bound to show up in the field as incorrect operation, program crashes, and unhappy customers. Purify detects and reports on these kinds of errors whether or not they cause the project to misbehave in easily detectible ways.
For Java and .NET programs, Purify helps you identify memory that your program is still using when it should have been released. This type of memory leak will cause your programs to be bloated, get slower over time, and eventually run out of memory.
The Quantify component
Quantify is the part of PurifyPlus that does performance analysis. It works for C/C++, Java, and .NET programs. Quantify measures and analyzes the time spent in your program to identify "hot spots" and inefficient code. The unique "River of Time" display leads you directly to the areas of your program that are consuming the most time.
The PureCoverage component
PureCoverage does code coverage analysis. Like Quantify, you can use it on all your C/C++, Java, and .NET programs. With PureCoverage, you can ensure that you are testing your whole program. After all, the quality of code you don't test is unknown, because your testing isn't reporting on its correctness or health. Bugs in untested code will "escape" into release, and they will be much more costly to find and fix. It's much cheaper to identify testing gaps, fill them, and find the bugs before release.
In addition, it's important to know that Purify only does error checking on parts of your program that run during testing. If you have gaps in test coverage you will also have gaps in Purify error checking. This is another good reason to use PureCoverage.
How developers and testers use PurifyPlus
Developers and testers use tools differently. A developer is more likely to want interactive, graphical results with "drill-down" capability, and PurifyPlus delivers that. A tester is more likely to want log files and summary reports for use in reports and for capturing and trending results. PurifyPlus produces those, too. In this section I'll describe some of the ways that developers and testers, respectively, can use the PurifyPlus components to optimize quality, and what advantages you'll gain by doing so.
Purify for developers
Developers can use Purify in two ways. First, they can use it as a debugging tool when confronted with a program bug or defect that seems to be related to improper memory usage. Purify can help identify the culprit when memory is being corrupted, used improperly, or leaked. In this way, Purify is like a doctor: you turn to it when you are sick and need a diagnosis and a cure.
But there is another way for developers to use Purify: like a writer uses a spelling checker, to find and eliminate errors you didn't even realize were there. By using Purify routinely, you can find mistakes in memory usage and fix them even before they lead to a program crash or other obvious problematic behavior. (Is a bug still a bug if you find and fix it before you check it in?)
One way that some programming departments use Purify is as an automatic step in the source code control process. Before a developer's checked-in code is allowed to proceed to testing, it has to be tested with Purify. If there are Purify error reports, the change isn't allowed to proceed into the build. This keeps the build healthy and keeps developers out of trouble. After all, you don't want to be the one who checked in a change that made all the tests crash. In this way Purify acts like vitamins or exercise: it's something you can use every day to keep your code healthy and keep a lid on problems so they don't get out of hand.
Quantify for developers
A tester or a test tool can tell you at a high level that system performance is inadequate, but finding and fixing the underlying causes falls to the developer. That's where Quantify comes in. Quantify highlights the "hot spots" in your code, such as places where an algorithm needs to be improved, where a linked list should be a hash table, or where a result needs to be cached so it isn't recomputed so often.
Quantify lets you compare two runs of your program, so you can easily see the difference in performance when you make a code change. This lets you do performance optimization using real data instead of guessing. For C and C++ programs, Quantify uses consistent timing figures so the performance data is not influenced by outside factors like load on the test machine. That means you can compare runs reliably and measure finer performance differences than you can with tools that only measure "wall clock time."
Another way to use Quantify, besides performance analysis, is to gain a better understanding of program logic and operation. Quantify's "River of Time" displays a graph that shows how the program's functions called each other. You can use this to follow the program logic and see relationships among subsystems. Sometimes the most valuable insights come from surprises like "I didn't expect this subsystem to call that one." Quantify's call graph shows you what really happened -- if it doesn't match what you expected then you're likely to learn something.
PureCoverage for developers
Developers are generally responsible for creating unit tests and sometimes functional tests. One of the goals of these tests is to exercise all the code in a given area of the project. You can examine the source code by hand and write test cases that you think will cover it all, but this is not a reliable or efficient way to develop tests. You need a code-coverage tool like PureCoverage to tell you that you're really exercising all of the loops and conditions in that area of the project. Remember, untested code is probably buggy code!
Once you've created unit tests that exhaustively exercise the target code, you aren't done writing tests, however. As the underlying code changes, your tests will inevitably fall behind. They won't exercise new features or different code paths that are introduced later. You can stay on top of this by using PureCoverage to analyze the test coverage routinely and watch for drops in the coverage level of a given subsystem. It's better to find this out earlier rather than later: if you get caught late in a release cycle with inadequate coverage, you'll be the one staying late to write more tests.
PureCoverage has another benefit when writing tests: it tells you when to stop. There's no benefit in over-testing a functional area or subsystem. If you write tests by eye instead of using measurement tools like PureCoverage, you can over-invest in test creation without getting any additional quality in return. PureCoverage can save you time and money by telling you when enough is enough.
A final benefit to PureCoverage for developers is that it meshes with the use of Purify as a backstop to keep memory usage errors out of the project. Remember that Purify only performs memory error checking on code that you actually execute during testing. If your tests fail to exercise some areas of the code, then you also aren't getting the benefits of Purify in those areas.
PureCoverage for testers
A test suite is only as good as the quality of the tests it contains. That may seem obvious, but it bears repeating. There's no point in spending time and money to run a test suite, even one that passes all its tests, if it doesn't really exercise the project. If the verdict from the test suite doesn't really tell you about the quality of the project, you're just wasting time and resources.
One way to measure the quality of the tests is to know what areas of the project they exercise and (more important) what areas they miss. Only by understanding the coverage achieved by your tests and the gaps that remain can you judge the value of your test verdicts. Gaps in testing are hard to identify without tools -- a test that fails or crashes is obvious and unmistakable, but gaps in testing are silent. If you don't measure your test coverage, you will keep running your tests and seeing them pass and never know your coverage level (and thus the value of the test runs) is deteriorating.
PureCoverage measures your test coverage and reveals gaps. It produces reports down to the individual line level; but for testers it is usually best to capture, summarize, and trend the coverage data at the file or subdirectory level. This can identify areas of the project that achieve less coverage over time, to show where testing needs to expand in line with code changes and added features.
Purify for testers
From the perspective of a tester or test manager, Purify is a dream tool, because it lets you get more value out of the work you're already doing. After all, you already have tests that exercise the project, verify its correctness, and report on its quality. Running those tests with Purify lets you make them do double duty. That is, while your tests are busy doing the same jobs they have always done, you also use Purify to monitor the program's execution to detect and report on memory usage errors.
As mentioned above, many types of memory usage errors are insidious: the program appears to behave normally, even though it's at risk for memory corruption and improper behavior. Very often, tests are less complex than the eventual production environment, so a program can contain a memory corruption error that will wreak havoc after release even though the tests all pass. Purify adds value to your existing correctness tests by simultaneously checking for memory access errors.
Here's another way that Purify is a dream tool: it reports bugs, pure and simple. Some tools dump out data you need to review or interpret or graph over time to see results. Purify isn't like that. Purify identifies real bugs in the program you are testing and points you right to them. There are two types of reports from Purify: those marked with yellow "caution" icons and those with red "error" markers. The "error" type unambiguously reflects a defect in the program logic: Purify is reporting a program bug and that's what you need to hear.
Because Purify's analysis of memory usage errors is so thorough and so complete, there is also a lot of value in a test run where Purify does not report any errors. When Purify doesn't report any errors, it means the program (as tested) doesn't have those errors. That's a great thing to know, an affirmative statement about the high quality of the project being tested.
Naturally this only applies to the types of memory errors that Purify detects, and to the ways that your tests exercised the program. There can still be memory bugs in the program that would happen along different code paths or using patterns of data that weren't tested.
This is why it's so important to use PureCoverage as well as Purify to identify gaps in testing. If the test suite exercising the program doesn't really put the code through its paces, a passing grade from Purify doesn't mean as much. Like a traffic cop, Purify can only report on the violations that happen while it's looking. PureCoverage lets you identify gaps in testing so you can fill them and let Purify see the whole picture.
Quantify for testers
Testing isn't only about correctness. Another factor in project quality is performance. Even if the software always works properly and gives the right answers, you don't have a high-quality product if it's unacceptably slow. Less obviously, performance can change from build to build or release to release, such that you can start out with good performance but lose it somewhere along the way.
Quantify lets you measure the performance characteristics of a project. You can use the data to trend results over time and spot eroding performance from build to build or release to release. For example, imagine you have a test that puts a server through its paces with a standard data set and transaction sequence. You can use Quantify to measure the test's performance, and you can track the performance over time as the project evolves. It's important to verify that new features and code changes aren't degrading overall performance. With Quantify's "graphical diff
1
" feature, you can use the call graph and the River of Time to compare previous runs to current ones and highlight new hot spots and time sinks.
If a test includes awareness of Quantify's measurement capabilities, it can even narrow the focus to eliminate uninteresting factors. For example, for many server applications it's not important if the server takes longer to start up: what's important is the run rate of transaction performance after startup. A Quantify-aware test could "warm up" a server with standard data and transactions (to fill the caches, etc.) and then tell Quantify to start data collection only after all that is finished. By collecting timing data only for the subsequent transactions, the data excludes one-time startup costs and gives a clearer idea of whether and how the system's core performance is changing from build to build or release to release.
Conclusion
It's a fact: developers and testers think differently. They also use tools differently. What they have in common is a focus on quality. Both these roles can benefit significantly from using Rational PurifyPlus because it is both a developer tool and a tester tool -- it's a quality tool.
Notes
1 Only the Windows version of PurifyPlus has this "graphical diff" feature.
Resources Learn
Discuss
About the author  | 
| | Allan Pratt is a Principal Engineer on the PurifyPlus product team in IBM's Rational software division. He has been a professional developer for over twenty years. He started out at Atari in the days of the 520ST, then joined the tools group at the old Taligent for a while before arriving at Pure Software in 1996. He's always stayed pretty close to the bare metal: his projects have included BIOS programming in 68000 assembly code, linkers and debuggers on multiple platforms, UNIX® scripts and tools, object-code instrumentation for Reduced Instruction Set Computer (RISC) and Explicitly Parallel Instruction Computing (EPIC) processors, and byte-code instrumentation for Java and .NET.
|
Rate this page
| 
