Level: Introductory Michael Kelly, Software Engineer, QA, Liberty Mutual
24 Nov 2003 The runtime analysis tools in Rational PurifyPlus offer big benefits for a small investment when used by both developers and testers. Learn where PurifyPlus fits in the development process and how to get developers and testers working in concert with the tools.
Rational PurifyPlus isn't just for testers. The runtime analysis tools it provides -- Purify, Quantify, and PureCoverage -- should ideally be owned and used by the development team as well as the test team. When developers start using PurifyPlus, more bugs are found sooner in the development cycle, and we all know how much time, money, and energy that can save. And when both testers and developers are using PurifyPlus in a coordinated way, communication overall between the two teams will improve.
But unfortunately, the PurifyPlus tools too often get overlooked in the feature-rich Rational suites (Rational Suite DevelopmentStudio, Rational Suite TestStudio, and Rational Suite Enterprise), which offer tools that span the entire software development lifecycle. It may be hard to understand the place of the PurifyPlus tools in the development process, and consequently it may be hard to convince test and development teams alike to take the time to learn how to use them. I know in the last company I worked for, we had PurifyPlus available to us but at the same time had no idea what the tools did.
This article will help you think about where the PurifyPlus tools fit in the development process. It will also give you some ideas about how to get both developers and testers working in concert with PurifyPlus. When that happens, you'll realize that you won't find another set of tools that offers more benefits and a shorter return-on-investment.
How to Think of the PurifyPlus Tools
Purify, Quantify, and PureCoverage are often referred to as testing tools and development tools at the same time. I've also seen them described as additional testing tools. None of these terms accurately say what the tools do. Testing tools doesn't really fit because this term is most often used to describe automation, scripting, or other black-box tools, while PurifyPlus really provides "white-box" testing, giving insight into the internal workings of the program under test. The term developer tools doesn't help either, because there are many developer tools out there and this name tells nothing specific about PurifyPlus.
Rational calls Purify, Quantify, and PureCoverage runtime analysis tools. This term describes the activity that developers, as the main target audience for the product, should include in their everyday work -- runtime analysis, also known as development testing.
Why is it important to do runtime analysis? It's not enough merely to check the quality of the code by verifying that no warnings or errors are generated during compilation of the program, or even to check the return values of the important functions by running unit tests, though these should be part of development testing. It's just as important to check whether the application uses its resources (memory, CPU) correctly as it runs, in order to avoid more serious problems like crashes and difficulties with scalability and durability later in development.
Runtime analysis with the PurifyPlus tools can detect such problems by closely monitoring the execution of the application and by creating extensive reports about memory usage, memory leaks, memory and performance bottlenecks, and code coverage. Furthermore, these runtime analysis tools not only detect problems but also find the root cause of misbehavior by pointing to the line of code where a problem originates.
While I don't want to describe in detail here how to use the tools, I think it may be helpful to some if I describe them briefly with what I've found their value to testers and developers to be. If you're already familiar with the tools, feel free to skip down to "How to Get Both Developers and Testers to Use PurifyPlus." (Incidentally, when Purify, Quantify, and PureCoverage are packaged either alone or as part of PurifyPlus, they have the same features as when incorporated in the Rational suites.)
Purify
Purify detects memory leaks and memory corruption errors in an application on both the Windows and UNIX platforms. It can also be used to profile the Java memory usage of your application on a Windows machine. I find it to be fairly easy to learn (see the online help and the tool mentors within whichever suite you may have) and to give surprising amounts of information. The only complaint I've heard about it is that in some cases it gives too much information, but never fear -- as with almost all the Rational tools, you can filter down to the information you really care about.
Developers will find value in this tool when looking for memory leaks during the development and unit-testing phase of the software lifecycle. Testers will find this tool helpful when they need to test a smaller application quickly, where it's normally not cost effective to perform full-blown automated testing and consequently scripting is skipped entirely. In these instances, it's normally a good practice to create a couple of high-level scripts (one that uses each major feature, and one that works its way through most of the different sections of the application) and execute them with Purify turned on. The tester doesn't need to have access to the code; even without the code, Purify gives surprisingly meaningful results. When I first started playing around with it I did something similar to this, setting the Purify feature in Robot, and ended up finding three memory leaks in the code. I spoke with the lead developer for the project and together we figured out a way to integrate Purify testing into some of the more critical projects the company was working on.
If you're a tester, never underestimate the power of taking a memory leak error to a developer. That same developer whom you annoy with what he thinks of as silly GUI changes or a minor deviation from the requirements that the end user would never have noticed will look at you with new respect. Now you're no longer just a tester, you're a tester who found a real bug! Even if what you found is the most trivial of memory leaks (array bounds errors for noncritical functionality, or poor garbage collection), because you found it and he didn't but should have, you'll become someone he looks forward to sending his software to.
Quantify
Quantify finds performance bottlenecks on both Windows and UNIX platforms. What that means is that Quantify looks at your application while it's running and monitors what it's doing in terms of the actual physical components of the computer, and then shows you the performance data for your program and its components. This will help you see where your program spends most of its time and then optimize it so that you get the best performance from those resources that are most critical.
If your development team is already using a tool that does something similar to this, they may not be very impressed. But your testing team can certainly benefit from doing high-level scripting with Quantify, just as with Purify and for the same reasons. Quantify is very easy to use (even if its results aren't always easy to interpret and understand) when integrated with Robot. Have your testers run the application under test through its paces, if for no other reason than to profile it on a box other than the developer's 8000 MHz PentiumVII machine with 4 GB of RAM and a 150 GB RAID-array hard drive optimized for performance. I don't need to say much to convince people that performance testing on a variety of computer configurations is a good thing; likewise, profiling an application's performance on different configurations is just as telling about the application.
PureCoverage
PureCoverage integrates with Purify and also works as a stand-alone application. PureCoverage tracks the components, data, and modules your application uses during its execution. You can view exactly which components or modules are being executed, down to the detail of the individual lines of code.
While your developers may find this useful, I think greater shouts of joy will be heard from your testing team. This gives them the ability to track the amount of code coverage offered by their test plans, and not just requirements coverage. I find this tool to be especially useful on larger applications and less necessary on smaller applications.
How to Get Both Developers and Testers to Use PurifyPlus
Your development staff may already have tools that perform functions similar to those performed by the PurifyPlus tools. If they do and they're actually using these tools, there's no point in asking them to learn PurifyPlus. But if they do and they're not using these tools, it would be worthwhile for them to learn to use PurifyPlus in concert with your testing team. The worst thing that will happen is that at some point management will have to decide which tools to keep and use. The best thing that will happen is that the developers will start doing this type of testing and more bugs will be found sooner in the development lifecycle.
In my experience, a phased approach to getting testers and developers on the same page with PurifyPlus works best. I'll outline my suggested approach here.
Phase 1: Get Them to See the Value
The first thing you need to do to get your developers and testers using this software is to get them to see the value in it. You have to find ways to expose them to PurifyPlus and get them to buy into the tools and want to use them. There are several ways of doing this, and your success will depend on the size of your IT staff, the size of the projects where these tools will potentially be implemented, and the amount of testing and process that currently takes place at your organization.
One of the easiest ways to get developers to buy into a runtime analysis tool or technique is to show them that it works. Having spent considerable time working with developers as a test engineer at my last company as well as being the lead developer at my current job, I know that results are what make developers stand up and pay attention. Typically, if you go to a developer and say, "Hey, I read about this new thing and I think we should try it . . . ," she will pay polite attention while you're there, take whatever information you give her, and add it to the "things to do when I have absolutely no work, can't take a long lunch, and have no technical articles to read" pile that sits on the corner of her desk. This is where unproven ideas go to die.
What's more effective is to have your test team learn the tools enough to get some sort of reasonable results and output (which will be easy with Purify and PureCoverage and somewhat more challenging with Quantify) and share it with the development staff. This will pique their interest enough so that they'll ask your testers to do more with the tools. At this point, they like the potential of what they see, but they still don't like it enough to do it themselves and potentially waste good development time.
Phase 2: Learn and Use the Tools Hand in Hand
In the next phase, your test team and your development team will start to work hand in hand while learning and using the tools. The test team will still be doing most of the work, but the development team will be looking at the results and asking questions about them. Truly curious developers may even want to see a test script execute so they can watch the results being created themselves.
The trick at this point is to be patient and allow an atmosphere of sharing to grow. The test team should be willing to spend time doing some of the testing that should technically be done by the developers, and likewise the developers should be patient enough to explain some of the more technical details of the application and language to the test team. If you work in an organization where this already happens, bully for you and your organization. If not, it will come with a little time.
Phase 3: Offload Some of the Testing to Developers
Once things seem as though they're going smoothly, the next step is to try to offload some of the testing onto the backs of the developers. Purify and Quantify should typically be run by the developers during unit testing and integration testing, with the exception of a single execution in acceptance testing. This means that you'll slowly have to convince them that it would be easier on the testers and quicker for them if they found bugs earlier in the process.
In actuality, this step will take place on its own. Once your developers see memory leaks in Purify, they'll more than likely want their own copy so they can run it for themselves to see if they can get the problem in their development environment. Likewise, once a significant performance bottleneck is found using Quantify, your developers will want to play around in Quantify to see if they can find others.
Once the developers start to show a genuine interest in using the tools, the testers should encourage it. They should continue to interact with the developers on a regular basis and start to ask them questions about some of the bugs they may be finding with the tools. For example, if Purify or Quantify shows no significant errors in testing, the testers should ask the developers if they found any earlier while they were doing their own testing. If the test plans don't show an acceptable amount of coverage in PureCoverage, the testers can ask the developers to start sharing their test logs with them and can use some of the developers' coverage to minimize their own risk.
Just as for requirements and defect tracking, there should be communication between the teams when dealing with testing at this technical level. You'll find that as use of the tools continues and the two teams communicate more, there will more than likely be better communication overall.
Moving Forward . . .
Even though Purify, Quantify, and PureCoverage don't attract a lot of attention in the feature-rich Rational suites, they offer great value to both developers and testers. Ideally, the PurifyPlus tools should be owned and used by both teams. Developers love them because they save precious time spent in debugging and looking for memory and performance bottlenecks. Testers praise PureCoverage because it helps them assess the quality of their testing procedures. And using the tools together creates a great channel of communication between the often-separated development and QA groups.
About the author  | | | Mike Kelly is currently a software testing consultant for the Computer Horizons Corporation in Indianapolis. He's had experience managing a software automation testing team and has been working with the Rational tools since 1999. His primary areas of interest are software development lifecycles, software test automation, and project management. Mike can be reached by e-mail. |
Rate this page
| 
