As Quality Engineering has become more advanced, using newer tools and methodologies for testing any software has become imperative.
For many teams, test automation has become the backbone of their software testing. However, even when solid test automation exists, you still need to maintain it: adding new tests, modifying existing tests, and adjusting scripts to keep up with the changes in the product. It is worth finding out how to get a good return on all this effort.
The size and complexity of software systems continues to increase, and test automation has followed the same trend. Test harnesses are becoming larger, warranting more maintenance and closer monitoring, so that fast, reliable, and accurate testing can occur. To do this, you need the right tools.
IBM Rational PurifyPlus is a collection of three tools: IBM Rational PureCoverage®, IBM Rational Quantify®, and IBM Rational Purify®. These products are well suited for testers (and developers) to use in the course of testing software.
More than test automation
Even if you understood that test automation alone is not enough to declare the product under test ready for shipping, you need to make the proper investment in the design, review, and upkeep of a test harness or your customers will still manage to find those elusive defects that seem to slip into applications.
When you (as a tester or quality engineer) are given a testing task, the following are some questions to ask yourself to better understand it:
- What do I have here?
- What resources are available to help me complete this task?
- Is the task going to be complete if I run all the planned tests against the application?
- Do the automated tests exercise most of the application?
- What else is available to help me test more thoroughly, so that I can complete my testing task?
Understanding the behavior of the application under test, and gleaning valuable information regarding how the automation is exercising it, can greatly improve testing effectiveness. This in turn reduces the number of defects that escape to your customers.
When testing, you should focus on finding the maximum number of defects in order to enhance the productâs quality, not just on completing a testing task.
This is runtime analysis: analyzing a running program to collect data that helps you understand the behavior of the software, and the relationships between various components in it.
In the course of software development, think of runtime analysis as an advanced debugging and testing technique.
During the execution of an application, you can collect and study the following runtime data:
- Code coverage
- Performance bottlenecks
- Threading problems
- Execution paths
- Runtime tracing
- Memory errors and leaks
- The incorrect use of memory
Following are details about how the components of PurifyPlus are designed to help you accomplish this:
- PureCoverage is for code coverage analysis: it measures how much of the product code was exercised in a test suite, and how much remains to be tested.
- Quantify is for performance analysis: it helps find performance bottlenecks in applications.
- Purify is for memory analysis: it is useful in finding memory leaks and incorrect use of memory in the application, which may be resulting in crashes.
PurifyPlus for Microsoft® Windows® can be used with C++, Java™, or managed code. It is also available integrated within the Microsoft® Visual Studio® .NET IDE (integrated development environment).
PurifyPlus for Linux® and UNIX® is available for C/C++ and Java applications built on Red Hat, SUSE, Solaris™, HP-UX, and AIX platforms. This article, however, will look at automation and analysis on the Windows platform only.
Code coverage analysis
Assume that you have already written and implemented tests matching every bullet point in the requirements or specifications document. Furthermore, you are running them in a stable automation environment, and results so far have been encouraging. However, customers are reporting problems for which there are no test cases. When code coverage analysis is used at such a juncture, the results are frequently eye opening. The analysis gives you invaluable feedback about the effectiveness of the test suite that you were relying on.
Some paths and portions of the code always remain untested, even with a large number of automated tests. This creates the danger of a problem being discovered by a customer that you could easily have found had it been "covered" in the test cycle. It is possible that some newly introduced features were not properly documented to enable testing. Another possibility is that some new code might have introduced a change in behavior that is leaving a large section of the product untested.
To perform code coverage analysis of a test suite, follow these steps:
- Profile an automation run with an application that is already instrumented by PureCoverage
- Collect coverage data
- Analyze the data to find out what portions of the source code are not exercised
- Add more tests to fill the gaps wherever possible
- Run test automation with the newly added tests
- Confirm that adding the new test improved your coverage
Here is an example of running a Java automation suite with PureCoverage. ClassicsJava is a sample program for ordering music CDs from a database. This program, included as a sample application in IBM Rational Functional Tester, was first instrumented by PureCoverage using this command:
Coverage/java java.exe âjar ClassicsJavaA.jar
Next, a test automation script was run on this application. One order was placed, and then the application was terminated. In Figure 1, methods that are covered are labeled Hit, and those not covered are labeled Missed.
Figure 1. PureCoverage showing hits and misses for an automated test program
Coverage analysis also helps find out if there are any redundant test cases: tests that exercise the same path in the code over and over again, causing unnecessary delays. In addition, it helps you generate smart verification test data as well. For example, a newly introduced change in code requires you to run the automation to verify that there are no regressions. You can examine the coverage analysis data to help you determine which subset of the automation suite needs to be run, allowing you to validate the new code in less time.
To drill down into a particular method, use the Annotated Source feature in PureCoverage. To access the Annotated Source view, double click the method. For this feature to work, source code needs to be available. One way to address this is to use a developerâs work environment and software, so that all the source paths and directories are easily accessible. Figure 2 shows the Annotated Source view for the method
ClassicsJava.getCustomerNames(). As shown in Figure 1, this method was missed (or not covered by the test), so the lines are in red font.
Figure 2. PureCoverage showing annotated source view for method
Performance analysis is another technique that, when used in conjunction with a test harness, yields enlightening results about the application. When test automation noticeably slows down with a new build, it is crucial to find out what is happening internally in the source code. What could have caused the slowdown? Here are some possible causes:
- Change introduced in the source code: There might be unnecessary routines that slow down the execution time. Maybe there is an algorithm that changed and affected performance.
- Change in test code: A test case that you added to improve code coverage may be causing the slowdown, possibly exercising a peculiar portion that is vulnerable to speed.
- Change in test data: A large amount of test data (or unusual data) may be stressing the code and causing bottlenecks.
- Change in environment or connectivity issues: If the network is slow, perhaps your hardware needs to be upgraded.
You should not rely on simply "noticing" that test harness runs are taking longer to report a performance issue. It makes sense to establish a method to measure performance. There are many ways to detect a performance problem:
- Log timing information alongside the automation
- Generate logs within test scripts to record time
- Write specific tests that measure performance
- Use a performance monitoring tool in conjunction with the test harness
The Quantify component of PurifyPlus is a performance monitoring tool that enables you to collect performance data. Figure 3 shows an example of the ClassicsJava program automation used with Quantify. The program was instrumented beforehand, and then automation was run (similar to the PureCoverage technique).
Figure 3. This portion of the Callgraph generated by Quantify shows methods that consume the most time
Quantify collects information regarding which methods are callers and descendants, and how much time they spend in those calls. You can examine this information to determine if there are any that can be eliminated.
It is important to note that proper filters should be used before running Quantify, otherwise the amount of data collected could be too large for analysis.
Follow these steps to do performance analysis using Quantify:
- Decide when to collect the data. Automation needs to be stable enough that it runs for a reasonable period of time without crashing.
- Set benchmarks, comparing expected time versus actual time for a specific number of tests.
- Run test automation with the instrumented program.
- Analyze the collected data, determining which tests or methods look like problems.
- Identify and investigate issues to see if they can be fixed, then repeat the process.
Send the results of performance analysis to your software developers to find out if there is anything that can be done to curb the slowdown. At times, you will find out that whatever change was introduced was a necessary evil. In that case, the benchmark should be adjusted so that any further slowdown will become apparent the next time.
There is so much value in doing this type of analysis, because it benefits both the automation suite and the application being tested. On rare occasions, it might be determined, for example, that the automation runs slowly and that this doesn't have much to do with the application being tested. Perhaps the tests could be trimmed down for each night of testing, so that not all tests run all the time. At times, using the profiling tool will decrease the performance incrementally, since tools interact with the software to collect the performance data. You will need to determine which tool best suits your application, and how to calculate the time spent by that tool to collect performance data during the automation run.
Runtime memory errors and leaks are among the hardest problems to detect in an application. They may be triggered by random events, and can be quite unpredictable. Purify helps pinpoint these types of errors. Purify is easy to use, and it has a great command line interface to help you get just the right information.
The following listings are very simple examples of incorrect memory usage. These may look simplistic, but their simplicity is deceptive. This is because, in a small module belonging to a very large application, these may show up randomly. If this piece of code is used over and over again in the course of application execution, problems will multiply, eventually causing an ugly crash.
Listing 1. A memory leak
// Allocate memory 1K (void) new char; â¦.. //Forget to de-allocate later in the program
Listing 2. Memory that is not initialized and used later
char *string1 = "start"; char *string2 = malloc(10); length = strlen(string2); //string2 is not initialized like string1
Suppose that test automation is running reasonably well, all tests are reporting success, and performance is fine. Then, all of a sudden, there is a crash that stops the test suite. The tester complains that, "There are these problematic issues that seem to appear during the course of running tests. The application crashes sometimes, and not always during the execution of some of the tests."
Many quality engineers have encountered the situation where it is neither easy nor straightforward to replicate an application crash. One of the reasons for this is that symptoms of incorrect memory use typically show up far from the offending code that actually caused them. Sometimes it is mere coincidence that the application seems to work correctly even with bad memory errors close to the starting point.
Here are just some of the many possible causes of a memory problem:
- Code dependencies and collisions occurring when different modules are running at the same time
- Incorrect memory use occurs when functions used for allocating and de-allocating memory do not match
- Bad data structure initializations going out of bounds of the allocated memory
In some organizations, it is expected that developers perform memory analysis before checking in their new code, to make sure that they did not cause an unintended error. When a solid regression suite is present for testing the application, incorporating a runtime error detection tool such as Purify in it is a smart thing to do. The tool will double-check interactions between different modules, ensure that they are working together correctly, and expose any code dependencies that were not apparent in the course of individual development of the programs.
To use Purify to perform memory analysis while running the test automation, follow these steps:
- Incorporate Purify in the test automation so that it runs in the background with minimal disturbance. This means that you would run your program prefixed with purify /savetextdata, similar to the PureCoverage and Quantify techniques described previously.
- Collect the runtime information. There might be leaks, dangling pointers, or incorrect memory usage.
- Analyze all the gathered data, summarize it in a simplified document that points out which test or portion of code caused the problem, and send this information to the developers.
- When issues are resolved, rerun Purify with the tests and report the results.
For C and C++ programs, the Purify results show warning, error, and info messages like MLK (Memory Leak), MPK (Potential Memory Leak), ABR (Array Bounds Read), and so on. Each of these Purify acronyms shows the kinds of memory errors plaguing the application. Figure 4 shows an example of a Purify file displaying errors in the sample application stocknt, which is included with the PurifyPlus software.
Figure 4. A Purify report on automated tests for a C++ application
For Java and managed code programs, Purify collects data regarding which methods or classes are responsible for allocating the largest amounts of memory, as shown in figure 5.
Figure 5. The Function Detail view for classicsjava automation when run under Purify
It is useful to get a picture of the overall memory use during the course of application execution, to find out which objects are unnecessary.
Make memory analysis a habit
Augmenting test automation with a memory analysis tool increases the overall efficiency in an organization. Test automation is already exercising your code in ways that the development team may not be aware of. In addition, when you use a memory analysis tool to gather relevant data, it can reduce the time that developers have to spend debugging and looking for problem areas.
At times, depending on the skill levels and time constraints in your organization, it might be advisable to collect the runtime data and just pass it along to the development team for analysis and investigation. Finding memory problems does not need to be restricted to the cases where a crash occurs. Analysis can be performed from time to time whenever source code or test code changes, to see what it might uncover.
Figuring out the best combination of when to use what
We, as quality engineers, don't necessarily have to wait for a problem to surface in order to perform runtime analysis. In fact, it is a good practice to assume that there is a problem, and then use runtime analysis to pinpoint what it is.
The earlier that you detect performance and memory bottlenecks, the easier it is for you to fix them. Some issues require re-programming and changing entire sections of code logic. Many programming languages have features that provide data collection for more information about application runtime (for example, asserts and exception handlers). Most programmers do incorporate memory problem finding tactics in debug versions of their applications. They might even have code within their program to measure the applicationâs performance.
Debugging and runtime analysis
However, there is a difference between debugging and runtime analysis. Debugging is usually done by developers, and can help them find small problems in the course of running their own module (even a single program at a time). Runtime analysis, on the other hand, is an approach that applies to the entire application, and it can be performed by developers or testers, or both.
Coverage analysis is more efficient when you use it with automation. Automated tests should be available in order to measure how much testing was done. However, you can perform coverage analysis using manual tests.
A loose guideline on which type of analysis is more desirable at a given time is to look at what type of customer issues you find, determine whether they belong to any or all of the following three categories, then choose the combination of analysis that needs to be done.
- Problem has been found for which no test exists: code coverage
- Problem is slowness, which occurs when a large amount of data or a certain kind of data is involved: performance issue
- Program dies or crashes abruptly when customer uses it: memory
Bridging the gap between developers and testers
Using test automation and runtime analysis together is a smart strategy that helps with the ultimate goal of delivering quality software. It can minimize the number of defects that your customer finds, as well as increase the level of confidence that your developers and testers have.
Prior to runtime analysis, sudden and unexpected crashes or bottlenecks caused your team a great deal of frustration. After, they gain a deeper understanding of the software behavior and intricacies involved. It may or may not be possible to fix all of the issues, or completely improve test coverage, but understanding the problem will help you plan for the future.
Using the test harness as one of the tools available, and then combining it with profiling tools, helps reveal pesky issues that may be hidden. Runtime analysis can be performed at any juncture of the software development lifecycle, even when there is no automation present. It can even be done when the complete source code is not available. Developers can use runtime analysis tools and techniques alongside quality engineers to identify and resolve issues during their debugging and unit testing stages.
This approach helps bridge the gap between developers and testers. Both will be working together to find defects in the software, rather than having their customers find them. This also helps improve communication between developers and testers. Testers benefit from getting a wider "white box" window into the source code to better understand the software they test. In addition, when testers report their runtime analysis findings (whether it is a performance problem, a memory issue, or code coverage), they are giving fact-based feedback to developers on where work needs to be done so as to enhance quality.
The author would like to thank David Lofgren for reviewing this article and Don Nguyen for helping with the Classics Java sample.
- Stay current with developerWorks technical events and Webcasts.
- IBM Rational PurifyPlus product page: Find technical documentation, how-to articles, education, downloads, and product information about Rational PurifyPlus.
- Browse the technology bookstore for books on these and other technical topics.
Get products and technologies
- Download a free trial version of Rational PurifyPlus V7.0.
- Download IBM product evaluation versions and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.
- Runtime Analysis with PurifyPlus Family of Tools forum: Ask questions about Rational PurifyPlus.
- Check out developerWorks blogs and get involved in the developerWorks community.
Dig deeper into Rational software on developerWorks
Get samples, articles, product docs, and community resources to help build, deploy, and manage your cloud apps.
Experiment with new directions in software development.
Software development in the cloud. Register today to create a project.
Evaluate IBM software and solutions, and transform challenges into opportunities.