An introduction to runtime analysis with Rational PurifyPlus

from The Rational Edge: This article discusses runtime analysis in the context of other Rational best practices and outlines its enormous benefits to software developers, testers, and managers.

Share:

Goran Begic, Senior IT Specialist, IBM, Software Group

Goran Begic joined Rational Software in the Netherlands in 1999. Since then, he has worked in technical support, field enablement, product management, and sales for the IBM Rational PurifyPlus family of developer testing tools. He also has expertise in implementing agile development practices. In 1996, he earned a bachelor's of science in electrical engineering from the University of Zagreb.



Allan Pratt, Principal Engineer, IBM

author photoAllan Pratt is a Principal Engineer on the PurifyPlus product team in IBM's Rational software division. He has been a professional developer for over twenty years. He started out at Atari in the days of the 520ST, then joined the tools group at the old Taligent for a while before arriving at Pure Software in 1996. He's always stayed pretty close to the bare metal: his projects have included BIOS programming in 68000 assembly code, linkers and debuggers on multiple platforms, UNIX® scripts and tools, object-code instrumentation for Reduced Instruction Set Computer (RISC) and Explicitly Parallel Instruction Computing (EPIC) processors, and byte-code instrumentation for Java and .NET.



15 March 2007

Also available in Chinese

illustrationRuntime analysis is the practice of looking at a program's behavior based on what it does when it runs. This is different from "static analysis," where you analyze program behavior by looking at its source code or high-level structure; or "crash analysis," where you look at what a program leaves behind when it fails.

Debugging is one classic example of runtime analysis: You set breakpoints, step through the program, and look at argument values and variables, all to gain an understanding of what's actually happening as the program runs. When you're debugging, you're not interested in all the possibilities of what might happen, but, rather, what really does happen. That's what separates runtime analysis from static analysis.

Debugging is highly interactive and can be powerful, but it's limited in what it can show you. That is, it's hard to identify patterns and integrate higher-level behavior when your focus is narrowed down to the size of a source line or a variable.

Testing is another kind of runtime analysis: You feed in data or events and check to see that you got the right behavior from the program. This sort of testing is the cornerstone of software quality.

Still, traditional testing only shows you so much. A program can pass all its tests (give the right answers) and still have serious quality problems.

These are the reasons that powerful runtime analysis tools like IBM Rational PurifyPlus were created.

What is PurifyPlus?

PurifyPlus is a runtime analysis tool suite that monitors your program as it runs and reports on key aspects of its behavior:

  • How much memory does it use?
  • Does it leak memory?
  • Does it contain memory access errors?
  • How long does it take to run; where are the bottlenecks?
  • What is the thread running/sleeping behavior?
  • How much of the source code is actually running?

These key analysis features are all separate from the core "correctness" question: "Does it work?" and "Does it produce the right answers?" Testing can show whether your program works at all, but it takes runtime analysis tools like PurifyPlus to bring all these types of problems to light.

The major components of PurifyPlus are:

  • Purify for memory usage tracking and memory error detection
  • Quantify for performance analysis, code flow, and thread visualization
  • PureCoverage for source code coverage analysis

Runtime analysis in the program development cycle

Runtime analysis has a place during the interactive compile-edit-debug development cycle and also in the automated builds and tests that are a key part of any project's quality plan.

Runtime analysis during development and implementation

During interactive development, Purify can report on memory errors before new code is committed to the project. Developers can use PureCoverage to identify areas where new tests are required. Quantify can reveal unexpected code paths and bottlenecks early in the development cycle.

Runtime analysis during automated testing

When running automated builds and tests, PurifyPlus lets you monitor quality and collect metrics to ensure your project stays on track. When your tests run under Purify without reporting any errors, you know your program is free of those memory access bugs and leaks. When running with Quantify shows that performance meets targets, you know you have not introduced bottlenecks. And when PureCoverage reports a high level of code coverage, you know you have not introduced new blocks of code without also adding automated tests to exercise them.

You can increase the quality of the runtime analysis data you collect during automated testing by using testing automation tools, such as IBM Rational Functional Tester. Automated testing can give you a better depth of testing for a single iteration, and also help assess the impact of newly introduced changes on product quality. If software quality drops between two consecutive iterations of the component, runtime analysis data makes it very easy to find the responsible feature or code change.

Runtime analysis for program understanding

Sometimes runtime analysis isn't about correctness or quality at all. For example, runtime analysis can help with program understanding. Quantify, in particular, can show a call graph that reveals how the pieces of a program fit together, or it can reveal surprising paths that can explain unexpected performance degradations or interactions.

Basic PurifyPlus capabilities

As noted above, IBM Rational PurifyPlus has three major components: Purify for memory analysis, Quantify for performance analysis, and PureCoverage for code coverage analysis.

For C/C++ programs, Purify automatically detects and reports on memory leaks and memory access problems like using memory after you've freed it, freeing the same block twice, or reading from memory before you've initialized it. All of these are software quality issues that can lurk in a program even if its tests all pass. Problems like this can cause expensive failures in production after you release the product.

For Java and Microsoft .Net programs ("managed programs"), Purify tracks memory usage and memory references so you can see where your memory bottlenecks are, see where you are holding memory you ought to release, and compare before-and-after images to detect unintended increases in memory usage ("leaks") in the program.

Quantify tracks program performance and calling behavior, so you can see execution flow and identify bottlenecks. Quantify highlights code paths that take the longest, using the River of Time(tm) feature. In addition, Quantify lets you visualize thread execution behavior in your program.

PureCoverage tracks code coverage, so you can identify gaps in your testing and areas of your program that the runtime analysis tools are not seeing. Since Purify and Quantify only see the code that actually runs, the code coverage metrics provided by PureCoverage are critical to knowing that you are verifying and improving quality across your whole project.

Examples of runtime analysis with PurifyPlus

The major goals of debugging are to find the root cause of defects and understand application behavior.

Runtime analysis provides additional capabilities that supplement traditional debugging:

  • Visualization of application execution.
  • Measurement of vital runtime parameters, including memory usage, performance, and code coverage.
  • Error detection in user code.
  • Documentation of runtime behavior.

We will examine these capabilities below.

Visualization of application execution

To understand this capability, we'll look at four examples.

Visualization example 1: Code coverage

Runtime analysis with a tool such as Rational® PureCoverage® (included in Rational PurifyPlus) provides various views to code coverage information, one of them being Annotated Source. This particular view shows the source file of the examined application; the color of the line indicates the line's status after the executed test case: hit, missed, dead, or partially hit.

As Figure 1 shows, the user can see code coverage and the execution path for this test case.

Figure 1: Rational PurifyPlus display of annotated source for the C#.NET application in Visual Studio.NET
click here to enlarge

The code fragment in Figure 1 shows the exact path the application took when executing the switch statement on line 111. This particular line is marked as partially hit because line 122 hasn't been executed.

Visualization example 2: Threads

A runtime analysis tool such as Rational Quantify (included in Rational PurifyPlus) provides thread visualization, which can assist in detecting multithreading problems by marking the state of each of the threads while debugging. As Figure 2 shows, this allows you to examine the status of threads visually, while debugging.

Figure 2: Rational Quantify thread analysis view in Visual Studio 6
click here to enlarge

Visualization example 3: Call graph

Runtime analysis tools can also detect and display performance bottlenecks. The big advantage of this approach, compared to traditional methods, is that you can get an excellent overview of the execution path as well as precise information about the number of calls to the methods involved in the scenario. As Figures 3A and 3B show, the Call Graph in Rational Quantify highlights a chain of calls in the most time-consuming execution path; that is the performance hotspot. The thickness of the line connecting methods is proportional to the ratio between the time (or memory if you are using Purify) spent in this chain of calls and the rest of the application.

Figure 3A: Rational Quantify call graph of a mixed VB.NET and C#.NET application in Visual Studio.NET
click here to enlarge

Figure 3B: Rational Quantify call graph of a C/C++ application on Solaris
click here to enlarge

Visualization example 4: Memory usage

The first step in handling memory leaks is to detect them. One very intuitive way to do this is to visualize overall memory usage and take snapshots of memory in the program under test (PUT). This lets you see potential memory leaks in the running application. (This feature is available in Rational Purify for Java and .NET managed applications.) For example, if snapshots of memory usage for the component running on the server show that overall memory usage increases after each client session, then it is very likely that this component leaks memory (see Figure 4).

Figure 4: Overview of thread status and memory usage in Rational Purify for Windows
click here to enlarge

Measurement of vital runtime parameters

Visual error detection is just the first stage of runtime analysis. We also need to understand exactly what happens during the run. For that purpose, runtime analysis should be based on exact measurements of parameters vital for the application's execution:

  • Runtime performance
  • Memory usage
  • Code coverage

Again, we will look at examples to understand this runtime analysis capability.

Measurement example 1: Function list view

Function List View is a typical runtime analysis view that can be generated with a specialized Runtime analysis tool such as Rational Quantify (see Figure 5). It presents all important methods and/or objects of an application in tables that can be sorted by number of measured parameters; this allows developers analyzing code to find what methods used the most available memory at that point in time, as well as the slowest functions, the age of objects, and so forth.

This view provides exact information about the number of calls to methods, time spent in methods only, time spent and memory accumulated in selected methods and all their descendants, and so on.

Figure 5: Rational Quantify function list view for a Visual C++ Application
( click here to enlarge

Measurement example 2: Function detail view

A runtime analysis tool such as Rational Quantify can also extend the information in Measurement Example 1 to include information about the distribution of measured data between calling methods and descendants. This is shown in the Function Detail View (Figure 6). This view highlights callers and descendants that contribute to a performance or memory hotspot -- information that can help detect the exact cause of a performance or memory bottleneck.

Figure 6: Rational Quantify function detail view for a Visual C#.NET application in Visual Studio.NET (with IBM Rational XDE)
click here to enlarge

Measurement example 3: Method coverage module view

As we explained earlier, in some cases -- and especially when assessing the value of available testing methods -- it is useful to measure the percentage of code covered while testing, or simply to mark all the methods that haven't been tested after a series of tests. You can do this with a tool such as Rational® PureCoverage®, which yields precise information about untested and dead code vs. tested code (Figure 7).

Figure 7: Rational PureCoverage display of code coverage on the method level for a mixed C#.NET and VB.NET Application in Visual Studio.NET (with IBM Rational XDE)
click here to enlarge

Error detection: Runtime memory corruption in user code

This is the crowning glory of runtime analysis for native C/C++ applications. Runtime analysis can not only help to detect problems by displaying performance, memory, thread, and code coverage data in different views, but it can also pinpoint the exact location in the user code where the error is generated and/or caused. Runtime memory corruption detection is essential to ensure proper functioning and high quality of native C and C++ applications on all platforms. IBM Rational tools for runtime memory detection are Rational Purify and Rational PurifyPlus. Again, let's look at some examples.

Error detection example 1: Rational Purify memory error and memory leak reports

Rational Purify can pinpoint the exact line of code where a developer has created a memory error. It doesn't even need source files to provide this information; Rational Purify detects errors in memory and uses debug information to trace these errors back to the responsible lines of code (see Figure 8).

Figure 8: Rational Purify memory error and memory leak report for a Visual C++ application
click here to enlarge

In this particular example, the developer forgot to take the termination string into consideration when building an array variable. This error was causing the release build of the application to crash, whereas the debug build worked fine. This example is just one of the many ways in which runtime analysis significantly reduces debugging time for C/C++ development.

Error detection example 2: Quantify annotated source

Rational Quantify has a unique capability to measure distribution of time recorded for each of the user methods per line of code. Quantify annotated source displays times measured for each line of code, along with times spent and inside functions called on the line. This information can help you narrow the performance bottleneck down to an individual line of code (Figure 9).

Figure 9: Rational Quantify annotated source for a mixed visual Basic 6 and Visual C++ application in Visual Studio 6
click here to enlarge

Error detection example 3: Purify object and reference graph

In Java and .NET managed code, it is not possible to make runtime memory errors such as out of bounds reads and writes and free memory reads and writes, because the automatic memory management in the runtime subsystem prevents developers from directly accessing allocated memory. However, this automated memory management doesn't prevent programmers from forgetting references to the objects' allocated memory. As long as there is a reference to such dynamically allocated objects somewhere in the code, they will stay in memory and will not be cleaned by the automatic memory management (garbage collector). The net effect of such errors is the same as the effect of C/C++ leaks: The memory becomes unavailable for this and all other processes running on the host operating system. By doing a runtime analysis with Rational Purify, however, you can pinpoint the exact line of code where the reference to the object in question has been created (Figure 10).

Figure 10: Rational Purify object and reference graph for a Java application
click here to enlarge

Documentation of runtime behavior

Yet another way to leverage runtime analysis is by documenting the application's runtime behavior for future use. This helps you assess the overall quality of the project and measure the influence of newly introduced features and code changes on overall application performance, reliability, and test harness completeness. This advanced way of practicing runtime analysis involves collecting runtime data for each iteration of the component or application under development and analyzing the data at different stages in the project lifecycle. This information can help in determining overall project quality as well as the effect of new feature additions and bug fixes on overall quality.

When you use runtime analysis data together with source control tools such as IBM Rational® ClearCase®, you can easily detect which changes to the source code database are responsible for a faulty build and/or failure of automated tests; you will know which portions of the source code base were changed between the successful set of tests and the set of tests that failed. Not only that: You can identify the owner of those code changes and the exact times and dates they were introduced.

Advanced runtime analysis tools such as Rational PurifyPlus provide features to analyze multiple test runs by, for example, allowing the user to merge code coverage data from various tests or test harnesses, or to create separate data sets for comparisons of consecutive iterations of test measurements, as shown in Figure 11.

Figure 11: Rational Quantify compare runs report
click here to enlarge

In Figure 11, Rational Quantify compares two data sets and highlights chains of calls where performance has improved (green line) and chains of calls where performance has dropped (red line). The calculated data is available in both the Call Graph view and in the more detailed Function List view.

Even if you are not in a position to create an automated test environment, you can still automate data analysis by taking advantage of runtime analysis data saved as ASCII files. Figure 12 shows an example of a performance profile imported into Microsoft Excel.

Figure 12: Rational Quantify performance report imported into Excel
click here to enlarge

You can easily automate data analysis in Excel by creating simple Visual Basic applications, or with any of the popular scripting languages: Perl, WSH, JavaScript, and so on. PurifyPlus for UNIX comes with a set of scripts that can help you manage and analyze data collected from various tests.

Runtime analysis: The emphasis is on quality

Runtime analysis expands standard software development activities along one key dimension: concern for quality. It paves the way for achieving higher software quality through better understanding of the internal workings of an application under development. Remember: Source code that compiles is not proof of quality; detailed, reliable, and precise runtime performance, memory utilization, and thread and code coverage analysis data are the only way to determine that an application is free of serious errors and will perform efficiently.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational
ArticleID=201863
ArticleTitle=An introduction to runtime analysis with Rational PurifyPlus
publish-date=03152007