Code coverage analysis

The cost of identifying and fixing a defect after a software system has been deployed is unpredictable but almost always significant. For example, if a financial trading portal shuts down due to a defect, the liability cost can be huge, and the pressure to fix it immediately will be immense. It is a lot easier and more economical to fix defects in the Development phase. Therefore it is extremely important that you systematically test your code as you develop it.

Code coverage analysis is an effective and commonly used technique to assess the quality of a test suite. It tells you which segments of your code have not been exercised by your test suite. For instance, if you add new functionalities in your software, but forget to add corresponding test cases to the test suite, it discovers these untested pieces of codes so that you can design test cases to execute them. It also provides a quantitative measure of the effectiveness of your test suite by reporting the percentage of the code that has been covered by the tests, so that you can establish code coverage quality benchmarks. If code coverage falls below these benchmarks, it indicates that testing has not been adequate, and you need to take corrective measures.

Code coverage analysis can be performed at various levels of granularity. Method-level code coverage reports whether a method was executed or missed by the test suite. It is a shallow measure that you can use in initial stages of testing to identify obvious deficiencies in your test suite quickly. Line-level code coverage is finer grained: for each line of source code, it reports whether the line was executed by the test suite. It provides you with vital clues to add test cases for executing uncovered lines. Essentially, it insures that the test suite executes each line of code at least once, so that no untested code is shipped with your product.

In rest of this article, you will learn how to use IBM Rational Application Developer (IRAD) to profile your Java™ applications and collect line-level code coverage data. You will examine several relevant views that display coverage statistics and assist you in identifying Java packages with a low level of code coverage, as well as show you annotated source code with each line marked as covered, uncovered, or partially covered. You will also learn how to analyze statistics for a single test case, and aggregated statistics for a set of test cases.

Back to top

Line-level code coverage using IRAD

This article will demonstrate IRAD's code coverage capabilities using a simple Java program that checks whether a given number is prime. This choice of a simple program is deliberate, so that you are not distracted by the complexity of the example and can rather focus on code coverage analysis. The Prime class is shown in Listing 1. The program takes a single command line argument and checks whether it is a prime number. Now, the isPrime method can be implemented more elegantly and compactly with fewer return statements. Also, a somewhat uncommon coding style is used (having a if statement and a return statement in the same line). These choices, however, are deliberate, in order to demonstrate finer aspects of IRAD's code coverage capabilities.

Create a Java project in your IRAD workspace, and add the Prime class to the project. Also add another class named PrimeSieve to the same project. The default file that gets generated upon adding a class will suffice; you don't need to add anything to it. This additional class will help in demonstrating how an untested class appears in IRAD's code coverage analysis.

public class Prime {
    public static boolean isPrime(int n) {
        if (n<2) return false; // 2 is the smallest prime
        if (n==2) return true;
        if (n%2 == 0) return false; // even number, but not 2

        int limit = (int) Math.floor(Math.sqrt(n));
        for (int i=3; i<=limit; i+=2) {
            if (n%i == 0) return false; // divisible by i
        }
        return true; // n is prime
    }

    public static void main(String[] args) {
        if (args.length != 1) {
            System.err.println("Usage: Prime <number>");
            return;
        }
        try {
            int num = Integer.parseInt(args[0]);
            System.out.print(num);
            if (isPrime(num)) {
                System.out.println(" is a prime number");
            } else {
                System.out.println(" is not a prime number");
            }
        } catch (NumberFormatException e) {
            System.err.println(args[0] + " is not a number");
        }
    }
}

Collecting code coverage data

Before you start profiling the Prime class, make sure that the IBM Agent Controller is running. It is packaged with IRAD and gets installed along with it. In the Microsoft® Windows® environment, it usually runs as a service (see Resources for more details on the IBM Agent Controller).

Here are the steps to profile your application for line-level coverage:

1. Start the Profile dialog by clicking Run menu and selecting Profile.
2. Create a configuration for Prime under the Java application.
3. Enter the project name, application name, and other details in the Main tab.
4. Enter 29 as Program arguments in the Arguments tab.
5. Go to the Profiling tab and select Code Coverage - Method and Line Level profiling set (Figure 1 shows the Profile dialog during this step).
6. Click the Profile button, and IRAD will switch to the Profiling and Logging perspective, execute your application, and collect line-level code coverage data.

Examining code coverage views

Once you have profiled your program and code coverage data has been collected, the Profiling Monitor view in the Profiling and Logging perspective will show a profiling resource named Method and Line Code Coverage for each execution. Select a profiling resource and right-click it, then select Open With from the context menu; this will bring up a list of views that can be used to visualize code coverage data.

Figure 2 shows a screen dump in the Profiling and Logging perspective from a code coverage analysis session. The pane at the bottom is the Console view, which shows the program output: 29 is a prime number. The pane on the left is the Profiling Monitor view. The first profiling resource in the view has been selected, and the pop-up menu is showing two options: Coverage Details and Coverage Statistics. If you select Coverage Details, IRAD will bring up two views: Coverage Navigator (right pane) and Annotated Source (center pane). Selecting Coverage Statistics will take you to the Coverage Statistics view (shown in Figure 3).

The Coverage Statistics view is the simplest, and it has a coverage statistics table. It shows whether a method was missed or hit, how many units were hit, and the percentage of total units that were hit. Later in this section, you will learn more about what constitutes a unit. Looking at Figure 3, you can tell that the main and isPrime methods were hit, and the constructor Prime was missed. You may be surprised because there is no constructor in the Prime class definition, but the Java compiler generates default constructors in such cases. Both the main and isPrime methods are static, and an object of type Prime is never created. Hence the constructor is never invoked, which is why it was missed. You should watch % Units Hit closely. It is a very effective indicator of code coverage: the higher the percentage, better.

By default, IRAD doesn't tell you how many calls were made to each method. You need to activate the count mode to get that information: you will learn how to do this in the next section. This section avoided adding the activation step in order to get you started quickly and in the simplest possible way, but screen dumps include that information to utilize the opportunity to explain a view in complete detail.

The Coverage Navigation view (right pane in Figure 2) graphically shows coverage levels of packages, classes, and methods, and it allows you to navigate through them. If you select any of the packages, classes, or methods, the Annotated Source view (center pane in Figure 2) will be updated to show relevant coverage information.

If you select the topmost (root) entry in the Coverage Navigation view, the Annotated Source view will show coverage statistics for all packages in a table as well as in a color-coded histogram. If you select a package in the Coverage Navigation view, the Annotated Source view will show you coverage statistics for all classes in the package. In Figure 2, the default package is selected in the Coverage Navigation view, and the Annotated Source view tells you that code coverage for Prime.java and PrimeSieve.java is 55.56% and 0% respectively. Any source file with 0% coverage (shown as a red bar in the histogram) requires your urgent attention, because it indicates that some files are completely missed by the test suite.

If you select a class or method in the Coverage Navigation view, the Annotated Source view shows annotated source code with each line marked as fully covered ( sign), not covered ( sign), or partially covered ( sign). Figure 4 shows annotated source for a test run with input 29.

A line is fully covered if all units in the line have been executed by the test, and is not covered if no unit has been executed. If some of the units are executed, then the line is partially covered. To understand this better, select the isPrime method in the Coverage Navigator view, and clear the Show Source with Annotation button at the top right corner of the Annotated Source view. You will see a coverage statistics pie chart for the isPrime method, and a table with unit-level breakdown for each line (Figure 5).

Look the details for line #3 (if (n<2) return false;). The table shows that there are two units and only one was hit. The two units are the condition and the return statement. Further, the first unit was hit once, and the second unit was not hit at all. It indicates that the condition was never true. Since only the first unit was hit, the line is marked as partially covered. Similarly, line #7 has 3 units: a call to the Math.sqrt method, a call to the Math.floor method, and assignment to the limit variable. All three units were hit, so the line is marked as fully covered.

Back to top

Advanced features

Aggregate statistics from several runs

As you noticed in the previous section, the test case with input 29 did not cover all lines. Typically, you will have a test suite with many test cases, and you would want to know aggregated coverage statistics for the test suite. Let's assume that, for the Prime program, the test suite has test cases with the following program arguments: 29, 25, 4, 3, 2, 1, "string-instead-of-a-number", and no argument. Profile the program for each of these arguments by following the steps given in the Collecting code coverage data section, but provide a different program argument in Step 4 each time.

Now you have eight sets of coverage statistics and you want to see the aggregated statistics. There is an option drop-down in the top-right corner of the Profiling Monitor view (see Figure 6). Select Distributed Layout instead of Simple Layout. In the Distributed Layout, all runs are organized according to profiling monitors and host machines. By default, all profiling data is collected under a profiling monitor named DefaultMonitor. The name of your machine will appear as the host under the DeafultMonitor, and all test runs for the Prime program will appear as profiling resources under that host. You can choose to aggregate data either at the monitor or host level, and the code coverage views will display results aggregated on all test runs.

In the Profiling Monitor view, select a monitor or a host (instead of a profiling resource), and follow the same steps as you would for seeing coverage for a single test run. Right-click and select Open With, then select Coverage Details or Coverage Statistics from the view list. Now these views will show aggregated coverage statistics instead of statistics for individual test cases. Figure 7 shows coverage statistics aggregated for all tests under the DefaultMonitor, and now 94.44% of the units are executed (note that the default Prime constructor is still not executed). The Annotated Source view in Figure 8 shows that all lines, except the first line, are marked as fully covered. The compiler-generated default constructor is mapped to the first line, and therefore that line is marked as not covered.

Fine tune data destination

You can change the monitor name from DefaultMonitor to something more suitable. In the Profile dialog, select the Profiling tab, then select the Destination tab. Change the Monitor text field to whatever name you want to give it (see Figure 9). The ability to organize test runs under various monitors is quite useful. It allows you to segregate test cases, and to examine code coverage within each segment. Let's say that part of your test suite is run nightly, another part is run weekly, and finally the whole suite is run once a month. For each of these test suite runs, you can set IRAD to collect coverage data under different monitors, and you can examine each of these individually for the extent of code coverage. This can help you in selecting test cases for nightly and weekly run such that you get maximum code coverage.

Activate count mode and specify method patterns

You used the Code Coverage - Method and Line Level   profiling set (Figure 1) to collect line-level code coverage data. This profiling set comes built-in with IRAD. It tracks whether a unit in a source code line was executed, but doesn't keep count of how many times each unit was executed. To count hits per unit, you can either modify this profiling set or create a new one by clicking the Edit or Add button, respectively, in the Profile dialog (Figure 1). When you do so, IRAD will start the Profiling Set wizard. Click Next, and you will see the dialog shown in Figure 10. This profiling set uses the Method and Line Code Coverage profiling type, which has a checkbox named Count mode activated. If you select that checkbox, IRAD will keep a counter for each unit and will track hits per unit for each unit that was shown in Figure 5. If you hover over a line in Annotated Source view, a tool-tip with the count information will appear.

In the same dialog (Figure 10), you can also specify Method patterns. IRAD collects coverage data continuously while your program is running, and it passes that data to code coverage views at periodic intervals. If you specify Method Patterns, instead of updating code coverage views at periodic intervals, IRAD will pass the data to code coverage views whenever a method matching one of the patterns is executed.

Back to top

Summary

Code coverage analysis measures the effectiveness of your test suite, and therefore is an indirect measure of the quality of your software product. IRAD can provide you line-level coverage statistics that are fine-grained to the extent that each line of source code is further broken into atomic units. It also provides you a simple yet powerful mechanism to organize test runs under different monitors, and an easy way to switch between statistics for individual tests and aggregated statistics. In addition, it has a tabular Coverage Statistics view and a graphical Annotated Source view for effective visualization of the coverage data, so that you can quickly discover untested code. By identifying the lines of code that have not been exercised, you can design test cases that will execute those lines, and add these tests to the test suite. In this way, you will insure that the quality of your software is not undermined by inadequate testing.

Resources

Learn

• For specific information on how to set up and start IBM Agent Controller, see the Rational Application Developer V6 Programming Guide.
  
  
• Learn about statement coverage techniques in the research paper A comparison of some structural testing strategies by Simeon Ntafos in IEEE Transactions on Software Engineering, vol. 14, No. 6, pages 868-874, June 1998.
  
  
• To get an overview of IRAD profiling capabilities, see Runtime problem determination with IBM Rational Application Developer.
  
  
• Learn about Code Review and Component Testing in the article Improving Software Quality with Rational Application Developer for WebSphere Software.
  
  
• Learn profiling concepts in Eclipse by reading An introduction to profiling Java applications.
  
  
• Read about other runtime analysis capabilities in IRAD: Java memory leak analysis andExecution time analysis.
  
  
• To get started with IRAD and IBM Rational Web Developer, read Introduction to IBM Rational Web Developer 6.0.
  
  
• Excellent guided tours on using IRAD are in the book An Introduction to IBM Rational Application Developer.
  
  
• Check out the IBM Rational Application Developer space at developerWorks.
  
  

Get products and technologies

• IBM trial products for download: Build your next development project using with IBM trial software, available for download directly from developerWorks.
  
  

Discuss

• Participate in the discussion forum.
  
  

About the author

Satish Chandra Gupta is a developer in the IBM Rational® PurifyPlus® group in Bangalore, India. His interests include compilers, programming languages, runtime analysis, Java memory leaks, type theory, software engineering, and software development environments. His research has been published in ACM/IEEE conferences. He received a B.Tech. from the Indian Institute of Technology in Kanpur (India), and an M.S. from the University of Wisconsin in Milwaukee (USA).

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• Code coverage analysis
• Line-level code coverage using IRAD
• Advanced features
• Summary
• Resources
• About the author
• Comments

Next steps from IBM

• Try: Rational Application Developer for WebSphere Software
• Demo: Build, test and deploy JAX-WS Web services
• Webcast: Webcast: Develop innovative applications efficiently using the right application foundation in today's economy
• Briefing: Software development conference
• Buy: Rational Application Developer for WebSphere Software

My developerWorks community

Dig deeper into Rational on developerWorks

Rate a product. Write a review.

Write a review for Rational products today!

Special offers