Introduction to the code coverage tooling in IBM Rational Application Developer

Generating code coverage statistics for your Java applications

Code coverage is an important metric used in software testing, because it provides insight into how well your application is being covered by a set of test cases. This article shows you how to use the code coverage offering in IBM® Rational® Application Developer to generate coverage results for your Java™ applications, and provides details on how to analyze the results in order to improve your tests.

Share:

Paul Klicnik (pklicnik@ca.ibm.com), Software Developer, IBM

Paul KlicnikPaul Klicnik is a Software Developer at IBM Toronto lab in Markham Ontario. He has worked on Code Coverage since 2008, and worked in the general area of performance and testing tools since 2006. Paul has worked on several key IBM products, including IBM Rational Performance Tester and IBM Rational Application Developer, as well as the Eclipse Test and Performance Tools Project (TPTP) project.



08 April 2010

Also available in Chinese

What is the Rational Code Coverage feature?

Code coverage is an important aspect of software testing, and can be considered fundamental to the overall system testing of a component. The motivation behind coverage tooling is simply to give you (as a developer or tester) more insight into the areas of code that are being exercised by a set of test cases. This information is useful because you can then use it to devise new test cases to achieve adequate coverage.

The IBM® Rational® Code Coverage feature is a tool that integrates with IBM® Rational® Application Developer. You can use it to generate and analyze coverage statistics for your Java applications. The tooling generates statement coverage statistics for the application under test (that is, the number or percentage of lines in your application that have been executed).

The current offering of the Rational Code Coverage feature is only available in Rational Application Developer Version 7.5 and later. This article assumes the use of Rational Application Developer V7.5.4. The section on configuring an IBM® WebSphere® Application Server for code coverage assumes the use of V7.0, although a variation of the provided instructions will work on previous versions.

Instrumentation

In order to properly analyze the coverage statistics in the Rational Code Coverage feature, it is important to understand the technology used behind the scenes.

Rational Code Coverage uses an instrumentation engine to manipulate the bytecode of a class and inject custom calls to the coverage data collection engine. Figure 1 provides a high-level overview of the process:

Figure 1. Overview of the Rational Code Coverage execution environment
Input, output, and callback

Basic blocks versus executable units

The instrumentation engine operates on units of bytecode called executable units. The definition of executable unit is slightly different than the traditional definition of basic block, but the differences are important to take into account when the results are analyzed.

By definition, a basic block is a set of instructions that cannot be branched into or out of. The key idea here is that when the first instruction runs, all of the subsequent instructions in that block are guaranteed to be executed without interruption. It follows that a basic block can be conceptually considered a single group or block of instructions. In general, basic blocks end on branch, call, throw or return statements.

An executable unit begins at the start of every basic block and at any instruction that corresponds to a line of source code that is different than the previous instruction. What differentiates an executable unit from a basic block is the condition that triggers the end of the executable unit. For example, the divide instruction is not considered to be the end of an executable unit despite the fact that it can throw an exception.

The instrumentation engine in Rational Code Coverage is used to inject custom code at the start of every executable unit. Consequently, you can customize the Rational Code Coverage feature to report statistics down to the executable unit level of granularity (in other words, block coverage). Figure 2 provides an overview of how the instrumentation engine modifies the bytecode to support code coverage.

Figure 2. Overview of the bytecode instrumentation
Overview of the bytecode instrumentation

Larger view of Figure 2.


Generating coverage statistics in Rational Application Developer

One of the major advantages of the Rational Code Coverage feature is that you can enable it on any Java project in Rational Application Developer by navigating to the Code Coverage panel in the project Properties, as shown in Figure 3.

Figure 3. Code Coverage panel in the project Properties
Set the acceptable coverage level percentages

Select the Enable code coverage check box in Figure 3 to enable code coverage for the project, and to instrument the classes in your project under the covers. You can also use this panel to customize acceptable coverage levels for each step of granularity. The supported levels of granularity are described following:

  • Type coverage: the percentage of types covered in a class
  • Method coverage: the percentage of methods covered in a class
  • Line coverage: the percentage of lines covered in the class file
  • Block coverage: the percentage of blocks covered in a class file. Note that a block refers to an executable unit (as described previously)

You can also specify custom filters, and they are used to control what gets instrumented in your project. By default, all of the classes in your project are instrumented, but you can create custom filters to exclude target packages or specific types, if there is a need to restrict the results.

Package Explorer

After you enable code coverage on a project, coverage statistics will be generated the next time that the application is launched. Note that statistics will not be generated for all types of launch configurations automatically. Table 1 displays the launch types that are supported from within Rational Application Developer.

Table 1. Supported launch configurations
Launch Type
Java Applet
OSGi Framework
JUnit
JUnit Plug-in Test
Java Application
Eclipse Application
Standard Widget Toolkit (SWT) Application

A sample application has been provided in the Downloads section and will be used throughout this article. The application is a simple representation of different vehicles (car, van, motorcycle, and so on) and the various parts associated with each vehicle. The UML diagram outlining the structure of this application is displayed in Figure 4.

Figure 4. UML diagram for the sample application
Types and parts of a vehicle

There are two JUnit tests already defined in the project: TestCar.java and TestCarImproved.java. As the names suggest, these tests target the Car.java class. While in the Java perspective in Rational Application Developer, you can start the TestCar.java test by right-clicking TestCar.java and selecting Run As > JUnit test. The results of the JUnit test will appear in the JUnit view as normal. The coverage results are integrated into the Rational Application Developer UI, and you can analyze them by switching back to the Package Explorer. Figure 5 displays a sample result set for the TestCar.java test.

Figure 5. Coverage results for TestCar.java displayed in the Package Explorer
Coverage percent shown next to elements in the Explorer

By default, the UI is annotated with only the line coverage information; however, you can change this in the workbench Preferences, and optionally choose to include coverage for packages, types, and blocks. The percentage beside each Java item is a breakdown of the line coverage for the last execution. You can drill down into the various Java artifacts (for example, classes, types, and methods) in the Package Explorer to get coverage statistics at a lower level of granularity.

The results are color-coded depending on the success rate: by default, red indicates that the acceptable coverage level has not been met while green indicates that the appropriate coverage level was achieved. Naturally, the goal of the test is to reach an acceptable coverage level on the classes of interest.

Based on the results shown in Figure 5, the first test was inadequate: the Car class (and abstract parents AbstractFourWheelVehicle and Vehicle) did not reach appropriate coverage level. Luckily, you have a second attempt to execute: TestCarImproved.java. Again, you can execute the test as a normal JUnit and the results are automatically updated in the Package Explorer (Figure 6).

Figure 6. Code coverage results for TestCarImproved.java displayed in the Package Explorer
More componenets have acceptable coverage and are green

Java editor

Line coverage results are also displayed and marked in the Java editor, and you can use it to give a more precise indication of which lines are covered in each class. After coverage statistics have been generated, you can open any class in your project with the Java editor, and the left ruler bar in the editor shows the coverage information. Figure 7 displays the results for Vehicle.java:

Figure 7. Coverage results displayed in the Java editor
green mark on left shows covered lines

The color indicators are the same as they are in the Package Explorer. That is, by default a green line was covered and a red line was not covered. There is a slight advantage in viewing the results in the Java Editor because it also indicates the partially covered lines. Partially covered lines can occur when there is more than one executable unit on a line of source code, but only one of them has been executed. As an example, look at the first line of code in the setTargetSpeed(int speed) method shown in Figure 7: the first executable unit is the if statement, and the second executable unit is the return statement. By default, a partial line is colored in yellow.


Generating reports

You can compile the code coverage results into reports and view them in Rational Application Developer, or save them to the file system for future analysis. You can generate two different types of reports: Workbench reports (Eclipse-based) and HTML reports. To generate a report, select Run > Code Coverage > Generate Report. Figure 8 shows the report generation dialog.

Figure 8. Report generation dialog
Select launch configuration from table with Name and Date

You can create and view a report in Rational Application Developer using the Quick View option on the dialog, or save it to the file system using the Save Report option.

Workbench reports

The workbench reports (also known as Eclipse-based reports) provide a consolidated view of all of the coverage statistics for your project, and contain coverage information for all of the classes in your project at execution time. Figure 9 shows a populated Eclipse-based report.

Figure 9. Coverage results in an Eclipse-based report
shows element, coverage, and covered and total lines

Workbench reports have the added advantage of being integrated in Rational Application Developer, so you can use them as a quick tool to provide insight into the parts of your code that require improved test coverage. As Figure 9 shows, the statistics in a workbench report contain coverage information for all levels of granularity: from a package to a method. Right-clicking any of the Java artifacts displays a pop-up menu with two additional actions: Show in Package Explorer and Open in Java Editor. These are useful tools for identifying and investigating areas of code with low coverage, because they highlight the selected area of code by opening it in the appropriate viewer or editor.

HTML reports

HTML reports display the same type of information provided in the Eclipse-based report, but in HTML format. These reports are particularly useful when saved to the file system, because they provide a way for the coverage results to be analyzed independently of Rational Application Developer, shared with team mates, or published to a Web site for viewing.


Generating statistics outside of the workbench

One of the major features of the Rational Code Coverage tool is its ability to generate statistics outside of Rational Application Developer. This provides extra flexibility and enables you to customize your environment to take advantage of the Rational Code Coverage feature in your systems. For example, one natural combination would be to set up a nightly build environment and generate statistics with JUnit tests on the nightly driver.

You can integrate the Rational Code Coverage feature into your environment by performing the following three steps: instrumentation, execution, and report generation.

Step 1. Instrumentation

There are two different approaches that you can use to instrument your application. The first is to use the instrument.bat/sh script provided in the <RAD_HOME>/plugins/com.ibm.rational.llc.engine_<date>/scripts directory. This article does not focus on this script, but you can reference the Rational Application Developer documentation for more information if necessary. The second approach is to use the instrumentation Ant task provided by the Rational Code Coverage feature. Listing 1 shows an example usage of the instrument task configured to target the sample application in this article.

Listing 1. Example usage of instrument Ant tasks on the sample application in this article
<target name="instrument">
  <taskdef name="instrument" 
    classname="com.ibm.rational.llc.engine.instrumentation.anttask.InstrumentationTask" 
    classpath="{path to com.ibm.rational.llc.engine plugin}"/> 
  <instrument saveBackups="true" 
    baseLineFile="project.baseline" 
    buildPath="VehicleProject" 
    outputDir="VehicleProjectInstr"/> 
</target>

A quick overview of the expected parameters is outlined in Table 2, following.

Table 2. Input parameters for instrumentation tasks
ParameterDescription
buildPathThe path to the project on the file system
outputDir(Optional) The output directory of the instrumented project. If not specified, the classes in the buildPath will be instrumented in place.
baseLineFile(Optional) The output location of the baseline project index file. See the following paragraph for more information on this file.
saveBackups(Optional) Set to true if the original class files should be backed up before instrumenting.

Both approaches to instrumentation will output a baseline file. A baseline file is a notion specific to the Rational Code Coverage feature. The baseline file contains an index of all of the classes in your project, and maintains additional metadata about each class. This file is used at the reporting step (Step 3 following) to determine which classes in your application were not covered. This step is necessary because the Rational Code Coverage data collection engine is only notified of a class when it is loaded by Java™ Virtual Machine (JVM), and so a list of the classes that were not executed cannot be determined without additional metadata. If the baseline file is not present at reporting time, the classes that were not loaded will be absent from the report.

Step 2. Execution

In order to execute the instrumented classes, the Java environment must be configured correctly at launch. The two specific parameters needed for execution are explained below:

  • -Dcoverage.out.file=<absolute path to output file>: the file specified by this JVM argument is the output location of the coverage statistics
  • Add the <Rational Application Developer HOME>/plugins/com.ibm.rational.llc.engine_<date>/RLC.jar to the classpath: because the code has been instrumented with callbacks to the the Rational Code Coverage data collection engine, the RLC.jar file needs to be on the classpath at runtime.

These parameters can be supplied to a JUnit Ant task. Listing 2 provides example usage.

Listing 2. Example of how to specify the the Rational Code Coverage feature arguments in an Ant launch
<target name="run">
 <junit showoutput="true" fork="yes">
  <jvmarg value="-Dcoverage.out.file={absolute path to the output file}"/>
  <classpath>
   <pathelement location="{absolute path to the 
    <Rational Application Developer HOME>/plugins/com.ibm.rational.llc.engine_<date>
        /RLC.jar file}"/>
   <pathelement location="{path to the project classes}"/>
   <pathelement path="{absolute path to the junit.jar}" />
  </classpath>
  <test name="com.ibm.vehicles.tests.TestCar" outfile="TestCar" />
 </junit>
</target>

Step 3. Report generation

You can generate reports using another Ant task provided by the Rational Code Coverage feature. This task uses the reporting functionality provided by the BIRT Eclipse.org project, and thus requires that you download the BIRT V2.3.2 Reporting Engine standalone offering. This can be done by navigating to http://www.eclipse.org/birt/download, select the V2.3.2 release and downloading the Report Engine offering. Note that this Ant task can only produce HTML reports.

Listing 3 provides sample usage of the reporting Ant task. Note that, as input, it requires the coveragedata file generated in Step 2 and (optionally) the baseline file generated in Step 1.

Listing 3. Example usage of the report generation Ant task on the sample application in this article
<target name="generate-report">
 <path id="lib.path">
  <pathelement location="{absolute path to the 
    <Rational Application Developer HOME>/plugins/
        com.ibm.rational.llc.common_<date>.jar plugin}"/>
  <pathelement location="{absolute path to the 
    <Rational Application Developer HOME>/plugins/
        com.ibm.rational.llc.report_<date> plugin}"/>
  <pathelement location="{absolute path to the 
    <Rational Application Developer HOME>/plugins/
        org.eclipse.equinox.common_<date>.jar plugin}"/>
  <fileset dir="{absolute path to the BIRT ReportEngine directory}\lib" includes="*.jar"/>
 </path>

 <taskdef name="code-coverage-report" 
   classname="com.ibm.rational.llc.report.birt.adapters.ant.ReportGenerationTask" 
   classpathref="lib.path"/>

 <code-coverage-report 
   outputDir="{absolute path to the report output directory}" 
   coverageDataFile="{absolute path to the coveragedata file generated in step 1}" 
   baseLineFiles="{absolute path to the baseline file generated in step 1}"/>
</target>

An example HTML report is displayed in Figure 10. Generating HTML reports using the Ant task provides a means by which users can view the statistics generated in an Ant environment independently of Rational Application Developer.

Figure 10. Coverage results in an HTML report
Navigation on left, results on right

Try it out!

The Downloads section of this article provides sample scripts and build files for an Ant environment that can be used to instrument, execute, and generate reports on the sample application. If you're interested in testing this environment out, refer to the README file in the Standalone.zip file.


Generating statistics on WebSphere Application Server

Generating code coverage statistics with WebSphere Application Server is supported, but unfortunately automatic configuration from within Rational Application Developer is not supported. However, the Rational Code Coverage feature offering is flexible enough to be integrated into server environments, including WebSphere Application Server. To configure your WebSphere Application Server for code coverage, you'll need to follow these instructions:

  1. Launch the server
  2. Log into the Administrative Console
  3. In the left option pane, expand Servers
  4. Expand Server Types
  5. Click WebSphere application servers
  6. Select the appropriate application server
  7. Expand Java and Process Management under the Server Infrastructure section in the option pane on the right of the page
  8. Click Process definition
  9. Click Java Virtual Machine under the Additional Properties section
  10. In the Boot Classpath section, add the RLC.jar file. As previously explained, this .jar file is in the Rational Code Coverage data collection engine and is located in <Rational Application Developer HOME>/plugins/com.ibm.rational.llc.engine_<date>/RLC.jar
  11. In the Generic JVM arguments, add the -Dcoverage.out.file={output file} JVM argument. As previously described, this argument specifies where the output statistics should be saved.
  12. Save the server configuration and restart the server.

Figure 11 displays a screen capture of the Administrative Console after the described modifications have been made. Note that the instructions must be followed on each server instance that is executing a code coverage-enabled application.

Figure 11. WebSphere Application Server configuration for the Rational Code Coverage feature
Properties include classpath and heap size

After the server is configured for code coverage, you can deploy code coverage-enabled applications to the server manually (from the Administration Console) or using the integrated support in Rational Application Developer. Note that the coverage results are not automatically imported into Rational Application Developer for analysis, so you need to perform the following steps to import the statistics back into the workspace:

  1. In the Java perspective in Rational Application Developer, right-click in the Package Explorer and select Import
  2. Expand Code Coverage
  3. Select Code Coverage Data File and click Next
  4. Select the Data is located on the file system option and click Next
  5. In the Coverage Data file field, select the coveragedata file on the file system that was produced by the server
  6. In the Into folder field, select a directory in the workspace to save the imported file.
  7. Select the appropriate project in the Associate with Project area. You should associate the statistics with the project in your workspace that contains the source code that was used to generate the statistics on the server.
  8. Click Finish

When the coveragedata file is in the workspace, you can display statistics in the UI or generate reports. This can be accomplished by right-clicking the coveragedata file and selecting Code Coverage > Show code coverage indicators or Generate Report. This functionality is beneficial because it provides access to all of the tools for analyzing results in Rational Application Developer.


Downloads

DescriptionNameSize
Sample application used in this articleSampleApplication_CCRAD.zip26KB
Sample standalone execution environmentStandalone_CCRAD.zip28KB

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational, Java technology, DevOps
ArticleID=479843
ArticleTitle=Introduction to the code coverage tooling in IBM Rational Application Developer
publish-date=04082010