Test Requirements Coverage
Value of this report
One of the important aspects of product quality is the test case coverage of the product’s requirements. This helps the product development team to understand if there has been enough testing from a breadth perspective of all the requirements that the product is supposed to meet. Although by itself without deep examination of the thoroughness of the test cases, this metric is not enough to know that the product is properly tested. Remember that test coverage only is typically measured as direct or indirect coverage of a requirement by a test case and does not speak to how completely the requirements have been tested. Test coverage is only valuable if the requirements have been decomposed to the point where verification points have been incorporated in a test case to test every aspect of the covered requirement. Most test cases are linked to a requirement based on a partial coverage criteria with makes the test coverage optimistic at best. Having a single test case for each requirement is a fairly low bar of test coverage and yet many test plans fail to meet this threshold.
Let’s now turn to how to build a test requirement coverage report. There are two ways to look at coverage. First let’s look at the content of a set of requirements modules or collections and how many requirements have links to test cases. Then we will look at a test plan (and its associated test cases) and see which requirements it tests.
Requirements with linked test cases
Go into Report Builder and choose to build a new report. Select Requirements of Type “CapabilityRequirement” as the artifact type. This is a custom requirement type that we built into our ALM Requirements project area in DOORS Next Generation.
For the Traceability links section add an option link type “Validated By Test Case”. I believe that this is a new feature that is being added in the coming release and is present instead of the “Related Test Case” choice since this article is based on a Report Builder 6.0.1M1 pre-release installation. Choose the appropriate value for your installation.
This is what the link diagram looks like after the link type has been chosen.
After you click continue, we next limit the scope to the requirement and quality management project areas that contain the linked artifacts on which we want to report.
Now we add a condition on which requirements we want to report. In this case we want to see which of the Key Capabilities for each of the products have linked test cases.
So we choose to select only those requirements contained in these modules (or collections). Then we click the Add and Close button to save these selections.
After clicking Continue button, we move to Format results. This is a list of the default columns chosen for us by the Report Builder.
Since we do not need the Requirements Project as we are only in one, we delete this entry as well as the Requirement Type as they are always the same. Then we move the Requirements Collection up to the top as the first column since it is how we are wanting to sort the data (which we set in a later step). The resulting list of columns looks like this picture.
We want to sort first by Collection or Module and then by Requirement and then by Test Case. This provides the most organized list based on analyzing the requirements collections and which have associated test cases. Click Continue to complete this step.
We give the report a descriptive name, tag it with a category of reports (in this case under my name), and set the privacy to Public so that it will show up in the shared report catalog for all to use. Click Save to save the report and publish it in the public catalog.
To run the report to check that the data looks correct, click Continue and page through the resulting data. Note that some requirements have many test cases linked while others have no test cases.
Next I chose to open the Preview at the top of the builder page and switch the view to Graph and choose the Collection or Module as the X axis categories. In addition I switched the graph type to Stacked bar and Orientation to Vertical. In addition I count the number of times each test case is linked to a requirement in that collection. Then I save these as default settings.
Finally I can run the finished report and get the graph view as the default which I can change to a table view whenever I run the report. In addition by clicking on an individual slice of a bar graph, I get tabular data on only that part of the graph. By the way the bottom chunk of each bar represents those requirements without test cases. Currently selecting this chunk displays all of the requirements data for the entire collection or module. In a future version, this should only display those requirements without linked test cases.
In another blog entry I will post the second requirement coverage report.