Topic
  • 3 replies
  • Latest Post - ‏2007-06-13T15:29:33Z by nagkumar
SystemAdmin
SystemAdmin
210 Posts

Pinned topic What metrics can be used to show Unit Testing reduces Defect Rate

‏2007-05-29T18:31:39Z |
I just had an interesting discussion with a coworker regarding our introduction of unit tests.

The impetus for introducing the tests is to reduce our defect rate, and all the ROI expected from catching defects early.
No surprises there.

I should mention that our working definition of defect rate is "Bugs Logged per month." Perhaps that ought to be "Bugs logged per month per new LOC" or something else?

My question is, once we have introduced unit testing, what metrics can we use to show whether it's working or not? Intuitively, I thought "that's easy, if the defect rate drops, testing is working."

Our discussion centered on the following confounding factors: i.e. The original untested codebase is stabilizing, newer features aren't exercised in the field as rigorously, refactoring is making the original codebase more robust, etc. Any of which could be contributing to a reduction in defect rates, in the absence of units tests.

Thoughts? Our project is an already existing code-base that is in maintenance, including regular addition of new features.

I'm interested in hearing other experience in this.
Updated on 2007-06-13T15:29:33Z at 2007-06-13T15:29:33Z by nagkumar
  • SystemAdmin
    SystemAdmin
    210 Posts

    Re: What metrics can be used to show Unit Testing reduces Defect Rate

    ‏2007-06-01T01:18:03Z  
    You could start to measure code coverage-- that way you can begin to assess what's being tested and what's not. I'd also begin to correlate defects to actual code and assess coverage against that code.

    Your point about refactoring and what not could certainly lead to fewer defects but without the assurance of actual tests to verify functionality you could be skating on thin-ice, don't you think? Defects can pop up that may have been caught earlier; plus, having tests in place sure does make refactoring a lot safer, not to mention easier too.
  • SystemAdmin
    SystemAdmin
    210 Posts

    Re: What metrics can be used to show Unit Testing reduces Defect Rate

    ‏2007-06-08T14:19:32Z  
    If you're measuring metrics, you have to have a baseline. Your baseline isn't "Bugs Logged per month", since unit testing has no impact on EXISTING code, only on TESTED code. You need to measure the number of bugs found in the code you've tested, and look to see if that number decreases each time that code is updated (and re-tested).

    Over time, your "Bugs Logged per month" should go down, but only once all of your existing code has been tested at least once.
  • nagkumar
    nagkumar
    3 Posts

    Re: What metrics can be used to show Unit Testing reduces Defect Rate

    ‏2007-06-13T15:29:33Z  
    If you're measuring metrics, you have to have a baseline. Your baseline isn't "Bugs Logged per month", since unit testing has no impact on EXISTING code, only on TESTED code. You need to measure the number of bugs found in the code you've tested, and look to see if that number decreases each time that code is updated (and re-tested).

    Over time, your "Bugs Logged per month" should go down, but only once all of your existing code has been tested at least once.
    You could look for Regression bugs due to new fixes as a measure.. The regression bug count should become zero (provided the code coverage of unit tests is 100%).

    Also, another factor is how many test cycles has gone thought to add new big features. In general for major features, the test cycles would be more if there are no unit tests.

    Regards,
    Raja Nagendra Kumar,
    C.T.O
    Competant & Reliable Offshore Code Verification Partners.
    www.tejasoft.com