I just had an interesting discussion with a coworker regarding our introduction of unit tests.
The impetus for introducing the tests is to reduce our defect rate, and all the ROI expected from catching defects early.
No surprises there.
I should mention that our working definition of defect rate is "Bugs Logged per month." Perhaps that ought to be "Bugs logged per month per new LOC" or something else?
My question is, once we have introduced unit testing, what metrics can we use to show whether it's working or not? Intuitively, I thought "that's easy, if the defect rate drops, testing is working."
Our discussion centered on the following confounding factors: i.e. The original untested codebase is stabilizing, newer features aren't exercised in the field as rigorously, refactoring is making the original codebase more robust, etc. Any of which could be contributing to a reduction in defect rates, in the absence of units tests.
Thoughts? Our project is an already existing code-base that is in maintenance, including regular addition of new features.
I'm interested in hearing other experience in this.
Pinned topic What metrics can be used to show Unit Testing reduces Defect Rate
Answered question This question has been answered.
Unanswered question This question has not been answered yet.
Updated on 2007-06-13T15:29:33Z at 2007-06-13T15:29:33Z by nagkumar
Re: What metrics can be used to show Unit Testing reduces Defect Rate2007-06-01T01:18:03ZThis is the accepted answer. This is the accepted answer.You could start to measure code coverage-- that way you can begin to assess what's being tested and what's not. I'd also begin to correlate defects to actual code and assess coverage against that code.
Your point about refactoring and what not could certainly lead to fewer defects but without the assurance of actual tests to verify functionality you could be skating on thin-ice, don't you think? Defects can pop up that may have been caught earlier; plus, having tests in place sure does make refactoring a lot safer, not to mention easier too.
Re: What metrics can be used to show Unit Testing reduces Defect Rate2007-06-08T14:19:32ZThis is the accepted answer. This is the accepted answer.If you're measuring metrics, you have to have a baseline. Your baseline isn't "Bugs Logged per month", since unit testing has no impact on EXISTING code, only on TESTED code. You need to measure the number of bugs found in the code you've tested, and look to see if that number decreases each time that code is updated (and re-tested).
Over time, your "Bugs Logged per month" should go down, but only once all of your existing code has been tested at least once.
nagkumar 120000NBPN3 Posts
Re: What metrics can be used to show Unit Testing reduces Defect Rate2007-06-13T15:29:33ZThis is the accepted answer. This is the accepted answer.
- SystemAdmin 110000D4XK
Also, another factor is how many test cycles has gone thought to add new big features. In general for major features, the test cycles would be more if there are no unit tests.
Raja Nagendra Kumar,
Competant & Reliable Offshore Code Verification Partners.