I am new to the SCM arena, the last six months. (I'm a wayward developer, on
a mission to improve engineer processes and practices.)
I do know a bit about metrics though. (Developed quality metrics for a
notebook manufacture in a previous life.) The concept of metrics is that
they are a measure of performance. In the manufacturing world, defects per
millions is a standard. In fact it is so much a standard that manufactures
can compare the quality of their product with their competitors products!
And from that measurable goals can be set, i.e. We will improve are quality
from 10000 DPM to 500 DPM. It is amazing how quickly quality improves
($aving) when you have quantitative measuring tools.
The last six months I have been spent on learning ClearCase, but my new
years resolution is to get some metrics.
Stuff like,
delta time of SCM process for engineers - Time spent checking in,
checking out, merging, paper work, etc.
delta time of SCM process for build team - Time spent building,
validating, releasing, paper work, etc.
delta time of SCM process for product release - Time spent
packaging, release notes, etc.
% time SCM process fails
% time build fails
Amount of time it takes to reproduce a previously released version
Cost of Implementing SCM - How long to set up a project
Amount of time it takes to back out (subtractive merge) a product
feature.
Average time to complete a task - What is the average time to
complete a task (bug/enhancement).
On time average - Can you predict when tasks are completed.
Number of active tasks - Displayed as a histogram might give a hint
on motivation or close to done
Number of active bugs being tracked - Histogram.
bug rate - What is the rate of bugs being reported.
bug distribution - What area of the code are the bugs collecting in.
Average time till bug detection - From the beginning of testing till
bug is found
% of bugs that are 'no problem found'
% of bugs seen more than once
and more???...
-----Original Message-----
From: Robbins, Bert [mailto:Bert_Robbins@NAI.com
Sent: Friday, January 28, 2000 11:31 AM
To: 'Aiello, Bob (Exchange)'; 'Pease, Kevin'; 'cciug@rational.com'
Subject: RE: [cciug] Suggestions for common metrics?
SCM practices allow you to identify where the defects were introduced with
result of the identification of the defects being that you not repeat the
actions that caused those defects in the future. Defects need to be
categorized to determine what areas need improvement. If a large number of
the defects are due to a deficiency in the design area then that design area
needs to be improved to reduce the defects. If an individual developer is
responsible for a large number of defects than that developer needs to be
scrutinized to see what is causing that developer to produce a large number
of defects.
The CM/SCM department is no different than any other department and when a
defect is found in the CM/SCM process or procedures it should be evaluated
just like any other defect.
Valid metrics are determined by an organization's culture and philosophy.
One organization may believe that lines of code is a valid metric and
another may dismiss this as irrelevant. This doesn't mean one organization
is wrong and the other is right.
Everyone is measured by their performance. Performance is related to the
policies and procedures. An individual may be performing poorly due to
following bad policies and procedures or an individual may be performing
poorly due to not following good policies and procedures. The key to
improving an individual's poor performance is finding the cause of the poor
performance.
Finding the cause of a defect should be given equal weight in the
development process as fixing the defect to ensure that the defect isn't
repeated.
Thanks
Bert Robbins
-----Original Message-----
From: Aiello, Bob (Exchange) [mailto:raiello@bear.com
Sent: Friday, January 28, 2000 12:57 PM
To: 'Pease, Kevin'; 'cciug@rational.com'
Subject: RE: [cciug] Suggestions for common metrics?
The real point is that SCM practices prevent defects and help insure the
quality of the code. The important metrics to consider are # defects
resolved by your development effort and an affirmation that these defects
are NEVER the result of a configuration management mistake. I have worked in
a number of companies where they had to create several releases (a week!)
just because they couldn't get the versions of the header files correct. We
fixed that.
The important thing is also to consider the validity of the metrics you
collect. A valid metric can be traced to some other criteria.
You mentioned some metrics that I think could be counterproductive. It's
best not to measure a person's performance using metrics. Measure the
process instead.
Good luck!
> -----Original Message-----
> From: Pease, Kevin [SMTP:Kevin.Pease@FMR.COM
> Sent: Friday, January 28, 2000 12:49 PM
> To: 'cciug@rational.com'
> Subject: [cciug] Suggestions for common metrics?
>
>
>
> > Hi all,
> >
> > I've received a rather vague request from the managers of a
> > ClearCase environment I provide support for, to "provide metrics" for
> this
> > environment. I have very little experience with metrics, however, and
> my
> > customer hasn't been very forthcoming about exactly what their
> > requirements are, so this is making it a bit hard to show them metrics.
> > What they've asked for is for me to come up with some suggestions as to
> > what might be useful metrics to track. So, this all leads up to my
> > question:
> >
> > What "common" metrics do you folks track? I've gotten some
> > suggestions already, namely:
> >
> > Number of builds, broken down by # of successes, # of failures; #
> > of failures broken down by "cause of failure":
> > 1. Development error - e.g., code change broke the build,
> > developers didn't check in all required code, etc.
> > 2. Systems error - e.g., build server crashed during build,
> > network died, etc.
> > 3. Scm error - e.g., release engineer didn't start the
> > build on time; build scripts failed; scm system crashed or hung in
> some
> > way, etc.
> >
> > Various ClearCase-specific numbers, e.g., # of branches, # of
> > elements, # of views, # of VOBs, etc etc.
> >
> > Changes between builds -- # of files changed, # of source lines
> > changed / deleted / added, etc.
> >
> > Can anybody offer any other suggestions about what kinds of metrics
> > you would care to collect (and would find useful)? Specifically, this
> is
> > a ClearCase 3.2.1 environment running on Solaris servers, with multiple
> > repositories on containing source in C, C++, perl, and various shell
> > scripting languages.
> >
> > Also, can anybody offer suggestions on good reference materials for
> > software metrics? (Title at least, preferably with author?)
> >
> > Any suggestions you folks can offer would be greatly appreciated!
> >
> > Thanks,
> >
> > Kevin Pease
> >
> > ----------
> > Kevin B. Pease FISC - SCME
> > Fidelity Investments Systems Company
> > kevin.pease@fmr.com (617) 392-9022
> > SkyTel Page: 1-800-759-8888, Pin # 4713841
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>
***********************************************************************
Bear Stearns is not responsible for any recommendation, solicitation,
offer or agreement or any information about any transaction, customer
account or account activity contained in this communication.
***********************************************************************
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
This archive was generated by hypermail 2b29 : Sun May 06 2001 - 00:22:41 EDT