Topic
  • 5 replies
  • Latest Post - ‏2012-04-14T22:02:19Z by M_Kevin_McHugh
Dr Laurence James
Dr Laurence James
3 Posts

Pinned topic Industry Productivity Metrics For Agile (Java) Projects

‏2012-04-11T12:52:01Z |
 Hi folks -
 
Could someone possibly point me at any studies which provide some visibility of the kinds of productivity rates that good agile teams can typically achieve or for that matter the spread of productivity that is achievable?  This could be based on story point velocity or some other measures e.g. lines of Java code per developer per 2 week iteration etc Thanks in advance.
Updated on 2012-04-14T22:02:19Z at 2012-04-14T22:02:19Z by M_Kevin_McHugh
  • M_Kevin_McHugh
    M_Kevin_McHugh
    22 Posts

    Re: Industry Productivity Metrics For Agile (Java) Projects

    ‏2012-04-12T18:53:51Z  
    Dr. James,
        My answer has several facets to it.
    1. I have a study with charts.  I just don't have permission to release it.  I will seek permission.  It may take a while.
    2. IBM published several articles regarding the Software Group's (SWG) Agile adoption.  I've included those links below.
    3. Story points vary from team to team based on how they decide to measure complexity.  In fact, not every team uses a numeric story point system.  My conclusion is that a straight point to point comparison is mathematically invalid.  The fact that you left open the measurement type indicates you have a feel for this.
     
       IBM Case Studies:
     
     
       Measurement Approaches:
           Please review the link below regarding cost, benefit, ROI for sprints.  Even with the outlook proposed, there is difficulty comparing between companies and teams.  However, ROI or at least business value should largely normalize across those boundaries.
           This is a fantastic Lean Six Sigma Blackbelt study opportunity.   My theory going in would be that I could statistically show the improvement in earned value vs. cost.
     
            Please take a look at this link for my thoughts regarding velocity measures.  I want to be on the same page in this regard.
     
     Regards,
    M. Kevin McHugh
     
    Updated on 2012-04-12T18:53:51Z at 2012-04-12T18:53:51Z by M_Kevin_McHugh
  • Dr Laurence James
    Dr Laurence James
    3 Posts

    Re: Industry Productivity Metrics For Agile (Java) Projects

    ‏2012-04-13T08:13:31Z  
    Dr. James,
        My answer has several facets to it.
    1. I have a study with charts.  I just don't have permission to release it.  I will seek permission.  It may take a while.
    2. IBM published several articles regarding the Software Group's (SWG) Agile adoption.  I've included those links below.
    3. Story points vary from team to team based on how they decide to measure complexity.  In fact, not every team uses a numeric story point system.  My conclusion is that a straight point to point comparison is mathematically invalid.  The fact that you left open the measurement type indicates you have a feel for this.
     
       IBM Case Studies:
     
     
       Measurement Approaches:
           Please review the link below regarding cost, benefit, ROI for sprints.  Even with the outlook proposed, there is difficulty comparing between companies and teams.  However, ROI or at least business value should largely normalize across those boundaries.
           This is a fantastic Lean Six Sigma Blackbelt study opportunity.   My theory going in would be that I could statistically show the improvement in earned value vs. cost.
     
            Please take a look at this link for my thoughts regarding velocity measures.  I want to be on the same page in this regard.
     
     Regards,
    M. Kevin McHugh
     
     Many thanks for your suggestions Kevin; I will investigate further.
     
    Scott recommended Capers Jones' book "Applied Software Measurement. Global Analysis Of Productivity and Quality"
     
    Would you also have any kind of "actuals" from the agile companies you've worked with to provide some kind of approximate mapping from say a single user story point to person days of effort? e.g. a client I am currently working with has 1 story point = 1.5 man days of effort.  The client is having problems with their 90 day development release cycles and seem to have to descope significantly every time. I suspect that their estimation techniques are not that good because of many reasons (e.g. lack of domain knowledge, greater complexity than originally anticipated, too many defects etc). If an industry average was say 1 story point = 2 hours effort then I could probably recommend to them that they need to drill a bit deeper into their higher level user stories and do some more detailed elaborations... Thanks again

     
  • M_Kevin_McHugh
    M_Kevin_McHugh
    22 Posts

    Re: Industry Productivity Metrics For Agile (Java) Projects

    ‏2012-04-13T14:51:43Z  
     Many thanks for your suggestions Kevin; I will investigate further.
     
    Scott recommended Capers Jones' book "Applied Software Measurement. Global Analysis Of Productivity and Quality"
     
    Would you also have any kind of "actuals" from the agile companies you've worked with to provide some kind of approximate mapping from say a single user story point to person days of effort? e.g. a client I am currently working with has 1 story point = 1.5 man days of effort.  The client is having problems with their 90 day development release cycles and seem to have to descope significantly every time. I suspect that their estimation techniques are not that good because of many reasons (e.g. lack of domain knowledge, greater complexity than originally anticipated, too many defects etc). If an industry average was say 1 story point = 2 hours effort then I could probably recommend to them that they need to drill a bit deeper into their higher level user stories and do some more detailed elaborations... Thanks again

     
    I have two sets of information in mind.  And, the details
     
    The first is the one I said I would seek release on.  If I get permission, it will be a more enterprise wide view.  It is a pair of charts that speak to feature / head count.
     
    The second seems a bit more associated to your follow on question.  I don't have charts.  But, I can provide more direct information since I was the ScrumMaster for that work.
    Let's go through your questions step by step.
     
    1 story point = 1.5 man days of effort
       - The team I ScrumMaster'ed for had a ratio that was more like 1 story point = 1 - 2 hours of work.  I would not say this ration is better or worse.  Just different.  
     
    Let's go into some of the drivers and possible solutions....
     
       - The Story only needs to be accomplish-able within a Sprint.  As long as the team can reach the Definition of Done within the sprint, I have no argument on the size.  
       - I recommend two levels of management (within the current scope of our conversation).  First is the Story.  Make sure to "point" it.  Sound like you are already doing that.  Second, break the story into tasks.  I try to keep my teams' tasks at sizes they can report completion within a day.  That is not a hard and fast rule.  But, I try to guide that way.   If you follow this then, one Plan Item (story) -> many Execution Item (tasks).  You will find the burn up for the sprint becomes clear.  And, you will not have to worry about the man days for the story as directly.
     
    The client is having problems managing their 90 day development release cycles and seem[s] to have to descope significantly every time.  This paragraph is intended to give you a view on velocity variance
       - This should be a straight velocity management.  Estimates of story points should start to provide an average velocity.  The theory is that the average complexity velocity (average story points / sprint) settles down quickly.  If you are not seeing it settle, then I recommend you build a control chart.   A run chart is a series of measures over time.  You velocity for each sprint would be the measure.  Put those on a chart and you get a line that zig zags.   The next step for a control chart is to find 3 std deviations up and 3 down from the mean.  You should be able to do this in Excel trivially.  If not, please let me know and I can provide assistance.   The reason for the control chart is to understand the variance in velocity.  If it is not settling down, then, to protect from the descoping, use the upper and lower control limits (the 3 std dev lines) to make a "more reasonable" estimate for the team's ability to hit their predicted velocity.
        - You could leave items on the backlog and pull them into the sprint.  That would be a scope add and might play better.  But, I still think you should try to improve estimation and squeeze those control limits together (through better estimation and execution).
        - My standard estimation to execution has two stable nodes.  I either complete my tasks in 80% of the estimate. Or, I complete them in 2x the estimate.  Overall, I complete all tasks in the original intended calendar time.  To improve that, I worked on my estimation and focused on bringing clarify to Analysis
        - You might try looking at improving clarify of the story.  I don't have your stories to look at.  But, you might need to improve Definition  of Done.  You might need to get your Test folks to help with Acceptance Criteria.  You might need to bring clarity to tasks (as I mentioned above).
     
       Domain complexity is an Agile Scaling factor.  I had not encountered a case where it impacted complexity velocity by driving variance.  It is a reasonable thing to consider though.  My first take is to get a grip on complexity velocity (defined as story points / sprint).  Next, you may want to encourage the team to conduct Architecture spikes sprints to explore the domain.  This should improve domain knowledge and reduce complexity velocity variance.
     
        A fifth ceremony for Scrum.  1) Daily Scrum; 2) Sprint Planning; 3) Sprint Demo; 4) Sprint Retrospective....  The book Elements of Scrum 1.0 recommends something called Story Time.  It is an hour a week to groom the product backlog.  This should start driving story clarity.  And reduce complexity velocity variance
     
        I would recommend that the Team know's where the issues are coming in from.  Look to the Retrospectives for the issue drivers.  In one case, I tried for a long time to handle requirements clarity.  Eventually, the PM stepped in and directed the requirements analyst to have story time w/ the developers and it smoothed right out.
     
    Industry Average points -> hours.  I don't have direct numbers on that.  I suggest it doesn't matter if you use Stories as plan items and tasks as execution items.  That will provide more clarity to the hour commitment.  Drilling into the stories is a good idea.  I think I said that just above.  My experience is 1 point = 1 - 2 hours.  I've seen teams with ratios like the one you quote.  The ratio did not make a difference to their execution.
     
    You may be seeing them trying to implement Epics which are too big to bite off in a sprint.  However, I don't think that is the case.
     
    I had another client who missed their story size dramatically in their first sprint.  We stopped the sprint, replanned, and they executed well after that.  The key for them to recognize the need to replan was emphasis on their need to demonstrate the story at the end of the sprint.  When I pushed on that, the developers asked for a resize so they could demo.  That got them the right sizing.  That sounds like your client's goal.
     
    Please let me know if I need to clarify anything.  The control chart section might not be intuitive for someone new to that technique.  I could provide an example spreadsheet if needed.
     
    -Kevin
        
  • Dr Laurence James
    Dr Laurence James
    3 Posts

    Re: Industry Productivity Metrics For Agile (Java) Projects

    ‏2012-04-14T14:44:58Z  
    I have two sets of information in mind.  And, the details
     
    The first is the one I said I would seek release on.  If I get permission, it will be a more enterprise wide view.  It is a pair of charts that speak to feature / head count.
     
    The second seems a bit more associated to your follow on question.  I don't have charts.  But, I can provide more direct information since I was the ScrumMaster for that work.
    Let's go through your questions step by step.
     
    1 story point = 1.5 man days of effort
       - The team I ScrumMaster'ed for had a ratio that was more like 1 story point = 1 - 2 hours of work.  I would not say this ration is better or worse.  Just different.  
     
    Let's go into some of the drivers and possible solutions....
     
       - The Story only needs to be accomplish-able within a Sprint.  As long as the team can reach the Definition of Done within the sprint, I have no argument on the size.  
       - I recommend two levels of management (within the current scope of our conversation).  First is the Story.  Make sure to "point" it.  Sound like you are already doing that.  Second, break the story into tasks.  I try to keep my teams' tasks at sizes they can report completion within a day.  That is not a hard and fast rule.  But, I try to guide that way.   If you follow this then, one Plan Item (story) -> many Execution Item (tasks).  You will find the burn up for the sprint becomes clear.  And, you will not have to worry about the man days for the story as directly.
     
    The client is having problems managing their 90 day development release cycles and seem[s] to have to descope significantly every time.  This paragraph is intended to give you a view on velocity variance
       - This should be a straight velocity management.  Estimates of story points should start to provide an average velocity.  The theory is that the average complexity velocity (average story points / sprint) settles down quickly.  If you are not seeing it settle, then I recommend you build a control chart.   A run chart is a series of measures over time.  You velocity for each sprint would be the measure.  Put those on a chart and you get a line that zig zags.   The next step for a control chart is to find 3 std deviations up and 3 down from the mean.  You should be able to do this in Excel trivially.  If not, please let me know and I can provide assistance.   The reason for the control chart is to understand the variance in velocity.  If it is not settling down, then, to protect from the descoping, use the upper and lower control limits (the 3 std dev lines) to make a "more reasonable" estimate for the team's ability to hit their predicted velocity.
        - You could leave items on the backlog and pull them into the sprint.  That would be a scope add and might play better.  But, I still think you should try to improve estimation and squeeze those control limits together (through better estimation and execution).
        - My standard estimation to execution has two stable nodes.  I either complete my tasks in 80% of the estimate. Or, I complete them in 2x the estimate.  Overall, I complete all tasks in the original intended calendar time.  To improve that, I worked on my estimation and focused on bringing clarify to Analysis
        - You might try looking at improving clarify of the story.  I don't have your stories to look at.  But, you might need to improve Definition  of Done.  You might need to get your Test folks to help with Acceptance Criteria.  You might need to bring clarity to tasks (as I mentioned above).
     
       Domain complexity is an Agile Scaling factor.  I had not encountered a case where it impacted complexity velocity by driving variance.  It is a reasonable thing to consider though.  My first take is to get a grip on complexity velocity (defined as story points / sprint).  Next, you may want to encourage the team to conduct Architecture spikes sprints to explore the domain.  This should improve domain knowledge and reduce complexity velocity variance.
     
        A fifth ceremony for Scrum.  1) Daily Scrum; 2) Sprint Planning; 3) Sprint Demo; 4) Sprint Retrospective....  The book Elements of Scrum 1.0 recommends something called Story Time.  It is an hour a week to groom the product backlog.  This should start driving story clarity.  And reduce complexity velocity variance
     
        I would recommend that the Team know's where the issues are coming in from.  Look to the Retrospectives for the issue drivers.  In one case, I tried for a long time to handle requirements clarity.  Eventually, the PM stepped in and directed the requirements analyst to have story time w/ the developers and it smoothed right out.
     
    Industry Average points -> hours.  I don't have direct numbers on that.  I suggest it doesn't matter if you use Stories as plan items and tasks as execution items.  That will provide more clarity to the hour commitment.  Drilling into the stories is a good idea.  I think I said that just above.  My experience is 1 point = 1 - 2 hours.  I've seen teams with ratios like the one you quote.  The ratio did not make a difference to their execution.
     
    You may be seeing them trying to implement Epics which are too big to bite off in a sprint.  However, I don't think that is the case.
     
    I had another client who missed their story size dramatically in their first sprint.  We stopped the sprint, replanned, and they executed well after that.  The key for them to recognize the need to replan was emphasis on their need to demonstrate the story at the end of the sprint.  When I pushed on that, the developers asked for a resize so they could demo.  That got them the right sizing.  That sounds like your client's goal.
     
    Please let me know if I need to clarify anything.  The control chart section might not be intuitive for someone new to that technique.  I could provide an example spreadsheet if needed.
     
    -Kevin
        
    Thanks again Kevin for a lot of useful content.
     
    To give you some more context, I recently (March) conducted a s/w development health assessment for this client. They have these 90 day release cycles for a programme of work involving 10 "agile" development teams producing components which get integrated into the 90 day release cycle. Each team is around 10-15 staff with a typical combination of roles, BAs, architects, testers, deveopers, PM etc They are relative new to agile and in fact not genuinely "agile" at all - they are very much more "Wagile" than anything else. They get the business requirements sorted out during the first 3 week or so, they then have 4 development iterations of 2 weeks each, followed by integration/regression testing sprints and a final "hardening" sprint (i.e. all hands to the pumps on defect fixing). Project teams do perform unit testing and integration testing on a daily basis and there is usually a weekly build of everything.. Project teams typically find around 5-10 defects each per day; they claim that most of the errors are down to incorrect programming.
     
    The business requirement user stories (they call them functional user stories) look fine; the "technical user stories", which are generated by the developers and are derived from the business reqs, did not look wonderful to me. The developers then create 4 or 5 tasks for each technical user story.At the end of a sprint there is some kind of product "demonstration"
     
    Complicating matters significantly is the fact that these 10 development teams working on Release 3.1 say, may also have to retrofit some key users stories into 2 or 3 older releases, R3.0 and R2.3.1
     
    I suspect that the volume of concurrent development work that the teams have to accommodate, is also having a negative impact on their development efficacy.
     
     Once I digest all the info you've sent me, then I;ll respond with some conclusions ...
     
    Thanks again
  • M_Kevin_McHugh
    M_Kevin_McHugh
    22 Posts

    Re: Industry Productivity Metrics For Agile (Java) Projects

    ‏2012-04-14T22:02:19Z  
    Thanks again Kevin for a lot of useful content.
     
    To give you some more context, I recently (March) conducted a s/w development health assessment for this client. They have these 90 day release cycles for a programme of work involving 10 "agile" development teams producing components which get integrated into the 90 day release cycle. Each team is around 10-15 staff with a typical combination of roles, BAs, architects, testers, deveopers, PM etc They are relative new to agile and in fact not genuinely "agile" at all - they are very much more "Wagile" than anything else. They get the business requirements sorted out during the first 3 week or so, they then have 4 development iterations of 2 weeks each, followed by integration/regression testing sprints and a final "hardening" sprint (i.e. all hands to the pumps on defect fixing). Project teams do perform unit testing and integration testing on a daily basis and there is usually a weekly build of everything.. Project teams typically find around 5-10 defects each per day; they claim that most of the errors are down to incorrect programming.
     
    The business requirement user stories (they call them functional user stories) look fine; the "technical user stories", which are generated by the developers and are derived from the business reqs, did not look wonderful to me. The developers then create 4 or 5 tasks for each technical user story.At the end of a sprint there is some kind of product "demonstration"
     
    Complicating matters significantly is the fact that these 10 development teams working on Release 3.1 say, may also have to retrofit some key users stories into 2 or 3 older releases, R3.0 and R2.3.1
     
    I suspect that the volume of concurrent development work that the teams have to accommodate, is also having a negative impact on their development efficacy.
     
     Once I digest all the info you've sent me, then I;ll respond with some conclusions ...
     
    Thanks again
     One more thought to go with the digestion.  :-)
     
    If the teams have defects coming in from "outside" the sprint.  I usually teach that the defects should have a story point estimate provided.  This will help identify the impact of defects from previous releases or even concurrent Independent Test.
     
    I usually (for myself) record defect records on unit test defects.  It keeps me disciplined.  However, I expect those defects as part of my development work.  Therefore, I would "story point" those.  I would just track estimate and actual hours.
     
    I shall await your conclusions.
     
    -Kevin