Topic
152 replies Latest Post - ‏2012-11-07T02:53:09Z by ned_from_california
richardknaster
richardknaster
8 Posts
ACCEPTED ANSWER

Pinned topic Virtual Roundtable - Agile Metrics [Event closed]

‏2012-10-24T13:46:20Z |
**Even though this event is over, you are welcome to browse this discussion**
 

Welcome to this round table discussion which will run until Friday 2nd November.
In this thread we are discussing Agile metrics.
1.  What are the most important metrics to help Agile Teams to improve?
2.  Are metrics against the agile value of simplicity --  "maximizing the amount of work not done"?
3   What kinds of metrics have you seen implemented that drive the wrong behavior?
4.  What is best way to determine what metrics should be tracked for Agile?
5.  What are the type of metrics that should be tracked for Agile?
  •     at the organizational level
  •     at the portfolio level
  •     at the team level
  •     at the individual level
 You will need to JOIN this group to reply to this topic.
 
You can return to the Welcome notice, join one of our breakout topics on enterprise level metrics metrics transparency and misuse, or business value achieved or discuss future topics.
Updated on 2013-01-07T22:34:50Z at 2013-01-07T22:34:50Z by sjpeich
  • richardknaster
    richardknaster
    8 Posts
    ACCEPTED ANSWER

    Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

    ‏2012-10-24T16:55:45Z  in response to richardknaster
     I thought I begin the roundtable with some thoughts about what makes good agile metrics to help frame our discussion:
      
    • Are easy to collect and don't become an impediment to the team's work
    • Measures business outcomes over activity (e.g. value delivered vs. number of tasks completed)
    • Looks at trends, not just a single point in time
    • Encourages whole team results over individual results.  If there are multiple teams for a project, use the rolled up numbers wherever possible to encourage collaboration over competition
    • Provides information for meaningful dialog and are not used to hammer the team
    • Are transparent and actionable
    • Encourages the right behavior; if you reward for finding defects, the team will document lots of meaningless defects, if you push the team to increase their velocity, the team will inflate Story point estimates and so on.  You reap what you sow!  Be careful what you ask for, you might just get it!
    Updated on 2012-10-24T16:55:45Z at 2012-10-24T16:55:45Z by Darrel Rader
    • Cherifa
      Cherifa
      30 Posts
      ACCEPTED ANSWER

      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

      ‏2012-10-24T22:00:14Z  in response to richardknaster
       Just adding to Rich's thoughts: What makes good agile metrics
      • Easy to understand: not too technical so everyone in the team or outside the team (managers) can  grasp
      • Not vague – Most can quickly analyze it and know what the measure represents
      • Useful – (different from meaningful) , should help address concerns/tough questions, and identify risks earlier
      • Encourage team collaboration and not defeating the purpose of agile adoption
       
       
       

       

      • JimDensmore
        JimDensmore
        24 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-25T16:53:15Z  in response to Cherifa
        Going back to being iterative in everything we do, which means being risk aware, I find that the first and biggest risk we have with most of my customers is they are measuring nothing.  Thus, they are operating open loop - they do not objectively know the degree to which their activities are having the desired effect on their business.  Usually, the right thing to do is pick exactly one thing to measure objectively, one metric, and get your customer to measure that.  It turns out to be incredibly difficult most of the time; they organizationally resist measuring at all if they're not already doing so.
         
        Once they're measuring something, the highest risk usually is that they're not measuring what they need to be measuring.  Now that they're measuring something, we can pick the next thing they need to measure, or maybe the next two things, and get them started measuring those things.
         
        Once they're measuring what they need to be, the next highest risk usually is whether they are use the information they get successfully (or, in the limit, do not abuse the information they're getting).  This is about feedback on the measurement program.  Aligning with the business helps, using the business/operational/practice framework I mention elsewhere in this thread.  Being thoughtful helps too, simply looking for what behaviors the current measures are promoting and figuring out how to fix it when improved behaviors are still needed.  
    • Darrel Rader
      Darrel Rader
      3 Posts
      ACCEPTED ANSWER

      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

      ‏2012-10-24T22:35:01Z  in response to richardknaster
       
      Daniel Haischt shared an interesting InfoQ article on the subject of agile measures. Great cartoons  http://www.infoq.com/articles/not-destroy-team-metrics
       
       
      Updated on 2012-10-24T22:35:01Z at 2012-10-24T22:35:01Z by Darrel Rader
      • DanielSHaischt
        DanielSHaischt
        5 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-25T08:53:23Z  in response to Darrel Rader
         I am asking myself how agile metrics should be established? For instance if you want to achieve team buy-in, and that's really what the InfoQ article covers, you want to consider letting the team drive and establish metrics they:
         
        • Want to use internally
        • Want to open to their stakeholders
        • Want to use to get feedback on team performance or individual performance (isn't agile about team first anyway?) 
        So if that assumption is true it would really require self-organized teams. Are we there yet? For instance are teams always striving for taking responsibility and thus ownership? So if metrics will be established/enforced by stakeholders or "the organization", does this sometimes happen cause stakeholders or "the organization" treat agile teams as immature in terms of taking responsibility/ownership and thus they try to look after teams instead of letting them make mistakes for the sake of a productive learning process towards true agility?
         
        So I guess my point is that it's essential at which level such metrics are getting established in terms of achieving team buy-in and to get back to the InfoQ article, in terms of avoiding teams cheating or optimizing their own metrics they are required to report to stakeholders. Finally I consider the process of finding the right metrics a continuous process because after a couple of iterations/hops you may figure out that some metrics are really having value where others don't. So there should be freedom to have a change process that allows to either add or drop metrics as required.
         
        Cheers
        Daniel
        Updated on 2012-10-25T08:53:23Z at 2012-10-25T08:53:23Z by DanielSHaischt
      • AlexXIE
        AlexXIE
        8 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-26T06:32:39Z  in response to Darrel Rader
         Darrel, this is one best paper I had ever read, thanks for sharing. I read the two recommend metrics focus on the ultimate value of agility -1. Time to market;   2. Volume of Value of your delivery.  These probably the CEO level metrics.The metrics regarding the interests of Directors, Managers, Team Lead, QA manager, Development Manager will be also valuable and had great potential needs to develop to align "Agile Metrics". Because I think only the effort and result would only be bought in after we think in their Govern position.
      • AlexXIE
        AlexXIE
        8 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-26T06:44:53Z  in response to Darrel Rader
        One another point - for those invalid Metrics areas that the paper had indicated, I guess it is just a waste if we stop find solution to make the "imperfect metrics" better.
        Like I still think some person will like to "Line of code" . Just we need guarantee the developer won't just populate the code from one line to mutiple lines.... here is a sample... as me will write the 3 lines of code 
        For (i=1,i<10,i++){
        print("The line number increase 1");
        }
        While somebody else will do only 1 line of code like below or do 4 lines of code to realize the same function point.
        For (i=1,i<10,i++){print("The line number increase 1");}
         Why don't we give a standard coding standard, (most IDE tool can do this) to format the code to one style, then no matter their intention, the number of line of code will not be impacted by "coding habit". And  similar way to "eliminate the dead code"  ....etc.
         
        And for "Defect Rate", I think QA manager need that, and some people manager also need that to reflect the quality  and productivity of different Agile Testers, while in case the misleading metric of "Defect Rate" will bring meaningless large number of defects, why don't we use the "Anti-Metrics" or "Fix-Metrics" just to fix the hole and prevent it mislead the team? I think everyone here had good ideas to do that, so save a hundred words typing here.... :-)
        Updated on 2012-10-26T06:44:53Z at 2012-10-26T06:44:53Z by AlexXIE
      • Reed_Fegg
        Reed_Fegg
        14 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-29T18:36:22Z  in response to Darrel Rader

        Darrel, thanks for sharing this article, http://www.infoq.com/articles/not-destroy-team-metrics

         
        Took a look and thought it was great article as well.   Wanted to pose a question to the Virtual Roundtable.
         
         

        What do you think of the authors assertion that we "We measure actual output rather than contributing factors."

        • Value Delivered:  Measures business impact and not performance by using Product Owner feedback.
        • On Time Delivery: Focus on measuring how accurately they can forecast their sprints over a long interval of time (like 1 year)
    • WalkerRoyce
      WalkerRoyce
      18 Posts
      ACCEPTED ANSWER

      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

      ‏2012-10-25T13:50:39Z  in response to richardknaster
       Here is my description of necessary and sufficient attributes of GOOD metrics:
      1. They are simple, objective, easy to collect, easy to interpret, and hard to misinterpret.
      2. Collection can be automated and non-intrusive.
      3. They are derived from the evolving product baselines rather than some subjective assessment.
      4. They are useful by both management AND engineers in communicating progress and quality in a consistent format.
      5. Their fidelity improves across the life cycle. 
      • ScottAmbler
        ScottAmbler
        24 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-25T14:02:59Z  in response to WalkerRoyce
         Gotta love pasting from MSWord into non-MS browsers.  ;-)
      • DougStewart
        DougStewart
        10 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-25T17:17:19Z  in response to WalkerRoyce
         Metrics should also be evolutionary.  Some of the metrics we establish for a project piloting an iterative approach may be more like traditional measures and less to our liking.  CxO's still have to make decisions across a wide range or projects during organizational transformation. The best way to lose executive support, other than failing to deliver, is to stop answering questions 'because we are Agile.' In the short term it may be necessary to map Agile metrics into more traditional ones and evolve organizational metric practice as the organization transforms.
        • M_Kevin_McHugh
          M_Kevin_McHugh
          22 Posts
          ACCEPTED ANSWER

          Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

          ‏2012-10-25T17:56:16Z  in response to DougStewart
          Agreed on metrics being evolutionary.
           
          Evolutionary metrics is implied in the "Goal" part of GQM.  Once you've reached a goal, the nature changes.  You're not trying to reach it any more.  You might be trying to maintain it.  But, new goals come up.
           
           
        • Cherifa
          Cherifa
          30 Posts
          ACCEPTED ANSWER

          Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

          ‏2012-10-25T20:28:13Z  in response to DougStewart
           Doug... I do not understand what you really mean by "evolutionary" when it comes to the metrics? It all depends on what you are measuring? Your agile adoption? .In that case you may want to use metrics that help you understand how well you are doing in terms of the agile process adoption, etc...Those set of metrics are going to be different from the ones you use for measuring your iterative development. Are you measuring if you are meeting your business objectives? Regardless of the process, you may end up with the same set of metrics that you want to monitor ( employee satisfaction, lowering the cost of maintenance...)...
          What I would add is that
          • metrics are specific to a duration ( in time t1 having X  measure makes sense but in time t2 the same X does not mean anything)
          • metrics are specific to a context
          Please let's get going with more insight/your views on this passionate topic...
          • DougStewart
            DougStewart
            10 Posts
            ACCEPTED ANSWER

            Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

            ‏2012-10-25T22:15:54Z  in response to Cherifa
            Cherifa .... Organizations make decisions about project health based on metrics that may be suitable for waterfall deliveries.  I have customers with early adopter agile projects being asked to report on their on-time performance of writing requirements or test cases.  Neither of these are interesting from an Agile perspective but it is not possible to ignore the requirement and at the same time too early to change it.  
             
            Originally these metrics were established to measure some aspect of project health.  As happens in many cases, the metric becomes the objective and the real objective is forgotten by most.  In the example of 'requirements done' this would be a measure of completeness of the business solution, namely scope and make versus buy.   In waterfall we prove we have a viable business solution, including scope and make v buy, by delivering a requirements spec. Using Agile approaches, we still need to measure that we have a viable business solution.  We just do it with a metric measuring how close we are to closing the window on requirements.
             
            When faced with the requirement for metrics that don't make sense for Agile I suggest pulling back to examine why the metric is collected for traditional projects and to find a way to measure that intent in an Agile context.  I also caution that Culture Shock may prevent us from jumping to the best metric for the early adopter projects.  We may have to pick interim metrics as we continue the transformation.  Over time we can evolve our metrics in a way that still allows executives to compare and make decisions across an entire portfolio that includes Agile and Waterfall projects.
             
            Of course we want to drive to good metrics over time.  Depending on the size and complexity of the organization under transformation it may take some time to get there. 
            • Cherifa
              Cherifa
              30 Posts
              ACCEPTED ANSWER

              Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

              ‏2012-10-26T16:15:38Z  in response to DougStewart
              Waterfall?? tell me about it. I understand where you are coming from and those were the days where people’s productivity was measured in terms of number of the reports and documents they created to show evidence of work being completed or done. Deciding whether a solution was "Viable " was based on delivering the  requirements specs ( among other things).
              Doug, you raised good discussion points that I would like to submit to our audience as most of organizations are in that situation of moving from waterfall to agile.
              • What metrics did we collect for traditional projects?
              • Why were they collected and  how  compared with the agile metrics?
              • What could be the interim metrics that will prevent us from this cultural shock of moving from Waterfall to  Agile?

              Walker/Scott? and all any thoughts?

              • DougStewart
                DougStewart
                10 Posts
                ACCEPTED ANSWER

                Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                ‏2012-10-29T16:17:52Z  in response to Cherifa
                 Another thing to consider is that continuous improvement suggests that the Agile 'transformation' never ends.  As we reach one level of capability, with the metrics to measure it, we immediately set our goals higher and the cycle repeats.  
      • DanPollitt
        DanPollitt
        4 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November) - Metric design

        ‏2012-10-26T13:22:02Z  in response to WalkerRoyce
         Hi All,
         
         A few years ago I attended an Agile conference in London that ran a session on metric design - specifically trying to pursue a metric that gave you a better chance of getting what you thought you were measuring. The format worked so well I took it away and applied it in several situations with positive results.
         
        The process recognised that a team being subjected to a metric (especially if connected to any sort of incentive or recognition) would immediately try to game the metric and as such this quite often undermined the original intent of introducing the metric, it actually relied upon that gaming behaviour and employed an iterative refinement process.
         
        The general idea was that a workshop of people would be split into small groups and each team given the initial proposal of a metric along with the accompanying description of what we wanted to measure (the objective of the metric). The teams would then put on their "devious hats" and work to describe how the metric numbers could be achieved whilst avoiding achieving the desired behaviour (how they would game it). The team would then trade the metric with another team and that new team would modify the metric to "close the loopholes". This process would be repeated several times, iteratively refining the metric - hopefully ending up with something that would give you a better chance of encouraging desired behaviours.
        • ScottAmbler
          ScottAmbler
          24 Posts
          ACCEPTED ANSWER

          Re: Virtual Roundtable - Agile Metrics (25 October to 02 November) - Metric design

          ‏2012-10-26T13:33:51Z  in response to DanPollitt
          Dan, this sounds like a take on Toyota's 5-Whys.  Maybe it should be called Agile's 5-Underminings.  ;-)
           
          You might be interested to know that a few years ago a couple of IBMers (at the time) wrote a paper that was published in the Agile 2008 conference proceedings about some of IBM's experiences with metrics.  One author was Bill Krebs, also known as Agile Bill, and the other Ted Rivera if I'm not mistaken.  Their findings, based on an internal study within IBM, was exactly as you describe.  When a team uses metrics for their own improvement they're motivated to keep the numbers "honest".  When the numbers are reported to management AND there's a perception that they're being used to judge or reward the team then there is a tendency to game the numbers.  Worse yet many teams were no longer making the actual improvements that they needed, yikes! 
          • DanPollitt
            DanPollitt
            4 Posts
            ACCEPTED ANSWER

            Re: Virtual Roundtable - Agile Metrics (25 October to 02 November) - Metric design

            ‏2012-10-26T14:03:57Z  in response to ScottAmbler
             Found a reference, it was this:  http://www.codemanship.co.uk/parlezuml/metrics/doyougetwhatyoumeasure.htm at XPDay05
            Updated on 2012-10-26T14:03:57Z at 2012-10-26T14:03:57Z by DanPollitt
          • WalkerRoyce
            WalkerRoyce
            18 Posts
            ACCEPTED ANSWER

            Re: Virtual Roundtable - Agile Metrics (25 October to 02 November) - Metric design

            ‏2012-10-26T14:38:04Z  in response to ScottAmbler
            This is a great point Scott/Dan. And that brings me back to one central theme of measurement: improving trust.
            Let me postulate a few things: 
            1. Open and honest measurement will be attractive in a high-trust environment.
            2. Open and honest measurement will be repelled in a low-trust environment.
            3. Incentives on achieving shared objectives can improve performance in a high-trust environment.
            4. Incentives in a low-trust environment will be gamed.

            Measurement is a tool that that can help, or hinder, an organization, depending on its culture. If you are trying to deploy measurement within a low-trust organization, you need to be very forthcoming about admitting your current state (low-trust) and openly stating your intent to shifting the culture to a higher-trust environment by deploying open and honest measurement. These leadership dimensions are the secret to realizing the big benefits of measured improvement.


            Updated on 2012-10-26T14:38:04Z at 2012-10-26T14:38:04Z by WalkerRoyce
            • SmartDJ
              SmartDJ
              4 Posts
              ACCEPTED ANSWER

              Re: Virtual Roundtable - Agile Metrics (25 October to 02 November) - Metric design

              ‏2012-10-29T19:17:42Z  in response to WalkerRoyce
               Walker, about the team climate is a very interesting factor for measurement, how can we know the team trust level , that is , how we know a team is high trust environment or low trust environment! thanks for comments!
              • WalkerRoyce
                WalkerRoyce
                18 Posts
                ACCEPTED ANSWER

                Re: Virtual Roundtable - Agile Metrics (25 October to 02 November) - Metric design

                ‏2012-10-30T14:46:10Z  in response to SmartDJ
                 DJ,
                How do you know you are in a high trust environment? 
                Here are a few signs:
                1. People speak the truth. Good news or bad news, there is no fear of the truth.
                2. People feel free of communicating to their peers, to their supervisor, and to their supervisor's peers. There is not a hierarchy or protocol of communication paths.
                3. Meetings happen in an open bay or with open doors.
                4. When managers exhibit trust in their peers.
                5. Teams help define their goals and targets.
                6. A track record of accountable performance.

                How do you know you are in a low trust environment? Here are a few signs of that:

                1. Discussions are preceded by "Please don't discuss this with others..."
                2. One on one closed door discussions.
                3. Managers noticeably don't trust their peers.
                4. Targets and goals are set outside the team, and without their involvement.
                5. A culture of blame, excuses and Pearl Harbor files.

                Trust may be difficult to measure objectively...but like natural beauty, you know it when you experience it.

          • Cherifa
            Cherifa
            30 Posts
            ACCEPTED ANSWER

            Re: Virtual Roundtable - Agile Metrics (25 October to 02 November) - Metric design

            ‏2012-10-26T14:44:53Z  in response to ScottAmbler

            My next take:

            It is important to have the team to decide (in collaboration and with participation of the stakeholders) what metrics will generate real conversation (during reviews) and needs to be captured! Not all are needed!!!

            Few principles to start with

            • Keep it simple!!!!
            • Set the right metrics, metrics that are useful/meaningful to the team, project executives and managers
            • Work with management and ask/challenge: What we will do with the data everyone is requesting and then how perceived
            • Change People Attitude towards metrics: provide awareness, make team understand the importance.
            4

            • WalkerRoyce
              WalkerRoyce
              18 Posts
              ACCEPTED ANSWER

              Re: Virtual Roundtable - Agile Metrics (25 October to 02 November) - Metric design

              ‏2012-10-26T14:59:33Z  in response to Cherifa
               This is good. And we probably should be a little clearer about what is "management." It is folly to think that all levels of management and stakeholders need to be aware of team measures and outcomes. When I speak of management, I tend to mean the direct  leadership team (for example, the PM, architect, Dev and test leads) who typically represent the face to less intimately involved stakeholders like users, executives, and buyers.
               
              It is valuable if all measures and metrics and forecasts can be used to communicate among external stakeholders. And, the more technically savvy they are, the easier it is to make better steering decisions. But we will always create simpler, more summary views of trends and forecasts for "outsiders" who don't have all the context. We always want accurate depictions that are not burdened with all the precision details that provide technical confidence in our judgments.
              • PaulBahrs
                PaulBahrs
                1 Post
                ACCEPTED ANSWER

                Re: Virtual Roundtable - Agile Metrics (25 October to 02 November) - Metric design

                ‏2012-10-26T16:35:12Z  in response to WalkerRoyce
                I had a recent discussion with a client on this topic. Specifically, how to provide a dashboard to meet the perspectives of different stakeholders - for the same measures. For example, how to provide a consistent representation of defect measures to the development team, program/department management and CxO team?  For discussion purposes, the team may be interested (pointing to Rich's focus above) on team improvement. Program / department management, might focus on resource allocation and root cause improvements. The CxO may be interested in how trends change with enterprise process/technology changes.
                 
                • MikeORourke
                  MikeORourke
                  10 Posts
                  ACCEPTED ANSWER

                  Re: Virtual Roundtable - Agile Metrics (25 October to 02 November) - Metric design

                  ‏2012-11-01T18:46:41Z  in response to PaulBahrs
                   The key aspect of these dashboards in an agile context is the following:
                  • They must come from the same data
                  • You can't "bother" the practitioners (meaning that the tools needs to automatically generate these metrics)
                  • You should stay away from comparing / contrasting teams (given diffrences is code complexity and languages) 
                  • ned_from_california
                    1 Post
                    ACCEPTED ANSWER

                    Re: Virtual Roundtable - Agile Metrics (25 October to 02 November) - Metric design

                    ‏2012-11-07T02:53:09Z  in response to MikeORourke
                     HI Mike.....
                    I really do have to agree on keeping the data the same.   I have seen mature co-located teams right up a very samll number of defects, because the development team is working closely with the test teams...   and issues are addressed without the need of a formal defect (through bullpen sessions, etc).  Less mature teams or teams that are not co-located often require more formal communication between test and development...  and because of that more defects are written...
                     
                    This information shown to upper management can be misread.....   the old idea of quality by defect count again raises its ugly head....
        • JimDensmore
          JimDensmore
          24 Posts
          ACCEPTED ANSWER

          Re: Virtual Roundtable - Agile Metrics (25 October to 02 November) - Metric design

          ‏2012-10-26T15:28:06Z  in response to DanPollitt
          Heh heh heh.  Please post picture of a Devious Hat.  :-)
      • KristofKloeckner
        KristofKloeckner
        2 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-11-03T20:23:14Z  in response to WalkerRoyce
         I think you nailed it. I would add one more characteristic, though:
         
        Metrics need to be tied to business value, i.e. not only be easily understood, but also be considered relevant by the stakeholders in an agile project.
        Here we have an interesting challenge. Some characteristics of a 'good' project are relatively easy to measure (like problems reported), but I find the industry struggles to measure output, or even productivity. 
        Over the lifetime of a product, some measures used are revenue, speed of adoption (for a second or later release), customer satisfaction..
        How predictive is stakeholder feedback or Beta user feedback? Would be interested in perspectives from the participants.
    • AlexXIE
      AlexXIE
      8 Posts
      ACCEPTED ANSWER

      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

      ‏2012-10-26T07:05:31Z  in response to richardknaster
      Hey all,
       
      Do you agree different type of product will had different Agility Metrics, as we may had different understanding about "value"?
       
      1. Middleware  - more integration quality
      2. Web application - more tolerance if you had defects slip out careless
      3. zOS application - more stability quality, and less timing needs..
      .....
       
      Updated on 2012-10-26T07:05:31Z at 2012-10-26T07:05:31Z by AlexXIE
      • ScottAmbler
        ScottAmbler
        24 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-26T13:37:16Z  in response to AlexXIE
        Different teams will be in different situations.  As a result they will work in their own way, be organized in their own way, and be dealing with different challenges.  As a result I think that it's reasonable to expect that they'll want to capture a collection of metrics which reflect the context of the situation they find themselves in.  In short -- no one metrics strategy fits all.
        • AlexXIE
          AlexXIE
          8 Posts
          ACCEPTED ANSWER

          Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

          ‏2012-10-27T06:04:24Z  in response to ScottAmbler
           Yep, good to hear you here, Scott, and I agree with you, no on metrics strategy fits all. And plus one more comment the "details are the value"
  • Reed_Fegg
    Reed_Fegg
    14 Posts
    ACCEPTED ANSWER

    Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

    ‏2012-10-25T00:25:04Z  in response to richardknaster
    One concern that I often run into when mentoring teams is the focus on "Velocity" as the primary measure of team success. Can you or others on the panel comment on the use of "Velocity" and at what level it should used and how
    • at the organizational level 
    • at the team level
    • at the individual level
      
    • GustavoGrillo
      GustavoGrillo
      8 Posts
      ACCEPTED ANSWER

      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

      ‏2012-10-25T18:00:53Z  in response to Reed_Fegg
       I don't like using Velocity reports outside of the team. It is often misinterpreted as "Productivity" and this is the first step in a journey with a very ugly ending...
      Velocity must be used for the team to know how much they can accomplish. People outside the team should read ROI, Features or something they can easily relate to.
      • Cherifa
        Cherifa
        30 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-25T20:11:11Z  in response to GustavoGrillo
         Gustavo, I think everyone agrees this should be the case...Velocity is meaningless outside the team.
        Thanks for sharing
    • richardknaster
      richardknaster
      8 Posts
      ACCEPTED ANSWER

      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

      ‏2012-10-29T15:24:29Z  in response to Reed_Fegg
      Reedy:
       
      Great question about Velocity.  I agree with Gustavo that Velocity is a team measure and should not be used at the individual or organizational level.  When Velocity is used at the organizational or individual level,  Velocity goes from being a useful calibration tool (what's our capacity for the next iteration?) to a productivity measurement tool. This means that the team's focus gets away from the important things: the quality of the customer experience (quantity of features vs. quality of customer experience) and improving the team's delivery engine.  Velocity should not be used for measuring predictability outside the team.  Instead, measure throughput, how many teams releases can our team do a year, or measure how much value did we deliver to our customers, customer satisfaction, etc
       
      Many managers are incorrectly using the Velocity metric to measure productivity and that is having the opposite effect and harming an Organization's agility.   I hear dysfunctional comments like this:  Your velocity fell from 32 to 24 this time, what’s wrong? Let’s get it back up, or even higher!  In response teams begin to add low value features, inflate their story point estimates and are afraid to remove stories that they've discovered aren't needed for fear that they will have to explain their drop in velocity, when in fact, they've eliminated waste and perhaps focused more time on the other stories delivering high value, quality features.  The Standish Group estimates that 80% of all features delivered are never used and using velocity for any other purpose in which it was intended will drive the wrong behavior.
  • M_Kevin_McHugh
    M_Kevin_McHugh
    22 Posts
    ACCEPTED ANSWER

    Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

    ‏2012-10-25T04:27:01Z  in response to richardknaster
    weeeelll, I'm not a panel member, but,....
     
    I see the use of velocity having a number of facets.
    In an early stage of Agile adoption, I tend to focus on "getting" to velocity.  I also tend to focus on the improvement of the velocity chart (i.e. how well does the actual velocity line "lie down" on the ideal line.)  
     
    I frequently find that organizations and teams in the early stage of the Agile adoption process worry about getting it perfect.  By showing them a series of velocity charts, the individuals, the organization can see that the expected behavior is improvement over time.
     
    A second goal, is to prompt better Story writing and practices.  What I find is that if stories are faulty, that the story point are either wrong or too difficult to assign points to.  And, so, I tend to gently but firmly nudge the team towards the behaviors needed to get velocity.
     
    A third goal is to help transform the organization by moving the team (and particularly their managers) away from command and control and into a constrained box with a certain amount of complexity (sprint and story points) inside it.  Many team that I coach speak to me about how their managers will simple tell them to work overtime or add work to sprints.  I have advised the team and Scrum Master to simple add stories or tasks to track the fact of the change.  This, I advised will help in future conversations with management about workload.  It will also allow any Agile support tool that handles workload management to "show up" the overload.
     
    Now, on the flip side, I've worked with teams that do not use story points.  They essentially constrain their story complexity ahead of time.  In essence, all stories are between 1 and 8 points.  And, they don't point them.  So, it's just a matter of closing out stories.  How many are open, how many are closed.  This works so long as certain assumptions of equivalent complexity or equivalent complexity mixes are maintained.  I prefer points.  But, in the art of work not done (i.e. a facet of Lean) I can (if you let me use the term) "allow" for the validity of the approach.
     
    On the other, other hand (yes, 2*other)....without velocity, I've seen team descend into simple task management.  In one, truly awful case, I watched two hard core Theory X managers override velocity indicators with threats to get ad doc tasks done when ever they wanted.  Task management is tough enough.  Theory X Agile is a trip to the dark side.
     
    Anthony Crain has a wonderful set of metrics.  I hope that he chimes into this round table.
     
    I've also waxed prosaic about Velocity in a series on the Agile Transform Zone.  The link to the first in a series of article is:
     
    As a parting thoughts
    1. Velocity is a tool.  Collect and use it IF it helps you reach a goal.  Some of the possible goals are listed or implied above.
    2. The smaller the team, the shorter the project, the more agile-ly mature the organization the less the metric might mean.
    3. I also see Velocity as a door to Whole Team....  :-)
     
     
    • Reed_Fegg
      Reed_Fegg
      14 Posts
      ACCEPTED ANSWER

      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

      ‏2012-10-25T07:11:27Z  in response to M_Kevin_McHugh
      For the sake of an arguement, let's say the management team overseeing the agile transformation wants to use "velocity" to compare the performance of two teams. How would you handle that situation.
      • RafaelSene
        RafaelSene
        4 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-25T11:39:24Z  in response to Reed_Fegg
        To compare two teams by velocity we need to have a common concept about what we are using to measure the velocity. I guess that we need set the same metric for both teams before compare them. 
        • DanielSHaischt
          DanielSHaischt
          5 Posts
          ACCEPTED ANSWER

          Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

          ‏2012-10-25T12:05:05Z  in response to RafaelSene
          QUESTION: Why do you want to "compare" teams at all? Will you use the result of a comparison just for information or will you use it to define action items in support of driving agile adoption?
           
          Lets assume both teams are able to deliver value within a timebox and their product owners aknowledge that both teams are able to deliver value. In such a case I would still asume that how product owners define value is highly subjective and relates to the individual context/environment in which a team is currently working. That said I would say you should rather focuse on value delivered than on velocity.
           
          E.g. what kind of velocity are we talking about anyway? Is it velocity that originates from an environment where a team realy managed to agree on done done done criteria and their stakeholders aknowledged that the criteria are valid? Or is it velocity from an environment where done done done criteria arent well defined?
          • RafaelSene
            RafaelSene
            4 Posts
            ACCEPTED ANSWER

            Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

            ‏2012-10-25T12:27:07Z  in response to DanielSHaischt
             I forgot to mention on my statement that to compare velocity, booth teams have to work within the same context or the same product. But I do believe that working software (delivering all priorities and helping client to increase the ROI )is the best approach to measure how quick is a team.
            • M_Kevin_McHugh
              M_Kevin_McHugh
              22 Posts
              ACCEPTED ANSWER

              Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

              ‏2012-10-25T13:07:40Z  in response to RafaelSene
               Correct 
               
              To extend on that any comparisons made should not weigh upon or alter the team's behavior 
               
              Thus if management wants to compare, put the effort to compare in the management tea and the complexity of the work to bring the values into a comparable state 
               
               
               
              :-) 
            • meerec
              meerec
              6 Posts
              ACCEPTED ANSWER

              Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

              ‏2012-10-25T13:17:31Z  in response to RafaelSene
              To me it makes sense to make comparisons of velocity of one and the same team between their own sprints.
              Comparing velocities of two different teams -- has no meaning to me. 
          • MikeORourke
            MikeORourke
            10 Posts
            ACCEPTED ANSWER

            Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

            ‏2012-11-01T18:54:38Z  in response to DanielSHaischt
             In the end we have to measure teams versus other teams. But to do so based on velocity doesn't make sense. In IBM we were forced for a long period of time to put teams in low cost countries because it was deemed we would save money. In fact we didn't, but it wasn't becuase the people were any less smart! Much of this replied on two issues: most of the low cost country savings were due to the fact that 50% of the engineers are fresh out of university, and that we take code / products from people who built it and have lived with it for years and give it to someone whose never seen it. Of COURSE you will lose productivity here. I find the best measurements within a team where different components or features are developed in different locations. is to let them self-police. If theyhave the data, the story points, the velocity measurements, they do a great job managing the productivity themselves.
        • ScottAmbler
          ScottAmbler
          24 Posts
          ACCEPTED ANSWER

          Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

          ‏2012-10-25T14:07:26Z  in response to RafaelSene
           When you set the same point system across teams you run several risks:
          1. There's an overhead to doing this.  This then begs the question whether the additional cost is justified by the value of the decisions that it enables, in this case the comparison of teams.  Gut feel tells me that this will happen rarely.
          2. You still won't be consistent.  Even if you can set a common baseline between teams over time they will diverge, in different ways, from that baseline.  The implication of this is that you'll be comparing apples with oranges even though you think you're comparing banannas with banannas.  
      • M_Kevin_McHugh
        M_Kevin_McHugh
        22 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-25T12:31:58Z  in response to Reed_Fegg
         First I would explain that Velocity is a measure of Complexity accomplished during a time period (see my article on Complexity vs Value Velocity for more details.)
         
        Then I would explain that what is difficult (ie complex for on team might be easy for another team. This naturally leads to Velocity differences.  Therefore comparing velocity has less meaning than might be originally supposed.  
         
        Usually I then get the argument that hours of work could imply complexity and therefore story points. This is a practice I try to dissuade teams from during adoption because it has detrimental impact on the use of thin slicing (the human ability to very rapidly evaluate thing in this case stories relative to each other. It leads instead back to detailed analysis and standard program management behaviors 
         
         
        All of that having been said I would propose that management compare Velocity between teams using a technique that avoid absolute values 
         
        Over time velocity for multiple sprints builds up. And as we know we get average velocity. But we also get standard deviation too. Charting average velocity along with 3 std dev up an 3 std dev down gives you the consistent / bound team behavior for velocity
         
        It is then possible to see which teams are more consistent meaning predictable. And predictability is VERY important. It is also possible to see if velocity is rising 
         
        Now be careful not to push rising velocity. That can cause other problems. Again I hope Anthony joins. He speaks wry well on metrics and counter metrics to keep unintended consequences out of a metrics system. 
         
        My short answer is essentially to normalize the charts and overlay them 
         
        We do have clients who setup a chart for X hours = Y story points. That will do it too. But brings in some adoption issues 
         
        Thanks 
        Kevin 
      • meerec
        meerec
        6 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-25T12:40:24Z  in response to Reed_Fegg
        Hmmm. I don't get the question: or is it a "tricky one"?  LOL
        Velocity of the team depends on how the team estimated the stories at the first place. 
        So it only makes sense to talk about this team's velocity. Velocity of two different teams? Velocity is not a universal metric. 
        Updated on 2012-10-25T12:40:24Z at 2012-10-25T12:40:24Z by meerec
      • ScottAmbler
        ScottAmbler
        24 Posts
        ACCEPTED ANSWER

        A Disciplined Approach to Velocity was Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-25T13:28:57Z  in response to Reed_Fegg
        Velocity is meant to be a measure used by a team to manage itself.  It really isn't meant to be an external facing metric that management can somehow use to govern it.
         
        In the Disciplined Agile Delivery (DAD) book we discussed how velocity can be used as the input into other measures that management can potentially use:
        1. AccelerationAcceleration is the change in velocity, which is an indication of productivity change.  Acceleration can potentially be used to compare teams because unlike velocity it is unitless (my points are different than your points, and rightfully so). and can even be monetized.
        2. Release burn down charts.  These were popularized by Scrum and can be used to provide a rough projection of the remaining schedule and by implication project cost.  Unfortunately burn down charts tend to provide overly optimistic projections in practice.
        3. Ranged release burn down chartsRanged release burn down charts are a more sophisticated take on release burn down charts which we suggest in DAD.  It uses both the gross velocity (which Scrum refers to as velocity) and the net velocity to calculate two projections, an optimistic and a pessimistic one, thereby giving you a ranged projection.  The estimation community, BTW, has suggested for many years now to give ranged not single estimates.
        4. Ranged burn down trend chartsRanged burn down trend charts show the evolving ranged estimate throughout a project.  From what I've seen, at least for healthy projects, this chart tends to track closely to the estimation cone predicted by estimation theory.
        • M_Kevin_McHugh
          M_Kevin_McHugh
          22 Posts
          ACCEPTED ANSWER

          A Disciplined Approach to Velocity was Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

          ‏2012-10-26T04:21:16Z  in response to ScottAmbler
           Scott, a variation on the ranged idea.  I do like the uncertainty projections.
           
          What about adding Upper and Lower Control Limits to the stew?  It comes for free and provides interesting, normalizable information to report out to management that probably wonders how to measure agile teams.
           
          At the risk of burdening the Product Owner, what about estimations of value in addition to complexity?  This would provide a different (and more valuable) sorting mechanism for prioritizing the backlog.  ROI burndown  :-)
           
          @Densmore @Walker - this goes towards some of the economics of software development.
           
          • ScottAmbler
            ScottAmbler
            24 Posts
            ACCEPTED ANSWER

            A Disciplined Approach to Velocity was Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

            ‏2012-10-26T13:50:25Z  in response to M_Kevin_McHugh
            Kevin, I've done exactly that in practice.  The challenge is generally around net velocity because it can vary widely, particularly at times when requirements change widely.  For example, say we have 200 points left on the backlog and a gross velocity of 10.  Consider the following scenarios:  No new functionality is added, therefore net velocity is 10 ==> No ranged estimate this iteration.  8 points added, therefore net velocity is 2 ==> Huge range in the estimate (20 iterations left vs 100 iterations).  10 points added, net velocity is 0 ==> Net velocity indicates an infinite schedule.  12 points added, net velocity is -2 ==> we finished 100 iterations ago.  The point is that we need some sort of control limits as you suggest.  In the DAD book we suggested on potential hueristic is to have a lower limit on net velocity to be half the gross velocity.  Another approach is to take averages over several iterations, thereby smoothing the results a bit.  And there's other strategies as well.  It can make for some interesting spreadsheet formulas at times (or some interesting report logic for vendors of such tooling). 
          • Cherifa
            Cherifa
            30 Posts
            ACCEPTED ANSWER

            A Disciplined Approach to Velocity was Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

            ‏2012-10-26T19:57:34Z  in response to M_Kevin_McHugh
             Kevin, caught this: Estimation based on "value" in addition to complexity? In my view, it is unsafe to estimate based on value. Who would estimate it? Your stakeholders? They may have so many different views on "value"...my "value" is not your "value". At the end of the day, we want the developers to do the estimate based on their skills, the complexity of the feature to be implemented. How do you you want to expense the feature to be developed? On value or complexity? Now you have the prioritization attribute which would help with the so called "value" +risk+ dependencies+...+ . This is where all stakeholders will have a say. My 2 cents.
            • JimDensmore
              JimDensmore
              24 Posts
              ACCEPTED ANSWER

              A Disciplined Approach to Velocity was Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

              ‏2012-10-26T20:29:54Z  in response to Cherifa
              Actually, without estimating and measuring based on value, I know of no way to objectively align IT (or whatever the development shop is) with the business.  Money is the only unit of value measure that is equitable during cost comparisons - cost is in currency, so value needs to be in currency.  The value model you create for this takes inputs from all the stakeholders who would accrue value, or cost, or both, from the investment(s) under consideration.  The key to enhancing safety, as you put it, is to allow the stakeholders to supply answers that are uncertain.  Holding them to a single value is where the problems come in.
               
              We've found that a triangular distribution is generally sufficient, so permitting a stakeholder to supply an uncertain answer only requires that s/he supply three numbers instead of one - low, high and likely.  The conversation often starts like this: So how much value do you expect to accrue from this investment?  "heck, I dunno."  Could it be a billion dollars?  "Oh, gee, no, not that much."  Well then you do have some idea then, right?  ... and you tease out the triangular distribution thusly.  (Sometimes they respond with a formula ... we can handle that too.)  I have only had this fail when it was considered politically inadvisable to supply any numbers.  Even in that situation it was clear the stakeholders were willing to give me a triangular distribution except for the politics.  One can't always solve the politics, but a set of stakeholders who have been directed to supply inputs to a value model will always be able to build one.  The COOL part is once you have a value model, (1) adding to it becomes part of every retrospective, (2) deciding how to beat down the biggest (highest priority) uncertainties (risk!) to which it alludes becomes part of every sprint plan, and (3) once your value model accounts for uncertainties, your stakeholders become willing to be accountable for what the model says!!!
            • M_Kevin_McHugh
              M_Kevin_McHugh
              22 Posts
              ACCEPTED ANSWER

              A Disciplined Approach to Velocity was Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

              ‏2012-10-29T02:17:43Z  in response to Cherifa
               Ahh!!!
               
              I agree that the Team should not estimate value.  It's not their job.  It's not their business.  The Team owns Complexity.
               
              However, ....
              The Product Owner owns the backlog and prioritization.  Ask and answer the question: "How is the Product Owner to decide on ranking?"
               
              And, Cherifa, you've hit on a couple possibilities.
               
              In summary, and of several factors can be used: Value, Risk, ...
               
              As I point out in my article (Velocity: what flavor would you like? -- part 2 in a series) there are several ways to think about Velocity.  I provided charts in that article to assist with visualizing the uses.
               
              1) Complexity Velocity -- owned by the Team, the "traditional" way we think of velocity.
              2) Value Velocity -- owned by the Product Owner -- leads to Agile Porfolio Management.  Investment Analyzer is a cool way to think about this space
              3) Risk Velocity -- owned by a combination of the Team and Product Owner
              4) Return on Investment (ROI) Velocity -- owned by the Product Owner.
               
              As I see it, Value and ROI are the right ways to "sort" stories.  In my models in my article, Value ended up being a better way to sort.  I need to spend more time modeling and working the math to convince myself that Value is better than ROI.  Actually, my intuition tells me ROI is better.  It's what the Product Owner really wants.
               
              • JimDensmore
                JimDensmore
                24 Posts
                ACCEPTED ANSWER

                A Disciplined Approach to Velocity was Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                ‏2012-10-30T19:11:13Z  in response to M_Kevin_McHugh
                Right, I see the problem, it's who needs the measures?  And I think you have it right Kevin, or at least reasonable.  If we make the common assumption that costs are reasonably well known (small uncertainty - it's not always true, but often it is true) then as value goes, so ROI goes, since ROI = value/cost = (benefits-cost)/cost.  And also thus, one always can be calculated from the other quickly.  Let's suppose two projects have the same value, and one is half the cost of the other, e.g. V1=10, V2=10, C1=4, C2=2, then B1=14, B2=12, ROI1=2.5, ROI2=5.  They rank the same on the value scale.  However one ranks significantly better than the other on the ROI scale.  That should make clear that ROI is the better ranking measure, since in project 2 the same value accrues for half the cost.  Sorry, I have been careless, saying value sometimes when I needed or meant to say ROI.
                • M_Kevin_McHugh
                  M_Kevin_McHugh
                  22 Posts
                  ACCEPTED ANSWER

                  A Disciplined Approach to Velocity was Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                  ‏2012-10-31T02:39:21Z  in response to JimDensmore
                   Jim.... can I quote you on that lead in?   lol   ;-)
                • M_Kevin_McHugh
                  M_Kevin_McHugh
                  22 Posts
                  ACCEPTED ANSWER

                  A Disciplined Approach to Velocity was Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                  ‏2012-10-31T02:41:23Z  in response to JimDensmore
                   Actually, Jim, ... it's possible to use either as a ranking system.  And, it is... helpful (I took me a moment to avoid using "valuable").... to be able to evaluate against either.  It opens options.
            • DougStewart
              DougStewart
              10 Posts
              ACCEPTED ANSWER

              A Disciplined Approach to Velocity was Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

              ‏2012-10-29T17:03:37Z  in response to Cherifa
               An interesting thread which I think exposes a gap in our practice as much as it provides many of the answers.  We often forget that software development is a business process and not just a project.  When we change a business process, there is usually a business case that defines the ROI = Value / Cost equation.  We have excellent tools now to allow us to control the denominator, cost, and to communicate our current progress towards the goal.  I agree that the team should not be trying to define or measure value, instead defining and measuring scope.  The scrum product owner does apply a qualitative indication of value through priority within the scrum team.
               
              What we don't seem to have is a good way to measure value in some quantitative sense so that Program Management can determine that the project is on target for the minimum ROI and funding should continue. (@Kevin I'll be reading your papers in a few minutes.) Is there a good measure of value that can be used by Program Management?

              • Cherifa
                Cherifa
                30 Posts
                ACCEPTED ANSWER

                A Disciplined Approach to Velocity was Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                ‏2012-10-30T00:17:58Z  in response to DougStewart
                 Doug, Measuring scope is definitely a way to track progress. At the end of the day you need to deliver on customer's requirements  and those are part of the scope that team has defined.
                When we talk about measuring scope do you mean measuring scope changes (how many features have we added to or removed from the scope, how many have we modified), the number of baselined features, number of features we delivered ( this comes back to the velocity thing....I do not want to go there...), and/or else?
                In addition, what i find useful and as part of delivering on functional/non functional requirements are the post ship reports. Let's say within a telco company there is a call center which tracks all reports/complaints. How many calls did the call center receive before the features were shipped versus after ...
                Requirements test coverage is another metric related to requirements......
                Any other suggestions?  .
                Updated on 2012-10-30T00:17:58Z at 2012-10-30T00:17:58Z by Cherifa
                • DougStewart
                  DougStewart
                  10 Posts
                  ACCEPTED ANSWER

                  A Disciplined Approach to Velocity was Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                  ‏2012-10-30T02:15:41Z  in response to Cherifa
                   I agree.  I was talking about something a little different, though.  We always have the option to cut low priority scope in order to keep on the original schedule whenever we understand that we are not going to meet the original schedule.  The question then becomes one of understanding if the scope that is now to be delivered still meets the minimum required for the business case ROI requirements.  
                   
                  If the project no longer meets the ROI the options are to accept the situation, cancel the project,  add an iteration, or add another team depending on which fits best  with the business case.  Other than human judgement, I don't know of a way to value the subject scope in order to compare it with the additional cost of a team or iteration.  Thoughts?
                  • M_Kevin_McHugh
                    M_Kevin_McHugh
                    22 Posts
                    ACCEPTED ANSWER

                    A Disciplined Approach to Velocity was Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                    ‏2012-10-31T02:50:59Z  in response to DougStewart
                    Do I hear you correctly if I restate it as a matter of the total being greater than the sum?
                     
                    Flipping it around, one model is to say that if a project is work $100k, and there are 100 user stories, then each story is worth $1000,
                     
                    It's one model.  Another model might be to divy up the $$ amongst the stories with more details understanding of their contribution to the project.
                     
                    Some stories might not have value with out other stories.  That's a thought worth figuring out how to model.
                     
                    But, you're correct, Agile goes for scope reduction.  And, it might be that with reduced scope, and thus push the project below the line.  The project might be cancelled or help until it gets back above the line.
                     
                    But, to attempt to get to answer your question, I propose that the project and each epic or story has $$ value.  This is very likely to have an uncertainty band (think about the hurricane we are seeing... an uncertainty band is like the probably tracking lines they show on the weather report.   A cone of possible tracking).
                     
                    So, if each story has a valuation band, then, we could use a tool like Investment Analyzer (in Focal Point) to aggregate the bands.  And, if enough contributors drop out of the model (oh!!! I mean enough stories get pushed out of the scope), then you can re-evaluate the continuation of the project.
                     
                    This does mean the Product Owner becomes more of an investment analyzer (sorry for the overload of terms)  BTW, the team and the Portfolio Mgr experience a similar job role change.
                    • Cherifa
                      Cherifa
                      30 Posts
                      ACCEPTED ANSWER

                      A Disciplined Approach to Velocity was Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                      ‏2012-11-02T19:02:29Z  in response to M_Kevin_McHugh
                       Doug, Kevin...in addition to the good points you made:
                      1. You mentioned a "business case (BC)" : Depending on the nature of the project, usually a BC provides the evidence, the project is a good investment (ROI we are talking about). This is a good thing and I am ok with that. My problem  is that a  BC has the connotation of a commitment from the business to fund a project with a "fixed scope". In agile we all agree the scope is flexible, changes can happen and we need to cater for that. The scope is a continual negotiation between the stakeholders. Do we go into a level of talking about stories and story points in a BC? I do not think so.
                      To me there is a disconnect between the BC when it is done and the backlogs estimation activities  at the team level.(stories, tasks). The BC may be listing the themes, epics....and when the team start their estimation activities they focus on stories and tasks. Usually and as we progress in the development cycle, we become more accurate ( there is less uncertainty) with our estimate. What is missing  is a "Value case" to be determined and which shows how this value is defined, how measured (talking about metrics here and to the $ value Kevin mentioned) at each step,  and what are the steps to reach an optimal value.This value case should be a live document that can be reviewed at end of each iteration, at end of each release...In addition to the story value, risks, impediments, uncommitted team members, defects.. all those have values  too (they cost money to the project!!)
                      Kevin, re-evaluating the value case and learning from each step  so you can improve is what matter.The value-case to me is the "middle man" between the BC and the backlogs.
      • GustavoGrillo
        GustavoGrillo
        8 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-25T18:21:13Z  in response to Reed_Fegg
         I would suggest a customer-centric metric, for example:
        • How much revenue each team delivered as assessed by the customer or
        • How quality is perceived by the customer for the deliveries of each team
        Can anyone thing of anything else on these lines?
        • DanPollitt
          DanPollitt
          4 Posts
          ACCEPTED ANSWER

          Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

          ‏2012-10-26T13:37:49Z  in response to GustavoGrillo
           I have a great book published by Intel that really helps people think about measuring cost/delivered value from this point of view: Measuring the Business Value of Information Technology
          • JimDensmore
            JimDensmore
            24 Posts
            ACCEPTED ANSWER

            Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

            ‏2012-10-26T15:34:37Z  in response to DanPollitt
            Dan, how does the book address uncertainty?  How does it address lag in business level measures?  I'm only asking because my reading backlog on this topic is already large, and this book isn't available electronically.  Trying to screen before I buy.  Thanks!
      • richardknaster
        richardknaster
        8 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-29T15:52:54Z  in response to Reed_Fegg
        Reedy:
         
        I would want to find out from the Managers what they hope to achieve by comparing the two teams and then find a more appropriate set of metrics or actions to take.  Comparing teams leads to unhealthy competition between teams and that is the enemy of collaboration.  Teams will begin to game the metrics, not share things that they've learned to improve their team, or even worse not help a blocked team because it might hurt their own velocity, instead of doing what's right for the greater good.
         
        Wanting to compare teams is also a sign that management needs some coaching on how they can help teams.  For example: helping remove organizational impediments, creating an environment that fosters learning, continuous improvement, inspection and adaption. If managers key focus is on productivity, then there is a high likelihood that the transformation to agile will fail.  Agile is a lot more than increasing productivity, it's about responding to change and customer feedback, empowering the team to determine how to do their work, and delivering high quality features in an iterative and incremental way.  Imagine using your speedometer to measure the productivity of someone's driving.  Will that drive the right behavior, or will it encourage going at unsafe speeds without considering the context of the situation.
         
        There is nothing wrong with measurement as long as it is properly used.  However, all too often teams are losing agility as managers are encouraging teams to meet target measures to conform to a plan vs. responding to change.  There is nothing wrong with being predictable or reducing variance, however that should not trump the team's ability to inspect and adapt.  Otherwise we are back to our waterfall ways.
        Updated on 2012-10-29T15:52:54Z at 2012-10-29T15:52:54Z by richardknaster
  • meerec
    meerec
    6 Posts
    ACCEPTED ANSWER

    Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

    ‏2012-10-25T15:05:36Z  in response to richardknaster
    Some discussion points from me -- Richard and Cherifa inspired.
    1. Motivation: if a team can measure an improvement, it gives them a reason to continue on this improvement path. A critical factor in motivation is not measurement itself but empowerment -- moving decisions to the lowest possible level in an organisation while developing the capacity of those people to make decisions wisely.  
    2. Noise: generating too much data can create noise -- what will you do with all this information? 
    3. Information radiators & extreme feedback are important: keeping meaningful measurement data visible will remind everyone of the value and goals and progress made. 
    4. Focus on the right level: the tendency to decompose a big job into smaller tasks and then optimising every task is often a flawed strategy -- like winning the race is not about winning all the stages --  may not be an attainable objective. 
    5. The real purpose of measuring is to reduce uncertainty. And the measurement does not need to be exact for it to help reduce uncertainty. We don't need perfect measurement, only such that help us answer questions.  
    6. But ... you get what you measure: if you cannot afford to measure all that is important, partial measurement shouldn't be your goal as you are in serious danger of encouraging incorrect behaviour; for instance, need to measure profitability of the business created by a product not its development cost.
     
     
    Updated on 2012-10-25T15:05:36Z at 2012-10-25T15:05:36Z by meerec
    • RickW
      RickW
      2 Posts
      ACCEPTED ANSWER

      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

      ‏2012-10-25T12:29:42Z  in response to meerec
      Great feedback on Velocity (fairly applied across teams) as a key metric. It seems we all agree on the importance of this metric.  
       
      What about other key metrics?  Would you classify "Burn Down" for a sprint/iteration as another key metric in our toolbox?
      • RafaelSene
        RafaelSene
        4 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-25T12:33:35Z  in response to RickW
         I don't believe that we could call "Burn Down" as a metric, but I do believe that what it shows must be the input for all metrics we use.
      • DanielSHaischt
        DanielSHaischt
        5 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-25T12:43:43Z  in response to RickW
         What kind of burndown? I made the experience that teams over here have more difficulties with getting the release burndown right in comparison to sprint burndowns which seem to be easier to accomplish.
         
        The are metrics which we defined over here back in 2010 or so:
         
        Continuous Integration (Scope: Team & Scrum Master)
          Build Usability Ratio

        Stakeholder Feedback (Scope: Team, Scrum Master, Product Owner)
          planned vs. actual stakeholder interactions

        Iteration Burndown (Scope: Team, Scrum Master)
          Progress within the iteration

        Release Burndown (Scope: Scrum Master, Product Owner, Executives)
          Remaining Work for Release

        Reflection
          Reflection (Scope: Team, Scrum Master)
          Actions from Reflection (Scope: Scrum Master, Product Owner)

        Quality Metrics
          Technical Debt (Scope: Team, Scrum Master, Product Owner)
          Unresolved Blockers (Scope: Scrum Master, Product Owner)
          Field Quality (Scope: Product Owner, Executive)
      • meerec
        meerec
        6 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-25T12:47:48Z  in response to RickW
         Agree with @RafaelSene ... not a metric as such. But important information to display on the status of the project. My point #3 above about extreme feedback calls for such information radiators.
  • DanielSHaischt
    DanielSHaischt
    5 Posts
    ACCEPTED ANSWER

    Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

    ‏2012-10-25T12:22:31Z  in response to richardknaster
    Would one agile metric or metric to meassure agile adoption of a particular team be the existence of emergent behaviors in such a team? The reational behind this question is that emergenze could be an indicator on whether how empowered teams are.
     
    See:

    http://c2.com/cgi/wiki?EmergentBehavior
    http://en.wikipedia.org/wiki/Emergence
    • Reed_Fegg
      Reed_Fegg
      14 Posts
      ACCEPTED ANSWER

      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

      ‏2012-10-25T12:47:38Z  in response to DanielSHaischt
      From my view I guess it depends on what type of "burndown" or better yet "burnup" you are referring to.
       
      Sprint burndown is not a metric that I would recommend using outside the team but Release Burnup is a measurement used to communicate value delivered.
       
      Product owners use that to evaluate if project team is delivering against their commitments and hopefully as a way of showing value delivered business.
  • ScottAmbler
    ScottAmbler
    24 Posts
    ACCEPTED ANSWER

    Goal-Question-Metric (GQM)

    ‏2012-10-25T13:46:32Z  in response to richardknaster
    Governance is baked right into Disciplined Agile Delivery (DAD) and your approach to metrics is an important part of your governance program.  An important metrics strategy that DAD promotes is GQM. The fundamental idea behind GQM is that you first identify a goal that you would like to achieve, then a set of questions pertinent to achieving that goal, and then the metric(s) associated with each question. Of course, as you fulfill your existing goals new goals will emerge, so GQM proves to be an evolutionary strategy in practice.   Let's consider an example.  Let's assume that your organization has a goal of reducing technical debt.  Potential questions that you may ask include what the current level of technical debt is and are you reducing technical debt over time?  To measure the current level of technical debt you could use code quality metrics, perhaps determines via code analysis tools, or defect density metrics.  To determine if you're reducing technical debt over time you might consider build health/trend metrics, defect trend metrics, and test coverage metrics.   Moving forward, if I may be so bold, I'd like to suggest that we take this conversation out of the metrics weeds for awhile and instead take a GQM approach to our discussion.  For some hints as to what that may look like, check out Chapter 20 of the Disciplined Agile Delivery (DAD) book
    • DanielSHaischt
      DanielSHaischt
      5 Posts
      ACCEPTED ANSWER

      Re: Goal-Question-Metric (GQM)

      ‏2012-10-25T14:39:23Z  in response to ScottAmbler
      Sonar (www.sonarsource.org) exactly does hat you are describing here. Yet I think depending on what the team agrees upon technical debt could be broadened to an extend where it's not only code centrig (i.e. based on static & dynamic code analysis).
    • ScottAmbler
      ScottAmbler
      24 Posts
      ACCEPTED ANSWER

      Re: Goal-Question-Metric (GQM)

      ‏2012-10-25T14:50:18Z  in response to ScottAmbler
      The goals that we identifies in the DAD book are:
      1. Deploying in a timely manner.  Notice how it's not worded "be on schedule".
      2. Spending IT investment wisely.  Notice how it's not worded "be on budget".
      3. Providing business value.  In DAD we promote the idea that the primary measure of progress should be consumable solutions that provide quantifiable value to your stakeholders, not just "potentially shippable software".
      4. Producing a quality solution.  
      5. Reducing technical debt.  Easy to talk about, but a lot harder to do systematically.
      6. Providing a healthy working environment.  Agile is a people-oriented paradigm.  We should likely measure how we're actually doing at that. 
      7. Being regulatory compliant. 
       
  • ScottAmbler
    ScottAmbler
    24 Posts
    ACCEPTED ANSWER

    Consider the Audience of a Metric

    ‏2012-10-25T13:51:42Z  in response to richardknaster
    An important implication of the GQM approach is to understand the audience for potential metrics.  This is a theme that we've seen come out in previous postings, several people have indicated that some metrics are for the team itself and some for management.  In DAD we observe that there are three potential audiences for agile metrics:
    1. Your team.  A DAD team wants timely and accurate metrics so that they can self organize more effectively.  For example they may refer to an iteration burndown chart during a coordination meeting to understand their current status or a defect trend chart to help them to understand how well they’re responding to incoming defects. 
    2. Your stakeholders.  Project stakeholders are often interested in answering fundamental questions such as when will the next release be shipped, is the solution of sufficient quality, what is the expected cost, and are we satisfied with the solution so far?  Ideally these questions are answered through a development intelligence (DI) strategy where stakeholders have access to an automated dashboard that focuses on stakeholder-specific issues. 
    3. The governors.  Your governing body will want to monitor project teams so they can make better decisions and hopefully guide project teams effectively.

    • WalkerRoyce
      WalkerRoyce
      18 Posts
      ACCEPTED ANSWER

      Re: Consider the Audience of a Metric

      ‏2012-10-30T18:47:15Z  in response to ScottAmbler
       This is an interesting thread. There seems to be some confusion (maybe just i am confused) around who metrics are aimed at. Some metrics may be targeted for team use, and some may be targeted for management/governor use. In the spirit of transparency, I don't think this "apparent pigeonholing" is a good idea.
      As I have managed programs and teams big and small, the most insightful measures were always the ones that the team used themselves for their own prioritization, planning and reporting. For sure, some measures are more useful for daily engineering activities and others for steering decisions, but I think we should always strive for everyone to have transparent access and understanding of cause and effect relationships between different measures.
      If this demands that we have more technically capable managers, so be it. 
       
      Click here to join the break-out discussion on this topic.

      Updated on 2012-10-30T18:47:15Z at 2012-10-30T18:47:15Z by richardknaster
      • Cherifa
        Cherifa
        30 Posts
        ACCEPTED ANSWER

        Re: Consider the Audience of a Metric

        ‏2012-10-25T14:23:32Z  in response to WalkerRoyce
         Walker, I must have posted my reply at the same time as yours. talking about insightful measures , what are they? In terms of quality metrics? which ones are of interest to the team vs the managers?
        Daniel posted the following for Quality Metrics
          Technical Debt (Scope: Team, Scrum Master, Product Owner)
          Unresolved Blockers (Scope: Scrum Master, Product Owner)
          Field Quality (Scope: Product Owner, Executive)
        Can you please share your experience? When collected, how used? which ones are easy to collect? which one are acted upon  for improvement?
        • WalkerRoyce
          WalkerRoyce
          18 Posts
          ACCEPTED ANSWER

          Re: Consider the Audience of a Metric

          ‏2012-10-25T14:31:58Z  in response to Cherifa
           The most insightful quality measures are 1) defect rates, 2) change rates, 3) cost of change trends. All of these are derived from the evolving code and test baselines.
          Agility means "can change rapidly" so we need to measure change. 
          And my belief is that these core quality metrics are interesting to all stakeholders
          • Cherifa
            Cherifa
            30 Posts
            ACCEPTED ANSWER

            Re: Consider the Audience of a Metric

            ‏2012-10-25T15:13:44Z  in response to WalkerRoyce
             Measuring change is an important aspect !. How do you measure changes within a team?
            do you measure them by the number of enhancement requests?  number of new requirements? requirements churn?
            What else? any insight of  measuring change rates?
            • WalkerRoyce
              WalkerRoyce
              18 Posts
              ACCEPTED ANSWER

              Re: Consider the Audience of a Metric

              ‏2012-10-25T15:46:12Z  in response to Cherifa
               I wrote extensively on this topic in my SW Project Management book (Appendix C). And Appendix D provides a thorough case study to help with understanding usage models. I also published a paper on Measuring Agility and Architectural Integrity in 2011 on this topic.
               
              A simple answer is probably too simplistic. Nevertheless, here is the tip of that iceberg. You need to measure the volume of change (in SLOC, FPs, UCs, User stories, files, TCs, bang, or other measures) and understand types of change (defect, enhancement, new feature, performance, upgraded infrastructure, etc.). When instrumented in your change management system, one can extract short-term and long-term trends that illustrate progress and quality trends.
              The only factual change measures come from code and test baselines. Change rates in requirements and models are much more speculative measures.
               
              • JimDensmore
                JimDensmore
                24 Posts
                ACCEPTED ANSWER

                Re: Consider the Audience of a Metric

                ‏2012-10-25T16:18:52Z  in response to WalkerRoyce
                 I now confess the sin of having never noticed Appendix C before :-) ... lots of thoughtful stuff in there.  Audience is part of context, and I bet we all have had good experiences with being careful to set context well ... and bad experiences when one fails to do so.  So I'm all in there, audience must be part of the measurement plan.
                 
                The thought that occurred to me reading this is that maturity is about how much Quality is affected by a given Change Volume.  So, make a big change.  Is there a big quality hit?  That's lower maturity than if the quality hit is small.  I've never cottoned to the idea that bang is bad.  Bang is a power tool, and it's very easy to misuse.  If I have use case points available I'll use that instead.  But UC or Function points are difficult to get to, and I'm happy to start with bang.  Just don't draw hasty conclusions from it.
                 
                Finally, 'agility means "can change rapidly"' ... yes, and: agility means 'can change rapidly without suffering too great an adverse quality consequence'.  But that's probably what you meant.  I have had many conversations about Agile vs. agile: Agile is a means to be agile.
            • RickW
              RickW
              2 Posts
              ACCEPTED ANSWER

              Re: Consider the Audience of a Metric

              ‏2012-10-25T16:04:46Z  in response to Cherifa
              Good question Cherifa - The metric I think is most relevant to the amount of change is requirements volatility.  (This could be decomposed into the things you list above.).  I've seen some groups able to measure this well and some groups struggle with trying to track this at a level that is valuable. 
               
               
          • AlexXIE
            AlexXIE
            8 Posts
            ACCEPTED ANSWER

            Re: Consider the Audience of a Metric

            ‏2012-10-26T06:25:21Z  in response to WalkerRoyce
            I agree on "Can change rapidly"  is agility capacity indicator, and definitely the volume ways and algorithm of Walker had contributed are real good stuff. While I am opposite to encourage unnecessary change, for example, some error zone
            1. Shorten the iteration cycle without adequate customer involvement heartbeat, like the iteration cycle is 2 weeks, the customer engage only every per month-  actually even there were customer involve frequently, here we will only spoil the customer  in giving incomplete thoughts, have the business analyst shrunk responsibilities, and exhaust your team.
            2. Think the 2 months iteration  development process is not as agility as the 2 weeks iteration one - since the iteration length is decided by "how long you can't protect your team from change" in stead of the "capability of team rapid change" .
            • ScottAmbler
              ScottAmbler
              24 Posts
              ACCEPTED ANSWER

              Re: Consider the Audience of a Metric

              ‏2012-10-26T14:00:05Z  in response to AlexXIE
              Alex, here are some thoughts for you:
              1. Stakeholders can be actively involved every single day.  In Agile Modeling we promoted the practice of Active Stakeholder Participation, an extension to XP's On-site Customer practice.  To make this happen you have to have access to your stakeholders, implying that they understand the importance of doing so; team members need to have adequate communication and collaboration skills; and you need to adopt inclusive tools and techniques that enable stakeholders to actively participate.
              2. Having stakeholders actively involved can be challenging but can also be very rewarding.  Yes, they will get incomplete information at times, but this is an issue that they can easily overcome once they understand the realities of development and they can see quick feedback on their suggestions.  The benefit of active stakeholder participation is your increased ability to reduce the feedback cycle and thereby increase your productivity and increase the chance that you'll build something that meets their real needs.
              3. Shorter iterations are generally better than longer ones.  I'm incredibly leery of iterations longer than 4 weeks and generally promote two or one week iterations.  I've run several IT surveys over the years that show that the vast majority of agile teams have iterations of 4 weeks or less and that the most popular iteration length is 2 weeks.
          • KristofKloeckner
            KristofKloeckner
            2 Posts
            ACCEPTED ANSWER

            Re: Consider the Audience of a Metric

            ‏2012-11-03T20:27:00Z  in response to WalkerRoyce
             Reminds me of a great metric that I believe Mary Poppendieck brought up - how long does it take to deploy a change in production (I think she even said 'a single changed line').
            As we look at agile delivery encompassing an extended lifecycle including deployment into production, this becomes a very important test of how agile you truly are..
        • M_Kevin_McHugh
          M_Kevin_McHugh
          22 Posts
          ACCEPTED ANSWER

          Re: Consider the Audience of a Metric

          ‏2012-10-31T02:58:06Z  in response to Cherifa
           There is some excellent thought/efforts out there on measuring technical debt.  Unfortunately, my bookmark is on my home computer and my google search is not turning up the same hit.  I will repost if I get the chance in the 24 hours I have at home this weekend.
           
          @Jim @Walker - it is up in the territory of thought that IA gets into.  Hope I can find that link.
      • ScottAmbler
        ScottAmbler
        24 Posts
        ACCEPTED ANSWER

        Re: Consider the Audience of a Metric

        ‏2012-10-25T14:25:49Z  in response to WalkerRoyce
        BUT, from an outside-in-development (OID) point of view you do want to understand the potential audience.  Surely your team will have different goals than your business stakeholders, who will have different goals than your operations stakeholders, who have different goals yet again than your governors, and so on.  The implication is that there will be different questions to address those different goals.  In Chapter 20 of the DAD book we described 20+ potential agile metrics, 7 potential goals (just an exemplar list), questions for those goals, then mapped metrics to the questions.  As you would expect any given metric mapped to one or more questions. 
         
        Yes, we could have a dashboard that shows all the potential metrics for a project and I have no doubt that some people will find that very useful.  I highly suspect that many people will want tailored dashboards that show the subset of the metrics that address the questions that they are interested in.  When we look at public dashboards, such as the ones used by the Jazz team, these are in fact tailored, presumably, to meet the needs of the audience for those dashboards.
        • Cherifa
          Cherifa
          30 Posts
          ACCEPTED ANSWER

          Re: Consider the Audience of a Metric

          ‏2012-10-25T15:04:34Z  in response to ScottAmbler
           Point taken Scott!! Considering the audience is very important when it comes to select the metrics.
        • WalkerRoyce
          WalkerRoyce
          18 Posts
          ACCEPTED ANSWER

          Re: Consider the Audience of a Metric

          ‏2012-10-25T15:33:26Z  in response to ScottAmbler
           If the team has different goals than other stakeholders, then you have a big problem. I think you meant that they have different perspectives on the same set of shared goals. You listed 7 goals earlier: timeliness, efficiency, value, user quality, maintainability, morale, compliance. I summarized your 7 goals with one word each, but each of these must be a shared, measurable goal among all constituencies.They are not equally important to each constituency, but they are all connected and you can't change one without affecting some others. Everyone needs some level of understanding in these tradeoffs and consequently transparency is the key to maintaining a high-trust environment across the whole team.
          Now, how we view progress in achieving those goals may branch into various measures or tailored perspectives that are more appropriate to a specific audience (like end users, operators, testers, designers, funders, IVVers, et al). 
           
          Good measurement reduces uncertainty which increases trust which reduces overhead which improves efficiency, which leads to higher morale, quality, and business value.
          • JimDensmore
            JimDensmore
            24 Posts
            ACCEPTED ANSWER

            Re: Consider the Audience of a Metric

            ‏2012-10-25T16:44:42Z  in response to WalkerRoyce
             Perhaps this means that audience is partly about focus - who are you focusing on more.  Other stakeholders count, but not as much ...
          • Cherifa
            Cherifa
            30 Posts
            ACCEPTED ANSWER

            Communication...communication

            ‏2012-10-25T17:08:06Z  in response to WalkerRoyce
             Good point ! If team has different goals than other stakeholders then you have a pb!! Definitely!! but the reality is that team working on projects are clueless about managers/exec goals. having different perspective on the same set of goals is fine but not bridging the  gap between different audience is a problem. So my take on this : You need a good communication mechanism ( top-down and bottom up) to suceed with whatever measurement framework you put in place. You have to get rid of the syndrome: "i was told" to collect this metric but I do not know why?
             
             
            • GustavoGrillo
              GustavoGrillo
              8 Posts
              ACCEPTED ANSWER

              Re: Communication...communication

              ‏2012-10-26T12:13:10Z  in response to Cherifa
               Maybe asking stakeholders outside the team help building metrics that are meaningful for THEM might be a good communication/involvement tool. You'd get metrics that are actually monitored and useful, and these might send the right message about what's valued by the stakeholders
      • SmartDJ
        SmartDJ
        4 Posts
        ACCEPTED ANSWER

        A basic Metric Set based on GQM method

        ‏2012-10-26T00:29:01Z  in response to WalkerRoyce
         Followed Walker's defination of    the good metrics and GQM method , I am designing a basic measurement set,the customer focus on Product Competetiveness, Poductivity, Quality and TTV, here is the basic thought any comments are welcome:   

        Basic Measurement:

        1.Size:SP or xx Point
        2.Finshed Story point | Requiement over time and Value Point over time
        3.Change number and impact over time
        4.WI progres and Build Usability Rate
        5.Test Case finish rate for FITUTFVTSVT
        6.Defect # for development phase and after release 
        Then we could get the indicator for  Product Competetiveness, Poductivity, Quality and TTV
        Updated on 2012-10-26T00:29:01Z at 2012-10-26T00:29:01Z by SmartDJ
      • richardknaster
        richardknaster
        8 Posts
        ACCEPTED ANSWER

        Re: Consider the Audience of a Metric

        ‏2012-10-30T18:46:04Z  in response to WalkerRoyce
         Since there's been some much discussion about this topic, I've created a separate discussion thread, so we can go deeper in detail.
         
        Click here to join the break-out discussion.
         
        I think it would be good to discuss how to best deal with the transparency of metrics and how to avoid their misuse by those outside the team.  Agile principles suggest we are open and transparent with plans, progress and problems, yet many people have experienced that certain metrics like Velocity are being misused.and are proposing that certain metrics not be shared.  How has your organization dealt with this problem?
    • Cherifa
      Cherifa
      30 Posts
      ACCEPTED ANSWER

      Re: GQM and Consider the Audience of a Metric

      ‏2012-10-25T14:09:52Z  in response to ScottAmbler
       Good discussion Scott: A measurement framework at the enterprise level is based on the audience of the metric  and  is indeed   a GQM .kind of top down  framework ( business goals will translate into operational goals which themselves translate into project/practices goals). Deciding to reduce the technical debt  may have been derived from a top business goals which is related to quality improvement. Let's hold this discussion for a mn and see if we can decide first at the project level what metrics are of interest to the team and their managers. We had good discussion on velocity,  can we summarize and see what else is being used by teams and based on your experience which other metrics are useful at the team/project level. Are those metrics discussed during retrospective? Oiutside of the retrospective, where else they are discussed?
      Updated on 2012-10-25T14:09:52Z at 2012-10-25T14:09:52Z by Cherifa
      • JimDensmore
        JimDensmore
        24 Posts
        ACCEPTED ANSWER

        Re: GQM and Consider the Audience of a Metric

        ‏2012-10-25T16:42:29Z  in response to Cherifa
         Yes, Cherifa.  As always, we have lots of discussions about ideas and principles, but these discussions can be frustrating for people who aren't yet able to take the principles and apply them to develop some real measurements they can leverage to get feedback on how they're doing.  We have to enable this, and I say this because when I look around in the industry, too few are measuring, and when they are measuring, too few are measuring what they really need to measure.  (In fact, Douglas Hubbard says that essentially no one is measuring what they need to measure.)
         
        I use four frameworks to understand concretely what to do.  The first is GQM - already discussed here.  Second is business/operational/practice goals-then-metrics as a way to ensure, when you build your feedback loop, that you are aligning with your business.  This second framework is as follows: articulate your business goals (e.g. revenue, or profit (not the same)), then your operational goals (e.g. improved productivity, improved quality, earlier time-to-market), then trace the two sets and prioritize each, then build from your top priority operational goals a list of practices you decide you will adhere to in an effort to meet those operational goals.  Now you've gone down the feedback loop, and you must now return to the top, which we do with metrics.  Practice metrics tell you that you are executing the practices you have decided to execute.  For example, if you decided to do MDD then one practice measure e might be the percentage of model-generated code, with a goal being a higher percentage over time.  Then we have operational metrics to tell you whether the practices you are executing (and which you now KNOW you're executing because you put practice metrics in place as required in order to do so) are getting you the operational benefits/goals you sought.  And finally, though they lag and are affected by lots more than what your development shop might be doing, you put business metrics in place to tell you whether your business is improving because you are meeting your operational goals.  (I use Microsoft as an example here: I do not know whether this is true, but it might be true that Microsoft decided not to focus so much on quality because they were getting good business results without spending that much money on quality.  Thus their quality is annoying sometimes but they still succeeded in the marketplace.)
         
        Third framework is simple but important: it's about the difference between metrics and measures.  In appxC Walker calls the measures Statistics, and this might be a better word, I'm considering using it.  For me, a measurement is closer to raw data.  A metric is information you have decided to determine from crafting an understanding of some combination of measures and of their trends.  Measures are more about implementation, metrics are more about plans.  So, Statistic or Measure: volume of code (SLOC) that's model-generated.  Statistic or Measure: total volume of code (SLOC).  Practice Metric: percentage or fraction of code that's model-generated as a means to determine whether we're following the MDD practice, derived from model-generated volume over total volume. Practice metric goal example: trend improving up to 80%.  (Choose your max desirable percentage carefully.)  Could give similar examples at operational and business levels.
         
        Fourth framework is the measurement plan template, this plan is a crucial artifact.  I won't outline its content here but can do so separately if needed.  Of course, it includes audience :-) .
    • mjlyons
      mjlyons
      1 Post
      ACCEPTED ANSWER

      Re: Consider the Audience of a Metric

      ‏2012-10-29T16:45:00Z  in response to ScottAmbler
       I agree that agile metrics should be viewed from different perspectives and for different needs, 1) the team wants to monitor progress, help them reduce inefficiencies, build a high performing team,  velocity, story point completed are examples that we are know at this level.  Once we move up to 2) stakeholders and  3)management thing become more complex. These groups I believe are really looking for transparency.  They want to know if the release dates are going to be met, is the budget on schedule, all of those normal PM metrics.  I don't think they are concerned about what the specific metrics are as long as they answer, am I going to get what I asked for, in the agreed upon time and with the budget allocated.  I think this is much more critical today as agile moves into more traditional organizations where a team working on a wall board with little outside metrics is not acceptable.  At the request of one client,  I tried normalizing velocity across 30 or so teams at a client, but it became very easy for the scrum master to  manipulate the numbers.   I don't think comparing one team to another is a practical or useful exercise, however, metrics that give transparency are.
  • PeterRhysThomas
    PeterRhysThomas
    1 Post
    ACCEPTED ANSWER

    Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

    ‏2012-10-25T15:54:16Z  in response to richardknaster
     One important point to note here is the difference between a measure and a target:
    •   A measure allows monitoring some value, with respect to diagnosis or analysis of a system.  For example a doctor taking your temperature or pulse to aid diagnosis of health.  Collection of measures allows for analysis to inspect and adapt a process, however an effective measure is one which cannot easily be directly influenced (due to Goodhart's law - more on that below).
    • A target is a specific level of a measure and is intended to define a certain behaviour.  However as been noted, specifically by Charles Goodhart (who formed Goodhart's law based on political policy in the UK in the 1970s) that when a measure becomes a target it ceases to be a good measure.

    I have seen many examples of Goodhart's law in practice, for instance if you set targets for velocity - teams will increase thier estimates. My favourite metric - code coverage - I cover in this blog post

     So maybe to take these questions back a to basics a little, my main questions are
     
    • What is the purpose of the metrics, measures, targets which we want to analyse
    • Do we want these to influence the behaviour of our teams, and if so are we sure that we fully understand the consequences of what we want to measure.
    • HazelWoodcock
      HazelWoodcock
      10 Posts
      ACCEPTED ANSWER

      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

      ‏2012-10-25T17:16:42Z  in response to PeterRhysThomas
       Peter, you raise a good point here.  Metrics should be informing decisions and influencing behaviour (of the teams, or management, or sales force...).  Collecting metrics and setting targets without careful thought about the consequences is a very dangerous thing for the health of a project and I am glad to hear that there is a law (cautioning) against it.
      We can measure to compare project progress against a known curve and use the results to take corrective action if we see things going wrong.  We can measure to inform process improvement.  
      Walker says above that 'Good measurement reduces uncertainty which increases trust which reduces overhead which improves efficiency, which leads to higher morale, quality, and business value.'  We should also consider that Bad measurement decreases trust, subverts effort and causes long term damage.
      Perhaps the question we are really asking here is 'What is is safe to measure?'
      • JimDensmore
        JimDensmore
        24 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-25T17:32:07Z  in response to HazelWoodcock
         LOL.  Nothing!  Nothing is safe to measure.  It's just that it's way more unsafe not to measure.  Yes, that chain is a keeper, and once again we're back to the iterative fundamental -- Plan, then Do, then Retrospect, and Repeat.  Just like shampooing only different: keep repeating.
        • M_Kevin_McHugh
          M_Kevin_McHugh
          22 Posts
          ACCEPTED ANSWER

          Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

          ‏2012-10-26T04:13:17Z  in response to JimDensmore
          Jim's right
          Heisenberg
          Godel
          Murphy...
        • AlexXIE
          AlexXIE
          8 Posts
          ACCEPTED ANSWER

          Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

          ‏2012-10-26T06:51:52Z  in response to JimDensmore
           A single metric is not safe, how about proposing two competing metrics to balance?
          • M_Kevin_McHugh
            M_Kevin_McHugh
            22 Posts
            ACCEPTED ANSWER

            Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

            ‏2012-10-31T03:03:08Z  in response to AlexXIE
             Let me tell you a story....
             
            I once worked at a place where this one subcontractor promised no more than 3 defects per programmer per year.
             
            They then told the programmers that if they went over 3 they would get no raise that year.
             
            What do you suppose they did?  Well, they certainly didn't report more then 3 defects.
             
            In fact, they opened a single defect with a huge list of problems in it.  They closed it out at the end of the year.
             
            Their reaction of hostility was understandable when I asked them to record even unit test defects in ClearQuest.  This would go WAY over 3.
             
            A counter measure was needed.  And, indeed the original metric which intended to express high quality was poorly thought through.
             
            Metrics issues from the trenches....
            • JimDensmore
              JimDensmore
              24 Posts
              ACCEPTED ANSWER

              Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

              ‏2012-10-31T03:55:57Z  in response to M_Kevin_McHugh
              My suspicion is counter metrics or something else to counter behavior unintentionally promoted by metrics is always needed.  This spiral may be a big reason clients don't measure.  But the biggest metrics stories seem to involve personal metrics.  Big help if we follow Agile and keep away from metrics at the personal level.  (I suspect everyone here has had experience with this already.)
    • JimDensmore
      JimDensmore
      24 Posts
      ACCEPTED ANSWER

      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

      ‏2012-10-25T17:33:42Z  in response to PeterRhysThomas
       Let's add Target to the simple measure or statistic vs. metric framework.  Perfect, good stuff.
    • Cherifa
      Cherifa
      30 Posts
      ACCEPTED ANSWER

      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

      ‏2012-10-25T19:30:33Z  in response to PeterRhysThomas
      4It is important to set some metrics and have targets for continuous improvement. Let me explain it with an example: I have 20 defects at end of my release 1. I want to reduce the number of defects by 25% in my release2: reducing the number of defects by 25% is my target and it usually enforces the team  to focus on improving quality by reducing the number of defects. I do not see that as bad behavior. I do understand they are some metrics for which you cannot simply set some targets…(because you are afraid those can be distorted, i.e. velocity –this shows bad behavior). IMHO, defects burn-down is a good chart as it measures actual progress of reducing the number of defects and it enforces the team to focus on quality.Any other experience?
      Updated on 2012-10-25T19:30:33Z at 2012-10-25T19:30:33Z by Cherifa
      • DougStewart
        DougStewart
        10 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-25T20:20:36Z  in response to Cherifa
         defect burn down should account for both quantity and severity of defects to be the most effective.  Ready to release thresholds can be considered using this approach as well as measuring team capability.
        • Cherifa
          Cherifa
          30 Posts
          ACCEPTED ANSWER

          Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

          ‏2012-10-25T20:39:36Z  in response to DougStewart
           Agree!! and this probably can be done with RTC by having a filter according to the severity attribute.
      • AlexXIE
        AlexXIE
        8 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-26T06:56:09Z  in response to Cherifa
         Release by release the defect number comparison not wisely indicate the quality improvement.  As the quality not only depends on Developing Quality,  Testing Coverage and most importantly, also the easy to neglect thing is "Product or Feature themselves"
        • Cherifa
          Cherifa
          30 Posts
          ACCEPTED ANSWER

          Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

          ‏2012-10-26T15:17:15Z  in response to AlexXIE
           Good point: The defect rate is not the only measure that indicates quality improvement. There is more to Quality than focusing only on testing and finding defects.

          Updated on 2012-10-26T15:17:15Z at 2012-10-26T15:17:15Z by Cherifa
          • JimDensmore
            JimDensmore
            24 Posts
            ACCEPTED ANSWER

            Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

            ‏2012-10-26T15:51:54Z  in response to Cherifa
            A list I borrowed from Tox that I have found useful in addressing the various aspects of quality:
            • defect rate
            • production defect rate
            • frequent regression test failures
            • various health metrics like build health (and rate), and code health, including static and dynamic (think Purify+) analysis metrics
            • coverage
            A  very different metric I don't see very often relates to architecture and reuse; these factors contribute to quality through improved robustness generally, and using something already debugged specifically:
            • (development resources) / (use case points or story points)
            This metric should improve over time if your architecture is improving.  The same size use case shouldn't take as long next time, if you are able to reuse some of the assets you already developed.  In a way, this is a direct measure of agility.  In the limit, once your architecture is complete, the next use case should consist of nothing but building another orchestration of the operations available through the assets you already have.  In that limit, the only thing to debug is the orchestration.  (I'm reminded of the old analysis level control, entity and boundary stereotypes - orchestration refers to the control idea.)
             
             
            • ScottAmbler
              ScottAmbler
              24 Posts
              ACCEPTED ANSWER

              Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

              ‏2012-10-28T20:03:51Z  in response to JimDensmore
              Jim, the improvement over time of the velocity per person would actually be the acceleration metric
              • JimDensmore
                JimDensmore
                24 Posts
                ACCEPTED ANSWER

                Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                ‏2012-10-28T23:02:02Z  in response to ScottAmbler
                 Hmm ... where's the Like button.  Yes, absolutely, it would have been clearer for me to say it that way.
              • Cherifa
                Cherifa
                30 Posts
                ACCEPTED ANSWER

                Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                ‏2012-10-29T02:19:37Z  in response to ScottAmbler
                 Scott, my only problem with the acceleration metric is with its interpretation or misinterpretation.
                I would be interested in hearing your thoughts on the possible interpretation of this metric and what are the drawback?  Is higher acceleration always a good thing? or else? Please expand on how this can be analyzed ( as a single metric) or in combination of other metrics such as defects, technical debt....
                • M_Kevin_McHugh
                  M_Kevin_McHugh
                  22 Posts
                  ACCEPTED ANSWER

                  Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                  ‏2012-10-29T02:51:11Z  in response to Cherifa
                   If Acceleration is measured without a counter metric, the Team might (will) inflate their numbers in order to meet the measurement.
                   
                  Danger!!! Will Robinson!!! Danger!!!
                  • Reed_Fegg
                    Reed_Fegg
                    14 Posts
                    ACCEPTED ANSWER

                    Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                    ‏2012-10-29T16:30:52Z  in response to M_Kevin_McHugh
                     One counter metric I recommend to be used with Velocity is "Defects" or better yet "Technical Debt". Just posted a couple blogs and one was focused on "Why Agile Teams collect metrics?" where I go into a little more detail.
                     
                    Click here. 
                    Plan on sharing some of the metrics used by Agile CoE monitoring adoption as well in an upcoming blog.
                • ScottAmbler
                  ScottAmbler
                  24 Posts
                  ACCEPTED ANSWER

                  Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                  ‏2012-10-29T15:25:13Z  in response to Cherifa
                  Fundamentally there are always several problems with metrics:
                  1. They can be misinterpreted.  Education and experience are great ways to counter this challenge.
                  2. They can be gamed.  Acceleration is a simple calculation based on velocity, so you'd need to be gaming your velocity to game acceleration.  This happens in practice unfortunately.  Best way to counter this problem is for teams to not share their velocity outside of the team itself.  This is easier said than done if the team has adopted an information radiator strategy within itself to communicate effectively.  If the team believes that they're not being judged on their velocity they're less likely to game it. 
                  3. Scalar values are problematic.  It is far better to look at trends than singular values in most cases.
                  4. Single metrics are problematic.  As Kevin implies you need to look at several metrics, not just a single one, to get an effective picture as to what's going on.
                      Acceleration, like other metrics, suffers from these challenges.  So act accordingly.
                • DougStewart
                  DougStewart
                  10 Posts
                  ACCEPTED ANSWER

                  Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                  ‏2012-10-29T17:43:16Z  in response to Cherifa
                   Team evolution can can give misleading results also.  A team in the Storming stage will see things as more complex than a team that has matured to Performing.  Assuming the team moves from Storming to Norming to Performing (to High Performing with luck), acceleration my appear to be negative even though the team is obviously more productive from other measures.
                  • GustavoGrillo
                    GustavoGrillo
                    8 Posts
                    ACCEPTED ANSWER

                    Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                    ‏2012-10-29T18:26:59Z  in response to DougStewart
                     That's some great insight! Something that sounds counter-intuitive, but is very much present in every agile transformation.
              • M_Kevin_McHugh
                M_Kevin_McHugh
                22 Posts
                ACCEPTED ANSWER

                Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                ‏2012-10-29T02:44:31Z  in response to ScottAmbler
                 Scott, I think there's room for further application of physics terms.  In this context, Acceleration works.
                 
                But, I think that Velocity is actually a misnomer.  Agile "Velocity" when based on Complexity is a measure of the ability to do work.  This implies that it is more like Horsepower or Torque than Velocity.
                 
                But, since, I don't think the term will change, let's leave that aside.  Acceleration is a great example of an extended metric.  It is also wide open for problems of negative behavior based on the thing being measured.  And, frankly so is Velocity.
                 
                I actually expect to see either Acceleration as the Team becomes more capable with the domain.  Or, I expect that the team will lower their Complexity estimates on stories that are now "easier".  If the Complexity lowers, then we do not see Acceleration.  If Complexity stays the same, then the upper end of the story point range gets pushed up over time.  This would reflect Acceleration.  However, given some of the techinques for estimating Complexity, this "breaks" the original range of the story points.
                 
                This looks to me like a fundamental problem with velocity in the mid-term use.  However, I believe that if the Team is allowed to evolve, the range will eventually settle.  I would also prefer the lowering of Complexity instead of inflation.  But, this means Acceleration will not be evident as easily.
                 
                 
                @Scott - if we see Acceleration, we should see an upward trend of the Velocity chart.  And, if we kick off an initiative (like adding Agile Modeling to the Team mix) we should see an uptick on the Average Velocity as well as the Upper and Lower Control Limits.
                 
                More, variance is expensive.  So, in addition to seeing the impact of Acceleration as an up swing on the average line, we also want to see the upper and lower control limits tighten.
                 
                Velocity give predictability.
                 
                @Scott
                So as you said before in talking about the "projection cone" with Velocity, we can get very very accurate information on how a team will perform as we project into the future.
                 
                @Jim
                This essentially gives you a means to see the uncertainty.


                 
                • ScottAmbler
                  ScottAmbler
                  24 Posts
                  ACCEPTED ANSWER

                  Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                  ‏2012-10-29T15:27:52Z  in response to M_Kevin_McHugh
                  Yes, a positive value in acceleration co-relates to an improvement in velocity.  However, a negative value co-relates to lower velocity.
              • Reed_Fegg
                Reed_Fegg
                14 Posts
                ACCEPTED ANSWER

                Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                ‏2012-10-29T16:37:34Z  in response to ScottAmbler
                 Scott,
                 
                With regards to acceleration. I haven't been able to narrow acceleration down to a specific practice or person while coaching so would be interested in hearing more on that topic.
                • ScottAmbler
                  ScottAmbler
                  24 Posts
                  ACCEPTED ANSWER

                  Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                  ‏2012-10-29T16:58:13Z  in response to Reed_Fegg
                  Acceleration is a reflection of the entire team's ability, or inability as the case may be, to improve the way that they work.  So it will very likely be that many things are simultaneously motivating changes in productivity, hence the difficulty to tease out just a single influence.
                  • MikeORourke
                    MikeORourke
                    10 Posts
                    ACCEPTED ANSWER

                    Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                    ‏2012-11-01T19:00:21Z  in response to ScottAmbler
                     Agree with Scott here... and the key area here is the word ENTIRE!. To me acceleration can be measured in the sense that the ability to take a change request from a customer and measure how long it takes to get that change requestt hrough to the production system.  Even if its a single line of code. Most organizations have no clue how long this takes them and then try to become Agile. If that cycle for your organization is 2 months, then having a 4 week sprint makes no sense. You need to accelerate your rate of change throughout the ENTIRE system (NOT JUST DEVELOPMENT) in order to get the foundational charateristics for Agile.
                    • Cherifa
                      Cherifa
                      30 Posts
                      ACCEPTED ANSWER

                      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                      ‏2012-11-01T19:07:45Z  in response to MikeORourke
                       Mike, this is in line with what I posted this morning in the other thread (metrics at the enterprise level)...and related to the aging of the items in the backlog. Unfortunately we are not always in a new development environment but most of the time... the backlog contains change requests, defects...and so on...so my point : to measure the aging of work items and the average time needed for a WI to be processed and delivered is important in my sense
                      • DougStewart
                        DougStewart
                        10 Posts
                        ACCEPTED ANSWER

                        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                        ‏2012-11-02T00:04:59Z  in response to Cherifa
                         this does remind me of an old practice we had many years ago, that of escalating priority based on aging.  A minor bug from a customer might take on a higher priority than its severity level would indicate if it had aged for some period of time. Has anyone seen this practice used recently?  Of course this is already at the discretion of the PO.
                        • ScottAmbler
                          ScottAmbler
                          24 Posts
                          ACCEPTED ANSWER

                          Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

                          ‏2012-11-02T00:36:14Z  in response to DougStewart
                           In Disciplined Agile Delivery we promote several ways to prioritize work, not just Scrum's value driven strategy. With a leaner approach you might choose to consider aging issues.  
      • M_Kevin_McHugh
        M_Kevin_McHugh
        22 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-31T03:04:04Z  in response to Cherifa
         Control charts....
         
        Reduction of variation
         
        Movement of the mean in the desired direction
         
        Reaction to new tools and process
         
        yummy
  • M_Kevin_McHugh
    M_Kevin_McHugh
    22 Posts
    ACCEPTED ANSWER

    Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

    ‏2012-10-26T04:25:41Z  in response to richardknaster
     I am interested in hearing about Whole Team and Adoption metrics.
     
    I can measure skill up take.  But, the progression of Whole Team adoption seems softer to me.  I know Agile coaches to go at Whole Team as their primary coaching vector.  I take an Iterative Development primary vector with a follow up to Whole Team.
     
    My Whole Team sensing is:
    Smiles and laughs in the ceremonies (retrospectives, daily scrum ...)
    Good impediment flow and servant leadership
    Lack of formal boundaries on who can do what (formalized templates for work items worry me in this space).
       - but note, this is a negative measurement.
    • BethFriday
      BethFriday
      1 Post
      ACCEPTED ANSWER

      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

      ‏2012-10-29T00:46:09Z  in response to M_Kevin_McHugh
      Great Conversation.  Many of the clients that I speak with want to understanding how to capture and quantify errors that escape in production code.  These failures span issues with both functional and non-functional requirements.  What advice do you give to clients to make incremental progress against these kinds of costly errors ??
      • ScottAmbler
        ScottAmbler
        24 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-29T15:16:44Z  in response to BethFriday
        Funny you should ask this as I recently wrote detailed articles about addressing non-functional requirements (NFRs) on disciplined agile projects.  The first article, Strategies for Implementing Non-Functional Requirements goes through the techniques required to properly address NFRs throughout a disciplined agile lifecycle.  I believe the primary reason why NFR-related defects get into production in the first place is because methods such as Scrum and XP have very little to say about NFR activities during development.  The second article, Strategies for Verifying Non-Functional Requirements, overviews disciplined agile testing and quality practices for catching NFR-related defects before they get into production.  So, the best advice I can give is to bake quality right into your process from the very beginning and not ship buggy solutions to begin with. 
         
        If you take a few minutes to read those articles I have no doubt that they provide ample insights into potential improvements that they can make to their agile process.   First step, and potentially hardest, is likely for them to start thinking outside of the Scrum box and start moving towards a Disciplined Agile Delivery (DAD) approach.
        • Cherifa
          Cherifa
          30 Posts
          ACCEPTED ANSWER

          Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

          ‏2012-10-29T16:13:39Z  in response to ScottAmbler
           Beth, errors/defects/ bugs will slip out for so many reasons:
          1. The PO decides ( and in accordance with his/her "Doneness" criteria ) to deliver/ship a product with such a number of defects. Mind you, I do not think the PO is foolish to ship a product with high severity defects.
          2. Team missed or failed to deploy some of the agile practices and using a "test first approach" that help track errors earlier and before they can escape. (as listed by Scott..those are about testing early, doing some ACDD , CI, having short iterations to see the effect of any change/collecting feedback early, engaging an independent testing team as early as possible, doing automated regression testing and so on...)
          Running Tested Features (RTF), defects rates, defects trends (and other metrics)  being tracked and monitored on a regular basis can help catch some of the bugs you mentioned.
           
          Updated on 2012-10-29T16:13:39Z at 2012-10-29T16:13:39Z by Cherifa
    • Reed_Fegg
      Reed_Fegg
      14 Posts
      ACCEPTED ANSWER

      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

      ‏2012-10-29T16:39:17Z  in response to M_Kevin_McHugh
      From what I have seen, most teams reach a steady state with regards to "real skills growth" after 3 or 4 construction sprints but before that they go through learning pains as they learn better estimate.   Would be interested in hearing the round tables thoughts on this?
  • MariaEricsson
    MariaEricsson
    2 Posts
    ACCEPTED ANSWER

    Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

    ‏2012-10-29T13:54:32Z  in response to richardknaster
     
    I have had several very interesting discussions with clients lately on the good and bad of metrics for agile.
    For example, I have been directed to Craig Larman and Bas Vodde's work on scaling lean and agile, where they quite strongly argue the negative effects of metrics and talk about how an organization that steers through metrics is a command and control culture which is in conflict with agile values and self-organizing teams. 
    So - coming at it from that angle, is there any value in metrics? Maybe, if we think of metrics for the purpose of providing more transparency to what is going on, so that teams can be more informed in how the decide to self-organize ... thoughts?
    • WalkerRoyce
      WalkerRoyce
      18 Posts
      ACCEPTED ANSWER

      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

      ‏2012-10-29T15:09:56Z  in response to MariaEricsson
      Hmm,
      "... an organization that steers through metrics is a command and control culture which is in conflict with agile values and self-organizing teams"
       
      Sorry. I find this statement to be a non sequitur on two fronts. 1) steering through metrics implies command and control, and 2) steering through measures is in conflict with agile values.
      For sure, many cultures DO use metrics to inflict a command and control culture onto a creative team, but that does not have to be so. And what agile values are in conflict with measurement? Measuring code and test baseline changes, build times and change trends are explicitly the target of the agile manifesto's statement that "We value working software over comprehensive documentation"
      If software development is a sandbox, it needs no measure, but if it is part of a business, it must make forecasts, and it must deliver measured outcomes--or nobody will invest in it. 
      • ScottAmbler
        ScottAmbler
        24 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-29T15:34:41Z  in response to WalkerRoyce
        I agree with Walker.  I suspect what's happened is that Craig and Bas's experience is limited to a small number of environments so they haven't had the opportunity to see metrics applied successfully in agile environments.  
         
        Their argument is reminicent of the "I've only seen big dysfunctional requirements documents therefore documentation must be a really bad idea" or the "I've seen teams waste a lot of time developing detailed architecture models therefore modeling is a really bad idea" lines of thought we still see amongst some "agilists".   A bit more experience will hopefully bring greater insight their way.
      • Reed_Fegg
        Reed_Fegg
        14 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-29T16:25:39Z  in response to WalkerRoyce
        I agree with Scott and Walker this seems unusual but collecting metrics   may depend on management's ability to not use metrics incorrectly and definitely requires more coaching especially at the mid and executive levels.  If a team can't use "velocity" to assess their ability to deliver, then as Scott has said "it will be gamed". Actually have first hand experience where a team learned they were being compared to another team, something that taboo in agile, and suddenly their velocity doubled the next sprint with no other changes.  Hard to explain..... 
      • Cherifa
        Cherifa
        30 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-10-29T16:38:05Z  in response to WalkerRoyce
         Great insight Walker!!!
        "Metrics do not manage...people do"...( can't remember who said it!). One thing we want to avoid is using the "metrics" for the wrong reasons!
        Walker, you mentioned something of interest: Forecasting, measuring outcomes and making timely decisions and improve based on those outcomes! Those are the main reasons why we should be collecting metrics.
         
    • Reed_Fegg
      Reed_Fegg
      14 Posts
      ACCEPTED ANSWER

      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

      ‏2012-10-29T16:19:09Z  in response to MariaEricsson
      Metrics serve many purposes and have many owners and depending on who uses the metrics. The answer can't be "we can't collect metrics". No technical or business transformation can be successful without being able to "learn and adapt" based on real feedback. 
       
      Have had similar discussions over the last couple years, while helping several with agile transformations efforts. During this time have had to group metrics by three distinct groups
      • Delivery Team 
      • Agile CoE (adoption team)
      • Management
       Some of the common concerns / questions that I have seen from these groups are
       
       Agile Delivery Team
      • Are we on track to deliver what we committed to (Iteration Burn down)?
      • How much value have we delivered to business? (Release Burndown)
      • How much work can we commit to (Velocity)?
      Agile Coe (Agile Adoption Team)
      • To assess the effectiveness of the CLM Development practices being adopted
      • What other areas need improvement (e.g., improve estimating process)?
      • Do we have any roadblocks to our adoption impediments (skills, resources, training)?

      Management Team

      • Will the project be a success (triple constraints)
      • Are we making the best use of our resources
      • Can we show the business that CLM is helping us to be better, faster, and cheap 

      Each may use the same metric but for different purposes. The problem with most of the metrics is they can be "gamed" if used incorrectly. For example, "velocity" during several engagements I have had to use "acceleration" to help track progress but also had to coach management that "positive/negative" acceleration isn't a performance measurement tool especially when the team is new to agile.  We used acceleration to help understand if your "learn and adapt" activities were helping the team improve and were used inconjunction with the "Team Retrospective" to make sure we weren't interpreting a velocity/acceleration to some "one time" event that happened on the team, like a new team member or build up of technical debt.

  • SmartDJ
    SmartDJ
    4 Posts
    ACCEPTED ANSWER

    Question:How can we measure the business value achieved since Agile always focus on continuous value delivery

    ‏2012-10-30T18:51:42Z  in response to richardknaster
     Question:How can we measure the business value achieved since Agile always focus on continuous value delivery , do we use business value estimation for each user story or ??
     
    Please join the breakout discussion on this topic.
    Updated on 2012-10-30T18:51:42Z at 2012-10-30T18:51:42Z by richardknaster
    • AlexXIE
      AlexXIE
      8 Posts
      ACCEPTED ANSWER

      Re: Question:How can we measure the business value achieved since Agile always focus on continuous value delivery

      ‏2012-10-30T02:46:30Z  in response to SmartDJ
       How about rely on the product owner/manager to give every iterative feature a score, the higher the more business value.
      • JimDensmore
        JimDensmore
        24 Posts
        ACCEPTED ANSWER

        Re: Question:How can we measure the business value achieved since Agile always focus on continuous value delivery

        ‏2012-10-30T19:21:49Z  in response to AlexXIE
        That score would tend to be subjective.  Using money as the scoring factor is less subjective.  Allowing uncertainty yields more honest estimates and a more proactive response to risk.  That said, it is not easy since business value lags and is associated with many more factors than the project at hand.  Since value lags, estimates are used.  This is possible because we allow uncertainty, otherwise no one would be comfortable being accountable for the estimates.  Even so, it is not until we begin to gain experience with actual results that we can begin to calibrate the estimators and their estimates.  As the calibrations take hold, business accountability can begin.
    • richardknaster
      richardknaster
      8 Posts
      ACCEPTED ANSWER

      Re: Question:How can we measure the business value achieved since Agile always focus on continuous value delivery

      ‏2012-10-30T04:12:00Z  in response to SmartDJ
       Hi DJ:
       
      This is a very complex question and depends upon the reasons for measuring business value.
      1. Are you trying to prioritize the features that are being delivered?  Consider using Business Value points.  Business Value points can be used to measure the relative value of User Stories, features, epics, etc..  Value Points can help you to prioritize a product backlog and convey the relative importance of User Stories to the Development Team.  The issue with Value points is that they can't be easily translated into financial data, nor can you compare my business value points to your team's business value points.  The MoSCoW prioritization technique is also a good way to align stakeholders and the Development Team on the importance of each feature.  There are definitely other techniques to aid in setting priority that may sometimes be referred to as value.  You can track the number of clients asking for this feature or experiencing this problem, number of won/lost deals because of this item, number of prospects interested in this release or feature set (epic, or theme), etc.
      2. Do you want to measure the return on the investment from the continuous delivery of the software?  It is not feasible to to measure the business value of an individual Feature or at even more granular level at the User Story level. The problem is we don't sell individual features or user stories, we sell a product or service.  Therefore, we need to measure the business value at the level of which the product or service is being sold.  We can look at quarterly revenue trends and also chart the delivery of our software to the market and see if there is a correlation to between  the two.  We can do win/loss revenue analysis and conduct Customer surveys to understand what prompted the purchase decision and also track customer satisfaction trends.  Other measures could be considered as well, such as the improvement in quality and the reduction of production issues (e.g downtime, etc.) which could also be seen as value.

      This one question could be an entire roundtable.  Anyone else have some quick thoughts on DJ's question?

      Updated on 2012-10-30T04:12:00Z at 2012-10-30T04:12:00Z by richardknaster
      • Holitza
        Holitza
        7 Posts
        ACCEPTED ANSWER

        Re: Question:How can we measure the business value achieved since Agile always focus on continuous value delivery

        ‏2012-10-30T17:57:34Z  in response to richardknaster
         I think the most feasible way to measure customer value on a regular basis is to use lightweight surveys like Rich mentioned and another way might be to incorporate ratings of some kind.  Similar to how we solicit ratings for our products on ibm.com (see http://www-01.ibm.com/software/rational/products/rtc/) or how Apple does it in the App store.  These can be easy to maintain and provide more continuous feedback on business value.  These should also be combined with the other metrics like sales and win/loss data to make a more comprehensive evaluation of business value.
      • JimDensmore
        JimDensmore
        24 Posts
        ACCEPTED ANSWER

        Re: Question:How can we measure the business value achieved since Agile always focus on continuous value delivery

        ‏2012-10-30T19:26:21Z  in response to richardknaster
        Richard:
        1. I suggest (and I think Kevin would suggest) using ROI not value, see my post above.
        2. Business value of an individual feature and/or user story can be estimated, and that IS a measure.  I think what you are saying is that we cannot measure it with 100% confidence; with that I agree.  Focal Point's most important use cases are associated with evaluating (including business value) each feature in a portfolio of features in order to prioritize them
        Yes.  Could be a whole roundtable ...
        • GustavoGrillo
          GustavoGrillo
          8 Posts
          ACCEPTED ANSWER

          Re: Question:How can we measure the business value achieved since Agile always focus on continuous value delivery

          ‏2012-10-30T19:47:30Z  in response to JimDensmore
           I agree with using ROI, but I think it is important to make this decision with the product owner, and it must be something he/she is comfortable with. This metric must be his/her view of what it is value.
          As in any agile practice, the chosen metric must be continually reviewed, in order to assess and calibrate its validity.
  • MikeORourke
    MikeORourke
    10 Posts
    ACCEPTED ANSWER

    Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

    ‏2012-11-01T19:17:44Z  in response to richardknaster
     I'd like us to think about not just "metrics" in the traditional sense, but metrics that help teams do the right thing. I'll give you a for instance. Ine of the areas we see constantly is the fact that while the development team is doing some form of agile development, the product management and testing teams aren't, so the benefits are poor overall. One thing we use effectively is the notion of "what is done". In other words, we "force" the product manager, developer and tester to define on a work item by work item basis, what is done BEFORE they start coding or testing. This creates a level of team communication up front and far less rework on the back end.  Technically, its not a measurement, but a process that enforces good behavior. Aren't these types of things just as important in terms of ensuring the right things are happening on the teams?????
    • M_Kevin_McHugh
      M_Kevin_McHugh
      22 Posts
      ACCEPTED ANSWER

      Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

      ‏2012-11-01T19:48:27Z  in response to MikeORourke
       Mike,
          I need you to clarify.  Let me take a couple shots at what I think I am hearing to help with that.
       
          Are you asking for measures/metrics regarding adherence to Definition of Done?  This strikes me as an quality issue to be handled via defects.  But, it also seems to me that it's not what you are asking.  Or maybe you are. 
          It is also sort of what I was asking about for Whole Team.  How to measure the behaviors we want.  Maybe the counter example will make sense (below)
       
      OR
       
          I've certainly seen teams and their management structure working against each other.  For instance, one company had 5 layers of management for Test and 5 layers of management for Requirements and 5 layers of management for Development.  So, it took 6 levels of escalation to reach a common manager.  The result (one of them) was that, for instance, the Test management structure asked for "Tester burndown"
       
         We spent a lot of time trying to put together a report to help with this.  And, there were other cases.
       
         Metric: How many waterfall reports do we have.  Plot this over time.
       
      ## At the risk of inciting a riot, a measure is a single point.  A metric is a set of measures (I like to use time as the other axis).    :-)
       
      • MikeORourke
        MikeORourke
        10 Posts
        ACCEPTED ANSWER

        Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

        ‏2012-11-01T19:55:37Z  in response to M_Kevin_McHugh
         Hey Kevin... its mostly around the first (although the second is interesting). My point is that while not specifically a metric, having teams be "forced" to define what is done upfront, causes behavior patterns that are good for the team and good for management (causes less waste and rework, etc). I think sometimes when we think of metrics and measruements we think in black and white terms, but measurements are mostly about getting data so you can react and modify. The "definition of done" isn't a measurement per se, but will cause many other measurements to improve. How do we capture such things as that?
        • M_Kevin_McHugh
          M_Kevin_McHugh
          22 Posts
          ACCEPTED ANSWER

          Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

          ‏2012-11-02T00:31:30Z  in response to MikeORourke
           A couple thoughts... hang with me...
           1st - I would like Jim Densmore to weigh in on this question from the view point of Asset Governance. In my opinion, Agile is leans heavily in the Asset Governance direction (as opposed to Process Governance).  Jim has spent more time thinking in that space.  I still tend to think reflexively in the Process Governance space.
           
          2nd - I observed and worked with a team that found that that they needed a Definition of Ready.  That's right, Definition of Ready, not Done.  In this particular case, they were struggling to get the impact and definition of services to a Services team.  The service consumers were getting services that did not match their needs.  Thus, there was a lot of swirl and rework.  They measured the impact of their DoR and DoD by how the rework/swirl dropped off.
           
          Remember that Agile is not only about the ability to change direction frequently but also about Lean - avoiding Non-required waste (sorry to Lean experts to trim down the Lean space so far).... :-)
           
          So, this company split into three teams.  Services, and two application teams that use the services.  They found that they struggled to create what we called Services Stories (not user stories) that were clear.
           
          The situation improved when the DoR went into effect.  It was a bit more waterfall-ish than I wanted.  It was more waterfall-ish than they wanted.  But, at a time when they were learning this new SOA thing, it was a reasonable approach in light of Agile and Lean.
           
          What they saw as a result was increased..........wait for it.......
           
          Velocity from all three teams.    Yeah!!! Velocity!!!!
           
          They were disciplined about not taking credit for stories that were not complete.  So, as quality rose, they could take more credit for stories.
           
          If they wanted to, they could have measured rework of stories or defects.
           
          Thoughts?
  • Anthony_Kesterton
    Anthony_Kesterton
    1 Post
    ACCEPTED ANSWER

    Re: Virtual Roundtable - Agile Metrics (25 October to 02 November)

    ‏2012-11-01T21:59:23Z  in response to richardknaster
     Seen some really interesting discussion here - very useful.  I do think we are sliding away from the first few questions - and suggest the following as an answer to "What are the most important metrics to help Agile Teams to improve?"
     
    1) Burndown (time based) - as an indication of estimated time remaining and flagging new issues when it jumps back up (or down) - allowing teams to see a form of progress towards final delivery.
    2) Velocity (story points for that specific team) - how well the team is doing at both estimating complexity and converting that complexity to something that can be delivered
     
    Another measure might be defects found in the code once it has been delivered in the sprint (and any subsequent errors found in production).  This might be useful to examine as a total, and then perhaps to categorize as "wrong thing", "built wrong" or "used wrong".
     
    anthony