Topic
  • 31 replies
  • Latest Post - ‏2012-11-02T14:20:10Z by Reedy Feggins
WalkerRoyce
WalkerRoyce
18 Posts

Pinned topic Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures [Event closed]

‏2012-10-30T14:54:43Z |
**Even though this event is over, you are welcome to browse this discussion**
 
Let's break out into a different thread for discussion around enterprise measures. Productivity, Quality, Time-to-value and Predictability are four distinct measurement areas to consider.
 
Let me start with a simple assertion. 
Project or team level metrics tend to be centered on looking at changes from release to release. Defects, change speed, change cost, and changes in the variance on the estimate to complete are good examples. The object of measurement is essentially the code/test base and the subject of measurement is largely the same team. Even though the code and team may be dynamically changing over time, they represent a relatively consistent context for measurement.
 
Now let's switch to an enterprise context. We have already seen warnings about comparing metrics across projects. Why? Largely because their is not a comparable basis for comparison--namely the code/test base on different projects and the team makeup on different projects can provide a wildly incomparable context.
 
What do we do? How do we normalize so that we can manage our business against reasonable norms?
 
 You will need to JOIN this group to reply to this topic.
 
You can return to the Welcome notice, go to the main agile metrics topic, join one of our breakout topics on metrics transparency and misuse, or business value achieved or discuss future topics
Updated on 2013-01-07T22:34:08Z at 2013-01-07T22:34:08Z by sjpeich
  • HazelWoodcock
    HazelWoodcock
    10 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-30T18:59:39Z  
    Walker, these four areas are distinct, but seem to be very tightly coupled.  Without predictability it is hard to achieve any of the other three.  Time to value, productivity and quality all belong together in an equation alongside a magic constant.  I can't argue against these being useful at an enterprise level, but they are also valuable at a project level.  With historical metrics from previous projects to compare against, these will also point to projects getting into trouble before they are dangerously out of control. 
    Perhaps these are universally useful measures, but they are not simple to calculate.  How many individual numbers do we need to collect to get a meaningful view of these, and are the things being measured transferable between organizations?  Is there a transferable formula we can create that can be used across organizations to indicate performance in each area?  Can the complexity of the project be taken into account in the numbers?  If we produce a transferable formula, do we then have the issue of senior managers with an incentive to game the metrics as well as developers?
  • Cherifa
    Cherifa
    30 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-30T20:04:22Z  
    Walker, these four areas are distinct, but seem to be very tightly coupled.  Without predictability it is hard to achieve any of the other three.  Time to value, productivity and quality all belong together in an equation alongside a magic constant.  I can't argue against these being useful at an enterprise level, but they are also valuable at a project level.  With historical metrics from previous projects to compare against, these will also point to projects getting into trouble before they are dangerously out of control. 
    Perhaps these are universally useful measures, but they are not simple to calculate.  How many individual numbers do we need to collect to get a meaningful view of these, and are the things being measured transferable between organizations?  Is there a transferable formula we can create that can be used across organizations to indicate performance in each area?  Can the complexity of the project be taken into account in the numbers?  If we produce a transferable formula, do we then have the issue of senior managers with an incentive to game the metrics as well as developers?
     Walker,  normalization is a goodness for benchmarking!! To have a comprehensive list of metrics at the enterprise level that can be normalized by industry/sector (banking, insurance, distribution, mobile/telco, etc..) can be a great tool for benchmarking and comparing one organization with another. Any thoughts ?
  • GustavoGrillo
    GustavoGrillo
    8 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-30T20:11:42Z  
    Walker, these four areas are distinct, but seem to be very tightly coupled.  Without predictability it is hard to achieve any of the other three.  Time to value, productivity and quality all belong together in an equation alongside a magic constant.  I can't argue against these being useful at an enterprise level, but they are also valuable at a project level.  With historical metrics from previous projects to compare against, these will also point to projects getting into trouble before they are dangerously out of control. 
    Perhaps these are universally useful measures, but they are not simple to calculate.  How many individual numbers do we need to collect to get a meaningful view of these, and are the things being measured transferable between organizations?  Is there a transferable formula we can create that can be used across organizations to indicate performance in each area?  Can the complexity of the project be taken into account in the numbers?  If we produce a transferable formula, do we then have the issue of senior managers with an incentive to game the metrics as well as developers?
     I like to look at these metrics from outside the "IT department".
    Why is Productivity an important metric? It may seen obvious but in a agile context it is healthy to ask this, if only to check that we're not doing things "because we've always done that way".
    Quality is one metric that might really be useful if seen from the business's or customer's point of view. Some features might not be as important to the customer as others, so they may be of lower quality in technical terms.
    Time-to-value could be assessed on the long run by checking if the organization is delivering faster than its competitors or at the right pace for the market/customers.
    For Predicability I would ask the same question that I asked for Productivity. If you delivering at the market's pace, does it matter if you're on schedule? Two weeks early? A month late?
    I know these are more questions than answers, but I think it's healthy to ask "Why we're doing this, anyway?" sometimes  
  • WalkerRoyce
    WalkerRoyce
    18 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-30T20:27:17Z  
    • Cherifa
    • ‏2012-10-30T20:04:22Z
     Walker,  normalization is a goodness for benchmarking!! To have a comprehensive list of metrics at the enterprise level that can be normalized by industry/sector (banking, insurance, distribution, mobile/telco, etc..) can be a great tool for benchmarking and comparing one organization with another. Any thoughts ?
    I don't see obvious ways of normalizing across industries.
     
    I see one simple way of normalizing that avoids the direct cross-project comparisons. Various teams will measure themselves differently. But however they measure, we can hold them to measured improvements that are roughly equal in terms of improvement over time. Most big organizations demand a 5% improvement year over year. In big macro terms, this is a fair thing to do. However you measure, we want you to improve that measure by a certain percentage. Note that an immature organization has a purely speculative basis for setting such a target. A mature organization with a strong foundation of analytic experience can provide a credible case for improvement that can be negotiated rapidly between governors, team leaders and practitioners as a shared, achievable target.
     
    Now, one can provide unfair counter examples. If one team has a a zero-defect track record and another has an abysmal quality record, applying 5% improvement to both teams is a context-independent mistake. However, such counter examples occur rarely in our business and smart people can adjust.
  • WalkerRoyce
    WalkerRoyce
    18 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-30T20:35:53Z  
     I like to look at these metrics from outside the "IT department".
    Why is Productivity an important metric? It may seen obvious but in a agile context it is healthy to ask this, if only to check that we're not doing things "because we've always done that way".
    Quality is one metric that might really be useful if seen from the business's or customer's point of view. Some features might not be as important to the customer as others, so they may be of lower quality in technical terms.
    Time-to-value could be assessed on the long run by checking if the organization is delivering faster than its competitors or at the right pace for the market/customers.
    For Predicability I would ask the same question that I asked for Productivity. If you delivering at the market's pace, does it matter if you're on schedule? Two weeks early? A month late?
    I know these are more questions than answers, but I think it's healthy to ask "Why we're doing this, anyway?" sometimes  
     We are all being asked to do more with less. That is why some measure of productivity is needed. How much more and how much less constrains our plans, targets and morale. If we don't know how productive we are now, it is difficult to convince people we can do better.
     
    Quality is directly correlated with your outside-in reputation. If you want to sell anything, your reputation for quality is a key parameter for persuading clients.
     
    Time-to-value is, in my view, the measure that is most correlated with agility. Why are we all so enamored with agility? Because we all know that the quicker we can make "value" tangible in development, the quicker and better we can translate it into business outcomes.
     
    Predictability is another metric correlated to reputation. If you don't plan and execute predictably, you will have a harder time selling your investors on the funding, selling your teams on the plans and establishing trust among everyone.
     
    My simple answers to your very insightful questions.
     
    Updated on 2012-10-30T20:35:53Z at 2012-10-30T20:35:53Z by WalkerRoyce
  • GustavoGrillo
    GustavoGrillo
    8 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-31T01:54:26Z  
     We are all being asked to do more with less. That is why some measure of productivity is needed. How much more and how much less constrains our plans, targets and morale. If we don't know how productive we are now, it is difficult to convince people we can do better.
     
    Quality is directly correlated with your outside-in reputation. If you want to sell anything, your reputation for quality is a key parameter for persuading clients.
     
    Time-to-value is, in my view, the measure that is most correlated with agility. Why are we all so enamored with agility? Because we all know that the quicker we can make "value" tangible in development, the quicker and better we can translate it into business outcomes.
     
    Predictability is another metric correlated to reputation. If you don't plan and execute predictably, you will have a harder time selling your investors on the funding, selling your teams on the plans and establishing trust among everyone.
     
    My simple answers to your very insightful questions.
     
    Thanks Walker,
    I agree with you about Quality and Time-to-value.
    I have problems with Productivity and Predictability because I've never seen them being used in a way that was effective in an enterprise environment.
    For example, I've seen Function Points being used for calculating Productivity (Function Points per week). What happened is that people began inflating the FP calculations, so Productivity increased.
    I've also seen more than my share of buffers in schedules, beyond reasonable risk-management, just to give a false impression of predictability. So, if you estimate 3 times more hours for every task on your project you will be pretty much on schedule, no matter what.
    I obviously chose extreme examples, but that's my experience.
    Is there a way to have a Productivity/Predictability metric that is scroundrel-proof?
    Am I doomed to misbehaving projects? 
  • WalkerRoyce
    WalkerRoyce
    18 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-31T10:36:45Z  
    Thanks Walker,
    I agree with you about Quality and Time-to-value.
    I have problems with Productivity and Predictability because I've never seen them being used in a way that was effective in an enterprise environment.
    For example, I've seen Function Points being used for calculating Productivity (Function Points per week). What happened is that people began inflating the FP calculations, so Productivity increased.
    I've also seen more than my share of buffers in schedules, beyond reasonable risk-management, just to give a false impression of predictability. So, if you estimate 3 times more hours for every task on your project you will be pretty much on schedule, no matter what.
    I obviously chose extreme examples, but that's my experience.
    Is there a way to have a Productivity/Predictability metric that is scroundrel-proof?
    Am I doomed to misbehaving projects? 
    No doubt, that all of these can be gamed. Humans are subject to human nature.
    In a low trust environment, there will be more gaming. But there is an even bigger answer to achieving honest measures. People call this empowerment, but the term is so misused, I will use a different synonym: ownership.
    When a team "owns"  the results, they will be more honest because it is they who benefit.
    Owners of a home will treat their property better than renters of a home.Neighborhood watch programs work because neighborhoods have shared ownership in the security of their properties.
     
    So how do we get teams to own their targets?
    Mandates from above don't work. Team commitments to plans and incentives can work. One key is to recognize that many of the targets are interrelated. If we target only one dimension, the others may suffer. So good incentives are outcome based where the team can tradeoff performance across multiple dimensions.
    Again, this gets back to creating a high trust environment and shared ownership for goals.
     
    Is IBM good at this? Think about your local team. How much say do you have in your own team's performance destiny? 
  • GustavoGrillo
    GustavoGrillo
    8 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-31T12:35:28Z  
    No doubt, that all of these can be gamed. Humans are subject to human nature.
    In a low trust environment, there will be more gaming. But there is an even bigger answer to achieving honest measures. People call this empowerment, but the term is so misused, I will use a different synonym: ownership.
    When a team "owns"  the results, they will be more honest because it is they who benefit.
    Owners of a home will treat their property better than renters of a home.Neighborhood watch programs work because neighborhoods have shared ownership in the security of their properties.
     
    So how do we get teams to own their targets?
    Mandates from above don't work. Team commitments to plans and incentives can work. One key is to recognize that many of the targets are interrelated. If we target only one dimension, the others may suffer. So good incentives are outcome based where the team can tradeoff performance across multiple dimensions.
    Again, this gets back to creating a high trust environment and shared ownership for goals.
     
    Is IBM good at this? Think about your local team. How much say do you have in your own team's performance destiny? 
    That's exactly my point.
    It is difficult to create enterprise-level metrics because they are owned by the managers or CIOs, not by the team. You can't have effective metrics without an environment where trust can thrive.
    When we are recommending this or that metric in a big company this issues keep coming to haunt us and it's hard to come up with more metrics than working software and the PO's view of R.O.I.
  • WalkerRoyce
    WalkerRoyce
    18 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-31T13:06:30Z  
    That's exactly my point.
    It is difficult to create enterprise-level metrics because they are owned by the managers or CIOs, not by the team. You can't have effective metrics without an environment where trust can thrive.
    When we are recommending this or that metric in a big company this issues keep coming to haunt us and it's hard to come up with more metrics than working software and the PO's view of R.O.I.
     It may be difficult, but it is still important.
    CIOs and Execs that rule by fiat will not create a high trust environment and such measures will not produce great results. But it can be done. 
    It is obvious to most teams when hey have been listened to, when they are consulted on what the probability distribution of possible outcomes is likely to be, and when there is a negotiated target where all constituents understand the risks and rewards. We may not pick the easy path, but we pick a path where everyone understands the upsides and downsides--and everyone is communicating honestly about progress trends, quality trends and obstacles.
  • Reedy Feggins
    Reedy Feggins
    14 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-31T14:29:33Z  
     It may be difficult, but it is still important.
    CIOs and Execs that rule by fiat will not create a high trust environment and such measures will not produce great results. But it can be done. 
    It is obvious to most teams when hey have been listened to, when they are consulted on what the probability distribution of possible outcomes is likely to be, and when there is a negotiated target where all constituents understand the risks and rewards. We may not pick the easy path, but we pick a path where everyone understands the upsides and downsides--and everyone is communicating honestly about progress trends, quality trends and obstacles.
     Hi Folks, just catching up on this topic been running from Hurricane SANDY
     
    Read throught initial assertion and agree that most team metrics are team  based and mostly  focused on release to release but agile is all about "qualiy", "predictability" and customer satifaction.  Many of the executives I have worked with want measures similar to traditional projects, namely Earned Value.
  • Cherifa
    Cherifa
    30 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-31T14:43:18Z  
    I don't see obvious ways of normalizing across industries.
     
    I see one simple way of normalizing that avoids the direct cross-project comparisons. Various teams will measure themselves differently. But however they measure, we can hold them to measured improvements that are roughly equal in terms of improvement over time. Most big organizations demand a 5% improvement year over year. In big macro terms, this is a fair thing to do. However you measure, we want you to improve that measure by a certain percentage. Note that an immature organization has a purely speculative basis for setting such a target. A mature organization with a strong foundation of analytic experience can provide a credible case for improvement that can be negotiated rapidly between governors, team leaders and practitioners as a shared, achievable target.
     
    Now, one can provide unfair counter examples. If one team has a a zero-defect track record and another has an abysmal quality record, applying 5% improvement to both teams is a context-independent mistake. However, such counter examples occur rarely in our business and smart people can adjust.
     Good point....on the same line I would like your insight on what does it take to build this analytic experience when you are doing an Enterprise Agile transformation? What are the pre-requisite for this analytic foundation to be built progressively and that can help with the  transformation. I understand we need to focus on the three key areas: People, Process and Technology. Today we have invested on the T side (the tooling are more and more sophisticated..you can basically create any report you want...with improved ETL, data mining, etc.. capabilities..). My main concern (and this is related to this discussion topic), is having the right people with the right skills to understand what BI is needed for better decision making and how to go about deploying it. Agile metrics management (as any data management or other discipline) requires dedicated  and knowledgeable people ( at all levels of the org), in particular, at the IT level   such as  metric modelers, metric architects...etc to be able to architect and design the metric models required for an enterprise. Any thing that I am missing and that I am not aware of?
    Updated on 2012-10-31T14:43:18Z at 2012-10-31T14:43:18Z by Cherifa
  • HazelWoodcock
    HazelWoodcock
    10 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-31T18:34:17Z  
    • Cherifa
    • ‏2012-10-31T14:42:56Z
     Good point....on the same line I would like your insight on what does it take to build this analytic experience when you are doing an Enterprise Agile transformation? What are the pre-requisite for this analytic foundation to be built progressively and that can help with the  transformation. I understand we need to focus on the three key areas: People, Process and Technology. Today we have invested on the T side (the tooling are more and more sophisticated..you can basically create any report you want...with improved ETL, data mining, etc.. capabilities..). My main concern (and this is related to this discussion topic), is having the right people with the right skills to understand what BI is needed for better decision making and how to go about deploying it. Agile metrics management (as any data management or other discipline) requires dedicated  and knowledgeable people ( at all levels of the org), in particular, at the IT level   such as  metric modelers, metric architects...etc to be able to architect and design the metric models required for an enterprise. Any thing that I am missing and that I am not aware of?
     Cherifa, I completely agree with your concerns.  Metrics should not be gathered without a real understanding of the effect.  We really cannot measure without affecting what we are measuring.  Design of metrics is not to be taken lightly. There are some basic things to consider with any metrics collection.
    • Don't collect data you have no intention of using
    • Measure what you want more of (reduce the chance of gaming the metric)
    • Consider the effect on the things NOT being measured
    • Educate the audience on what the metric means and on what it DOESN'T mean
    • Decide what you will do with the information - what decisions will you take based on the metrics, what are your thresholds for action etc.
    • Abandon a metric that is not working
    A program of metrics, in any organization is often a damaging thing because the people designing it do not understand the wider effects of what they are doing.  You don't get your software team to design the building they sit in, you don't get your project managers to control the marketing effort.  As you said, dedicated and knowledgeable people are needed.
     
    At an enterprise level we probably want meta-metrics.  Metrics on metrics.  How many organizations do that? Are there any metrics on meta-metrics?  My head hurts now, so I will climb off my soapbox!
  • Reedy Feggins
    Reedy Feggins
    14 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-31T19:51:39Z  
     Cherifa, I completely agree with your concerns.  Metrics should not be gathered without a real understanding of the effect.  We really cannot measure without affecting what we are measuring.  Design of metrics is not to be taken lightly. There are some basic things to consider with any metrics collection.
    • Don't collect data you have no intention of using
    • Measure what you want more of (reduce the chance of gaming the metric)
    • Consider the effect on the things NOT being measured
    • Educate the audience on what the metric means and on what it DOESN'T mean
    • Decide what you will do with the information - what decisions will you take based on the metrics, what are your thresholds for action etc.
    • Abandon a metric that is not working
    A program of metrics, in any organization is often a damaging thing because the people designing it do not understand the wider effects of what they are doing.  You don't get your software team to design the building they sit in, you don't get your project managers to control the marketing effort.  As you said, dedicated and knowledgeable people are needed.
     
    At an enterprise level we probably want meta-metrics.  Metrics on metrics.  How many organizations do that? Are there any metrics on meta-metrics?  My head hurts now, so I will climb off my soapbox!
     Agree with concern that team can't feel threatened by metrics and that if so they will be gamed in some way. However most organization need some way understanding where they are against "the Plan" and of "predicting" their future outcomes. Project managers are typically asked for monthly updates and projects for this reason.  Its pretty hard to come up with meaningfull metrics and alot of traditional teams have useless metrics, like % percentage complete. Agile metrics have to be derived from metrics the team collects anyways (easy) and are meaning ful to them (things like burndown) but they may need to be normalized against the project own goals
  • WalkerRoyce
    WalkerRoyce
    18 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-11-01T13:34:39Z  
     Hi Folks, just catching up on this topic been running from Hurricane SANDY
     
    Read throught initial assertion and agree that most team metrics are team  based and mostly  focused on release to release but agile is all about "qualiy", "predictability" and customer satifaction.  Many of the executives I have worked with want measures similar to traditional projects, namely Earned Value.
     Earned value is a very reasonable metric. But there are very few software teams that have  honest usage models for reporting earned value. Most software projects using EVM report a monotonically increasing EV until....they start integrating, then the roof caves in and they hang around 80-90% complete for a long, long time while they try and shoehorn in a bunch of late fixes, corrupt the architecture and deliver a fragile and unmaintainable product or release.
       
    Honest earned value can be achieved.  You should see progressions and digressions in a healthy project. A digression in EV is what modern architects call "refactoring" where we can regress in perceived progress to correct design and requirements flaws. When progressions continue and subsequent digressions require less and less rework, the team can deliver better architectures, and more changeable systems, more predictably.
     
    Honest earned value is a good title for "best practices in Agile measurement"
  • Rob Suritis
    Rob Suritis
    2 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-11-01T16:01:08Z  
     Earned value is a very reasonable metric. But there are very few software teams that have  honest usage models for reporting earned value. Most software projects using EVM report a monotonically increasing EV until....they start integrating, then the roof caves in and they hang around 80-90% complete for a long, long time while they try and shoehorn in a bunch of late fixes, corrupt the architecture and deliver a fragile and unmaintainable product or release.
       
    Honest earned value can be achieved.  You should see progressions and digressions in a healthy project. A digression in EV is what modern architects call "refactoring" where we can regress in perceived progress to correct design and requirements flaws. When progressions continue and subsequent digressions require less and less rework, the team can deliver better architectures, and more changeable systems, more predictably.
     
    Honest earned value is a good title for "best practices in Agile measurement"

    I like "Honest Earned Value". Value is a core metric to add to Walker's initial list. Highly productive teams can still be working on low value capabilities from the business perspective. Enterprise level management may care less about Productivity or Time-to-value if the Value (thinking Innovation as well) delivered outweighs them.

    Earned "Value" (a misnomer) metrics are problematic because they are often implemented to measure the wrong thing - activity. So you have finished 80% of the activity, but not delivered the business value. The Planned/Earned Value part of the EVM equation needs to be changed to reflect business value of the delivered capabilities. This is a problem we have seen before e.g. Iterative development with federal/DoD projects and rigid implementation of EVM contractually tied to a WBS.

    How do we apply this to Agile? Product and Release backlog/burn down are a place to start but the difficulties start when trying to identify the unit of value measurement. $ Value seems the most universal to me, but it is always a struggle for enterprises, and even more so IT organizations, to tie what they are developing back to $ value. Perhaps we should talk about Earned Business Value.

    Well EBV exists... found the topic in Collabnet article "Monitoring Scrum Projects with AgileEVM and Earned Business Value Measures" http://www.open.collab.net/servlets/OCNDirector;jsessionid=87FFD7B67E4C0371FB108247CF9B2FD1?id=WP-AGILE-MS

    The article maps value to Story Points - I'm not happy with that as a measure of value. Thoughts?

    Updated on 2012-11-01T16:01:08Z at 2012-11-01T16:01:08Z by Rob Suritis
  • Cherifa
    Cherifa
    30 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-11-01T16:52:29Z  

    I like "Honest Earned Value". Value is a core metric to add to Walker's initial list. Highly productive teams can still be working on low value capabilities from the business perspective. Enterprise level management may care less about Productivity or Time-to-value if the Value (thinking Innovation as well) delivered outweighs them.

    Earned "Value" (a misnomer) metrics are problematic because they are often implemented to measure the wrong thing - activity. So you have finished 80% of the activity, but not delivered the business value. The Planned/Earned Value part of the EVM equation needs to be changed to reflect business value of the delivered capabilities. This is a problem we have seen before e.g. Iterative development with federal/DoD projects and rigid implementation of EVM contractually tied to a WBS.

    How do we apply this to Agile? Product and Release backlog/burn down are a place to start but the difficulties start when trying to identify the unit of value measurement. $ Value seems the most universal to me, but it is always a struggle for enterprises, and even more so IT organizations, to tie what they are developing back to $ value. Perhaps we should talk about Earned Business Value.

    Well EBV exists... found the topic in Collabnet article "Monitoring Scrum Projects with AgileEVM and Earned Business Value Measures" http://www.open.collab.net/servlets/OCNDirector;jsessionid=87FFD7B67E4C0371FB108247CF9B2FD1?id=WP-AGILE-MS

    The article maps value to Story Points - I'm not happy with that as a measure of value. Thoughts?

     Rob, I agree that most of the literature on Value refers to the mapping with story points...
    I have seen others  looking at the EV by understanding the backlog and measuring its "maturity: the size of the backlog when the project started, the rate of backlog processing, the aging of work items in the backlog..etc. .it is costly to have items in your backlog and keeping them  there for a certain amount of time without  being processed (for the reason, they have not being prioritized). In the queuing theory, those are the measure they care for, in addition to the average waiting time in the queue....Is this conflict with some of the agile practices?
    Updated on 2012-11-01T16:52:29Z at 2012-11-01T16:52:29Z by Cherifa
  • JimDensmore
    JimDensmore
    24 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-11-01T18:21:37Z  

    I like "Honest Earned Value". Value is a core metric to add to Walker's initial list. Highly productive teams can still be working on low value capabilities from the business perspective. Enterprise level management may care less about Productivity or Time-to-value if the Value (thinking Innovation as well) delivered outweighs them.

    Earned "Value" (a misnomer) metrics are problematic because they are often implemented to measure the wrong thing - activity. So you have finished 80% of the activity, but not delivered the business value. The Planned/Earned Value part of the EVM equation needs to be changed to reflect business value of the delivered capabilities. This is a problem we have seen before e.g. Iterative development with federal/DoD projects and rigid implementation of EVM contractually tied to a WBS.

    How do we apply this to Agile? Product and Release backlog/burn down are a place to start but the difficulties start when trying to identify the unit of value measurement. $ Value seems the most universal to me, but it is always a struggle for enterprises, and even more so IT organizations, to tie what they are developing back to $ value. Perhaps we should talk about Earned Business Value.

    Well EBV exists... found the topic in Collabnet article "Monitoring Scrum Projects with AgileEVM and Earned Business Value Measures" http://www.open.collab.net/servlets/OCNDirector;jsessionid=87FFD7B67E4C0371FB108247CF9B2FD1?id=WP-AGILE-MS

    The article maps value to Story Points - I'm not happy with that as a measure of value. Thoughts?

    Rob ... what's the problem with story points?  They're simple.  Everything can be gamed, but no more so or less so with story points.  Cherifa ... IS it costly to keep things in the backlog?  From a raw productivity standpoint, I'm not sure.  Going up to the enterprise level though, it can be.  At that level someone needs to prioritize the backlog.  Hopefully that's not being done with EV, it's being done with projected (and therefore uncertain) business value.  If a story isn't needed, then it's never going to be prioritized for development.  It depends, then, on WHY the long queue wait time. 
  • JimDensmore
    JimDensmore
    24 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-11-01T18:33:02Z  

    I like "Honest Earned Value". Value is a core metric to add to Walker's initial list. Highly productive teams can still be working on low value capabilities from the business perspective. Enterprise level management may care less about Productivity or Time-to-value if the Value (thinking Innovation as well) delivered outweighs them.

    Earned "Value" (a misnomer) metrics are problematic because they are often implemented to measure the wrong thing - activity. So you have finished 80% of the activity, but not delivered the business value. The Planned/Earned Value part of the EVM equation needs to be changed to reflect business value of the delivered capabilities. This is a problem we have seen before e.g. Iterative development with federal/DoD projects and rigid implementation of EVM contractually tied to a WBS.

    How do we apply this to Agile? Product and Release backlog/burn down are a place to start but the difficulties start when trying to identify the unit of value measurement. $ Value seems the most universal to me, but it is always a struggle for enterprises, and even more so IT organizations, to tie what they are developing back to $ value. Perhaps we should talk about Earned Business Value.

    Well EBV exists... found the topic in Collabnet article "Monitoring Scrum Projects with AgileEVM and Earned Business Value Measures" http://www.open.collab.net/servlets/OCNDirector;jsessionid=87FFD7B67E4C0371FB108247CF9B2FD1?id=WP-AGILE-MS

    The article maps value to Story Points - I'm not happy with that as a measure of value. Thoughts?

     We have some research we in IBM Rational been participating in, in collaboration with IBM Research.  It might apply here.  I'm honestly not sure yet, it's just a thought.  Consider backlog to be a form of trouble or change tickets (though in this context, normally there is no "trouble" or defect involved).  The research was done by looking at support shops, who traditionally measure themselves using mean time to close trouble tickets.  We have found that this single measure was not giving support shops very good information on how to be more effective.  The five measures Research settled on which improves the feedback a support shop gets are these.  Note that clearly trending is important for each.
    • How old are the tickets when they're closed over the past few time periods - 80% measure
    • How many tickets were closed
    • How many tickets were opened
    • How old are the tickets in backlog - 80% measure
    • How large is the backlog.
    The two 80% references are there because we don't measure the mean age of all the tickets when we do this.  This is because we have found that, on average, about 20% of the backlog is present for reasons out of the control of the support shop.  They should not be measuring themselves on this because they can't control it.  Maybe the same factor is at play here in Agile development backlogs.  The 80%/20% might not be the same, we would need to measure that empirically.  i.e. suppose it turns out to be 90/10 instead - then we should use that number.  This relates back to projected business value again, we'll have to tie that in to see how the backlog prioritization is being performed.
  • Reedy Feggins
    Reedy Feggins
    14 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-11-01T18:56:52Z  
     We have some research we in IBM Rational been participating in, in collaboration with IBM Research.  It might apply here.  I'm honestly not sure yet, it's just a thought.  Consider backlog to be a form of trouble or change tickets (though in this context, normally there is no "trouble" or defect involved).  The research was done by looking at support shops, who traditionally measure themselves using mean time to close trouble tickets.  We have found that this single measure was not giving support shops very good information on how to be more effective.  The five measures Research settled on which improves the feedback a support shop gets are these.  Note that clearly trending is important for each.
    • How old are the tickets when they're closed over the past few time periods - 80% measure
    • How many tickets were closed
    • How many tickets were opened
    • How old are the tickets in backlog - 80% measure
    • How large is the backlog.
    The two 80% references are there because we don't measure the mean age of all the tickets when we do this.  This is because we have found that, on average, about 20% of the backlog is present for reasons out of the control of the support shop.  They should not be measuring themselves on this because they can't control it.  Maybe the same factor is at play here in Agile development backlogs.  The 80%/20% might not be the same, we would need to measure that empirically.  i.e. suppose it turns out to be 90/10 instead - then we should use that number.  This relates back to projected business value again, we'll have to tie that in to see how the backlog prioritization is being performed.
    While I like story points as a meaure of value, don't think measuring mean would work for the Product backlog.  Most Agile teams often use Product Backlog as a wishlist of every possible capability and many are never completed. However if you are referring o the Release backlog which usually has prioritized and estimated stories then we could use it as the source of the such metrics.
     
    Agile teams already report Team Velocity each sprint using points. After 4 or 5 sprints many teams use the average velociy to project how many points may be done for the entire release. Most program and projects managers want to be able to predict their estimate to complete along with their cost factors.
  • MikeORourke
    MikeORourke
    10 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-11-01T19:03:33Z  
     Agree with concern that team can't feel threatened by metrics and that if so they will be gamed in some way. However most organization need some way understanding where they are against "the Plan" and of "predicting" their future outcomes. Project managers are typically asked for monthly updates and projects for this reason.  Its pretty hard to come up with meaningfull metrics and alot of traditional teams have useless metrics, like % percentage complete. Agile metrics have to be derived from metrics the team collects anyways (easy) and are meaning ful to them (things like burndown) but they may need to be normalized against the project own goals
     The first key to at least slow the "gaming" process is to ensure that all key metrics are generated by the system. If they are manual, you can guarantee both the fact that they will be wrong and that they will be gamed. The second key here is to create a "set" of measurements to make gaming the system harder. In other words, keeping measurements around just things related to delivery time, means that you WILL deliver on time, but the quality may be crap.
  • MikeORourke
    MikeORourke
    10 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-11-01T19:06:27Z  
    While I like story points as a meaure of value, don't think measuring mean would work for the Product backlog.  Most Agile teams often use Product Backlog as a wishlist of every possible capability and many are never completed. However if you are referring o the Release backlog which usually has prioritized and estimated stories then we could use it as the source of the such metrics.
     
    Agile teams already report Team Velocity each sprint using points. After 4 or 5 sprints many teams use the average velociy to project how many points may be done for the entire release. Most program and projects managers want to be able to predict their estimate to complete along with their cost factors.
     I think most of the measurements we discuss here are valid, but in the end, measuring story points starts to become useless. If a story has 100 points associated with it, and you've completed 90 of them, what do you have.... probably a half-baked feature or function. I think they key is to decide to measure things that HAVE value to your customers.... features running and testing in production or at least in the SCM trunk. Things like that make more sense to me as being enterprise worthy measurements
  • JimDensmore
    JimDensmore
    24 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-11-01T19:40:38Z  
     I think most of the measurements we discuss here are valid, but in the end, measuring story points starts to become useless. If a story has 100 points associated with it, and you've completed 90 of them, what do you have.... probably a half-baked feature or function. I think they key is to decide to measure things that HAVE value to your customers.... features running and testing in production or at least in the SCM trunk. Things like that make more sense to me as being enterprise worthy measurements
    Mike, I was going to post this in business value ... and sorry it's long ... but your comment causes me to wish it posted here. 
     
    As we've mentioned, story points generally correlate only to the amount of work to complete.  I am not sure what the answer is yet but I have some thoughts on it based on the canonical (woo hoo!) ATM example.  There is a mix here between trying to be efficient and productive (team) and trying to be effective, build the right thing, build the thing right (enterprise, and business alignment).  We have to determine where to do each.
     
    Suppose we build an ATM with two use cases, (1) transfer funds and (2) get cash.  Turns out the transfer funds user stories are complex to build and the cash user stories are easy.  Lots of EV by SP measures for transfer funds, but not so much for get cash.  Later we release our new ATM, first with only the transfer funds bits implemented.  Customers already have ATM cards.  But no one comes to the ATM.  Guess there wasn't much business ("biz") value in that.  But then we release the next version of the ATM and it has get cash implemented.  Whoa, customers flock to our bank, open a new account to have access to the cash option of the ATM and everything.  BIG biz value in that.
     
    Ok that's fairly obvious as far as it goes, but how to measure it?  We have to go back to the original biz plan (hopefully the biz planners have been keeping it up to date as things develop).  And the result that we get, prior to our release, must be an estimate, possibly with large uncertainty.  We must include the uncertainty in the estimate.  Hindsight being what it is, wouldn't we work to get the Get Cash bits released first, if we had it to do over?  Even at the expense of including the transfer funds in the first release?  Surely Shirley.
     
    Next thought, suppose (as is probably true in real life) that the implementations of transfer funds and get cash use similar bits of the underlying architecture?  As transfer funds is developed, some of the components under construction will later be used in get cash.  What is the biz value of those components?  Whatever it is, it's not just associated with the biz value of the transfer funds stories.  It must also include the biz value of the get cash stories.
     
    So if we somehow manage to pick good biz value measures, then we automatically lean toward building the things first that will make us more revenue, and we also lean toward building the things right in the architectural sense, because our biz value measures do better under those conditions.  Clearly the biz value measures should relate to revenue, or profit, or ROI, or something like that.  Just as clearly, they're very uncertain numbers until after the fact, so we need to include that uncertainty.  Unfortunately, uncertainty relates back to honesty - it's hard to stay honest.  Long-term calibration helps.
     
    Now why am I bringing this up in an Agile Metrics context? The answer is that Mike took it up a level, up to the Enterprise level, where doing the work efficiently isn't any longer the only thing that 's important.  At the enterprise level we need to consider managing the portfolio as well, that is, doing the right work, doing the work that will best align our development shop with the business.  Your comment about becoming useless, Mike, applies.  The team needs to be responsible for efficiency and productivity, and needs to be responsive to the enterprise portfolio management prioritization.
  • Cherifa
    Cherifa
    30 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-11-01T19:47:43Z  
     I think most of the measurements we discuss here are valid, but in the end, measuring story points starts to become useless. If a story has 100 points associated with it, and you've completed 90 of them, what do you have.... probably a half-baked feature or function. I think they key is to decide to measure things that HAVE value to your customers.... features running and testing in production or at least in the SCM trunk. Things like that make more sense to me as being enterprise worthy measurements
    Good point Mike..in addition......if a story is worth 100 points (by the way we do not recommend such a big story)..and if we have delivered 90 points...meaning the story is not done yet...would not that be waste?  This story worth of 100 points must have X acceptance criteria (AC)  that have been identified by the product owner. Delivering on a certain number of acceptance criteria instead of  story points ( in accordance with the PO decision) can be a way of having delivered value... some value not all!!! why not? We can track the number of AC and related tests coverage.
    Updated on 2012-11-01T19:47:43Z at 2012-11-01T19:47:43Z by Cherifa
  • MikeORourke
    MikeORourke
    10 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-11-01T19:51:55Z  
    • Cherifa
    • ‏2012-11-01T19:46:18Z
    Good point Mike..in addition......if a story is worth 100 points (by the way we do not recommend such a big story)..and if we have delivered 90 points...meaning the story is not done yet...would not that be waste?  This story worth of 100 points must have X acceptance criteria (AC)  that have been identified by the product owner. Delivering on a certain number of acceptance criteria instead of  story points ( in accordance with the PO decision) can be a way of having delivered value... some value not all!!! why not? We can track the number of AC and related tests coverage.
     Its not waste IMHO just because its not finished. It's just not "there" yet. You could potentially still test it, stub it out, try some of the scenarios against it.... so it has value... but notvalue from a customers perspective yet. The value is that its closer to being done than not. However, to your point, until it can get stakeholder feedback... we've got no understanding of how close it is. The point I was making was closer to where Jim discussed.... which is that we have to use the same "data" for burndown charts and other practitioner measurements but as an executive... I don't care about those measurements... I need measurements that translate to how I am being measured (Time to market, ROI, Costs, etc).