Topic
  • 31 replies
  • Latest Post - ‏2012-11-02T14:20:10Z by Reedy Feggins
WalkerRoyce
WalkerRoyce
18 Posts

Pinned topic Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures [Event closed]

‏2012-10-30T14:54:43Z |
**Even though this event is over, you are welcome to browse this discussion**
 
Let's break out into a different thread for discussion around enterprise measures. Productivity, Quality, Time-to-value and Predictability are four distinct measurement areas to consider.
 
Let me start with a simple assertion. 
Project or team level metrics tend to be centered on looking at changes from release to release. Defects, change speed, change cost, and changes in the variance on the estimate to complete are good examples. The object of measurement is essentially the code/test base and the subject of measurement is largely the same team. Even though the code and team may be dynamically changing over time, they represent a relatively consistent context for measurement.
 
Now let's switch to an enterprise context. We have already seen warnings about comparing metrics across projects. Why? Largely because their is not a comparable basis for comparison--namely the code/test base on different projects and the team makeup on different projects can provide a wildly incomparable context.
 
What do we do? How do we normalize so that we can manage our business against reasonable norms?
 
 You will need to JOIN this group to reply to this topic.
 
You can return to the Welcome notice, go to the main agile metrics topic, join one of our breakout topics on metrics transparency and misuse, or business value achieved or discuss future topics
Updated on 2013-01-07T22:34:08Z at 2013-01-07T22:34:08Z by sjpeich
  • HazelWoodcock
    HazelWoodcock
    10 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-30T18:59:39Z  
    Walker, these four areas are distinct, but seem to be very tightly coupled.  Without predictability it is hard to achieve any of the other three.  Time to value, productivity and quality all belong together in an equation alongside a magic constant.  I can't argue against these being useful at an enterprise level, but they are also valuable at a project level.  With historical metrics from previous projects to compare against, these will also point to projects getting into trouble before they are dangerously out of control. 
    Perhaps these are universally useful measures, but they are not simple to calculate.  How many individual numbers do we need to collect to get a meaningful view of these, and are the things being measured transferable between organizations?  Is there a transferable formula we can create that can be used across organizations to indicate performance in each area?  Can the complexity of the project be taken into account in the numbers?  If we produce a transferable formula, do we then have the issue of senior managers with an incentive to game the metrics as well as developers?
  • Cherifa
    Cherifa
    30 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-30T20:04:22Z  
    Walker, these four areas are distinct, but seem to be very tightly coupled.  Without predictability it is hard to achieve any of the other three.  Time to value, productivity and quality all belong together in an equation alongside a magic constant.  I can't argue against these being useful at an enterprise level, but they are also valuable at a project level.  With historical metrics from previous projects to compare against, these will also point to projects getting into trouble before they are dangerously out of control. 
    Perhaps these are universally useful measures, but they are not simple to calculate.  How many individual numbers do we need to collect to get a meaningful view of these, and are the things being measured transferable between organizations?  Is there a transferable formula we can create that can be used across organizations to indicate performance in each area?  Can the complexity of the project be taken into account in the numbers?  If we produce a transferable formula, do we then have the issue of senior managers with an incentive to game the metrics as well as developers?
     Walker,  normalization is a goodness for benchmarking!! To have a comprehensive list of metrics at the enterprise level that can be normalized by industry/sector (banking, insurance, distribution, mobile/telco, etc..) can be a great tool for benchmarking and comparing one organization with another. Any thoughts ?
  • GustavoGrillo
    GustavoGrillo
    8 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-30T20:11:42Z  
    Walker, these four areas are distinct, but seem to be very tightly coupled.  Without predictability it is hard to achieve any of the other three.  Time to value, productivity and quality all belong together in an equation alongside a magic constant.  I can't argue against these being useful at an enterprise level, but they are also valuable at a project level.  With historical metrics from previous projects to compare against, these will also point to projects getting into trouble before they are dangerously out of control. 
    Perhaps these are universally useful measures, but they are not simple to calculate.  How many individual numbers do we need to collect to get a meaningful view of these, and are the things being measured transferable between organizations?  Is there a transferable formula we can create that can be used across organizations to indicate performance in each area?  Can the complexity of the project be taken into account in the numbers?  If we produce a transferable formula, do we then have the issue of senior managers with an incentive to game the metrics as well as developers?
     I like to look at these metrics from outside the "IT department".
    Why is Productivity an important metric? It may seen obvious but in a agile context it is healthy to ask this, if only to check that we're not doing things "because we've always done that way".
    Quality is one metric that might really be useful if seen from the business's or customer's point of view. Some features might not be as important to the customer as others, so they may be of lower quality in technical terms.
    Time-to-value could be assessed on the long run by checking if the organization is delivering faster than its competitors or at the right pace for the market/customers.
    For Predicability I would ask the same question that I asked for Productivity. If you delivering at the market's pace, does it matter if you're on schedule? Two weeks early? A month late?
    I know these are more questions than answers, but I think it's healthy to ask "Why we're doing this, anyway?" sometimes  
  • WalkerRoyce
    WalkerRoyce
    18 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-30T20:27:17Z  
    • Cherifa
    • ‏2012-10-30T20:04:22Z
     Walker,  normalization is a goodness for benchmarking!! To have a comprehensive list of metrics at the enterprise level that can be normalized by industry/sector (banking, insurance, distribution, mobile/telco, etc..) can be a great tool for benchmarking and comparing one organization with another. Any thoughts ?
    I don't see obvious ways of normalizing across industries.
     
    I see one simple way of normalizing that avoids the direct cross-project comparisons. Various teams will measure themselves differently. But however they measure, we can hold them to measured improvements that are roughly equal in terms of improvement over time. Most big organizations demand a 5% improvement year over year. In big macro terms, this is a fair thing to do. However you measure, we want you to improve that measure by a certain percentage. Note that an immature organization has a purely speculative basis for setting such a target. A mature organization with a strong foundation of analytic experience can provide a credible case for improvement that can be negotiated rapidly between governors, team leaders and practitioners as a shared, achievable target.
     
    Now, one can provide unfair counter examples. If one team has a a zero-defect track record and another has an abysmal quality record, applying 5% improvement to both teams is a context-independent mistake. However, such counter examples occur rarely in our business and smart people can adjust.
  • WalkerRoyce
    WalkerRoyce
    18 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-30T20:35:53Z  
     I like to look at these metrics from outside the "IT department".
    Why is Productivity an important metric? It may seen obvious but in a agile context it is healthy to ask this, if only to check that we're not doing things "because we've always done that way".
    Quality is one metric that might really be useful if seen from the business's or customer's point of view. Some features might not be as important to the customer as others, so they may be of lower quality in technical terms.
    Time-to-value could be assessed on the long run by checking if the organization is delivering faster than its competitors or at the right pace for the market/customers.
    For Predicability I would ask the same question that I asked for Productivity. If you delivering at the market's pace, does it matter if you're on schedule? Two weeks early? A month late?
    I know these are more questions than answers, but I think it's healthy to ask "Why we're doing this, anyway?" sometimes  
     We are all being asked to do more with less. That is why some measure of productivity is needed. How much more and how much less constrains our plans, targets and morale. If we don't know how productive we are now, it is difficult to convince people we can do better.
     
    Quality is directly correlated with your outside-in reputation. If you want to sell anything, your reputation for quality is a key parameter for persuading clients.
     
    Time-to-value is, in my view, the measure that is most correlated with agility. Why are we all so enamored with agility? Because we all know that the quicker we can make "value" tangible in development, the quicker and better we can translate it into business outcomes.
     
    Predictability is another metric correlated to reputation. If you don't plan and execute predictably, you will have a harder time selling your investors on the funding, selling your teams on the plans and establishing trust among everyone.
     
    My simple answers to your very insightful questions.
     
    Updated on 2012-10-30T20:35:53Z at 2012-10-30T20:35:53Z by WalkerRoyce
  • GustavoGrillo
    GustavoGrillo
    8 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-31T01:54:26Z  
     We are all being asked to do more with less. That is why some measure of productivity is needed. How much more and how much less constrains our plans, targets and morale. If we don't know how productive we are now, it is difficult to convince people we can do better.
     
    Quality is directly correlated with your outside-in reputation. If you want to sell anything, your reputation for quality is a key parameter for persuading clients.
     
    Time-to-value is, in my view, the measure that is most correlated with agility. Why are we all so enamored with agility? Because we all know that the quicker we can make "value" tangible in development, the quicker and better we can translate it into business outcomes.
     
    Predictability is another metric correlated to reputation. If you don't plan and execute predictably, you will have a harder time selling your investors on the funding, selling your teams on the plans and establishing trust among everyone.
     
    My simple answers to your very insightful questions.
     
    Thanks Walker,
    I agree with you about Quality and Time-to-value.
    I have problems with Productivity and Predictability because I've never seen them being used in a way that was effective in an enterprise environment.
    For example, I've seen Function Points being used for calculating Productivity (Function Points per week). What happened is that people began inflating the FP calculations, so Productivity increased.
    I've also seen more than my share of buffers in schedules, beyond reasonable risk-management, just to give a false impression of predictability. So, if you estimate 3 times more hours for every task on your project you will be pretty much on schedule, no matter what.
    I obviously chose extreme examples, but that's my experience.
    Is there a way to have a Productivity/Predictability metric that is scroundrel-proof?
    Am I doomed to misbehaving projects? 
  • WalkerRoyce
    WalkerRoyce
    18 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-31T10:36:45Z  
    Thanks Walker,
    I agree with you about Quality and Time-to-value.
    I have problems with Productivity and Predictability because I've never seen them being used in a way that was effective in an enterprise environment.
    For example, I've seen Function Points being used for calculating Productivity (Function Points per week). What happened is that people began inflating the FP calculations, so Productivity increased.
    I've also seen more than my share of buffers in schedules, beyond reasonable risk-management, just to give a false impression of predictability. So, if you estimate 3 times more hours for every task on your project you will be pretty much on schedule, no matter what.
    I obviously chose extreme examples, but that's my experience.
    Is there a way to have a Productivity/Predictability metric that is scroundrel-proof?
    Am I doomed to misbehaving projects? 
    No doubt, that all of these can be gamed. Humans are subject to human nature.
    In a low trust environment, there will be more gaming. But there is an even bigger answer to achieving honest measures. People call this empowerment, but the term is so misused, I will use a different synonym: ownership.
    When a team "owns"  the results, they will be more honest because it is they who benefit.
    Owners of a home will treat their property better than renters of a home.Neighborhood watch programs work because neighborhoods have shared ownership in the security of their properties.
     
    So how do we get teams to own their targets?
    Mandates from above don't work. Team commitments to plans and incentives can work. One key is to recognize that many of the targets are interrelated. If we target only one dimension, the others may suffer. So good incentives are outcome based where the team can tradeoff performance across multiple dimensions.
    Again, this gets back to creating a high trust environment and shared ownership for goals.
     
    Is IBM good at this? Think about your local team. How much say do you have in your own team's performance destiny? 
  • GustavoGrillo
    GustavoGrillo
    8 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-31T12:35:28Z  
    No doubt, that all of these can be gamed. Humans are subject to human nature.
    In a low trust environment, there will be more gaming. But there is an even bigger answer to achieving honest measures. People call this empowerment, but the term is so misused, I will use a different synonym: ownership.
    When a team "owns"  the results, they will be more honest because it is they who benefit.
    Owners of a home will treat their property better than renters of a home.Neighborhood watch programs work because neighborhoods have shared ownership in the security of their properties.
     
    So how do we get teams to own their targets?
    Mandates from above don't work. Team commitments to plans and incentives can work. One key is to recognize that many of the targets are interrelated. If we target only one dimension, the others may suffer. So good incentives are outcome based where the team can tradeoff performance across multiple dimensions.
    Again, this gets back to creating a high trust environment and shared ownership for goals.
     
    Is IBM good at this? Think about your local team. How much say do you have in your own team's performance destiny? 
    That's exactly my point.
    It is difficult to create enterprise-level metrics because they are owned by the managers or CIOs, not by the team. You can't have effective metrics without an environment where trust can thrive.
    When we are recommending this or that metric in a big company this issues keep coming to haunt us and it's hard to come up with more metrics than working software and the PO's view of R.O.I.
  • WalkerRoyce
    WalkerRoyce
    18 Posts

    Re: Virtual Roundtable-Agile Metrics Breakout--Enterprise Level measures

    ‏2012-10-31T13:06:30Z  
    That's exactly my point.
    It is difficult to create enterprise-level metrics because they are owned by the managers or CIOs, not by the team. You can't have effective metrics without an environment where trust can thrive.
    When we are recommending this or that metric in a big company this issues keep coming to haunt us and it's hard to come up with more metrics than working software and the PO's view of R.O.I.
     It may be difficult, but it is still important.
    CIOs and Execs that rule by fiat will not create a high trust environment and such measures will not produce great results. But it can be done. 
    It is obvious to most teams when hey have been listened to, when they are consulted on what the probability distribution of possible outcomes is likely to be, and when there is a negotiated target where all constituents understand the risks and rewards. We may not pick the easy path, but we pick a path where everyone understands the upsides and downsides--and everyone is communicating honestly about progress trends, quality trends and obstacles.