Topic
  • 4 replies
  • Latest Post - ‏2012-11-02T20:55:14Z by JimDensmore
SmartDJ
SmartDJ
4 Posts

Pinned topic Virtual Roundtable - Agile Metrics Breakout - How can we measure the business value achieved since Agile always focus on continuous value delivery [Event closed]

‏2012-10-29T19:28:10Z |
**Even though this event is over, you are welcome to browse this discussion**
 
How can we measure the business value achieved since Agile always focus on continuous value delivery , do we use business value estimation for each user story or ??
 
 If you are not a member of Developerworks, you will need to JOIN this group to reply to this topic.
 
You can return to the Welcome notice, go to the main agile metrics topic, join one of our breakout topics on enterprise level metrics, or  metrics transparency and misuse or discuss future topics.  
Updated on 2013-01-07T22:33:30Z at 2013-01-07T22:33:30Z by sjpeich
  • JimDensmore
    JimDensmore
    24 Posts

    Re: Virtual Roundtable - Agile Metrics Breakout - How can we measure the business value achieved since Agile always focus on continuous value delivery

    ‏2012-10-30T19:53:05Z  
    This was one of several business analytics research topics recently during collaborations between Rational and IBM Research.  The result is the Investment Analysis component of Rational Focal Point that is available with 6.5.1 + and now vetted through a couple of launch clients (one of whom also helped during development, NYC HHS).  I'd post a link to the paper in Connections except Connections is down.  (Just what is a Network Spike, anyway?)  Murray's cattail section will work for now.  Take a look at the ACM preprint first.  If that's too heavy for you try some of the other stuff there on measuring business value, e.g. the ROI Pulse 2012 presentation.  Glad to engage in more detail as required.
     
    The approach is to use benefit estimation including uncertainty, leveraging iterative techniques to reduce that uncertainty over time.   Note that costs also are uncertain, at least if there are any future costs, as will always be true for a project or feature development in progress.  In many cases benefits can be cast in units of dollars (or your favorite country's currency unit); when so, direct comparisons to cost are possible yielding the equations
    • value = benefits - cost
    • ROI = value / cost
    These results are not single numbers, they are distributions calculated from random variables (I'm using the math definitions of these terms).  This makes using a spreadsheet correctly difficult (requires Monte Carlo simulation plugin ... and then it is still difficult) so we created the Investment Analysis component of Focal Point, which makes doing all this and more quite straightforward.
     
    It does NOT make the estimation process easy.  What it does do is take an estimation process most of our clients find inviting but impossible to execute, and makes that process possible, even straightforward.  The only thing that remains is to convince your client that such estimation results in accountability and prioritization that makes it worth the effort (i.e. applying Investment Analysis to the use of Investment Analysis).
     
    All this has little to do with Agile ... except that behaving iteratively is essential to achieving good results from doing the estimations since iterative behavior attempts to incrementally reduce risk.  If you're not behaving iteratively, then uncertainty is too often not addressed until there's little that can be done about it.
     
    Think about it with this example: your product owner thinks that the business value of this project is $10m at best, ($20m) -- negative $20m -- at worst, and has a likely value of $2m.   Your first question is probably "what uncertainties (or risks) are causing you this big swing into negative value?"  The product owner should be able to list them and their estimated contribution, after all they were central to the estimation process.  Now, if you decide to proceed with this project, surely the first thing you would do is address at least one of the big risk contributors in iteration one?  Probably so, and, of course, please stop calling me Shirley.
  • JimDensmore
    JimDensmore
    24 Posts

    Re: Virtual Roundtable - Agile Metrics Breakout - How can we measure the business value achieved since Agile always focus on continuous value delivery

    ‏2012-11-01T19:48:10Z  
    This was one of several business analytics research topics recently during collaborations between Rational and IBM Research.  The result is the Investment Analysis component of Rational Focal Point that is available with 6.5.1 + and now vetted through a couple of launch clients (one of whom also helped during development, NYC HHS).  I'd post a link to the paper in Connections except Connections is down.  (Just what is a Network Spike, anyway?)  Murray's cattail section will work for now.  Take a look at the ACM preprint first.  If that's too heavy for you try some of the other stuff there on measuring business value, e.g. the ROI Pulse 2012 presentation.  Glad to engage in more detail as required.
     
    The approach is to use benefit estimation including uncertainty, leveraging iterative techniques to reduce that uncertainty over time.   Note that costs also are uncertain, at least if there are any future costs, as will always be true for a project or feature development in progress.  In many cases benefits can be cast in units of dollars (or your favorite country's currency unit); when so, direct comparisons to cost are possible yielding the equations
    • value = benefits - cost
    • ROI = value / cost
    These results are not single numbers, they are distributions calculated from random variables (I'm using the math definitions of these terms).  This makes using a spreadsheet correctly difficult (requires Monte Carlo simulation plugin ... and then it is still difficult) so we created the Investment Analysis component of Focal Point, which makes doing all this and more quite straightforward.
     
    It does NOT make the estimation process easy.  What it does do is take an estimation process most of our clients find inviting but impossible to execute, and makes that process possible, even straightforward.  The only thing that remains is to convince your client that such estimation results in accountability and prioritization that makes it worth the effort (i.e. applying Investment Analysis to the use of Investment Analysis).
     
    All this has little to do with Agile ... except that behaving iteratively is essential to achieving good results from doing the estimations since iterative behavior attempts to incrementally reduce risk.  If you're not behaving iteratively, then uncertainty is too often not addressed until there's little that can be done about it.
     
    Think about it with this example: your product owner thinks that the business value of this project is $10m at best, ($20m) -- negative $20m -- at worst, and has a likely value of $2m.   Your first question is probably "what uncertainties (or risks) are causing you this big swing into negative value?"  The product owner should be able to list them and their estimated contribution, after all they were central to the estimation process.  Now, if you decide to proceed with this project, surely the first thing you would do is address at least one of the big risk contributors in iteration one?  Probably so, and, of course, please stop calling me Shirley.
     I also had some thoughts about business value and its use as backlog prioritization, but over in another thread.
  • dsharrock
    dsharrock
    2 Posts

    Re: Virtual Roundtable - Agile Metrics Breakout - How can we measure the business value achieved since Agile always focus on continuous value delivery

    ‏2012-11-02T20:46:09Z  
     This highlights the difficulty of talking about value in many organizations. Though there are some highly numerate organizations in which value is well understood, underpins the majority of business decisions, and is measurable across the development lfecycle, this is rare.
     
    Most organizations have  a poor understanding of business value to begin with, so measuring value is a challenge. A couple of observations on this: 
     
    1) Tracking perceived or agreed business value of the stories (or epics etc) of an backlog can give you an estimate of value delivered by the development process. This first requires estimation of business value - which at a story level is certainly a waste of time, although at  a portfolio level it makes a lot of sense. The benefit of this approach is that it can empower the Product Owner to make prioritization decisions 'against the run of play' but that deliver the maximum estimated value. The downside is that it has no relation to real value as perceived by the customer.
     
    2) Track value as delivered to the customer - revenue or conversion lift, increased customer engagement, whatever the goal is for a given product or release. This has the advantage of being a true measure of tangible value. The downside is that few organizations have the analytics tied together well enough to fully close the loop on the features delivered. The exception might be multi-variate testing around UX/UI work... 
     
    3) A transition from (1) to (2). And this is typically what I see happening. Encouraging an organization to start  using the vocabulary of value, for creating hypotheses for features which can be tested, for setting up the analytics to test these hypotheses as they go live, and for providing the quantitative feedback to the business stakeholders when planning each consecutive release, so that the vocabulary of demonstrable or realized value becomes a part of the discussion. It doesn't take many release cycles for this to have an impact for organizations that truly want to deliver value quickly.
  • JimDensmore
    JimDensmore
    24 Posts

    Re: Virtual Roundtable - Agile Metrics Breakout - How can we measure the business value achieved since Agile always focus on continuous value delivery

    ‏2012-11-02T20:55:14Z  
    • dsharrock
    • ‏2012-11-02T20:46:09Z
     This highlights the difficulty of talking about value in many organizations. Though there are some highly numerate organizations in which value is well understood, underpins the majority of business decisions, and is measurable across the development lfecycle, this is rare.
     
    Most organizations have  a poor understanding of business value to begin with, so measuring value is a challenge. A couple of observations on this: 
     
    1) Tracking perceived or agreed business value of the stories (or epics etc) of an backlog can give you an estimate of value delivered by the development process. This first requires estimation of business value - which at a story level is certainly a waste of time, although at  a portfolio level it makes a lot of sense. The benefit of this approach is that it can empower the Product Owner to make prioritization decisions 'against the run of play' but that deliver the maximum estimated value. The downside is that it has no relation to real value as perceived by the customer.
     
    2) Track value as delivered to the customer - revenue or conversion lift, increased customer engagement, whatever the goal is for a given product or release. This has the advantage of being a true measure of tangible value. The downside is that few organizations have the analytics tied together well enough to fully close the loop on the features delivered. The exception might be multi-variate testing around UX/UI work... 
     
    3) A transition from (1) to (2). And this is typically what I see happening. Encouraging an organization to start  using the vocabulary of value, for creating hypotheses for features which can be tested, for setting up the analytics to test these hypotheses as they go live, and for providing the quantitative feedback to the business stakeholders when planning each consecutive release, so that the vocabulary of demonstrable or realized value becomes a part of the discussion. It doesn't take many release cycles for this to have an impact for organizations that truly want to deliver value quickly.
     Difficult, sure, but necessary for our customers to align themselves with their business.  In many cases we can consider value to be an estimate of future revenue over time.  I think this does track to the value as perceived by the customer.  After all if they don't buy it, they didn't perceive any value from it. We can lead customers to begin this estimating process and then also help them work to close the loop so that they can begin the process of becoming accountable for the estimates.  Reading your item (3), I think perhaps you and I are saying the same thing.