Demos: The Achilles Heel of Agile Projects

from The Rational Edge: The Agile Manifesto and other Agile guidance suggests frequent product demonstrations, such as at the end of each iteration. But much of this advice is broad and non-specific, leaving development teams to learn by trial and error. This article provides a wealth of practical wisdom on when, how, and in front of whom to perform demos on Agile projects. This content is part of the The Rational Edge.

Theodore F. Rivera, Software Group Strategist, IBM

Ted Rivera is part of a core team of engineers with the IBM® Software Group that has developed our standard Agile education, as well as resources for leaders and project managers on Agile teams. He has personally worked with dozens of IBM teams around the world, helping them make the transition to Agile. He has experience in development, test, documentation, user experience, and management; virtually every role associated with the development of commercial software.



12 November 2008

Also available in Chinese Russian

illlustrationIt is a given that Agilistas promote frequent demos -- certainly at the end of each iteration or sprint -- as the key way to make sure the development team stays on track with what their customers will want.1 Trouble is, much of the Agile guidance is unclear about how to deliver demos, how much work is involved in getting demos together and scheduled, or who should see the demos. Worse still, the assumption is that you're building software for one customer or one organization. What if you're developing software commercially, or developing applications for a large and complex organization with widely ranging and even competing interests?

Despite the assertion in the Agile Manifesto that we will value 'customer collaboration over contract negotiation,' truth be told, many Agile teams are far less in touch with their customers than they should be.2

What guidance is out there regarding demos on Agile projects? Not much. For example, the answer to too many questions in the Scrum community is to 'let the team decide.' But what happens when your team is large, or distributed, or both? How does such a team decide how to run demos effectively; or perhaps more importantly, how to get ongoing feedback? The XP community has advocated the notion of the 'on-site customer.' How well has it worked in practice to get users to live and work shoulder-to-shoulder with the development team for months on end? Assuming you can find such brave souls, will an on-site customer be suitable to represent all the many different customers purchasing commercial software, or a large and complex organization's multiple competing interests?

If getting feedback is really an urgent matter for Agile teams, the following questions need thoughtful attention in order to make the aim of the Agile Manifesto work in practice:

  • Who is my customer?
  • How do I get feedback using demos?
  • Are there other ways to get feedback besides demos?

In this article I'll address each of these questions in detail.

Who is my customer?

This might appear to be a stupid question. It is featured prominently here, though, because not all customers are the same -- there are categories of customers to think about. And what's more, since you're not interested in perspectives only from customers (as the Agile Manifesto might lead you to believe), I'll broaden the scope beyond customers to stakeholders. Here are four different stakeholder types, which I've used elsewhere:3

  • Principals are those people who will make the determination to pay for your software to achieve specific business goals, or who are perhaps key influencers in this purchasing decision.
  • End users are of course the people who will use your software to do a part, or even all, of their job.
  • Partners are people who will use your software to help others meet their business goals. Business partners might be one example (e.g., installing, maintaining, or deploying your software for a customer); another example might be a particular customer's IT shop. The IT shop perhaps didn't buy your software or doesn't use it, but they have to successfully get it running so that end users can.
  • Insiders are the people you work with in your own organization who have views about what should be in the software. This could be sales, marketing, executives, corporate, other developers -- anyone you work with who has an opinion about what your product should do or how it should be built.

These various stakeholder categories often support or conflict with one another in the context of developing a commercial software product or application. I'd add that among these four categories, we often tend to pay more attention to insiders and end users -- too often failing to take into consideration our business sponsors (principals) or partners.


How do I get feedback in demos?

In an effort to work through a challenging example, let's assume you want to get feedback for a commercial software product that is being developed using Agile methods. If you operate in such an environment, you realize right up front that just the idea of a demo comes with an inherent challenge -- it's called intellectual property. Depending on the country you are planning to do a demonstration in, if you haven't worked with a lawyer to protect your IP, you will lose that IP as soon as it is demonstrated to someone outside of your organization.

Consider the implications of this one dynamic. Let's say that your team uses short iterations (as it should): perhaps two weeks. Let's say further that you want to do demos every two weeks. Are your interactions with your legal counterparts tuned for this kind of pace? The point here, which is not limited to intellectual property, is that a demo can be very complicated -- and if you're not careful, your demos can get shut down before they've even started.

Varying stakeholder feedback

But for the moment, let's assume you've handled legal and other issues. How might you get feedback from your various stakeholders by means of demos? Consider the illustration in Figure 1.4

Shows sequence of iterations.

Figure 1: Feedback intervals during a project

If you consider a demo to be not just a one-time event, but a frequent one, it becomes clear how those stakeholder categories can be helpful. Here's an approach you might consider as a 'mental model' to get stakeholder feedback:

  • At the end of iteration 1, you might demonstrate to an insider stakeholder, such as an executive. There is a good deal of value in using the first iteration in this way. To begin with, it is clarifying and galvanizing for the team to understand that at the end of the very first iteration, their software needs to be stable enough to demonstrate, because 'someone important' is going to be interested in seeing it work. Second, if the first demo is given to an insider stakeholder, there is no need to address legal issues associated with intellectual property. And third, let's assume the demo is a failure in some way -- what have you really lost? Yes, perhaps you have egg on your face, but you have not failed in front of a paying customer, which is a far more significant matter.
  • At the end of iteration 2, we might demonstrate to a cross-organizational set of interested parties. Many different people you work with are highly opinionated about your software, and a failure to recognize and acknowledge this is foolhardy. What is more, it is urgently important to engage with this cross-functional set of interested parties early in the development of a new software release, since it helps to set expectations. Importantly, this also helps to reset the organizational expectation that the demo is a review of what will be in the product to seeing it as a review of what is now in the product. Once again, this second iteration demo being offered to insider stakeholders buys us an additional iteration prior to needing to obtain legal clearance for our intellectual property.
  • For iteration 3, we might do a demo for business partners. It is increasingly the case, especially for developers of commercial software, that business partners will potentially help sell, deploy, troubleshoot, configure, customize, and train others to use that software. As a result, it is often the case that such partners have a viewpoint that is seasoned with the experiences of many different customers of your software. Such a perspective is invaluable. What is more, failure to consider the perspectives of such partners will potentially predispose them to work with other vendors with whom they will have better opportunities for collaboration. Given the importance of satisfying business partners to maintain your own revenue stream, such demos are not optional, but essential. Sadly, such demos are too rarely sought out.5
  • For iteration 4, it's certainly necessary and appropriate to demonstrate your software to end users. Many Agile teams wrongly assume that this means shipping the software to one or more customers, having them install it, and then providing the developers with feedback (like a beta program). While for some teams, frequent betas will be useful and should be considered, an online demonstration to one or many end users can be conducted via conference call and or web tooling, allowing for feedback that can spur dialogue among the various end users. It is often the case in such demonstrations that end users will 'correct' extreme viewpoints of what should be in the product or how a given function should work.6
  • At the end of the fifth iteration, it would be great to do a demonstration for one or more principals -- in the case of commercial software, the people who have either made the decision to buy your software in the past, or who are considering such an acquisition at present. Let's face it: these demonstrations are hard to set up. But if you prepare several iterations in advance, it makes a huge difference. In the case of principals who have not yet bought your software, you will always find enthusiastic support for such demos from people who sell your software. In the case of principals who have already purchased your software, you will always find enthusiastic support for such demos from people who support your software.
  • What about the sixth (or subsequent) iteration? Bear in mind that all of this is just one among many ways to approach the Agile demo problem. You could instead do a demonstration to executives and other insiders at the same time, or demo to end users every other iteration, or orchestrate user group meetings around demos, or whatever. Mix and match. The important thing is that you have a meaningful demo each iteration to someone who cares. And also, while this example highlights an approach that starts with internal stakeholders and gradually moves outward, the best feedback you will get is rarely from internal parties -- so be aggressive and thorough in soliciting feedback from those who will buy and use your software.

One additional point is worth making in this context. What about the case when the work done in a particular iteration is somewhat hidden work, such as when work is done on an API? The key is to consider: who is interested in the work that has been done in this iteration, or in the last several iterations? Even in the case of an API or a similarly unglamorous set of functions, someone else cares: a partner development team, a consuming application team, whomever. Always, always do a demo to people who care about the stability and value of that function.

Making decisions with stakeholder feedback

There's one other aspect of doing demos that will not be intuitively obvious. Consider the diagram in Figure 2.

Shows different intervals for addressing feedback

Figure 2: Feedback intervals on an idealized project

In this simple diagram, an ideal world is envisioned: feedback is received from stakeholders, and in the iteration after the one you're about to work on, that feedback is acted on. Trouble is, the real world is a messy place. Not all feedback is created equal. Let's say for example, that after the end of the first iteration, you are able to do a demo in front of ninety end users, taking advantage of a small user conference or some similar activity. It may be the case that the input you receive is of such substance that you know to act on it immediately. It might therefore be the case that in iteration 2, you would set to work implementing their recommendations.

Now, let's assume that after iteration 2, you get feedback from a principal whose perspective you were unsure of. Were you to act on that feedback immediately, you could easily build the wrong capabilities into your product. It would be wiser to validate that perspective with feedback obtained in one or more subsequent iterations.

The real goal is this: by doing meaningful demonstrations for many varied stakeholders over the course of many iterations, your whole team can begin to understand viscerally what will and what won't resonate with them all. Demos will begin to serve as sanity checks and innovation moments, rather than profoundly significant and rare opportunities for the team to obtain feedback.

Consider what this end state might look like. A high-ranking executive goes to an important customer, and comes back with information about a key function pushed for by that customer. What would you do now in this situation? Very likely, you'd sit back and take it, pretending that you could get this function into the current release with quality. If instead, you had a history of interaction with varied stakeholders, you could say to that executive:

'I see you've asked for function x. We are happy to do whatever you'd like us to work on. But here are the business objectives (rather than functions) we have heard from our discussions with others. For one of these business objectives, we have twenty customers who have asked us to include this in the product, and we can't get to this yet. We have another for which ten customers have made a similar request. The function you are asking for no one else has yet requested. Which would you prefer we work on?'


Are there other ways to get feedback besides demos?

Agile methodologies emphasize the iteration demo as a primary mechanism to obtain feedback, but beyond this, one other approach warrants reflection. The Scrum methodology offers to us the notion of the product owner. Simply put, the product owner has the final decision with respect to what will go into the software product or application. For complex projects, the idea of a chief product owner in concert with product owners over various subcomponent or applications might be organized in a hierarchical manner.

It's great, and perhaps urgently important, to have a final decision-maker with respect to what will go into our software. But my fear is that the product owner will not foster the kind of demo cadence highlighted above, or will perhaps listen too closely to a trusted source (such as an industry analyst) as opposed to the varied perspectives of various stakeholders.

My ultimate concern, though, is less with the idea of the product owner than it is with our desperate need to get meaningful feedback. Demos are important, but they are just not enough. As a consequence, three other approaches that can augment demos warrant brief mention: transplant testing, residencies, and reverse residencies:7

  • In transplant testing, client artifacts are transplanted into your test environment in an effort to better mimic how the clients are using your software. On the surface, it might appear as if this is a difficult or esoteric activity. On the contrary, consider the possible 'footprint' your software may leave on a client's systems: preference files, log files, data sets, configuration files, trace files; you name it. What is more, consider the performance characteristics of your software in the client's environment. Are there peak usage periods? What kinds of flows are there in and out of your system? The more creatively you can emulate the environment your clients work in, the more insight you will gain into how your software is really being used.
  • Residencies provide a profound opportunity for you and your software development organization to really understand how your software is being used. In residencies, you bring in one or more people who work with your software from a particular client or partner, and have them spend perhaps a week testing the new release under development. Invariably, they find issues you would not have found, and they gain insight into both what is new in the product as well as advanced aspects as to how it can be used.
  • Reverse residencies allow you to send members of our technical team -- architects, developers, testers, information developers, whomever -- to spend time with people who use your software in a production environment. Simply put, there is no better way to understand how people want to use your software than to see them using it in practice. There is just no substitute. In our reverse residencies, we have often focused on helping out with important deployments or migrations, or have become involved with customers using more advanced implementations of our software. Regardless of who your stakeholders are, get out there with them and see first-hand how they benefit from and/or struggle with your software.

These three methods, while explained briefly here, should be thought of as key ways in which the iteration demos you are now conducting can be supplemented with additional, deep insight into how your software is used and the business value it delivers. Such insight is indispensable.

Are there yet more ways to get feedback? Absolutely. We find internal betas tremendously helpful for many of our products and applications. We have informal mechanisms that allow for the sharing of code, components, and whole products prior to release. Make the last iteration of your code available for others to access and use. Be creative.


Parting thoughts

The chief objection to this kind of frequent interaction by means of demos, transplant testing, residencies, and reverse residencies -- an approach that suggests we listen closely to our various stakeholders -- is that it will somehow diminish what is referred to as 'breakthrough thinking.' In other words, some would assert that if all you're doing is listening to the people who buy, use, install, and support your software, you'll never come up with the 'next great thing.'

First of all, I'd suggest that an approach that fosters incremental learning is likely a model with far more potential to succeed for the typical team -- it's a way to ensure solid progress that is not dependent solely on that divine spark of insight that is so elusive. That said, if your teams begin to interact as intensely with their various stakeholders as has been suggested here, is it not the case that they will have a better sense as to what they really need, as opposed to what they need right now?

  1. Some view the term "Agilista" as pejorative. I view it as a call to action, and proudly sport it as a title.
  2. The Agile Manifesto can be found at http://agilemanifesto.org/.
  3. The origin of these categories is Carl Kessler and John Sweitzer, Outside-in Software Development (Upper Saddle River, NJ: IBM Press/Pearson, 2008). Scott Sehlhorst also offers a helpful perspective at http://tynerblain.com/blog/2007/10/11/stakeholder-goals/.
  4. This illustration is adapted from Outside-in Software Development, and it is used with permission.
  5. Bear in mind that the stakeholder category of partners includes more than simply business partners. The members of an IT shop that deploys your software can certainly be considered partners -- they didn’t buy your software, but they have to install, configure, maintain, upgrade it, and so forth, just as a business partner might. It should also be noted that in many organizations, legal agreements are already in place with business partners, once again potentially mitigating the need for intellectual property clearances that might be needed for other demonstrations.
  6. Note that demonstrations to end users will require legal clearance for your intellectual property. If it appears as if too much emphasis is being placed on legal issues throughout this example, bear in mind that you will lose any and all intellectual property if demonstrating to people outside your company without such precautions.
  7. I have written and spoken about these three approaches, developed along with IBM® employees Adam Tate and Scott Will, in various papers and at various conferences. Most recently, these three approaches have been discussed and expanded on by Carl Kessler and John Sweitzer in the aforementioned Outside-in Software Development.

Resources

  • A new forum has been created specifically for Rational Edge articles, so now you can share your thoughts about this or other articles in the current issue or our archives. Read what your colleagues the world over have to say, generate your own discussion, or join discussions in progress. Begin by clicking HERE.
  • Global Rational User Group Community

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational
ArticleID=351579
ArticleTitle=Demos: The Achilles Heel of Agile Projects
publish-date=11122008