Agile planning in real life
First-hand advice for developing releases with agile planning
"We're agile, we do iterations." You hear this statement often from teams claiming to do agile development. But there's a lot more to being truly agile than doing your development in iterations.
Traditionally, development teams have specific releases with pre-determined content and pre-determined dates. At the start of each release, the management team meets with the development team for the ritual arm-twisting for commitments on both the release date and the content.
The challenge in this scenario is that management is trying to dictate both the release date and the content, while the development team's belief is that they can achieve one or the other. In reality, the content side of this equation may be more flexible than the development team initially believes it to be. Unfortunately, development teams do not always realize this, and people start working wonky hours to get things out the door.
This article serves as a roadmap that can help you overcome this difficulty and move your team from doing plain iterative development to agile development. The roadmap includes:
- Suggestions for helping your team transition to using agile planning.
- Guidance on iterations. How long should they be? Should a team only work in iterations during a given release? What should the team be doing between releases?
- Tips on creating user stories and understanding the role of the product owner. What are user stories? Who should write them? What is the difference between an epic and a story? How long should the work contained in a story be? Who is the product owner? What is the role of the product owner in the process?
- How-to advice for planning and managing the backlogs at the product, release, and iteration levels. Each planning level has its own backlog. What should these different backlogs contain? What level of estimation is required for each planning level?
- Techniques for measuring and interpreting team velocity. What kind of information can you extrapolate from team velocity?
Easing the transition
Here are some things to consider before you start to transition:
- Understand what you want to get out of the process. Change is hard for some people, and you need to understand where you are willing to make compromises early in the process in order to get the results you want. You can always push for more once you've demonstrated you are getting the desired results; express that what you want to add is simply the next step in the evolution of the process.
- Don't start the transition on a whim. If the process is being forced on the team, and the team has not been properly trained, this can create a lot of bad feelings and a general lack of confidence in the new process.
- Know that education is your friend. Learn what you need to learn, and get comfortable with it. Try using agile methods on a small-scale project first to work out some of the kinks. Once you've learned, educate the rest of your team. Ensure that everyone uses the same vocabulary and has the same understanding of the process before you begin. This education process will help you find the gaps in your own knowledge.
- Commit to the process. Once you start, don't look back. Yoda must have had this in mind when he said: "Do or do not ... There is no try." If things start to go wrong, make adjustments to the new process, rather than reverting to your old ways.
You need a level of commitment from your team to be successful. To get their commitment, you need to believe in the process yourself. Agile planning can work for your team if you are committed to doing it right.
Scheduling your iterations
Let's start with a popular question: "How long should your iterations be?" The goal of doing iterations is to allow the team to get feedback early and often from your stakeholders (customers, the product owner, and others).
A common iteration length is two or four weeks (sometimes up to six weeks). Shorter iterations tend to be easier to plan because the quick turnaround conveys a sense of urgency. Longer iterations give the false impression that there's a lot of time for people to do their work—this can be deadly.
Iterations are the heartbeat of your team. You should always be doing iterations, even if you are not currently working on a release. This allows the team to schedule early research (also known as architecture spikes), prototyping, or user story creation for the next release. Doing these activities before actually working on the "official" iterations of the release can help you identify risks early and devise mitigation plans if needed.
Try not to vary iteration lengths during a release just to meet a specific release date. You may encounter some resistance to change on your team, and constantly changing the iteration length does not demonstrate that your team leaders have confidence in the new process. Also, calculating team velocity (which we'll cover later under Measuring velocity) is more difficult with variable iteration lengths because you are not using a constant period of time to measure against.
The exception to preferring a shorter iteration length is the final iteration. If the final iteration contains mainly endgame activities such as globalization and bug fixing, it is less disruptive to the team to have a shorter iteration at this point.
As an alternative to scheduling a shorter iteration for the final iteration, you can schedule a shorter length (like two weeks) for all iterations. By using this approach, if the release date is not at the end of the iteration, you will be able to wrap up the work for the release a week earlier.
On the last day of an iteration, schedule time for a demonstration and retrospective of the iteration, and include the developers, management team, and stakeholders. The demonstration of features helps everyone see the progress made by the team and also provides an opportunity for early feedback that can lead to valuable adjustments during the course of the release.
Creating user stories
User stories are one-liners that state customer requirements. Imagine them as what you would tell a customer to explain what you are doing. User stories focus on what the customer needs; they do not spell out implementation details.
The product owner talks with customers to understand their requirements and creates an initial set of high-level user stories. To create more fine-grained user stories, you may want to create a customer team with a product manager, testers, real users, and interaction designers. This template is useful for writing user stories:
As a <role>, I want to <goal> to achieve <business value>.
User stories can have a hierarchy as well. Epics are user stories that talk about features or functionality at a higher level. You can iterate over these, break them down, and generate new user stories and epics. Figure 1 illustrates an epic that was decomposed into multiple smaller user stories.
Figure 1. An epic broken into multiple user stories
User stories represent the work the team must accomplish to meet the requirements of the product. How big should the user stories be? For epics, it doesn't necessarily matter; these will be broken down in smaller pieces later. For stories that the team will work on during the iteration, the work should take two to three days to build and should be functionality driven whenever possible.
You should not size the user stories to fit an iteration. For example, if you have a four-week iteration, this will potentially be a big story that you cannot contain in your iteration. At the end of the iteration, you may find that the team is unable to contain the work it committed to.
It is normal to have a lot of user stories in your release backlog. Look at it as a reality check. If you have fewer stories. but these stories take more time to implement, you don't have a good representation of the work that you need to accomplish.
You should try taking the building blocks approach when building user stories. What is the minimum functional behaviour you can deliver? Start from that point, then create another story for the next set of functional behaviour. and keep going until you have created stories to cover the entire feature.
Note that stories may cut through the different components of your product. In this case, your team just needs to create the tasks for each component during the iteration planning phase.
Identifying the product owner
The next thing to do is identify your product owner. The product owner is someone who is in contact with the customers and has an understanding of their needs. The product owner identifies the features that are required in the product for it to be successful on the market and meet customer needs.
The relationship between the product owner and development team is very similar to a customer and provider relationship. The important point to remember about agile planning is that the development team should always be working on items the product owner considers important. This is why both the product backlog and release backlogs are prioritized lists.
Even if the product owner is the ultimate decider in terms of priorities, there still needs to be a healthy relationship between the development team and the product owner. The development team must ensure that its needs, in terms of architecture work that is required to deliver on the release backlog, are understood and properly prioritized.
The product owner's responsibility is ensuring the product backlog and release backlog are properly prioritized to ensure the team is always working on the most important work items.
The product owner is responsible for the product backlog, a prioritized set of user stories that need to be implemented in the product.
Your team may already have a requirements, or line-item, database. If this is the case, these are a good starting point for building your product backlog. To convert your existing database, the product owner should go over the requirements database and identify the high-level user stories associated with each of them.
The backlog needs to be prioritized because it allows the product owner to select the pieces of a requirement that are more critical to implement. One requirement may break down into ten user stories, but if the development team completes the five most critical stories, that may be sufficient to meet the customer requirement.
Figure 2. How the product backlog feeds all the others
The high-level stories in the product backlog need to be estimated by the team using story points. All groups involved in delivery of the feature (development, quality assurance, and documentation) participate in the estimation process. These estimates give the product owner a picture of the relative cost of each feature.
The product backlog is meant to be dynamic with items being added, removed, and reprioritized. At the same time, it needs to be maintainable. The work it represents should not exceed 18 months. Beyond that threshold, lower priority items should get knocked off the list.
Why should the backlog be kept short? This suggestion comes from Lean principles (see Agile software development in a nutshell). There's no value in keeping more work in a queue than you can realistically do in a reasonable amount of time. If a good idea falls off the backlog, it will return if it is a really good idea and you are in touch with your customer needs. So let stuff fall off the list and don't try to keep these in some secret, hidden, back-pocket backlog list.
At the start of a release, the product owner looks to the product backlog and identifies the user stories to be included in the next release. These items are then moved to the release backlog.
Usually, the ritual arm-twisting begins at this point. The development team must now work closely with the product owner to ensure that the work that was identified looks appropriate for the size and duration of the release. Tracking the team velocity during the release will provide a clearer picture of how much of this work will be achieved in the end.
While creating the release backlog, break down any high-level user stories into smaller user stories. For this task, consider creating a customer team with a product manager, testers, real users, and interaction designers.
The customer team works in concert with the product owner to prioritize the new stories and add them to the release backlog. The development team then estimates these new stories and assigns them story points. The goal at this stage is to give development a good sense of what is required in the release in smaller chunks to estimate.
As the development team works through user stories during the iterations, the product owner and development team gain knowledge. This knowledge may generate new user stories or demonstrate that some stories are less critical. For new stories, the product owner and development team need to go through the estimation and prioritization process once again. For less critical stories, the product owner should remove them from the release backlog or lower their priority in the backlog. The team can also use this new knowledge to change estimates of existing stories in the backlog.
The release backlog is dynamic, but once the iteration starts, the product owner can no longer change the work that was selected for the iteration. If, after starting the work, the team gains knowledge that indicates that certain items will not fit in the release or in the iteration, this is a different situation (discussed in the next section). During the iteration, the development team must be able to focus on their deliverables.
At the start of the iteration, the product owner can re-prioritize the release backlog so the development team knows the items it needs to work on next. The product owner and development team should take a half-day on the first day of the iteration for the iteration planning meeting. During this meeting, the team should select as many high-priority items as it can contain from the release backlog and add these to the iteration backlog.
The trick to limiting this activity to a half-day is to have your release backlog ready to go. If the team is creating the user stories from iteration to iteration (instead of as early as possible), you can waste valuable time, and it can take longer to do your planning. You'll also never really know how much work is planned for your release.
During the planning meeting, for each user story in the iteration backlog, the team needs to break down all the tasks required for the story to be considered complete. The tasks from everyone involved in the story (development, quality assurance, documentation) need to be identified. The team needs to estimate each of these tasks in hours and add these estimates to the plan.
The biggest concern here is over-commitment. In your initial iterations, look at the number of hours in the estimates to ensure the work will fit in the time allocated for the iteration. If the team finishes work ahead of schedule, it can always go back to the release backlog and pick some additional work items that can be completed inside the iteration.
The other concern is not identifying all the tasks required to complete a story. Unfortunately, this will happen in your initial iterations. These forgotten items pile up and may cause you to not be able to complete a story in an iteration. Here are two examples of this common mistake:
- Having a bucket called "Testing" that doesn't include the time for writing the test cases.
- Not including a task for the development team to write automated unit tests.
Before the team goes back to the release backlog well, ensure the stories in the iteration are really done (then ensure they are done two more times!). In other words, are all the tasks complete? If some testing is outstanding for a user story, it will be to your advantage to have the developer help the tester complete the testing. If there are defects remaining for user stories developed in the iteration, fix them now before moving on. Build quality in; it's not how much work you do in an iteration, it's all about how bug free the delivered work is.
You will want to establish exit criteria for your iterations. For example, all high-priority (severity one and two) defects must be resolved for the story to be considered complete. Any unresolved low-priority (severity three and four) defects must be resolved in the following iteration and will be added to the top of the iteration backlog. It's important not to carry your defects for too long a time. This way, you will be in a much better place during the end game part of your release.
If the team realizes during the course of the iteration that a user story is more complicated than originally anticipated and cannot be contained in the iteration or even in the release, then you need to raise this issue as early as possible. It is important to identify your options. Is there a way to scale down the story and still deliver something of value to the customer in the current release? If so, create appropriate user stories and discuss them with your product owner. The scaled-down version may just be an interim measure: What would you do if you had the time to do it right? Identify the user stories for these items and discuss them with your product owner. They may get into subsequent releases of the product.
The team can add any kind of work in the iteration plan. If an architecture spike is required for a feature, add a user story for it to the release backlog and have tasks such as "Design," "Building Prototype," or "Research." The good thing about adding a user story to the release backlog for the spike is that it allows the product owner to prioritize it. If the product owner feels the cost of doing the research is too high, the architecture spike story will be lower on the list, or the feature will be removed from the release.
Assigning story points to user stories
In the product backlog and release backlogs, the development team estimates and assigns story points to each user story. Why use story points instead of units of time such as hours? The main reason is that in the early stages of development, you don't know how much work you really have for a story. And if you do all the analysis and design up front, you may be preventing the design from evolving as you know more about the feature.
Another reason to use story points versus units of time is that the ideal time from one person to another can vary. If the person doing the estimate is not the person doing the work, the estimate may be wrong.
Finally, another reason to use story points is that what was originally an ideal-time estimate may be misinterpreted as an elapsed-time estimate. Elapsed time is the time it actually takes to do the work after factoring in all of the interruptions the employee has in a normal workday. So, for example, a task estimated at five days may actually take eight to nine days to complete once the daily meetings, e-mails, phone calls, and reviews are taken into account.
When using story points, what you are actually doing is comparing the relative complexity of one feature to another. If you take a story worth one point and compare it with a story worth five points, what you are essentially saying is that the second story is five times bigger to do than the first one. Notice that it was not stated that it would necessarily take five times more time to do.
There may be user stories that the team cannot estimate because there are too many unknowns. In these cases, the team can add a user story to the backlog to do some additional research. Once the research is done, the team can revisit the story to assign points.
Initially, it will be challenging to estimate story points because your team has no reference points. As the team does more and more estimates, those estimates will improve in accuracy. To do the story point estimates, you can use the Planning Poker technique.
Planning Poker is a consensus-based estimation technique for estimating effort or relative size of tasks in software development. It attempts to minimize anchoring (early comments by team members that "anchor" a task time by expressing it aloud). Each team member plays their estimate card (this card expresses the values 0, 0.5, 1, 2, 3, 5, 8, 13, 20, 40, 100 but not units; units are decided by the moderator) such that the card cannot be seen by the other players. After each player has played a card, all cards are exposed at once.
Velocity is the long-term tracking of how much work has been done by a team per iteration. It is measured in the same units as in the release backlog, meaning story points.
The team initially assigns story points for each user story in the release backlog. At the start of each iteration, the team selects the stories it will work on during the course of the iteration. To earn the points assigned to a story, the team must complete all tasks for that story within the iteration.
If the team does not complete one or more tasks of a user story, the team earns zero points for the story at the end of the current iteration, and the story will be deferred to the next iteration. The team will earn the points for the story in the iteration in which it completes the remaining tasks.
As the team completes iterations, tallies up the story points, and looks at the data of previous iterations, a picture such as the one in Figure 3 begins to emerge. Notice how the team does not earn the same number of points in each iteration.
Figure 3. Velocity over multiple iterations
The nice thing about velocity is that after a few iterations, you can start using the numbers in your tracking charts to extrapolate how much of your remaining release backlog items you will be able to complete.
Table 1. Story points per iteration
Table 1 shows story points per iteration. Using the data in Table 1, you can extrapolate the following information:
- The average number of story points per iteration is 33.
- The current velocity of the team is 35 story points.
- The average of the three slowest iterations is 30 story points.
So assuming that you have 6 iterations left in your release, you can begin to make the following assumptions:
- At average velocity, the team will complete an additional 198 story points (6 x 33).
- At current velocity, the team will complete an additional 210 story points (6 x 35).
- At slowest velocity, the team will complete an additional 180 story points (6 x 30).
In Figure 4, you can see the release backlog split into three sections. The middle section represents the work that can be contained in the last 6 iterations using the assumptions above.
Figure 4. Extrapolating where the work ends
What Figure 4 also communicates is that the development team cannot complete all the user stories in the release backlog. However, if the team always works from the top of the list down, the odds are high that the team has always been working on the most important items for the release.
The main goal of agile planning is having as much of the "known" work as possible on the table and visible to all. This article emphasizes the "known" because as you gain knowledge about what you are doing, you may be adding new stories to the backlog. This allows the product owner to continually prioritize the release backlog and ensure development is always working on what is most important.
Product owners usually survive even if not all stories make it into a release. They tend to get upset about it, though, when the team spends too much time on items that are not critical and then cannot fit all the important stories in the release because they ran out of time. So treat product owners as your customer and continually give them what they want first.
Any opinions I've expressed in this article are my own and not necessarily those of IBM.
- Check out these articles used in the
creation of this article:
- Agile Estimating and Planning (Prentice Hall PTR, 2005) is a guide to estimating and planning agile projects, complete with a philosophical and practical take on agile estimating and planning.
- User Stories Applied: For Agile Software Development (Addison-Wesley, 2004) focuses on the importance of user stories in agile development.
- Also see training for agile development.
- Planning Poker is a technique that helps distributed teams estimate together.
- The series "Architectural manifesto:
Adopting agile development" (developerWorks, March 2008-January 2009) can
help you decide if agile development is right for your needs:
- Part 1: About the processes, benefits, and requirements
- Part 2: How its used for different projects; customer effects
- Part 3: The role of stakeholders
- Part 4: Defining requirements
- Part 5: User stories and prioritization
- Part 6: From the customer's viewpoint
- Part 7: Estimating work effort
- Part 8: Modeling
- Part 9: The agile-SOA link
- In this podcast, "Scott Ambler on Agile development" (developerWorks, April 2007), IBM's Agile Development Practice expert explains this iterative and incremental approach to development, lays out the business case for it, and dispels some myths in the process.
- "Agile software development: A tour of its origins and authors" (developerWorks, March 2007) aims to show you what "agile" truly means.
- In the developerWorks Linux zone, find more resources for Linux developers (including developers who are new to Linux), and scan our most popular articles and tutorials.
- See all Linux tips and Linux tutorials on developerWorks.