The waterfall project methodology is deeply entrenched within many organizations, so much so that the decision to migrate to agile processes is often beyond the control of development teams who implement the code. This is particularly true of large projects that span several organizations.
Despite several propositions to move one recent major project to agile development, no progress was made in doing so until the development team responsible for one of the project's critical applications decided to move ahead and apply agile methodologies to those pieces of the project over which they had control. The result of this decision illustrated the value and benefits of agile development in a way that the greater project management team could appreciate and understand within the context of the overall project.
This article describes how that development team was able to successfully implement agile development within a larger waterfall project. This example, including the concepts and techniques that were implemented, the challenges that were faced, and the benefits that resulted, might help you find a similarly useful manner with which you could introduce agile concepts and processes to your organization.
This article assumes a general familiarity with basic project management, and with agile development concepts and terminology.
The particular application at the center of this case study is an existing sales contract application used by a worldwide sales team. The application has been in production for several years, and typically has three to four releases a year, both major and minor. The application’s client base is large and diverse, with 3,000 power users plus thousands more occasional users.
The ongoing project is managed by the CIO’s office, with solution leaders representing the clients. The various members of the overall project team -- users, designers, architects, developers, testers, production operations, and so on -- are from different organizations within the company.
The integrated development process for this waterfall project included these phases:
During pre-concept phase, the development team provides a rough order of magnitude (ROM) for proposed release content and (not committed) target dates. In concept phase, the client prioritizes the list of candidates for the release, business and system requirements are created and reviewed, and the development team provides level of effort (LOE) for business requirements (BRs) and system requirements (SRs). In plan phase, the SR candidate list is firmed up, and the development team builds low-level sizings for these requirements.
For this project, a typical schedule would be something like this:
- Concept phase: 2 months
- Plan phase: 2 months
- Development phase: 6 weeks
- Qualify phase: 6 weeks
The candidate lists of customer wants and needs (CWN), BRs, SRs, and, eventually, change requests, are constantly updated and reprioritized throughout these phases, even into the six week development phase.
In terms of technologies used in the application, it was built on a Service Oriented Architecture (SOA) platform utilizing the high availability capabilities of IBM® WebSphere® Process Server V184.108.40.206, IBM WebSphere Portal V6.1, and IBM DB2® 9.1.6. The application leveraged Web services to integrate with other internal company applications. In addition, WebSphere Process Server technology was utilized to manage business rules and process workflows. Coordinating with the requirements and release schedules of the other internal applications contributed to the complexity of planning and scheduling releases for the project application.
Why be agile in a waterfall?
A significant amount of time of each typical release of this application (four months out of a seven month cycle) was spent in the concept and plan phases. Even then, requirements still changed. The development team decided to pursue an agile approach because it accounted for changes to business priorities and needs, and for development re-factoring as the project evolved. Such an approach would enable the development team to eliminate significant time from the concept and plan phases, add more time to the development phase, and focus on delivering more function and higher quality.
Due to restricted deployment windows and the availability of users to participate in acceptance testing, deployment dates were limited and usually decided early in the concept phase. Lengthy planning and re-planning typically resulted in a shortened development phase, which in turn resulted in the development phase being consistently frantic and reliably late. Frequent changes to release priorities during the development phase did not help.
A major goal for the development team was to adopt better development processes to control the chaos from both external influences (late requirements, changing requirements, and so on) and their own poor procedures (inadequate development processes, low code quality, and so on). Adopting an agile approach would be a means toward changing their development culture and gaining control over their processes.
Beyond that, the development team was aware of the other benefits of the agile approach: improved quality, faster implementation, and greater customer satisfaction. They also believed that if they could move forward with agile processes, they would gain momentum in convincing the greater project to move to an agile methodology overall. Their resulting lessons learned would aid them in working out any kinks in their tools and processes, and speed up the eventual shift to agile processes.
How much agile methodology was really implemented?
When the project first started, the development team attempted to adopt some (though not all) of the concepts of the Scrum framework, an agile process based on empirical process control theory that focuses on transparency, inspection, and adaptation. The team implemented daily stand-up meetings and burndown charts, but chose not to adopt components to enable inspection, such as sprint planning, sprint review, and retrospective meetings.
As the project moved into the waterfall qualify phase -- in which the development team needed to handle a constant stream of testing defects -- the team’s scrum process ended. They would have to pick it back up when the next release started its development phase. This failure of the initial Scrum attempt was ultimately a consequence of not having adopted the processes of inspection; this lesson learned helped the team in their follow-on Scrum efforts.
In the development phase of a more recent release, the development team decided to start fresh and introduce the complete agile process based on the Scrum concepts and hallmarks. The team established a commitment to sprint planning, sprint review, and retrospective meetings. More importantly, they committed to process inspection and self-governance.
To fully adopt the Scrum way of doing things, the team needed to identify who on the scrum team would assume the three required roles of Product Owner, ScrumMaster, and Team. Initially, the development project manager was positioned as the primary Product Owner. The Team consisted of the existing development team members. A ScrumMaster was selected who established a sprint calendar of three- to four-week cycles, with start and end dates that didn’t necessarily line up with the waterfall process milestones; the team learned that sprint meetings scheduled close to waterfall milestones result in the team focus being shifted to one of these events, but not both, leaving an unfavorable result for the one that became neglected.
For the sprint planning meeting, the development team started with the product backlog of candidates that the client had identified (in the concept and plan phases) for future release of the application. The Product Owner selected a subset of these items, which constituted the possible product backlog to be considered for the sprint. During the sprint planning meeting, the development team committed to the product backlog items they believed they could complete during the sprint, and from that built the sprint backlog, which was the tasks, task estimates, and assignments that the team would actually work on during the sprint.
At the end of each sprint, the development team held a sprint review meeting. The purpose of this meeting was to demonstrate to the stakeholders what had been completed in the sprint, and provide a timely opportunity for the stakeholders to comment and ask questions, possibly resulting in a list of desired changes. Such changes and any missing or incorrect functionality were added to the product backlog for the next sprint.
At the end of each sprint, the development team also held a sprint retrospective meeting to review what went well during the sprint and what could be improved upon. Based on this feedback, they changed their processes as appropriate and added non-functional action items to the product backlog.
The typical schedule was to hold the sprint review and sprint retrospective meetings on the same day at the end of the sprint, followed by the sprint planning meeting for the next sprint on the following day. These meetings were conducted within the Scrum time-boxing guidelines.
Client representatives (solution leaders) attended the sprint planning and sprint review meetings. Initially, the client’s interests were represented by the development project manager, but in later sprints, the development team was able to pull in the solution leaders and get direct client input and feedback through these key participants. This approach was purposeful; the team wanted to understand the processes and methodology well before expanding the circle of participants to include the solution leaders.
- User stories
The development team also incorporated user stories into their planning. Although the client representatives (solution leaders) should be responsible for creating user stories, the development team did this instead, based on existing requirements, because the greater project was not yet committed to agile processes. The team then engaged the solution leaders in validating the user stories, collaborated to refine them and gain a greater understanding of the requirements. This practice has greatly improved the understanding and accuracy of the requirements, and greatly reduced rework after the fact.
- Quality assurance
While the development team has long used continuous integration, they recently incorporated automated testing into the development process. Work items were not considered complete until an automated test was available and merged into the testing tool. Daily builds and automated testing ensured that the code base was not broken as code was checked in. This resulted in higher quality code and fewer defects being found in the qualify phase. Coding items were not considered complete until the functionality could be demonstrated on a live server. Non-coding items, such as new processes or documentation, were published on the development team’s internal wiki and distributed to the team for review prior to being presented as complete at a sprint review meeting.
Other agile elements
Because the development team operated within a rigid, waterfall process, they could not adopt all aspects of agile development. The team still went through a requirements cycle of refining CWNs to BRs and SRs, prioritizing, ROM, LOE, and low-level sizings, adding new requirements, and circling around repeatedly.
The development team was able to drive some improvements in the requirements cycle; for example, they were able to move to prioritized lists vetted by the client representatives, and to size users stories at different points in the requirements cycle to build candidate scope lists for client review. The team leveraged the Scrum processes to address this activity and generated the needed work products for the requirements cycle, ultimately reducing the dreaded requirements churn during the development phase.
When the project moves fully to agile processes, the expectation is that user stories will be built early in the cycle, prioritizing will be limited to one or two cycles, and more time will be spent in the development phase. This state of agile adoption supports reviewing the implemented features with the client and updating the functionality based on their feedback, rather than reworking the various levels of requirements multiple times on paper.
Rather than completely rely on the sprint backlog and burndown chart tooling to assign and track work, the development team used a traditional Microsoft® Project plan to schedule tasks during the development phase. There were a couple of reasons for this. First, this demonstrated to the client that resources allotted to the project were fully utilized to deliver as much content as possible. Second, Microsoft Project was the expected and more recognizable form of communicating status in a waterfall project. However, by referencing burndown in addition to a project plan in status updates, the development team was taking steps toward exposing and transitioning the other teams to agile tracking methods.
Of course, working with two inherently different styles (and cultures) of project management presented a number of challenges to the development team:
A key challenge in transitioning from a waterfall process to an agile process is bridging the terminology. For this project, the development project manager did an excellent job of doing this translation, both in terms of “speaking waterfall” to the client and in minding the gap between tools and status in the agile world to their counterparts in the waterfall world. With this “bridge” in place, the development team had more freedom to adopt their own processes and take the incremental steps described here.
The development team’s standard mode of operation was to constantly be in a sprint cycle -- whether or not they were specifically working on a release candidate. Therefore, the sprints covered the sizing work, production support activity, and qualify work. Because of this, the team tried to not have the sprints start and stop on the development cutover milestone (DCUT). Similarly, it was useful to not have the sprint end on the DCUT date; taking time at that point for the required two days of meetings was not conducive to getting the target release into qualify phase. The sprint typically ended the following week.
Initially, the development team struggled with the inevitability of handling production support within the sprints. Some of the developers were dedicated to handle Level 3 Support, but the majority of the production defects that occurred needed to be addressed by members across the development team. While all of this work could not be fully planned, a general level of support-related work could be anticipated. This work was addressed in two parts: analysis and resolution. The ground rule that helped move the team forward with integrating production support into the Scrum model was to set the priority of any production defect to the highest possible value, thus initial analysis trumped anything else on a developer’s plate or task list.
On the analysis side, when a production defect was created, the ticket was added to the current sprint and assigned based on severity. Root cause analysis was performed to determine the effort involved in providing a solution, and this analysis resulted in identifying the cause of the problem (or if not obvious, what was needed to identify the cause), an estimated sizing to implement the fix and create a unit test, and the difficulty of implementing the fix. Once this was done, the ticket was removed from the current sprint and the Product Owners determined the priority in relation to other items in the product backlog. Root cause analysis was targeted to be performed within existing Service Level Agreement requirements.
The Product Owners then assigned the issue to a sprint backlog (which could be the current sprint, or the product backlog for a future sprint), which enabled the fix to then be delivered within a release, a fix pack, or via a hot fix.
The development team estimated ten defects per week during the two weeks after a new release was deployed, and three defects per week otherwise, with four hours for analysis.
Periodically, production issues resulted in a change request. The development team’s guidelines set forth that these must be prioritized by the Product Owner for inclusion in a sprint. If the Product Owner believed a change request was the highest priority for the development team, the ScrumMaster and Product Owner decided whether the current sprint should be adjusted or (in a rare case) halted to accommodate it.
A subset of the development team was an infrastructure team that handled build tools, server administration (both test and production), database administration, and so on. This team had assignments within the sprint, but they also periodically received work requests from other teams on the project. This violated the principle of Scrum which states that the team is to be left alone to perform their sprint work.
To alleviate this problem, the development team enlisted three tactics:
- The team identified infrastructure tasks that should be included in every sprint and included them in the sprint backlog (for example, alpha deployments, beta deployment support, production online backups, and so on).
- Any infrastructure tasks related to product backlog items were identified and included in the sprint backlog.
- The Infrastructure Team Lead was designated as the focal point for external requests for infrastructure skills, worked with the Product Owner to determine the priority of such work items, and included them in the sprint backlog as deemed necessary.
Infrastructure team members learned to refuse the interruptions, deferring to the process above for such requests. This enabled them to remain fully engaged in the Scrum process.
The development team eventually moved to a hierarchy of Scrum teams because of its size, dividing into four teams:
- Business Process/Backend Services
- User Interface
- Life Cycle Management
These teams conducted daily scrums among themselves. A biweekly scrum of scrums was held in which one representative from each of the Scrum teams participated and summarized their team’s status since the last meeting, the team’s plans before the next meeting, any impediments that existed, and what obstacles they would impose on other Scrum teams.
The development team utilized a bug/issue tracking system that provided an integrated Wiki with convenient reporting facilities. IBM Rational Team Concert™ provides this functionality. The team created tracking tickets for all development work items and generated burndown charts (among other reports) for the sprints. The team was also able to extend the tool and the Wiki to accommodate the team’s requirements. For example, the team created a tool for user stories that was incorporated into the Wiki.
The development team heavily utilized a development Wiki to document processes, provide access to progress reports, and interface to their tracking system. Because of the interface to the tracking system, the Wiki was a particularly good mechanism for documentation and collaboration.
The development team built their project plans in Microsoft Project. The project plan referenced a tracking ticket for each low-level task included in the plan; this was an evolution that started before moving to agile processes. Instead of referencing a CWN with its associated hours, a tracking ticket was created for each task involved in delivering a CWN in a release. Where possible, tasks were subdivided and hours were distributed among two or more developers to increase productivity and reduce overall coding duration. Each subdivided task received its own tracking ticket and was individually referenced in the project plan (the intent was that each ticket/task should have a maximum of one owner). The project plan also identified the priority of tasks among individual developers and the overall priority of release candidates. A custom priority field was created on the tracking tickets, updated based on the project plan, and dictated to each developer in the order they should perform their tasks.
As expected -- and as advertised -- adopting agile processes resulted in improved code quality, faster implementation, greater customer satisfaction, and better development processes. The development team achieved other benefits as well, including some that are not usually thought of as agile benefits:
- Handling large tasks
A key benefit of adopting Scrum was that it has enabled the development team to fit big ticket items into releases. This might seem counter-intuitive; after all, according to some, limiting sprints at two to four weeks eliminates the ability to handle large pieces of functionality. On this project, large features were hard to fit into the prioritized list for a release; the client often preferred ten smaller items to one large item, even if that large item was important. Scrum forced the development team to regard these big features incrementally, breaking them down into analysis or research, and then incremental implementation. The team was able to fit the analysis or research into sprints not tied to the development phase, providing them with the discipline to finish the upfront tasks, and making it easier to then fit the incremental implementation into the release priorities.
This approach was utilized for non-functional items as well, such as code re-factoring and automated integration testing. Convincing the client that such work was important was challenging, but the development team was able to break the re-factoring work into small pieces, build a gradual plan for the changes, and then fit the incremental changes into sprints and releases.
Related to this is the benefit of gaining the ability and discipline to deconstruct large tasks; sprints forced the development team to think in terms of smaller-sized tasks. With smaller tasks, the team was better able to plan release content, as well as reallocate resources in the event that tasks slipped. Large tasks tended to require a broader range of skills and expertise, but when broken down into smaller tasks, many of these could be assigned to developers with narrower skills, which enabled greater flexibility in assigning resources. After several releases and multiple sprint planning sessions, this “small task” thinking had become ingrained on the entire team.
- Process improvement
The Scrum process also provided a framework for incremental process improvement that the development team previously lacked. The sprint retrospective meetings provided useful feedback, and process changes were added to the next sprint. In some cases, such as production support and the infrastructure team’s tasks, it took several sprints to refine the processes to the point of success. The team committed to the process, worked through the issues, and built processes that work well for the team.
- Handling change
The development team became better able to deal with change, even mid-sprint. During one release, they began the development phase with a clear understanding of the priorities, then in the middle of a sprint, the client came forward with a new number one priority. Normally such churn could demoralize a team, but they utilized the principles of Scrum to talk about the options (halting the sprint and starting over versus carrying on with an incorrect priority list). Because the development team was empowered to make the decision, they were able to shift gears and accept the new priority with success.
- Work environment
The adoption of Scrum processes had a significant impact on team morale, team discipline, and team interaction. Because the development team was empowered to improve their own processes, they strongly committed to the processes and put energy and enthusiasm into everything. Creativity was applied to process improvements, and this creativity carried over into the team’s development efforts. In addition, the updated processes greatly improved the development cycle, reduced the panic at the DCUT milestone, and reduced the number and severity of defects found in both the qualify phase and in production. The release process became more of a steady, well-planned effort, and less a time of the chaos, as the team had experienced before. Work hours were more predictable, even as the team hit key milestone dates, which improved morale and reduced tension between the developers and the other teams working on the project.
With the value and benefits of agile development made clear – and with impressive results -- the overall project intends to move to agile processes using Scrum with the next release of the application; a kickoff agile workshop for that release has just been held. The development team will be able to easily retain their development processes, adjusting in minor ways to accommodate the larger agile processes.
As part of the development planning and tracking, the development team has a lengthy backlog of candidates (CWNs and defects) that did not make the cut. These will be stored in the team’s issue tracking tool and will be pulled into the new agile processes for consideration in a future release.
A big change for the development team will be the development of user stories by the Product Owners. This effort started with the kickoff agile workshop, using Rational Team Concert as the tool of choice. Because the user story process provided higher quality requirements, the Product Owners have committed to this and driving it will improve the processes greatly.
Another significant change will be shifting the Scrum teams from the current silo structure (based on functional areas in development) to a structure that includes representatives across the project. Previously, since Scrum was limited to the development team, the functional division was natural. During the kickoff, the team structures were discussed and mapped out for the next release. The development team has taken steps to move to a mixed-team structure for the current release, and they will review and revise the process as they go, applying lessons learned to the next release.
The development team plans to become a more test-driven development shop, and has taken steps to that end with the test team (which is in separate organization). The development team will work with the test team, bring them into their sprints, teach them the Scrum concepts, and jointly work toward the goal of agile adoption across the project, regardless of timeline.
Finally, future plans include evangelizing Scrum and agile principles to the partner teams across the overall project. The development team’s experience puts them in a unique position to train others, not only because they have the agile knowledge and have learned the lessons, but also because they are the most knowledgeable with the workings of the application, and have lived the cultural experience of using agile development in the organization.
- Scrum Guide from Scrum Alliance
- Agile Project Management with Scrum, Ken Schwaber, Microsoft Press, 2004
- Agile and Iterative Development: A Manager's Guide, Craig Larman, Addison-Wesley, 2003
- An Introduction to Agile, Kevin Aguanno, Multi-Media Publications, 2005
- Education: Succeeding with Agile
- IBM developerWorks WebSphere
- IBM WebSphere Developer Technical Journal