From Traditional to Iterative to Agile to Disciplined Agile
Getting back to the theme of agile and architecture (before I took a detour into leadership!) - I've been trying to pull together a simple visualization of the practices I apply when helping organizations improve their development capability. In a previous post I mentioned some "traditional" practices that are part of our architecture heritage: Multiple Views, Quality Attribute-Driven Development, Component-Based Development, Asset Reuse, Decision Capture and Architecture Proving. Let me complete the story.
In short, here are the key practices that I believe iterative development has given us:
- Iterative Development. As a project progresses, releases provide incremental improvements in capability until the final system is complete. An iterative development process is similar to "growing" software, where the end product matures over time. Each iteration results in a better understanding of the requirements, a more robust architecture, a more experienced development organization, and a more complete implementation.
- Risk-Value Lifecycle. There is more to an iterative development process than a stream of iterations; there must be an overall framework in which the iterations are performed, which represents the strategic plan for the project and drives the goals and objectives of each of the iterations. Such a framework is provided in the Rational Unified Process, whose phases are labeled Inception, Elaboration, Construction, and Transition. Each phase concludes with a major milestone and an assessment to determine whether the objectives of the phase have been met. A satisfactory assessment allows the project to move to the next phase. Phases also answer the question “what should an iteration focus on?” RUP phases support a risk-value approach that results in risk being removed from the solution early in the lifecycle. This is particularly manifest in the Elaboration phase whose key measure is a stable architecture and which is focused on removing technical risk, ensuring that those elements of the solution that are costly to change are considered sooner rather than later. A phase-based approach therefore supports convergence of a number of elements as the project progresses. For example, risks reduce over the project lifecycle, and any cost and schedule estimates become more accurate.
The RUP (and other iterative methods) introduce other practices that are also relevant to the work of the architect. Although these practices apply equally to plan-centric (traditional) methods, they have been given a particular emphasis when applied in conjunction with an iterative approach:
- Shared Vision. This practice ensures that all stakeholders, both consumers and producers of the solution, share a common view of the problems being solved and the key characteristics of the solution (albeit at a high level). In essence, this practice is focused on aligning expectations and ensuring market acceptance. This practice ensures that there is an alignment between product strategy and the output from the development team.
- Use Case-Driven Development. The practice of describing primarily functional requirements with use cases is well-documented. Aside from the technique itself, use cases (and flows through each use case) make natural units of implementation in an iterative development approach since use cases influence planning, architecture, development, and testing.
- Release Planning. This practice is focused on the just-in-time project planning needed to scope the release of executable software within an iteration. This iteration-specific planning complements any high-level planning that considers the project as a whole.
Although the foundations of the agile movement are well known (Manifesto for Agile Software Development, 2001), my own reaction was somewhat-mixed once mainstream agile methods appeared – especially since they seemed to reinvent the wheel (and terminology) in parts. Why did Scrum use the word “sprint” when “iteration” would do? I guess it was all part of the story. Let’s face it, rugby players sprint when going for the line. I can’t imagine for one minute a rugby player receiving the ball and then iterating as fast as he could :)
“We now have the agile method wars – branded methods based on metaphors, unique terminology, and sometimes questionable certification programs”. (Ambler & Lines, Disciplined Agile Delivery, 2012)
Over time, of course, agile methods have proven to add incredible value. In particular, I believe they ground us in what matters most in building software in a timely, cost-efficient and quality manner. Whether we’re talking Extreme Programming (XP), Scrum or some other agile method, they each live and breathe certain principles and each draw upon practices that have stood the test of time. So what’s so different about agile? Specifically, how does it differ from iterative development? The fact is – agile approaches introduce new practices and for good reason. Schwaber tells us of his experiences in working with the process theory experts at the DuPont Experimental Station:
“They were amazed and appalled that my industry, systems development, was trying to do its work using a completely inappropriate process control model. They said systems development had so much complexity and unpredictability that it had to be managed by a process control model they referred to as ‘empirical’ … I realized why the industry was in such trouble and had such a poor reputation. We were wasting our time trying to control our work by thinking we had an assembly line when the only proper control was frequent and first-hand inspection, followed by immediate adjustments”. (Schwaber & Beedle, 2002)
Of course, we could say that iterative development addresses this finding. However, agile methods take the notions of continuous delivery, team development and stakeholder interaction to a new level. Here's what I see as the key practices that distinguish an agile method from other methods and, in particular, iterative development. Of course, there are other practices too, some that are method-specific and others that are not, but they are not as core as those listed below.
- Test-Driven Development (TDD). The “test first” approach advocated by TDD is primarily targeted at programmers, but is a cornerstone of any agile method. Creating tests that, essentially, define the specification of what the code should do first, helps focus programmers on meeting this specification. “This develops into a natural and efficient rhythm –test, code, refactor, test, code, refactor”. (Beck, 2005)
- Continuous Integration. This practice encourages frequent the integration and testing of programming changes. “Team programming isn’t a divide and conquer problem. It is a divide, conquer, and integrate problem”. (Beck, 2005)
- Refactoring. This practice is focused on changing an existing body of code in order to improve its internal structure. In a sense, this practice is focused on addressing technical debt, albeit at a local level (since it is typically applied to isolated bodies of code, rather than the system as a whole). In practice, any teams that also perform a level of design (and create models) also update these designs where relevant.
- Whole Team. Agile methods focus on the value of highly-collaborative teams as exemplified by Scrum’s daily standup meeting. It is also the team that decides how the project goals will be met and will self-organize accordingly. This means that team composition may change over time since the team, as a whole, should always have the right skills at any point in time to complete the tasks at hand. This practice also instills of sense of collective ownership and responsibility.
- User Story-Driven Development. This practice describes the capture of both functional and non-functional requirements in a lightweight manner (more lightweight than use cases) and encourages collaboration with the relevant stakeholders throughout a project. User stories influence planning, development, and testing.
- Team Change Management. This practice supports the logging of defects or new requirements, by any member of the team, that are within the scope of the current iteration. Such requests are captured as work items and placed on a product backlog (as exemplified by Scrum) which is ultimately reviewed by the team as part of their planning.
In an ideal world, processors would also be infinitely fast and disk drives would have infinite capacity. Teams would also be co-located. The point is – there are real-world considerations that limit the art of the possible in an agile world. For example, when a high degree of complexity is involved in large-scale efforts, some have noted the need to introduce new techniques and, in particular, the importance of architecture:
“When developing complex, large-scale applications, many have reported that agile methods must be adapted to include more kinds of architectural information. We see evidence of this in the zero-feature release, the architectural spike, and agile practices that recognize the architect role”. (Nord & Tomayko, 2006)
“Architecture is an important topic, especially when development is distributed over several sites, and/or when the development is based on legacy systems”. (Pei Breivold, Sundmark, Wallin, & Larsson, 2010)
Scaling factors, such as team distribution and solution complexity, apply to both small and large organizations, and a consideration of these factors can be found in the Disciplined Agile Delivery (DAD) approach, a hybrid of agile and traditional methods as documented in (Ambler & Lines, Disciplined Agile Delivery, 2012). This approach introduces an Agile Scaling Model (ASM) which considers various factors and I suggest you read the book which is quite excellent.
“These factors force adjustments in important areas to the practical
realizations of agility at scale. They encourage a disciplined approach
to the scaling of agile techniques. My experience with several large
enterprise software delivery organizations suggests that agile
approaches can be successful in almost every kind of development team
and in almost all organizations. Each situation has its own specific
needs and variations, yet the basic core agile principles can provide
significant results when applied with care, and with realistic
expectations”. (Brown, 2012)
A consideration of these scaling factors has resulted in the introduction of additional practices. In several cases, these are not new practices at all but recognition that large, complex, enterprise-wide distributed development and the like require us to embrace certain approaches, even though they take us away from an idealized view of agile. Some examples are:
- Measured Performance. This practice allows project and portfolio-level measurements to inform key business decisions. For example, knowing that the burndown of a project’s product backlog is not on a downward trajectory may indicate that more (or better-skilled) resources are required on the project. Similarly, observing a lot of requirements churn (changes to already-defined requirements) may indicate that the project’s business representative may not have the domain knowledge required. Measured performance is good for both the project team and the business.
- Formal Change Management. This practice provides a controlled approach for managing changes and is (as the name suggests) more formal than the Team Change Management practice. This practice is often applied when approval is required from stakeholders outside of the project team, or when a deliverable has been baselined as part of a contract and the deliverable needs to be modified.
- Concurrent Testing. While a project team may embrace test-driven development, there is often an independent test team present (especially in larger organizations) that typically provide a level of user acceptance testing before the solution is put into production. The purpose of this practice is to bring this “external” team in sync with the project team and have them work in step with the project so that their work isn’t compressed into a separate activity at the end of an iteration or release.
Concluding a whirlwind tour of the heritage of certain practices, a summary of waterfall, iterative, agile and disciplined agile practices that have been mentioned is given in the table below, which I use for two key purposes:
In fact, standard configurations of the alignment of architecture practices and agile practices are emerging – Nord and Tomayko show an alignment of the Software Engineering Institute’s approaches (such as Attribute-Driven Design) within XP (Nord & Tomayko, 2006) and Madison discusses the application of architecture practices within a “hybrid of Scrum, Extreme Programming, and sequential project management”. (Madison, 2010)
- Enablement. Considering the set of practices can help practitioners in their agile journey. Conveying the principles and practices of agile to a practitioner that has used an iterative approach such as RUP is a very different affair from someone that’s only ever followed a waterfall approach. This then translates into a different focus of any enablement activities.
- Practice selection. It should have become clear by now that there is no one-size-fits-all method that caters for all situations. Furthermore, the set of practices discussed in this article can be seen as a list from which to choose for any given situation. In particular, we can see that, rather than viewing waterfall, iterative, agile and disciplined agile as competing approaches they are, at least from a practice perspective, highly complementary in that the practices they espouse can be combined as required.
“Our experience is that “core” agile methods such as Scrum work wonderfully for small project teams addressing straightforward problems in which there is little risk or consequence of failure. However, “out of the box”, these methods do not give adequate consideration to the risks associated with delivering solutions on larger enterprise projects, and as a result we’re seeing organizations investing a lot of effort creating hybrid methodologies combining techniques from many sources”. (Ambler & Lines, Disciplined Agile Delivery, 2012)
“Although each approach has a home ground of project characteristics within which it performs very well, and much better than the other, outside each approach’s home ground, a combined approach is feasible and preferable”. (Boehm, Get Ready for Agile Methods, with Care, 2002)