Agility@Scale: Strategies for Scaling Agile Software Development
ScottAmbler 120000HESD Tags:  agility-at-scale xp outside-in-development modeling scrum disciplined-agile-deliver... agileadopt amdd analysis agile 3 Comments 7,177 Visits
The popular Agile literature can often seam naive when it comes to how Agilists work with project stakeholders:- Extreme Programming (XP) has a practice called On-Site Customer where one or more people work closely with your team to provide information and to make decisions in a timely manner.- Scrum has the role of Product Owner who is the one single person that the development team goes to for decisions about requirements. - Agile Modeling (AM) has the practice of Active Stakeholder Participation which extends On-Site Customer to get the stakeholder(s) actively involved with the modeling effort through the use of inclusive tools and techniques.
These are great strategies for small, co-located teams doing straightforward development, but they quickly fall apart at scale. This occurs for several reasons:1. Stakeholders are a diverse group. Your stakeholders include end users, business management, project funders, enterprise architects, operations staff, support staff, other system development teams, and many others. Different people have different, and often contradictory, requirements and they certainly have different priorities. It's questionable whether a single person, or a handful of persons, can adequately represent this diverse group.2. One person becomes a bottleneck. Even with a small co-located team this is a problem, let alone one that is geographically distributed or one that is very large. There's no way that a single person can be available 24/7 in a responsive manner to support distributed teams.3. It's a difficult role. The Product Owner/Customer (POC) is responsible for representing the business to the development team. They're making important decisions on a regular basis, decisions which they'll be held accountable for.4. One person becomes a serious project risk. Not only is it questionable whether a single person can fairly represent all stakeholders, even if they could what happens if you lose that person? They effectively become a single point of failure for your team.
To scale this role, consider the following strategies:1. Recognize the true scope of the POC role. Not only are they stakeholder proxies they also are a development team representative to the stakeholder community as a whole. As stakeholder proxies they'll make decisions and prioritize the work, they'll run requirements elicitation sessions, they'll negotiate priorities, and they'll put the development team in contact with stakeholders who have expertise in specific aspects of the domain. As team representatives they'll often demo the current version of the system to other stakeholders, communicate the status of the project to people, and respond to various requests for information from the stakeholders.2. Have multiple people in it. A single POC works well for small, co-located teams developing simple software. At scale you'll soon discover that you need multiple people in this role so that they don't become a bottleneck. For distributed teams it's common to see each subteam have one or more POCs who are managed by a primary/chief POC. The primary POC typically works on the coordinating team with the chief architect (I'll talk about this role in a future blog posting) and the program manager (also a topic for a future blog posting).3. Train them in business analysis skills. The person(s) in the POC role need good business analysis skills. If fact, it's common for people who were formerly BAs for traditional teams to step into the POC role, particularly with BAs who originally come from the business side of your organization. This strategy has its advantages and disadvantages. As a BA they've likely got solid business knowledge but their instincts may motivate them to take a documentation-driven approach to providing information to the development team instead of a collaboration-based approach. Be careful.4. Consider the full system development lifecycle. There's far more to the POC role than supporting the development team during Construction iterations. During "Iteration 0", the Inception phase for an Agile RUP project or the warm-up phase for an Eclipse Way project, the POC(s) will often lead the initial requirements envisioning efforts. The product backlog, or better yet your work item list, needs to come from somewhere after all. During the release iteration(s), the Transition phase for RUP or the End-Game phase for Eclipse Way, the POC(s) will focus on communicating the upcoming release to the stakeholder community, will be actively involved with any final user acceptance testing (UAT), and may even be involved with training end users.
In my January 2008 column in Dr Dobb's Journal, posted at http://www.ddj.com/architect/204801134 , I provide detailed advice about how to scale the way that you work with stakeholders on Agile projects by applying the practices of Agile Model Driven Development (AMDD). There's no magic solution, you just need to choose to organize yourself effectively. The good news is that you can easily work with stakeholders at scale.[Read More]
It's customary to start a blog by describing the vision for it. Although this vision will undoubtedly evolve over time, it's always good to put a stake in the ground to get things started. Agile software development is clearly taking off and in my opinion is becoming the dominant development paradigm. Furthermore it appears that Agile approaches enjoy a higher success rate, providing better value for your IT investment, than do traditional approaches. Although organizations are succeeding at simpler projects with agile, many are struggling when applying Agile in more complex situations. They're finding that the "Agile rhetoric" doesn't always live up to its promises once you move into these complex situations. My goal with this blog is to share strategies for applying Agile techniques at scale.
When applying Agile strategies at scale you are likely to run into one or more of the following complexity factors:1. Geographical distribution. Is your team, including stakeholders, in different locations? Even being in different cubicles within the same building can erect barriers to communication, let alone being in different cities or even on different continents.2. Regulatory compliance. Regulations, including the Sarbanes-Oxley act, BASEL-II, and FDA statutes, to name a few, can increase the documentation and process burden on your projects. Complying to these regulations while still remaining as agile as possible can be a challenge.3. Entrenched policies, people, and processes. Most agile teams need to work within the scope of a larger organization, and that larger organization isn't always perfectly agile. Hopefully that will change in time, but we still need to get the job done right now. Your existing culture and organization can really hinder your ability to scale agile approaches, then a few "simple" changes can really help your efforts.4. Legacy systems. Although the politically correct term would be "proven assets" the reality is that it can be very difficult to leverage existing code and data sources due to quality problems. The code may not be well written, documented, or even have tests in place, yet that doesn't mean that your agile team should rewrite everything from scratch. Some legacy data sources are questionable at best, or the owners of those data sources difficult to work with, yet that doesn't given an agile team license to create yet another database.5. Organizational distribution. When your teams are made up of people working for different divisions, or if you have people from different companies (such as contractors, partners, or consultants), then your management complexity rises.6. Degree of governance. If you have one or more IT projects then you have an IT governance process in place. How formal it is, how explicit it is, and how effective it is will be up to you. IBM has been doing a lot of work in this topic over the past few years, and just recently Per Kroll and I have done some work around Lean Governance strategies. 7. Team size. Large teams will be organized differently than small teams, and they'll work differently too.8. System complexity. The more complex the system the greater the need for a viable architectural strategy. An interesting feature of the Rational Unified Process (RUP) is that it's Elaboration phase's primary goal is to prove the architecture via the creation of an end-to-end, working skeleton of the system. This risk-reduction technique is clearly a concept which Extreme Programming (XP) and Scrum teams can clearly benefit from.
It is definitely possible to scale Agile software development to meet the real-world complexities faced by modern organizations. Based on my experiences, I believe that over the next few years we'll discover that Agile scales better than traditional approaches. Many people have already discovered this, but as an industry I believe that there isn't yet sufficient evidence to state this as more than opinion. My goal with this blog is to provide advice for scaling Agile so as to increase your chances of success.
So, it looks like I have my work cut out for me. My strategy will be to address common questions which I get when working with customers and with internal IBM development teams. I have the privilege to work with a variety of software development teams worldwide, helping them to become more agile. They're all struggling with the same basic issues although don't recognize it because they're too focused on their own situation. So hopefully I'll be able to spread the word about what's actually working in practice.
I hope that you stay tuned.
- Scott[Read More]
I'm happy to announce that I've accepted the role of Managing Director of the Scrum Alliance (SA), a part-time position in addition to my duties here at IBM. On the surface this must appear to be a radical and unpredictable departure for me, considering my history of being critical when it comes to some of the past activities of the Scrum Alliance. To be fair, I've actually been critical of the Certified Scrum Master (CSM) scheme, and rightfully so. But I have also actively embraced the good ideas contained in Scrum and have incorporated them, with attribution, in my writings about Disciplined Agile Delivery (DAD) and other agile topics. I believe that I've made this very apparent in this blog and in other sources such as the Agile Modeling site. So, it really isn't such a radical departure for me afterall, although still arguably one that was difficult to predict. In fact, one of the reasons why the Scrum Alliance approached me to be Managing Director is the fact that I have been critical of many of the Scrum community's behaviors.
So, over the next few months you're going to see what I believe to be some welcome changes at the Scrum Alliance. Our first step at serving you better will be to apply agile strategies and principles in the way that we work. Importantly, we'll be taking a three pronged strategy based on respect, clarity, and integrity. We have dubbed this strategy "Scrum Alliance 2.0".
To be more respectful of existing and potential SA members, we will begin executing the following activities:
Clarity through openness and honesty
We are also starting to execute on four activities for improving the clarity of how we operate:
The fundamental idea here is that the Scrum Alliance should have nothing to hide from our membership. We've preached open and honest communication for years, now we're going to start actually living by those words. Yes, it may be a bit painful to work to this level of clarity, but we feel that you deserve this.
Integrity through actions, not words
Finally, we're taking three actions to increase the overall integrity of the Scrum community:
As I hope you have guessed by now this blog is an April Fool's joke. I have no intention of becoming the Managing Director of the Scrum Alliance and my condolences go out to anyone who would take on this position. This blog posting does however reflect what I would do to bring greater clarity, integrity, and respect to the Scrum community. The Scrum Alliance can and should choose to do a lot better. I hope it has been food for thought.
My current agile newsletter for Dr. Dobb's Journal, How Agile Are We? discusses some of the results from my recent How Are Are You? survey. The survey investigated how well "agile teams" met the following five agile criteria:
Some interesting results include:
I believe that there are several important implications:
ScottAmbler 120000HESD Tags:  regulatory-compliance agility-at-scale asm agile 2 Comments 12,403 Visits
One of the scaling factors called out in the Agile Scaling Model (ASM) is “regulatory compliance”. This name is a bit of a misnomer because this scaling factor really addresses two issues: complying to regulations imposed upon you from external sources and choosing to adhere to internal regulations willingly adopted by your organization. It is relatively common for agile teams to find themselves in such situations. For example, in the 2009 Agile Practices Survey one third of respondents said that they were applying agile on projects where one or more industry regulations applied.
First let’s consider external regulatory compliance. In these situations you may face the need to undergo an audit by an external regulatory body with consequences for non-compliance ranging from anywhere to a warning to a fine or even to legal action. Sometimes even a warning may be a grave thing. A few years ago I was working with a pharmaceutical company which had discovered that a warning from the FDA for non-compliance with their CFR 21 Part 11 regulation, when reported in major newspapers, resulted on average in a half-billion dollar loss to their market capitalization as the result of a dip in their stock price. There are financial regulations such as Sarbanes-Oxley and Basel II, informational regulations such as HIPAA which focuses on health information privacy, technical regulations such as ISO 27002 for security practices, and even life-critical regulations such as some of the FDA regulations.
External regulations are typically managed by a government organization or industry watchdog will range in complexity and can have a myriad of effects on project teams. For example, you may need to be able to prove that you had a documented process and that you followed it appropriately; you may need to produce extra artifacts, or more detailed artifacts, than you normally would; you may need to add extra features to your solution, such as tracking financial information, that you wouldn’t have normally implemented; you may need to produce specific reports to be submitted to the regulatory body; or you may even need to submit your team to audits, sometimes scheduled and sometimes not, to ensure regulatory compliance. Interestingly, even though many of those requirements go against the agile grain, the 2009 Agility at Scale Survey found that organizations were successfully applying agile techniques while still conforming to external regulations. So yes, it is possible to scale your agile strategy to address regulatory compliance.
Second, let’s consider compliance to internally adopted, or sometimes even developed, “regulations” which you will be potentially evaluated/appraised against. Perfect examples of these are process improvement frameworks such as CMMI and ISO 900x. Similar to external regulations, the 2009 Agility at Scale Survey found that some agile teams are succeeding in situations where they have chosen to adopt such frameworks. It’s important to note that frameworks such as CMMI aren’t primarily about ensuring the compliance of development teams to a standard process, regardless of what CMMI detractors may claim, but instead about process improvement. Process improvement at the IT department (or beyond) is an enterprise discipline issue from the point of view of ASM, implying that frameworks such as CMMI affect more than one scaling factor.
When you find yourself in a regulatory situation, whether those regulations are imposed or willingly adopted, the best advice that I can give is to read the regulations and develop a strategy to conform to them in the most agile manner possible. If you let bureaucrats interpret the regulations you’ll likely end up with a bureaucratic strategy, but if you instead choose to take a pragmatic approach you will very likely end up with a very practical strategy. Part of that strategy is to treat the regulatory representative(s) within your organization as important stakeholders whom you interact with regularly throughout the project.
The SEMAT vision has recently been posted online. SEMAT is short for Software Engineering and Method and Theory (SEMAT) and I am one of several signatories and a provider of input into the effort.
There are several reasons why I'm involved with SEMAT:
I have several thoughts about the SEMAT vision which I'd like to share with you:
The Scrum community has adopted a different set of terms than the other agile methodologies. This is done on purpose to help people realize that Agile approaches are different than traditional approaches, which can help in their adoption, but it can also hinder people's understanding because some of the terminology is not only non-standard it really doesn't make much sense. Because of this I'm often asked by people that I'm coaching to convert back and forth between terms, and recently wrote a detailed article on the subject. The following summarizes the mapping:
For more details read my article Translating Scrum Terminology which includes explanations of a wider range of Scrum terms and discussions of why some of them really are questionable.
Recently I visited a customer who had adopted Scrum. They were a few sprints, what Scrum calls iterations, into the project and were running into some difficulties. Although I was primarily brought in to educate senior management on disciplined agile software development, I was also asked to sit in on the team’s daily stand-up meeting so that I could hopefully provide some suggestions as to how to address the problems they were running into.
Their work area was fairly typical. They had some whiteboards which they were using for project planning and tracking, with sticky notes to indicate what work had been taken on by each team member. The current status of the task (not yet started, in progress, and completed) was indicated by putting each sticky note in a corresponding column for the status and corresponding column for the team member. This allowed everyone on the team to easily share their status and to see the status of everyone else. On the sides were sketches of the architecture as well as some business oriented models. In addition to Scrum the team had adopted several practices from Agile Modeling, in this case they had done some initial requirements envisioning and architecture envisioning, as well as practices from Extreme Programming (XP) for construction. In short, they had followed a fairly common strategy of combining practices from various agile methods.
This would have worked perfectly fine if they had tailored the practices to reflect the situation that they were in, but instead they adopted them "straight out of the book". First, the team was distributed, with most of the team in the location that I was visiting but some people located in two other distant cities. Therein was the source of most of their problems. The people at the other two locations weren’t getting much value out of the daily stand-up meetings, even though they would dial in, because they couldn’t see the project status information. Although people at this location were trying their best to represent these distant people in the daily stand-ups it wasn’t working well – their status information wasn’t being kept up to date and for some people it was a bit of mystery as to what they were actually working on at all.
This team also had 30 people in it, which isn’t a big deal although it can stretch the limits of the simple modeling and planning tools (in this case paper and whiteboards) that they were using. Because the team was larger they were investing a fair bit of time creating burn down charts at both the iteration/sprint and project levels. One of the unfortunate implications of using manual tools for project management is that any associated metric/status reporting in turn becomes manual as well. Considering how the agile community is so concerned with working efficiently, I find it comical that we have a tendency to overlook our own potentially unnecessary bureaucracy such as this.
The problem was that the team was applying strategies, in this case using sticky notes and whiteboards to capture the detailed iteration plan, applying similar strategies to capture key models, and were verbally relaying of status information between sub-teams. There are perfectly fine strategies for smaller co-located teams, but not so good for large or distributed teams. The solution was to recognize that they were in an Agility@Scale situation and needed to tailor their approach to reflect this fact. In this case they needed to forgo some of the manual tools and instead use electronic tooling such as Rational Team Concert (RTC) to share information across disparate locations, in particular the work assignment and corresponding status information. RTC also creates common agile reports such as burn-down charts based on the activities of the developers, providing accurate (nearly) real-time information while removing the burden of status reporting. The RTC project dashboard does more than just this, to see an actual example of one visit www.jazz.net to see the dashboard for the RTC development team itself. You can also see their actual work item list too, a more advanced version of Scrum’s product and sprint backlogs.[Read More]
ScottAmbler 120000HESD Tags:  velocity governance acceleration agileexec agile metrics 2 Comments 7,900 Visits
I've been getting a lot of questions lately about applying the acceleration metric in practice. So, here's some answers to frequently asked questions:
1. How do I monetize acceleration? This is fairly straightforward to do. For example, assume your acceleration is 0.007 (0.7%), there are five people on the team, your annual burdened cost per person is $150,000, and you have two week iterations. All these numbers are made up, but you know how to calculate acceleration now and IT management had darn well better know the average burdened cost (salary plus overhead) of their staff. So, per iteration the average burdened cost per person must be $150,000/26 = $5,770. Productivity improvement per iteration for this team must be $5,770 * 5 * .007 = $202. If the acceleration stayed constant at 0.7% the overall productivity improvement for the year would be (1.007)^26 (assuming the team works all 52 weeks of the year) which would be 1.198 or 19.8%. This would be a savings of $148,500 (pretty much the equivalent of one new person). Of course a 20% productivity increase over an entire year is a really aggressive improvement, regardless of some of the claims made by the agile snake oil salesman out there, although at 10-15% increase is a reasonable expectation. What I'd really want to do is calculate the acceleration for the year by comparing the velocity from the beginning of the year to the end of the year (in Western cultures I'd want to avoid comparing iterations near to the holidays). So, if the team velocity the first week of February 2008 was 20 points, now the same team's velocity the first week of February 2009 was 23 points, that's an acceleration of (23-20)/20 = 15% over a one year period, for a savings of $112,500.
2. Is acceleration really unitless? For the sake of comparison it is. The "units" are % change in points per iteration, or % change in points per time period depending on the way that you want to look at it. Because it's a percentage I can easily monetize it, as you see above, and use it as a basis of comparison.
3. How do I convince teams to share their data? This can be difficult. Because acceleration is easy to calculate for agile teams, and because it's easy to use to compare teams (my team has .7% acceleration whereas other teams down the hall from mine have accelerations of .3% and -.2% of teams), people are concerned that this metric will be used against them. OK, to be fair, my team might be OK with this. ;-) Seriously though, this is a valid fear that will only be addressed by an effective governance program based on enablement, collaboration, and trust instead of the traditional command-and-control approach. Management's track record regarding how they've used measurements in the past, and how they've governed in general, have a great effect on people's willingness to trust them with new metrics such as acceleration. The implication is that you need to build up trust, something that could take years if it's possible at all.
4. Why does this work for agile teams? Agile teams are self organizing, and an implication of that is that they will be held accountable for their estimates. Because of this accountability, and because velocity is a vital input into their planning and estimation efforts, agile teams are motivated to calculate their velocity accurately and to track it over time. Because they're eager to get their velocity right, and because acceleration is based on velocity, there's an exceptionally good chance that it's accurate.
5. What about function points or similar productivity measures? Function points can be calculated for projects being developed via an agile approach, or other approaches for that matter, but it's a very expensive endeavor compared to calculating acceleration (which is essentially free) and likely will be seen as a bureaucratic overhead by the development team. My rule of thumb is that if you're not being explicitly paid to count function points (for example the US DoD will often pay contracting companies to create estimates based on function point counts) then I wouldn't bother with them.
6. What about calculating acceleration for iterative project teams? Iterative project teams, perhaps following Rational Unified Process (RUP), can choose to calculate and track their velocity and thereby their acceleration. The key is to allow the team to be self organizing and accountable for their estimates, which in turn motivates them to get their velocity right just like agile teams (RUP can be as agile as you want to make it, don't let anyone tell you differently).
7. What about calculating acceleration for traditional project teams? In theory this should work, in practice it is incredibly unlikely. Traditional teams don't work in iterations where working software is produced on a regular basis, they're typically not self organizing, and therefore there really isn't any motivate to calculate velocity (even if they do, there is little motivation to get it right). Without knowing the velocity you can't calculate acceleration. If you can't trust the velocity estimate, and I certainly wouldn't trust a traditional team's velocity estimate, then you can't trust your acceleration calculation. So, my fall back position to calculate productivity improvement would be to do something like function point counting (which is expensive and difficult to compare between teams due to different fudge factors used by different FP counters) and then looking at change in FPs delivered over time.
8. How can I apply this across a department? It is fairly straightforward to roll up the acceleration of project teams into an overall acceleration measure for a portfolio of teams simply by taking a weighted average based on team size. However, this is only applicable to teams that are in a position to report an accurate acceleration (the agile and iterative teams) and of course are willing to do so.
9. What does a negative acceleration tell me? If the acceleration is negative then productivity on the team is going down, likely an indicator of quality and/or team work problems. However, you don't want to manage by the numbers so you should talk to the team to see what's actually going on.
10. What does a zero acceleration tell me? This is an indication that the team's productivity is not increasing, and that perhaps they should consider doing retrospectives at the end of each iteration and then acting on the results from those retrospectives. Better yet they can "dial up" their process improvement efforts by adopting something along the lines of IBM Rational Self Check.
ScottAmbler 120000HESD Tags:  agility-at-scale agileadopt architecture reuse disciplined-agile-deliver... soa 2 Comments 7,820 Visits
A common question that I keep running into with customers is whether you can take an agile approach to service oriented architecture (SOA). The quick answer is yes, because Agile is orthogonal to the implementation technologies used. You can take an agile approach developing COBOL applications running on mainframes, fat-client Java applications, multi-tier J2EE applications, and yes, even services. Granted, it's easier to do with some technologies than others, either because of the nature of the technology or because of the supporting tools.
The long answer is "yes, but". You don't adopt an SOA approach for the sheer joy of doing so, instead you very likely want to improve the level of reuse within your organization. To succeed at SOA-driven reuse you need an enterprise focus, something that doesn't appear to be very common on many agile teams. Therein lies the challenge. Several strategies for improving your chances with Agile SOA, and SOA in general, follows:1. Invest in some initial enterprise architecture modeling. You don't need to identify all of the details up front, that would take too long and actually put the effort at risk, but you do need to set a starting point to guide development teams. Identifying the technical architecture is critical, and identifying a few basic services which would provide immediate business value to one or more teams is critical. Involve people from several application project teams to ensure that you get a wide range of input. See http://www.agiledata.org/essays/enterpriseArchitecture.html for a streamlined approach to enterprise architecture modeling. Creating big, detailed models often proves to be a waste of time because development teams are rarely motivated to read mounds of documentation.2. Build out the initial infrastructure on a real application development project. This proves that your SOA strategy actually works and puts the technical foundation in place for future teams. During this period you'll be tempted to try to support several development teams, which is feasible but dramatically increases your risk. It's also tempting to focus simply on getting the infrastructure in place without delivering any business functionality, but this risks producing an ivory-tower architecture that nobody is interested in.3. Spread the service architects out onto application development teams. The people that formulated and then proved your SOA should be actively involved on the development teams that are working with it to ensure that the teams use it appropriately and to ensure that the architects get concrete feedback which they can use to evolve the architecture. When working on agile teams, these people will need to work in a collaborative and evolutionary approach just like other team members.4. Fund reuse separately. I've lost track of the number of organizations that I've run into that fail at reuse because their development teams never have the resources to develop reusable assets. That's simply the nature of the beast -- project teams will always be more interested in addressing their own specific requirements than they are in investing the time and effort to make something reusable. The real problem here is that you expect them to act differently. A better strategy is to have a separate reuse engineering team that has the resources to monitor existing projects to look for potentially reusable assets. When they find said assets this team does the work to harvest the asset, to reengineer it to make it reusable, and then to integrate back into the original source project. The goal is to make it as painless as possible to produce reusable assets such as services. If you expect project teams to do this work out of the goodness of their hearts then you're effectively punishing them when they do the right thing. That's not a very good governance strategy, IMHO.5. The reuse team now owns the asset. Any reusable asset, including services, will need to be maintained, evolved over time, and supported. This isn't free nor is it viable for project teams to do so.
If you're interested, I provide agile strategies for both enterprise architecture and strategic reuse in the book "Enterprise Unified Process". Although written under the assumption that you're taking a RUP-based approach to development, the reality is that the EUP can extend any evolutionary/agile software development process so that it addresses the larger-scale needs of modern IT organizations.
- Scott[Read More]
During 2007 Per Kroll and myself invested a significant amount of time development a framework for lean development governance. This effort resulted in a series of three articles that were published in Rational Edge and a recently published white paper. The articles go into the various practices in detail whereas the paper provides an overview aimed at executives. I also recently did a webcast which is now available online. The URLs are at the bottom of this blog posting.
Development governance isn’t a sexy topic, but it critical to the success of any IT department. I like to compare traditional, command-and-control approaches to governance to herding cats – you do a bunch of busy work which seems like a great idea in theory, but in the end the cats will ignore your efforts and stay in the room. Yet getting cats out of a room is easy to accomplish, as long as you know what motivates cats. Simply wave some fish in front of their noses and you’ll find that you can lead them out of the room with no effort at all. Effective governance for lean development isn’t about command and control. Instead, the focus is on enabling the right behaviors and practices through collaborative and supportive techniques. It is far more effective to motivate people to do the right thing than it is to try to force them to do so.
This framework is based on the philosophical foundation provided by the 7 principles proposed in the book “Lean Software Development” by Mary and Tom Poppendieck. The 7 principles are:1. Eliminate Waste. The three biggest sources of waste in software development are the addition of extra features, churn, and crossing organizational boundaries. Crossing organizational boundaries can increase costs by 25% or more because they create buffers which slow down response time and interfere with communication. It is critical that development teams are allowed to organize themselves, and run themselves, in a manner which reflects the work that they’re trying to accomplish. 2. Build Quality In. If you routinely find problems with your verification process then your process must be defective. When it comes to governance, if you regularly find that developers are doing things that you don’t want them to do or are not doing things that they should be then your approach to governance must be at fault. The strategy is not to make governance yet another set of activities that you layer on top of your software process but instead should embed into your process to make it as easy as possible for developers to do the right thing. 3. Create Knowledge. Planning is useful, but learning is essential. 4. Defer Commitment. You do not need to start software development by defining a complete specification, but instead work iteratively. You can support the business effectively through flexible architectures that are change tolerant and by scheduling irreversible decisions to the last possible moment. This also requires the ability to closely couple end-to-end business scenarios to capabilities developed in potentially several different applications by different projects. 5. Deliver Fast. It is possible to deliver high-quality systems fast and in a timely manner. By limiting the work of a team to their capacity, by not trying to force them to do more than they are capable but instead ask them to self-organize and thereby determine what they can accomplish, you can establish a reliable and repeatable flow of work. 6. Respect People. Sustainable advantage is gained from engaged, thinking people. The implication is that you need a human resources strategy which is specific to IT, that you need to focus on enabling teams not on controlling them. 7. Optimize the Whole. If you want to govern your development efforts effectively you must look at the bigger picture, not just individual project teams. You need to understand the high-level business process which the individual systems support, processes which often cross multiple systems. You need to manage programs of interrelated systems so that you can deliver a complete product to your stakeholders. Measurements should address how well you’re delivering business value, because that is the raison d’etre of your IT department.
Based on our experiences, and guided by the 7 principles, Per Kroll and I identified 18 practices of lean development governance. We've organized these practices into 6 categories:1. The Roles & Responsibilities category: - Promote Self-Organizing Teams. The best people for planning work are the ones who are going to do it. - Align Team Structure With Architecture. The organization of your project team should reflect the desired architectural structure of the system you are building to streamline the activities of the team.
2. The Organization category: - Align HR Policies With IT Values. Hiring, retaining, and promoting technical staff requires different strategies compared to non-technical staff. - Align Stakeholder Policies With IT Values. Your stakeholders may not understand the implications of the decisions that they make, for example that requiring an “accurate” estimate at the beginning of a project can dramatically increase project risk instead of decrease it as intended.
3. The Processes category: - Adapt the Process. Because teams vary in size, distribution, purpose, criticality, need for oversight, and member skillset you must tailor the process to meet a team’s exact needs. - Continuous Improvement. You should strive to identify and act on lessons learned throughout the project, not just at the end. - Embedded Compliance. It is better to build compliance into your day-to-day process, instead of having a separate compliance process that often results in unnecessary overhead. - Iterative Development. An iterative approach to software delivery allows progressive development and disclosure of software components, with a reduction of overall failure risk, and provides an ability to make fine-grained adjustment and correction with minimal lost time for rework. - Risk-Based Milestones. You want to mitigate the risks of your project, in particular business and technical risks, early in the lifecycle. You do this by having throughout your project several milestones that teams work toward.
4. The Measures category: - Simple and Relevant Metrics. You should automate metrics collection as much as possible, minimize the number of metrics collected, and know why you’re collecting them. - Continuous Project Monitoring. Automated metrics gathering enables you to monitor projects and thereby identify potential issues so that you can collaborate closely with the project team to resolve problems early.
5. The Mission & Principles category: - Business-Driven Project Pipeline. You should invest in the projects that are well-aligned to the business direction, return definable value, and match well with the priorities of the enterprise. - Pragmatic Governance Body. Effective governance bodies focus on enabling development teams in a cost-effective and timely manner. They typically have a small core staff with a majority of members being representatives from the governed organizations. - Staged Program Delivery. Programs, which are collections of related projects, should be rolled out in increments over time. Instead of holding back a release to wait for a subproject, each individual subprojects must sign up to predetermined release date. If the subproject misses it skips to the next release, minimizing the impact to the customers of the program. - Scenario-Driven Development. By taking a scenario-driven approach, you can understand how people will actually use your system, thereby enabling you to build something that meets their actual needs. The whole cannot be defined without understanding the parts, and the parts cannot be defined in detail without understanding the whole.
6. The Polices & Standards category: - Valued Corporate Assets. Guidance, such as programming guidelines or database design conventions, and reusable assets such as frameworks and components, will be adopted if they are perceived to add value to developers. You want to make it as easy as possible for developers to comply to, and more importantly take advantage of, your corporate IT infrastructure. - Flexible Architectures. Architectures that are service-oriented, component-based, or object-oriented and implement common architectural and design patterns lend themselves to greater levels of consistency, reuse, enhanceability, and adaptability. - Integrated Lifecycle Environment. Automate as much of the “drudge work”, such as metrics gathering and system build, as possible. Your tools and processes should fit together effectively throughout the lifecycle.
The URLs for the 3 articles:Principles and Organizations: http://www.ibm.com/developerworks/rational/library/jun07/kroll/Processes and Measures: http://www.ibm.com/developerworks/rational/library/jul07/kroll_ambler/Roles and Policies: http://www.ibm.com/developerworks/rational/library/aug07/ambler_kroll/
The URL for the white paper:https://www14.software.ibm.com/webapp/iwm/web/preLogin.do?lang=en_US&source=swg-ldg
The URL for the webcast:https://www14.software.ibm.com/webapp/iwm/web/preLogin.do?lang=en_US&source=dw-c-wcsdpr&S_PKG=112907C[Read More]
There seems to be a lot of misunderstanding around the topic of "Agile RUP", so I thought I'd put in my $0.02 on the subject. I suspect that some of the misunderstanding stems from lack of knowledge about RUP, either because the person either hasn't looked at RUP and is simply parroting the misinformation they heard from other people or because they've seen or been involved with questionable implmentations of RUP in the past. For the first problem I suggest reading "Agility and Discipline Made Easy" by Per Kroll and Bruce MacIsaac because it gives a pretty good overview of applying RUP in an Agile manner.
The second problem is a bit more challenging to overcome because of the very nature of RUP. RUP isn't a software process, it's actually a software process framework from which you instantiate software processes. Big difference. You select, and tailor where appropriate, the process elements appropriate to your situation. Unfortunately many organizations appear to have struggled with this concept. A common anti-pattern which are organizations that look at the RUP and say "there's a lot of really good stuff here" (absolutely true) and then conclude "we need to do it all" (yikes). That's the equivalent of going to a buffet and trying to eat all of the food in it, very clearly a bad dietary strategy. Just like you need to pick and choose only the food that you should eat, hopefully being mature enough to choose food that is good for you, you need to pick the appropriate process elements which are good for you. This requires significant experience and process knowledge to do effectively because software development is a complex endeavor and the best approach for one situation may be completely different for another situation. A second anti-pattern is when organizations assign their existing process engineers, who are often used to document-heavy serial processes, and ask them to tailor the RUP. It isn't surprising that they often produce a document-heavy and serial version of the RUP (at that point I would argue that it's no longer RUP).
The point is that these problems are self-inflicted, that these organizations could just as easily have chosen to instantiate the RUP in a light and effective manner, and better yet in a truly agile manner. In practice the RUP can be as agile as you want it to be, but you need to choose to work this way.
Some important observations:1. RUP socialized many of the concepts that Agile was built on. Although the concept of iterative development was around long before RUP, for many organizations RUP made the concept palatable through its mature approach (particularly when compared to some of the RAD/Spiral strategies at the time). In many organizations RUP also socialized testing throughout the entire lifecycle, delivery of working software each iteration, and collaborating closely with stakeholders throughout the project (to name but a few). These ideas seem straightfoward today, and they've been taken to even greater extremes in some cases , but back in the mid-90s this was pretty heady stuff for the vast majority of practitioners within the IT community. 2. RUP has adopted many of the "new" agile techniques. RUP is a process framework containing a wealth of IT practices, including both agile and traditional practices (and a lot in between). RUP continues to evolve, capturing industry best practices from many sources. So naturally RUP has adopted many agile concepts such as test-driven development (TDD), continuous integration, embracing change, and others. Check it out and see for yourself.3. RUP is as agile, or non-agile, as you want to make it. I've said it before and I'll say it again -- the RUP is a process framework from which you instantiate processes. You've got complete control over how agile the RUP is.4. RUP contains many of critical techniques for scaling agile. In a previous blog posting I overviewed the issues around scaling, not only is team size an issue but so is geographical distribution, regulatory compliance, application complexity, legacy systems/policies, governance, and organizational distribution.
I'm sure some people are reading this and thinking to themselves "of course this is what Scott is going to say, that's his job." Well, think what you want, but I was writing about how to take an agile approach several years before joining IBM. In fact, I believe that I'm the first to do so, writing about it in print in my Software Development column back in 2001 and more importantly in my book Agile Modeling: Effective Practices for XP and UP in 2002. And, if you go poking around the web a bit, you'll see a lot other have written about this too, including Craig Larman, Ivar Jacobson, Bob Martin, Gary Evans, Doug Rosenberg and many more.
I'd like to leave you with a sound bite: "The RUP done right is Agile. The RUP done wrong is just plain wrong."[Read More]
ScottAmbler 120000HESD Tags:  agile offshoring testing distributed geographic-distribution scaling-agile 1 Comment 5,142 Visits
Over the past few months I've had several people ask me whether it makes sense to offshore agile testing, and more importantly when it makes sense to do so. So I thought I would share my thoughts on the subject:
I suspect that you're going to find yourself in serious trouble if:
In short, it can make sense to offshore agile testing in a very narrow range of situations. Be very careful.
ScottAmbler 120000HESD Tags:  enterprise agile disciplined-agile-deliver... 1 Comment 4,636 Visits
On April 25, 2013 I gave a webcast for the Global Rational User Community entitled Disciplined Agile Delivery: Going beyond Scrum . During the webcast a large number of questions were asked but unfortunately I couldn’t get to all of them. So I’ve taken the opportunity to write up the answers in this blog posting.
I have a young daughter and she's at the age where she wants to dress herself. The problem is that if we pick a single outfit and try to get her to wear it she refuses (I've lost count of the times I've heard "I don't want that"). At the other extreme if we let her pick her own outfit from the closet she'll be there for hours trying everything on. As experienced parents advise what we need to do is present her with two or three choices and ask her to pick what she wants.
So how does this relate to software development? Once again, let's look at extremes. First, consider Scrum's approach of prescribing a single way of doing things. For example, Scrum prescribes that you hold a daily meeting, called a Scrum, where everyone stands up and answers the same 3 questions. Scrum also prescribes a single change management strategy where you have a stack of requirements prioritized by business value. Scrum prescribes three roles - ScrumMaster, Product Owner, and Team Member - as well as other things. Don't get me wrong, these strategies are all great in certain circumstances but not for all. Prescribing one way of doing things is an extreme, so perhaps we shouldn't be surprised when people refuse to do it that way or struggle to make it work given the situation that they face.
At the other extreme consider RUP's approach where it presents repository of techniques from which to select the ones appropriate for you. The problem is that now we have an overwhelming way of doing things from which to choose, all of them good options in certain situations. So why are we surprised when teams struggle to identify a coherent tailoring of RUP?
Now let's consider the middle ground. The Disciplined Agile Delivery (DAD) process decision framework takes a goals-driven approach. So, instead of saying "hold a daily stand up meeting and answer these three questions" it says to regularly coordinate within the team and there are several ways of doing so (hold a Scrum meeting, hold a Kanban-style meeting, and so on). Yes, DAD does provide a large number of techniques to choose from (as does the agile community in general) but it also provides a straightforward way to choose between them. DAD does this by describing the advantages and disadvantages of each technique and suggests when, and when not, to use each approach. When people are presented with viable options, and the trade-offs associated with each, it's much more likely that they'll choose an approach that is better suited for their situation.
Scrum's single prescribed strategy works well only when that strategy is appropriate for the situation at hand. Similarly, telling my daughter exactly what to wear works well only when she's in the mood to wear that outfit. RUP's cafeteria approach to software process works well when you have the expertise, and time, to choose what's best for you. Similarly, asking my daughter to pick out her outfit from all the choices in her closet only works well when I've got a lot of time to wait for her. In both situations a better strategy is to present options, describe the trade offs, and then let people pick what's right for them given the context of the situation that they face. This is exactly what the DAD framework promotes.
I believe the goals-based approach of Disciplined Agile Delivery (DAD) represents an important step forward in the software process realm. It's time to recognize the extremes for what they are and move to a more viable middle ground.
ScottAmbler 120000HESD Tags:  agileadopt disciplined-agile-deliver... training 1 Comment 10,422 Visits
I'm happy to announce that IBM Rational's RP252 Advanced Disciplined Agile Delivery (DAD) workshop is now available. This is a 3-day, hands-on workshop which teaches students the fundamentals of Disciplined Agile Delivery (DAD). This workshop is offered both publicly and privately.
Rolf Nelson recently recorded a short (5 min) podcast about IBM Rational Team Concert (RTC). RTC is a complete agile collaborative development environment providing agile planning, source code management, work item management, build management, and project health, along with integrated reporting and process support. I've worked with RTC for a couple of years now and have been truly impressed with it. What should be of interest to many people is the Express-C version which is a free, fully-featured, 10-license version of RTC which can be easily downloaded from www.jazz.net.
ScottAmbler 120000HESD Tags:  estimating agileadopt agile agileexec management 1 Comment 13,502 Visits
For several years now I've written various articles and newsletters on the topics of estimating and funding strategies for software development projects, and in particular for agile software development projects. Whenever I talk to people about agile software development, or coach them in how to succeed at it, some of the very first questions that I'll be asked, particularly by anyone in a management role, is how to fund agile software development projects. Apparently a lot of people think that you can only apply agile strategies on small, straightforward projects where it makes sense to do a time and materials (T&M) approach. In fact you can apply agile strategies in a much greater range of situations, as the various surveys that myself and others are showing time and again. My goal with this blog posting is to summarize the various strategies for, and issues surrounding, the funding of agile software development projects.Modified by ScottAmbler
There are three basic strategies for funding projects, although several variations clearly exist. These strategies are:
The point is that there are several strategies for funding agile software development projects, just like there are several strategies for funding traditional software development projects. My experience is that fixed-price funding strategies are incredibly poor practice which increases the risk of your software development projects dramatically. I recognize how hard it can be to change this desire on the part of our business stakeholders, but have also had success changing their minds. If you choose to perservere, which is a difficult decision to make, you can help your organization's decision makers to adopt more effective strategies. Like you they want to improve the effectiveness of your IT efforts.
Further reading: (In recommended order)
Yesterday I was involved with a workshop around agile development at scale. At one point in the conversation we started talking about the relationship between cost and quality. Some of the people in the workshop were relatively new to agile and still believed the traditional theory that to build in high quality it costs more, sometimes substantially more. This does appear to be true on traditional waterfall projects, but some people were making the mistake that this was an "natural law of IT" which also must apply to agile project teams. I naturally jumped on that idea and described how agile developers have found that writing high quality code leads to lower development costs and shorter time to value, in direct contradiction to traditional theory. A few people struggled with the idea for a bit, and one was pretty adamant that in some cases the need for very high quality does in fact lead to greater cost and time. He talked about his experiences on large-scale Rational Unified Process(RUP) projects and in particular how some URPS (usability, reliability, performance, and supportability) requirements can increase your cost. At this point Per Kroll, co-author of Agility and Discipline Made Easy: Practices from OpenUP and RUP, jumped into the conversation and pointed out although higher quality does lead to lower cost in most cases, using Toyota's lean approach to manufacturing as an example, that the agile community didn't completely have the relationship between quality and cost completely correct. My spidey sense told me that a learning opportunity was coming my way.
Per and I had an offline discussion about this to explore what he'd been observing in practice. In most situation it appears to be the case that higher quality does in fact lead to lower costs and shorter time for delivery, something that Per and I had observed numerous times. This happens because high quality code is much easier to understand and evolve than low quality code -- the agile community has found that it is very inexpensive to write high quality code by following practices such as continuous integration, developer regression testing [or better yet test-driven development(TDD)], static code analysis, following common development conventions, and agile modeling strategies. When you "bake in" quality from the start through applying these techniques, instead of apply traditional techniques such as reviews and end-of-lifecycle testing (which is still valid for agile projects, but should not be your primary approach to testing) which have long feedback cycles and therefore prove costly in practice. But, as we've learned time and again, when you find yourself in more complex situations of Agility@Scale sometimes the mainstream agile strategies fall down. For example, in situations where the regulatory compliance scaling factor is applicable, particularly regulations around protecting human life (i.e. the FDA's CFR 21 Part 11), you find that some of the URPS requirements require a greater investment in quality which can increase overall development cost and time. This is particularly true when you need to start meeting 4-nines requirements (i.e. the system needs to be available 99.99% of the time) let alone 5-nines requirements or more. The cost of thorough testing and inspection can rise substantially in these sorts of situations.
In conclusion, it does seem to be true in the majority of situations, which is what the level 1 rhetoric focuses on, that higher quality leads to lower development costs. But at scale this doesn't always seem to hold true.
PS -- Sorry for the corny title, but a couple of days ago at the Rational Software Conference I had the pleasure of interviewing Jamie Hyneman and Adam Savage from the Discovery Channel's Mythbuster's show as part of the conference keynote. They're great guys, BTW, who have had a really positive impact on motivating children to be interested in science (apparently kids like to see stuff get blown up, go figure).[Read More]
I just wanted to share with you the Manifesto for Software Craftsmanship which extends the Agile Manifesto. The Manifesto for Software Craftsmanship states:As aspiring Software Craftsmen we are raising the bar of professional software development by practicing it and helping others learn the craft. Through this work we have come to value:
That is, in pursuit of the items on the left we have found the items on the right to be indispensable.
I view this manifesto as an important step in the maturation of software development. More on this in a future blog posting.[Read More]