The SEMAT vision
has recently been posted online. SEMAT is short for Software Engineering and Method and Theory (SEMAT) and I am one of several signatories and a provider of input into the effort.
There are several reasons why I'm involved with SEMAT:
- The industry clearly needs something like this. Ivar Jacobson has been writing and speaking for awhile now about how our industry behaves in a similar manner to the fashion industry
-- we lurch from one cool idea (the fashion) to another and few of us
observe that the current fashion is often just a rehash of fashion(s) from the
- The right people are involved. The SEMAT initiative has
attracted a wide range of industry experts, including both the old
guard, the new "agile guard", and people in between. Arguably we may
be short a few people, the data community isn't well represented and
I'm not sure we have the systems community covered well, but we can
address those challenges in time.
We'll achieve something. We may not pull of the entire vision, but we
are going to produce something of value and I hope that it has a
positive impact on the industry. Time will tell.
I have several thoughts about the SEMAT vision
which I'd like to share with you:
- The vision is coherent. A lot of reasoned effort went into it's development as you can see when you read it.
- It will be a challenge to identify a non-trivial kernel. Even developing a kernel language will be hard as people will often stick to their preferred terms. For example, is it an iteration, a sprint, or a time box?
- Practitioners may not notice. It will also be a significant challenge to get practitioners interested in the SEMAT effort and more importantly to leverage the material. For example, the patterns community has a long history of producing great work which for the most part is ignored by the vast majority of practitioners, with the exception of a handful of the hundreds of patterns out there. So, although SEMAT is likely to produce some great ideas, will anyone care?
- Academics may not notice. I suspect that this will be less of a problem than practitioners not noticing, but it's still a possibility.
- Position papers from many of the signatories, including myself, will soon be published at the SEMAT site. Several non-signatories have been invited to submit papers as well.
- A workshop is being held in the third week of March in Zurich. At this workshop the people who published position papers will flesh out our ideas.
- I'll blog about these things as they occur.
When you’re inside, safe in the warmth of your home watching snow fall on your driveway outside, all snowflakes look the same. But, when you look at a snowflake up close, particularly when you do so under a microscope, you quickly discover that all snowflakes are in fact unique.
It’s the same with IT projects.
When you look at them from afar, particularly
from a very high level, they all look the same.
However, when you look at them up close, you quickly discover that they too
The agile scaling factors
, which are really just general scaling factors applicable to all types of IT
project regardless of paradigm, help to make this very clear.
For example, when it comes to team size some teams
are small, less than ten people perhaps, some are medium sized, and some are very
large (with hundreds of people).
comes to distribution some teams are co-located in the same room, some teams
have team members in different cubicles in the same building, some have people
working in different buildings, and some even have people working in different
Many agile teams work in
regulatory environments, in fact the July 2009 DDJ State of the IT Union survey reports that one third of agile teams must
comply to industry regulations, although clearly many agile teams do not have
this as a concern. That’s only three
scaling factors. The point is that a
small, co-located team working in a non-regulatory environment will work much
differently than a fifty-person team working in three different locations,
which in turn works differently than a two hundred person team in the same
building working in a regulatory situation.
Different teams, facing different scaling issues will work in different
ways – unique snowflakes from a process point of view.
During the second week of August the Agile 2011 conference was held in Salt Lake City (SLC). As you likely know the Agile Manifesto was formulated 10 years ago in SLC so it was apropos to hold it there. There was some excitement around the 10 year anniversary of the manifesto, with a panel session with the 17 authors of it. Sadly there seemed to be little excitement around the efforts of the 10th anniversay agile workshop
in February which proposed a potential path forward for the agile community. I found the conference to be an evolutionary improvement over the conferences of the past few years, which is a very good thing because the focus since 2008 has moved beyond the "cool" new programming techniques to include the issues that enterprises face.
Starting at the Agile 2008 conference I've seen an uptick in interest in what I would consider some of the more mature topics in agile development, although I'm unfortunately still seeing significant confusion out there too, in part due to over-exuberence of people new to agile. For example, there's people still asking about basic issues about agile architecture
and agile database
techniques, although I was really happy to see more coherent discussions around scaling agile
. My own presentation about the Agile Scaling Model
was well attended and I suspect I opened a few people's eyes regarding the realities that we face (yes, there's a lot more to it than holding a "scrum of scrums", yeesh). We have a long way to go until people really start to understand scaling issues, but we're clearly on the path to getting there.
The conference show floor was interesting, with a wide range of vendors offering services and products focused on agile and lean. One thing that I noticed was many vendors had large monitors showing off their ability to support lean task boards, which for the most part they all looked the same. At the IBM booth we were showing off some of the Jazz tools
, in particular Rational Team Concert (RTC)
. For a long time now we've been giving away fully functional, with no time limit, licenses of RTC for teams of up to 10 people. Something worth checking out.
The Agile 201x conferences hosted by the Agile Alliance are always a good investment of your time and money, and Agile 2011 was no exception. See you at Agile 2012 in the great state of Texas!
My January 2010 DDJ Agile Update, Tragic Mistakes When Adopting Test Driven Development (TDD)
, is now online. In the article I summarize what I consider to be common, and tragic, mistakes that I'm seeing organizations make when they attempt to adopt TDD.
These mistakes include:
The article also goes into potential benefits of TDD as well as potential challenges that you're face when adopting it.
On Nov 16 2011, Kim Werner, Agile Coach from ATSC and Liz Parnell, Solution Design Manager from Blue Cross Blue Shield of North Carolina (BCBSNC), gave a webcast
sharing their experiences regarding how BCBSNC adopted a few Agile techniques, with the help of some good coaching, and adopted some IBM Rational Jazz tools
(Rational Team Concert and Rational Requirements Composer) to reduce time to market and lower development costs. BCBCNC works in the highly-regulated healthcare industry, so faced a few interesting constraints when adopting agile.
A recurring discussion that I have with experienced agile developers is what it means to take a disciplined agile approach. The conversation usually starts off by some saying "but it already requires discipline to do agile", something that I fully agree with, followed by "therefore 'disciplined agile' is merely a marketing term", something which I don't agree with. The challenge with the "standard" agile discipline is that it is often focused on construction activities within a single project team, clearly important but also clearly not the full picture. There's more to an agile project than construction, and there's more to most IT departments than a single development project. In short, there are many opportunities for IT professionals to up their discipline, and thereby up their effectiveness, opportunities which we make explicit in the Disciplined Agile Delivery (DAD) framework.
Let's explore the many aspects to taking a disciplined agile approach:
You adopt "standard" agile discipline
. Aspects of agile which require discipline
include adopting practices such as test-driven development (TDD), active stakeholder participation, working collaboratively, shortening the feedback cycle
, and many more. These strategies are a great start to becoming disciplined IT professionals.
You take a goal-driven approach
. When we first started working on the DAD framework I didn't want to create yet another prescriptive framework, particularly given Rational's track record with the Rational Unified Process (RUP) framework. Rational has been pilloried for years for the prescriptive nature of RUP, which is unfortunate because there are a lot of great ideas in RUP that agile teams can benefit from, some of which we adopted in DAD and many of which are being actively reinvented with the agile community even as you read this. Furthermore, there are many prescriptive elements of the Scrum method that can get teams in trouble. For example, Scrum prescribes that you hold a daily stand up meeting, often called a Scrum meeting, where everyone should answer three questions. That's a great approach for teams new to agile, but it proves problematic in many situations due to it's prescriptive nature. Do you really need to do this once a day? I've been on teams where we held coordination meetings twice a day and others only once a week. Do you really need to stand up? I've been on geographically distrubited agile teams where many of us were sitting down during coordination calls. Do you really need to answer three questions, two of which are clearly focused on status regardless of claims otherwise? I've been on lean teams where we met around our Kanban board and focused on potential blockers. The answers to these questions depends on the context of the situation you find yourself in. The challenge, at least from the point of view of a process framework, is how do you avoid falling into the trap of being overly prescriptive. The strategy we adopted in DAD is to take a goal-driven approach. The observation is that regardless of the situation you find yourself in there are common goals your team will need to fulfill. For example, at the beginning of a project common goals include developing an initial plan, initially exploring the scope, initially identifying a technical strategy, and securing initial funding (amongst others). Throughout construction you should coordinate your activities, improve the quality of your ecosystem, and produce a potentially consumable solution on a regular basis (more on this below). So, instead of prescribing a daily stand up meeting the DAD framework instead indicates you should coordinate your activities, and gives several options for doing so (one of which is a Scrum meeting). More importantly DAD describes the advantages and disadvantages of your options so that you can make the choice that's best suited for the situation your team finds itself in (see this blog posting
for a detailed example of the types of tables included in the DAD book to help you through such process tailoring decisions). In short, our experience is that it requires discipline to take a goal driven approach
to agile delivery over the prescriptive strategies in other agile processes.
You take a context-driven approach
. There are many tailoring factors, which I describe in the Software Development Context Framework (SDCF)
, which you need to consider when making process, tooling, and team structure decisions. For example, a large team will adopt a different collection of practices and tools than a small team. A geographically distributed team will adopt a different strategy than a team that is co-located. You get the idea. Other tailoring factors include compliance, team culture, organization culture, technical complexity, domain complexity, and project type. It requires discipline to recognize the context of the situation you find yourself in and then act accordingly.
You deliver potentially consumable solutions
. One of the observations that we made early in the development of the DAD framework was that disciplined agile teams produce potentially consumable solutions, not just potentially shippable software. Although delivery of high-quality, working software is important it is even more important that we deliver high-quality working solutions to our stakeholders. For example, not only are we writing software but we may also be updating the hardware on which it runs, writing supporting documentation, evolving the business processes around the usage of the system, and even evolving the organizational structure of the people working with the system. In other words, disciplined agilists focus on solutions over software
. Furthermore, "potentially shippable" isn't sufficient: not only should it be shippable but it should also be usable and should be something people want to use. In other words it should be consumable (a concept DAD adopted from IBM's Outside In Development
). Minimally IT professionals should have the skills and desire to produce good software, but what they really need are the skills and desire to provide good solutions. We need strong technical skills, but we also need strong "
such as user interface design and process design to name just two.
The incremental delivery of potentially consumable solutions on an incremental basis requires discipline
to do successfully. DAD teams focus on repeatable results not repeatable processes
You are enterprise aware
. Whether you like it or not, as you adopt agile you will constrained by the organizational ecosystem, and you will need to act accordingly. It takes discipline to work with enterprise professionals such as enterprise architects, data admistrators, portfolio managers, or IT governance people who may not be completely agile yet, and have the patience to help them. It takes discipline to work with your operations and support staff in a DevOps
manner throughout the lifecycle, particularly when they may not be motivated to do so. It requires discipline to accept and potentially enhance existing corporate development conventions (programming guidelines, data guidelines, UI guidelines, ...). It requires discipline to accept that your organization has an existing technology roadmap that you should be leveraging, building out, and in some cases improving as you go. In short, enterprise awareness requires a level of discipline
not typically seen on many agile teams.
You adopt a full delivery lifecycle
. Empirically it is very easy to observe that at the beginning of an agile project there are some activities that you need to perform to initiate the project. Similarly at the end of the project there are activities that you need to perform to release the solution into production or the marketplace. The DAD process framework addresses the effort required for the full delivery effort, including project initiation, construction, and deployment. Our experience is that it requires discipline on the part of IT professionals to include explicit phases
for Inception/Initation, Construction, and Transition/Deployment and more importantly to focus the appropriate amount of effort on each. One danger of explicit phases is that you run the risk of taking what's known as a Water-Scrum-Fall
approach, a term coined by Dave West the person who wrote the forward for the DAD book, where you take an overly heavy/traditional approach to inception and transition in combination with a lighter agile approach to construction. Water-Scrum-Fall occurs because many organizations haven't made a full transition to agile, often because they think it's only applicable to construction. Our experience is that you can be very agile in your approach to inception and transition, experience we've built into the DAD framework. Having said that it clearly requires discipline to keep inception activities short
and similarly it requires discipline to reduce the "transition phase" to an activity
You adopt a wider range of roles
. An interesting side effect of adopting a full delivery lifecycle is that you also need to adopt a more robust set of roles. For example, the Scrum method suggests three roles - Scrum Master, Product Owner, and Team Member - a reflection of the Scrum lifecycle's construction focus. DAD suggests three primary roles - Team Lead, Product Owner, Team Member, Architecture Owner
, and Stakeholder - as well as five secondary roles which may appear at scale.
You embrace agile governance
. Governance establishes chains of responsibility, authority and communication in support of the overall enterprise’s goals and strategy. It also establishes measurements, policies, standards and control mechanisms to enable people to carry out their roles and responsibilities effectively. You do this by balancing risk versus return on investment (ROI), setting in place effective processes and practices, defining the direction and goals for the department, and defining the roles that people play with and within the department. It requires discipline to adopt an agile approach to governance
, and that's something built right into the DAD framework.
I have a young daughter and she's at the age where she wants to dress herself. The problem is that if we pick a single outfit and try to get her to wear it she refuses (I've lost count of the times I've heard "I don't want that"). At the other extreme if we let her pick her own outfit from the closet she'll be there for hours trying everything on. As experienced parents advise what we need to do is present her with two or three choices and ask her to pick what she wants.
So how does this relate to software development? Once again, let's look at extremes. First, consider Scrum's approach of prescribing a single way of doing things. For example, Scrum prescribes that you hold a daily meeting, called a Scrum, where everyone stands up and answers the same 3 questions. Scrum also prescribes a single change management strategy where you have a stack of requirements prioritized by business value. Scrum prescribes three roles - ScrumMaster, Product Owner, and Team Member - as well as other things. Don't get me wrong, these strategies are all great in certain circumstances but not for all. Prescribing one way of doing things is an extreme, so perhaps we shouldn't be surprised when people refuse to do it that way or struggle to make it work given the situation that they face.
At the other extreme consider RUP's approach where it presents repository of techniques from which to select the ones appropriate for you. The problem is that now we have an overwhelming way of doing things from which to choose, all of them good options in certain situations. So why are we surprised when teams struggle to identify a coherent tailoring of RUP?
Now let's consider the middle ground. The Disciplined Agile Delivery (DAD) process decision framework takes a goals-driven approach. So, instead of saying "hold a daily stand up meeting and answer these three questions" it says to regularly coordinate within the team and there are several ways of doing so (hold a Scrum meeting, hold a Kanban-style meeting, and so on). Yes, DAD does provide a large number of techniques to choose from (as does the agile community in general) but it also provides a straightforward way to choose between them. DAD does this by describing the advantages and disadvantages of each technique and suggests when, and when not, to use each approach. When people are presented with viable options, and the trade-offs associated with each, it's much more likely that they'll choose an approach that is better suited for their situation.
Scrum's single prescribed strategy works well only when that strategy is appropriate for the situation at hand. Similarly, telling my daughter exactly what to wear works well only when she's in the mood to wear that outfit. RUP's cafeteria approach to software process works well when you have the expertise, and time, to choose what's best for you. Similarly, asking my daughter to pick out her outfit from all the choices in her closet only works well when I've got a lot of time to wait for her. In both situations a better strategy is to present options, describe the trade offs, and then let people pick what's right for them given the context of the situation that they face. This is exactly what the DAD framework promotes.
I believe the goals-based approach of Disciplined Agile Delivery (DAD) represents an important step forward in the software process realm. It's time to recognize the extremes for what they are and move to a more viable middle ground.
Modified by ScottAmbler
Recently I have been asked by several customer organizations to help them to understand how to account for the expense of agile software development. In particular, incremental delivery of solutions into production or the marketplace seem to be causing confusion with the financial people within these organizations. The details of accounting rules vary between countries, but the fundamentals are common. In order to get properly account for the costs incurred by software development teams you need to keep track of the amount of work performed and the type of work performed to develop a given solution. Time tracking doesn't have to be complex: at one customer developers spend less than five minutes a week capturing such information.
Why is Time Tracking Potentially Valuable?
There are several financial issues to be aware of:
- Capitalization. For public companies capital expenses (CapEx) can potentially boost book value through the increase in assets (in this case a software-based solution) and increase in net income (due to lower operating expenses that year). On the other hand, operational expenses (OpEx) are accounted for in the year that they occur and thereby reduce net income which in turn reduces your organization's taxes for that year.
- Matching. One of the goals of good accounting is to accurately reflect the net income of the enterprise and to prevent income manipulation or "smoothing". As such a key tenet is the principle of matching revenues with the appropriate expenses. For software this means that we expense the cost of the software over the lifetime of the asset against the income at that time. An implication of this is that capitalizing software development, when appropriate, before the software goes into production clearly violates the matching principle since there is no benefit of the asset until such time.
- Tax Credits. In some countries you can even get tax credits for forms of software development that are research and development (R&D) in nature.
The point is that the way that a software developer's work is accounted for can have a non-trivial impact upon your organization's financial position.
What Do Agilists Think of Time Tracking?
So, I thought I'd run a simple test. Last week on LinkedIn's Agile and Lean Software Development group I ran a poll to see what people thought about time tracking. The poll provided five options (a limitation of LinkedIn Polls) to choose from:
- Yes, this is a valuable activity (33% of responses)
- Yes, this is a waste of time (39% of responses)
- No, but we're thinking about doing so (2% of responses)
- No, we've never considered this (18% of responses)
- I don't understand what you're asking (5% of responses, one of which was mine so that I could test the poll)
The poll results reveal that we have a long way to go. Of the people inputting their time more of them believed it was a waste of time than understood it to be a valuable activity. When you stop and think about it, the investment of five minutes a week to track your time could potentially save or even earn your organization many hundreds of dollars. Looking at it from a dollar per minute point of view, it could be the highest value activity that a developer performs in a given week.
The discussion that ensued regarding the poll was truly interesting. Although there were several positive postings, and several neutral ones, many more were negative when it came to time tracking. Some comments that stood out for me included:
- It's a colossal waste of time unless you're billing a customer by the hour.
- We record time spent on new development work (as distinct from other tasks such as bug fixing in legacy code and so on) as this is capitalised as an asset and depreciated.
- I think the *most* pointless example was where the managers told us what we should be putting in.
- One day we will move past the "just do it" mentality and have some meaningful conversations and the reasons for what we do.
- In my experience time tracking is a massive waste... of time. It's a poor substitute for management.
- Why do you need to know more than the info available through Sprint Backlog, Sprint burndown and the daily standup?
- Some of my teams (I am SM for three teams) are skeptical about this. They do not think that keeping track of task hours this way will be any more useful than the daily standup reports. And they do not believe that Management can resist the temptation to use task hours as a measure.
I think that there are several interesting implications from this discussion:
- Agilists need to become more enterprise aware. It's clear to be really effective that agile delivery teams need a better understanding of the bigger picture, including mundane things such as tax implications of what they're doing. In Disciplined Agile Delivery (DAD) this is something that we refer to as being enterprise aware. There's far more to enterprise awareness than understanding pertinent accounting principles, for interest disciplined agile teams work towards a common technology roadmap and common business roadmap, but appreciating why time tracking is a potentially valuable activity would be a good start.
- Management needs to communicate better. It's also clear that management needs to communicate more effectively regarding why they're asking people to track their time. To be fair, management themselves might not be aware of the tax implications themselves so may not be making effective use of the time data they're asking for.
- Management needs to govern more effectively. Several people were clearly concerned about how management was going to use the time data (by definition they are measures) which could be a symptom of both poor communication as well as poor governance (unfortunately many developers have experiences where measures have been used against them, a failure of governance, and no longer trust their management teams to do the right thing as a result).
- Time tracking should be streamlined. It was obvious from the conversation that several people worked in organizations where the time tracking effort had gotten completely out of hand. Spending 5 minutes a week is ok, and to be quite blunt should be more than sufficient, but spending fifteen minutes or more a day doing so is far too much. Over the years I've helped organizations design measurement programs and I've seen a lot of well-intention efforts become incredibly onerous and expensive for the people they were inflicted upon. I suspect it's time for a reality check in some of these organizations people were alluding to. A good heuristic is that for any measurement you should be able to indicate the real cost of collecting it, the use(s) that the information is being put to, and the value resulting from those uses. If you can't quickly and coherently do that then you need to take a hard look at why you continue to collect that metric. The lament "we might need it one day" is a symptom that you're wasting time and money.
- Agile rhetoric is getting in the way. Some of the team-focused agile practices, such as burndown charts (or better yet ranged burndown charts) and stand up meetings may be preventing people from becoming enterprise aware because they believe that all of their management needs are being met by them.
- You may be missing out on the benefits of time tracking. Many organizations are potentially leaving money on the table by not being aware of the implications of how to expense software development.
Disciplined agilists are enterprise aware. This is important for two reasons: First, you want to optimize your organizational whole instead of sub-optimize on project-related efforts; second, you can completely miss opportunities to add real value for your organization. In the anecdote I provided it was clear that many agile developers believe that an activity such as time tracking is a waste when that clearly doesn't have to be the case. Worse yet, although someone brought up the issues around capitalizing software development expenses early in the conversation a group of very smart and very experienced people still missed this easy opportunity to see how they could add value to their organization.
Granted, time tracking on an agile project team is nowhere near as sexy as topics such as continuous integration (CI), TDD, the definition of done, continous architecture, or many more. But you know what? Although it's a mind-numbingly mundane issue it is still an important one. 'Nuff said (I hope).
When you are first adopting agile techniques in your organization a common strategy is to run one or more pilot projects. When organizing these projects you typically do as much as you can to make them successful, such as finding:
- Projects where the stakeholders are willing to actively work with you.
- IT people who are flexible, willing to try new things, and willing to collaborate with one another.
- IT people who are generalizing specialists, or at least willing to become so.
- Finding a project which is of medium complexity (therefore it's "real" in the sense that it's significant to your organization) but not one where it can make or break your organization (therefore it's safe to experiment with).
In North America we refer to this as "cherry picking" because you're picking the cherry/best situation that you can find.
- Being agile may not have been the primary determinant of success. You set up an environment where you have a good relationship with your stakeholders, where you have good people who want to work together, and the project is challenging but not impossible. Oh, and by the way you adopted a few agile techniques as well. Sounds to me that situation you could have adopted a few not-so-agile techniques instead and still succeed. Although my various project success surveys, see my IT surveys page for details, have shown time and again that agile project teams are more successful than traditional project teams I haven't been able to tease out (yet) whether this success is attributable to agile or just attributable to improved project initiation efforts.
- When adopting agile/lean widely across your organization, you can't cherry pick any more. For the past few years I've been working with IT organizations that are in the process of adopting agile/lean strategies across their entire organization, not just across a few pilot projects. What these organizations are finding is that they need to find ways to adopt agile where the business isn't as willing to work with IT, where some of the people aren't so flexible or collaborative, where some of the people are narrowly specialized and not as willing to expand their skills, or where the project exhibits scaling factors which motivates you to tailor your agile approach. It's harder to succeed with agile in these situations because they're not as "cherry" as what you've experienced previously. Luckily, if you've been successful previously then you now have some agile experienced people, you have successes to reference, and you've likely overcome some problems even in the cherry situations which you have learned from. So, your cherry successes will hopefully improve your ability to succeed even in "non cherry" situations.
- You need to work smarter, not harder. If the source of your success was actually from improved project initiation practices and not from agile, then recognize that and act accordingly. Realistically part of your success was from that and part was from agile, and the organizations that adopt a measured improvement approach potentially have the data to determine which practices lead to success and which didn't. Without the metrics you're effectively flying blind when it comes to deciding how to improve. There is clearly a mandate for smarter work practices within IT, within your organization as a whole for that matter.
If you want to gain more insight into some of the issues that you'll face when adopting agile across your organization, I suspect that you'll find my recent paper Scaling Agile: An Executive Guide
to be interesting. I've got a more detailed paper in the works, so stay tuned to this blog.