Best practices for lean development governance, Part III: Roles and policies

from The Rational Edge: The third in a series of articles covering IBM Rational's recommended approach to governing modern software development efforts, this article presents the roles and responsibilities, as well as the policies and standards to be adopted in lean software development governance. This content is part of the The Rational Edge.

Share:

Scott W. Ambler, Practice Leader, Agile Development, Rational Methods Group, IBM

author photoScott W. Ambler is the Practice Leader Agile Development with IBM Rational and works with IBM customers around the world to improve their software processes. He is the founder of the Agile Modeling (AM), Agile Data (AD), Agile Unified Process (AUP), and Enterprise Unified Process (EUP) methodologies. He is the (co-)author of nineteen books, including Refactoring Databases, Agile Modeling, Agile Database Techniques, The Object Primer 3rd Edition, and The Enterprise Unified Process. He is a senior contributing editor at Dr. Dobb’s Journal. His personal home page is www.ibm.com/rational/bios/ambler.html and his Agile at Scale blog is www.ibm.com/developerworks/blogs/page/ambler.



Per Kroll, Manager of Methods, IBM Rational, IBM

author photoPer Kroll is Chief Architect for IBM Rational Expertise Development and Innovation, an organization leveraging communities, portals, methods, training and services assets to enable customers and partners in software and systems development. Per is also the project leader on the Eclipse Process Framework Project, an open source project centered on software practices, and he has been one of the driving forces behind RUP over the last 10 years. Per has twenty years of software development experience in supply chain management, telecom, communications, and software product development. He co-authored The Rational Unified Process Made Easy - A Practitioner's Guide, with Philippe Kruchten, and Agility and Discipline Made Easy - Practices from OpenUP and RUP, with Bruce MacIsaac. A frequent speaker at conferences, Per has authored numerous articles on software engineering.



15 August 2007

Also available in Chinese

illustrationIn Part I of this three-part article series, we introduced lean software development governance, and described the mission and principles of lean governance, along with the organization and stakeholder collaboration required for project-by-project success. In Part II, we focused on the practices surrounding the essential processes and measures to be used in lean software development governance. Here, in Part III, we will conclude by examining the roles and responsibilities, as well as the policies and standards, that need to be adopted for lean software development governance.

By "roles and responsibilities," we mean defining who has the authority to perform certain tasks, how these responsibilities are supported by human resource departments, and how team organization must support and add value to IT architecture. By "policies and standards," we mean defining guidelines to support consistent operations across the software development effort, which specifically address an Integrated Lifecycle Environment, the management of IT assets, and the use of loosely coupled architectures to allow rapid changes in support of evolving business requirements.

Roles and responsibilities

The first value of Agile software development is "individuals and interactions over processes and tools"1, a value that we've adopted in this framework. When it comes to development governance our experiences are similar -- the focus should be on enabling IT professionals, to promote effective collaboration between them instead of trying to control or directly manage them. The practices in this category focus on enabling development while still making it clear who is responsible, who has authority, and to whom they are accountable. These practices are:

  • Promote self-organizing teams
  • Align human resource policies with IT values
  • Align team structure with architecture

Lean Software Development

In Implementing Lean Software Development (Addison Wesley 2006) Mary and Tom Poppendieck show how the principles of lean manufacturing can be applied to software development to optimize the whole IT value stream. The seven principles of lean software development are:

  1. Eliminate Waste. The three biggest sources of waste in software development are the addition of extra features, churn, and crossing organizational boundaries. Crossing organizational boundaries can increase costs by 25% or more because they create buffers which slow down response time and interfere with communication. It is critical that development teams are allowed to organize themselves, and run themselves, in a manner which reflects the work that they're trying to accomplish. This principle maps to the practices Promote Self Organizing Teams, Pragmatic Governance Body, Business-Driven Project Pipeline, Risk-Based Milestones, Continuous Improvement, Integrated Lifecycle Environment, and Valued Corporate Assets.
  2. Build Quality In. The Poppendiecks make a simple observation: if you routinely find problems with your verification process then your process must be defective. When it comes to governance, if you regularly find that developers are doing things that you don't want them to do or are not doing things that they should be then your approach to governance must be at fault. The strategy is not to make governance yet another set of activities that you layer on top of your software process but instead should embed into your process to make it as easy as possible for developers to do the right thing. You build quality in via the practices Iterative Development, Risk-Based Milestones, Embedded Compliance, and Valued Corporate Assets
  3. Create Knowledge. Planning is useful, but learning is essential. You want to promote practices such as Iterative Development which enables teams to discover what stakeholders really want and then act on that knowledge. Having standards, guidelines, and reusable assets which people want to follow, and can modify to meet their specific needs, is critical, hence the importance of the Valued Corporate Assets practice. Consistent and timely feedback is also important, both within the team by developing working software on a regular basis but also at the program level through the practices Continuous Project Monitoring and Simple and Relevant Metrics. Knowledge creation is supported by the practices Continuous Improvement because you actively strive to learn and improve and by Self Organizing Teams because individuals are given more opportunities to think through what they are doing.
  4. Defer Commitment. You do not need to start software development by defining a complete specification, but instead work iteratively. You can support the business effectively through flexible architectures that are change tolerant and by scheduling irreversible decisions to the last possible moment. This also requires the ability to closely couple end-to-end business scenarios to capabilities developed in potentially several different applications by different projects. As you can see the practices Iterative Development, Flexible Architectures, and Scenario-Driven Development enable you to defer commitment to the best possible time.
  5. Deliver Fast. It is possible to deliver high-quality systems fast and in a timely manner. By limiting the work of a team to their capacity, by not trying to force them to do more than they are capable but instead ask them to self-organize and thereby determine what they can accomplish, you can establish a reliable and repeatable flow of work. At an organizational level, one needs to enable programs to deliver business value at a pace defined by the fastest moving projects, rather than at the speed of the slowest project, the latter of which is all too common. The practices Self-Organizing Teams, Staged Program Delivery, Iterative Development, Valued Corporate Assets, and Flexible Architectures enable you to deliver fast.
  6. Respect People. The Poppendiecks observe that the sustainable advantage is gained from engaged, thinking people. The implication is that you need a human resources strategy which is specific to IT, that you need to focus on enabling teams not on controlling them. This principle motivates the practices Adapt The Process, Promote Self-Organizing Teams, and Align HR Policies with IT Values.
  7. Optimize the Whole. If you want to govern your development efforts effectively you must look at the bigger picture, not just individual project teams. You need to understand the high-level business process which the individual systems support, processes which often cross multiple systems. You need to manage programs of interrelated systems so that you can deliver a complete product to your stakeholders. Measurements should address how well you're delivering business value, because that is the raison d'etre of your IT department. To optimize the whole you need to follow various lean development practices, including Staged Program Delivery, Scenario-Driven Development, Continuous Improvement, Simple and Relevant Metrics, Continuous Project Monitoring, Integrated Lifecycle Environment, Valued Corporate Assets, Flexible Architectures, and Align Organization Structure with Architecture.

Promote self-organizing teams

A "self-organizing team" has the authority to choose the work that it will perform within the scope of the governance structure in which it operates, and it assumes the responsibility to do that work in the way it chooses. This is a participatory approach to decision-making in which everyone has the opportunity to provide input and to listen to the decision-making process. Important aspects of a self-organizing team are:

  1. The team selects its own work. At the beginning of an iteration the team collectively selects the work items from its prioritized work items list. The work items list includes requirements defined and prioritized by the project stakeholders, coupling the actual business needs to what the team executes on.
  2. Individuals select their own work. Individuals are empowered to select their own work from the iteration's work items. Workers choose tasks for a variety of reasons. For example: a) because they are good at it and know that they can do the work effectively; b) because they want to gain more experience at something and hope to improve their skill set by working with someone with such experience; or c) simply because they know that the work needs to be done and understand that it's their turn to do it. Note that the project manager may always make suggestions as to what work someone should consider choosing, particularly if they wish the individual to round out their skill set.
  3. The team determines how to perform the work. The team determines the general strategy for doing the work at hand and the related tasks required. More detailed planning, if required, will be done on a just-in-time (JIT) basis by the individual(s) doing the work. High-level, long-term planning is initially done by the team at the beginning of the project and then evolved as appropriate by the project manager.
  4. Everyone commits to the work. The team commits to accomplishing the work that it has agreed to do by the end of the iteration, and individuals commit to doing the work that they say they will do.
  5. The team coordinates regularly. To ensure that the work is accomplished and that all appropriate team skills are fully leveraged, the team must coordinate its efforts effectively through regular meetings.
  6. The manager is still accountable. Although the team takes collective responsibility, the ultimate responsibility and associated decision rights lie with the manager. At the same time, the manager should step in and assert authority only when required, which should be rarely. The team is responsible for planning and performing the work; the manager is responsible for communicating the plan and coordinating with groups external to the team.

Benefits

There are several benefits to this practice:

  1. Improved software economics. Because team members have control over their own work, motivation is higher, resulting in increased productivity.
  2. Improved decision making. Decisions are made at the right place within the organizational structure because teams are empowered with both the responsibility and the authority to get the job done.
  3. Improved quality. Contributions from a broader audience results in better quality plans, architectures, and end products.
  4. Fewer opportunities for errors. Self-organizing teams typically require fewer transitions of work products between individuals and better communication, which decreases the chance of defects being introduced into the product.
  5. Greater opportunities to grow individual skills. Individuals have greater opportunities to work on activities which are new to them, thereby improving their skill set and overall productivity.

Trade-offs

This practice requires organizational maturity and a culture of cooperation between people. As a result, there are several trade-offs associated with this practice:

  • You need responsible and mature team members. Team members must have the ability to self-organize in the first place, implying that they must be willing to take responsibility for their decisions and the maturity to accept that they still need to do the "not so exciting" tasks that keep the project on track.
  • You must trust the team. The project manager will provide less day-to-day direction and more high-level guidance to the team.
  • The project manager becomes more of a coach or leader. This may prove difficult for some traditional project managers who feel that part of a manager's job is to push teams toward the highest productivity levels possible, or to incent a few individuals toward heroic achievements. Over the long run, these efforts usually prove counter-productive for the team as a whole.
  • The manager is still responsible for the overall effort. Someone still needs to ensure that everything that needs to be done is accomplished, such as any required documentation to meet regulatory guidelines and conformance with any previously agreed schedule or cost estimate.

Anti-patterns

The following anti-patterns are related to team organization:

  • Responsibility without authority. Teams are told that they are responsible for delivering a system, but are not given the authority to perform critical tasks or make critical decisions. For example, a team may be told to work in an agile manner to develop a mission-critical application but is forced to work with the very non-agile data administration group that insists that only it can develop the database and a testing group which insists on comprehensive requirements documentation before any testing may commence. The "agile" team is effectively hamstrung, because team members do not have the authority over the data professionals and testers required to perform the work in the manner which reflects how the rest of the work is done.
  • Adults get treated like children. A team of highly paid, well-educated professionals has a detailed project plan forced on them by a well-meaning project manager. The underlying assumption is that these professionals are unable to determine the best course of action by themselves, even though they have the skills to do the actual work. Although the project manager may be ultimately responsible for the plan, they are also responsible to do the planning with the people who are going to do the actual work.
  • Mediocrity by consensus. Consensus-based decisions often result in lowest common denominator, poor quality decisions that everyone can agree to but that won't get the job done effectively. Self organization isn't a consensus-based approach. Sometimes individuals will disagree with a decision but will choose to go along with the will of the team, and that can be a mistake. Managers must consider minority opinions at times, and occasionally work to convince the majority to give an unpopular approach a chance to succeed. An effective manager doesn't always require consensus, though it's certainly not something to rule out, either.
  • Disaster from anarchy. When teams are given the freedom to decide every issue, they often make decisions which do not reflect the overall organizational strategy. The concept of self-organizing teams does not imply anarchy. Self organization is tempered by the guiding hand of a good project manager. It is also tempered by organizational standards, infrastructure, and external regulations. "Self-organizing" doesn't mean that you have complete freedom to do what you want.

Recommended default

Project teams must have the authority to organize themselves, their work environment, and their overall approach as they see fit to effectively achieve the project results which they are responsible for. Any constraints placed on the team, such as organizational guidance, must be described to and negotiated with the team as appropriate throughout the project.

Align HR policies with IT values

Effective software development is dependent on talented individuals being motivated to collaborate effectively in a team environment. For this to work well, you need to ensure that HR concerns, such as employee incentives and rewards, career paths, and responsibilities are well aligned with the values of the IT organization.

A focus on collaboration changes how you reward and measure people. Rather than measuring only individual productivity, you instead need to measure how much value an individual provides to the team. The most valuable team members often spend most of their time mentoring other team members, a value-add activity which needs to be reflected in their overall evaluation.

Effective development requires close collaboration among a set of people who regard each others as peers. In some organizations, a job promotion from developer to architect, for example, moves your top technical talent away from day-to-day development to a separate architecture team that may not be closely tied to reality. This makes it more likely that architects create ivory tower architectures. In effective development organizations, these specialized and experienced people also serve as mentors and technical leaders, which requires them to work side-by-side with more junior people.

In each of the two above examples, the HR system needs to encourage the right behavior to implement IT values, and to ensure effective collaboration among team members.

Benefits

Aligning HR policies with IT values carries many benefits:

  • Improved software economics. Good software is built by motivated and talented people. Your HR policies are crucial to ensure that you have motivated as well as talented people, and to ensure that compensation and rewards reflects accepted values.
  • Improved motivation and increased staff retention. People are motivated to do the right things when those things are recognized, rewarded, and supported. This also leads to reduced staff attrition.

Trade-offs

There are a number of important trade-offs associated with aligning HR policies with IT values:

  • Willingness to change HR policies. An HR department is responsible for the HR policies of many different organizational units. Adjusting HR policies to the specific needs of any single department is challenging and requires investment. There are also legal implications that need to be considered, which may vary from country to country. Changes to HR policies can, however, pay off greatly through a more effective IT organization.
  • Investing in technical career path. To address top technical talent you typically need to make sure that you provide an appealing technical career path, not to force people to either choose a management career path they are not interested in or suitable for, nor to force top talent to have a stagnant career. Providing viable career paths for IT professionals is a non-trivial investment.
  • Willingness to modify behavior to attract top talent. Motivation for software engineers is often different from that of other job categories. Rewards, such as allowing an employee to build skills or develop really exciting innovative products, can mean more than traditional benefits alone. To attract top talent, it may be worth considering what type of software engineers you want to attract, and modify your behavior to be attractive to that target segment.

Anti-patterns

The following anti-patterns are relevant here:

  • Treating IT like any other part of the business. HR policies need to be adapted to the motivational needs of each target employee group. Since motivation for software engineers may differ from the motivation of other developers, it is important to understand any gap, and adjust HR policies to address those gaps. It should be noted that there may be legal restrictions on what changes can be made, and that these restrictions may vary from country to country.
  • Providing no individual flexibility. Even within IT, all individuals are not motivated by the same drivers. To reflect this, you should offer a variety of incentives aligned with teaming goals and let the employee choose which plan fits best with personal goals and circumstances.

Recommended default

Make sure that IT and HR managers engage in an ongoing discussion on how to better align HR policies to IT values.

Align team structure with architecture

Conway's law, defined by Melvin Conway in the late 1960s, tells us that any piece of software reflects the organizational structure of the group that produced it. This occurs because people will work in a manner that reflects the way they are organized. In other words, a decentralized group is likely to produce a system with a decentralized architecture. Any strengths or weaknesses in a project team's organizational structure will inevitably be reflected in the resulting system that they produce, implying that if you want to work towards an effective IT architecture, then you need to have an effective IT organizational structure.

Examples:

  1. Geography-driven architecture. A geographically distributed organization should consider an enterprise architecture that distributes ownership of each major component (frameworks, databases, applications, ...) to a single location. For example, a team in India may own the data warehouse, several applications may be owned by a team in Switzerland, and the security framework may be owned by an infrastructure team in the US. By organizing ownership of components by geographical distribution you reduce communication barriers.
  2. Quality-driven organization. If you want to achieve a high level of quality within your systems then you need to embed quality professionals within the development teams to iteratively review and advise on the ongoing work (thereby reducing communication time and decreasing opportunity for information loss via documentation). Note that you may still have an independent test team which externally validates the project team's work, but the majority of quality professionals would be embedded with the team. If quality is a secondary priority for your organization but you require documentary evidence that you reached a specific level of quality, then an external quality group that merely reviews the work of development teams may be sufficient.
  3. Usability-driven organization. If usability is a primary concern of your system then you should organize teams so that they are close to end users so as to ensure they can easily obtain feedback from the end users. You should also embed usability engineers on the project team.
  4. Reuse-driven organization. If reuse -- perhaps based on component-technology or services-based technology -- is a critical factor then you should have teams which own and support those reusable architectural elements available to help integrate the components with other projects.

Benefits

The primary benefits of aligning your organizational structure with your desired architecture are:

  • Streamlines your development efforts by making it as easy as possible for development teams work to that architecture
  • Provides opportunities for shorter iterations (thereby reducing project risk)
  • Reduces endless cycles of hand-offs, reviews, rework, etc. between groups
  • Improves overall communication on the team

Trade-offs

There are several trade-offs associated with this practice:

  • Contradictory architectural priorities. Sometimes one architectural priority, such as security, will motivate you to choose one organizational strategy, while an equally important priority, such as distribution, will motivate a completely different organizational strategy. Just as you need to make architectural tradeoffs, you will need to make organizational ones, too.
  • Enterprise restrictions. Your organization may have decided long ago on a centralized data architecture and data administration team yet now be developing decentralized systems with development teams all over the world. Luckily, no organizational structure is carved in stone and you can choose to negotiate new structures which better reflect your current needs.
  • Architecture isn't the only factor. There are many factors, not just architecture, which must be taken into account when defining organizational structure. These include availability of personnel, applicable regulations and employment laws, personalities of individuals, culture, and even something as basic as availability of office space.

Anti-patterns

The following anti-patterns are related to this practice:

  • One organizational strategy fits all. Just as systems will vary in their underlying architectures, teams will vary in their overall organizational structure. For example, one development team may be co-located in a single room and work closely with stakeholders, whereas another development team may be geographically distributed across several sites. Each team is in a different situation, working on different systems, and in different manners even though they both work for the same organization. By trying to force-fit a single organization structure on all teams you potentially put some if not all projects at risk due to organizational/architectural dissonance.
  • Organization inhibits architecture. Your organizational structure is used to justify the architectural approach for a system without consideration of the true requirements for that system. For example, because you have a centralized data administration group, you force all systems to have a centralized data store even though it may make more sense for some applications to have several data stores distributed across various sites. This decision potentially increases development and operational costs, not to mention overall technical risk, in order to streamline a relatively straightforward administration effort.
  • Architecture inhibits organization. The architecture of one or more systems is used to justify your IT organizational approach without consideration for non-architectural concerns. For example, your organization has purchased, installed, and is running a mission-critical CRM system that has a distributed architecture. Now you wish to reduce IT costs by centralizing your IT department in one location, yet the inflexible architecture of your CRM system prevents you from doing so.

Recommended default

You should organize around your architecture as the default, with cross-functional teams responsible for each major component. Alternatively your company may choose to have several "base" team structures to choose from as a starting point, then allow the team to self-organize (see the Self-Organizing Teams practice) to develop a structure which meets their specific situation.

Policies and standards

The practices in this category describe specific guidelines to support consistent operation between the various parties involved with development. The first value of Agile Software Development is "individuals and interactions over processes and tools" doesn't imply that you won't have any processes and tools at all, it simply states that they aren't as important as people and the way that they work together. With a lean approach to governance your organization will adopt tools and policies which enable developers to collaborate effectively, which motivate them to develop and reuse existing infrastructure and assets, and to do high quality work. The practices surrounding policies and standards are:

  • Integrated lifecycle environment
  • Valued IT assets
  • Flexible architectures

Integrated lifecycle environment

Software development relies on intense collaboration between team members, frequently with a temporal separation in terms of different time zones or working hours, and a spatial separation with development occurring at different sites. Collaboration is also made difficult by the size of many development efforts, where you need effective mechanisms for managing large-scale interaction. Add to this the need for management oversight and the necessary book-keeping to ensure regulatory compliance.

Integrated lifecycle environments are built to enable the above, at lowest possible cost, and provide the enabling infrastructure for most if not all of the other governance practices. Key components of an integrated lifecycle environment include:

  • Software configuration management (SCM). Most projects have thousands of moving parts, each frequently evolving with new versions, and which need to be integrated into a working application ideally many times per day. This requires tightly integrated configuration management, change management, and build management capabilities. SCM constitutes the backbone of an integrated lifecycle environment.
  • Design and construction. Developing code effectively requires a powerful IDE. Design and architectural capabilities allow you to manage the complexity of the code, and developer test capabilities ensure that the code is of high quality. Tight integration with the SCM is crucial to manage change and to enable frequent builds.
  • Test and quality assurance. As applications become increasingly mission-critical, the need for improved quality in terms of reliability, performance, stability, user experience, and function increases. Functional and performance test tools aim at achieving exactly this. On top of being tightly integrated with SCM and your design and construction environment, it is also crucial that your test and quality assurance environment are tightly integrated with the operations and systems management environment to ensure the right quality attributes in the operations environment.
  • Process, project, and portfolio management. Process integrated into the rest of the lifecycle environment helps provide consistency as well as improve efficiency and productivity by allowing us to learn from past experiences. Project and portfolio management capabilities aid in ensuring that investments are made in the right projects, projects are executed properly, and the right metrics are provided to ensure effective oversight. A tight integration with the lifecycle environment enables painless governance, in which mundane tasks -- including metrics collection and analysis -- are automated to avoid overhead. This style of management also includes "embedded compliance" (described in Part II of this series) where the process required for compliance is seamlessly integrated into the development process, with required reports being automatically generated.
  • Business and requirements analysis: To add business value, application development needs to be done in the context of a good understanding of current and target business processes, as well as a shared understanding of evolving requirements. Tight integration with the lifecycle environment is necessitated by today's rapidly evolving business environment.

Benefits

A good governance solution should manage value to ensure that what is developed is aligned with the strategy of the company in a cost-effective manner. Integrated lifecycle environments offer a number of benefits related to governance:

  • Enables effective implementation of governance practices. Integrated lifecycle environments facilitate implementation of all or almost all of the identified governance practices, especially when considering complexities, such as distributed development, compliance requirements, and large-scale development faced by most development organizations.
  • Lower Total Cost of Ownership (TCO). Integrated tool environments lower TCO by reducing support and licensing cost relative to a collection of individual tools, by reducing the costs of upgrades, training, and moving people around within your organization.
  • Enables collaboration. Especially as teams become geographically distributed and/or part of large-scale development efforts, collaboration becomes a major enabler of effective development. Integrated lifecycle environments can greatly facilitate effective collaboration in these situations.
  • Painless governance. By automating mundane bookkeeping, such as who has changed what, and metrics collection and analysis, such as quality or cost metrics, integrated lifecycle environments make governance reasonably painless once the right environment as been established.
  • Practice enabler. Many of the other practices described are enabled or facilitated by integrated lifecycle environments, including iterative development, simple and relevant metrics, and flexible architectures.

Trade-offs

Deploying an integrated lifecycle environment has a number of trade-offs:

  • Greater skills. Moving from an environment with sparse point applications to an integrated lifecycle environment requires investments in additional skills to work with the new environment.
  • Initial investment. There is also an initial investment in acquiring the lifecycle environment.
  • Continuous modernization. Tools grow old, and you need to upgrade to keep up with improved and ever-evolving technology. Since an integrated lifecycle environment consists of many closely integrated applications, you need to carefully monitor when to upgrade what tool, to make sure that you do not break integrations. This can be addressed by pushing the upgrade consistency problem to a vendor that provides a continuously updated lifecycle environment.

Anti-patterns

The following anti-pattern should be avoided:

  • Cobbled-together toolset. Is your business to build integrated tool environments, or applications? Vendors spend hundreds of millions on integrated development environments, and if you do not have very deep pockets, the risk is that you will never be able to match the specialized expertise of tool providers in the software industry, even when you're using "free" open source tools.

Recommended default

We recommend the IBM® Rational® software development platform by default.

Valued IT assets

An "IT asset" is any software component, service, template, enterprise model, reference architecture, example, pattern, guideline, or standard which the organization desires IT professionals to apply in their everyday work. A "valued IT asset" is one that IT professionals actually want to apply, typically because it is viewed as being of sufficiently high quality and relevant to the task at hand. In other words, IT professionals take advantage of the IT assets available to them because the assets actually enable them to increase their productivity. They follow the enterprise architecture and appropriate reference architectures because those assets save them time, their systems invoke existing services, and they take advantage of existing technical infrastructure all because it's easier to reuse these things than to build them again.

Ideally, IT professionals follow corporate standards and guidelines because it helps them to streamline their work, not because they're forced to do so for fear of failing an audit. True, some standards are practices which must be followed by law, giving IT professionals no choice in the matter. Luckily there is often significant leeway in most legislated standards, providing your organization the opportunity to implement them in a manner that maximizes value to IT professionals (or alternatively minimizes the pain of conformance).

Benefits

There are several benefits to supporting valued IT assets:

  1. Increased consistency. Following common guidance, such as coding standards and modeling guidelines, and applying common patterns increases the overall consistency of the work products produced by IT. This in turn leads to decreased support and maintenance costs. Guidance that is willingly followed will be applied.
  2. Increased efficiency. Increased reuse of technical assets enables you to increase overall efficiencies through the standardization of common functionality. Following consistent guidance enables you to standardize non-value add activities and thereby focus on new opportunities.
  3. Improved time to market. Increased reuse, consistency, and efficiency lead to decreased time to market, thereby improving your organization's competitiveness.
  4. Improved communication. Increased consistency of work products, the common vocabulary provided by patterns, and a common enterprise architecture help to promote communication between IT professionals. This in turn helps to streamline your overall IT effort and decreases the chance of defects resulting from miscommunication.
  5. Decreased governance costs. When IT professionals willingly apply valued IT assets on a regular basis, overall governance costs are decreased because less investment is required to enforce and validate usage of those assets.

Trade-offs

There are several trade-offs associated with this practice:

  • Investment in quality. IT assets are valued by people because they are high quality and applicable to whatever they are trying to achieve. High quality doesn't come about by accident, but instead by investment through hard work or effective procurement.
  • Quality costs more. It costs more to create a reusable asset, or to write concise guidance, than it does to create project-specific assets. You must adopt a funding strategy that doesn't punish teams for creating these assets.
  • Investment in asset support. To apply an IT asset effectively, people will need help. The asset must be easy to find and there must be people available to go to for help understanding and applying the asset.
  • Investment in asset evolution. All IT assets go stale over time without continued investment in them. For example, coding standards for Java 1.1 are of little value if developers are currently working with Java 5. A template for formal, detailed use cases will either slow down development teams or will be ignored by them if what they really need is a template for simple, light-weight use cases.
  • Reduced freedom. When IT assets are available and readily applied by IT professionals, it makes it difficult for developers to deviate from the corporate direction. Many developers want the freedom to choose the most appropriate persistence strategy for their system; they don't want to be told what existing database to use.
  • Need to come to agreement. There must be a general consensus that an IT asset is valuable. For example, Java coding standards should be written and supported by a group of respected leaders within your organization. They will also be reviewed and commented on by others, improving them further. Everyone might not agree with all of the recommendations in your Java coding standards, but they should recognize the importance of code consistency and willingly agree to follow them.

Anti-patterns

The following anti-patterns are related to valued IT assets:

  • Enforced guidance. Senior IT management declares that all IT professionals must follow and conform to a common set of standards and guidelines, then puts a rigid review process in place to ensure compliance. Although this can lead to many of the benefits indicated above, it often comes at increased enforcement costs.
  • Declared reusable. Technical assets, such as components, services, or even your entire IT infrastructure, are declared to be reusable by their owners. Something isn't reusable until it has been successfully reused, not because someone says that it's reusable. Developers will not willingly reuse assets unless they are of high quality and applicable to the task at hand.
  • Ivory tower assets. An "ivory tower asset" is one that is perfect, or nearly perfect, until it's made available to the rest of IT. An example is a highly detailed enterprise data model which took months of work to complete, a security framework which addresses all possible aspects of security required by development teams, or a highly polished set of documentation templates for requirements and design artifacts. Ivory tower assets are often provided too late to be of value to their intended audience or have far too much baggage to be of use in practice.
  • Reuse disincentive plans. There are many ways you can motivate IT professionals to not reuse existing assets. For example, measuring them on lines-of-code (LOC) motivates them to write new code instead of reusing existing code. Charging them to reuse an asset in order to fund the maintenance and support of it motivates them to simply write something specific to their needs which is directly less expensive for them.

Recommended default

We recommend that IT departments maintain coding standards for their primary development languages, a library of common patterns that they wish to promote within their systems, reference architectures for critical aspects of their technical infrastructure, and a description of the vision for their enterprise architecture. There should also be people in place to support, evolve, and purchase or harvest these valued IT assets. Wherever possible, we recommend you buy rather than build an IT asset, particularly if you can easily acquire something that meets your needs, and thereby put the maintenance burden on someone else.

Flexible architectures

One of the aims of lean software development governance is to enable flexibility and responsiveness in your business, which in turns necessitates flexibility and responsiveness in how you organize and evolve your IT systems. Lean governance is enabled by flexible architectures because they enable development teams to react effectively to the changing needs of the business. Currently the most flexible approaches to architecture are through open computing and service-orientation.

Open computing consists of three major components:

  • Open standards. Standards promote interoperability by using open, published specifications for API's, protocols, and data and file formats.
  • Open architectures. Good architectures enable us to build loosely coupled, flexible, reconfigurable solutions. Service-oriented architecture (SOA) is an example of an open and flexible architectural style, which also has the advantage of providing a close coupling between business processes and software capabilities, to ensure strategic alignment.
  • Open source software (OSS). OSS tools are crucial to driving innovation by leveraging community development and collaboration.

SOA builds on open computing, which allows systems to be dynamically composed by a set of loosely coupled services closely linked to the business processes they are supporting. This provides good alignment between IT assets and business needs, while delivering the flexibility and responsiveness required for the agile business.

It should also be noted that while SOA has obvious benefits, it does add complexity to the governance process, since moving from vertical silos to an SOA requires us to manage a many-to-many relationship between business units and IT assets in terms of funding, management of service-level agreements (SLAs), and requirements management.

IBM Software Group is one of many organizations that recently has gone through a transformation of the architecture underlying its products. In the past, IBM's software products were built on proprietary hardware, operating systems, and application components. IBM Software Group has broadly adopted standards and open source efforts, and embraced SOAs. This has catapulted innovation as well as our ability to provide integrated solutions, and it has provided us and our customers with increased flexibility and responsibility. As an example, the recently launched product IBM WebSphere® Service Registry and Repository was developed in less than a year with excellent customer reception, an endeavor that in the past would have taken us years.

Benefits

Flexible architectures carry several benefits:

  • Supports evolving business needs. The combination of rapidly evolving businesses and IT being an increasingly crucial part of the business puts great pressure on IT to evolve fast. Flexible architectures provide the technical underpinnings to enable that to happen.
  • Reduced time-to-market. Flexible architectures enable you to more rapidly assemble applications from reusable components/services since they are loosely coupled. This enables broader reuse which in turn greatly reduces time-to-market.
  • Better ROI and lower TCO. By improving reuse and making it easier to assemble applications, you lower overall development costs. Reliance on industry standards also ensures availability of multiple offerings to address your needs, keeping prices lower versus proprietary technology solutions.
  • Reduced technical risk. Since flexible architectures increasingly rely on industry standards, they reduce your dependence on any one vendor. This provides you with increased operational freedom.

Trade-offs

Deploying flexible architectures has a number of trade-offs:

  • Requires investment. Flexible architectures do not come for free. You need to invest in skills, in technology and in transformation from today's assets to tomorrow's loosely coupled assets. To achieve the promise of reuse you also need to invest in reference architectures working examples of a portion of your overall architecture.
  • Requires evolution. Flexible architectures need to evolve with technology and standards. This requires ongoing skills and technology investments.
  • Requires quality. Flexible architectures facilitate reuse, but reuse requires a stronger emphasis on quality. Without sufficient quality, your reusable components will require more maintenance and support overhead than you save on reuse.

Anti-patterns

The following are anti-patterns related to flexible architectures:

  • Stove-piped architectures. Having vertical or stove-piped architectures reduces reuse and business flexibility. Just as technology has leveled the playing field for business and business opportunities the world over (see Thomas Friedman's The World is Flat: A Brief History of the 21st Century), our architectures need to accommodate the latest modes of social networking (e.g., Web 2.0) and distributed and highly outsourced business models allowing 24x7 operations.
  • Proprietary technology. Even though you can build flexible architectures with proprietary technology, it creates vendor lock-in, reduces flexibility in future sourcing, and likely results in higher total cost of ownership. The consequence: you are left behind as technology evolves beyond the proprietary solution.
  • Never committing to a technology. It is hard to choose a technology, in part because whatever choice you make may prove sub-optimal tomorrow. Never committing will, however, be worse, since you need to take decisions to evolve.

Recommended default

By default, we recommend service-oriented architectures, emerging architectures, open source technologies, such as Eclipse, and standards such as those associated with Java 2 Enterprise Edition (J2EE), the Object Management Group (OMG), and the Institute of Electrical and Electronics Engineers, Inc. (IEEE).

Conclusion

Every IT organization has a development governance program in place, but this program may not be explicit and it may not be very effective. In the past, traditional approaches to development governance were akin to herding cats -- management would do a lot of work providing direction but the developers would act on their own accord anyway. These traditional approaches to development governance haven't worked out because developers are intellectual workers who don't work well under command-and-control approaches. For a governance program to work it must reflect the actual environment in which it is being applied, and as always when it comes to management the primary factor is the people involved.

Effective governance isn't about command-and-control but instead is focused on enablement through collaborative and supportive techniques. This is akin to leading cats and it proves to be far more effective to motivate people to do the right thing instead of trying to force them to do so. In this article series we overviewed a collection of practices for lean development governance which reflects the realities of modern software development. These practices showed how to embed governance in your tools, processes, and development guidance to make it as easy as possible for people to do the right thing. They are a new way of looking at development governance which reflects modern approaches to development such as Lean Software Development, the IBM Rational Unified Process® (RUP®), Extreme Programming (XP), Dynamic Systems Development Method (DSDM), and Scrum. Development governance is only part of the overall IT governance picture, but an important one. Stay tuned.

Notes

1 The values of Agile software development are posted at www.agilemanifesto.org

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational
ArticleID=245869
ArticleTitle=Best practices for lean development governance, Part III: Roles and policies
publish-date=08152007