From the IBM WebSphere Developer Technical Journal.

What is scope creep?

Scope creep (also called requirement creep) in project management refers to uncontrolled changes in a project's scope. This phenomenon can occur when the scope of a project is not properly defined, documented, and controlled. (from IEEE Spectrum – Why Software Fails)

As a software engineering practitioner, architect, or project manager, this topic probably strikes a chord with you. Any seasoned IT professional undoubtedly has stories to tell about projects that grew so much in scope to ultimately incur costs and resources far exceeding even the most prudent expectations. While there almost certainly is no cure-all for the scope creep phenomenon, there are strategies for minimizing such situations from occurring.

Why is it significant?

If you are a developer, you may see yourself as the keeper of the code who simply provides technical skills to the greater organization; in other words, the requirements for avoiding scope creep is the responsibility of those in management or high-level architectural roles. I hope to change the common perspective that requirements only serve to formalize documentation and are just of concern to a particular role in an IT organization.

In my work as a WebSphere Consultant, scope creep is of paramount concern. Studies show that only 5-15% of enterprise systems make it from requirements to production. Enterprise software development is hard; it takes the careful alignment of enormous resources, collaboration, and discipline to put, for example, a banking system on Wall Street into production. Although I do keep up with some of the premier technologies of the information age, such as Web services, Service Component Architecture (SCA), and J2EE, I have found that the most value I can bring to an organization I work with is not so much to provide technical best practices, but to perform risk analysis and solidify requirements that are realistic. Targeting scope creep early and knowing best practices for enabling an organization to combat it is often a key insight to bringing a project into that 5-15% bracket of solid success.

Back to top

Techniques for defining realistic requirements

As the definition above stated, the fundamental cause of scope creep more often than not revolves around requirements. If the requirements are loosely defined, then consider the risk of scope creep exponentially increased. This observation alone, however, is not sufficient. I have seen some extremely detailed requirements documents that don't help avoid scope creep. In one case, a documentation specialist may write requirements without getting feedback from development or the end user; in another more common case, the requirements are simply ignored by practitioners -- but who can blame them when the requirements are contained in a 100-page technical document that is extremely difficult to read?

Consider the following recommendations to help you develop effective requirements that can make a difference.

1. Solidify requirements with test-driven development

My favorite best practice for combating scope creep is an underpinning of agile methodologies, such as Extreme Programming: Test assertion code that is written before the application code can serve as unambiguous, solid requirements. For instance, a good test assertion is a JUnit test that returns true if and only if the SOAP envelope returned by a Web service is identical to a pre-defined XML document.

Although this is a great way to define requirements that can be easily understood by a development organization and clearly validate software system functions, test cases alone are rarely appropriate as complete requirements. Projects that are best suited for test assertions as requirements are those where the end users of the application are themselves technical. In fact, if the customer is not technical, using test assertions alone can quickly turn into an anti-pattern where test case green lights have little meaning to the end user.

For example, an ideal situation for using test assertions to solidify requirements is the development of a component that is called by another application. Imagine the development of a custom J2EE JCA connector for accessing a legacy system, to be used remotely by a workflow application. The end users in this scenario are likely to be the architects and developers of the workflow application, and thus using executable test assertions as their requirements contract with the JCA connector development team is appropriate.

I am a strong proponent of test frameworks, such as JUnit, httpUnit, and Cactus. If you often develop J2EE applications and have not put a JUnit.jar on your classpath and run through the JUnit.org Test Infected Tutorial, I highly recommend you take twenty minutes to write your first assertion.

2. Use use cases that truly illustrate the end user's needs

It is my experience that most enterprise development shops are very aware of the need for use cases as part of the requirements process, even though they are often regarded as a necessary evil and the use case documents that are produced provide little value during development. Bulleted text is often a poor mechanism for writing effective use cases. As technology practitioners, we have a wealth of tools available for defining an end user's needs. For example, if the user is a call center specialist using a Web portal, dedicate a meeting solely to what form the use cases will take with an agreement by everyone that the content of the use cases will be deferred. In this particular scenario, some suggestions ways to represent use cases are:

• User stories. Extreme Programming requires user stories that fill the role of use cases. These are short paragraphs written by the end-users only that do not include any technical terms.

• Problem questions. Studies show that when an issue is framed as a question, an environment is created where the parties involved can brainstorm possible solutions. Consider asking questions that come directly from the end users, such as "How can we get a list of previous customer calls on the status screen?" Each question can be numbered as a use case with additional context surrounding the problem added to each. The context should not include the solution or any technical jargon.

• Process diagrams. UML tools or business process tools can be used to create process diagrams that illustrate an end user scenario in terms of the business need.

• User interface views. Implementing simplified view components of the pending application can be very effective as part of the requirements process. For example, very basic HTML without graphics or advanced formatting -- even white-board sketches, story board drawings, or paper prototypes for that matter -- can help the end user visualize what the system will resemble.

3. Get it working THEN get it right

Consider either an iterative or incremental software engineering approach. Regardless of an organization's culture and chosen software development methodology, implementing a vertical slice of an application (that is, some functionality that touches each component of the prospective final system) can greatly reduce risk and help identify key issues that affect the scope of a project. The vertical slice can then be refactored until the final system requirements are met. Technologies that support a component architecture (such as J2EE, SCA, JSF, and the Eclipse plug-in architecture) can greatly reduce the complexity of refactoring.

Although I strongly believe in the principle of a vertical slice, following this principle properly takes a lot of discipline and should not be done in a rush. The difficulty arises in the common case where the enterprise system requires many underlying services and prerequisite components for the vertical slice to function.

For example, consider the case where the initial development of a system focuses on getting a customer account. Truly implementing this function properly requires connection to an LDAP server, building an object-relational mapping layer to abstract a database, constructing Web services, security, proper error handling, stateless business logic to facilitate clustering, integration with CICS, and so on. To build a vertical slice that honestly reflects the final application, each of the prerequisite components and services required must either be built such that they can be efficiently refactored for the final application, or implemented with extensive foresight so they can be used as-is by the final distributed system. Having this discipline and convincing the project stakeholders of this perspective is key to avoiding scope creep.

4. Plan to change the plan

Few organizations actually accept the likely possibility that the initial requirements for a system will continuously change. Unfortunately, the terminology borrowed from brick and mortar architecture often causes serious problems in our field: software requirements are not blueprints. In traditional building construction, the requirements are generally static; the space accommodations and location of the building will always stay the same, the building does not have to rely on the architecture of another building, and even a young child can immediately determine the status of a building's construction with a single glance. Compare the accuracy of these methods of determining a building's construction status with that of any metric available to a software development project manager. Spectacular errors are commonplace. A 90% complete status estimate on a project that will require years of additional development due to unforeseen complexity is typical.

A critical best practice for avoiding scope creep, therefore, is to facilitate the change of requirements at any stage of the development process. Changing a single use case should not require dozens of meetings and executive approval. Consider these recommendations:

• Have a process by which anyone involved in the project can suggest a requirement improvement. It is critical that everyone have access to the requirements and that everyone is empowered to suggest changes. For instance, a wiki or mailing list dedicated to the project's requirements can be effective in this regard. At the very least, most change management products can save the project schedule or requirements in HTML form for easy periodic publication. My personal opinion is that requirements should be numbered, brief, and to the point so busy developers can easily keep track of them.

• Build in user feedback every step of the way. Strive to have checkpoint sessions with the end user at least as much as you do with management or other stakeholders. If a requirement change is suggested, user feedback should be solicited.

• Incorporate risk analysis. Most popular software engineering methodologies strongly advocate that risk analysis be tracked throughout the development lifecycle. For instance, the Rational Unified Process requires that major risk factors be identified and tracked by architectural impact. Similar to requirements tracking, I strongly recommend that risks be brief, easily available to (and updateable by) everyone, and prioritized by severity.

Back to top

Conclusion

Scope creep is a natural phenomenon in enterprise software development. Expect it and build mechanisms into your software development process to dynamically react to it. Hopefully, you have found something useful in this commentary to help make your work more successful, rewarding, and under control.

Resources

• IEEE Spectrum: Why Software Fails
  
  
• Wikipedia.org Scope Creep definition
  
  
• XP and Unit Testing
  
  
• Simple JUnit Tutorial
  
  
• Iterative & Incremental Development
  
  
• Rational Unified Process
  
  

About the author

Robert R. Peterson is part of the enablement team under IBM Software Services for WebSphere. He works to ensure that the WebSphere portfolio of products brings IBM's clients the greatest value possible. Robert is an accomplished inventor and co-author of WebSphere Application Server V6: Performance and Scalability. He is an alumni of IBM's prestigious Extreme Blue Program and holds a M.S. in Computer Engineering from the University of Florida.

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• What is scope creep?
• Techniques for defining realistic requirements
• Conclusion
• Resources
• About the author
• Comments

Next steps from IBM

Spring, JRuby, and Ajax development is easier with WebSphere Application Server - a smart Java 5 and J2EE Web services-based application server.

---

• Try: The no-charge WebSphere Application Server Community Edition is a pre-integrated, lightweight Java 5 application server built on Apache Tomcat and other best-of-breed open source software such as OpenEJB, Apache Axis, and Apache Derby.
• Article: The article leverage the Spring Framework and the WebSphere Application Server to improve your J2EE project productivity.
• Tutorial: See how the free WebSphere Application Server and XML can improve the efficiency of your JRuby on Rails and Ajax development.
• Buy: WebSphere Application Server - Express

My developerWorks community

Dig deeper into WebSphere on developerWorks

DB2 pureScale

See how true transparent scaling architecture can be achieved on distributed platforms

Special offers