Einstein once said that if he had one hour to save the world, he would spend 55 minutes defining the problem and only 5 minutes looking for a solution. Except for the rate of time spent on problem and solution, I wholeheartedly agree with the importance that he placed on understanding the problem before trying to solve it.
Consequences of not fully understanding the problem
In software engineering, understanding a problem is something that must be done early in the System Requirements definition stage. A problem that is not well-understood causes poorly defined and incomplete requirements that lead to unsuccessful projects. For years, the statistics related to failing projects and their related costs have been discouraging. To mention just a few of them:
According to the Standish Group (see Resources):
- 41% of projects fail to add value and adequate return on investment (ROI)
- 49% of projects exceed the initial cost estimates
- Only 28% of projects are finished on time and within budget
The Standish Group also associates that lack of success with the following factors:
- Lack of user engagement
- Unclear business objectives
- Failure to capture basic requirements
- Lack of control over project scope
- Lack of executive support
IDC (International Data Corporation) is even more specific when pointing out causes for failure. They found that more than 80% of failures in software development are a direct result of problems in requirement discovery, management, or analysis.
And according to IAG Consulting, more than 40% of the IT budget will be spent on poor requirement specifications.
These figures show that Einstein was right.
Why there is resistance to investing the time
In terms of software development, investing time in the understanding the problem means understanding the business objectives, correctly identify the stakeholders, ask the right questions to explore the problem, and use suitable techniques to describe what the system should do and why it is being created. Although it appears obvious, why is it often so difficult to spend time understanding the problem, even though people in the software industry understand the need to do so?
I believe the difficulty is for the following reasons:
- The anxiousness to find a fast solution for something required is typical of human nature. An inexperienced analyst feels especially challenged to solve the issue without a detailed understanding of the requirement. Often, he thinks about whether he will use that component that he made for the previous project while the user keeps explaining his difficulties in vain. This anxiety over a solution occurs at many levels. The project manager and the area coordinator both perceive progress only when they see something tangible, like lines of code. This is not a problem as long as they do not encourage starting the implementation while ignoring the importance of understanding business problems.
- System analysts and engineers do not feel comfortable venturing into knowledge areas that they are often unfamiliar with, such as business processes and industry expertise. It is always easier to talk about tables, reports, functions, and data.
- IT professionals do not receive suitable training in this subject area from most universities and colleges. We are well trained in terms of computer science and engineering. We understand operating systems, statistics, calculus, physics, compilers, programming languages, etc. There is no focus on humanities or psychology so that we can learn to communicate with people who lack this technical knowledge.
- Low user engagement during the project is often related to the culture imposed by the cascade development process (also known as waterfall development), still widely used in Brazil, which engages the user only at the beginning of the project for definition of requirements and at the end for system acceptance of the system
Three ways to change the culture
The good news is that there are ways to overcome these difficulties. The culture shift needed to implement or increase the maturity in a software engineering discipline requires investment in building these three pillars: process, tools, and people.
Invest in the process
In terms of process, there has been a great evolution in the last few years. Some noteworthy examples are the traditional Rational Unified Process, agile processes such as the Agile Unified Process or XP (eXtreme Programming), or scrum project management methods. Each has individual characteristics and, in varying degrees of emphasis, defend basic points related to requirements, such as:
- Ongoing user and stakeholder engagement
- Not specifying all requirements at the beginning of the project, but doing it in iterations instead
- Using appropriate techniques to facilitate the process.
Scott W. Ambler, in his book title Agile Modeling: Effective Practices for Extreme Programming and the Unified Process, explains the importance of modeling and documentation in any software project, including projects that use the agile approach. He emphasizes that traditional techniques, such as joint application development (JAD), use cases, interviews, and observations, among others, can be adapted to agile processes, and that the ongoing engagement of stakeholders is essential for success. What changes, depending on the process, is the degree of formality in using the technique and the time spent on activities.
Invest in the tools
As for tools, some extreme agilists defend the use of paper only. Others use a number of techniques and tools. Considering that the system will remain in the organization for years or decades and will undergo maintenance, we must use resources that make it easier to document and change it. Aside from the fact that geographically distributed development is becoming commonplace, the need for tool support for collaborative development is imperative.
In that respect, IBM® Rational® Requirements Composer is an excellent option, both for agile development in projects that demand less formality and for traditional projects that often are subject to regulation and auditing. It is built on IBM® Rational® Jazz™ technology, which is a collaborative platform, and it includes features to support problem understanding, requirement discovery and requirement definition stages.
Invest in the people
As for people, we need first to select analysts with the appropriate experience and to provide training and mentoring. There are techniques and best practices that can improve teamwork. Donald Gause and Gerald Weinberg (see Resources) define a problem as the existing gap between the perceivedcurrent situation and the desired situation. Given this definition, there are different ways to solve a problem.
Providing the desired situation is not always the best solution. A change in the perception of the current situation might change definition of the problem. Most requests for improvements received by Microsoft for products such as Word, for instance, ask for features that the product already has.
The better solution would be to change current perceptions, perhaps through training. Explaining the costs and other aspects involved in a solution might change the client's perception of the desired situation. Sometimes, clients ask for features that will not solve their problems, so they will not be used because their perceptions of the desired situation might be distorted. Evidence of this is a figure published by the Standish Group: "Typically, between 20-40% of code lines in systems go unused."
Other benefits of analyzing the problem
Problem analysis techniques teach us to ask the right questions, shift the focus to explore different points of view, and break down the problem to find the root of what looks like a problem but is only a symptom.
What is most impressive is that these skills generally prepare professionals for success in other fields. Success in sales is directly related to the ability to understand the client's business problems. Marketing professionals must understand market needs and consumer behavior. Good communication and problem analysis skills help us in our family relations. Being able to understand the reasons behind your teenage child's behavior can be harder than understanding the resistance and difficulties that you face in understanding your client's situation. You need to know how to deal with people, with differing perceptions, and how to put yourself in someone else's shoes to really understand their difficulties and needs. Some have an innate ability, while others have to recognize their limitations and hone their skills through appropriate techniques and training.
Curiously, 37 years ago Gerald Weinberg published his book called The Psychology of Computer Programming (see Resources), which is still relevant today, and the same basic and obvious issues have persisted for decades. According to him, a little application of psychology and the humanities can be the key to personal success, as well as to the success of software development teams.
Why executive-level support is essential
For the three pillars -- process, tools, and people -- to work in harmony within the organization, executive support is key, because the anxieties and the pressure to "deliver it yesterday" must be properly managed. The figures show that maintaining the status quo is bad for both business and IT. Think of something to change the future of software engineering, thereby changing the perception of IT, as being a cost center to a sector that adds value to business through technological innovation.
- Publications cited:
- Ambler, Scott. Agile Modeling: Effective Practices for eXtreme Programming and the Unified Process (Wiley; 2002)
- Gause, Donald C. and Weinberg, Gerald M. Exploring Requirements: Quality Before Design (Dorset House, 2011)
- Weinberg, Gerald M. The Psychology of Computer Programming: Silver Anniversary Edition by Gerald M. (Dorset House, 1998)
- Studies cited:
- Companies with Poor Requirements [Specifications] Spend an Average of $2.24M More per Project (IAG Consulting, 2008)
- 2004 CHAOS Report, the Standish Group
- International Data Corporation (IDC)
- To learn more about Rational Requirements Composer, start with the developerWorks page, and then review the product overview page and the features and benefits. For details and documentation, check the information center.
- Visit the Rational software area on developerWorks for technical resources and best practices for Rational Software Delivery Platform products.
- Stay current with developerWorks technical events and webcasts focused on a variety of IBM products and IT industry topics.
- Improve your skills. Check the Rational training and certification catalog, which includes many types of courses on a wide range of topics. You can take some of them anywhere, any time, and many of the "Getting Started" ones are free.
Get products and technologies
- Download the Rational Requirements Composer free trial or try it on the cloud.
- Evaluate IBM software in the way that suits you best: Download it for a trial, try it online, use it in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement service-oriented architecture efficiently.
- Join the Rational Requirements Composer forum to ask questions and participate in discussions.
- Rate or review Rational software. It’s quick and easy.
- Share your knowledge and help others who use Rational software by writing a developerWorks article. Find out what makes a good developerWorks article and how to proceed.
- Follow Rational software on Facebook, Twitter (@ibmrational), and YouTube, and add your comments and requests.
- Ask and answer questions and increase your expertise when you get involved in the Rational forums, cafés, and wikis.
- Get social about thought leadership. Join the Rational community to share your Rational software expertise and get connected with your peers.