We must be more like psychologists than engineers to succeed in software engineering

Why investing time in understanding the problem that software is to solve is a problem in itself, and how to solve it

In software development, Einstein's advice to invest time in understanding the problem means to understand the business objectives, correctly identify the stakeholders, ask the right questions to explore the problem, and use suitable techniques to describe what the system should do and why it is being created. But investing time in the problem can be a problem in itself, says the author, even though software engineers understand the need. She suggests why this happens and how to overcome the resistance to investing the time.


Marília Coelho, IT Specialist, IBM  

Marília Coelho is an IT Specialist at IBM, in Brazil, and a certified IBM Rational Consultant for Rational Unified Process v2003. Marília has been working closely with customers to improve their ability to develop and deliver software applications with quality, on time and within budget.

20 March 2012

Also available in Chinese

Einstein once said that if he had one hour to save the world, he would spend 55 minutes defining the problem and only 5 minutes looking for a solution. Except for the rate of time spent on problem and solution, I wholeheartedly agree with the importance that he placed on understanding the problem before trying to solve it.

Consequences of not fully understanding the problem

In software engineering, understanding a problem is something that must be done early in the System Requirements definition stage. A problem that is not well-understood causes poorly defined and incomplete requirements that lead to unsuccessful projects. For years, the statistics related to failing projects and their related costs have been discouraging. To mention just a few of them:

According to the Standish Group (see Resources):

  • 41% of projects fail to add value and adequate return on investment (ROI)
  • 49% of projects exceed the initial cost estimates
  • Only 28% of projects are finished on time and within budget

The Standish Group also associates that lack of success with the following factors:

  • Lack of user engagement
  • Unclear business objectives
  • Failure to capture basic requirements
  • Lack of control over project scope
  • Lack of executive support

IDC (International Data Corporation) is even more specific when pointing out causes for failure. They found that more than 80% of failures in software development are a direct result of problems in requirement discovery, management, or analysis.

And according to IAG Consulting, more than 40% of the IT budget will be spent on poor requirement specifications.

These figures show that Einstein was right.

Why there is resistance to investing the time

In terms of software development, investing time in the understanding the problem means understanding the business objectives, correctly identify the stakeholders, ask the right questions to explore the problem, and use suitable techniques to describe what the system should do and why it is being created. Although it appears obvious, why is it often so difficult to spend time understanding the problem, even though people in the software industry understand the need to do so?

I believe the difficulty is for the following reasons:

  • The anxiousness to find a fast solution for something required is typical of human nature. An inexperienced analyst feels especially challenged to solve the issue without a detailed understanding of the requirement. Often, he thinks about whether he will use that component that he made for the previous project while the user keeps explaining his difficulties in vain. This anxiety over a solution occurs at many levels. The project manager and the area coordinator both perceive progress only when they see something tangible, like lines of code. This is not a problem as long as they do not encourage starting the implementation while ignoring the importance of understanding business problems.
  • System analysts and engineers do not feel comfortable venturing into knowledge areas that they are often unfamiliar with, such as business processes and industry expertise. It is always easier to talk about tables, reports, functions, and data.
  • IT professionals do not receive suitable training in this subject area from most universities and colleges. We are well trained in terms of computer science and engineering. We understand operating systems, statistics, calculus, physics, compilers, programming languages, etc. There is no focus on humanities or psychology so that we can learn to communicate with people who lack this technical knowledge.
  • Low user engagement during the project is often related to the culture imposed by the cascade development process (also known as waterfall development), still widely used in Brazil, which engages the user only at the beginning of the project for definition of requirements and at the end for system acceptance of the system

Three ways to change the culture

The good news is that there are ways to overcome these difficulties. The culture shift needed to implement or increase the maturity in a software engineering discipline requires investment in building these three pillars: process, tools, and people.

Invest in the process

In terms of process, there has been a great evolution in the last few years. Some noteworthy examples are the traditional Rational Unified Process, agile processes such as the Agile Unified Process or XP (eXtreme Programming), or scrum project management methods. Each has individual characteristics and, in varying degrees of emphasis, defend basic points related to requirements, such as:

  • Ongoing user and stakeholder engagement
  • Not specifying all requirements at the beginning of the project, but doing it in iterations instead
  • Using appropriate techniques to facilitate the process.

Scott W. Ambler, in his book title Agile Modeling: Effective Practices for Extreme Programming and the Unified Process, explains the importance of modeling and documentation in any software project, including projects that use the agile approach. He emphasizes that traditional techniques, such as joint application development (JAD), use cases, interviews, and observations, among others, can be adapted to agile processes, and that the ongoing engagement of stakeholders is essential for success. What changes, depending on the process, is the degree of formality in using the technique and the time spent on activities.

Invest in the tools

As for tools, some extreme agilists defend the use of paper only. Others use a number of techniques and tools. Considering that the system will remain in the organization for years or decades and will undergo maintenance, we must use resources that make it easier to document and change it. Aside from the fact that geographically distributed development is becoming commonplace, the need for tool support for collaborative development is imperative.

In that respect, IBM® Rational® Requirements Composer is an excellent option, both for agile development in projects that demand less formality and for traditional projects that often are subject to regulation and auditing. It is built on IBM® Rational® Jazz™ technology, which is a collaborative platform, and it includes features to support problem understanding, requirement discovery and requirement definition stages.

Invest in the people

As for people, we need first to select analysts with the appropriate experience and to provide training and mentoring. There are techniques and best practices that can improve teamwork. Donald Gause and Gerald Weinberg (see Resources) define a problem as the existing gap between the perceivedcurrent situation and the desired situation. Given this definition, there are different ways to solve a problem.

Providing the desired situation is not always the best solution. A change in the perception of the current situation might change definition of the problem. Most requests for improvements received by Microsoft for products such as Word, for instance, ask for features that the product already has.

The better solution would be to change current perceptions, perhaps through training. Explaining the costs and other aspects involved in a solution might change the client's perception of the desired situation. Sometimes, clients ask for features that will not solve their problems, so they will not be used because their perceptions of the desired situation might be distorted. Evidence of this is a figure published by the Standish Group: "Typically, between 20-40% of code lines in systems go unused."

Other benefits of analyzing the problem

Problem analysis techniques teach us to ask the right questions, shift the focus to explore different points of view, and break down the problem to find the root of what looks like a problem but is only a symptom.

Problem analysis techniques teach us to ask the right questions, shift the focus to explore different points of view, and break down the problem to find the root of what looks like a problem but is only a symptom.

What is most impressive is that these skills generally prepare professionals for success in other fields. Success in sales is directly related to the ability to understand the client's business problems. Marketing professionals must understand market needs and consumer behavior. Good communication and problem analysis skills help us in our family relations. Being able to understand the reasons behind your teenage child's behavior can be harder than understanding the resistance and difficulties that you face in understanding your client's situation. You need to know how to deal with people, with differing perceptions, and how to put yourself in someone else's shoes to really understand their difficulties and needs. Some have an innate ability, while others have to recognize their limitations and hone their skills through appropriate techniques and training.

Curiously, 37 years ago Gerald Weinberg published his book called The Psychology of Computer Programming (see Resources), which is still relevant today, and the same basic and obvious issues have persisted for decades. According to him, a little application of psychology and the humanities can be the key to personal success, as well as to the success of software development teams.

Why executive-level support is essential

For the three pillars -- process, tools, and people -- to work in harmony within the organization, executive support is key, because the anxieties and the pressure to "deliver it yesterday" must be properly managed. The figures show that maintaining the status quo is bad for both business and IT. Think of something to change the future of software engineering, thereby changing the perception of IT, as being a cost center to a sector that adds value to business through technological innovation.



Get products and technologies



developerWorks: Sign in

Required fields are indicated with an asterisk (*).

Need an IBM ID?
Forgot your IBM ID?

Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.


All information submitted is secure.

Dig deeper into Rational software on developerWorks

Zone=Rational, DevOps
ArticleTitle=We must be more like psychologists than engineers to succeed in software engineering