Why is it that some software test teams succeed while others fail? Is it simply the nature of specific tasks the team is assigned? In other words, is the difference between success and failure preordained, or otherwise out of the team's control? Or is the key to success simply a matter of assembling a collection of "superstars" and then getting out of their way? Conversely, maybe success depends completely on effective team leadership. What's the answer here?

The short answer is that there is no simple solution. Any team has to balance its autonomy with the need to respond to external governing forces and inter-team responsibilities. Hiring stellar performers can obviously help in building a team, but it's not a guarantee of success.

This article examines some specific characteristics that set high-performance test teams apart. The objective is to help software test engineers and managers understand these characteristics and how to cultivate them in their own teams.

Characteristics of good test teams

There are many clichés people use to describe what makes a team a high-performance team: "There's no 'I' in team" and "The whole is greater than the sum of its parts," for example. These recognize that a good team acts not as a collection of individuals, but as a cohesive unit. However, effective software test teams also share more subtle (and less clichéd) characteristics. We'll start by examining these.

Test teams have well-defined roles
To paraphrase John Donne, no software engineer -- and no team of software engineers -- is an island. You may enjoy writing code for hours on end with no one but your iPod for company, but your success still depends upon your fellow team members' efforts. And your team's success depends upon the efforts of other teams.

For interdependent software development and testing efforts to succeed, a team has to clearly define sets of responsibilities, deliverables, and protocols that govern how team members will interact with one another as well as how the team will interact with other teams. In other words, a team has to establish social contracts for intra-team and inter-team tasks; these define the roles of team members as individuals and the role of the team in relation to other teams.

Why is this necessary? First, it enables the team to concentrate on achieving its goals; no one has to play detective to find out what the goals really are or play lawyer to defend them.

Second, in the absence of such a social contract, a software project -- or any other complex human enterprise, for that matter -- will end up functioning as a "nation of men, and not of laws." In other words, the processes that control how the project team and its members operate will be based solely on the personal experiences, judgment, and failings of the specific set of people involved. There won't be a well-defined, well-documented, set of rules that everyone can rely on.¹ Project success will rely solely on those people's ability to anticipate problems and each other's needs, empathize with each other's problems, and be selfless in pursuing a path for the project's greater good. How many of the people you know would you trust in such an environment?

The degree to which software projects are governed by a defined development and test process varies widely, depending upon the project environment. Some projects are organized around a well-defined, comprehensive process such as IBM Rational Unified Process®, or RUP®²; others approach things in an ad hoc manner. In the latter environment, the software test team has to start from scratch to define its role and establish those social contracts with other project teams. In the former environment, the task of defining the testing team's role is smaller, but the project leaders cannot simply declare, "We'll follow the process." They must take into account the project's unique needs and requirements. As Thomas Watson, Sr., always said, "You have to remember to THINK"!

Test teams are diverse
These days, when people hear the word "diversity," they think of relatively recent efforts to ensure that a company's workforce reflects the ethnic, racial, and social makeup of society at large. When you're assembling a software testing team, you also have to think of diversity in terms of the team members' skills, personalities, and experiences. Although you might think software testing team members should be relatively homogenous, the strongest teams are composed of a diverse set of individuals. Let's take a closer look at this; then we'll review the personality types and skill sets that you'll want to combine in assembling your team.

People often use sports analogies to describe business or engineering team dynamics; in fact, these analogies have become clichés in business communications.The funny thing about many clichés, however, is that at one time they reflected accurate and original thinking. For example, if you grow up near Boston (Massachusetts, USA) as I did, and you have any interest in sports, then you quickly learn the story of the 1967 Boston Red Sox baseball team. That year, the team won its first league championship pennant after more than twenty years of failure. They did it not by assembling a team composed only of superstars, but rather by assembling a team with a small number of bona fide "stars," a large number of younger players with potential but not much experience, and experienced, but not famous, players that filled a variety of different roles. It was a very diverse, and, coincidentally, a very exciting team. It was successful, not just because some individual players had stellar seasons, but also because of how the team members' diverse skills and personalities meshed.

This same type of diversity is important in building a successful software testing team. It's relatively easy to quantify skills such as proficiency in a specific programming languages such as Java, or experience with an architecture such as J2EE. But what about the other experiences, thought processes, habits, practices, and outlooks, that you'll want in your team members?

Desirable types for test teams
In an ideal world, you'd be able to assemble a test team by selecting from a catalog of "character types," based on your project's requirements, almost as if you were casting characters for a play or movie. What types would you want on your team to help ensure its success? Let's take a look at some possibilities.

• The Early Adopter. One aspect of working in software engineering is the simultaneous joy and pain of "constant change." Just when you think you're up to date with a technology or tool, a new version or new product gets released, and you're behind the curve again. And there is no choice about what to do: You must strive to keep up with the latest advances. If you stand still, you'll quickly fall behind. Therefore, you need someone on your team who revels in exploring the latest software and recommending additions for your team's software test environment.
  
  
• The Constant Adapter. This person complements the Early Adopter, by working to adapt both new and existing software tools to your team's specific needs. For example, suppose the Early Adopter found a tool intended to help database administrators recover from catastrophic failures. The Constant Adapter might learn how to use this tool and then apply it to another purpose, enabling testing team members to reconstruct databases and restore a "clean" testing environment after they perform a series of tests.
  
  
• The Happy Integrator. What's the one type of testing that always takes more time than management plans, uncovers serious, hard-to-debug problems, and is generally considered to be less important than other types of tests? The answer: integration testing. These days, it seems that no teams actually write all the software for their products. Instead, they build significant parts of their products using either code supplied by other companies or open source code. Performing integration testing on code supplied from multiple sources is quite different from testing code from a single source. You have to verify the product's operation in terms of the boundaries (and potential gaps³) between the integrated subsystems. This type of testing drives some people crazy, as it involves understanding and deciphering data and process flows between subsystem pieces. Fortunately, however, some folks actually revel in this type of "detective work."
  
  
• The Experienced Miner. There's a story about an old coal miner who didn't need geologic data to find coal deposits. He had been a miner for so long, that he could intuitively locate the coal and loosen it with a single swing of a pick-axe. He could sense the right place to hit so that the rock crust would cleave and reveal the coal, in the same way that a sculptor can sense exactly where to hammer a chisel into a stone. Some people are like that when it comes to testing software; regardless of the circumstances, they're able to find just the right place to exercise the software under test to find critical bugs. Sometimes this skill is based on experience with the type of software under test. For example, people who have direct experience with the design and operation of Java servlets are good at finding flaws in servlet-based applications.⁴ In other cases, a software test engineer may know exactly where to find "killer bugs," based on experience with similiar types of projects. They may also have worked on projects with similar design and project management problems. For example, some software designs simply must change and evolve throughout development and testing⁵ in response to market and technological changes.
  
  
• The Indefatigable Innovator. This is a person who always finds new ways to approach a problem. I can say, immodestly, that I have played this role on many project teams. A long time ago, I was asked to switch assignments on very short notice, from a fun project to a much less fun project. In trying to make the best of a bad situation, I decided to design and build a new test server software configuration definition and tracking tool, based on an XML database. When I discussed this idea with another member of this unhappy project's unhappy team, she shot me a sad look and warned, "Don't try anything creative." She felt so confined by the current (troubled) state of the project that she wasn't able to see the advantages of trying new approaches to old problems. Being new to the project, and also being determined to learn a useful new skill, I had no such problem with trying something different. Although I was never able to achieve my ultimate goal -- to make a tool that would automatically build complex test server configurations -- I was able to learn a good deal about XML DTD⁶ design and define a (largely manual) approach for quantifying complex server configurations.
  
  
• The Visionary. Whereas the Indefatigable Innovator always sees new ways to approach tactical problems, the Visionary can see solutions to higher-level, strategic problems. Teams need someone who sees "the big picture"-- someone who has a broad vision of how software testing in general, as well as testing for your specificic application, can be done. It's easy to fall into the trap of trying to apply existing solutions to new problems. Sometimes this is the most practical approach to follow, but at other times, you need to "think anew"⁷ and imagine new solutions. Technological changes are happening at an ever-increasing pace; "in the past" might refer to yesterday, and it's all too easy to fall behind by standing still.⁸ Every team needs someone with the vision to keep it looking and moving forward.

Have you noticed what characteristics these folks have in common? They're hungry. I'm not referring to their dietary habits (although, as a rule, testers do tend to devour any free food that's brought into their office), but rather to how they approach the practice of software testing. They are inquisitive, proactive, adaptive, and not afraid to try new things or ask "Why?" They are always growing technically and learning. They never fall behind by simply standing still.

Types that need redirection
Before we move on, let's also take a look at some types of folks who can actually impede work on a test team, and examine how to redirect their energies into more productive tasks.

• The Status Junkie. An important task for a software test engineer is to accurately track progress during testing, which includes reporting all defects, and test coverage for specific hardware and software configurations. This progress and defect information, commonly referred to as "metrics," is vital input for decisions regarding whether software under test should advance from alpha test to beta test, from beta test to release to the field, and so forth. It's important to remember that this status-related information is always a means to an end, where the end is the development, testing, and release of software products to generate revenue. Collecting and presenting status information should not become an end in itself. If you see a software test engineer spending more time and effort on status-related tasks than on actually designing, writing, executing, and debugging software tests, then you may have a "Status Junkie"⁹ on your hands. What other symptoms might indicate a Status Junkie? First, these people will put a great deal of effort into telling you in great detail why they can't accomplish a task -- instead of trying new ways to work around whatever problems they are having. Second, they spend a lot of time fussing with the form they use for status reporting. I've seen status reports that included pages of text and a dizzying array of multi-colored charts and diagrams but contained little useful information. A few years ago, during an especially difficult project, a co-worker of mine joked that "We should just ship our status reports; they're much prettier than our product!"
  
  How do you reform a status junkie? On the positive side of the equation, you're dealing with a person who likely has a good work ethic and a high energy level, and who also understands the importance of keeping track of defects, test progress, and other metrics. Your task is to focus all that energy in the correct direction and get him or her to keep the status reporting aspects of software testing in perspective. Start with an "intervention" and explain that status reporting is important, but not at the expense of actually building and testing the product you actually have to ship. Then, limit the scale of his or her status reporting. I've had good results when I tell folks to imagine that when they're writing a status report, they're actually writing a newspaper story. That makes them concentrate on communicating only important issues, and in a concise form. Finally, make sure they understand that the "rules" that you're imposing on them are flexible. The amount and detail of their reporting should vary in response to the situation. If their testing is going more or less according to plan, then a brief report is sufficient. If they are tracking down a severe problem late in the development cycle, then daily (or even more frequent) reports may be necessary.
  
  
• The Sage. While your team is discussing how to respond to a new technical challenge, someone may announce that, "I've seen this exact situation before. I already solved the same problem on the Aardvark project a long time ago. Let me tell you what we did." He then proceeds to regale everyone with a long description of his past successes. The problem is that he can only contribute a verbal description of the solution, perhaps augmented by diagrams he draws on a whiteboard. He can't actually deliver a set of working programs or detailed documents. When you ask for something concrete, he just waxes poetic and returns to the whiteboard to draw yet another diagram. In addition, the approach from his past project may not even apply if he doesn't really understand the current problem.
  
  How do you extract concrete results from such a person? Despite his seemingly unhelpful talk and diagram drawing, the Sage's experiences -- at least some of them -- may actually be a useful resource for your team. What you have to do is address each of his problems separately. First, if he does not really understand your current tasks but is trying to force-fit an old solution to them, stop him dead in his tracks and work through his discussion, point by point. Make him explain his understanding of the current project and play the questioning student to his teacher. Odds are, you'll soon convince him that he has some work to do in order to understand the current situation. Second, propose that, in the absence of a working code model from his past project, he quickly build a new prototype for your project, based on his past work. In other words, make him "walk the talk." At worst, this will force him to learn about the technical aspects of solving the problem at hand; at best, he will build a prototype that your team can use.
  
  
• The Spread-Too-Thinner. As we have noted, a challenge of working in software engineering is the need to respond to constant change. Sometimes the reasons for change originate outside of your organization -- if a new technology comes on the market, for example. Other times, changes originate inside your organization. For example, you may need to make design changes or change the planned test sequence as the software evolves. Dealing with these changes often involves multiple sets of tasks. The testers must be able to prioritize their tasks and then manage their time to complete those tasks. When something obstructs progress on their highest-priority task, they should seamlessly move on to the task with the next highest priority, then return to first one later. Some test engineers are responsible for a large number of tasks but are able to make progress on only a small number of them at any given time. Which brings us to the "Spread-Too-Thinner." This person is honest, hardworking, and wants nothing more than to help the team progress toward its goals. When his work is blocked, he constantly volunteers for additional tasks. Eventually, he takes on so many tasks, and is so busy trying to do everything, that he is unable to actually complete anything or be accountable for his high-priority work. He is simply spread too thin. A variation on this theme is The Artful Dodger. The Dodger differs from the Spread-Too-Thinner in one important way: He intentionally volunteers for more tasks than he can possibly perform. Why do this? So that he can hide behind one task to avoid working on another: the one to which he is currently assigned. If you ask him about his progress on task "A," he'll respond that he was too busy to do that task. He was working on task "B."

How do you convert a Spread-Too-Thinner or an Artful Dodger into a productive team member? The solution is more or less the same: Get him to focus. Impress upon him the need to define a manageable set of tasks and goals, and then direct his engeries toward completing them in an organized manner. At first, you may have to do what bartenders do to customers who have had too many drinks: "Cut him off" and reduce his list of assignments. Then work with him on a regular basis to ensure that he's actually making progress on the remaining assignments. If you're dealing with an Artful Dodger, then you may have to apply more drastic measures and reduce his list to one task at a time. This will force him to focus and make it impossible for him to dodge.

Let's return to the characteristics of a good test team.

The team maintains continuity
It can be difficult to build a team that's successful for a short time. You must do hiring, training, and then retraining as technologies change, and planning and then replanning as project goals and requirements change. It's even more difficult to build a team that is repeatedly successful over time. Team members mature, change jobs within the team, or even leave, taking their skills, experiences, and memories with them.

How does a team continue to be successful over time? It maintains continuity in the face of all these changes. Another sports analogy is useful here. In American baseball, the most famous team is the New York Yankees. Why? Because they've won so many championships over the past eighty years that people refer to them as a "dynasty." What are their ingredients for success? Owners that are dedicated to winning, excellent managers, and a personnel system that has consistently found and developed succeeding generations of star players.

These same qualities can help software development and testing teams succeed, as we will see below.

• Staffing. We've already discussed how the best way to build a successful team isn't simply to assemble a group of "stars," but rather to build a team of people with different, complementary skills, experiences, and outlooks. It's also important to ensure that your team is always preparing for the future. To continue our sports analogy, one key to the Yankees' dominance in the middle of the twentieth century was the way in which the careers of the team's lead players overlapped. By the time the team's leading player was in decline, the coach was already grooming the next great player. In the early 1920s, the team was led by Babe Ruth. In the latter part of that decade, the early years of Lou Gehrig's career overlapped with the last years of Ruth's career. In the mid-1930s, an ailing Lou Gehrig was augmented by a young Joe DiMaggio. And, in the early 1950's, Mickey Mantle served as an understudy in center field for the last year of Joe DiMaggio's career. The overlapping nature of these great players' caeers helped ensure that the team remained successful, even as other players came and went. Your software test team should plan for its future in the same way. Your team's managers and senior members should always be grooming its next (overlapping) generation of leading players. This will ensure that your team can maintain its success even when it has to deal with short-term outages among the team's leading members (e.g., for illness or the birth of a child), or even with those people's resignations. This overlapping approach also ensures that team members are cross-trained, and that no one person is a "single point of failure." In countless instances I've seen over the years, it was impossible to perform a specific task on a given day because, "Only Harry knows how to do that, and he's out sick today."
  
  
• Institutional Memory. Maintaining continuity for a team extends beyond staffing to hardware, software, and documentation resources on which the staff depends to do its job. Building test networks, developing test tools, and writing the documentation to explain their uses are important tasks. An often overlooked and less glamorous one is maintaining those resources, especially the documentation. However, investing the time and effort to do this enables the team to maintain continuity; that documentation represents the team's institutional memory. People come and go, but having a library of up-to-date documents that accurately describe the team's role, tasks, and tools will help succeeding "generations" of team members maintain the team's success.

We'll continue discussing the importance of effective management and managers in the next section. (How to establish dedicated ownership for your company is a topic beyond the scope of this article.)

The team has both managers and management
I've chosen this section's title carefully; although the performance of a team's manager is crucial to the team's success, the manner in which all team members, including the manager, practice management is even more important.

A manager must perform several tasks in order for a team to be successful. He must manage budgets, arrange for training, and oversee setup of the physical work environment (e.g., hardware). But he has even more important responsibilities.

• Enable self-management within the team framework. Some people approach management in a rigid, top-down manner. They maintain centralized control of all decision making and direct all team members' day-to-day actions. Although in some instances this type of control is necessary (e.g., if the team is composed of extremely junior, inexperienced engineers), it is a very inefficient model. If no one can act without express permission from the manager, the team can become paralyzed when the manager is unavailable or when it is waiting for explicit directions from the manager. The expression "speed kills" applies in reverse to situations in which team members have to react quickly to problems but don't because they are conditioned to wait for instructions from their manager. If you watch basketball, you have probably seen this scenario: A player fights for a rebound, gains possession of the ball, and then, instead of quickly passing or advancing with the ball and catching the opposing team off-guard, he stops and looks to the team manager for an indication of what to do next. In the time it takes for him to do this, the opposing team sometimes has time to recover, and the opportunity to gain advantage is lost. As a manager, you can provide a well-defined framework of guidelines and principles for your team to avoid a similar sort of paralysis. Encourage members to be self-managing so that they can respond quickly at decision points. This involves writing effective software test plans. Some people write plans that implicitly assume things will go right. As we well know, however, when you're testing software, things often go wrong. As a manager, you must ensure that all test plans have contingency plans for dealing with problems. This means having flexible plans that allow team members to make use of individual initiative. The saying, "Plan is also a verb" holds very true for software testing.
  
  
• Provide the "unfinished agenda." One of the most important managerial tasks is to set a direction for the team's work. My favorite quote on this subject is from the former US President, John F. Kennedy. In his 1960 election campaign, he said the president's responsibility was to "set before the people the unfinished agenda" for the nation. For a software test team, the unfinished agenda can comprise several courses of action, such as building test automation or learning new technologies or platforms (such as J2EE). The key here is for the manager to provide both short-term tactical leadership for the team's current projects and tasks and a long-term strategic vision for the team's future.

Back to top

In conclusion

We started this discussion by asking, "Why is it that some software test teams succeed while others fail?" In looking beyond simple answers and well-worn clichés, we discussed how teams define their role and establish intra-team and inter-team relationships, the types of individuals that managers should want on their team, and the role that management, as practiced by the project manager and team members, plays in team success.

All in all, there is no simple answer to what makes a team successful. Just as for individuals, it's a combination of environment, characteristics, and leadership. Oh, and, it also takes hard work!

Back to top

Notes

¹ The experience of trying to build and run a team in a project environment without any defined "rules" can be akin to what happens to the main character in Franz Kafka's The Trial; he is accused of a crime but cannot find out what the crime is.

²http://www-136.ibm.com/developerworks/rational/products/rup/

³ "Orchestrating Integration Tests," Software Testing and Quality Engineering, (http://stqe.net), July 2003.

⁴ "Testing Java Servlets," Dr. Dobbs Journal, August 2004.

⁵ For a great description of conceptual integrity with regard to software design, take a look at Fredrick Brooks' classic book, The Mythical Man-Month.

⁶ Document Type Definition. A good tutorial for XML and DTDs is available here: http://www-106.ibm.com/developerworks/xml/library/x-dtdint/.

⁷ I've always liked this quote from Abraham Lincoln: "The dogmas of the quiet past are inadequate to the stormy present. The occasion is piled high with difficulty, and we must rise with the occasion. As our case is new, so we must think anew and act anew."

⁸ "Four Lessons for Software Testers and Their Managers," The Rational Edge (http://www-106.ibm.com/developerworks/rational/library/4865.html)

⁹ A co-worker of mine, who is fond of alliteration, says that people addicted to collecting and reporting metrics are "Maniacal Misanthrophic Metrics-aholics." Now, that's a real mouthfull!

About the author

Len DiMaggio is a software quality engineer and manager at IBM Rational. Prior to joining Rational, he led software testing teams at BBN, GTE, and Genuity. Len has published articles on software testing in STQE, Software Development, Dr. Dobbs Journal, and the QAI Journal.

Rate this article

Comments

Back to top

Table of contents

• Characteristics of good test teams
• In conclusion
• Notes
• About the author
• Comments