September 10, 2018 | Written by: Chip Davis
Categorized: Requirements Management
Share this post:
With the growing number of smart and connected products being developed, more and more systems engineers must now deal with the verification and validation of embedded systems. Of course, the smarter those products get, the more complex those testing activities become. At the same time, new regulations constantly come into effect to prevent faulty products reaching the consumer. This is adding even more burden on the testers that need to ensure the functional safety of those smart things. Fortunately, the right test management process can help!
What is test management?
Test management is the practice of organizing and controlling the process and artifacts required for the verification and validation efforts. Traditional tools used for test management include:
- Pen and paper
- Word processors
Larger testing efforts may use home-grown software test management solutions, usually built on spreadsheets or databases, or commercial test management applications such as IBM® Rational® Quality Manager.
The general goal of test management is to allow teams to plan, develop, execute, and assess all testing activities within the overall product development effort. This includes coordinating efforts of all those involved in the testing effort, tracking dependencies and relationships among test assets and, most importantly, defining, measuring, and tracking quality goals.
Test management challenges
One way to sum up the objectives of test management is answering the following questions:
- Why should I test?
- What should I test?
- Where do I test?
- When do I test?
- How do I conduct the tests?
While this may seem straightforward enough at a high level, there are many obstacles that frequently arise in typical system engineering. These challenges are described below.
Not enough time to test
Except for certain specialized or highly mission-critical applications, very few projects include sufficient time in the development lifecycle to achieve a high level of quality. Very often, the almost inevitable delays in a complex project get assigned to the already short ‘testing cycle.’ Even the best projects are very likely to have difficult time constraints on testing tasks. The effects of this obstacle on test management are constantly changing priorities and shifting tasks, as well as reduced data for test results and quality metrics.
Not enough resources to test
In addition to the shortages in time, there is quite often a difficulty getting the right resources available to perform required testing activities. Resources may be shared on other tasks or other projects. While hardware resources for testing can add delays and difficulties, a shortage of human resources can be even more difficult to resolve. The effects of this obstacle on test management are roughly the same as those for time shortages.
Testing teams are not always in one place
More often these days, testing resources might be available but not at the same geographic location. Leveraging talent around the globe to reduce costs has become commonplace, but this introduces considerable technical obstacles. How do teams on another continent share artifacts and stay coordinated without delays and discord affecting the overall project? How can a project maximize efficiency with geographically distributed development?
Difficulties with requirements
While there are many testing strategies, validating requirements is typically the primary, highest priority testing that needs to be completed. To do this requires complete, unambiguous, and testable requirements. Less-than-perfect requirements management can lead to more profound issues in the testing effort. Using a tool such as IBM Rational DOORS Next Generation can significantly help improve requirements management and facilitate the development of good requirements.
For test management to be effective, there must be seamless access to the latest changing system and business requirements. This access must be not only to the wording of the requirements, but also to the priority, status, and other attributes. In addition, this requires the utmost coordination and communication between the teams developing the requirements and the teams performing the testing. This communication must go in both directions to ensure quality.
Keeping in synch with development
Another team coordination that is required to allow for product quality is between testers and systems architects and software developers. Aside from critical defects, it is almost a tradition in system and software development that the testing team’s work is only the tester’s concern. However, it is essential for everyone, especially the developers, to understand both the current level of quality and what has and has not yet been tested.
For testing teams to use their precious time efficiently, they have to keep up with constant changes in code, builds, and environments. Test management must identify precisely what build to test, as well as the proper environments in which to test. Testing the wrong builds (or functions) results in wasted time and can severely impact the project schedule. Testers must also know what defects are already known, and should therefore not be re-tested, and which are expected to be fixed. Testers must then communicate the defects found, along with sufficient information to facilitate resolution, back to the developers.
Reporting the right information
A testing effort is only useful if it can convey testing status and some measures of quality for the project. Generating reports is simple enough, but presenting the right information (at the right time, to all the appropriate people) can be trickier than it seems for several reasons:
- If there is too little information, then the project stakeholders will not fully understand the issues affecting quality, in addition to the reduced perceived value of the testing team.
- If there is too much information, then the meaning and impact of key information becomes obscured.
- There are often technical hurdles that can impede how to share information to different roles in different locations.
Another consideration in reporting results is exactly how the information is arranged, and in what formats (that is, the information can be tool based, browser-based, or in documents). Project stakeholders’ understanding of the testing and quality information will be reduced if there are technical or other restrictions limiting the arrangement or format of reports. Data should be presented in a clear and logical design that conveys the appropriate meaning, not in a layout constrained by tools or technology. It is therefore essential for test management to consider the need for flexibility and capability in providing a wide range of reporting formats.
What are the quality metrics?
One of the primary goals of a testing team is to assess and determine quality, but how exactly do you measure quality? There are many means of doing this, and it varies for the type of system or application as well as the specifics of the development project. Any such quality metrics need to be clear and unambiguous to avoid being misinterpreted. More importantly, metrics must be feasible to capture and store, otherwise they might not be worth the cost or could be incomplete or inaccurate.
What’s up next
Stay tuned for the next blog in this series where we’ll explore recommendations for how to improve test management. In the meantime, you can discover more about IBM Rational Quality Manager and IBM Rational DOORS Next Generation on our website.