 | Level: Introductory Liz Augustine, Technical Writer, IBM
Sam Guckenheimer, Senior Director of Technology, IBM
15 Feb 2002 from The Rational Edge:Focusing on subjects ranging from software testing goals and the value of Rational testing tools to how XP (eXtreme Programming) addresses testing, this interview concludes with a look at trends that may affect future testing products.
I recently caught up with Sam
Guckenheimer, Senior Director of Technology for Automated
Test at Rational. We talked about subjects ranging from the goals
of software testing, to how XP (eXtreme Programming) addresses testing,
to the value of Rational's offerings for the testing community.
We even spoke briefly about trends we might see in future Rational
offerings.
Liz:
Sam, let's start
by talking about the goal of testing. The answer to the question
is probably obvious to you, but why is testing important?
Sam:
You test so
that you can assess the software under development for the benefit
of the project team. James Bach uses the analogy that the testing
team acts as the headlights for the project. The results of your
testing can help you determine what to do next--you need to base
that decision on the real data testing provides. Bach also introduced
the concept of a "Testing Mission" as the charter of the test team.
Of course, the mission changes from one iteration to the next over
the course of a project.
Unfortunately, in many organizations, testing activity often does
not have this focus and can devolve into checking off tasks on a
test plan, or filling in color codes on a test report. But if it's
done right, testing can tell you about the state of your project.
In the Rational Unified Process®(RUP®)
version 2002, we emphasize the importance of being clear about the
testing mission. It's worth quoting RUP here:
Here are some ideas for mission statements you might adopt for
a given iteration:
- find as many bugs as possible
- find important problems fast
- assess perceived quality risks
- advise about perceived project risks
- advise about perceived quality
- certify to a standard
- verify a specification (requirements, design or product claims)
- satisfy stakeholders
- fulfill process mandates
Thinking about this list, it should occur to you that many missions
are mutually exclusive. For example, the best approach to the "find
important problems fast" mission will likely conflict with the "verify
a specification" mission. To successfully achieve one mission often
negates other possible missions, and requires a different supporting
test approach.
Liz:
You've talked
about establishing a testing mission for each iteration. Could you
talk about why you advocate iterative development and testing, as
compared to using waterfall development?
Sam:
In the waterfall
approach, you do all your analysis, then all your coding, then all
your testing, then release. One problem with this approach is that
it's hard to understand your progress, because you often end up
working on the riskiest parts of the project at the very end. These
risky areas include integration testing, determining whether your
architecture is sound, figuring out whether you've met customer
requirements, and so on. If something goes wrong in any of these
areas and you're near the end of your schedule, it's quite likely
that you'll deliver your product late.
In contrast, by using an iterative approach, you can address known
risks and uncover unidentified risks early. (In fact, one guideline
is to work on the riskiest parts of the project first.) But if you
use subjective methods to assess progress, you may not notice when
things are going wrong. To solve this problem, you can establish
iteration exit criteria and then test against these criteria to
assess your readiness to move to the next iteration.
Liz:
What sort of
testing strategy does XP offer to a software development organization?
Sam:
XP encourages
test-first design, with a focus on unit testing. You're expected
to write unit tests to verify your code before you actually
write the code. So you work in microcycles: Write a test, write
the code, run the test, write the next test, and so on. As a result,
you don't write code that you don't know how to test. Also, this
style of coding forces you to confront--and resolve--ambiguities
in the requirements.
One place where XP falls short is in independent testing. It speaks
only of unit testing and acceptance testing. For acceptance testing,
XP expects testers to work with customers to create tests that validate
that the "user stories" (equivalent to what RUP calls use cases)
have been implemented correctly. Also, XP recommends that these
tests be automated. Using this approach, you create an automated
suite of acceptance tests that cover "happy-day" scenarios and a
few obvious exception scenarios.
But as you know, there are not as many "happy days" in real life
as we'd like. That's why good testers spend eighty percent of their
effort thinking about what can go wrong. Good test strategies (automated
or manual) focus on seeking out potential problems, many of which
are due to faults of omission--failure to foresee certain issues
in design. Good testing involves more than counting on customer
expectations of desirable outcomes; it requires an iterative approach
involving exploration, discovery, error-seeking techniques, and
process, too. XP is silent on these issues. There's a very active
Yahoo Groups discussion forum on agile testing, where testers confront
the "XP missionaries," asking about coverage, test data, scenarios,
fault conditions, negative tests, and so on. So far, the discussion
hasn't led to a resolution.
Another intriguing aspect of XP is that developers work in pairs,
in effect creating a continual inspection process. Inspired by this
practice, James Bach has led some experiments in paired testing.
I've participated in those and have become an enthusiastic advocate
of paired testing as a result. It's worth noting, however, that
this is paired exploratory testing, which is inherently a manual
technique. So it's actually contrary to the official XP practice
of automating all tests.
To be clear, the XP and Agile Methods community has a lot to teach
us, and we worked some of their lessons into RUP in Version 2002.
For example, Brian Marick (one of the Agile Alliance members) contributed
a concept page on Test-First Design that considers the question
"Can all test design be done at component design time?" Marick's
answer, by the way, is negative.
Liz:
So you're implying
that test-first design is helpful but not sufficient.
Sam:
Yes--test-first
design is primarily a design strategy, and only secondarily
a testing strategy. Kent Beck writes about this in the September/October
2001 issue of IEEE Software.
As I just mentioned, there's a lot more to testing. Brian Marick
gives this rule of thumb: If the purpose of your test is to improve
the quality of your own code, then you as the developer (or the
developer and the tester together) should be performing it. If the
purpose of the test is to uncover problems that span multiple peoples'
work, then independent testers should perform it.
Liz:
Can you give
an example of how the developer should do testing?
Sam:
Sure. A developer
works in microcycles of edit, compile, debug, and test. In XP, this
cycle becomes test, edit, compile, retest, where debug and retest
become the same activity. For example, Martin Fowler says that if
you're thinking of putting in an assertion for debug, make it a
test instead. Many of these microcycles lead to a checkin, and eventually
to a delivery. With each checkin, you run all your tests. So over
time, you can build up a library of tests that you reuse regularly.
And all your tests should pass each time you make a delivery.
Liz:
So eventually,
all this test code makes its way to the testers.
Sam:
Ideally, yes.
This requires a test management process (and often a tool such as
Rational's TestManager®) that allows release engineers and testers
to reuse the unit test suites of developers. Whenever release engineers
make a build--nightly or after a certain number of deliveries
--either they or the tester runs smoketests, all of which must
run green to trigger a testing cycle.
Think of these microcycles and larger cycles as different-sized
gears, all working at the same time to run a larger machine.
Liz:
We've talked
about developers and testers--do analysts ever get involved with
testing efforts?
Sam:
Yes, sometimes
they do, and sometimes their work can make direct contributions
to the testing effort. Ivar Jacobson and I gave a talk at RUC 2001
called "Escaping Software Hell," in which we described a technique
called activity-driven verification. The main idea is that for each
activity in a process, there should be a step that validates the
completion of that activity by applying verification criteria. Think
about extending the test-first design strategy we discussed earlier.
Now think about writing a use case. As you write each flow in the
use case, you also write a step that verifies it. This technique
applies to requirements, too. While you write requirements, you
should keep in mind how they'll be tested.
Liz:
Is it possible
to automate this kind of testing?
Sam:
Yes; one of
the most underused artifacts is the visual model. Everyone wants
to do model-based testing, but most people don't know how to do
it. Rational is playing a leadership role in this area. For example,
UML 2.0 will provide semantics so that you can use sequence and
activity diagrams as test inputs. We're actively developing that
profile with a consortium of about ten other companies, which includes
customers and competitors. Think of the test definition as being
like a sheet of acetate on top of the model--so that the test
definition annotates the model. This allows you to move test activities
to a point earlier in the lifecycle, from today's test scripts written
in a 3GL language to tomorrow's full participation in the visual
modeling activity.
In fact, in five years, I think the vast majority of good testing
(which I define as tests with thorough coverage, clear exit criteria,
and the ability to pinpoint information quickly) will be done from
models, and it will all be UML-based.
Liz:
How would you
characterize the state of software development today? How reliable
is the software development process?
Sam:
We're at about
the same place that hardware development was twenty to thirty years
ago. Sometime in the 1980s, a few changes happened in hardware design:
- We saw the rise of a component marketplace. Designers started
building systems from commercial components rather than from scratch.
We saw this in PCs, in cars, and just about everywhere else.
- At the same time, the packaging of electronic components became
so complex that black box functional testing - stimulating the
component with test vectors applied purely from the outside--
no longer provided adequate coverage. Much of the circuitry simply
could not be exercised by applying these external test vectors.
So testing turned into a design activity, and design for testability
(DFT) became an essential part of design engineering. Nowadays,
no one would consider buying a computer-aided engineering system
for circuit design without carefully evaluating its DFT features.
Now, a few decades later, we're starting to see these same trends
in software. I believe that in a few years, we'll start developing
design-for-testability standards in the software world. We're already
seeing some evolutionary steps -- for example:
- Both J2EE and .NET expose interfaces to allow inspection. You
can get test access without needing to apply extra instrumentation
to the target.
- In UML, we're developing the mechanics to do test specification,
and we're developing the ability to create reports so that we
can mine value out of test specifications.
Liz:
So what is Rational
offering today that's changing how we test?
Sam:
Rational's doing
a lot. I've already covered RUP. Here are some more examples:
- With Rational Test RealTime, you can observe a running system
as a UML sequence diagram built on the fly, and you can analyze
it after the test run.
- With Rational Quality Architect (RQA), you can generate tests
from class diagrams and sequence diagrams. This allows you to
test integrations and interactions very early in the development
cycle.
- Rational PurifyPlus is an observation product that allows you
to inspect memory and determine code coverage of a target seamlessly.
- Rational Robot and TestManager allow you to capture and execute
tests at the GUI or API level for both functionality and load
tests.
- TestManager provides a full suite to support both internally
and externally developed tests, so that, for example, you can
run ready-made tests on a third- party component.
And all these tests can fit into the same test suite.
Liz:
Looking into
your crystal ball, what should we look forward to from Rational?
Sam:
Just recently,
we released Version 2002 of Rational Suite, and there are some great
improvements there.
- In the Rational Unified Process, we've redesigned the test discipline
so that it is the first well-instantiated, industrial-strength
test process that supports iterative development. (Other processes
typically assume that you're doing waterfall development.)
- Over the last year, we've made a heavy investment in helping
to accelerate testing earlier in the lifecycle. We've introduced
Rational Quality Architect, which helps you do unit and integration
testing with J2EE and COM. In Rational Test RealTime, we help
you do early testing of real-time embedded systems and of C++
systems.
Looking a little further out, the types of testing that people
do now--GUI, load testing, test reporting--won't go away, but
will be considered incomplete. We've already seen a lot of support
for developing complex systems in visual modeling tools, IDEs, and
so on, but support for testing has lagged behind. I think we'll
concentrate on how we can use UML to help with the testing process.
We'll also need to consider the complexities of design for testability
and runtime test access that have been successfully tackled in the
hardware world. For example in e-commerce, about ten percent of
code covers "happy day" scenarios, but ninety percent of the code
covers "rainy day" scenarios--fault conditions, errors, and so
on. So we'll also need to support testing at lower levels: interfaces,
for example.
References for Testing
Print
Kent Beck, "Aim, Fire [Test-First Coding]." IEEE Software, September/October
2001.
Cem Kaner, James Bach, Bret Pettichord, Lessons Learned in Software
Testing. John Wiley & Sons, 2001.
Web Sites
Agile Alliance Web site: www.agilealliance.org
Rational Unified Process Web page: http://www.rational.com/products/rup/index.jsp
XP (Extreme Programming) Web site: www.extremeprogramming.org
See http://www.testing.com/writings.html
for papers by Brian Marick and other testing authorities.
For a discussion group on agile testing, go to http://groups.yahoo.com/group/agile-testing/.
About the authors | | | Liz Augustine joined Rational in 1995 as a technical writer for Rational ClearCase®. While with that group, she rewrote the product tutorial, worked with engineers to design native Windows user interfaces, and served as documentation project leader for ClearCase on Windows. Since joining the Rational Suite team, she has written the Rational Suite® tutorial, a manual about using Unified Change Management® (UCM®) with Rational Suite, and material about upgrading from earlier versions of Rational Suite. In addition, she works with writers across the company to bring greater consistency to Rational documentation. After studying French at Swarthmore College, she received a B.S. degree in computer science from Boston University. |
| | | As Senior Director of Technology for IBM Rational's Automated Software Quality Group, Sam Guckenheimer is responsible for formulating Rational's product vision and strategy for software testing products as well as standards for testability. Before joining Rational in 1995, he held various marketing, engineering, and general management positions at both US and European software companies. A frequent speaker at testing conferences and participant in testing workshops, he has guest lectured at Florida Tech, MIT, Harvard, and Yale. He holds an A.B. (Phi Beta Kappa) from Harvard University. |
Rate this page
| |
