This is the eighth installment of a multiple-part article (as outlined below) that demonstrates using Rational's tools in a distributed, J2EE-based project.
- Part 1: project introduction; high-level planning
- Part 2: detailed planning; risk management; requirements management
- Part 3: Rational Rose model creation and access control; requirements analysis
- Part 4: use-case refinement; report generation; tool and technology selection
- Part 5: architecture and design
- Part 6: detailed design; early development; round-trip engineering; early unit tests
- Part 7: continued development; early builds; demonstrations
- Part 8: strategies for unit testing; function tests; GUI test scripts
- Part 9: system builds and testing; defect tracking; product acceptance
- Part 10: project results; conclusions; future work
The fictional premise of the article is that we.re a software company, Lookoff Technologies Incorporated, and our customer, Audiophile Speaker Design, Inc. (ASDI), has hired us to meet their burgeoning IT requirements. For a more detailed introduction to the project, see Part 1.
This Part 8 installment embellishes on the subject of testing, originally introduced in Part 6. Part 6 reviewed how, during early development, we instituted the use of Rational Purify and Rational Quantify to inspect memory use and performance bottlenecks. It also discussed many of the details of our early unit-testing efforts. This installment describes how those efforts progressed and reviews our use of Rational.s testing tools, primarily Rational PureCoverage and Rational Robot, to reduce the cost of testing through automation. At this stage we focused on function tests (including GUI tests), although we also carried out some early load tests.
Note that the Rational Unified Process (RUP) terms used here reflect two different dimensions of testing: "unit test" refers to the stage of development of the software being tested -- in this case referring to the smallest testable software units -- whereas "function test" and "load test" refer to the specific test objective, regardless of the development stage of the tested software. Much of what this article describes in the context of unit testing also applies to the testing we did in later stages of development (for example, during integration tests combining different components or subsystems).
As development proceeded, we found that our unit-testing effort was outweighing our development effort because of the complex user interactions that were possible with the application we were building. Unit testing always requires a significant amount of effort, and perhaps we didn.t budget enough time for it. We found that our GUI tests, which involved a tedious manual process, were especially slowing us down.
Although Phase 1 of the ASDI project was only meant to be a proof of concept, we were having a hard time tolerating visible bugs in the system. It was hard to find a good balance between having solid, credible software and pulling together frequent, quick demonstrations. We.d defined the scope of Phase 1 such that it had to be more than a technology demonstration; the customer expected it to be a usable beta version.
Here we.ll look at the scope of our unit-testing effort, the automation capabilities we chose to use (or not), and our use of Rational PureCoverage to determine the thoroughness of our testing.
The scope of unit testing -- what, how much, and when to test -- is a function of several variables, including:
- the complexity of the software
- the product.s features and interfaces
- how critical particular software is (or is not) to the overall product
- the team's tolerance for defects
In terms of how much of the code to test, on previous projects we hadn.t aimed for 100% code coverage in our software. Full coverage was very expensive and difficult to achieve, because we had to hand-inspect our code to determine which lines were and were not hit by the test. Small changes to the unit tests would require us to audit the code again to ensure that full coverage had been maintained. However, thanks to Rational PureCoverage, on the ASDI project we were easily able to come close to 100% coverage (with a small amount remaining untested due to project budget and scope). PureCoverage greatly simplified the task of inspecting code coverage, making it easy to identify which parts of the code had or had not tested.
As mentioned in Part 6, our unit-testing effort started almost as early as core development itself. Most of our developers started writing unit tests as soon they had a software unit that could conceivably be tested. They liked to test the internals while the code was fresh in their minds.
Execution of unit tests always preceded builds; it was unacceptable to introduce into a build simple defects that could have been caught through unit tests. Unit tests were always performed (and results logged) before code was distributed for review. Furthermore, developers ran their unit tests on a regular basis to make sure their periodic changes weren.t breaking the software. We weren.t as radical in our approach as some disciplines -- such as Extreme Programming (XP), in which unit test development often precedes code development -- but we saw unit testing as an important early step.
There.s also, of course, the issue of how to do the testing, especially the extent to which tests should be automated. Developers who were writing "behind-the-scenes" (non-GUI) code were lucky in that they could typically write automated tests consisting of Java drivers, stubs, and scripts. However, as mentioned earlier, the GUI developers had it much harder.
For our non-GUI tests, we observed the convention of having unit tests associated with each class; we wrote driver code to flex the class and report success or failure. We experimented with using testing frameworks that would automatically generate, manage, and run the tests for us, such as those touted by XP, but found mixed results. Although the philosophies were sound, the frameworks (including, for example, JUnit) weren.t mature enough for our purposes. Nevertheless, our developers were confident that if they had the time to become familiar with and tailor these utilities, they.d find testing frameworks to be invaluable for our unit tests. We.ll probably work more with these frameworks in future projects. On the ASDI project, adopting full-blown XP principles would probably have been too risky; instead, we eased toward some of its concepts in a controlled and risk-reduced manner. You can find a number of ideas related to this on the XProgramming.com site.
We didn.t use Rational.s testing tools to test our non-GUI code because we felt they didn.t go down to a low enough level. Where these tools really shone was for automated testing of our GUI-driven code. User interface tests are very manual and interactive and require a lot of content validation; Rational.s testing tools made it much easier for us to repeat these tests in a consistent and rapid fashion.
By using Rational PureCoverage, we were able to greatly reduce the amount of time we spent determining code coverage -- that is, which methods and lines were executed ("hit") or missed -- during execution of our unit tests. This tool enabled us to monitor the coverage provided by our tests and then decide whether to improve the tests or test the code through manual inspection instead.
PureCoverage 2001A works with Java code in addition to C or C++ applications, and is configured and run in the same way as Quantify and Purify (discussed in Part 6). This tool enabled us to look at our unit tests. code coverage in two ways: with the Coverage Browser or the file-specific Coverage Fileview.
With the Coverage Browser (Figure 1), we could see statistics summarizing the code coverage for each of the files (and directories) involved in the unit tests. These statistics included how many calls were made on all the method(s) therein, how many of the methods were hit or missed, and how many lines of code (excluding "dead," or nonexecutable, lines) were hit or missed.
|
| Figure 1: Coverage Browser |
| (click here to enlarge) |
We could then double-click a method in this summary to drill down into the Coverage Fileview, showing just which lines were hit or missed in that method. As shown in the example in Figure 2, missed lines are highlighted in red (and hit lines in blue).
|
| Figure 2: Coverage Fileview |
| (click here to enlarge) |
Function tests verify that the software works properly by testing against requirements defined in use cases. To ensure we.d test all requirements thoroughly, we used Rational Robot and Rational TestManager in designing and organizing our function tests. (A good discussion of functional testing strategies using Robot can be found in the Robot user.s guide.) Robot enables test scripts to easily be recorded and then played back later for automated testing. Two types of scripts can be created with Robot:
- GUI scripts record your user-interface actions and, when played back, launch and control windows as if a user were manipulating the application. Some client-side testing -- for instance, of an ActiveX control embedded within our Web application -- requires using these scripts. Because GUI tests posed special problems for us, this type of function test is discussed in detail in the next section.
- VU scripts monitor your application while you use it and track what network information is sent and received at the packet level. When played back, these scripts rerun the tests without having to launch the application, making them faster and lighter than GUI scripts. We used VU scripts for non-GUI function tests and for load tests of the B2B interface enabled by the command gateway.
We started creating our function test suites as early as possible, since these tests were an invaluable tool for the engineering team. Our goal of having the test scripts in place before the code was distributed for review was sometimes difficult to achieve, however, because some portions of code and some components were taking longer to complete than others, or because the integration and test (I&T) team was too busy to finish the test scripts in time. In one case, the test scripts were finished while the user interface was still changing; although some rework was necessary, the changes turned out to be fairly easy to integrate.
Once we had a strong set of function tests, it was easy to retest the system. This paid off in later iterations when we needed to retest small changes and make sure the changes didn.t break other pieces of code. We mapped our function test scripts to use cases so that we.d immediately know which tests to run when requirements associated with a given use case were modified.
While the developers of non-GUI code were being very productive with their unit-testing effort, the JSP/servlet developers were having a hard time keeping on schedule. What was hurting them the most was the process of testing GUI fields, which involved repeatedly carrying out the following steps:
- Set up test data in the database.
- Log in to the application through the Web browser.
- Navigate to the screen to be tested (a form with fields to be filled).
- Fill in one or more fields on the screen, depending on the specific test.
- Click the appropriate button to submit the form.
- Inspect the results.
- Return to step 4 with a different test (different field values or fields), until all reasonable combinations have been tested.
Some of the GUI tests were especially cumbersome, involving entry of the same, lengthy value into several fields. Using cut-and-paste helped, but it was still a chore for the team to repeat these tests over and over again. Some screens had 50 different tests that had to be performed on them, each consisting of many steps.
Although we hadn.t originally intended to use Rational.s testing tools, we felt that our difficulty with GUI testing was a good justification for delving into them. We decided to use Rational Robot to automate most of our GUI unit tests. We coordinated these tests in the following manner:
- The developers drafted a unit test specification for each test.
- The junior I&T team member took the unit test specification and created scripts based on the current build of the software.
- The unit test scripts were configuration-managed so that changes to them could be tracked.
- Periodically, a developer would give the I&T team the task of executing the unit test scripts and providing a report indicating success or failure against the unit test specification.
This approach worked wonderfully. Not only did it give the I&T team more work during the early stages of the development process, when they didn.t have much to do otherwise, but it also gave them an opportunity to ramp up on the software through exposure to the unit test specifications. Furthermore, our experience using Robot for GUI tests prepared us for other function test scripts that would be required.
When Rational Robot starts up, it requires the selection of an existing project (created using the Rational Administrator) with which the test scripts will be associated. In our case, the project location had to be accessible on the network so that we could share the project database with the entire team. We created the ASDI project in the Administrator tool as shown in Figure 3, and we selected this project when prompted for one by Robot.
|
| Figure 3: Creating a project with the Rational Administrator |
As a very simple example of a GUI test, we needed to confirm that the client-side validation was behaving correctly for our partSearch.jsp screen, which executed JavaScript code to validate its fields. Part numbers entered into this screen, for instance, had to be integers greater than 0, per ASDI.s requirements; if a user entered an invalid part number, the JavaScript code was supposed to report the error and not send the bad value to the server. Our JavaScript code followed the approach presented in Netscape.s sample code for form validation, although we wrote our own validation code in place of their comprehensive FormChek.js file. With this code in place, the entry of a negative part number brought up the window shown in Figure 4.
|
| Figure 4: Proper partSearch.jsp screen validation |
We created a script in Robot to test whether the part number field validation was functioning properly. First Robot prompted us to supply a name for the script (Figure 5).
|
| Figure 5: Recording a new GUI script |
After clicking OK, we were provided with a recording control
(Figure 6) and could then use our window-based application while Robot recorded
our actions. In this example, we started Internet Explorer 5, navigated to the partSearch.jsp page, entered a bad part number (-123456), and clicked Submit on the page.
|
| Figure 6: Robot recording control |
Robot recorded our actions in a GUI script (Figure 7). This is an oversimplified example; our actual tests involved much more comprehensive test scripts. We didn.t create hundreds of individual scripts, but instead tried to design one large script per screen. The documentation on Robot goes into test design in great detail, and we borrowed heavily from its recommendations.
|
| Figure 7: Sample Robot GUI script |
| (click here to enlarge) |
With a human-readable and easily editable script in hand, we didn.t have to re-record the entire script in response to a few small changes in the test. Sometimes we directly edited the script without rerunning the test, or we re-recorded a small portion of a test and pasted it into the existing script.
To run the test again, we simply clicked the "play" button on the Robot recording control (or chose Playback from the File menu). The script reran the test just as we.d originally performed it and, when finished, started up Rational TestManager to summarize our results. For the simple example above, TestManager displayed the report shown in Figure 8 after the script executed successfully.
|
| Figure 8: TestManager reporting Robot test success |
| (click here to enlarge) |
Suppose we.d broken the code such that the bad part number was no longer being caught -- that is, instead of displaying a window indicating the error, our application was posting the data to the servlet to perform the search. When we reran the GUI test by playing back the script, Robot would catch this problem, and the resulting TestManager report (Figure 9) would show that the test had failed.
|
| Figure 9: TestManager reporting Robot test failure |
| (click here to enlarge) |
When we started the ASDI project, we.d expected to use Rational.s analysis and design tools but not their testing tools to any great extent. By this point in Phase 1, we were surprised and pleased by how much we could automate and improve our testing process using Rational.s testing tools. These tools had a substantial learning curve, but once we sorted out the specifics of each tool, our team.s productivity significantly increased. In some cases, individual developers used tools such as Rational Purify, Quantify, and PureCoverage to supplement their unit-testing effort. Other tools, such as Rational Robot, required the greater expertise and attention that our I&T team could give to them.
The team still had to undertake integration and testing at the system level. Most of the unit-testing effort was wrapping up, but we needed to test the subsystems and overall system more thoroughly and formally. This meant formal builds, final build documentation, and significant focus on I&T. Load testing in particular had played only a small part in our testing efforts so far but would figure prominently in our testing of the full system. Fortunately, our test suites were 95% complete, and the software was slightly ahead of schedule, so we could focus extra effort on system testing to ensure a top-notch Phase 1 deliverable.
Our risk list was fairly short at this point. The customer was so tuned in to the evolving system that there weren.t many surprises, and our technical risks were few, thanks to the excellent progress made by the engineering team.
We now had to finish up code development, perform system tests, address any defects that were identified, and deliver Phase 1 to the customer on schedule. As a team nears completion of a major phase, it.s often difficult to wrap up all the details in a timely fashion. If we were to avoid having schedule overruns, we.d need to have only a small number of defects and quickly rectify any problems uncovered by tests.
Steven Franklin has an extensive background in software design, architecture, and engineering process, which he usually applies to large, distributed information management and command and control systems. He's been using Rational tools since 1997, and his primary areas of interest include XML, J2EE, wireless, and software engineering methodologies. Steven can be reached via e-mail.
Comments (Undergoing maintenance)
Table of contents
Next steps from IBM
Rational Application Developer features provide the comprehensive Integrated Development Environment (IDE) used to develop J2EE applications requiring Servlets and JavaServer Pages (JSPs).
- Try: Try the free Rational Application Developer, which helps developers quickly design, develop, test, analyze, and deploy high-quality Java, Javaserver pages, and J2EE applications.
- Article: Learn how you can deliver accurate, higher-quality J2EE apps with Rational Application Developer, open source JUnit and JUnitEE test frameworks.
- Tutorial: Build a Facebook application using PHP, the Spring framework, and J2EE.
- Buy: Rational Application Developer
My developerWorks community
Tags
Use the slider bar to see more or fewer tags.
Popular tags shows the top tags for this particular content zone (for example, Java technology, Linux, WebSphere).
My tags shows your tags for this particular content zone (for example, Java technology, Linux, WebSphere).
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
