This is the ninth installment of a multiple-part article (as outlined below) that demonstrates using Rational's tools in a distributed, J2EE-based project.
- Part 1: project introduction; high-level planning
- Part 2: detailed planning; risk management; requirements management
- Part 3: Rational Rose model creation and access control; requirements analysis
- Part 4: use-case refinement; report generation; tool and technology selection
- Part 5: architecture and design
- Part 6: detailed design; early development; round-trip engineering; early unit tests
- Part 7: continued development; early builds; demonstrations
- Part 8: strategies for unit testing; function tests; GUI test scripts
- Part 9: system builds and testing; defect tracking; product acceptance
- Part 10: project results; conclusions; future work
The fictional premise of the article is that we're a software company, Lookoff Technologies Incorporated, and our customer, Audiophile Speaker Design, Inc. (ASDI), has hired us to meet their burgeoning IT requirements. For a more detailed introduction to the project, see Part 1.
At this point in the ASDI project, we were nearing completion of Phase 1. ASDI had been involved in a number of our demonstrations and was feeling quite comfortable and satisfied with the product. (In fact, we had some concern that Phase 1 may have been so usable that ASDI would delay or cancel any Phase 2 follow-on; this would be an engineering victory, but certainly not a financial one.) The final factor in the customer's comfort would be to have fully compliant system tests and acceptance tests.
Our lines-of-code output had been reduced dramatically by this point. We were in the fix-and-finesse stage, with small changes dominating the team's effort. As the integration and test (I&T) team discovered software defects, they were entered and prioritized in a Rational ClearQuest database based on the DefectTracking schema. These defect reports were reviewed by the engineering team. The team lead and project engineer typically decided the prioritization of defects and maintained a plan outlining which ones were to be fixed for a given build. The biggest battle at this point was to discourage developer "goldplating" and the last-minute tweaks that were always tempting to try to fit in.
The frequency of performing full, from-scratch system builds -- building the system from the ground up on a clean environment - increased sharply toward the end of Phase 1. Originally scheduled to be monthly, these builds were sometimes performed as often as once a week. This wouldn't have been feasible on a larger project or with a less talented team, given the overhead of setting up a clean build environment, retrieving software from the source repository, building the system, and testing its behavior. However, thanks to the tight integration of our tools, our well-documented build process, and our rapid test execution using Rational's testing tools, we were able to make this increased frequency work.
Scripting in particular made it practical to have such frequent builds. At the very least, we ran scripts to test all areas of the system that might have been affected by defects we'd addressed. We often went further than that, running scripts against most or all of the system. We were fortunate that we could test the bulk of the system through automated scripts; Rational's testing tools were very compatible with the technologies we were dealing with, whereas some other technology combinations would undoubtedly make such far-reaching use of test scripts quite challenging or even impossible.
By the time we got to the stage of doing system builds, the I&T team was fully ramped up and competently leading the process. Their readiness was due in large part to our having brought them into the project team full-time during the development cycle (as noted in Part 6). On previous projects, we'd brought in the I&T team later, near the end of the development cycle, but this had its problems. We eventually realized that the I&T team needs ramp-up time just like any other technical members of the project team. Although for our earlier builds the I&T team tended to be slower than the engineering team at assembling the system, this was expected as part of their learning process, and having them do the builds freed the engineering team to continue with their development efforts.
The I&T team defined the goals of an upcoming build based on their perception of the engineering team's progress. They discussed the build with the project engineer to ensure that it would meet his expectations. For instance, if the engineering team was unprepared for function tests because some components or subsystems weren't ready, the build would simply be a dry run for assembly, checking that everything compiled, interfaces matched, and third-party versions (JDKs, libraries, OTS tools) were consistent among thread and subsystem team members. For builds where the system was sufficiently mature, the team would perform load tests or function tests against specific aspects of the system.
Near the end of the development cycle, the I&T lead was one of the most important members on the project team - perhaps even more so than the team lead or project engineer. The I&T lead held the plan for test execution, understood the weak and strong areas of the system, and constantly monitored the defect metrics. Also, he managed a full traceability map of the requirements, components, threads, test scripts, and defects, which helped him plan and prioritize his team's actions.
Fixing defects was never a trivial task. Each defect raised the following questions:
- Is this really a defect?
- What type of defect is it? It could fall into any of these categories (from the DefectTracking schema): cosmetic, corruption, data loss, documentation, operation, installation, missing feature, performance, stability, unexpected behavior, or unfriendly behavior.
- Do we need to fix it now or can we defer it?
- Is the related requirement incorrect?
- How can we fix the defect?
- How soon should we fix it?
- What other pieces of software will be affected by the fix?
- Who should fix it?
Whenever the I&T team found a defect in the system, they raised it in the ClearQuest database, where they could capture many of the above details. They connected to the database in ClearQuest and entered the defect in the Submit Defect form, as shown in Figure 1.
|
| Figure 1: Raising a defect in ClearQuest |
| (click here to enlarge) |
The defect database was shared by all members of the engineering team and could be accessed by them at any time through the network. For instance, the owners of the defect raised in Figure 1, who worked on the product's search capabilities, were grouped together in the user information as the Search Team. To keep track of any open test issues assigned to their thread by the I&T team, they used a ClearQuest query. Figure 2 shows the results of the search team's query, with the defect from Figure 1 selected in the result set and displayed. The query results (which always reflected the latest contents of the database as updated by the I&T team) were filtered to include only the search team's defects, sorted by priority and date of submission.
|
| Figure 2: Filtered defect query results |
| (click here to enlarge) |
Other thread teams similarly queried ClearQuest for defects in their areas and received appropriately filtered results. Only the I&T team, project engineer, and team lead looked at the full list of defects on an ongoing basis.
The Actions button shown in Figure 2 offers options to modify, assign, close, duplicate, postpone, or delete the currently selected query result. Who could perform which operations depended on their positions on the project team:
- Engineering team members could only modify or assign defects. For instance, the search team lead could assign the defect to a specific member of his team.
- The I&T team could perform all actions: raise defects in the database and modify, assign, close, duplicate, postpone, or delete them.
- The project engineer had the right to perform all actions, although closure was always left to the I&T team.
System tests ran through the entire system using the recorded test scripts. This testing was important not only in preparation for the customer acceptance tests, but also because it exercised the system heavily and provided insight into flaws in the test scripts. No matter how closely we tracked changes, there were often small changes that had an impact beyond the expected scope, breaking other pieces of code in unanticipated ways.
We always did our system builds on a clean environment set up by the I&T team, so we thoroughly tested the build documentation, which we followed to the letter. If any errors were encountered in the build documentation, an issue (in the "documentation" category) was raised in the defect database along with any identified test defects.
We tested the system's functional requirements, as we had ever since the unit-testing stage (as discussed in Part 8), as well as its nonfunctional requirements, which now received far more attention than during earlier testing. The primary nonfunctional areas we tested were usability and performance (load tests), as discussed below.
We made use of our office's only usability expert. She hadn't joined us with the rest of the I&T team, although she was involved in the early user interface planning and mockup efforts, helping with the human-machine interface. Her present job was to work with the system as a user and identify usability issues, which were also raised in the defect database (usually in the "unfriendly behavior" category).
Some of the usability issues had to be postponed because they were out of the Phase 1 scope or were very costly. However, numerous small usability problems were found that were easy to fix and would have caused the customer some confusion. This was some of our most cost-effective testing effort, because often only minor code changes were needed to vastly improve the product from the customer's perspective. The usability recommendations included new error messages, layout improvements, tweaks to button titles and menus, documentation fixes, and screen workflow modifications.
Our system didn't have demanding performance requirements, but we wanted it to be usable under peak loads. We'd done some load testing earlier, but this type of testing peaked near the end of the development cycle. We wanted to be sure the system would scale beyond ASDI's expectations. Hoping we'd get follow-on work in the form of Phase 2 of the project, we didn't want to create performance problems that could come back to haunt us. We load-tested two parts of the software: the Web application interface and the B2B interface enabled by the SSL/XML-based command gateway (introduced in Part 5).
Web Load Tests
For the Web load tests, we used Rational SiteLoad. This tool enabled us to record a script consisting of a series of Web transactions we performed and then duplicate those steps as any number of virtual users. We reviewed the anticipated load patterns with ASDI to determine how many users might access their Web-based application at once, and we decided to test for 20 users (more than enough, but it happened to be the number of simulation licenses we'd purchased).
With SiteLoad, we were easily able to emulate 20 concurrent users of the system and extract metrics regarding system performance. When we started up SiteLoad, it launched in our browser and prompted us to create a test (by recording a new script or combining existing scripts) or run an existing test (see Figure 3).
|
| Figure 3: SiteLoad's main screen |
| (click here to enlarge) |
When we chose to record a new script, SiteLoad brought up a Java-based browser and recorded all our actions in it. For instance, when we went to our partSearch.jsp screen in this browser, SiteLoad loaded the page from the server (as shown in Figure 4) and recorded our actions, including any values we entered and buttons we clicked. We designed this particular test to perform a number of searches against the database based on multiple parameters. This was obviously a simplistic test, since the database would cache the queries; other tests, however, were more rigorous and challenging.
|
| Figure 4: SiteLoad recording browser actions |
| (click here to enlarge) |
For each script we recorded, we were also able to set some general performance requirements for that test. We decided that when the script that tested the partSearch.jsp screen was played back, we wanted at least 90% of our pages to load in four seconds or less (see Figure 5). Again, this wasn't to meet any hard requirements, but rather to meet our informal goals for usability and overall quality.
|
| Figure 5: Setting SiteLoad performance requirements |
| (click here to enlarge) |
Figure 6 shows how we set up SiteLoad to simulate having 20 concurrent users repeatedly perform the partSearch.jsp screen actions we'd recorded. SiteLoad could have done more sophisticated performance modeling to help us identify the "performance wall" for our system, but we chose to go straight to the maximum of 20 on our first try. If we'd encountered problems, we would have reduced the initial user count to 5 and stepped up the user load by 1 every minute or two. We also could have set criteria for halting the tests, but we didn't bother, since we were visually monitoring the progress of the test to see how our system was behaving.
|
| Figure 6: Setting SiteLoad user characteristics |
| (click here to enlarge) |
While the test was running, SiteLoad displayed and constantly updated statistics like those shown in Figure 7. In this example, a histogram indicates these performance results for our test script: almost all of the page loads performed in under .8 seconds (in other words, in the range 0-20% of our performance limit of 4 seconds), with a small fraction taking up to 2.4 seconds. Using the options provided in the green menu bar near the top of the screen, we could have chosen to see more detailed reports covering, for example, page access, CPU load, and average load times.
|
| Figure 7: SiteLoad test results |
| (click here to enlarge) |
B2B Load Tests
For the SSL/XML B2B load tests, we used Rational Robot to record virtual user (VU) scripts. We entered the command we wanted Robot to monitor and ended up with a script that was quite different from a GUI script created by Robot (discussed in Part 8). Unlike GUI scripts, VU scripts record low-level information pertaining to received and transmitted information. By running the VU script from multiple machines, we could simulate having concurrent B2B client sessions manipulating the system. ASDI doubted that more than two concurrent sessions would ever take place, so we were able to go a few steps beyond this to ensure good performance.
With two weeks remaining in our schedule, system tests done by the engineering team had passed all requirements, and the I&T team had closed all issues except for some minor ones we'd already discussed with the customer and decided to defer. The next milestone for us to pass was test readiness review (TRR). We performed two TRRs: one internal and one with the customer. The internal TRR was basically a dry run, serving as a check of the following:
- Was all documentation complete?
- Was all code checked in and tested?
- Were all code review and unit-testing checklists archived for future verification?
- Did any undeferred test issues remain?
- Were all change requests either closed or incorporated into the requirements?
- Was the Rational Rose model complete, documented, and suitable for delivery?
- Had all aspects of the system been demonstrated to the customer?
In addition to reviewing the above items, we reviewed and rehearsed a system demonstration that would serve as a final presentation of the product to the customer at the external TRR. We were very proud of the product we'd built, and we wanted to be sure to show all its key features because we knew some of the upper management from ASDI would be present at the external TRR.
During the external TRR, we went through the same agenda as during the internal TRR. We ran through checklists with ASDI to show that everything was complete, and we ended with the demonstration. Not surprisingly, more ideas came up at this final demonstration, and we noted them in the minutes for future consideration.
The system had to pass some final acceptance tests in order for ASDI to agree that Phase 1 had been successfully completed. We were reasonably confident it would pass these tests, since we'd written a suite of scripts for performing end-to-end tests of the system with full requirements coverage. These scripts had been created using Rational Robot and had been thoroughly reviewed by ASDI. We figured the only thing that could hamper our success in the acceptance tests would be last-minute changes by the engineering team that broke other pieces of code. But we received a surprise before even getting started, when ASDI told us at the external TRR that they wanted us to run the acceptance tests manually. We thought our intention to use the test scripts had been made clear in the acceptance test plan, but we now realized our wording in that plan was ambiguous.
When we stated at the external TRR that the CATs (customer acceptance tests) would be fairly short and that we'd be able to execute and review script results very rapidly, ASDI said they wanted to see all the tests executed step-by-step so that they'd know what was happening. Although we hadn't expected this, it did seem fair to us, and doable. We'd documented all the test procedures and the plans for our test scripts, and we'd even maintained our test plans as scripts were updated, so it wouldn't be a problem for us to perform the tests manually.
What we weren't prepared for was how long it took to do the acceptance tests by hand. We'd saved so much time with the automated test scripts that we'd become spoiled. Now that we were doing manual tests according to our documented test procedures, we realized just how much time we'd saved.
We found that there were some missing details that had to be provided in the documentation of our test procedures. We hadn't always captured enough information to create an unambiguous and repeatable test. We also realized that sometimes we'd updated the test script without modifying the test plan. After making small changes to the test plans, we delivered them to the customer for a quick review; we agreed that the changes were small enough that another TRR wouldn't be needed.
The acceptance tests took place in our development environment, starting with clean builds according to the build documentation and leading into execution of the test procedures. These tests took a total of about two and a half days to execute. Three members from our team were involved (the I&T lead, project engineer, and team lead), as well as three from ASDI (QA, project manager, and technical lead).
We were proud that no software defects showed up during the acceptance tests. Only minor issues were raised, usually in the categories of "documentation" and "unfriendly behavior." All requirements were satisfied, and the customer was pleased (albeit tired) at the end of the tests.
This was probably the first project on which our team didn't have to work Herculean hours near the end of development and testing. Among the factors contributing to this were that we had better tools, familiar technologies, and an engineering team that had worked together on earlier projects and was finding its "groove."
The testing process in particular had been a big success. The most impressive aspect of the Rational tools may have been the testing functionality. This was our first introduction to automated testing, and we were surprised at how well it worked. The biggest pain with automated testing was script modification in response to changes in requirements; however, this responsibility was delegated to the I&T team and so didn't hinder our engineering efforts.
We now had to install the software at the customer's site and give them some time to evaluate the system there. Although there was no formal warranty stage, ASDI would be raising issues that would be entered into the ClearQuest database. Finally, an agreement had to be reached on the question of whether to proceed to Phase 2.
We felt there were no major risks at this point. We were confident that all the big problems had already been addressed, and we were fully prepared for whatever might come up in the brief remainder of this project.
Steven Franklin has an extensive background in software design, architecture, and engineering process, which he usually applies to large, distributed information management and command and control systems. He's been using Rational tools since 1997, and his primary areas of interest include XML, J2EE, wireless, and software engineering methodologies. Steven can be reached via e-mail.
Comments (Undergoing maintenance)
Table of contents
Next steps from IBM
My developerWorks community
Tags
Use the slider bar to see more or fewer tags.
Popular tags shows the top tags for this particular content zone (for example, Java technology, Linux, WebSphere).
My tags shows your tags for this particular content zone (for example, Java technology, Linux, WebSphere).
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
