System test, as we refer to it in this article, is the final product testing that is done before a software product is made available to customers. This testing typically starts after the completion of development and functional verification testing. Certain entry criteria are used to determine code readiness for system test (e.g. , regression status, defect backlog, etc.). These criteria must be met to officially start system test and claim results.

In a typical system test, the objective is to drive customer-like workloads on a system under high stress in a customer-like configuration (e.g., multiple platforms, multiple software versions, and so on). F ormal system test execution encompasses stress, longevity runs, memory leak analysis, multiple applications, complicated and varied topologies, high availability and outage testing, mixed release testing, full platform testing, product integration testing, and documentation testing.

This article describes the approach taken by one team for including a subset of system test as part of our agile development environment and in parallel with code development activities. We’ll show that our approach delivered a better system test application, allowed experimentation to maximize the value and efficiency of including a subset of system test in parallel with code development, and improved communications and skills across the team. These successes lead led to the anticipated benefits of improved full software verification team ( SVT ) ramp-up time and improved product quality at full system test entry.

The system test core team was engaged in parallel with the development team over six iterations. This enabled a subset of traditional system test application development and execution runs during every iteration. This parallelization led to the areas of success described here.

An agile process implementation model

After a year in development, using existing waterfall practices and delivering an open alpha and an open beta, our project switched gears and implemented an agile development model. The entire team was asked to embrace agile development principles as a “live experiment” to meet the requirements of a particular customer. The first iteration was a transitional one, mostly focused on delivering code that had been under the waterfall development model for multiple months. At the kickoff for the first iteration, the entire team consisted of approximately fifty-five people, including designers, developers, testers, information developers, and customer delivery teams.

Iteration 1 was used as a transition iteration, to finish deliverables that were already in development. The agile development transition started conservatively, with moderate development commitments for the first one or two iterations. The primary focus was on building and communicating a new project and interpersonal culture across the teams and integrating processes, such as parallel early system test, that would be used to make the teams successful in this new environment.

Nine system testers, a subset of the entire system test team, were involved part-time during the previous waterfall development cycle. While the code was being developed, they focused on building technical skills, planning, test case development, and application unit test. In doing so, they built skills in the technologies used by the project, but didn’t get any hands-on experience.

At the time of the change to agile development, five of the nine system testers were assigned full-time to the project. So prior to the first iteration, there was a system test team already learning new technologies, creating the test application, and working together as a team. These early test and development activities allowed higher productivity upon beginning the agile cycle.

Goals for system test team involvement

The challenge given to the five system testers was to ensure code quality sufficient for a beta-level delivery of functions developed in any given iteration. To meet the challenge, they decided to run a subset of the traditional system test application development and execution during every iteration, in anticipation of a final iteration fully dedicated to a more complex set of customer-specific, scenario-based tests. The system test team met with the senior architects of the project and together they selected one of the existing system test applications for enhancement and execution. The application would then be used to focus on stress and longevity runs throughout the iterations, and would be heavily used during the final iteration. The intention was that this limited and simultaneous engagement of system test resources would establish a solid foundation for the final, full-system test iteration as the project ultimately approached worldwide delivery.

Project and iteration timelines

Iterations were six weeks long. Table 1 shows the general iteration layout and the specific system test activities planned for each week. During week 1 of an iteration, the senior architects presented the proposed candidate function list. The list was primarily based on the specific customer requirements, and was described through use cases. The list was kept online so that all team members could refine the use cases as the iteration progressed. Each team member would review the candidate list, discuss design and issues with team members and senior architects and, by end of the week, commit the function to be delivered at the end of the iteration (or not). A daily scrum meeting was held. Senior architects were available at most of the daily scrums, and were available to all team members throughout all iterations.

The last iteration, as previously mentioned, was planned to focus on final defect cleanup and complex testing that was not practical during development iterations. In order to improve stability, no new function was targeted for the last iteration. Six iterations were completed. Also, in the last iteration, a significant number of developers were expected to join the system test team members in completing the final iteration by providing assistance with test execution and debug. The remaining developers would handle defects. This meant that during the development phase in earlier iterations, some developers needed to be trained to do system test in preparation for the last iteration.

Table 1: Activities per iteration: development and system test

Back to top

Successes

Here we’ll begin a discussion of some of the key successes realized, challenges encountered, and benefits conferred in relation to our agile system test activities. Let’s start with the successes.

More robust and realistic system test application

The agile development cycle started with the customer. The architects met frequently with this customer in order to understand their business objectives, IT strategies, application direction, and priorities. The system test team analyzed the iteration’s use cases and designed the test application enhancements based upon the subset of use cases that were considered to be critical to the external customer. If the test application enhancements were extensive, those enhancements were developed on a previous iteration to allow time for development of the test application. If the application enhancements could be done quickly, they were developed on the current iteration.

During week 2 the system test team presented the application design enhancements to the architects. This resulted in a dialog that provided the system test team the opportunity to incorporate the architects’ recommendations. It also provided the architects with a system test perspective that they may not have considered and which, at times, resulted in product design changes. Due to the system test team’s close working relationship with the architects and the development team, the system testers gained a much deeper understanding of the customer requirements.

Inclusion of the small system test team in parallel with development activities enabled the system test team to develop a much more robust, “customer-like” test application in an incremental fashion. This was a more realistic approach than what had historically been done during the waterfall development cycle, since it allowed the system test team to participate in the creation of the deliverables, instead of being forced to consume and test a large deliverable in a short amount of time. The agile development cycle allowed the system test team to act like a real customer, from the application design right through coding and unit testing of the system test application.

Closer communication between development and system test teams

A significant benefit derived from the adoption of agile development practices and the inclusion of system test activities was more frequent and deeper interactions between the development and system test teams through participation in the daily scrum. As a result, the system test team gained insight into current build problems, recently discovered defects, design modifications, etc. that would impact their test plans and/or the test application. The scrum also benefited the development team when system testers reported issues found during concurrent system test. Daily scrum communication allowed system test to briefly describe to the entire audience problems that may have required coordination or understanding by multiple developers. Finally, the scrum allowed development to indicate when the system test approach was inconsistent with the current code base functionality.

Another key benefit of the close communication between development and system test is that the system test team was immediately aware when new functionality was first available in a build. The communication of available functionality was critical since there were only three weeks per iteration to perform system test application design, code, and unit test. This was important because system test was sometimes developing test applications in parallel with the product functionality for that iteration. Remember, system test would make enhancements to test applications and unit-test those improvements very quickly. Because system test received timely notification that the required functionality was in a build, they were able to perform application testing as soon as possible.

The close communication between the architects, developers, and system testers allowed the entire team to benefit from multiple points of view and recommendations, improving code quality and completeness.

Deeper system test skills

Another advantage of including a system test subset during the agile development cycle is that it fostered deeper system test application development and test skills. This smaller, “core” system test team was assigned to the project much earlier in the release cycle. The iterative nature of the agile development cycle allowed the system test team to understand the product over a longer timeframe in smaller, more readily consumable pieces.

Continual subset of system testing throughout iterations

A subset of system test was performed throughout all iterations, even though the system test timeframe per iteration is limited. Various automation techniques were employed to enable stress/load testing, memory leak analysis, data integrity testing, and regression testing throughout the test effort. A stress/load tool allowed the system test team to run test applications under a simulated number of clients, for a specified duration. There were execution scripts that captured browser actions, which were played back, allowing automated execution to be performed. These scripts also allowed random data to be generated, which simulated diverse customer interactions. Related configuration scripts allowed the testers to specify the desired duration and number of simulated clients for each run. They monitored the run progress by periodically checking the product logs, as well as the logs of the stress tool. The stress tool also generated statistics, which provided potential indications of problems and unexpected activity. The stress tool was used during every test run after the first iteration.

Success with code stability, and therefore ability to do longevity test runs, resulted in the addition of memory leak analysis at the completion of an iteration starting with the third. Prior to each test, the system testers configured several product parameters to allow the memory leak analysis information to be collected during testing. Following each run, they would input the run’s log into the memory leak analysis tool. A set of graphs were generated which allowed viewing of the memory usage during the run and checking for indications of memory leaks. If memory leaks were discovered, bug reports were opened and pursued. Once the final run completed without problems, testers included information in the completion records of the test tracking tool, indicating the results of the final memory leak analysis.

Data integrity testing was also done during some of the test scenarios to ensure a subset of system testing was being performed. The test application relied heavily on IBM® DB2® for data persistence. The application consisted of a set of transaction workloads which updated the same set of tables in the application’s database. These transaction workloads were being run simultaneously, thereby driving load and intentionally creating database contention. The intention was to force transaction rollbacks to occur, so system testers could ensure that recovery completed successfully. When updates were attempted on records which were already locked, the transaction was rolled-back and retried at a later time. At the completion of the run, all the database activity was complete and all transactions had to be reconciled. Database consistency checking was built into the test application. The data persisted by the application had interrelated values across the set of tables it used. At the end of the run, the tester ran a set of additional tests which verified the data in each of the application’s tables were consistent with one another.

Regression testing was also used during iteration development starting with the second iteration. Iterative regression testing was executed each iteration by running the previous iteration’s version of the system test application on the current build for the current iteration. As we proceeded into later iterations, these test runs quickly became a collection of earlier tests running simultaneously. This allowed regression defects to be identified and handled throughout the iterative process, rather than at the end of test as is the case in a waterfall model.

Therefore, even though this agile release cycle consisted of a series of relatively short iterations, a subset of system testing was still able to be performed. Stress/load testing, memory leak analysis, data integrity testing, and regression testing were all performed due to the extensive automation capabilities exploited by the team. These types of tests provided confidence of the stability of the product code in simulated customer environments.

Bugs removed earlier in product cycle

Because system test executed regression, stress, and duration runs during the iterations, defects were identified and removed very close to the time of injection. System test received immediate fixes from development since the product code was fresh in developers’ minds. System test more easily verified fixes in an agile environment, due to defect turnaround time being faster and since the exact test environment was still in place. Collaboration between developers and system testers was better, since both had the same priorities as a result of working on the same use cases in the same timeframe. This facilitated defects to be identified, fixed, and verified more quickly.

Earlier iterations tested initial and more basic product functionality. During this time, the system test application was also basic and simpler; however, it was able to be run under a certain level of stress and duration in order to find defects. Subsequent iterations introduced increasing product functionality and more complicated product configurations. The system test application increased in functionality and complexity as well. The levels of stress and testing duration increased during the subsequent iterations. Typically, more complex defects were found in these subsequent iterations, but still very close to the time they were injected into the product.

In general, there was more development focus on system test defects during the agile development cycle. This resulted in earlier detection of defects and faster defect fix turnaround time.

Process evolution

The agile environment itself allowed for continual enhancements. There was a “lessons learned” session at end of an iteration that allowed all parties to identify problem areas and to suggest overall improvements and efficiencies.

One key characteristic of the agile model was that the development process was expected to evolve. This was true across all teams associated with the product deliverable, including system test. Insights were realized every iteration regarding how productivity could be improved. Each team was asked to offer their thoughts about what was learned and what could be improved. The overall team met and discussed each pain or success point. During that dialog, additional improvements were identified, captured in an online repository, and integrated into the next iteration.

One specific example of process evolution related to system test occurred during the first iteration where the system test team formalized a process for test tracking. In the past, a test tracking tool was used to define system test scenarios and document test execution results. In iteration 1, the system test team defined templates for those test scenarios, and created common configuration documents. These configuration documents allowed the system test team to detail the steps necessary to setup and execute common system testing procedures from one common documentation source. This single definition enabled individual testers to avoid creating and maintaining duplicate documents. The common documents were pointed to by all the relevant test scenarios.

Another example of how the system test team improved their process incrementally occurred when memory leak analysis was introduced in iteration 3 rather than waiting for the last, system test-only, iteration. This change was possible due to the code stability experienced, as previously mentioned. The necessary tools and setup instructions were added to a common configuration document, which was referenced by each planned test scenario. Also, the test completion details included the results of memory leak analysis following each run.

The system test team defined conventions and well-documented processes to assist in achieving repeatable and consistent testing. Many of these improvements could not have been known or defined prior to beginning our test effort for this project. Some of these ideas were based on “discoveries,” and others came purely from a desire to improve productivity for the following iterations. The frequency of “lessons learned” and the focus on model flexibility allowed the overall team to avoid earlier pain points, improve efficiency, and experiment with solutions.

Stop-ship identification

Beginning with iteration 3 the system test team drove identification of stop-ship scenarios. Starting with iteration 4, during the iteration commitment meeting, the system test team’s scenarios were classified as either stop-ship or non stop-ship. All stop-ship scenarios had to run successfully without defects or patches in order to successfully complete an iteration. The stop-ship test scenarios were generally those test cases which ran in a previous iteration and were carried forward as regression scenarios. These test scenarios were also combined to form a suite of tests which were run simultaneously. These combined tests were typically run under higher load than in earlier iterations and also run for at least twenty-four hours.

Non stop-ship test scenarios were generally based on customer requirements that needed to be delivered in a subsequent iteration, but were based on functionality being developed in the current iteration. This allowed initial system testing and problem determination to be performed very shortly after functional verification completed. This additional flexibility in the agile environment allowed complex problems to be identified, isolated, traced, patched, and run in the system test environment in a timeframe that spanned multiple iterations.

The system test team distinguished between stop-ship and non stop-ship test scenarios in their test tracking tool. During the commitment meeting in week 1 of the iteration, the senior architects provided guidance in classifying each commitment as stop-ship versus non stop-ship. Those classifications were noted in the online commitment page and were later used when creating test execution records for each of those commitments. For the stop-ship scenarios, the test phase had a completion date at the end of that iteration’s system test cycle, which occurred in weeks 5 and 6. The non stop-ship scenarios spanned iterations, but all remaining non stop-ship scenarios then became stop-ship scenarios in the final iteration. This distinction between the test scenarios helped manage risk, by allowing additional resource to be applied when unexpected problems occurred, at the expense of test scenarios of lower priority. This allowed the iteration deliveries to be shipped on schedule, but required careful management of the non stop-ship scenario backlog.

Back to top

Challenges

Following are some of the most important challenges faced in the course of agile system test activities.

Short iterations

Regression testing of system test scenarios occurred during iteration weeks 2 through 6. During weeks 2 through 4, system test also needed to enhance and unit-test the system test application so that it was ready to be used to test the new functionality during weeks 5 and 6.

System test scenario execution for new functionality only occurred during weeks 5 and 6. This length of time proved to be challenging for the subset of scenarios tested during this “live experiment” when the system test runs became more complicated in later iterations. For example, in the cases where the testing required more complicated topologies combined with durations longer than three days and/or complex workloads with outage and failover testing, it proved difficult to contain these complex scenarios within the allotted timeframe. These more complicated scenarios tended to expose more difficult bugs, which were harder to debug and sometimes were intermittent. Re-creation of these types of defects with tracing was time-consuming, as was applying patches and verifying fixes integrated in a driver. This became very challenging given the small number of testers assigned. As a result, some of the more complex scenarios and defects needed to span iterations.

Even though only one application was used for application development and scenario execution, application development consumed a significant amount of time in an iteration. This was partially mitigated by spanning two iterations for some functional enhancements to the application, doing application development during one iteration and scenario execution during the following iteration when needed.

Another challenge was the timing of the iteration results review and demos. In week 6, the system test team presented their iteration results to the overall team. During some of the iterations, the testing was not completed to present the results in the final week and the presentation of system test results needed to be moved to week 1 of the subsequent iteration. Also, at times the system test team was responsible for a portion of a demo to product management to showcase the iteration deliverable. These demos required significant preparation, and system test team participation became more difficult in later iterations.

An additional factor that made weeks 5 and 6 very challenging was that system test needed to train developers to do system test execution so that those developers could assist system test in later iterations. However, this required experienced system testers to mentor developers assigned to system test during that particular iteration, increasing the difficulty of completing the planned system test commitments. Since this involved a learning curve of approximately 1-2 weeks for a developer new to system test, the benefit to system test was only realized if the same developers were assigned to system test during multiple consecutive iterations, which was not always possible.

First agile development experience

This project was the first experience in agile development for most of the team members, and the parallelization of development and system test activities for this project was new to everyone on the team. The system test team of five people worked more overtime than expected. Throughout the project, every individual, including the system test team members, was learning how to size the design, development, and test tasks per iteration. Over-commitment or erroneous estimations created stress points on the system test team, as well as the other members of the overall team.

Use cases

Standardized naming conventions for use cases across iterations needed to be defined and implemented early in the agile cycle. In order to ensure that the system test team didn’t lose track of any of the use cases, and to allow all teams to be able to clearly identify each use case, it would have been helpful to have standardized “use case identifiers” used consistently, starting with the first iteration through the final iteration.

A template was put in place to consistently document use cases across the entire team. While the concept of a use case was not entirely new to the overall team, stating use cases in terms of external customer value was new to most team members. As a result, the template and wording of use case descriptions continually changed as a result of ‘lessons learned’ feedback at the end of the iterations which included feedback from the entire team including the customer delivery team.

During the project, customer documentation was a formatted version of the use case documentation. Since system test didn’t have the standard external customer level documentation available, the system test team was not able to test this type of customer documentation. If this standard product documentation was produced in the future, it would require an additional test effort.

Back to top

Benefits

Among the specific benefits associated with agile system test efforts were the following.

Better system test application

The system test application developed had significantly higher quality due to a series of ongoing enhancements over several iterations. When making all of these improvements, a variety of other aspects of the application were improved as well, including validation performed, logging provided, and the exception handling. The test application was more robust since it was developed over many months, as opposed to being developed during a traditional, concentrated test preparation phase.

Improved full system test ramp-up time

A core system test team working closely with development in an agile environment offers significant potential benefits to the development organization. The core team can produce a subset of the system test applications, scenarios, and automation scripts for use during final system test prior to product release. The creation and execution of these materials in parallel with development will help facilitate the broader system test team to become productive more quickly.

From a skills transfer standpoint, the system test core team, at the appropriate time, can introduce the new technology and release content to the broader system test team. The core team can present the customer-like application design and demo the application to the broader team, as well as offer pointers to documentation developed and to established development contacts, thus improving ramp-up time for the full system test iteration.

Improved product quality at full system test entry

Because the system test core team was executing a subset of system test early in the product development cycle, the code quality should be improved at the formal entrance to full system test, positively impacting the productivity of the broader system test team. Additionally, a release cycle using a system test core team engaged in early system test activities, followed by a concentrated full system test before product release, can offer improved quality compared to traditional development/test methods. Note that at the time of this writing, the final system test iteration has not yet been entered. During the development iterations, a subset of stress-related and memory leak defects were removed before the formal system test entry time due to the early testing by the system test core team.

Back to top

Conclusion

The project discussed in this article was small. It consisted of about fifty-five people including managers, architects, developers, and testers. There were about twelve different component teams, including teams working on two open-source technologies, which served as the basis for the technical function. The component teams were in different geographic locations; however, the small size of each team made it fairly easy to adopt agile development principles across the project. The prior year on the project using waterfall methods served as an education in the technologies used on the project. Transition to agile concepts was made easier by this prior technical education. The team did not have the additional challenge of a technology learning curve.

Iteration 1 was used as a transition iteration, to finish deliverables that were already in development and start operating as an agile project. The agile development transition started conservatively, with moderate development commitments for the first two iterations. The primary focus was on building interpersonal culture across the teams and establishing processes, such as parallel early system test application development and execution, which would be used to make the teams successful in this new environment. Based on the results of this project, time spent on a transition phase from waterfall development to agile development is encouraged.

The system test core team was engaged in parallel with the development team over six iterations. This enabled a subset of traditional system test application development and execution runs during every iteration. This parallelization led to the following areas of success:

• A more robust and realistic system test application incrementally built across multiple iterations.
• Closer communications between development, functional verification test, and system test enabled multiple points of view to be heard and shaped the system test execution.
• Deeper skills in the technologies and environments used in the project.
• Removal of some system test defects earlier in the product cycle.
• Iteration by iteration process or testing improvements.

Resources

• Participate in the discussion forum.
  
  
• A new forum has been created specifically for Rational Edge articles, so now you can share your thoughts about this or other articles in the current issue or our archives. Read what your colleagues the world over have to say, generate your own discussion, or join discussions in progress. Begin by clicking HERE.
  
  
• Global Rational User Group Community
  
  

About the authors

Mary Ellen Kerr (mkerr@us.ibm.com) is an IBM WebSphere Application Server System Tester with twenty-five years of IBM experience.

Curt J. Rousse (crousse@us.ibm.com) is a WebSphere Application Server System Tester with thirty years of IBM experience.

Donna P. Dynes (dynes@us.ibm.com) is a WebSphere Application Server Development manager with twenty-six years of IBM experience.

David Booz (booz@us.ibm.com) is an IBM Senior Technical Staff Member with nineteen years of experience in operating systems and middleware.

Rate this article

Comments

Back to top

Table of contents

• An agile process implementation model
• Successes
• Challenges
• Benefits
• Conclusion
• Resources
• About the authors
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

Dig deeper into Rational on developerWorks

Try IBM PureSystems. No charge.

Experience today!

---

Get started with the cloud-based trial and pattern development kit.

Special offers