Manage operational requirements for production

Part 2. Build test cases for pre-production testing


Content series:

This content is part # of # in the series: Manage operational requirements for production

Stay tuned for additional content in this series.

This content is part of the series:Manage operational requirements for production

Stay tuned for additional content in this series.

In the previous article, we saw how a collaborative approach between development and operations teams helps to identify the set of non-functional requirements (NFR) that have impact on the execution of the new system in the production environment. Identifying the requirements is however only half of the story.

To minimize risks when introducing a new system in an IT portfolio and infrastructure, we must identify the pre-production tests that can prove the robustness of the new system and build our confidence in its adequacy for business use.

Our first consideration is that each non-functional requirement is a technical consequence of the business need identified by stakeholders of the new system (cit. Rozansy and cit. Losacco). For example, when business people say, "one category of users must have access to the system at any time, any day of the year" that translates in IT terms to, "this system must be available 24x7.". Other users, such as employees who use the system from the back office during business hours, might require different availability windows. If the topology of the physical architecture includes separate channels for Internet connections and back-office connections, the 24x7 availability NFR will not impact the web servers that run the back-office channel.

Our second consideration is that the effects of certain NFRs on the execution of the new system are seldom independent from each other. For example, guaranteeing sub-second response time to a single active use, is not the same as guaranteeing the same performance to 10,000 concurrent users. In the same way, if the system topology includes the communication between internal components via a queuing system, the response time tests must include the workload generated by the expected number of concurrent users with the same physical constraints on the size and speed of the queue that will be in the production system.

Other articles (cit. Alexander and cit. Cripps) have expressed this fact with the notion of conflicting forces, which is similar to the loads and forces that exist on a building and are the base for the structural design work of civil architects. We will examine combinations of these forces to identify the corresponding stress cases, i.e. the combinations of NFRs that can actually happen during production time.

In our proposed approach, the relationships among stress cases, actual use cases of our system, and a specific test environment identifies our pre-production tests.

Identify pre-production tests

Non-functional requirements (NFRs) originate from a business need of system stakeholders. Since stakeholders typically assume different roles in an organization (for example, end-users, application developers and IT Operations), a collaborative approach among these stakeholders can lead to an agreed upon set of requirements.

To understand how requirements contribute to stress cases and how they relate to use cases, a good point to start is the operational context diagram (cit. Losacco). This system context diagram focuses on non-functional characteristics of a system. Its actors are typically UML actors (users and external systems), and the various IT professional categories that eventually interact with the actual system for its operations.

For each actor, the diagram identifies the requirements that originate from system use or by the channel used to access the system. Additional NFRs may originate from stakeholders who do not interact with the actual system but only with its development process, like IT architects or security officers.

We will continue here with the example we used in the first article. We consider the case of introducing a new system (For example, an Auction-based e-marketplace) in an existing data center.

Figure 1. Operational context diagram for e-Auction
The diagram extends the system context to NFRs
The diagram extends the system context to NFRs

Larger view of Figure 1.

As it is easy to see, the diagram highlights next to each actor his own non-functional requirements

For example, concurrent unregistered users are expected to be 10 times the number of registered users.

Registered users need a very responsive system during the auctions; on the other hand, this is not so important for unregistered users that can only browse future auctions.

Examine the non-functional requirements

Our first step is to build the conflict table for non-functional requirements. Our starting point is the available list of NFRs from the Operational Context Diagram. The table is split into the following zones:

  • In the first area, rows list the NFRs ordered by actor;
  • In the second one, rows represent key use cases for the pre-production test.

The columns contain all NFRs, including the ones that pertain to actors and the ones that are originated by other (non-actor) stakeholders.

Figure 2. Conflicting requirements matrix
The matrix identifies conflicting NFRs
The matrix identifies conflicting NFRs

Larger view of Figure 2.

For each row, a plus sign indicates where the NFR (column) adversely affects the actor NFR in the row heading by putting more stress on the system.

The sum of the plus signs in a single row identifies a stress case. It represents a combination of loads that can stress the system. These stress cases will be implemented in our pre-production tests.

Identify conditions for stress cases

The second step of the process is to identify the functional conditions that lead to the appearance of a stress case. Functions are typically represented through use cases; therefore we start from a list of (relevant) use cases. For each use case, we identify the NFRs that can occur during its execution. For example, the Bid Item use case is performed by the Registered Provider actor. In this case, 200 users might all bid at the same time on the same item of the same auction.

In real life there is no single way to define what are the "relevant" use cases and, by consequence, the "right" list. On one side the list cannot include all use cases of the new system, on the other it cannot only include a single use case per stress case. As a simple rule, we will start from use cases that were automated in previous tests. If those are not available, we can use architecturally significant use cases that were identified during the design of the system and deemed critical.

A "good" list of use cases for pre-production test must meet the following criteria:

  • Covers all listed NFRs
  • Includes all major functions of the new system
  • Includes the functions that are high-risk or highly-sensitive

Identify use cases

The third step in defining our pre-production test plan is to identify which use cases to associate with each stress case. For example, our first stress case attempts to demonstrate that our system can sustain nine hours of auctions by registered suppliers with no interruptions. Transactions are registered in the contract system within the batch window while random R/O access is performed by interested suppliers.

To reproduce this scenario, we need to use at least the following use cases: Bid Item, Reorder Item, Browse Auction. All of them must run concurrently.

On the other hand, some situations require even more sophisticated judgement: The requirement "Up to 200 concurrent R/W Users" can be verified by several choices of use cases. To make a good selection, you must take into account the relevance of those cases and their likelihood to produce the specific stress conditions in real operations. In some situations, more than one test case might be required to adequately test the stress case (For example, a test case based on Bid Item and another one based on List Active Auctions). In addition, both the "Up to 200 concurrent R/W Users" tests must have a "noise" script that runs in parallel to produce the level of background load needed by the stress case.

Some use cases can be chosen for more than one stress case. While the use case might be the same, the test cases can have different test environments, such as duration, loads, number of users, and so on, as each stress case brings its own perspective on how viable the new system is for production use.

To describe our stress cases in a formal way, we need therefore a tool that is able to link use cases to test environments to the planning and execution of the resulting test cases.

In the remaining of this article we explore the capabilities of Rational Quality Manager to support this task.

Manage testing activities with Rational Quality Manager

Rational Quality Manager is a collaborative solution that supports the various phases of quality management in a software development project. Each activity is assigned to different roles or team members. Definitions, assets and artifacts are collected centrally, so that they are always available to team members and can be reused across test cases, test plans or projects. The progress of each task as well as progress of the overall quality management project is tracked and analyzed in real time.

In addition to supporting the test team in planning and constructing test plans, the tool provides a lab management capability that is used to manage resources, such as physical and virtual systems that are available for testing.

These resources are what you use to build your test environments: they can be single systems or complex, multi-system configurations. Testers can either reserve pre-defined resources, or request specific configurations and builds.

In the execution phase, the tool coordinates the testing activities, supports the execution of manual tests and integrates with external tools for automated testing. The results of all tests are stored and there are numerous ready-to use reports for the following analyses:

  • To monitor the work assignment and execution progress, to optimize the test process and resource allocation
  • To monitor the application quality (number of defects, severity, defects that have been fixed, etc.), how it changes across the builds, up to matching of the exit criteria
  • To check that the test plan adequately covers all requirements

Translate stress cases into test cases

To build our pre-production test plan, we used several functions and resources offered by Rational Quality Manager :

  • Requirements: Our pre-production test has to verify if the new system can really support the production requirements. As we have seen so far, the production requirements are combinations of mutually-reinforcing, non-functional requirements. At this point, we do not trace the test of use cases or atomic NFRs, but we do trace the test coverage of the stress cases. These are our RQM requirements.
  • Scripts: Scripts translate a use case into a sequence of manual or automated steps. Scripts are the elements that we use to stress the lab test environment to a level that verifies the stress case. When the use cases needed to reproduce our stress conditions are identified, we need to build manual and automated scripts to run them.
  • Test environments: Atomic NFRs become the specifications of our test environment. For example, by using the information in row RS1 of the spreadsheet, we can build the graphic in Figure 3 (below), which is the description of the key elements of the test environment for our availability testing. We use such graphics as a high-level view of the test environment that we need to build.
Figure 3. An availability test environment
The test environment is a consequence of NFRs
The test environment is a consequence of NFRs
  • Lab resources: Lab resources are the building blocks of test environments. They can be physical or virtual machines. What we need to do in our RQM test environment specification is to identify the resources to build it. This is shown in Figure 4 (below) and in Figure 5 for availability test, and again in Figure 6 for response-time tests. Notice that a lab resource definitions can be much more complex than what is represented in our sheets; however, when defining the test environment we focused only on those characteristics which are important to our test objective i.e. on the ones related to our operation-related requirements.
Figure 4. Lab resources for an availability test
Lab resources make up the test environment
Lab resources make up the test environment

Larger view of Figure 4.

Figure 5. Lab resources definitions in Rational Quality Manager
Each lab resource has its own set of definitions
Each lab resource has its own set of definitions

Larger view of Figure 5.

Figure 6. Lab resources for response time testing
A second example of a test environment
A second example of a test environment
  • Machines and cells: Machines are the representation of real or virtual test systems. Machines are made available to the test through reservations. The (real) test environment for a test case is represented by a collection of test machines and is called a test cell. A test cell is the physical implementation of a test environment. The machine is the physical implementation of a lab resource with matching characteristics.
  • Test cases: Requirements, scripts and test environments all relate to an RQM test cases. A test case contributes to the verification of a requirement by linking an appropriate test environment to the execution of functions that can stress the system as needed. By tracking the execution of the resulting test cases, we check the satisfaction of our production requirements.

And now, you can have a good pre-production test (...and, hopefully, an uneventful production)!.


We have examined how to identify pre-production tests that represent real stress conditions on new systems. We used techniques to determine which non-functional requirements are mutually reinforced, build a stress case for the new system, and then determine the test cases necessary to adequately test it.

Then, we used Rational Quality Manager to translate the stress case into test scripts and test environments for our pre-production tests, ensuring that the test phase adequately verifies the expected loads and execution constraints of the new system.

Downloadable resources

Related topics


Sign in or register to add and subscribe to comments.

Zone=Rational, DevOps
ArticleTitle=Manage operational requirements for production: Part 2. Build test cases for pre-production testing