Compare test data generation methods
For a performance test, you must find efficient ways to create large data volumes for testing during the early stages of software development. The specific data volumes must simulate a real product environment, and the data sizes must vary from small to large so that you can find the performance bottleneck of the product that is being tested.
Table 1 contains examples of IBM® Rational® Quality Manager data volume definitions.
Table 1. Data volume definitions from Rational Quality Manager
Vaughn Rokosz, an IBM Rational systems verification tester, has defined the following approaches for adding test data into a system:
- Use test automation tools, such as IBM® Rational® Performance Tester or IBM® Rational® Functional Tester, to populate the test environment by simulating user activities.
- Use public APIs or command line tools to develop specialized population tools.
- Import test data from customers if the application supports importing.
- Use the bulk load tools that are provided with relational databases to initialize database tables.
- Use internal APIs to develop specialized population tools.
Some of these approaches for adding test data to a system can be challenging to implement. For example, applications that are based on Jazz Foundation Services access repositories through storage services. This logic data model makes it impractical to directly load database tables by using bulk load tools that are described in the fourth method.
Another challenge can occur at the beginning of the application development lifecycle. The user interface is typically not available, or it is unstable to use. This challenge makes methods 1 and 3 not feasible because both methods are based on client operations. Even when the application is stable enough, these two methods are time-consuming.
To address these challenges, you need to develop a method for populating data by using the API against the current build code. The risk of using this method is that it can require a large effort to accommodate build changes. To address this problem, you need to find an effective tool that can work independently from build changes.
Discover the value of Jazz Foundation Services
IBM® Jazz™ technology-based applications are considered to be extensions of Jazz Foundation Services, which are based on a Representational State Transfer, or RESTful specifications, and resource-oriented architecture. Each application provides services and data as REST APIs. In these cases, data can be exposed as addressable resources, especially as a URI. On the other side, it works as a Web service application that provides a create, read, update, and delete interface for its resources. These interfaces are implemented by the POST, GET, UPDATE, and DELETE methods within the HTTP protocol. This loosely coupled architecture enables different products to integrate with each other and enables clients to communicate with the Jazz Team Server in the same way no matter how the service is implemented.
Figure 1. Jazz Foundation Services
Populate data based on the REST API
Based on the REST API that is provided by Jazz-based applications, we proposed a data population method that takes the advantage of approaches 1 and 2, which were outlined in the first section. To illustrate how you can achieve this method, you can use Rational Quality Manager as the targeting server.
For this exercise, the data population goal is to create 20 test plans and add 1000 test cases to each plan. Other tools that we can use to illustrate this method include Rational Performance Tester, the Firefox browser, and the Firefox Poster plug-in.
For this example, the authors assume that you can create a template that can be assured before a system verification test (SVT) entry. The template file for this test plan example is testplan_001, which contains one test case, named testcase_001.
Get a resource template through REST API
Some applications provide the REST API during installation. For example, you can get an XML template of various resources from the Samples folder, which is in the installation directory. You can also access the XML schema of the test plan by URI:
If you do not have this template information, you can get the XML template of the target resource through the REST API as long as there is a target resource in the repository. You must also ensure that the URI of this resource is available. You can get the URI through one of the following methods:
- An API document that you can get from a developer
- A link to information that is on the client side
- The HTTP protocol data that is recorded by Rational Performance Tester to get the URI of the specified resource
After you get the URI, you can use it to get the resource template for each record type. The template is independent of the product build change because it is retrieved from the current repository. Here is an example from the record type test case in Rational Quality Manager:
- Use the URI to get the test case list in the browser.
Figure 2. Get the test case list
- In the results, select testcase_001. To open it, right-click on the page and select View > Page source. As Figure 3 shows, you can get the source code for this test case record in the XML format, save this page source to a file named testcase_001.xml which will be the resource template file for test case record.
Figure 3. Code in the XML template of testcase_001
- You can use these same steps to get the XML template file for the test plan, testplan_001.xml.
Record operations with the Firefox Poster plug-in
Poster is a plug-in for the Mozilla Firefox browser that can simulate the interaction between an application and an HTTP server. You can send and receive content from a URI by using the standard HTTP actions: PUT, POST, GET, or DELETE.
To use Rational Performance Tester and the Poster plug-in to automatically create the data repository, follow these steps:
- Start the Rational Performance Tester HTTP recorder. Firefox with the Poster add-on opens and records all of the requests and responses that are sent and received by the browser.
- If you are connecting to the Jazz server for the first time, you will be prompted to log in. Because Poster runs as a Firefox add-on, it shares the authenticated session from your login to the Jazz server. Rational Performance Tester can record and handle this session automatically.
- Check the resource operations that were performed by Poster. Review these three areas in the Poster dialog (see Figure 4):
- URL: Make sure that this corresponds to the resource that you want to interact with.
- Action: Select GET/POST/PUT to respectively retrieve, add, or modify the content.
- Content to send: Used in a POST/PUT where content can be added or modified. An element value such as identifier or title must be changed in this step to make the result correct and easily verified,
- Click Go, and it will send the request to the server. If you get a response code of 201, then a new resource was created by the POST action. If you get a response code of 200, then the resource was modified by the PUT action.
Figure 4. Use the Poster plug-in to generate testcase_001
By using Rational Performance Tester and Poster to record operations as below, you have generated a test script that simulates the scenario for our example. See yellow highlited page names of Rational Performance Tester test script in Figure 5:
- Create testplan: create one test plan (POST by Poster)
- Get testplan: get new created test plan (GET by Poster or by browser)
- Create testcase: create one test case (POST by Poster)
- Modify testplan: add new created test case to test plan (PUT by Poster)
Edit the test script
When editing test scripts, you must edit these elements (Figure 5) to incorporate the Rational Performance Tester data pool, custom code, reference, and other all functions that can be used.
- Add a loop or make a schedule to generate large data volumes.
- See the loop example in Figure 5. It can generate 20 test plans that each have 1000 test cases.
- Enter some values in the request URI or data content.
- The script is independent of resource type. You can substitute URI with other resource types to generate various resources. A resource ID can be substituted to generate resources with different IDs. To make the resource unique, edit the resource title, description, and other elements. See the right panel of Figure 5 for an example.
- Create a data correlation to support the resource relationship.
Figure 5. Edit the test script
In Figure 5, notice the custom code ModifyCase and ModifyPlan to make each test plan has its 1000 test cases. In each test plan XML data content, one test case is linked by adding one XML element shown as test case template in Figure 6.
The custom code in ModifyCase uses test case template as the input parameter to generate the 1000 test cases XML document. Its output will be merged into an test plan XML document that is retrieved from the get testplan request by the ModifyPlan custom code to generate a modified test plan. The code merges the XML documents, you can also find flow diagram of how the XML file merges in Figure 6. Flexible input argument settings in Rational Performance Tester make the target resource content independent.
Figure 6. Create a resource relationship by using custom code and data correlation
This data generation method can automatically populate data that is independent of the development environment and application interface. To implement this method, all you need is the REST API of the product, because this method is independent of any build change. Also, because this method works directly on the server resource through the API, its performance is better than other approaches that were outlined in the first section. For example, you can populate 1,000 Rational Requirement Composer resources in 4 minutes by using this method. If we had used one of the other uploading approaches that we described in the first section, the process would have taken longer than one hour.
As Jazz and RESTful services become popular in application development, the method that you learned in this article can be valuable not only for testing but also for other stages of the application development lifecycle.
- Read these articles in the Jazz.net library to learn more (requires Jazz.net registration):
- Find out about these Jazz-based products:
- Get A Brief Introduction to the REST specification.
- Learn about other applications in the IBM Rational Software Delivery Platform, including collaboration tools for parallel development and geographically dispersed teams, plus specialized software for architecture management, asset management, change and release management, integrated requirements management, process and portfolio management, and quality management. You can find product manuals, installation guides, and other documentation in the IBM Rational Online Documentation Center.
- Visit the Rational software area on developerWorks for technical resources and best practices for Rational Software Delivery Platform products.
- Explore Rational computer-based, Web-based, and instructor-led online courses. Hone your skills and learn more about Rational tools with these courses, which range from introductory to advanced. The courses on this catalog are available for purchase through computer-based training or Web-based training. Some of the "Getting Started" courses are available free of charge.
- Subscribe to the IBM developerWorks newsletter, a weekly update on the best of developerWorks tutorials, articles, downloads, community activities, webcasts and events.
Get products and technologies
- Download the Jazz Foundation Server and the Server SDK to get started.
- Download trial versions of other IBM Rational software.
- Download IBM product evaluation versions and get your hands on application development tools and middleware products from DB2®, Lotus®, Tivoli®, and WebSphere®.