Manage test data more effectively with Rational Quality Manager 4.0.1
Use new features to organize test data and define and override selection criteria
Tests generally use input parameters of various types, such as valid inputs, invalid inputs, normal case, or edge cases. To check how your system works under different conditions, you need to run tests under different inputs. Testers might also want to check the behavior of the same product from different vendors. In that case, you need a way to execute the same test scripts multiple times with different sets of inputs.
IBM® Rational® Quality Manager introduced test data in version 1.0. However, test teams have complex requirements, such as being able to use only a subset of records, categorizing the test data for easy search ability, and so on. Starting with Rational Quality Manager 4.0.1, you can take advantage of new features of test data, including the ability to:
- Organize test data so that you can find a particular test data while authoring test scripts
- Define selection criteria for test data records
- Associate selection criteria with test scripts
- Override the selection criteria in test cases for increased multiplicity
- Override selection criteria at run time
- Override selection criteria for test data records before execution
This article shows you how you can use these new features to use test data more effectively and efficiently.
To use the new capabilities of test data, you must have Rational Quality Manager 4.0.1 or later. A comma-separated values (CSV) file is required to create test data.
Background of test data
If you have not used Rational Quality Manager's test data capabilities before, this section provides an overview of its capabilities. If you are familiar with its test data capabilities, skip to Organizing Test Data.
Let's get started. To create test data, you need a CSV file.
Listing 1. Sample CSV file
Card Number:Number,Card Type:String,Card Holder Name:STRING 1111666644443333,VISA,Cooper 2222456789021111,Maestro,Hayden 3452678290346712,Masestro,Cooper 5555123767891234,VISA,Hayden 8888987612346789,Maestro,Lee 3214678923460985,VISA,Lee 6723456789012345,Master,Patel 3214234789047689,Amex,Xing 6523198725460215,VISA,Kurtis 3124768903451278,Master,Ayush 2145890724561111,Amex,Patel 1111333343562345,VISA,Xing
The first line of the CSV file represents the field name and its type. For example:
Card Number:Number represents a field whose name is
Card Number and the field type is
Number. The next lines represent values for each
field, separated by comma.
Create new test data
To create test data in Rational Quality Manager, navigate through Construction > Create > Test Data. Under the section Data File, use the Browse button to navigate to the directory where the CSV file exists. Select it to upload to Rational Quality Manager.
The Data Record section shows the contents of the CSV file. For example, the CSV file Card_Details.csv has 20 records, and of those, the initial 10 records are shown on the Rational Quality Manager user interface. Enter the name of the test data as Card Details and click Save. See Figure 1:
Figure 1. Test data creation
Tip: To see all the records of the CSV file in Rational Quality Manager, change the drop-down "Items per page" from 10 to Show All.
Use test data in manual tests
Now that you have the Card Details test data, you can use it when you create a test script.
To do so, navigate through Construction > Create > Test Script. Name the test script Card Validation. For Type, select Manual from the script type drop-down list.
Next, provide the test data you just created by clicking the Edit Test Data button. In the Manual Steps section, click the Insert Test Data Column button to insert a test data column in a particular step of the test script. See Figure 2:
Figure 2. Create test script using test data
Run the manual test with test data
To execute the test, first navigate to Construction > Create > Test Case and create a new test case. Associate the Card Validation test script with the test case and click the Run button in the test case editor.
Next, look at the manual execution page. Notice that Running Test Data Iteration 1 of 20 and the test data column within the script have been replaced by the first record of the test data. Click Pass for each step.
The script steps are repeated with two records from test data. Because the test data has 20 records, this process is repeated 20 times. See Figure 3:
Figure 3. Test case execution having test data
Organize test data
Your organization might need many test data input values. You can organize test data so it is easily searchable.
When you create a manual test script, you can associate test data with the test script. To search efficiently, you must first define categories for the test data and then assign these categories to each test data artifact.
Note: Categorization is a free form of organizing artifacts. You can define multiple categories and category values, depending on your project's needs. You can also organize categories in a hierarchical manner.
Categories can be created in two ways:
- Use the Manage Test Data Categories icon in the test data editor or Test Data table view.
- If you are a project administrator, navigate through Manage Project Properties > Artifact Categories > Test Data.
Tip: You must have permission to create categories.
After you create the categories for the test data, you can assign the categories to each test data artifact. You have two ways to do this:
- Use the test data editor to assign categories to one test data.
- Use the bulk update option from Browse Test Data.
The Browse Test Data table can filter test data based on category, which is helpful when you associate test data with the manual test script. From the manual test script editor, click the Edit Test Data button. All the existing test data in the project area will be listed in tabular format, so you can filter it using categories to find what you want.
To see how this works, look at Figure 4. The test data category is Component, and three values were created for it: Card, Inventory, and Payment. Each test data has one value for the Component category.
The left-hand side of the dialog shows the Filter By Categories option. If you click on any category value, the table lists only those test data that contain that category value. An in-line filter is provided in the table if you have multiple categories. See Figure 4:
Figure 4. Search test data using categories from the manual test script editor
Construct test script with record selection criteria
By organizing your test data, you can search a particular test data that's of interest. But as the number of test data increases, maintaining this organization becomes more difficult. Suppose a project has hundreds of test data, and you decide to change the predefined test inputs? You'd have to change all the test data – the CSV files associated with those test data.
Form master test data
Maintaining and changing test data are time-consuming tasks. To simplify those tasks, you may want to define common test data that can be reused across multiple scripts. You can identify similar kinds of test data artifacts and combine their records to form a master test data that has many records. Having master test data reduces the number of test data artifacts present in the project. If you want to change the test inputs with a new set of values, the changes need be made in only one place – in the master test data.
Record selection criteria
Having master test data simplifies test asset maintenance. But master test data might have hundreds of records, and not all records will be applicable for all test scripts for which this test data is used.
For example, one test script may require only five records and another test script may require five different records from the same test data. So you need a way to associate selected records from the test data with the manual test scripts. With Rational Quality Manager, you can define and associate record selection criteria with the manual test script and test data while authoring the manual test script. By default, all records are selected.
In the manual test script editor, the record selection criteria is shown as Test Data Usage, which is by default "Use all records." You have two buttons to edit Test Data Usage:
- The edit button creates or edits the record selection criteria
- The clear button selects all records from the test data
For an example, see Figure 5:
Figure 5. Test data usage in manual test script editor
If you click the edit button under Test Data Usage, you get the Record Selection Criteria dialog. This dialog has two attributes:
- "Where column name is": This combo box lists all the columns from the test data
- "And value is": Enter a value against the column name in this value text box
Think of this as a filter criteria:
Where column name is == <selected column> AND value == <something you define>
For this example, I chose Card Type as the column name with VISA as the value. Click Show to see the records that match the criteria defined from the test data. If you click Clear, the data selection criteria is reset and shows all records. Once you define a selection criteria, click OK to close the dialog. The criteria will be set. See Figure 6:
Figure 6. Record selection criteria creation in manual test script editor
Note: If you click OK and the record selection criteria doesn't match any record, you'll get a warning: No record found based on the record selection criteria. It should return at least 1 record.
Execute test with record selection criteria
So far you created record selection criteria (Card Type = VISA) and associated that to the test script (Card Validation). Six records in the test data have VISA as the Card Type, whereas the overall test data contained more records. It's time to execute this script by adding it to the test case. To execute the test:
- Open an existing test case. Under the Test Scripts section, click the plus icon to bring up the Test Scripts selection dialog.
- Select the test script you created (Card Validation).
- Save the test case.
- Click Run to execute it. In the manual execution page, notice that Running Test Data Iteration 1 of 6 has only six records, which are filtered by the record selection criteria.
- Click Apply All and select pass as the verdict for all steps.
For an example, see Figure 7.
Figure 7. Test execution with record selection criteria
In the result, 24 steps were generated: the test script had four steps, which were iterated six times based on your selection criteria, although the test data contains 20 records. With each iteration, the test data record values – based on your selection criteria – are substituted in the manual script.
Construct a test case with record selection criteria
By associating record selection criteria with a manual test script, you can reuse the same test data for different manual test scripts. For example, say you want to run a test for valid and invalid inputs. You can create test data that has both valid and invalid inputs, and then duplicate the test script and associate different record selection criteria with each to support the valid and invalid inputs. But if in the future you plan to change the test script, you'll also have to change the duplicated test script. This may not be scalable, and discrepancies can occur and updates missed.
With Rational Quality Manager, you can associate record selection criteria at the test case level. Consequently, you can use the same test script in multiple test cases, and – based on the test case you're executing – the record selection criteria of that test case comes into play.
You can create two test cases and associate the same test script – but different record selection criteria – for each. Create one test case for valid input and another for invalid input. Associate the same test script with both test cases. In the test script section of the valid input test case, associate record selection criteria to filter the valid inputs; in the test script section of the invalid input test case, associate record selection criteria to filter the invalid inputs. You have created two different tests using the same test script and the same test data. By doing this, you can reuse test script and test data efficiently.
Associate record selection criteria at the test case level
To associate record selection criteria at the test case level:
- Go to the Test Scripts section of the test case editor, which shows the test scripts in a tabular format. By default, the column Data Records shows the record selection criteria defined at test script level.
- Note that the test script table has row-level action to define or edit record selection criteria. Click Edit Data Record to bring up a dialog that associates record selection criteria with the test script for the test case. Another test case can associate a different record selection criteria with the same test script.
- Click Reset Data Record to script level to reset the record selection criteria to the test script level.
Record selection criteria that is defined at the test case level has precedence over record selection criteria defined at the test script level. See Figure 8:
Figure 8. Test script section in test case editor
Now change the record selection criteria from Card Type=VISA, which is defined at the test script level, to Card Type=Master. The new record selection criteria filters five records. See Figure 9:
Figure 9. Test case with record selection criteria
When you execute the test case, you'll see:
- Running Test Data Iteration 1 of 5 on the manual execution page instead of Running Test Data Iteration 1 of 6
- The test script steps will be repeated five times, because record selection criteria defined at the test case level has precedence over record selection criteria defined at the test script level
- 20 step results are generated: 4 (number of steps in the test script) * 5 (records filtered by the selection criteria defined as test case level)
For example, see Figure 10:
Figure 10. Test execution with record selection criteria defined at the test case level
Override record selection criteria before execution
Associating record selection criteria with the test case and test script allows you to select a subset of records from test data. Rational Quality Manager provides a way to allow you to select records directly from test data before test execution. This facility is controlled by a precondition defined for the operation Execute Test Case. The precondition name is Allow Test Data Record Selection. This precondition is enabled by default, but project administrators can enable or disable this feature for any role.
Enabling and disabling the advisor
To enable or disable a predefined rule, or advisor, navigate to Manage This Project Area. (You must be a project administrator to do this. You can enable or disable an advisor from the Project Administration page.) You can also reach the project area by navigating to https://<server>:<port>/<contextRoot>/admin and selecting the project area. Then:
- Click Preconditions & Follow-up Actions
- Click the edit button for Execute Test Case action.
- Enable or disable the Allow Test Data Record Selection precondition by clicking the Add or Remove button, respectively.
For an example, see Figure 11.
Figure 11. Precondition to select records before execution
Select records from the test data before executing the test case
Say you want to select records from the test data. Assume that the precondition is enabled and you run the test case, which has a test script that is associated with test data.
First, open a test case and click Run. On the Run Test Case dialog, check the box for Select Records from Test Data. The dialog converts to a Wizard. (If you changed the process settings while you were already logged into the Rational Quality Manager project area, you may need to log out and log in and then try again.)
The buttons Previous, Next, Finish, and Cancel will appear. See Figure 12:
Figure 12. Run Test Case dialog to select records from test data
If you click Next, test data appear in tabular format based on the record selection criteria defined at the test case/test script level. You can decide to execute with the existing selection, or you can use a different filter to select a new set of records.
You can clear the existing record selection criteria to visualize all the records in tabular format and then select a set of records manually. Records that are selected before execution take precedence over the record selection criteria defined at the test case or test script level. If you click Finish, test case execution will start. The test script steps will be repeated based on the number of records that were selected before test execution.
The ability to change the test behavior before test execution by selecting records from test data gives you great flexibility. You also get this functionality while running a test case execution record for Run with Options and Create Result without Execution.
Tip: Teams that follow a stringent process of defining the test data before execution may not require that this process setting be enabled. They may keep this capability disabled.
To see an example of the Run Test Case dialog after you click Next, see Figure 13:
Figure 13. Run Test Case dialog after you click Next
Select records from test data before executing the test suite
You also have the ability to select records from test data before execution when you run the test case as a part of a test suite. To do this:
- Create a test suite by navigating to Construction > Create > Test Suite.
- In the Test Case section of the test suite, click Add.
- Select that test case and save it.
- Run the test suite.
- In the Run Test Suite dialog, click OK. Now you are in the Test Suite Execution Console.
- Find the columns Test Case Progress and Name. In the drop-down menu option, click Select Record And Start Test.
For an example of what this looks like, see Figure 14.
Figure 14. Select records from test data while executing test suite
After choosing Select Record And Start Test, the Select Records from Test Data dialog appears. You can select or deselect any records from the test data. In this example, only two records of five have been selected. See Figure 15:
Figure 15. Record Selection Criteria dialog
Execution is based on the records you selected, and test script steps are repeated based on the number of records you selected before test execution. Because only two records were selected, test scripts steps will be repeated twice.
In this example, 8 step results are generated: 4 (number of steps in the test script) * 2 (records filtered by the selection criteria defined as test case level).
The same functionality is also available from the Execution Console when running a test. See Figure 16:
Figure 16. Execution result containing 8 steps
This article describes how to use and reuse test scripts and test data effectively. If you use a large number of test data, you can organize it into categories, which will help you search for it faster. If you have the same kinds of records available in different test data, combine them into a single set of master test data. Employ record selection criteria to use master test data in different test scripts and test cases. Finally, records selected before execution take precedence over the record selection criteria defined at test case or test script level. Record selection criteria defined at test case level takes precedence over the record selection criteria defined at test script level.
- Explore A complete testing process with Rational Quality Manager to learning the testing process using Rational Quality Manager.
- Explore the Rational Quality Manager Information Center for technical
details, check the Rational Quality Manager Roadmap. For how-to tips, read these developerWorks articles:
- Lab Management using IBM Rational Test Lab Manager: Extending IBM Rational Quality Manager for collaborative, comprehensive test planning and test asset management (developerWorks, October 2008)
- Getting started with IBM Rational Quality Manager (developerWorks, December 2008)
- Managing your first project with IBM Rational Quality Manager (developerWorks, October 2008)
- Download the free trial version of Rational Quality Manager, which includes Rational Test Lab Manager.