Editor's Notes: This article was written using IBM Rational Performance Tester 6.1 using Microsoft Windows XP Pro SP2 and the BookPool.com website as it existed on February 24, 2005. Please note that as you walk through this article, the BookPool.com website may appear differently because they are constantly updating content. In addition, this is a very basic walkthrough of script creation with no modifications suitable for testing Web applications where data is only being read from the system. Tests that involve data input require additional steps, such as datapooling, not covered in this article.
Would you like to know, before deployment, how your application and its operating environment will perform when many people try to use it at the same time? Do you wonder what kinds of response times users will experience in a real-world situation? Or do you just want to ensure that your code meets performance expectations? Performance testing will give you the answers you seek. This article shows you how to do some basic performance testing using IBM® Rational® Performance Tester 6.1. Specifically, it covers how to do load testing -- measuring an application and its operating environment against specific response criteria.
The ultimate goal of performance testing and monitoring is to create an excellent end-user experience. To reach this goal you need to plan and design tests, record some test scripts, create automated or manual performance test suites, execute these test suites against various configurations, and then review and analyze the reports. This article is intended for the novice and covers the topic from start to finish. If you are already familiar with performance testing and need more in-depth information, take a look at the "User Experience, Not Metrics" series and the "Beyond Performance Testing" series by Scott Barber.
Planning and designing your performance tests
Planning your performance test is a complex task during which you establish a test plan that answers these questions:
- What types of performance tests do I need to run to ensure the agreed-upon level of quality?
- What performance benchmarks are we interested in and how will I measure whether they have been met?
These questions are the basis of performance test planning. The first question directly addresses the quality of the product, while the second question addresses how to know when you have met the quality guidelines and how to report your results.
Your performance test plan should evolve slowly. Start with some simple objectives. Then, as your application grows and you become more comfortable with your performance-testing tools, grow your suite of tests and extend your view into the application-under-test. The article, "End-to-End Testing of IT Architecture and Applications," will help you learn the process of performance test planning.
After you have established your benchmarks and decided on the types of performance tests you need, the next step is to design your tests. You must identify the user navigation paths to test, how to implement these scenarios, what setup is involved in testing, and what the acceptance criteria are. Realize that designing performance tests is more of an art form than designing functional tests. Again, the two series mentioned previously by Scott Barber illustrate how to design excellent performance tests.
Creating a performance test project
The first step is to create a project within IBM Rational Performance Tester. A performance test project is a Java™ project created with a source folder, which serves as a container for performance tests.
To create a new performance test project:
- Select File > New > Performance Test Project. This opens the New Performance Test dialog box.
Note: If the New menu does not display the Performance Test Project option, you are not in the Test perspective. You can switch to the Test perspective or you can select File > New > Other > Test > Performance Test Project. However you select this option, it switches to the Test perspective.
- In the Project Name field, type a name for the project. In this article, we create a performance test for the BookPool.com website named "BookPool."
Figure 1. New Performance Test Project dialog box
Warning: Do not put the BookPool.com website under any serious load (keeping it under ten users is safe). BookPool.com may think you are attempting a denial of service attack. If you have any concerns, you can use one of your company websites, if one is available.
- Click Finish. The project is created, you are switched to the Test perspective, and you are asked whether you want to record a performance test now.
Creating a performance test script
When you start the recorder, a test creation wizard combines into a single flow these actions: the recording of a session with a Web application, the generation of a test from the recording, and the opening of the test in the test editor. You can record tests from Internet Explorer (the default on Microsoft® Windows®, which is automatically started and configured) or from another browser (such as Mozilla, Netscape, or another browser).
- At the Create Performance Test Prompt, click Yes. This starts the test creation wizard.
In the HTTP Proxy Recorder window, enter the name for the script. In this test, we perform a basic search and add a book to the shopping cart. I chose a name that reflects that basic flow.
Figure 2. HTTP Proxy Recorder
- Click Finish.
This starts the Recorder, which may take a couple of minutes.
Figure 3. Starting the Recorder
If you switch back to Performance Tester, you can see that the Recorder Control is logging the actions you take while recording.
Figure 4. Performance Tester Recorder Control
After the Recorder is started, Performance Tester opens the browser to a ReadMe page that outlines some common practices for performance testing. (Note: Depending on your configuration, it may also just load an "about:blank" page.)
Figure 5. Performance Tester ReadMe page
This is the official launch pad for your test. In the Address Bar, enter the URL www.BookPool.com and start recording tests.
In the search dialog, enter
Software Testingand click Search.
Figure 6. Search dialog box
When the results page loads, click the Add to Basket button for the first book returned. When I created this test, that book was How to Break Software: A Practical Guide to Testing (an excellent book).
Figure 7. Example of first book returned
After the Shopping Cart page loads, close the browser. Closing the browser signals to Performance Tester to stop recording. Performance Tester then generates your script and opens it in the Test editor.
Figure 8. Performance Test - Test Editor
The test editor lists by title a test's HTTP pages that, when expanded, show the request and response data in each page. You can use the editor to inspect or customize a test that was automatically generated from a recorded session, or if you are brave you can code a test from scratch.
The test editor pane has two main areas. The area on the left, Test Contents, displays the test's HTTP page hierarchy. The area on the right, Test Element Details, displays details about the currently selected item (test, page, page request, page response, or connection) in the test hierarchy. When you expand a test page, you see a list of the page's requests, in separate folders, whose names are the full URL request string minus the initial http://.
Figure 9. Expanded test page
The primary request, listed in bold, is the one from which the page title came. If the primary request does not return a page title, the test generator creates a unique name for it from the first node of the URL.
When you expand a request, you see the request's Response data and possibly Connection data.
Figure 10. Request response and connection data
The Response data inside each request shows the data that the Web server returned to the browser based on that request. Collectively, the requests listed inside a page are responsible for everything that was returned by the Web server for that page. The Connection data contains information about the connection to the Web server, such as SSL, authentication, or proxy options.
Schedules let you group and order tests, and run tests at a remote location. A schedule can be as simple as one virtual user running one test, or as complicated as hundreds of virtual users in different groups, each running different tests at different times. With a schedule, you can:
- Group tests to emulate the actions of different users.
- Set the order in which tests run: sequentially, randomly, or in a weighted order.
- Set the number of times each test runs.
- Run tests at a certain rate.
- Run one test, or a number of tests, at a remote location.
After you create a schedule that describes the behavior for your system, you can run this schedule using successive builds of the application-under-test or using an ever-increasing number of virtual users.
To create a schedule:
- Right-click your project, and then click New > Performance Schedule.
In the Performance Schedule wizard, enter the name of the schedule, and then click Finish.
Figure 11. Performance Schedule Wizard
A new schedule appears with one user group.
Figure 12. New Performance Schedule
User groups let you group tests in a logical order by using various characteristics representing various types of users on your system. Figure 12 shows one group that contains 100% of the users during execution. So with what we have right now, this would be 100% of the users searching, adding an item to their cart, changing their mind at the last minute, and leaving to check the price on another website. For a more realistic distribution on users, you might make groups for browsers, buyers, merchants, and users who are checking order status. You would then create scripts to represent each of these actions.
- Right-click the user group, and then click Add > Test.
In the Select Performance Tests dialog box, expand the project name to display SearchAndAddToCart.
Figure 13. Select Performance Test dialog box
Select the test and click OK. The test appears in the schedule.
Figure 14. Test added to schedule
Next, we need to alternate the start times for the users. Select ScheduleOne, the root node for the schedule.
In the Schedule Element Details section, under User Load, check the box Add a delay between starting each user.
Figure 15. User Load dialog box
In Delay, enter 100 milliseconds. Also, note that the Number of users is set to five. See the warning on the DOS attack in the Editors Note before you increase the number of users.
Our schedule is now ready to run.
Running a schedule and reviewing and analyzing the reports
Hard part's over! To run the schedule, do the following:
- In the Test Navigator, select ScheduleOne.
- Right-click the schedule, and then click Run > Run Performance Schedule.
This launches your test and report generation. One of the nicer features of this tool is that reporting takes place real-time. Eight reports run by default. We will take a brief look at each of them.
Figure 16. Overall report
The Overall report provides information on the progress of the state of the run and a bar chart showing the overall success of the run. By default, you should always see the percent page element status code success and the percent page status code success for the entire run. Optionally, if the run contains page title verification points or if the run contains response code or response size verification points, you will also see those percentages that passed.
Figure 17. Summary report
The Summary report is very WYSIWYG. It shows the data that is most significant to the test run and lets you analyze a test's final or intermediate results at a glance. In this report, you will find information such as run status, duration, active/completed/total users, max/min/average response times, total hits, and percent verification points passed.
Figure 18. Page Performance report
The Page Performance report shows the average response of the slowest 10 pages in the test as the test progresses. It lets you to evaluate the system's response for the test duration, as well as after a test. This tab has two sections: a graph and a table.
The graph shows the average response time of the 10 slowest pages. Each bar represents a page that you visited during recording. As you run the test, this graph changes, because the 10 slowest pages are updated dynamically during the run. After the run, the tab shows the 10 slowest pages for the entire run.
The table underneath the graph lists each page in the graph. The table supplements the graph by listing:
- The minimum response time for each page in the run.
- The average response time for each page in the run; that is, the same information that is graphed.
- The maximum response time for each page in the run.
- The number of attempts per second to access each page. The attempt is counted whether or not it is successful.
- The number of total attempts to access the page. The attempt is counted whether or not it is successful.
Figure 19. Response vs. Time Summary report
The Response vs. Time Summary report lets you see the average response trend as graphed through time. It contains two line graphs with corresponding summary tables:
- The line graph on the left shows the average response time for all pages during the run. Each point on the graph is an average of what has occurred during that interval. The table under the graph shows one number: the total average response time for all pages in the run.
- The line graph on the right shows the average response time for all page elements during the run. Each point on the graph is an average of what has occurred during that interval. The table under the graph shows one number: the total average response time for all page elements in the run. You set the Statistics sample interval in the schedule, as a schedule property.
Figure 20. Response vs. Time Detail report
The Response vs. Time Detail report lets you see the response trend as graphed through time. A separate line represents each page. The line graph shows the average response of each page through time. The table under the graph contains the following supplementary information: the minimum, average, and maximum page response times for the run, the rate of page attempts per interval for the most recent sample interval, and the number of page attempts per interval.
Figure 21. Page Throughput report
The Page Throughput report gives you an overview of the frequency of requests being transferred per interval.
- The line graph at the left shows the page attempt rate and page hit rate per interval for all pages. A page attempt means that the primary request was sent; it does not include requests within the page. A hit means that the server received the request and returned any response. The summary table under the graph shows the total hit rates and counts for each page in the run.
- The line graph at the right shows active and completed users over the course of a run. The summary table under the graph shows the most recent sample interval. You set the Statistics sample interval in the schedule, as a schedule property.
Figure 22. Server Health Summary report
The Server Health Summary report lets you see an overall indication of how well the server is responding to the load. The bar chart on the left shows the total page attempts, page hits, and status code successes for the run. The table under the chart shows the same information. The bar chart on the right shows the total page element attempts, page hits, and status code successes for the run. The table under the chart shows the same information.
Figure 23. Server Health Detail report
The Server Health Detail report provides specific details for the 10 least successful pages. The summary table under the chart shows, for all pages, the number of attempts, hits, and successes in the run, as well as the attempts per second during the run.
The reports are stored under Performance Test Runs, which is in the pane under the Test Navigator.
Figure 24. Performance Test Runs pane
From the Test Navigator, you can access any past runs, open existing reports, or drill down for more information.
This article presents a beginner's look at load testing with IBM Rational Performance Tester. You can do other kinds of performance testing with the tool as well, but this should get you started. When you get comfortable with the basics of performance testing, you can start mixing in some of the other highlighted features. As a general rule, if you investigate one new feature every time you open the tool, it should never get too overwhelming.
The online help for the tool is outstanding, and there are also good resources on IBM developerWorks related to performance testing. I would recommend looking at some advanced articles or just browsing the Performance and VU Testing forum. You will find much of the content is geared towards IBM Rational Robot, but the concepts are still valid. Good luck!
- Participate in the discussion forum.
- For next steps in performance testing, see Scott Barber's series "User Experience, Not Metrics" and "Beyond Performance Testing."
- For help in determining what to performance test, see "End-to-End Testing of IT Architecture and Applications" by Jeff Bocarsly, Jonathan Harris, and Bill Hayduk and "User Community Modeling Language (UCML) for performance test workloads" by Scott Barber.
- Find more resources about IBM Rational Performance Tester.
- Get involved in the developerWorks community by participating in
developerWorks
blogs.
- Browse for books on these and other technical topics.
Michael Kelly is currently a senior consultant for Fusion Alliance in Indianapolis. He's had experience managing a software automation testing team and has been working with the Rational tools since 1999. His primary areas of interest are software development life cycles, software test automation, and project management. Mike can be reached by e-mail.
Table of contents
Next steps from IBM
My developerWorks community
Tags
Use the slider bar to see more or fewer tags.
Popular tags shows the top tags for this particular content zone (for example, Java technology, Linux, WebSphere).
My tags shows your tags for this particular content zone (for example, Java technology, Linux, WebSphere).
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
