Advanced Rational Performance Tester reports

5 ways to get deeper insight into performance test results

By default, IBM® Rational® Performance Tester provides essential performance metrics, such as throughput, response times, concurrency, and success rate. However, it also includes several advanced features for detailed analysis, many of which are not commonly used. Proper use of these options provides deeper insight when analyzing test results. This article gives five tips for using some of these advanced features, all of which have helped tremendously in real-world performance testing projects with large companies.

Jigar N Kapasi (kjigar@in.ibm.com), Senior Software Architect, IBM India

Photo of Jigar KapasiJigar Kapasi is a software performance architect in the IBM India Software Lab, working with the HiPODS team. He works with large companies to lead critical performance engineering projects, which include architecture and design recommendations, performance testing, infrastructure and middleware analysis and tuning. He has been with IBM for 12 years and holds a bachelor’s degree in Information Technology from Bangalore University.



Priyanka Arora (prarora803@gmail.com), Performance Analyst, I.B.M.

Photo of Priyanka AroraPriyanka Arora is a Solution Performance Analyst. She has more than four years of experience, with strong technical analysis, tuning, troubleshooting, tools, and load test creation and execution of end-to-end performance engagements for several large telecom and public sector customers of IBM in India and in the Unites States. She earned an MSc Tech degree in Information Systems from Birla Institute of Technology and Science in India and is a certified WebSphere Application Server and Portal Server Administrator.



01 March 2011

Also available in Chinese

This article provides five tips that a performance test engineer can use to gain deeper insight into performance results:

  1. Handling spikes in results
  2. Generating and analyzing percentile reports
  3. Creating and using business transactions
  4. Determining volumes during a subset of a run
  5. Correlating client-side results with server activities

1. Handling spikes in performance test results

In an IBM® Rational® Performance Tester report, the Response vs Time Detail tab plots response time for each page. Extraordinarily high response times, or spikes (as shown in Figure 1), can be a concern if they occur frequently. Analyzing the raw response time data can give useful information about application performance.

Figure 1. Spikes in page response times
Graph of instances of very high response times

To determine the percentage of such outliers, you need the raw data for the test run that Rational Performance Tester uses to plot the performance report graphs. These are the steps to get this data:

  1. Right-click any tab of the report, and select Export Report Counters to CSV.
  2. Browse to a location to save the output, and name the file.
  3. For Export Type, select Full to get all counters.
  4. Check Export overall results and Split output if columns exceed250 (250 is the maximum column limit for a Microsoft Excel file).
  5. In the location that you just specified, multiple .csv files will get created. Browse the files for the one with this column name: All_Hosts Pages Response Time Page_Name Average interval.
    Note:Page_Name here will be replaced by the page titles in your test schedule, and there will be a separate column for each page in the test schedule.

Look for data on pages that show spikes in the graph. This data can now be used to determine the percentage of outliers for each page and troubleshoot potential bottlenecks.


2. Generating and analyzing percentile reports

Percentile response times are often better indicators of the application performance than average response times. Understanding how to generate and analyze performance reports will help you understand system behavior more accurately.

Generate percentile reports

After a test is finished and the performance report has been generated, you can generate a percentile report.

  1. From the Performance Test Runs window, under the report, right-click All Hosts.
  2. Select HTTP Reports > Percentile Report, as shown in the drop-down menu selections in Figure 2, to open the percentile report with default analysis targets.
Figure 2. Generating percentile report
Three menu selections as described

By default, a percentile report shows the 85th, 90th, and 95th percentiles. You can define a new value or modify the default percentile analysis target.

  1. Select Window > Preferences.
  2. Expand Test and click Percentile Analysis Targets.
  3. Either click Add and enter a required percentile value, as shown in Figure 3, or select an existing target and click Edit.
  4. Click OK, and then click Apply to implement your changes.
Figure 3. Custom percentile analysis targets
Defining percentile analysis target in preferences
  1. Regenerate the percentile report to see your custom percentile target values.

Analyze percentile reports

Theoretically, if the nth percentile of a set equals x, it means that n% values in the given set are below the value of x. So, applying that to percentile response time reports in Rational Performance Tester, if the 85th percentile response time for a page is 5 seconds, it means that 85% of virtual users got a response time of 5 seconds or better.

Percentile response times are more relevant than the average times when it is important that each user of the system gets acceptable response times consistently. The more stringent a customer is on each user's experience, the higher the percentile target to be considered. As the number of spikes in the response time set increases, the probability of getting a high percentile response time value for that page also increases.

Also see the Handling spikes in performance test results section.


3. Creating and using business transactions

Customers often characterize non-functional requirements in terms of business transactions. A business transaction typically spans multiple pages; whereas, by default, Rational Performance Tester reports metrics on a page basis. Generating performance results represented in terms of business transactions makes it easier for business users to benefit from what the reports reveal.

For business transaction-based metric reports, define the business transactions first, and then define the required performance metrics first. Then you can generate the transaction reports.

Define business transactions

A typical business transaction is representative of a use case consisting of several pages (navigating through required pages to raise a service request, for example). After you have identified a business transaction, you can define it by following these steps:

  1. Record the scenario in Rational Performance Tester.
  2. Select all sequential pages in the script that form the transaction.
  3. Right-click and select Insert > Transaction.
  4. In response to the question in the pop-up window that asks if you would like to move the selected objects into a new transaction, click Yes.
  5. Give the transaction a meaningful name, and then Save the test.

Enable performance requirements

A performance requirement specifies acceptable thresholds of performance and helps validate service level agreements. You can define performance requirements on several test elements.

Follow these steps to enable performance requirements on business transactions:

  1. Select the transaction in the test contents.
  2. Under Test Element Details, click the Advanced tab.
  3. Click the check box for Enable Performance Requirements.
  4. Now define all of the transaction performance requirements that will determine the transaction status (Pass or Fail) of your test. For example, if it is a requirement that the transaction must complete in not more than 120 seconds:
    1. Select the Maximum Transaction Execution Time [ms] performance requirement.
    2. Choose the <= operator (less than or equal to operator).
    3. Enter the value in milliseconds. In this case, enter 120000, as shown in Figure 4.
  5. Ensure that the page think times and pace times in this transaction align with these requirements. Thus, the sum of all think times, pace times, and expected page response times should fit within this limit.
Figure 4. Business transaction performance requirements
Selections for Test Contents, Test Element Details

Larger view of Figure 4.

Generate transaction performance reports

If you have transactions defined in your test suite, then the Summary tab of the performance report details a section on transaction performance metrics, such as average execution time, maximum or minimum execution time, and so on, as Figure 5 shows.

Figure 5. Business Transaction Summary report
Table with transaction execution time and volumes

To get a detailed report for each business transaction, generate the Transaction Report and the Transaction Percentile Report:

  1. In the Performance Test Runs window, right-click the test run.
  2. Select either Display Transaction Report or Display Transaction Percentile Report.
  3. The Transaction Report gives the transaction throughput, transaction count for the run, and a plot of transactions at run time. The Transaction Percentile Report shows execution (run) times by transaction at the defined percentile analysis targets. See the Generating and analyzing percentile reports section for more details.

4. Determining volumes for a specified subset duration

Some performance tests have non-functional requirements for volumes of business transactions achieved over a specified duration in a steady state. By default, a Rational Performance Tester report gives the total number of transactions achieved across the total duration of the run. These results include statistics gathered during the ramp-up phase, where the page hits are much lower. Getting performance metrics during the steady state gives you a more realistic measure of how a system will perform.

You can use the steady state as the time range for your report. Alternatively, you can also define your own time range in the test run by specifying start and end times. By doing this, you will then see metrics in the report that pertain only to that time range.

Follow these steps to define your own time range:

  1. Click the Server Health Detail tab on the report. The Hits -- Count [for run] column lists volumes against each page or transaction.
  2. Right-click the report and select Change Time Range.
  3. In the Select Time Range view (Figure 6), click the New Time Range button to define a new range.
  4. You then have two choices:
    • Either click Set to Steady State
    • Or enter values of your choice for Start Time and End Time from within the test run duration
Figure 6. Changing the time range for the report
Description, Start Times, End Times columns
  1. Finish and wait for the report to generate with the new time range.

The report will then show statistics only from samples in the range that you specified.

(Also see the Creating and using business transactions section.)


5. Correlating reports with server activities and logs

When monitoring or analyzing performance reports, it is very useful to correlate the reported performance with activities on the server side (through logs, traces, for instance). By default, the Rational Performance Tester performance report shows time across all tabs in seconds (starting at zero). Matching the time scale in the report with that of the server helps correlate performance results with server events and aids troubleshooting.

It is easier to correlate client-side performance metrics from Rational Performance Tester with the server-side log activities if the test report shows absolute times rather than relative times starting from zero.

To customize the time, open a performance report and complete these tasks:

  1. Go to the Response Vs Time Detail tab on the report.
  2. Right-click and select Customize.
  3. For the X Axis field, click the arrow to open the drop-down menu, and select Date & Time (Absolute) to change the template, as shown in Figure 7.
  4. Click OK and let the report refresh to reflect the change.
Figure 7. Customizing time on reports
Setting X axis template to Date and Time in report

Important:
To ensure precise correlation, synchronize the system time on the Rational Performance Tester host controller, agent, and all servers under test.

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational
ArticleID=629505
ArticleTitle=Advanced Rational Performance Tester reports
publish-date=03012011