Updated guidelines for performance testing and tuning with Rational ClearQuest Web 7.1

IBM® Rational® ClearQuest® architecture changed substantially in Version 7.1. This article updates performance testing and tuning guidelines and also introduces a systematic approach for designing test scenarios, creating test scripts, and tuning performance at various layers.

Jun Chang Ma (majunc@cn.ibm.com), Advisory Software Engineer, IBM

Dr. Jun Chang Ma is an advisory software engineer for the IBM China Software Development Lab. He joined IBM in April 2007 and has worked on ClearQuest Web development since then. He has a PhD degree in computer science. Dr. Jun Chang Ma works on ClearQuest Web development. He is an Advisory Software Engineer at the IBM China Software Development Lab.



11 January 2011

Also available in Chinese

Introduction

The IBM® Rational® ClearQuest® defect- and change-tracking system captures and tracks all types of changes for any type of project. The Rational ClearQuest Web feature provides access to repositories through a web-based interface.

Performance has often been a concern for Rational ClearQuest Web customers. Before deploying a new IBM® Rational® ClearQuest® version, customers will usually test whether the new version meets their performance requirements. If it does not, they try to determine the source of the performance problems and to do the necessary performance tuning.

Rational ClearQuest Web 7.1 architecture has changed substantially with the introduction of Web 2.0 technology and Change Management (CM) Server. Because of these architectural changes, the performance testing and tuning techniques for ClearQuest Web v7.0 (that is, the techniques described in CQWeb: Performance and Tuning) no longer apply to Version 7.1.

This article describes a systematic approach to performance testing and tuning that is suitable for ClearQuest Web 7.1. Most of the content in this article has been validated by using it on a customer site.


What's different in ClearQuest Web 7.1

There are two main differences in ClearQuest Web 7.1 that affect performance testing and tuning methods significantly:

  • Rich client. The web client in the previous version is a dumb terminal that simply renders the HTML sent from the server. However, the client in v7.1 is a Web 2.0 rich client that stores state and data in browser memory and communicates with the server in JavaScript Notation (JSON) format. This smart client caches reusable data, asynchronously prefetches data that will be needed, defers initialization until an object is needed (also called "lazy loading"), and so forth. Because of these features, the performance that is measured by a testing tool can differ from the performance that is perceived by the user.
  • CM Server. This provides server-side support for Rational ClearQuest and IBM® Rational® ClearCase® WAN interfaces. It replaces the Request Manager in ClearQuest Web v7.0 and has different tuning guidelines from that version. The About Change Management Server topic in the Rational ClearQuest Information Center provides a general introduction to CM Server. A link to the topic is available in the Resources section of this article.

How to design test scenarios

A test scenario must emulate the situations that occur in the actual production environment.

The test scenarios designed by the Rational Performance Engineering team are available in the developerWorks article IBM Rational ClearQuest Web Version 7.1 Performance Reports. Review these performance tests before you start to create your own.

It is essential to understand the application before testing it. Teams often design test scenarios based on their own experience or guess about how others use ClearQuest Web. This approach often does not reflect the real usage model, which you can obtain by analyzing the ClearQuest Web IHS log file, access.log.

  • In v7.0, using the Microsoft Windows operating system, the file is located at C:\Program Files\Rational\Common\rwp\logs
  • In v7.1, the file is located at C:\Program Files\IBM\RationalSDLC\common\IHS\logs.

The log file contains information such as the IP addresses that have accessed ClearQuest Web, the date and time that ClearQuest Web was accessed, the HTTP access method (for example, GET or POST), and the URL that was accessed.

The file has the following format:

9.51.12.24 - - [14/Jul/2010:14:31:53 +0800] 
"GET /cqweb/cqhelp.cq?action=welcomepage HTTP/1.1" 200 7604

It is usually safe to assume that each different IP address corresponds to a different end user. Use the Firebug add-on to the Firefox browser to analyze ClearQuest Web HTTP requests and help you map URLs to end-user transactions. A link to download the Firebug add-on is available in the Resources section of this article. To use the Firebug add-on:

  1. Open Firebug.
  2. Execute one transaction in ClearQuest Web, watch the corresponding HTTP requests in Firebug, and log the requested URLs.
  3. Repeat Step 2 until you get the URL information for all of the transactions of interest or for all of the URLs listed in the access.log file.
  4. Use this information to map the URLs to the transactions. This is feasible because different transactions usually have different URLs. However, if more than one transaction shares the same URL, group them as one combined transaction.

After you obtain the list of users, access times, and transactions, write a program to analyze it and get enough information to help design test scenarios to gather data for real production uses, such as these examples:

  • User load How many users accessed the system within a specified hour? How did the user load change over 24 hours in a specified day?
  • TPH (transactions per hour)
    How many transactions did one user execute, on average, during a particular time period (e.g., from 9am to 10am on 2010-08-09)? How the TPH changes over time (e.g., from 2010-08-09 9am to 2010-08-10 9am)?
  • Transactions
    What were the main transactions? What was the percentage of each main transaction? For a combined transaction, after you get its percentage, distribute the percentage to member transactions based on your experience when you design the percentage of test transactions.

Pick the peak time, analyze the corresponding access.log file, and get this information so that you can design better test scenarios. Repeat this analysis periodically to get a better understanding of your production environment and guide your future testing.


How to create test scripts

After you have designed test scenarios, the next step is to transform them into test scripts. The importance of this step is often underestimated. People assume that a performance testing tool such as Rational Performance Tester can generate a test script accurately. Unfortunately, sometimes that is not true. Use the following guidelines to create an accurate test script:

  • Do not record a transaction once. Instead, record each transaction two times and measure the performance separately. For example, for a "Find record by ID" transaction, search record 1, search record 2, make them two transactions, and measure the performance separately. You record transactions twice because of ClearQuest Web rich client, which does caching and prefetching at both client and server side. Therefore, the HTTP requests sent the first time can differ from those sent the second time.
  • Become familiar with Firebug and with the HTTP requests and responses for the transactions of interest.
  • Examine the test script that is generated by the testing tool. For each transaction, compare the test script with what you actually see in Firebug to determine that they are consistent. In addition, compare Firebug output with your actual perceived performance.

Occasionally, HTTP requests themselves do not take much time, but the user perceives the performance as poor because additional time is spent on the browser itself or on JavaScript.

On the other hand, Firebug sometimes indicates detects many ongoing HTTP requests, but the user perceives the performance as good because the requests occur in the background and have no impact on the user experience.

  • Test your script to verify that it is correct. Check the error log file SystemErr.log in ClearQuest Web to make sure that there are no error messages. The log file in Windows is by default located at C:\Program Files\IBM\RationalSDLC\common\CM\profiles\cmprofile\logs\server1\

How to tune performance

After you are satisfied with the test script, the next step is running the performance test. If the result meets your requirements, then congratulations! Otherwise, follow these steps to tune the performance:

  1. Tune your usage of ClearQuest. ClearQuest is flexible, and has few limitations on what you can do with query or hook code. The downside, of course, is that this flexibility can be abused and result in slow performance.
  2. Tune CM Server and WebSphere Server.
  3. Tune the underlying SQL database.

Tuning ClearQuest

This section covers query optimization and schema optimization. Query optimization concerns how to improve the performance of slow queries. Schema optimization concerns how to improve the performance of record-related operations.

Query optimization

A slow ClearQuest query is generally caused by underlying SQL syntax that is slow. Once you realize this, the optimization becomes straightforward: optimize the SLQ query and then transform between the SQL query syntax and the ClearQuest query syntax. There are many resources on how to do SQL optimization, e.g., SQL Performance Tuning by Peter Gulutzan. A link to this book is available in the Resources section of this article.

To view the underlying SQL statements:

For a ClearQuest query, get the generated SQL in the ClearQuest Eclipse client or ClearQuest client for Windows (note that it's not supported in ClearQuest Web).

  • For an Eclipse client, select the query, and from the context menu, select SQL > View SQL.
  • For ClearQuest client for Windows, verify that the menu item View > View SQL pane is checked. Then run or edit the query to display the various editors for the query, and click SQL editor.

If the ClearQuest query contains prompts, you must perform a preliminary step to view the statement: Copy the original query to a temporary folder, and replace the dynamic filter with a static filter. Then obtain the SQL statement by using either of the previous method.

After you obtain the SQL statement, you will find that the performance issue is caused by:

  • The ClearQuest query itself. In this case, tune the query.
  • The ClearQuest query does not have the capability to generate efficient SQL. In this case, transform the query to SQL, as instructed in the help topic Editing a query as SQL text. A link to this topic is available in this article's Resources section.

You usually do not need to optimize the SQL statements. Most of the issues can be resolved by tuning the query filter definition in the ClearQuest client. Slow SQL is usually caused by an inefficient WHERE clause, which corresponds to inefficient query filter definition in ClearQuest. To solve this problem, examine the whole filter tree systematically:

  1. Examine the filter execute plan.
  2. Check each query filter, including its field name, operator, and filter value.
  3. Consider whether the filtering operation can be done efficiently. If not, try to transform the filter to one that equivalent but more efficient.

The following tips can help you avoid some common mistakes.

  • Database index is critical to query performance. A query is often slow because it has no index or an inefficient index. Determine whether a field has an index and, if so, whether the query can efficiently use the index.
  • ClearQuest creates an index for ID, DBID, STATE, REFERENCE, and REFERENCE_LIST fields, as well as for fields with unique keys.
  • ClearQuest does not create an index for INT, DATE_TIME, SHORT_STRING, or MULTILINE_STRING fields. Therefore, if a query filter field is one of these types, and the associated table is large, the query can be very slow. To improve the query performance, consider manually creating an index for these fields.
  • Use history table fields very sparingly as filters (for example, history.action_name). This is because the history table is usually quite large, and ClearQuest does not create index for most of the fields. Manually creating an index is not recommended because the history table is frequently updated and the index could hurt update performance. However, ClearQuest does create a combined index for fields entitydef_id and entity_dbid by default in versions later than 7.0.0.0, if your database is created prior to 7.0.0.0, you should follow the guideline in Indexing the History table for ClearQuest to manually create one. A link to this tech note is available in the Resources section of this article.
  • Use the filter operator CONTAINS sparingly, because the CONTAINS operator does not use the index.

Schema optimization

For most record-related operations, slow performance is caused by inefficient schema design and hook code.

Some good materials are available about how to design schema for performance, such as the developerWorks article IBM Rational ClearQuest general schema design performance and the help topic Performance considerations for using hooks. A link to the help topic is available in the Resources section of this article. This article will not duplicate the main content but will highlight the following points:

  • Limit the use of Recalculate choicelist
  • Reduce use of the call $session->GetEntity
  • Use the following format to get a nested field value: $entity->GetFieldStringValue("project.leader.full_name")
  • Do not build a query to get a field choice list. Instead, use the following: $entity->GetFieldChoiceList("Owner")
  • Use the REFERENCE_LIST field sparingly. Each use adds two SQL queries.
  • Avoid using Field hooks
  • Prudently use back-references
  • Make sure that the attachment table has an appropriate index. If your ClearQuest database is created prior to 7.0.0.0, the Attachments table is not indexed by default, you need to follow the guideline in the support document Indexing the Attachments tables for ClearQuest to manually create one. A link to this support document is available in the Resources section of this article.

To increase your understanding about when ClearQuest executes each kind of hook and to help you evaluate the hook code, see the Information center topic Execution order of field and action hooks. A link to this topic is available in the Resources section of this article.

Tuning CM Server and WebSphere Server

The tech note Change Management Server (CM Server) configuration guidelines for ClearQuest Web contains important guidelines on configuring the CM Server for optimal performance. A link to this tech note is available in the Resources section of this article.

Two CM Server parameters in particular, maximumActiveServers and activeHttpSessionThreshold, affect performance. The maximum number of users supported by a server is the product of the two parameters values. Given a desired number of users that you want the server to support, you can adjust the two parameters to achieve the best response time. When you adjust activeHttpSessionThreshold, adjust serverWorkerThreadCount also, a good starting point is to keep them same, which allows one thread to serve one session when each session has one on-going task.

WebSphere performance tuning is a big topic. Both the WebSphere Application Performance web site, and the developerWorks article Case study: Tuning WebSphere Application Server V7 for performance offer helpful guidelines. A link to the WebSphere website is available in the Resources section of this article. Two parameters, in particular, are useful in practice:

  • JVM heap size. It's recommended to increase both the initial and maximum heap size to 1.5GB.
  • Thread pool size. In one customer environment, we start with the default maximum value of 200 and do testing by increasing/decreasing 50 each time and finally found the maximum size of 250 for both Default and Web Container thread pool achieves the best performance. But it could be very different for your environment, it's recommended to follow WebSphere performance tuning guideline and do your own performance testing and get the best suitable value for your environment.

Tuning the database

As mentioned before, a database index is critical to performance. ClearQuest creates an index for certain types of fields, which should cover most use cases. However in edge use cases, you might want to manually create indexes to speed up certain operations. Be careful when you manually create indexes. Document every change that you make, and consult IBM support/service.

To analyze database-related issues, enable ClearQuest core tracing so that you can track slow SQL statements. In Windows, set the following registry keys to enable tracing:

[HKEY_USERS\.DEFAULT\Software\Rational Software\ClearQuest\Diagnostic]
"Trace"="SQL;DB_CONNECT;"
"Report"="MESSAGE_INFO=0x40B;DIAG_REPORT=0x01"
"Output"="c:\\trace.txt"

Summary

This article introduces a systematic approach to performance testing and tuning the new ClearQuest Web 7.1 architecture. In terms of testing, it introduced two main phases: how to design test scenarios to emulate production usage and how to make sure test scripts reflected both test scenarios and user-perceived performance. In terms of tuning, this article introduces tips for query optimization; schema optimization; and CM Server, WebSphere, and database tuning. Many performance issues are caused by inefficient use of ClearQuest, and thus can be resolved by following best practices, as well as the tuning methods and related resources highlighted in this article. But, like any tuning guidelines, your own environment can differ and you must therefore validate these methods.

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational
ArticleID=607254
ArticleTitle=Updated guidelines for performance testing and tuning with Rational ClearQuest Web 7.1
publish-date=01112011