The IBM® Rational® ClearQuest® defect- and change-tracking system captures and tracks all types of changes for any type of project. The Rational ClearQuest Web feature provides access to repositories through a web-based interface.
Performance has often been a concern for Rational ClearQuest Web customers. Before deploying a new IBM® Rational® ClearQuest® version, customers will usually test whether the new version meets their performance requirements. If it does not, they try to determine the source of the performance problems and to do the necessary performance tuning.
Rational ClearQuest Web 7.1 architecture has changed substantially with the introduction of Web 2.0 technology and Change Management (CM) Server. Because of these architectural changes, the performance testing and tuning techniques for ClearQuest Web v7.0 (that is, the techniques described in CQWeb: Performance and Tuning) no longer apply to Version 7.1.
This article describes a systematic approach to performance testing and tuning that is suitable for ClearQuest Web 7.1. Most of the content in this article has been validated by using it on a customer site.
What's different in ClearQuest Web 7.1
There are two main differences in ClearQuest Web 7.1 that affect performance testing and tuning methods significantly:
- CM Server. This provides server-side support for Rational ClearQuest and IBM® Rational® ClearCase® WAN interfaces. It replaces the Request Manager in ClearQuest Web v7.0 and has different tuning guidelines from that version. The About Change Management Server topic in the Rational ClearQuest Information Center provides a general introduction to CM Server. A link to the topic is available in the Resources section of this article.
How to design test scenarios
A test scenario must emulate the situations that occur in the actual production environment.
The test scenarios designed by the Rational Performance Engineering team are available in the developerWorks article IBM Rational ClearQuest Web Version 7.1 Performance Reports. Review these performance tests before you start to create your own.
It is essential to understand the application before testing it. Teams often design test scenarios based on their own experience or guess about how others use ClearQuest Web. This approach often does not reflect the real usage model, which you can obtain by analyzing the ClearQuest Web IHS log file,
- In v7.0, using the Microsoft Windows operating system, the file is located at
- In v7.1, the file is located at
The log file contains information such as the IP addresses that have accessed ClearQuest Web, the date and time that ClearQuest Web was accessed, the HTTP access method (for example, GET or POST), and the URL that was accessed.
The file has the following format:
126.96.36.199 - - [14/Jul/2010:14:31:53 +0800] "GET /cqweb/cqhelp.cq?action=welcomepage HTTP/1.1" 200 7604
It is usually safe to assume that each different IP address corresponds to a different end user. Use the Firebug add-on to the Firefox browser to analyze ClearQuest Web HTTP requests and help you map URLs to end-user transactions. A link to download the Firebug add-on is available in the Resources section of this article. To use the Firebug add-on:
- Open Firebug.
- Execute one transaction in ClearQuest Web, watch the corresponding HTTP requests in Firebug, and log the requested URLs.
- Repeat Step 2 until you get the URL information for all of the transactions of interest or for all of the URLs listed in the
- Use this information to map the URLs to the transactions. This is feasible because different transactions usually have different URLs. However, if more than one transaction shares the same URL, group them as one combined transaction.
After you obtain the list of users, access times, and transactions, write a program to analyze it and get enough information to help design test scenarios to gather data for real production uses, such as these examples:
- User load How many users accessed the system within a specified hour? How did the user load change over 24 hours in a specified day?
- TPH (transactions per hour)
How many transactions did one user execute, on average, during a particular time period (e.g., from 9am to 10am on 2010-08-09)? How the TPH changes over time (e.g., from 2010-08-09 9am to 2010-08-10 9am)?
What were the main transactions? What was the percentage of each main transaction? For a combined transaction, after you get its percentage, distribute the percentage to member transactions based on your experience when you design the percentage of test transactions.
Pick the peak time, analyze the corresponding
access.log file, and get this information so that you can design better test scenarios. Repeat this analysis periodically to get a better understanding of your production environment and guide your future testing.
How to create test scripts
After you have designed test scenarios, the next step is to transform them into test scripts. The importance of this step is often underestimated. People assume that a performance testing tool such as Rational Performance Tester can generate a test script accurately. Unfortunately, sometimes that is not true. Use the following guidelines to create an accurate test script:
- Do not record a transaction once. Instead, record each transaction two times and measure the performance separately. For example, for a "Find record by ID" transaction, search record 1, search record 2, make them two transactions, and measure the performance separately. You record transactions twice because of ClearQuest Web rich client, which does caching and prefetching at both client and server side. Therefore, the HTTP requests sent the first time can differ from those sent the second time.
- Become familiar with Firebug and with the HTTP requests and responses for the transactions of interest.
- Examine the test script that is generated by the testing tool. For each transaction, compare the test script with what you actually see in Firebug to determine that they are consistent. In addition, compare Firebug output with your actual perceived performance.
On the other hand, Firebug sometimes indicates detects many ongoing HTTP requests, but the user perceives the performance as good because the requests occur in the background and have no impact on the user experience.
- Test your script to verify that it is correct. Check the error log file
SystemErr.login ClearQuest Web to make sure that there are no error messages. The log file in Windows is by default located at
How to tune performance
After you are satisfied with the test script, the next step is running the performance test. If the result meets your requirements, then congratulations! Otherwise, follow these steps to tune the performance:
- Tune your usage of ClearQuest. ClearQuest is flexible, and has few limitations on what you can do with query or hook code. The downside, of course, is that this flexibility can be abused and result in slow performance.
- Tune CM Server and WebSphere Server.
- Tune the underlying SQL database.
This section covers query optimization and schema optimization. Query optimization concerns how to improve the performance of slow queries. Schema optimization concerns how to improve the performance of record-related operations.
A slow ClearQuest query is generally caused by underlying SQL syntax that is slow. Once you realize this, the optimization becomes straightforward: optimize the SLQ query and then transform between the SQL query syntax and the ClearQuest query syntax. There are many resources on how to do SQL optimization, e.g., SQL Performance Tuning by Peter Gulutzan. A link to this book is available in the Resources section of this article.
To view the underlying SQL statements:
For a ClearQuest query, get the generated SQL in the ClearQuest Eclipse client or ClearQuest client for Windows (note that it's not supported in ClearQuest Web).
- For an Eclipse client, select the query, and from the context menu, select SQL > View SQL.
- For ClearQuest client for Windows, verify that the menu item View > View SQL pane is checked. Then run or edit the query to display the various editors for the query, and click SQL editor.
If the ClearQuest query contains prompts, you must perform a preliminary step to view the statement: Copy the original query to a temporary folder, and replace the dynamic filter with a static filter. Then obtain the SQL statement by using either of the previous method.
After you obtain the SQL statement, you will find that the performance issue is caused by:
- The ClearQuest query itself. In this case, tune the query.
- The ClearQuest query does not have the capability to generate efficient SQL. In this case, transform the query to SQL, as instructed in the help topic Editing a query as SQL text. A link to this topic is available in this article's Resources section.
You usually do not need to optimize the SQL statements. Most of the issues can be resolved by tuning the query filter definition in the ClearQuest client. Slow SQL is usually caused by an inefficient WHERE clause, which corresponds to inefficient query filter definition in ClearQuest. To solve this problem, examine the whole filter tree systematically:
- Examine the filter execute plan.
- Check each query filter, including its field name, operator, and filter value.
- Consider whether the filtering operation can be done efficiently. If not, try to transform the filter to one that equivalent but more efficient.
The following tips can help you avoid some common mistakes.
- Database index is critical to query performance. A query is often slow because it has no index or an inefficient index. Determine whether a field has an index and, if so, whether the query can efficiently use the index.
- ClearQuest creates an index for
REFERENCE_LISTfields, as well as for fields with unique keys.
- ClearQuest does not create an index for
MULTILINE_STRINGfields. Therefore, if a query filter field is one of these types, and the associated table is large, the query can be very slow. To improve the query performance, consider manually creating an index for these fields.
- Use history table fields very sparingly as filters (for example,
history.action_name). This is because the history table is usually quite large, and ClearQuest does not create index for most of the fields. Manually creating an index is not recommended because the history table is frequently updated and the index could hurt update performance. However, ClearQuest does create a combined index for fields
entity_dbidby default in versions later than 188.8.131.52, if your database is created prior to 184.108.40.206, you should follow the guideline in Indexing the History table for ClearQuest to manually create one. A link to this tech note is available in the Resources section of this article.
- Use the filter operator
CONTAINSsparingly, because the CONTAINS operator does not use the index.
For most record-related operations, slow performance is caused by inefficient schema design and hook code.
Some good materials are available about how to design schema for performance, such as the developerWorks article IBM Rational ClearQuest general schema design performance and the help topic Performance considerations for using hooks. A link to the help topic is available in the Resources section of this article. This article will not duplicate the main content but will highlight the following points:
- Limit the use of
- Reduce use of the call
- Use the following format to get a nested field value:
- Do not build a query to get a field choice list. Instead, use the following:
- Use the
REFERENCE_LISTfield sparingly. Each use adds two SQL queries.
- Avoid using Field hooks
- Prudently use back-references
- Make sure that the attachment table has an appropriate index. If your ClearQuest database is created prior to 220.127.116.11, the Attachments table is not indexed by default, you need to follow the guideline in the support document Indexing the Attachments tables for ClearQuest to manually create one. A link to this support document is available in the Resources section of this article.
To increase your understanding about when ClearQuest executes each kind of hook and to help you evaluate the hook code, see the Information center topic Execution order of field and action hooks. A link to this topic is available in the Resources section of this article.
Tuning CM Server and WebSphere Server
The tech note Change Management Server (CM Server) configuration guidelines for ClearQuest Web contains important guidelines on configuring the CM Server for optimal performance. A link to this tech note is available in the Resources section of this article.
Two CM Server parameters in particular,
activeHttpSessionThreshold, affect performance. The maximum number of users supported by a server is the product of the two parameters values. Given a desired number of users that you want the server to support, you can adjust the two parameters to achieve the best response time. When you adjust
serverWorkerThreadCount also, a good starting point is to keep them same, which allows one thread to serve one session when each session has one on-going task.
WebSphere performance tuning is a big topic. Both the WebSphere Application Performance web site, and the developerWorks article Case study: Tuning WebSphere Application Server V7 for performance offer helpful guidelines. A link to the WebSphere website is available in the Resources section of this article. Two parameters, in particular, are useful in practice:
- JVM heap size. It's recommended to increase both the initial and maximum heap size to 1.5GB.
- Thread pool size. In one customer environment, we start with the default maximum value of 200 and do testing by increasing/decreasing 50 each time and finally found the maximum size of 250 for both Default and Web Container thread pool achieves the best performance. But it could be very different for your environment, it's recommended to follow WebSphere performance tuning guideline and do your own performance testing and get the best suitable value for your environment.
Tuning the database
As mentioned before, a database index is critical to performance. ClearQuest creates an index for certain types of fields, which should cover most use cases. However in edge use cases, you might want to manually create indexes to speed up certain operations. Be careful when you manually create indexes. Document every change that you make, and consult IBM support/service.
To analyze database-related issues, enable ClearQuest core tracing so that you can track slow SQL statements. In Windows, set the following registry keys to enable tracing:
[HKEY_USERS\.DEFAULT\Software\Rational Software\ClearQuest\Diagnostic] "Trace"="SQL;DB_CONNECT;" "Report"="MESSAGE_INFO=0x40B;DIAG_REPORT=0x01" "Output"="c:\\trace.txt"
This article introduces a systematic approach to performance testing and tuning the new ClearQuest Web 7.1 architecture. In terms of testing, it introduced two main phases: how to design test scenarios to emulate production usage and how to make sure test scripts reflected both test scenarios and user-perceived performance. In terms of tuning, this article introduces tips for query optimization; schema optimization; and CM Server, WebSphere, and database tuning. Many performance issues are caused by inefficient use of ClearQuest, and thus can be resolved by following best practices, as well as the tuning methods and related resources highlighted in this article. But, like any tuning guidelines, your own environment can differ and you must therefore validate these methods.
- Here are the resources that are mentioned in this article:
- Rational ClearQuest Performance Improvement
- developerWorks articles:
- IBM Rational ClearQuest general schema design performance
- WAN Performance, reliability, and scalability improvement for IBM Rational CM Server for Rational ClearCase Remote Client and ClearQuest Web
- IBM Rational ClearQuest Web Version 7.1 Performance Reports
- Performance Resources for IBM Rational ClearCase and ClearQuest
- Case study: Tuning WebSphere Application Server V7 for performance
- Information center topics:
- WebSphere Performance web site
- ClearQuest tech notes:
- SQL Performance Tuning by Peter Gulutzan
Get products and technologies
- Get the trial download of Rational ClearQuest.
- Get the trial download of Rational Performance Tester.
- Evaluate IBM software in the way that suits you best: Download it for a trial, try it online, use it in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement service-oriented architecture efficiently.
- Join the Rational ClearQuest Forums and Communities to get and give tips to other developers and testers who use ClearQuest. You can subscribe by email, and subscribers can also post by email.
- Get involved in the My developerWorks community. Connect with other developerWorks users while exploring the developer-driven blogs, forums, groups such as the Rational Café, and wikis.