WAN Performance, reliability, and scalability improvement for IBM Rational CM Server for Rational ClearCase Remote Client and ClearQuest Web

A comparison of the 7.1.0.0 and 7.1.0.2 releases

This article discusses the benefits and the performance, reliability, and scalability gains from changes made to IBM® Rational® ClearCase® and Rational® ClearQuest® CM Server in Version 7.1.0.2.

Yuhong Yin (yyin@us.ibm.com), Software Engineer, IBM Corporation

Yin Yuhong photoYuhong Yin is a senior software engineer and project lead in the Rational Change and Configuration Management development organization. She joined Rational software in 2000. Before working on her current project of the Rational ClearCase and ClearQuest Change Management Server, she worked on ClearCase Unified Change Management (UCM), ClearCase and ClearQuest UCM integration, the Rational Green Thread Initiative, and other projects.



Knut Radloff (knut_radloff@us.ibm.com), Software Engineer, IBM Corporation

Knut Radloff photoKnut has been with IBM since 2000 and with the Rational Change and Configuration Management organization since 2007. He has been working on performance, reliability, and platform porting aspects of CM Server.



David Li (dqli@us.ibm.com), Software Engineer, IBM Corporation

David Li photoDavid is a software performance engineer working in the Rational Change and Configuration Management product group. He joined Rational software in 2000 and specializes in software load, stress, and performance testing.



Grant Covell (gcovell@us.ibm.com), Performance Engineer, IBM Corporation

Grant Covell photoGrant has been the team lead for ClearQuest Web performance and reliability testing for six years.



26 October 2009

Also available in Chinese

Introduction

As demands for productivity in the use of software have increased, the ability to follow processes efficiently has become increasingly important. The efficient performance of software processes enhances product quality and stakeholder satisfaction.

This article discusses the benefits and the performance, reliability, and scalability gains from changes made to IBM® Rational® ClearCase® and Rational® ClearQuest® Change Management (CM) Server in the V7.1.0.2 release. Information provided shows the advantages of migration or upgrade to 7.1.0.2 for Rational ClearCase Remote Client and ClearQuest Web from earlier releases.


Fixes and features in Rational ClearCase and ClearQuest 7.1.0.2

  • Added ClearCase Remote Client/CM Server load balancing support with IBM HTTP server

    In addition to the CM Server load balancing with IBM® WebSphere® Application Server Network Deployment Edge Component in Version 7.1, ClearCase Remote Client and CM Server load balancing can be accomplished with the IBM® HTTP Server that ships with the Rational ClearCase and ClearQuest software.

    For more information about configuring ClearCase Remote Client and CM Server load balancing his with the HTTP server for ClearCase Remote Client, see IBM Technote 1377474.

  • Implemented compression of downloaded cleartext

    Loading a ClearCase Remote Client view or updating a ClearCase Remote Client view is one of the most commonly operated ClearCase Remote Client operations. Time spent for this type of operations includes computing what is to be loaded and transmitting the files. The time for transmitting is much longer than computing the contents. Compressing the downloaded cleartext resulted in an average of 30% increase in speed for ClearCase Remote Client view load or update over a WAN connection. One thing to note is that the size of the download is the major factor in the speedup. So the larger the download, the faster the speed.

  • Forced a synchronous data transfer in CM Server

    The use of asynchronous data transfer requires an excessive number of buffers to send data over a TCP/IP connection. This mainly occurs when files are being transferred, but it also occurs with normal responses from IBM® WebSphere® Application Server where the CM Server resides. In the 7.1.0.2 release, CM Server forced the synchronous data transfer by setting the WAS attribute: com.ibm.ws.webcontainer.channelwritetype=sync.=

    This solved the excessive native memory use in the WebSphere Application-Change Manager Server, which was a problem observed with CM Server 7.1.0.0 and 7.1.0.1 releases.

  • Optimized some heavily used CM Server code paths to improve performance, reliability and to fix memory leaks

    CM Server architecture contains multiple layers, as seen in a typical Java™ 2 Platform, Enterprise Edition (J2EE) application, such as servlet, Enterprise JavaBeans™, Java™ Connector architecture (J2C) connection, and so on. The layered implementation, however, presents problems such as code complexity, data accumulation, and difficulty of debugging. In the 7.1.0.2 release, we removed one layer from the call stack, which resolved a JVM heap OutOfMemory issue discovered in long run or heavily loaded test scenarios and resulted in great improvement in performance, reliability, and transaction throughput.

  • Fixed memory leak problem and deadlock issue in CM API code and in ClearQuest Web code

    A ClearQuest Web 5-user as fast as possible test showed a deadlock situation in the CM API code. This issue was addressed by tightening the scope of some synchronized method calls.

    The ClearQuest Web 130-user test showed memory leaks in CM API code and in ClearQuest Web code, which have been fixed.

  • Fixed a problem of excessive CCRPC process creation during high concurrent load test scenario.

    There was a flaw in the CM Server lifecycle management algorithm that could result in excessive ClearCase RPC (CCRPC) processes and OutOfMemory errors in a CM Server during a multi-user, high-load ClearCase Remote Client test scenario. This algorithm was improved to take into consideration all known CCRPC processes when determining if the system has capacity to allocate a new one. If the system resource limit or the capacity MBean attributes are reached, CM Server issues a "Server is busy" error. Wait for a few minutes and retry the operation.

  • Improved the ClearCase Remote Client checkin and AddToSourceControl (mkelem) performance by increasing the data transfer chuck size

    For ClearCase Remote Client checkin and AddToSourceControl (mkelem) operations, there are multiple threads involved in streaming the data from the request socket to the back end. They are all activated once for each "chuck." By increasing the data transfer chuck size (from 1000 bytes to 32K), and thus reducing the context switching, performance was greatly improved.

  • Delivery of CM Server Administration GUI

    CM Server Admin GUI is a browser-based tool for monitoring and managing CM Server behavior and for reporting active ClearCase Remote Client users and ClearCase Web active users. This tool can be used to easily change CM Server MBean attributes without any scripting.

    For more information about CM Server Administration GUI, see IBM Technote 1377925.

These changes, collectively, improved the CM Server performance, reliability, and scalability, and enhanced the usability. See Comparison of sample test data for the ClearCase Remote Client, Versions 7.1.0.0 and 7.1.0.2, and Sample test data for ClearQuest Web, comparison between 7.0.1 and 7.1.0.2, for detailed test reports of the 7.1.0.2 release.


New test cases added in the 7.1.0.2 release

Since Version 7.1.0.0, the Rational Performance Engineering Team and Rational System Verification Team have added additional test configurations and the following test cases:

  • High transaction rate CM Server stress test

    This test simulation tested users working as fast as machine speed will allow to test the CM Server.

  • High user load concurrent load view tests

    Load view is one of the most demanding ClearCase Remote Client operations. We have run up to 100 concurrent load view operations, loading directories of varying sizes from the server.

  • IHS load balance tests on supported CM Server platforms

    IHS load balance support was added in V7.1.0.2. We have run ClearCase Remote Client load balance tests on almost all supported OS platforms and validated this improved load balance solution.

  • ClearQuest Web 75-, 150-, and 225-user scalability tests on the Windows platform
    Rational Performance Engineering ran a scalability test to demonstrate the 7.1.0.2 release's improved scalability and performance by using a schema of medium complexity on the Microsoft® Windows® operating system (OS).
  • Rational ClearQuest Web 130-user, 3-day test on the Linux platform.

    Our System Verification Test team conducted a 130-user, 3-day, 15 transaction per hour test of Rational ClearQuest Web and CM Server against a populated application lifecycle management (ALM) schema database on the Linux® platform.

  • ClearCase Remote Client 200-user, 3-day test run on the Linux platform
    System Verification Test team conducted a 200-user, 3-day, 15 transactions per hour ClearCase Remote Client test against CM Server on RedHat Linux 5. Tests were executed by using the CM API for ClearCase.
  • CM Server Administration GUI load test
    Rational Performance Engineering and System Verification teams ran load tests to demonstrate that constant use of the CM Server Administration GUI has no adverse effect on CM Server performance.

Comparison of sample test data for the ClearCase Remote Client, Versions 7.1.0.0 and 7.1.0.2

Table 1. Hardware configuration for the tests
TestConfiguration
Test driverIBM x336 with 2 Intel Xeon 3.4 GHz CPUs, 4 GB RAM,
OS: RHEL 4
CM ServerIBM x366 with 4 Intel Xeon 3.67 GHz CPUs, 8 GB RAM,
OS: RHEL 5.3
VOB serverIBM x366 with 4 Intel Xeon 3.67 GHz CPUs, 8 GB RAM,
OS: RHEL 5.3

The directory size in the LoadView operation was 22 MB. The size of the file in MakeFile, Checkin, and Checkout operations was 100 KB. A FreeBSD DummyNET WAN simulator was between the IBM® Rational® Performance Tester driver machine and the CM Server. We simulated 100 ms round-trip latency and 3 Mbps network bandwidth for these tests.

The graph in Figure 1 shows ClearCase Remote Client CM API performance improvements in 7.1.0.2 in single user benchmark tests.

Figure 1. Single user CM API benchmark
comparing single-user CM API response time

Larger view of Figure 1

In multiuser tests, the combined impact of aforementioned performance improvements in 7.1.0.2 is much more visible than the single user benchmark tests. In the 25 user tests, there was 400% performance improvement for load view operation, and 10% to 50% performance improvements for other operations, 7.1.0.2 over 7.1.0.0.

Figure 2. The 25 user CM API benchmark
comparing 25-user CM API response times

Larger view of Figure 2


Sample test data for ClearQuest Web, comparison between 7.0.1 and 7.1.0.2

Table 2. Hardware configuration for the tests
TestConfiguration
Test driverLenovo ThinkCentre with two dual 3 GHz Intel® Pentium® 4 CPUs, 2.5 GB memory
Operating system: Microsoft Windows XP Professional, Version 2002 Service Pack 2
Rational ClearQuest Web and CM ServerIBM® IntelliStation® Z Pro with two dual 2.40 GHz Intel® Xeon® CPUs, 4 GB memory, Physical Address Extension enabled
Operating system: Microsoft Windows 2003 Server, Enterprise Edition with Service Pack 2
Database serverIBM® xSeries® 345 with two dual 3.06 GHz Intel Xeon CPUs, 8 GB memory, Physical Address Extension enabled
Operating system: Microsoft Windows 2003 Server, Enterprise Edition with Service Pack 2
Database server: Microsoft SQL Server 2005 Service Pack 3

The graph in Figure 3 shows ClearQuest Web single-user performance improvements between 7.0.1.0 and 7.1.0.2:

Figure 3. Rational ClearQuest Web 7.1.0.2 compared to ClearQuest Web 7.0.1 single-user benchmark
comparing ClearQuest Web single-user response time

The graph in Figure 4 shows ClearQuest Web scalability for the 7.1.0.2 release.

Figure 4. ClearQuest Web 7.1.0.2 scalability
response times of ClearQuest Web server under load

Fixes and features planned for the next patch releases

This section lists the performance-, reliability-, and scalability-related work that we are working on now. No feature or fix is guaranteed.

  • Better cleartext upload compression to further speed up ClearCase Remote Client view load and update
  • ClearCase Remote Client client-side code optimization
  • Improve ClearCase Remote Client data transfer by increasing the size of file being uploaded. Preliminary test showed significant reduction in time for uploading large files and good increase in network utilization.
  • Efficient data fetching for ClearQuest record fields that contain large number of references.
  • Fast ClearCase Remote Client discordance detection.
  • Continue to simplify CM Server call stack .
  • Increase reliability by guarding against memory corruption in CM Server CQ Web backend
  • Fast record form loading for ClearQuest Web.
  • Performance improvement to ClearCase Remote Client UCM activity and bound CQ record in CQ-enabled UCM environment by efficient look up and caching.
  • Performance improvement to ClearCase Remote Client pending change view. Additional options in Pending Changes view configuration to scope by components (for UCM) or directory structure (for base CC).
  • Additional scalability improvements.

In addition, the next release of Rational ClearCase and ClearQuest CM Server will include FixPack 25. The Rational Performance Engineering and Rational System Verification teams have tested WebSphere Application Server FixPack 25. Various tests have shown great improvement in performance, reliability, and transaction throughput with FixPack 25. An IBM Technote on how to upgrade your CM Server WebSphere Application Server installation to FixPack 25 has been published. See Resources for a list and links.


Summary

IBM continues to work diligently to improve Rational CM Server, ClearCase Remote Client, and ClearQuest Web performance. As we have stated in this report, there have been dramatic improvements made to CM Server since the 7.1 release. These improvements make ClearCase Remote Client and ClearQuest Web perform better and make the server more reliable and scalable.

Data and facts provided by:

David Qiang Li, Grant Covell, Paul Weiss
IBM Rational Performance Engineering

Mark Duquette, Denice Ortege, Joey Zhou, Lily Shi
IBM Rational System Verification Team

Development work by:

Kevin Esler, Jon Freyberger, Steve Rehrauer, Rob Title, Yuhong Yin, Doug Mckenzie, Masa Koinuma, Terry Sun, Shaila Golikeri, Daniel Wang, Leon Liang, Jack Ma, Knut Radloff, Herb Miller, Matt Lennon, Adina Adler, Rob Stanzel, Mike Prentice, Mike Mole, Liz Bonesteel, Jim Wu, Howie Bernstein, Eleonora Ludin, Hans Heilman
IBM Rational CM Server Team, ClearQuest Web Team, and ClearCase Remote Client Team

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational
ArticleID=439818
ArticleTitle=WAN Performance, reliability, and scalability improvement for IBM Rational CM Server for Rational ClearCase Remote Client and ClearQuest Web
publish-date=10262009