Skip to main content

By clicking Submit, you agree to the developerWorks terms of use.

The first time you sign into developerWorks, a profile is created for you. Select information in your profile (name, country/region, and company) is displayed to the public and will accompany any content you post. You may update your IBM account at any time.

All information submitted is secure.

  • Close [x]

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerworks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

By clicking Submit, you agree to the developerWorks terms of use.

All information submitted is secure.

  • Close [x]

Results of IBM Rational Requirements Composer 2.0 performance and scalability tests

Plus tips for how to get optimal performance

Nam Le (namle@us.ibm.com), Staff Software Engineer, IBM
Nam Le photo
Nam Le is a staff software engineer at IBM in Research Triangle Park, North Carolina, in the U.S. He works on the Rational Requirements Composer Server team with an emphasis on both client and server performance.
Tom Mutdosch (mutdosch@us.ibm.com), Advisory Software Engineer, IBM
Tom Mutdosch photo
Tom Mutdosch is an advisory software engineer at IBM in Research Triangle Park, in North Carolina, in the U.S. He works on the Rational Requirements Composer Server, focusing on querying and searching requirements data.
Devang Parikh (dparikh@us.ibm.com), Software Engineer Manager, IBM
Devang Parikh photo
Devang Parikh is a Software Engineer Manager at IBM in Research Triangle Park, NC. He leads a Requirements Server team who is responsible to build server platform used by Rational Requirements Composer.
Balakrishna Kolla (brkolla@us.ibm.com), Advisory Software Engineer, IBM
Balakrishna Kolla photo
Bala Kolla is an advisory software engineer at IBM, in Research Triangle Park, North Carolina, in the U.S. He works on the Rational Requirements Composer Server team.
Chris McGraw (chris.mcgraw@us.ibm.com), Staff Software Engineer, IBM
Chris McGraw photo
Chris McGraw is a software engineer at IBM Rational in Edinburgh, Scotland, in the U.K. He works on the Rational Requirements Composer Server team.
Mark Goossen, Software Engineer, IBM
Mark Goossen photo
Mark Goossen is a software engineer at IBM, based in Pennsylvania in the U.S. He works on the Rational Requirements Server team, focusing on process and configuration.

Summary:  Learn how IBM® Rational® Requirements Composer, Version 2.0, performs with various 32-bit and 64-bit configurations. This report compares response times for configurations of varying resources and user loads.

Date:  28 Jan 2010
Level:  Intermediate PDF:  A4 and Letter (564KB | 34 pages)Get Adobe® Reader®
Also available in:   Chinese  Portuguese

Activity:  23468 views
Comments:  

What this article covers

Performance and scalability are of paramount importance in any server-based environment. Depending on the complexity of the environment, performance can dramatically impact the effectiveness of software that you have in place. The IBM® Rational® Requirements Composer Version 2.0 server is composed of several pieces that work in conjunction to form the back end of the tool. The V2.0 server components can be set up and configured in various topologies and configurations that affect performance and scalability.

Because we had already gone through the initial release, a large focus of this testing was on performance in the server and its underlying infrastructure. This article describes the server components in detail and explains the various server configurations and topologies that can affect performance. We also describe the techniques that we used to record performance measurements and scalability metrics under different scales and topologies and investigate the results of the tests.

We cover details of our findings through\ comparisons of different topologies, databases, and Web application servers, and we consider the effects of scaling both the number of users and resources on the server. Additionally, we explain how you can recreate our testing setup in your own environment to do your own performance tests. Lastly, we offer tips for performance-tuning and specifications for optimal Rational Requirements Composer server performance.

Disclaimer

The information in this document is distributed AS IS. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer' s ability to evaluate and integrate them into the customer' s operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk. Any pointers in this publication to external Web sites are provided for convenience only and do not in any manner serve as an endorsement of these Web sites. Any performance data contained in this document was determined in a controlled environment, and therefore, the results that may be obtained in other operating environments may vary significantly. Users of this document should verify the applicable data for their specific environment.


Recommendations for optimal performance

Topology

Dual-tier environment: One server for the application server and one server for the database server

Database

IBM® DB2® or Oracle

Application server

IBM® WebSphere® Application Server Version 6.0
Minimum machine specifications

Application server

  • Quad core 2.8 GHz CPU with 8 GB of RAM
  • Disk: High-performance SAS Disk, RAID, SAN, or NAS direct connected disk subsystem

Database server

  • Dual core 2.9 GHz CPU with 4 GB of RAM
  • Disk: High-performance SAS Disk, RAID, SAN, or NAS direct-connected disk subsystem

Summary

The main performance bottleneck is the application server machine for various test runs with varying amounts of load. So it is better to use a dual-tier environment with the better-performing machine hosting the application server. In our tests, IBM DB2 and Oracle databases handled the user loads similarly. Therefore, use whichever database software that you and your customer are most comfortable with using. We also found that the performance of the application servers is similar for various user loads. We prefer the WebSphere server because of the enterprise needs (security and administration) of Rational Requirements Composer administrators, as well as its features for customized performance-tuning.

Note:
We used a plain configuration of IBM WebSphere Application Server without any additional tuning.


Overview of the server components

The Rational Requirements Composer server creates, handles, and stores requirement resources that are used by Rational Requirements Composer clients and provides mechanisms to query and search that data. It also provides specific services for creating, managing, and retrieving folders, tags, comments, links, and project snapshots.

There are several parts that make up the server, each with specific responsibilities. At the core, the Rational Requirements Composer server is an application server that runs a set of applications. There are two main applications: a Requirements Composer server application and an IBM® Jazz® Foundation Server application. There is an additional application that generates Rational Requirements Composer resources into viewable images on the Web.


Figure 1. Rational Requirements Composer overview
Topology of components

Web application server

The Rational Requirements Composer server consists of one or more Java™ Enterprise Edition (JEE)-enabled Web servers. Rational Requirements Composer server supports both IBM WebSphere Application Server and Apache Tomcat. The Jazz technology platform and the Rational Requirements Composer server application are both Web modules that are installed on and run on the configured Web server. The applications can be run on two independent Web servers, depending on topology. The performance of each of these application servers will be covered later in this article.

Jazz and Jazz Foundation Services

The Jazz repository component handles all of the storage of Rational Requirements Composer artifacts and information. Jazz Foundation Services is a set of capabilities used to handle user and project administration, security, collaboration, query, and other generic cross-tool capabilities.

The repository backend is a relational database, which is used to store all of the server resources and artifacts. Jazz Foundation Services can extract various structured data from XML and Resources Description Framework (RDF) documents and then information is indexed, which allows for easy retrieval and querying for these properties. Additionally, Jazz Foundation Services uses a separate text index for full-text searching across resources (through Apache Lucene).

In order to act on or query the data, server side requests are translated into SPARQL Query Language statements and then run on the database.

Rational Requirements Composer server application

In addition to the core foundation capabilities provided by the Jazz technology platform, requirement-specific functions needs to be handled on the server. Jazz Foundation Services provides a mechanism to utilize a fronting application, which is a server-side Web application that contains application and presentation logic and uses Jazz Foundation Services for storage, queries, and security. The Rational Requirements Composer server application is a fronting Web application that runs on a Java EE-compliant server, separate from the Jazz Web application.

In addition to providing a requirement-specific function, the fronting application mechanism enables the Rational Requirements Composer server to enhance performance by using caching strategies specific to the requirements domain. This is in addition to standard caching provided by Jazz Foundation Services.

Rational Requirements Composer Web application

The Rational Requirements Composer server includes the functionality that allows access Rational Requirements Composer from a Web browser. An additional Web application that runs on the server is used for handling the conversion of Rational Requirements Composer resources into thumbnail-sized images.

Database

A relational database is used for persistence of server resources. We performed our testing on two enterprise databases supported by IBM Rational Requirements Composer 2.0: DB2 and Oracle. Our tests focused on enterprise scenarios that involved a large number of users (we did not use the Derby database in our testing). Microsoft® SQL Server was not tested, because it was not part of the supported operating environment at the time that we ran our tests.

Techniques and tools used to measure performance and scalability

Notice that the data discussed in this paper is a result of running simulated users against the Rational Requirements Composer server. Actual client experiences might differ, depending on customer environments and use.

Rational Performance Tester

IBM® Rational® Performance Tester is used to simulate and measure Rational Requirements Composer client/server interactions. The test environment is set up to handle up to 500 virtual users who are interacting with a Rational Requirements Composer server.


Figure 2. Rational Performance Tester environment
Diagram of the topology

Rational Performance Tester host machine

The host machine runs the Rational Performance Tester workbench and is responsible for distributing the user load and gathering the performance data. These are the machine specifications:

  • Lenovo ThinkCentre, Microsoft® Windows Server 2003 Enterprise Edition, SP2
  • Dual CPU 2.66 GHz, 4 GB RAM

The host machine distributes the user load evenly between each of the five Rational test agents.

Rational Performance Tester agent machines

The environment includes five Rational Performance Tester agent machines. Each machine receives requests for a set number of users and sends the requests to the Rational Requirements Composer server. The machine records the response times and results, and then sends the data to the host machine. Each of the agent machines has the following specifications:

  • Lenovo ThinkCentre, Microsoft Windows Server 2003 Enterprise Edition, SP2
  • Dual CPU 2.66 GHz, 4 GB RAM

For the test data, to eliminate the agent machine as a performance bottleneck, each agent machine simulates no more than 100 users.

Test scenario performed

There are two use cases that each user will perform: an Author use case and a Reviewer use case. Each user performs a use case and then waits for 120 seconds before repeating the use case. There was no additional " think time" added between individual operations in the use cases. All actions on resources are randomized. Based on real customer use, our test scenario consists of a ratio of 1:4 Authors to Reviewers. A typical test scenario run for 500 users will produce 705 resources, update 2,287 resources, and create 5,876 comments per hour. This is along with the other server interactions. The average number of requests is around 85,000 requests per hour. Each test runs for 60 minutes after all of the users are in the system.

Author use case

In this use case, the user simulates what a typical Rational Requirements Composer Author would do. The user navigates to find a resource, views the resource, and then decides to either update the resource or create an entirely new resource. Table 1 shows the actions that the user performs.


Table 1. Rational Performance Tester Author script
Description Test script operation Frequency
1. View the folder structure GetAllFolders 100%
2. Run a query for resource URLs in a folder RunFolderQuery 100%
3. Run a query to retrieve data about resource URLs GetMultiDescribe 100%
4. Select a resource and retrieve its contents GetArtifactContents 100%
5. Run a query to retrieve data about the resource FetchArtifactInfo 100%
6. Perform one of the following:
  • Modify a resource and Save
ApplyAttribute 75%
  • Create a new use case, requirement, or sketch part
CreateResource 25%

Reviewer the use case

In this use case, the user simulates what a typical Rational Requirements Composer Review would do. The user navigates to find a resource, views the resource, and then decides to either add a comment to the resource, tag the resource with a relevant tag, or move on to another use case without further action. Table 2 shows the user' s actions.


Table 2. Rational Performance Tester Reviewer script
Description Test script operation Frequency
1. View the folder structure GetAllFolders 100%
2. Run a query for resource URLs in a folder RunFolderQuery 100%
3. Run a query to retrieve data about resource URLs GetMultiDescribe 100%
4. Select a resource and retrieve its contents GetArtifactContents 100%
5. Run a query to retrieve data about the resource FetchArtifactInfo 100%
6. Perform one of the following:
  • Add a comment
CreateComment 50%
  • Tag the resource
TagResource 25%
  • View the resource, but take no further action
25%

Rational Performance Tester performance schedule

The schedule (Figure 3) consists of multiple test cases used to simulate the Author and Reviewer use cases.


Figure 3. Rational Performance Tester schedule
Screen capture of the schedule

Larger view of Figure 3.

Each test case interacts with the server by using the Rational Requirements Composer Samples, which are a collection of Java samples that demonstrate how customers and partners can interact with Rational Requirements Composer resources by using RESTful requests. The Samples are available on Jazz.net. For each server interaction, each user reuses two Apache HttpClient objects to communicate with the server through HTTP.

User ramp time and metrics

For each test run, users are introduced to the system at a normal set interval; this is called the " ramping" phase. This is to ensure that an introduction of a large number of users to a system will not overload the system. The response times gathered during the ramping phase will not be included in the metrics. The metrics will reflect the data gathered when all of the users are in the system.

Failure acceptance

Because the scripts are randomized, the amount of load on the server varies at different times. Under an abnormally large load, the server will return failure codes appropriately. If the failure count is less than 5% of the total number of requests, then the data is considered valid

System resource monitoring

VMware ESX server

On the VMware ESX server, the system resource monitoring is done by using the VMware Infrastructure Client performance tooling, as shown in Figure 4. The tooling allows monitoring of CPU, memory, and disk usage.


Figure 4. VMware Infrastructure Client performance tooling
Screenshot of VMware performance tooling

Larger view of Figure 4.

Windows servers

On Windows servers, the resource monitoring is done by using the Windows performance monitoring tool (perfmon) shown in Figure 5.


Figure 5. Windows performance monitoring tool
Screen capture of performance monitor tool

Test machine specifications

VMware ESX Server, Linux environnent

  1. Application server specifications
    • SuSE Linux® 4.1.2 64-bit VM
    • WebSphere Application Server 7.0.0.3
    • Quad-core Intel Xeon X5365 at 3.0 GHz CPU
    • 8 GB RAM
    • 50 GB SCSI hard disk
  2. Database server specifications
    • SuSE Linux 4.1.2 64-bit VM
    • DB2 Version 9.5
    • Quad-core Intel Xeon X5365 at 3.0 GHz CPU
    • 8 GB RAM.
    • 50 GB SCSI hard disk

* See appendix for ESX server specifications

Windows environment

  1. Application server specifications
    • Windows 2003 Enterprise Server, 32-bit
    • WebSphere Application Server V6.1
    • Intel Core Duo E6750 @ 2.6 GHz CPU
    • 4 GB RAM
    • 232 GB SATA hard disk
  2. Database server specifications
    • Windows 2003 Enterprise Server, 32-bit
    • DB2 Version 9.5.1
    • Oracle 10g
    • Intel® Core Duo E6750, 2.6 GHz CPU
    • 4 GB RAM
    • 232 GB SATA hard disk

Rational Requirements Composer setup

User repositories

To eliminate the variable of remote user authentication, the application servers mentioned in this article are local user repositories (that do not use LDAP). The WebSphere Application Server uses the federated user repository, and the Tomcat server uses the local Tomcat user directory.

Project structure

The project structure described in this article reflects the typical customer use, given that the average user will not have access to all of the resources in the repository. The projects are set up with data projects, which contain about 80% of the total resources in the repository. Each user has access to test projects, which account for about 20% of the resources in the repository.

IBM WebSphere Application Server settings

On 64-bit machines where additional memory is available, the Java™ Virtual Machine (JVM) maximum heap size is increased from the default 1536 MB. This is to ensure maximum efficiency. On 32-bit machines, the default settings are used.

Tomcat application server settings

The default settings are used for the Tomcat application server.

Database optimization

To ensure optimal database performance, it is necessary to ensure that the database is fully optimized. With both DB2 and Oracle, you can run statistics to allow the databases to analyze the shape of a schema and optimize the contents.

Databases generally manage statistics automatically (in a scheduled overnight operation, for example). However, to ensure that the databases were fully optimized during our testing, we made sure to manually run the statistics:


DB2
DB2 REORGCHK UPDATE STATISTICS ON TABLE ALL 


Oracle
EXEC DBMS_STATS.gather_schema_stats(' JAZZDBUSER' ); 


Performance test results

Database vendor comparison

In this trial, we compare the performance of database vendors. The environment is identical; the database server machine has both IBM DB2 and Oracle installed. The database servers are hot or cold, depending on which vendor is being tested. The data is backed up through IBM® Jazz™ as a file system backup and restored on both databases.

The database vendor comparison is performed in the Windows environment with 10,000 resources and 50 and 200 users.

Separate IBM WebSphere Application Server and DB2 database server (Windows)

The Windows 32-bit environment has the IBM application server configured to communicate with the DB2 database server. The average request response time for 200 concurrent users is 1,381 ms. There are natural fluctuations in response times throughout the 60 minute run.


Figure 6. Average request response time for 200 users on DB2 database with 10,000 resources
Graph & table, Rational Performance Tester results

Separate IBM WebSphere Application Server and Oracle database server (Windows)

The Windows 32-bit environment has the IBM WebSphere Application Server configured to communicate with the Oracle database server. The average request response time for 200 concurrent users is 1,321 ms. There are natural fluctuations in response times throughout the 60-minute run.


Figure 7. Average request response time for 200 users on the Oracle database with 10,000 resources
Graph and table, Rational Performance Tester data

Database comparison


Figure 8. Comparison of average response times between database vendors
Graph of database comparisons, 50 and 200 users

There is not much difference between DB2, Oracle, and SQL (supported in V2.0.0.1) databases in request response times (Figure 8). Correlated with the data in Table 4, which indicates that the database server is the least-used component in the system, either database vendor will provide the highest level of performance. So use whichever database you or your team is prefer.

Results for scalability of users and resources

Separate WebSphere Application Server and DB2 database server (Linux)

These test runs show the effect of scaling the number of concurrent users and the number of resources in the repository. The response times and server resources are monitored. The Linux environment is used for these runs, with the combinations of 200, 350, and 500 users and 10,000, 50,000, and 100,000 resources.


Figure 9. Linux environment response times for repository size vs. number of users
Graph of scalability results

Table 3. Average response times for repository size vs. number of users data, Linux
Sizing 200 Users 350 Users 500 Users
10,000 resources 97.2 ms 123.9 ms 236.2 ms
50,000 resources 101.4 ms 150.6 ms 318.5 ms
100,000 resources 115.6 ms 190.2 ms 436.2 ms


Analysis of performance metrics

Comparison of Web application servers

When comparing the Tomcat application server to the IBM WebSphere Application Server, we saw negligible difference in performance.

Note:
We used the default WebSphere Application Server configuration. No additional performance tuning was performed on WebSphere Application Server when running our tests.

The difference in average response times for each user grouping is never greater than 55 ms. For smaller-scaled shops with fewer than 300 users, choose the application server that is available and suits your particular needs.


Figure 10. Application server comparisons
Graph showing average response time to number of users

Number of users

In Figure 10, as the number of users increases, the average response time increases. The comparison of the rate of increase in response times does not show a large difference between Tomcat and IBM WebSphere application servers. When users are scaled up to 300, use the application server that you prefer.

Scalability

Scalability of users

In Figure 9, the introduction of users has a significant impact on response times. The largest degradation in performance occurs when the number of users is increased from 350 to 500. On the 100,000 resource repository, the response times increased by 130% when moving from 350 to 500 users, as compared to 65% from 200 to 350 users. This degradation indicates that, for that particular system, a load of 350 concurrent users is near the optimal load for a repository with 100,000 resources.

Scalability of number of resources

In Figure 9, as more resources are introduced, the response time degradation is minimal when there are 200 users in the system. But when there are 500 users in the system, an increase from 50,000 to 100,000 resources amounts to a 37% increase in response time; whereas, for 200 users, the increase is only 14%. To conclude, for our Linux environment, introduction of more resources has a significant impact on the performance when there are more than 200 users in the system. This increase is due to the fact that data indexing and searching occurs on the application server; therefore, it is affected as the number of resources increases.

Scalability breakdown

Comparing the effects of introducing more users vs. more resources, the introduction of more users had a more profound effect on performance (see Table 3). But if we separate the server interactions between database interactions and query interactions, we can see a much greater effect of increasing the number of users rather than the number of resources.


Figure 11. Sizing impact for repository actions and queries, Linux
Bar chart showing Sizing impact for repository actions and queries

For repository interactions, where the number of rows in the database will differ greatly between 10,000 resources and 100,000 resources, there is a substantial increase in the response time. But for query interactions that involve the Jena query library, the difference is not as apparent.

Summary

For a scalable server with better performance, use the dual-tier deployment, with one server for the application server and another for the database server. For large-scale deployments, use the either NAS or SAN storage solution for better performance under heavy server load.

Sizing

System resource usage

For system resources, we compare the WebSphere Application Server (" WAS" in Table 4, which follows) machine resource use and DB2 database server machine when all of the users are active in the system.


Table 4. Linux environnent resource use
WAS* CPU use % WAS memory use % WAS disk use (KBps) DB2 CPU use % DB2 memory use % DB2 disk use (KBps)
10,000 resources
200 users
18.00 58.6 214.7 1.16 9.67 127.70
10,000 resources
500 users
45.26 61.4 635.4 2.55 11.50 326.570
100,000 resources
200 users
18.90 63.8 827.5 1.73 10.80 302.20
100,000 resources
500 users
49.50 68.0 1795.0 2.89 10.96 435.67

*WAS = WebSphere Application Server

Increasing the number of resources has little effect on the WebSphere Application Server machine' s CPU or memory usage, but does affect the disk usage. For the DB2 database server, increasing the number of resources affects only the disk usage. Increasing the number of users affects both the CPU and the disk usage on the WebSphere Application Server machine but only the disk usage on the DB2 server machine. When considering how to size the Rational Requirements Composer environment properly, the disk usage is a concern for both WebSphere Application Server and DB2 server machines. Specifically, for the WebSphere Application Server machine, take care to ensure that the CPU usage isn' t a bottleneck for performance.

Disk space usage

The disk space usage for the Rational Requirements Composer server is composed of the disk space used for the Rational Requirements Composer server indexed data and the disk space needed for the database instance. By default, the indexed data is located on the same machine as the application server. Therefore, giving consideration to adequate disk space is important.

For example, for our Linux repository loaded with 100,000 Rational Requirements Composer resources, the indexed data takes up 10 GB of disk space. The database instance must be located on a server that also has appropriate disk space. For example, the database that housed 100,000 Rational Requirements Composer resources amounts to approximately 17 GB of disk space.

Note:
In our test, we used simplified resources, and the resources did not have prior revisions persisted, so actual disk space usage varied, based on the contents and the number of revisions of resources.

Network bandwidth use

During testing, we monitored the network use on both the server that was running the IBM WebSphere Application Server software, as well as the DB2 server.

Note:
By default, including all of our testing, all content transferred from the Rational Requirements Composer server to client connections uses HTTP compression, the gzip compression algorithm.

All client connections used HTTPS. Figure 12 shows the network bandwidth use of both the WebSphere Application Server and the DB2 server during our tests, using increasing numbers of resources and users.


Table 5. Linux environnent network use
Resources and users WebSphere network use (KBps) DB2 network use (KBps)
10,000 resources, 200 users 209.6 145.7
10,000 resources, 500 users 564.8 399.4
100,000 resources, 200 users 214.9 149.6
100,000 resources, 500 users 557.2 387.9

From the results, it is observed that the WebSphere Application Server server network usage was more than 40% higher than that of the DB2 server. Given these numbers, consideration should be given when setting up your server environment where network usage is a factor – the WebSphere Application Server will utilize more network bandwidth than the DB2 server.

Hardware sizing

By using the performance test data that we compiled, we created the tables that follow, based on the various hardware and software configurations for optimal deployment of Rational Requirements Composer server. When considering sizing options, Version 2.0 supports both single and multitier configurations. You can choose the lower-cost single-tier configuration with the option of adding a machine (dual-tier configuration) when your team grows.

The three lists that follow show the sizing for different enterprise deployments.
Small-scale enterprise configuration, 10,000 resources and up to 100 users:

  • 2 systems: Quad core CPU 2.4 GHz or higher, 64-bit
  • Memory: 4 GB or higher
  • Operating system: Linux or Windows server
  • Web application server: Tomcat 5.5 or WebSphere Application Server 6.1 or later
  • Database: Oracle 10GR2 or DB2 V9.1 or DB2 V9.5 Fix Pack 4

Medium-scale enterprise configuration, 50,000 resources and up to 250 users:

  • 2 systems: Quad core CPU 2.4 GHz or higher, 64-bit
  • Memory: 8 GB or higher
  • Disk: High-performance SAS disk (15K), RAID
  • Operating system: Linux or Windows server
  • Web application server: Tomcat 5.5 or WebSphere Application Server 6.1 or later
  • Database: Oracle 10GR2 or DB2 V9.1 or DB2 V9.5 Fix Pack 4

Large-scale enterprise configuration, 100,000 resources and up to 500 users:

  • 2 systems: Quad core CPU 2.4 GHz or higher, 64-bit
  • Memory: 8 GB or higher
  • Disk: High-performance SAS disk (15K), RAID, SAN or NAS direct-connected disk subsystem
  • Operating system: Linux or Windows server
  • Web application server: Tomcat 5.5 or WebSphere Application Server 6.1 or later
  • Database: Oracle 10GR2 or DB2 V9.1 or DB2 V9.5 Fix Pack 4

Network connectivity

Choosing network connectivity in dual-tier configurations is to minimize latency between the application server and the database server (no more than 1–2 ms), with the servers located on the same subnet. When using external storage, minimize connectivity latencies (the optimal configuration is through a Fibre Channel).

Disks

Based on the performance test results, we found that an increase in resources along with increase in concurrent users causes an increase in the load on the hard disc. Therefore, for storage for large-scale deployments, consider a SAN or NAS solution with the storage directly connected through a Fibre Channel (FC) connection to avoid the latency between the storage and the server. The benefits of using this configuration will allow offloading all disk I/O from the server to the NAS, while providing a complete fault tolerance disk subsystem with instantaneous snapshot backup capabilities and high availability and disaster recovery capabilities.

Availability

We set up a separate reliability test was to observe the server over an extended period of time. The configuration of the environment for this test was similar to that described previously. The configuration itself is specified in the Appendix: Reliability Test Environment.

The purpose of the reliability test was to determine the performance and stability of the Rational Requirements Composer server over 5–7 days of continual use. This test assessed the performance characteristics of the Rational Requirements Composer server by applying various workloads to the server over a long test period, and it measured the memory consumption and CPU use of the server.

After an initial startup, the memory usage stays roughly within a given band, with increases attributed to load, with subsequent easing off in accordance with load (see Figure 12). The same can be seen with processor utilization (Figure 13). Under load, the processor use can spike but does not stay consistently high.

During the course of the test, the server maintained 100% uptime.


Figure 12. Server Memory Available chart
Graph of server memory over time

Larger view of Figure 12.


Figure 13. Server Processor Time chart
Graph of server processor use over time

Larger view of Figure 13.

Fault tolerance

As previously noted under Failure Acceptance, the tests ran with an allowable threshold of 5% for error codes returned. These errors generally occur only under heavy load where the number of simultaneous users and size of the data store approach the higher limits of the test parameters (500 users, 100 KB resources).

Observation of the server over extended periods of time (with variation in load) shows that the errors that occur under load do not leave any lasting impact or have adverse effects on other requests. This behavior can be attributed to the " stateless" architecture of the Rational Requirements Composer server.

OAuth timeout errors under load

The common cause for errors under heavy load can still be traced to response times. In this case, it is not the response time between the Rational Requirements Composer server and client, but rather the response time between the Rational Requirements Composer server and Jazz Foundation Services.

Each request that the Rational Requirements Composer server makes against Jazz Foundation Services is authenticated by using the OAuth protocol (see Resources for where to find details). There is a period of time between the Rational Requirements Composer server being granted an Access Token and having the request serviced by Jazz Foundation Services. If this period of time is greater than the configured OAuth Access Token Timeout, then the request will fail, because Jazz Foundation Services will reject the Rational Requirements Composer server request when it eventually gets around to servicing it. In these cases, the server logs should report that OAuth Access Tokens are timing out.

Reducing OAuth timeout errors

On a loaded server that is producing OAuth Access Token Timeout errors, you can increase the OAuth Access Timeout by using the Jazz Foundation Services Admin Web UI. To find the setting, select Server > Configuration > Advanced Properties.

Figure 14. Advanced server properties
Advanced Properties selected in menu

The OAuth settings can be found under the OAuthServiceProvider heading, as shown in Figure 15.


Figure 15. OAuth Timeout properties
Segment of table with arrow

Increasing the Access Token timeout allows more requests through and reduces the number of timeout errors, thereby increasing overall reliability. This will, of course, be at the expense of increasing average response times between the Rational Requirements Composer client and server, because requests are given a longer time to complete.

Note:
The required timeout value will vary, depending on server configuration.

Performance tweaks

Modifying frequency of scheduled server tasks

There are several server tasks that are performed according to a schedule. When these tasks occur, they generate heavier load on the server. Depending on the Rational Requirements Composer resource use, the amount of load on the server might be reduced by changing when the scheduled tasks run. You can follow this guidance on the schedule tasks and how to modify them:

  1. Recent feeds tasks

There are four tasks that are updated on a schedule: recent artifacts, recent requirements, recent comments, and tag counts. These tasks run individually for each Rational Requirements Composer project. Depending on how often artifacts, requirements, and comments are created or updated, the performance impact of the tasks can be reduced. Use these guidelines:

    • If a particular type of operation is not performed very often, the frequency can be reduced.
    • If a particular type of operation is performed often, the frequency of the corresponding task can be increased to reduce the amount of work each task has to perform.
    • If a particular type of task is not as important, the frequency can be decreased to reduce the amount of work
    • Staggering the occurrence of the tasks will prevent all of the tasks from being performed at the same time, thus producing a situation where, for a given time period, server requests might be processed slowly

To modify the frequency of the tasks, the fronting.properties file must be modified. The file is located here:
<server installation directory>/conf/rdm/fronting.propreties.

These are the corresponding properties (all property values are in milliseconds, or ms):

Recent comments feed

This feed determines which comments appear in the recent comments feed. It is triggered by the creation or update of comments.

com.ibm.rdm.fronting.server.RDMRecentCommentsUpdate.interval=360000

Value used during testing: 300,000

Recent artifacts feed

This determines which artifacts appear in the recent artifacts feed. It is triggered by the creation or update of any Rational Requirements Composer resources.

com.ibm.rdm.fronting.server.RDMRecentArtifactsUpdate.interval=420000

Value used during testing: 420,000

Recent requirements feed

Determines which artifacts appear in the recent requirement feed. Is triggered by creation or update of Rational Requirements Composer requirement resources

com.ibm.rdm.fronting.server.RDMRecentRequirementsUpdate.interval=300000

Value used during testing: 540,000

Tag counts

Determines the use count of each public tag. Is triggered by the tagging and untagging of resources.

com.ibm.rdm.fronting.server.RDMTagCountUpdate.interval=900000

Value used during testing: 900,000

  1. Jazz indexing persistence interval

This interval is meant to record the progress of the underlying index task. When the progression is saved, requests made during that time will take longer than usual for responses. Thus, by increasing the persistence interval, the amount of time during which there is sluggish behavior will be reduced. The downside is that when the progress is persisted, there will be more data to persist. Therefore, if there are large amounts of resource creation or updates, decrease the persistence interval. If the are repository use is mainly read-only operations, then increase the persistence interval to provide better performance.

The persistent property is modified using the Jazz Admin page and requires Administrator privileges.

Advanced Properties: Persist index progress every N ms

Default value is 60000 ms (1 minute)

Value used during testing: 600,000 ms


Figure 16. Index persistence interval Advanced setting
Screenshot of index persistence interval setting

Acknowledgements

Thanks to Mario Maldari, René Morrow, Jim Yen, Ronald Li, Bhavin Shah, and Li Ping Li on the System Verification Test team for supplying information about reliability tests.


Appendix: Reliability test environment

The Availability section discusses a separate environment that was used to perform the sustained 5–7 day period of testing. This environment is detailed in Figure 17.


Figure 17. Reliability test environment
Topology of reliability test environment

Reliability test: application server

  • XSERIES 3650, Intel® Xeon, CPU 5160 at 3.00 GHz, 8 GB of RAM (using PAE),Microsoft® Windows® Server 2003 Service Pack 2 (32-bit)
  • IBM Rational Requirements Composer Server Version 2.0.0 (20091105 1210)
  • WebSphere Application Server Version 6.1, Fix Pack 25
  • 1024 MB maximum heap per instance (only one instance was installed)
  • IBM DB2 database server (JRSXML index database, runs on the application server)
  • DB2 V9.5, Fix Pack 2a

Reliability test: Rational Requirements Composer Client and Rational Performance Tester

  • Intel Core 2Duo CPU E6750 at 2.66 GHz, 2.98 GB of RAM (using Physical Address Extension, or PAE, Microsoft® Windows® XP (32-bit)
  • IBM Rational Requirements Composer Client Version 2.0 (20091105 1210)
  • IBM Rational Performance Tester Version 8.0.1

ESX server specifications

  • IBM® System x® 3650 (7979MC1)
  • Dual Intel Xeon X5365 (3.0 GHz quad core)
  • 16 GB of RAM
  • Two 15 KB RPM SAS drives in RAID0


Resources

Learn

  • Start here to learn more about Rational Requirements Composer:
  • See the JFSCoreSecurity wiki on Jazz.net for a more detailed treatment of how Fronting Server applications interact with Jazz Foundation Services by using the OAuth protocol.

  • Learn about other applications in the IBM Rational Software Delivery Platform, including collaboration tools for parallel development and geographically dispersed teams, plus specialized software for architecture management, asset management, change and release management, integrated requirements management, process and portfolio management, and quality management. You can find product manuals, installation guides, and other documentation in the IBM Rational Online Documentation Center.

  • Visit the Rational software area on developerWorks for technical resources and best practices for Rational Software Delivery Platform products.

  • Explore Rational computer-based, Web-based, and instructor-led online courses. Hone your skills and learn more about Rational tools with these courses, which range from introductory to advanced. The courses on this catalog are available for purchase through computer-based training or Web-based training. Some of the " Getting Started" courses are available free of charge.

  • Subscribe to the IBM developerWorks newsletter, a weekly update on the best of developerWorks tutorials, articles, downloads, community activities, webcasts and events.

Get products and technologies

Discuss

  • Join the discussion in the Rational Requirements Composer forum about all aspects of requirements definition, including both general, tool-independent concepts and tool-specific information.

  • Check out developerWorks blogs and get involved in the developerWorks community.

About the authors

Nam Le photo

Nam Le is a staff software engineer at IBM in Research Triangle Park, North Carolina, in the U.S. He works on the Rational Requirements Composer Server team with an emphasis on both client and server performance.

Tom Mutdosch photo

Tom Mutdosch is an advisory software engineer at IBM in Research Triangle Park, in North Carolina, in the U.S. He works on the Rational Requirements Composer Server, focusing on querying and searching requirements data.

Devang Parikh photo

Devang Parikh is a Software Engineer Manager at IBM in Research Triangle Park, NC. He leads a Requirements Server team who is responsible to build server platform used by Rational Requirements Composer.

Balakrishna Kolla photo

Bala Kolla is an advisory software engineer at IBM, in Research Triangle Park, North Carolina, in the U.S. He works on the Rational Requirements Composer Server team.

Chris McGraw photo

Chris McGraw is a software engineer at IBM Rational in Edinburgh, Scotland, in the U.K. He works on the Rational Requirements Composer Server team.

Mark Goossen photo

Mark Goossen is a software engineer at IBM, based in Pennsylvania in the U.S. He works on the Rational Requirements Server team, focusing on process and configuration.

Report abuse help

Report abuse

Thank you. This entry has been flagged for moderator attention.


Report abuse help

Report abuse

Report abuse submission failed. Please try again later.


developerWorks: Sign in


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Select information in your profile (name, country/region, and company) is displayed to the public and will accompany any content you post. You may update your IBM account at any time.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


Rate this article

Comments

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational
ArticleID=464165
ArticleTitle=Results of IBM Rational Requirements Composer 2.0 performance and scalability tests
publish-date=01282010
author1-email=namle@us.ibm.com
author1-email-cc=
author2-email=mutdosch@us.ibm.com
author2-email-cc=
author3-email=dparikh@us.ibm.com
author3-email-cc=
author4-email=brkolla@us.ibm.com
author4-email-cc=
author5-email=chris.mcgraw@us.ibm.com
author5-email-cc=
author6-email=markgoossen@us.ibm.com
author6-email-cc=