Scalability of Rational System Architect 11.4

This article reports test results that demonstrate the scalability of IBM® Rational® System Architect when deployed in a Windows Terminal Server environment. It also serves as a guide to help predict hardware needs as the number of concurrent users is increased.

Share:

Kevin P. Calandrella (kevin.patrick@us.ibm.com), Manager, Enterprise Architecture Product Development, I.B.M.

Kevin Calandrella has been a member of the Rational System Architect development team since 1997. His background includes application deployment, database performance optimization, automated testing, and user interface design.



Ryan R. Schmitz (schmitry@us.ibm.com), Software Engineer, Rational System Architect, I.B.M.

Ryan Schmitz has been a member of the Rational System Architect quality assurance and development team since 2005. His background includes automated and manual testing, hardware and system environment management, and test design and analysis.



10 March 2011

Also available in Chinese

Introduction

This article reports test results that demonstrate the scalability of IBM® Rational® System Architect when deployed in a Windows Terminal Server environment.

Rational System Architect is a multiuser repository-based enterprise architecture modeling tool. Business analysts use it to visualize, analyze, and communicate all aspects of an enterprise architecture by applying industry standard frameworks, notations, and methods. All project information is stored in a multiuser repository called an encyclopedia, which is hosted on a Microsoft SQL Server or an Oracle database server.

Clarification

Throughout this article, we refer to the Windows Terminal Server used to host the Rational System Architect application as the application server.

Terminal Services is Microsoft's implementation of thin client terminal server computing, where Windows applications such as Rational System Architect are made accessible to a remote client machine. Users are able to access applications deployed on a terminal server from Windows 7, Windows Vista, Windows XP, or any other computer that has Microsoft's remote desktop connection software installed. When using terminal services, only the user interface of an application is presented on the client machine. All user input is redirected over the network to the terminal server, where all application execution takes place.

Terminal Services gives IT administrators the flexibility to centrally deploy applications to users, regardless of their locations. It helps reduce the costs and challenges of maintaining desktop machines with applications that are frequently updated, hard to install, or need to be accessed over low-bandwidth connections. Terminal Services provides for a secure environment by integrating with Windows authentication systems to prevent unauthorized users from accessing the applications or data, as well as allowing administrators to encrypt the data transmitted between the terminal server and client machines.


Objectives

Objective 1: Determine how many concurrent users can function effectively when using Rational System Architect in a given hardware configuration.

Objective 2: Develop a guide to help predict hardware needs as the number of concurrent Rational System Architect users is increased.


Performance metrics

To evaluate the scalability of Rational System Architect, the performance of the application and database servers need to be considered, as does the responsiveness of performing typical end user actions.

Server performance metrics

The following performance metrics are used to evaluate the responsive of both the application and database servers under a given load. These metrics are collected by the Performance Monitor utility included with Microsoft Windows Server 2003. A sampling rate of 15 seconds is used.

CPU

Counter used: \\Server\Processor(_Total)\%Processor Time

Description: % Processor Time is the percentage of elapsed time that the processor spends to execute a non-idle thread. It is calculated by measuring the duration the idle thread is active in the sample interval, and subtracting that time from interval duration. (Each processor has an idle thread that consumes cycles when no other threads are ready to run). This counter is the primary indicator of processor activity, and displays the average percentage of busy time observed during the sample interval. It is calculated by monitoring the time that the service is inactive and subtracting that value from 100%.

Memory

Counters used: \\Server\Memory\Committed Bytes, \\Server\Memory\%Committed Bytes In Use

Description: Committed Bytes is the amount of committed virtual memory, in bytes. Committed memory is the physical memory which has space reserved on the disk paging files. There can be one or more paging files on each physical drive. This counter displays the last observed value only; it is not an average. % Committed Bytes In Use is the ratio of Memory\\Committed Bytes to the Memory\\Commit Limit.

Disk

Counter used: \\Server\PhysicalDisk(_Total)\% Disk Time

Description: % Disk Time is the percentage of elapsed time that the selected disk drive was busy servicing read or write requests.

Network adapter

Counter used: \\Server\Network Interface(*)\Bytes Total/sec

Description: Bytes Total/sec is the rate at which bytes are sent and received over each network adapter, including framing characters. Network Interface\\Bytes Received/sec is a sum of Network Interface\\Bytes Received/sec and Network Interface\\Bytes Sent/sec.

Rational System Architect performance metrics

The following performance metrics are used to evaluate the responsive of Rational System Architect under a given load. These metrics are collected by the Loadgen engine, which is included. (See the Test Methodology section for additional information on Loadgen.)

Diagram Open Average
Average time in seconds to open diagrams.

Diagram Close Average
Average time in seconds to close and save diagrams.

Definition Open Average
Average time in seconds to open definitions.

Definition Close Average
Average time in seconds to close and save definitions.

Definition Add Average
Average time in seconds to create new definitions.

Definition Delete Average
Average time in seconds to delete existing definitions.

Explorer Refresh Average
Average time in seconds to refresh the explorer window.

Explorer Open Average
Average time in seconds to expand a random set of tree nodes in the explorer window.

Average Operation Time
Average time in seconds to perform all operations (an aggregate of the above metrics).

Average Transaction Time
Average time in seconds for database transactions.

Average Lock Time
Average time in seconds for database applocks. An applock is used to lock the database for exclusive access by a single user. In general, longer applocks mean that Rational System Architect is less responsive to the end user.

These averages are calculated for all users participating in the test. For example, if 20 users opened 50 diagrams each during the test, then the Diagram Open Average is calculated from all 1000 diagram opening actions performed.


Test environment

A baseline test environment was chosen to reflect the hardware and software that we concluded would be a common arrangement for the end user. An application server was set up to host Rational System Architect and to be used as the access point for users connecting via remote desktops. A separate database server was set up to host the SQL Server and the Rational System Architect encyclopedia used during the test. Details of the hardware and software configuration used are listed below.

Application server

Hardware:

  • CPU: Intel Xeon Dual Core 2.4 GHZ
  • RAM: 4 GB
  • Network adapter: 1 Gbps connection with measured throughput of 144 Mbps to the database server

Software:

  • Windows Server 2003 R2 SP2 (with Terminal Services enabled)
  • Rational System Architect Version 11.4

Database server

Hardware:

  • CPU: Intel Xeon 2.4 GHZ
  • RAM: 4 GB
  • Network adapter: 1 Gbps connection with measured throughput of 136Mbps to the application server

Software:

  • Windows Server 2003 R2 SP2
  • Microsoft SQL Server 2008

Test encyclopedia

Object count:

  • Diagrams: 211
  • Symbols: 10,893
  • Definitions: 123,741
  • Relationships: 1,017,246

Configuration:

  • Encyclopedia type: Professional
  • Workspaces: Disabled
  • Frameworks : None

Primary Methods:

  • Data modeling
  • BPMN (Business Process Modeling Notation)
Figure 1. Overview of typical Rational System Architect deployment
Diagram shows remote users connecting to server

Test methodology

Rational System Architect Loadgen

Loadgen is a proprietary tool built into Rational System Architect for the purpose of user simulation. Loadgen is capable of performing many common tasks, such as opening and closing diagrams and definitions or creating new definitions, without the need for user interaction. It also provides detailed reports on all actions and the time that it takes to complete them.

The tasks that Loadgen performs are based on a configuration file that identifies the number of operations and the "randomness" of those operations. The configuration used for this study is described here:

Diagram settings

  • Maximum number of diagrams open at once: 2
  • Time for diagram to be open, min/max: 60 sec / 240 sec
  • Time between diagram close and the next open, min/max: 30 sec / 90 sec
  • Number of symbols to change while diagram is open, min/max: 10 / 20

Definition settings

  • Maximum number of definitions open at once: 4
  • Time for definition to be open, min/max: 30 sec / 90 sec
  • Time between definition close and the next open, min/max: 60 sec / 120 sec
  • Percent of definitions that get saved: 75%
  • Percent of definitions that get cancelled: 25%

Add Definition settings

  • Maximum number of definitions to add at once: 1
  • Time between definition adds, min/max: 30 sec / 90 sec

Delete Definition settings

  • Maximum number of definitions to delete at once: 1
  • Time between definition deletes, min/max: 90 sec / 150 sec
  • Wait time before first delete: 300 sec

It's important to note that these settings apply to each user, not the test as a whole. For example, the configuration allows for 2 diagrams to be open at once, and if there are 10 users participating in the test, that means there can be a maximum of 20 diagrams open simultaneously during the test.

Test steps performed

Loadgen can be used to simulate the actions of any number of virtual Rational System Architect users. For this study we used Loadgen to repeat the same 2 hour test with an increasing number of virtual users (from 5 to 25 concurrent users).

A Loadgen test run consists of the following steps:

  1. From a Windows machine separate from the application and database servers, a remote desktop connection is made to the application server, logging in with a predefined user ID.
  2. Through the remote desktop connection, an instance of Rational System Architect is started on the application server, and the test encyclopedia is opened.
  3. Steps 1 and 2 are repeated for each virtual user desired for the test.
  4. After all virtual users have opened the test encyclopedia, Loadgen is enabled and the automated test begins.
  5. The test is allowed to run for 2 hours. As the test is executing, performance data is logged by both Loadgen and Windows Performance Monitor.
  6. After 2 hours, Performance Monitor is disabled, all virtual users are logged off the application server, and Loadgen and Performance Monitor data are archived for analysis.

Test results

Test results are broken down into three categories:

  • Rational System Architect
  • Application server
  • Database server

Rational System Architect performance

Table 1 lists the total number of actions performed by all users for each test run. The data gives an indication of the actual load being placed on the test environment. For example, during the 25-user test run, each user opened an average of 61.84 diagrams. The test run lasted 2 hours, so each user opened 30.92 diagrams per hour, or approximately 1 open action every 2 minutes.

Table 1. Total number of actions performed
5 Users10 Users15 Users20 Users25 Users
Operation count2948584186061078613477
Transaction count1003219945292723681746257
AppLock count38037561110041384917420
Explorer Refresh count2364666958731091
Explorer Open count135264384481609
Diagram Open count338671100312451546
Diagram Close count33767299912391537
Definition Open count9521888276434814349
Definition Close count9501880276134674345
Definition Add count6021191172621742746
Definition Delete count28957985110651354

Table 2 lists the average time in seconds that it took to perform each action. For example, during the 25-user test run, the average time to perform the 1546 diagram opens was 2.24 seconds.

Table 2. Average time to perform each action
5 Users10 Users15 Users20 Users25 Users
Average. Operation0.430.520.681.041.12
Average Transaction0.130.160.220.380.41
Average Applock0.160.170.180.220.22
Explorer Refresh average0.270.290.280.350.37
Explorer Open average0.270.290.260.330.37
Diagram Open average1.181.401.721.972.24
Diagram Close average0.570.720.921.281.64
Definition Open average0.300.370.551.051.02
Definition Close average0.300.370.490.870.92
Definition Add average0.190.280.501.131.31
Definition Delete average0.290.370.691.541.76

Application server performance

CPU use

Table 3 shows the load placed on the application server CPU during each test run.

  • Average % processor use = Average CPU use for test run.
  • Total % change = To eliminate any processor use from normal Windows operations, we calculate the average processor use for the current test run minus the average processor use of the previous test run. Thus, the total % change between 10 and 15 users is 46.44 - 27.62 = 18.82.
  • % Change per user = Total % change divided by 5 (increase in users between test runs).
Table 3. Application server CPU use
Test RunAverage % Processor UsageTotal % Change% Change Per User
5 users13.35NANA
10 users27.6214.272.85
15 users46.4418.823.76
20 users62.8116.373.27
25 users73.7010.892.18
Average % change per user = 3.01% (per core*)
* This number applies to each core, because this is a dual core system. If it were a single core system, this number would be about double.
Chart 1. Application server CPU use during 2-hour test runs
Chart showing resource use over time

Larger view of Chart 1.

Memory use

Table 4 shows the load placed on application server memory during each test run.

  • Peak % RAM use = Highest reported RAM use, used to see if there were any unusual spikes.
  • Average % RAM use = Average of all reported RAM use.
  • Peak MB RAM use = The highest-reported RAM use in megabytes.
  • Average MB RAM use = Average of all reported RAM use in megabytes.
  • Total MB RAM change = In order to eliminate any memory use from normal Windows operations, we calculate the average RAM MB use for the current test run minus the average RAM MB use of the previous test run. Thus, the total MB change between 10 and 15 users is 2586.51 - 1983.12 = 603.39.
  • % Change per user = Total MB RAM Change divided by 5 (increase in users between test runs).
Table 4. Application server memory use
Test RunPeak % RAM UsageAvg. % RAM UsagePeak MB RAM UsageAvg. MB RAM UsageTotal MB RAM ChangeMB RAM Change Per User
5 users37.6132.401649.721427.04NANA
10 users46.5444.172089.571983.12556.08111
15 users61.1657.452753.452586.51603.39120
20 users77.7171.643498.583225.33638.82128
25 users88.9784.364005.313797.76572.43115
Average MB RAM change per user = 118.5
Chart 2. Application server % of memory used during 2-hour test runs
Chart showing resource use over time

Larger view of Chart 2.

Chart 3. Application server memory used during 2-hour test runs
Chart showing resource use over time

Larger view of Chart 3.

Disk Use

Table 5 shows the load placed on the application server hard drive during each test run.

  • Average % disk time use = Average of all reported disk use.
  • Total % change = In order to eliminate any disk use from normal Windows operations, we calculate the average % disk time use for the current test run minus the average % disk time use of the previous test run. Thus, the total % change between 10 and 15 users is 4.31 - 2.66 = 1.65.
  • % Change per user = the Total % change divided by 5 (increase in users between test runs).
Table 5. Application server disk use
Test runAvg. % disk time useTotal % change% Change per user
5 users2.13NANA
10 users2.66 0.53.11
15 users4.31 1.65.33
20 users5.28 0.97.19
25 users6.93 1.65.33
Average % change per user = .24%
Chart 4. Application server % disk use during 2-hour test runs
Chart showing resource use over time

Larger view of Chart 4.

Network use

Table 6 shows the load placed on the application server network interface during each test run.

  • Peak network KB/sec = Highest reported network use, used to see if there were any unusual spikes.
  • Average network KB/sec = Average of all reported network use.
  • Total network KB/sec change = In order to eliminate any network use from normal Windows operations, we calculate the average Network KB/sec use for the current test run minus the average Network KB/sec use of the previous test run. Thus, the total KB/sec change between 10 and 15 users is 266.48 – 161.32 = 105.16.
  • Network KB/sec change per user = the Total Network KB/sec Change divided by 5 (increase in users between test runs).
Table 6. Application server network use
Test runPeak network KB/secAverage network KB/secTotal network KB/sec changeNetwork KB/sec change per user
5 users807.5978.78NANA
10 users810.98161.3282.5416.51
15 users9089.23266.48105.1621.03
20 users10063.78441.25174.7734.95
25 users11710.39513.7372.4814.50
Average network KB/sec change per user = 21.75
Chart 5. Application server network use during 2 hour test runs
Chart showing resource use over time

Larger view of Chart 5.

Database server performance

CPU use

Table 7 shows the load placed on the database server CPU during each test run.

  • Average % processor use = Average CPU use for test run.
  • Total % change = In order to eliminate any processor use from normal Windows operations, we calculate the average processor use for the current test run minus the average processor use of the previous test run. The total % change between 10 and 15 users is 17.58 – 13.24 = 4.34.
  • % change per user = Total % change divided by 5 (increase in users between test runs).
Table 7. Database server CPU use
Test RunAvg. % Processor UsageTotal % Change% Change Per User
5 users8.95NANA
10 users13.244.29.86
15 users17.584.34.87
20 users20.242.66.53
25 users23.032.79.56
Average % change per user = .71%
Chart 6. Database server CPU use during 2-hour test runs
Chart showing resource use over time

Larger view of Chart 6.

Memory use

Table 8 shows the load placed on database server memory during each test run.

  • Peak % RAM use = Highest reported RAM use, used to see if there were any unusual spikes.
  • Average % RAM use = Average of all reported RAM use.
  • Peak MB RAM use = The highest reported RAM use in megabytes.
  • Average MB RAM use = Average of all reported RAM use in megabytes.
  • Total MB RAM change = In order to eliminate any memory use from normal Windows operations, we calculate the average RAM MB use for the current test run minus the average RAM MB use of the previous test run. The total MB change between 10 and 15 users is 2120.29 – 2117.33 = 2.96.
  • % Change per user = Total MB RAM change divided by 5 (increase in users between test runs).
Table 8. Database server memory use
Test RunPeak % RAM UsageAvg. % RAM UsagePeak MB RAM UsageAvg. MB RAM UsageTotal MB RAM ChangeMB RAM Change Per User
5 users50.3050.232119.572116.54NANA
10 users50.2950.252119.072117.330.79.16
15 users50.4850.322127.362120.292.96.59
20 users50.4050.332123.732120.680.39.08
25 users50.3550.252121.782117.49-3.19-.64
Average MB RAM change per user = .05
Chart 7. Database server % of memory used during 2-hour test runs
Chart showing resource use over time

Larger view of Chart 7.

Chart 8. Database server memory used during 2-hour test runs
Chart showing resource use over time

Larger view of Chart 8.

Disk use

Table 9 shows the load placed on the database server hard disk during each test run.

  • Average % disk time use = Average of all reported disk use.
  • Total % change = In order to eliminate any disk use from normal Windows operations, we calculate the average % disk time use for the current test run minus the average % disk time use of the previous test run. The total % change between 10 and 15 users is 4.23 – 2.60 = 1.63.
  • % change per user = Total % change divided by 5 (increase in users between test runs).
Table 9. Database server disk use
Test runAverage % disk time useTotal % change% Change per user
5 users2.30NANA
10 users2.600.30.06
15 users4.231.63.33
20 users5.210.98.20
25 users6.341.13.23
Average % disk use change per user = .21
Chart 9. Database server % disk use during 2-hour test runs
Chart showing resource use over time

Larger view of Chart 9.

Network use

Table 10 shows the load placed on the database server network interface during each test run.

  • Peak network KB/sec = Highest reported network use, used to see if there were any unusual spikes.
  • Average network KB/sec = Average of all reported network use.
  • Total network KB/sec change = In order to eliminate any network use from normal Windows operations, we calculate the average Network KB/sec use for the current test run minus the average Network KB/sec use of the previous test run (e.g. - The total KB/sec change between 10 and 15 users is 263.54 – 157.11 = 106.43).
  • Network KB/sec change per user = the Total Network KB/sec Change divided by 5 (increase in users between test runs).
Table 10. Database server network use
Test runPeak network KB/secAverage network KB/secTotal network KB/sec changeNetwork KB/sec change per user
5 users802.2674.52NANA
10 users922.15157.1182.5916.52
15 users10095.64263.54106.4321.29
20 users11356.14435.65172.1134.42
25 users11755.05505.9170.2614.05
Average network KB/sec change per user = 21.57
Chart 10. Database server network use during 2-hour test runs
Chart showing resource use over time

Larger view of Chart 10.


Conclusions

Objectives

Objective 1. Determine how many concurrent users can function effectively when using Rational System Architect in a given hardware configuration.

Given the hardware configuration used for this study, we believe that up to 25 concurrent Rational System Architect users could function effectively. The response times for performing typical actions were acceptable, with diagram opens taking the longest at 2.24 seconds on average.

The main factors limiting the scalability of the test environment are CPU and memory use on the application server. With 25 concurrent users, CPU and memory use were at 74% and 84%, respectively. Additional users are likely to quickly cause these resources to be overtaxed and lead to noticeable performance degradation.

Objective 2. Develop a guide to help predict hardware needs as the number of concurrent Rational System Architect users is increased.

Table 11 displays the average resource use for each user in our test environment.

Table 11. Average resource use per user
Application serverDatabase server
CPU3.01%0.71%
Memory118.5 MB0.05 MB
Disk0.24%0.21%
Network21.75 KB/s21.57 KB/s

By looking at the average resource use for each user, it is possible to estimate the hardware that will be required to support additional Rational System Architect users. Let's consider two cases: 50 and 200 concurrent users.

50 users

Application server

Starting with the application server, we can see that both disk and network use per user is low and will not be limiting factors. Estimated disk use would be 12% (.24 * 50), while network use would be 8.7 Mbps (21.75 * 8 * 50).

With CPU use at 3.01% per user, under a 50-user load the dual core CPU used for this study would be severely overtaxed (3.01 * 50 = 151%). Additionally, with memory use of 118.5 MB per user, under a 50-user load all available memory would be used (118.5 * 50 = 5.9 GB). To mitigate these limiting factors, we recommend that the application server used for this study be upgraded to a quad core CPU with 8 GB of RAM.

Database server

Moving to the database server, we see that CPU, disk, and network use per user is low and will not be limiting factors. Estimated CPU use would be 35.5% (.71 * 50), disk use would be 10.5% (.21 * 50), and network use would be 8.6 Mbps (21.57 * 8 * 50).

When considering memory use on the database server, it more important to look at the size of the database than how much memory is consumed per user. Generally, you want to have enough memory to cover the largest database being used. The database used for this study was 2.8 GB in size. Allowing for modest database growth over time, a configuration of 4 GB of RAM would be reasonable.

In conclusion, the database server used for this study would be able to support 50 concurrent users.

200 users

Application server

Starting with the application server, we can see that both disk and network use per user is low and will not be limiting factors. Estimated disk use would be 48% (.24 * 200), while network use would be 34.8 Mbps (21.75 * 8 * 200).

With CPU use at 3.01% per user, under a 200-user load the dual core CPU used for this study would be severely overtaxed (3.01 * 200 = 602%). Additionally, with memory use of 118.5 MB per user, under a 200-user load all available memory would be used (118.5 * 200 = 23.7 GB). There are two approaches that can be used to mitigate these limiting factors:

  • The first approach is to deploy Rational System Architect on a single high-powered server consisting of 4 quad core CPUs with 32 GB of RAM.
  • A more practical approach could be to deploy Rational System Architect on 4 different application servers. Each application server would have a single quad core CPU and 8 GB of RAM and be capable of supporting up to 50 concurrent users. This approach is known as server clustering. Additional application servers can be added to the cluster as needed, allowing for increased scalability.

Database server

Moving to the database server, we see that both disk and network use per user is low and will not be limiting factors. Estimated disk use would be 42% (.21 * 200), while network use would be 34.5 Mbps (21.57 * 8 * 200).

Even though CPU use per user is low, at .71% under a 200-user load the single CPU on the database server would be severely overtaxed (.71 * 200 = 142%). To mitigate this limiting factor, we recommend upgrading the database server to a dual core CPU.

When considering memory use on the database server, it more important to look at the size of the database than how much memory is consumed per user. Generally, you want to have enough memory to cover the largest database being used. The database used for this study was 2.8 GB in size. Allowing for more rapid database growth over time, a configuration of 8 GB of RAM would be reasonable.

In conclusion, to support 200 concurrent users the database server used in this study would need to be upgraded to a dual core CPU with 8 GB of RAM.

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational
ArticleID=631996
ArticleTitle=Scalability of Rational System Architect 11.4
publish-date=03102011