DB2 performance testing and monitoring using Rational Performance Tester

Document options Document options requiring JavaScript are not displayed Discuss Sample code Rate this page Help us improve this content

Level: Intermediate

Mike Liu (mikezliu@ca.ibm.com), Software Engineer, IBM
Tony Lau (tktlau@ca.ibm.com), Software Engineer, IBM　

24 Jan 2008

Discover how you can use Rational® Performance Tester as a performance testing tool in an IBM® DB2® for Linux®, UNIX®, and Windows® benchmarking environment. Learn best practices to follow and general rules of thumb. The Trade6 benchmark application is used as an example workload.

Introduction

Performance engineering is a discipline that entails determining and reporting the current performance of a software solution. It involves simulating a large number of concurrent users, collecting diagnostic data, plotting graphs, doing performance analysis, and engineering performance improvements into the System Under Test (SUT). This process is repeated until the performance goal is reached. In a typical DB2 WebSphere® topology, there is easily more than ten, if not hundreds, of machines in the SUT. As a result, the testing cycle is often very tedious and error-prone.

Rational Performance Tester is the cornerstone of IBM’s strategy for performance testing and monitoring intended to simplify performance engineering. This article introduces the best practices of using the IBM testing solution to test DB2 in a WebSphere cluster environment.

Back to top

Goals

The major objectives of this article are:

• To describe the major pain-points in performance testing and monitoring without a performance test solution like Rational Performance Tester
• To describe Rational Performance Tester and Tivoli Performance Monitoring infrastructure as an IBM performance testing solution
• To go through some of the performance best practices and rules of thumb of using Rational Performance Tester in a DB2 WebSphere environment

Back to top

Prerequisites

Below is a listing of machines used in our testing environment, their roles, hardware used, and installed software.

Back to top

Performance engineering without RPT

Performance engineering is a discipline that deals with determining and reporting the current performance of a software solution. It involves setting up the System Under Test (SUT), simulating a large number of concurrent users, managing the SUT, collecting diagnostic data, analyzing the data collected, and engineering performance improvements into the SUT. This process is repeated until the performance goal is reached.

1. Setup a test environment. The test environment must be able to model the production environment. The measurements gathered later on are only as accurate as the models that are developed. It could be as simple as a single node WebSphere Community Edition with DB2 Express-C or it could be a 16-node cluster of WebSphere Application Server Extended Deployment Edition on DB2 9.
2. Simulate a user load. A realistic user load deals with consumer behavior analysis and implementation of efficient threading. A good workload simulator requires implementation of random functions. A good workload simulator for enterprise performance testing handles thousands of threads and/or processes, and it must be perfectly scalable.
3. Manage the System Under Test. On every trial, the SUT must be in the same state in order to produce repeatable results. In order for this to happen, you should recycle the cluster of WebSphere and DB2, clean log files in different directories, and restore the DB2 tablespaces. It is good practice to do all these consistently in order to produce repeatable results.
4. Collect diagnostic data on every single machine. At the very least, you should have vmstat, iostat, and periodic DB2 snapshots. The diagnostic tools must be started at the right time and, of course, with the right command. The output must be consolidated systematically for further analysis. Custom-made scripts are often written for this.
5. Crunch the numbers and plot graphs. This entails a good amount of copy-and-paste and spreadsheet manipulation. A reasonably complicated system involves more than 100 pieces of diagnostic data to graph and analysis. Proprietary graphing solutions are often integrated to a custom-made script to post-process the data to human consumable form.
6. Repeat for N times! The purpose of performance testing is to reveal the performance bottleneck and execute performance improvement. How many trials does it take to pass a customer acceptance test? It is our experience that a release can involve more than 100 runs.

Rational Performance Tester is the IBM strategy of performance testing and monitoring intended to simplify performance engineering. In the following sections, see how RPT can simplify the performance testing cycle from simulating the user load to the result graphs being plotted.

Back to top

Topology

It is important to first plan out a topology for your testing environment. A typical clustered Trade6 testing environment driven by RPT consists of a database, an HTTP server, a WebSphere deployment manager, WebSphere application server(s), driver(s), and a RPT workbench. The topology for our testing environment is as follows:

The RPT workbench should be installed on a separate machine from the workload driver machines to reduce overhead on the drivers

The drivers have a high overhead and should be separated from your Trade cluster. If the driver is on one of the machines running a Trade server, then your cluster becomes unbalanced in terms of resources, which may lead to performance issues if you are using a simple round-robin load balancing scheme.

There are many factors that affect the scaling of each component in your test environment to one another. For example, if your workload is database-heavy, then you expect a high ratio of database to WebSphere CPUs. For a our test environment, we found the following sizing ratios to work well in balancing resource utilizations across our machines:

1. Total Driver CPUs : Total DB2 CPUs
   4 : 1
2. Total HTTP Server CPUs : Total DB2 CPUs
   1 : 10
3. Total WAS CPUs : Total DB2 CPUs
   2 : 1 (4 : 1 for EJB applications)
4. Network for Drivers : Total DB2 CPUs
   We recommend a 100 Mb network when the database machine has less than 4 CPUs, and a gigabit network for more than 4 CPUs

Back to top

Setting up the Trade6 environment

To set up a Trade6 benchmark on a DB2 and WebSphere platform, including creating and populating the database, please refer to the "Set up and run a Trade6 benchmark with DB2 UDB" article (see Resources).

In the next few sections, discover how Rational Performance Tester can be used as the workload driver and learn about some monitoring utilities available for users.

Back to top

RPT as a workload driver

Using the IBM Rational Performance Tester as a workload driver simplifies and automates the process of running performance tests. RPT provides the framework for users to create and run different types of performance tests and provides built-in utilities to simplify the process of collecting and analyzing performance metrics.

For example, the HTTP protocol in RPT allows users to record, edit, and execute HTTP performance tests that measure metrics such as the page hit, page throughput, and page response times.

RPT also comes with integrated support for resource monitoring tools such as IBM Tivoli Monitoring, Windows Performance Monitor, and rstatd, which can be used to monitor resources across all the machines in your testing environment. IBM Tivoli Monitoring comes with optional components to monitor applications and databases in further detail. For example, the Tivoli Monitoring for Databases allows users to monitor a variety of metrics inside DB2 databases such as database snapshots, table spaces, and buffer pools.

The RPT workbench

The workbench serves as an interface from which users can configure, launch, and monitor performance tests. When running a performance test, the workbench deploys RPT execution code to target deployment machines and executes them using the RPT agent controller.

Installing the RPT workbench

1. Download the RPT 7 installation package and start the installation wizard by running launchpad.exe.
   
   
   Figure 2. Installing IBM Rational Performance Tester
   
   
   
2. Choose to install the IBM Rational Performance Tester (Includes Agent)
3. If you do not have the IBM Installation Manager installed, follow the instructions in the wizard to install it
4. When the IBM Installation Manager opens, choose to install IBM Rational Performance Tester 7.0.0
5. Agree to the license and choose an installation path for the shared directory and RPT
6. Choose a typical installation and follow the instructions on completing the installation

You need to obtain license keys for RPT if you want to run tests with a large number of users. You can do this by either using a licensing server containing license keys or using a license file. To point to a Rational license server or import a license file, you should run the IBM Rational License Key Administrator (found in All Programs -> Rational Software). The license key that your workbench uses is also automatically used by all agent controllers that the workbench uses.

The default workbench heap size may not be sufficient for larger tests depending on the amount of data that has to be transferred back to the workbench from the agent controllers and other monitoring utilities. If your workbench runs out of memory, you can increase the default workbench heap size by editing eclipse.ini located in your RPT home directory and changing the following line:

-Xms40m

-Xmx1500m

It is recommended that you set logging levels of the RPT schedule execution components to “WARNING” to reduce overhead.
To do this, go to Window -> Preferences -> Logging. Open the Loggers tab and change the logging level of the following to “WARNING”:

com.ibm.rational.test.common.schedule.execution
com.ibm.rational.test.lt.execution

The RPT agent controller

A RPT agent controller must be installed on each driver machine to provide a method for the workbench to control them. This allows the RPT workbench to deploy code onto them as well as to execute performance tests.

Installing the RPT agent controller

1. Download the agent controller installation package to the driver machine
2. Run the installation wizard (On Linux systems you should run install_linux.bin; On Windows systems you should run launchpad.exe and then choose to install the Agent)
3. Agree to the license and choose an installation path
4. Choose to use the default JVM and complete the installation

To start the agent controller on Unix/Linux, execute RAStart.sh:

$ /opt/IBM/SDP70Shared/AgentController/bin/RAStart.sh
  Starting Agent Controller
  RAServer started successfully

Be sure that your agent controller is using the JVM that comes packaged with RPT (RPT70/jdk/jre/bin/java). To check which JVM the agent controller is using, open <Agent Controller Home Dir>/config/serviceconfig.xml and check the value of JAVA_PATH. As good practice, the Java version used by your workbench should match the one used by your agent controllers.

On each machine containing an agent controller, you may need to apply the following tunings:

1. On Window machines, you should increase the limit on the number of tcpip ports:
   Open regedit (Start -> Run -> regedit) Edit the value of MaxUserPort to 65534 under the following key:
   
   

   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

   
   
2. On Unix/Linux systems you should increase the maximum allowed open files using ulimit (as root):
   
   

   $ ulimit -n 10050000

   
   

After applying tunings, you should restart the agent controller, as the listing below shows:

$ /opt/IBM/SDP70Shared/AgentController/bin/RAStop.sh
  RAServer stopped, pid = 27275
$ /opt/IBM/SDP70Shared/AgentController/bin/RAStart.sh
  Starting Agent Controller
  RAServer started successfully

Back to top

Running a performance test with RPT

Deploying the Trade6 performance test project

A sample RPT performance test project is provided (see Downloads). Using the scripts provided in this performance project, you can drive up your Trade6 environment through a simulated workload that repeatedly hits the /trade/scenario servlet of the Trade6 website.

To deploy these scripts, open RPT and choose to import an existing project into the workspace. Browse to where you unzipped TradeProject, select it, and select Copy projects into workspace. Alternatively, you can point directly to the zip file by clicking Select archive file. Figure 3 shows these steps:

Click Finish to let RPT copy this project into your workspace. Your workspace should now have a performance test project containing two Trade6 test scripts:

You need to make the following changes to these scripts so that they work for your environment:

1. Open TradeScheduleTest and expand the test until you see the HTTP request element. Change the host and port to point to your Trade6 web host.
   
   
   Figure 5. Browsing TradeScheduleTest
   
   
   
2. Open TradeSchedule and expand User Group 1. For each driver machine where you’d like to deploy this performance test on, click Add new and add that machine.
   
   
   Figure 6. Adding deployment machines
   
   
   

Enabling problem determination via the Problem Determination tab in the performance schedule allows agent controllers to produce logs for debugging purposes. For actual performance runs, the problem determination log level should be turned to “None” to reduce overhead on the agent controller machines, or turned to “Severe” and sampled from 1 user per user group.

For large performance tests you may need to increase the JVM heap for the agent controllers. For example, if you wanted to increase the heap to 1500 MBs, double-click on an agent controller machine (in Test Navigator), open the General Properties tab, and add a new property named RPT_VMARGS with a value of –Xmx1500m. It is recommended to set the maximum heap size to 1.5 GBs for Windows and 3 GBs for Linux.

Running the Trade6 performance test

To launch the performance test, right-click on the TradeSchedule test -> Run As -> Performance test schedule. Alternatively, you can press Alt+Shift+X, C while having the performance test selected.

The workbench then deploys the performance test and required RPT libraries to each of the driver machines and executes them. As the test is executing, you can view different pages of the performance report to monitor the test dynamically.

Figure 8 show the throughput page which displays information regarding the page hit rate and user load:

The Response vs. Time page plots the average response times of each request. Figure 9 shows this response time page:

Back to top

Monitoring tools in RPT

Rational Performance Tester 7 comes with integrated support for three monitoring tools:

• rstatd
• IBM Tivoli Monitoring
• Windows Performance Monitor

It is recommended that you install at least one type of monitoring tool for each machine in your performance test environment. Doing so allows you to monitor all resources and isolate any resource bottlenecks.

To enable monitoring, go to the Resource Monitoring tab of your performance schedule, as Figure 10 illustrates:

Click Add New to add a new machine to monitor. You will then be prompted for the host to monitor and the monitoring tool to use.

Most monitoring utilities collect statistics with timestamps based on the system under monitor. Because of this, you should first synchronize system clocks across all the systems being monitored.

On Linux and AIX systems you can do this by typing (as root):

$ ntpdate -u speedo1
  12 Apr 13:52:06 ntpdate[21596]: step time server 9.26.54.6 offset 8.096963 sec

Windows Performance Monitor

The Windows Performance Monitor is installed by default on all Windows machines and can be used to monitor a variety of system resources.

rstatd

The rstatd tool is used to collect some basic monitoring data from Linux and Unix operating systems. These include:

• Average number of jobs in run queue
• IOWait/Idle/System/User CPU Time
• Total collisions seen on all interfaces
• Total context switches
• Total disk transfers
• Total inbound/outbound errors on all interfaces
• Total inbound/outbound packets on all interfaces
• Total interrupts
• Total VM pages paged in/out
• Total VM pages swapped in/out

Installing rstatd

rstatd comes pre-installed on most Unix systems. To start the rstatd daemon, type:

$ rpc.rstatd

For Linux operating systems, an open source version of rstatd can be found here: http://rstatd.sourceforge.net/ To install rstatd:

$ tar xvf rpc.rstatd-4.0.1.tar
$ cd rpc.rstatd-4.0.1/
$ ./configure
$ make
$ make install

After this, start it by typing:

$ rpc.rstatd

To monitor resources in RPT using rstatd, open the Resource Monitoring tab of a performance test, choose to use UNIX rstatd monitor, and choose the counters you want to collect. Figure 11 shows these steps:

To see the monitoring in action, open the Resources tab of the performance report while a test is running. Below you can see a chart plotting the Idle, System, IOWait, and User CPU time on the database machine:

IBM Tivoli Performance Monitoring

IBM Tivoli Monitoring provides a much richer set of metrics to monitor and can be used to gather additional performance metrics from your testing environment not collected by rstatd. Different types of ITM monitoring agents can be installed such as Tivoli OS Monitoring, Tivoli Monitoring for Databases, or Tivoli Monitoring for Applications to provide further in-depth resource monitoring.

Installing IBM Tivoli Performance Monitoring

Before installing Tivoli Monitoring, you should choose a machine to act as the monitoring server. For our test environment we chose to use the database machine.

To install the monitoring server on Unix/Linux:

1. Copy the installation package onto the machine and run the install.sh script
2. Choose an installation directory for the monitoring server
3. Choose to install products to the local host and accept the license agreement
4. Select your OS and choose to install the Tivoli Enterprise Monitoring Server (TEMS)
5. Choose a TEMS name and complete the installation

$ /home/adm22237/ITM/bin/itmcmd server start MORTAL_HUB
  Starting TEMS...
  TEMS started...

Next you should install monitoring agents on each machine to monitor. The Tivoli OS monitoring agent is included with most installation packages containing the Tivoli monitoring server installation. Other monitoring agents such as Tivoli Monitoring for Databases or Tivoli Monitoring for Applications may need to be downloaded separately.

To install a monitoring agent on Unix/Linux:

1. Copy the agent installation package onto the machine and run the install.sh script
2. Choose an installation directory for the monitoring agent
3. Choose to install products to the local host and accept the license agreement
4. Select your OS and choose to install the monitoring agent
5. Follow the instructions and complete the installation.

$ /home/adm22237/ITM/bin/itmcmd config -A ux
  Agent configuration started...
        
  Will this agent connect to a TEMS? [YES or NO] (Default is: YES):
  TEMS Host Name (Default is: mortal): mortal
        
  Network Protocol [ip, sna, ip.pipe or ip.spipe] (Default is: ip.pipe):
        
  Now choose the next protocol from one of these:
  - ip
  - sna
  - ip.spipe
  - none
  Network Protocol 2 (Default is: none):
  IP.PIPE Port Number (Default is: 1918):
  Enter name of KDC_PARTITION (Default is: null):
        
  Configure connection for a secondary TEMS? [YES or NO] (Default is: NO):
  Enter Optional Primary Network Name or "none" (Default is: none):
  Are you installing this product into a clustered environment(Default is: NO):
  Agent configuration completed...

Where ux should be replaced by the ID of your agent. This ID is unique based on the agent type and your OS. To find out your agent ID, you can use the cinfo command:

$ /home/adm22237/ITM/bin/cinfo –I
        
  *********** Thu Apr 12 10:25:41 EDT 2007 ******************
  User      : mikezliu     Group: build pdxdb2
  Host name : mortal       Installer Lvl: 610 / 100
  CandleHome: /home/adm22237/ITM
  ***********************************************************
  ...Product inventory
        
  a4      Monitoring Agent for i5/OS
  tms     Version: 06.10.02.00
        
  ax      IBM Tivoli Monitoring Shared Libraries
  aix513  Version: 06.10.02.00
  aix516  Version: 06.10.02.00
        
  jr      Tivoli Enterprise-supplied JRE
  aix513  Version: 400 Rel: 100
  aix516  Version: 400 Rel: 100
        
  lz      Monitoring Agent for Linux OS
  tms     Version: 06.10.00.00
        
  ms      Tivoli Enterprise Monitoring Server
  aix513  Version: 06.10.02.00
        
  nt      Monitoring Agent for Windows OS
  tms     Version: 06.10.02.00
        
  sh      Tivoli Enterprise Monitoring SOAP Server
  aix513  Version: 06.10.02.00
        
  sy      Summarization and Pruning Agent
  tms     Version: 06.10.02.00
        
  tm      Monitoring Agent for IBM Tivoli Monitoring 5.x Endpoint
  tms     Version: 06.10.02.00
        
  ud      Monitoring Agent for DB2
  aix516  Version: 06.10.00.00
  tms     Version: 06.10.00.00
        
  ui      Tivoli Enterprise Services User Interface
  aix513  Version: 06.10.02.00
  aix516  Version: 06.10.02.00
        
  ul      Monitoring Agent for UNIX Logs
  tms     Version: 06.10.02.00
        
  um      Universal Agent
  tms     Version: 06.10.02.00
        
  ux      Monitoring Agent for UNIX OS
  aix516  Version: 06.10.02.00
  tms     Version: 06.10.02.00

Follow the instructions and enter the hostname of the Tivoli monitoring server when prompted.

Depending on the type of monitoring agent you are installing, you may need to install support for that agent to the monitoring server. For our cluster, we added support to the monitoring server for a monitoring agent for DB2 on Unix/Linux as follows:

1. Stop the monitoring server:
   

   $ /home/adm22237/ITM/bin/itmcmd server stop MORTAL_HUB Stopping TEMS... TEMS stopped...

   
   
2. Copy the agent installation package (which contain the installation files for adding agent support to the monitoring server) to the monitoring server machine and run the install.sh script
3. Choose the installation directory of the monitoring server
4. Choose to install products to the local host and accept the license agreement
5. Select your OS and choose to install the Tivoli Enterprise Monitoring Server support for Databases
6. Follow the instructions and complete the installation
7. Start the monitoring server:
   

   $ /home/adm22237/ITM/bin/itmcmd server start MORTAL_HUB Starting TEMS... TEMS started...

   
   
8. Add support for the monitoring agent to the server by typing:
   

   $ /home/adm22237/ITM/bin/itmcmd support -t MORTAL_HUB ud Copying cat and attr data... Product support installation started... Product support installation completed...

   
   Where ud should be replaced by the ID of your agent (use the cinfo command to find out the ID).

The following patch should also be applied to the Tivoli Monitoring Server and Tivoli Monitoring Agent to add support for DB2 V9:

DB2 6.1.0-TIV-ITM_DB2-LA0053 Agent Support Install (6.1.0-TIV-ITM_DB2-LA0053.tar)

To start the agent, use the itmcmd command. You may need to pass in additional options depending on the type of agent you want to start. For example, to start a Tivoli OS monitoring agent, you can type:

$ /home/adm22237/ITM/bin/itmcmd agent start ux
  Starting agent...
  Agent Started...

To start a Tivoli monitoring agent for DB2, you should follow these steps:

1. Log onto the instance user (for example, mikezliu)
2. Connect to the database and start the monitoring agent:
   
   

   $ db2 connect to trade6db Database Connection Information Database server = DB2/AIX64 9.1.2 SQL authorization ID = MIKEZLIU Local database alias = TRADE6DB $ /home/adm22237/ITM/bin/itmcmd agent -o mikezliu start ud Starting agent... Agent Started...

   
   

To monitor resources in RPT using Tivoli, open the Resource Monitoring tab of a performance test, choose to use IBM Tivoli Monitoring, and specify the host name of the monitoring server:

After doing this, you can select counters to collect using the Resource tab.

You can see the monitoring in action by opening the Resource tab of a performance report while a test is running. Figure 15 illustrates these steps:

Right-clicking on the graph allows you to customize it in different ways such as adding and removing counters. For example, if you would only like to view the buffer pool hit ratio then you can right click on the graph -> Add/Remove Performance Counters -> Resource Monitoring Counter and select only the pool hit ratio counter. The graph is then updated to plot only that counter:

Back to top

Summary of best practices

Planning out your test environment topology

RPT workbench and driver machines: The RPT workbench should be installed on a separate machine from the workload driver

Driver machines: The drivers have a high overhead and should be separated from your system under test.

RPT workbench

Workbench heap size: Increase the default workbench heap size for larger tests

Logging levels: Logging levels for the RPT schedule execution components should not be set above “WARNING” for actual performance runs

Agent Controller

JVM for agent controller: The agent controller should be using the JVM that comes packaged with RPT

Maximum open files on Linux/UNIX: Increase the maximum allowed open files

Maximum tcpip ports on Windows: Increase the maximum allowed tcpip ports

Running RPT performance tests

Problem determination: During performance runs, problem determination log level should be turned to “None” or turned to “Severe” and sampled from 1 user per user group.

JVM stack heap size on Agent Controllers: For large tests you will need to increase the JVM stack heap size of agents started on the agent controllers

Resource Monitoring

System clocks: All machines in your system under test should have their system time synchronized with each other so that resource monitoring uses the correct timestamps

Back to top

Conclusion

This article describes how the IBM Rational Performance Tester can be used as a performance testing and monitoring tool in a DB2 environment. RPT offers a large variety of tools that assist users in carrying out performance tests, from built-in support for monitoring utilities to the ability for users to write and execute their own workload driver scripts. Using RPT as an end-to-end performance testing tool can greatly improve productivity and simplify the process of conducting performance tests and collecting performance metrics from a DB2 testing environment.

Back to top

Acknowledgements

The authors would like to thank Kent Siefkes and Kevin Mooney for their help and advice in using Rational Performance Tester as a workload driver, as well as providing advice on best practices to follow. The authors would also like to thank Joseph P Toomey, Kevin Mooney, Judy Liu, and Adam Muise for their valuable feedback on the article.

Back to top

Download

	Information about download methods

Resources

• "Set up and run a Trade6 benchmark with DB2 UDB" (developerWorks, Jun 2005) is a step-by-step guide to setting up a Trade6 benchmarking environment with DB2
  
  
• Visit the IBM DB2 InfoCenter to find out more about DB2 for Linux, UNIX, and Windows.
  
  
• Go to the IBM WebSphere Application Server InfoCenter for more information on IBM WebSphere Application Server
  
  
• Visit the Rational Performance Tester page to find more articles, tutorials and downloads related to the Rational Performance Tester.
  
  
• Go to the DB2 for Linux, UNIX, and Windows zone on developerWorks to get the resources you need to advance your DB2 skills.
  
  
• Browse the technology bookstore for books on these and other technical topics.
  
  

• Download IBM product evaluation versions and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.

• Participate in the discussion forum.

About the authors

		Mike Liu is a software engineer in DB2 performance benchmarks and advanced solutions development at the IBM Toronto Lab. He is currently involved in performance benchmarking and the prototyping of a new protocol for Rational Performance Tester.

		Tony Lau has more than 5 years of experience in DB2 performance testing and monitoring. He is a software engineer in DB2 WebSphere Benchmarks and Advanced Solutions Development. He has been the DB2 technical leader of 3 world-record-breaking SPECjAppServer2004 benchmark teams. He is currently involved in the design and prototyping of new features for the next generation of Rational Performance Tester. Prior to joining IBM in Toronto, he was a research assistant at the Bell laboratory at the University of Waterloo.

Rate this page

Please take a moment to complete this form to help us better serve you. Did the information help you to achieve your goal? Yes No Don't know   Please provide us with comments to help improve this page:   How useful is the information? 1 2 3 4 5 Not useful Extremely useful

Back to top