Skip to main content
skip to main content

developerWorks  >  Rational  >

Using IPOT: Resource monitoring

developerWorks
Document options

Document options requiring JavaScript are not displayed

Discuss


My developerWorks needs you!

Connect to your technical community


Rate this page

Help us improve this content


Level: Intermediate

Russell Butek (butek@us.ibm.com), Developer, IBM

17 Apr 2007

This article describes the various resource monitoring data collectors available in IBM Rational Performance Tester, including a discussion of the architecture of each data collector and how each is configured when using Performance Tester.

Overview

Resource monitoring is a term used to describe the observation of a property or asset over time. Typically, these observations are numerical facts or data, also known as statistical data. Most often within the performance testing space a resource can be, but is not limited to, a physical system or a process executing on a system. Resource monitoring is crucial when trying to determine the problem with an application that is failing to perform at reasonable levels, because this type of monitoring allows a performance tester to determine whether there is a lack of system-level resources or an issue with the application itself.

The resource monitoring feature in IBM® Rational® Performance Tester allows real-time monitoring of systems and system processes. This granularity enables problem determination analysts to diagnose a problem accurately; as the resource monitoring data can help an analyst determine if the failure was caused by the lack of system resources (such as memory) or a particular process (such as a database) involved in an applications execution.

There are three resource monitoring data collectors provided in Rational Performance Tester:

  • Windows™ Performance Monitor for Microsoft® Windows® systems
  • rstatd for Linux®/UNIX® systems
  • IBM Tivoli® Monitoring (ITM) for monitoring a variety of platforms

Data collectors extend a generic Resource Monitoring platform, which is included in Performance Tester (See Figure1, the IBM Rational Performance Tester stack architecture). Additional data collectors can be added into the product using Eclipse™ Platform extensions, which will then be automatically loaded by the Resource Monitoring platform.


Figure 1. IBM Rational Performance Tester stack architecture
image of architecture stack

Because each data collector will have a specific mechanism for retrieving data from a remote host, the Resource Monitoring platform is designed to allow custom Java™ code to be written to implement the data collection mechanism using a common interface.

Once the data is retrieved, the data must be transformed into the Eclipse Test and Performance Tools Platform (TPTP) statistical model format. This model format is based on the Eclipse Modeling Framework (EMF) and is designed to be a generic method of storing statistical data using XML. The open source Eclipse TPTP project provides APIs that abstract the details of storing this data and make is easy to store retrieved data.

All of the data collectors provided in Performance Tester have a specific mechanism of retrieving data; however, once the data is gathered the collectors transform this data into the common statistical model format. Thereby, Performance Tester and other Eclipse-based views to render reports based on this data.

Configuring a performance schedule

Resource monitoring is enabled when configuring a performance schedule. Selecting the top-most node in the Schedule Contents section of the Performance Schedule Editor. Figure 2 displays configuration parameters available in the Schedule Element Details section. A tab labelled Resource Monitoring contains the configuration parameters that are used when the schedule in context is executed.


Figure 2. Performance Schedule Editor
image of workspace

Let us explain some of the choices and actions:

  • Enable resource monitoring: This check box activates the resource monitoring feature for the performance schedule you are working with. It can be used as an on or off switch for resource monitoring as a whole.
  • Ignore invalid resources when executing the schedule: Select this check box to suppress error messages about resource monitoring data sources. These errors can occur if the data sources are unreachable or invalid. If you select this option, you must view logs to see the error messages.
  • The Data Source table in the Resource Monitoring tab displays the configured data sources that are collected from when this schedule executes. If this is a new schedule, the Data Source table will be empty. If you have previously added resource monitoring data sources to this schedule, you can Edit or Remove them. Remove does not delete the data source from the file system; it merely removes it from this view because other test schedules or applications might still use the data source.
  • Clicking Add New (see Figure 2) creates and allows a user to configure a new resource monitoring location:
  • Host (Figure 3), indicates the target host name (Internet Protocol (IP) address for fully qualified host name) to monitor during the execution of the performance schedule.

Tips:

  • When using IBM Tivoli Monitoring as the data source, the host field is the location of the resource management collection agent (or monitoring agent) and not the Tivoli Enterprise Monitoring Server.
  • When using Windows Performance Monitor or UNIX rstatd monitor as the data source, the host field is the target Windows or UNIX system that has to be monitored.
  • Numerous data collectors are available in the Data Sources list. If required, several data sources can be configured for one location by selecting the desired data sources from the check box list on the left-hand side of the dialog.

Figure 3. Create and configure a new resource monitoring location
image of dialog box
  • Click Add Existing (Figure 2) if you have existing locations in your workspace that you want to add and configure to the current working performance schedule.

Figure 4. Add existing location to a performance schedule for resource monitoring
image of dialog box

Once you have enabled resource monitoring you have to specify the data sources to collect from.

Any configuration changes you make for a particular data source are stored with that location. This means that you have to set up a data source only once. If you export a schedule, it will contain the data source configuration information. This includes potentially sensitive information, such as stored passwords.

It is important to note that to capture accurate resource monitoring data, you must ensure that the clocks on all systems are synchronized. If you do not synchronize the clocks on the workbench and on all of the systems under test, resource counters are displayed inaccurately (with respect to time) in the reports.

Monitoring using Windows Performance Monitor

The Microsoft Windows operating system contains an embedded measurement infrastructure that can be accessed either through the Windows Registry or using Application Programming Interfaces (APIs). The measurement infrastructure is apart of the Windows Management Infrastructure (WMI) and allows for real-time monitoring of approximately one thousand performance counters.

Performance counters

Performance counters indicate how well the operating system, an application or service is performing. Each counter is designated a specific metric to collect, such as, a counter named % Processor Time will collect the percentage of elapsed time that the processor spends to execute a non-Idle thread and can be utilized as an indicator of processor activity. Capturing the measurements of such counters over time assists analysts to discover system bottlenecks and fine-tune a system or an application's performance.

Within the measurement infrastructure, counters are structured using a two-level hierarchy. The first level consists of the performance object, which is a unit that can model a physical device or abstract component in a system. The second level consists of the performance counter, which is an attribute of the performance object. Table 1 provides a list of three performance counters and the performance objects under which they are categorized.


Table 1. Windows Performance Monitor objects and counters
Performance objectPerformance counter
Processor% User Time
% Processor Time
Interrupts/sec
MemoryPage Reads/sec
Page Writes/sec
Available KBytes
Logical DiskDisk Read Bytes/sec
Disk Write Bytes/sec
% Disk Time

Data collection

The resource monitoring data displayed by Performance Tester when using Windows Performance Monitor is collected by leveraging the existing measurement infrastructure already present in the Microsoft Windows operating system. Therefore, this data collector is considered to be agent-less, because a user of Performance Tester does not have to install any additional software to the operating system that is being monitored.

Performance counter data is stored within the Windows Registry and can be programmatically accessed using two methods (Figure 5):

  • The first method is to access the Windows Registry directly, however, this approach can be complex and cumbersome.
  • The second method, which is the recommended method, is to use the Performance Data Helper (PDH) API. The PDH interface is an abstraction of the details required to retrieve performance object and performance counter data from the Windows Registry. In addition, the interface automatically returns values that are adjusted for appropriate units and scale. The interface is also the foundation for the Windows Performance Monitor (also known as Perfmon).

Figure 5. Microsoft Windows performance monitoring stack architecture
image of stack architecture

Physical devices (such as processors, memory, and network cards) and applications (such as operating systems, databases, servers, and third-party drivers) can have special counters built into them that are monitored by the Windows Kernel. This approach allows for extensibility of the standard set of countered monitored by the operating system alone.

Moreover, any application already using the Windows Registry or the PDH interface automatically are able to monitor these special counters, once the counters have been registered for collection.

In the case of Performance Tester, a custom application (pdh.dll) was built using the PDH interface to enable data collection from systems running Microsoft Windows. The custom application integrates with the Eclipse Platform in the form of a plug-in extension to Performance Tester. This plug-in interacts with the custom application using the Java Native Interface (JNI).

System configuration

In this section we describe how to configure the system for the Windows Performance Monitor.

Performance detail level

The measurement infrastructure contains approximately one thousand performance counters, and the quantity is increasing with newer releases of Microsoft Windows. Presenting a user with a large volume of counters poses a usability issue, and therefore, there is a configuration parameter that should be used with the PDH interface to help limit the number of performance counters presented to the user.

The following is a list of the different performance detail levels available from the PDH interface.

  • Novice: Indicates that this counter may be meaningful to most users. This is the most common counter detail level.
  • Advanced: Indicates that this counter is likely to be useful only to advanced users.
  • Expert: Indicates that this counter is likely to be useful only to the most advanced users.
  • Wizard: Indicates that this counter is not likely to be useful to any users.

In the case of Performance Tester, the custom application (pdh.dll) uses the Advanced detail level. This detail level includes generic counters (including processor, disk, memory, and more) and specific counters (including network card interfaces). In this release, this detail level is not configurable.

Network interface

A user of the Windows Performance Monitor data collector must ensure that the network interface on the physical machine that data is being collected from is configured correctly. Users must ensure that the connected network interface has File and Printer Sharing for Microsoft Networks enabled (See Figure 6).


Figure 6. Ethernet network connection properties in Microsoft Windows
image of dialog box

This is required because the PDH uses the Windows net use command to establish a connection to remote machines for data collection. This service can be enabled by opening the Properties dialog on a network connection listed in the Control Panel of any Microsoft Windows system.

A user of Performance Tester can collect performance counter data from any system running the Microsoft Windows operating system, only when the Performance Tester workbench client is executed on a Microsoft Windows operating system. This limitation is present since the PDH uses the operating systems at the client and target endpoint for communication.

Configuring a performance schedule

To configure resource monitoring using Windows Performance Monitor, create a new or add an existing location to a given performance schedule.

Enter a target Windows host name from which to collect data from and select Windows Performance Monitor from the list of available data sources (Figure 3).

Note: The host you want to monitor must be accessible through the Windows network. Typically, if you are able to connect to a shared hard disk drive on the remote host from the workbench, then you will also be able to use the remote host as a source of resource monitoring data. If the remote host is not accessible through the Windows network, an error message will display when you attempt to use the Resource page to select the type of data that you want to capture.

On the Location tab, type the user name, password, and domain (Figure 7).

  • The user name and password must match to a Windows user account on the target host system.
  • The domain is optional, only required if you need to perform cross-domain authentication.
  • Select Save Password to save your password locally. If you do not save your password, you might be prompted for it (depending on the host system configuration) when editing the configured location or when running performance schedules that use the location.

Figure 7. Windows Performance Monitor Location configuration
image of dialog box

On the Resource tab, select the type of data that you want to capture (Figure 8). The tree view shows the host and all of its respective counter groups and counters. The tree is organized by performance objects, denoted by folder icons, and performance counters, denoted by stop-watch icons.


Figure 8. Windows Performance Monitor Resource tab
image of dialog box

Clearing the Show only selected counters checkbox allows you to see all available counters (Figure 9). Be selective; monitoring all possible resource data requires substantial amounts of memory. Hover over a counter with your mouse to see details about what that counter measures.


Figure 9. Windows Performance Monitor Resource tab showing all available counters
image of dialog box

On the Options tab, the time interval properties can be configured as follows (Figure 10):

  • Enter the Polling Interval (in seconds), for collecting resource data. For example, if you accept the default of 5 seconds, counter information will be collected at 5 second intervals from the specified host during the schedule run.
  • Enter the Timeout Interval (in seconds). If the resource monitoring host does not respond within this amount of time during a schedule run, an error is logged.

Figure 10. Windows Performance Monitor Options tab
image of dialog box

Once the configuration is complete and saved, the resource monitoring location is added to the performance schedule (Figure 11).


Figure 11. Resource Monitoring configured using Windows Performance Monitor
image of dialog box


Back to top


Monitoring using rstatd for Linux/UNIX systems

The rstat daemon (herein referred to as rstatd and sometimes referred to as rpc.rstatd) is a Remote Procedure Call (RPC) server that returns performance statistics for a UNIX-based operating system. The performance statistics are retrieved from the kernel itself.

This daemon may already be installed and running on most Solaris and Linux installations, however, this configuration will depend on the vendor and distribution. The rstat daemon is normally started by the inetd daemon.

Performance counters

The statistics monitored are restricted to fifteen performance counters, which include processor, disk, memory, and network utilization. The available counters are shown in Table 2:


Table 2. Performance counters for the rstat daemon on Linux/UNIX
Performance counterDescription
Percentage User CPU Time Percentage user time is the percentage of non-idle processor time spent in user mode.
Percentage IOWAIT CPU Time Percentage IOWAIT time is the percentage of non-idle processor time spent waiting for I/O to complete.
Percentage of System CPU Time Percentage system time is the percentage of non-idle processor time spent in system mode.
Percentage of Idle CPU Time Percentage of time that the processor is idle.
Total Disk Transfers per second Total number of disk transfers on each of the disk interfaces (per second).
Total VM Pages Paged IN per second Total number of pages paged in per second. (Paging vs. swapping: see Note 1, below)
Total VM Pages Paged OUT per second Total number of pages paged out per second. (Paging vs. swapping)
Total VM Pages Swapped IN per second Total number of pages swapped in per second. (Paging vs. swapping)
Total VM Pages Swapped OUT per second Total number of pages swapped out per second. (Paging vs. swapping)
Total Interrupts per second Total number of interrupts per second.
Total Inbound packets on all interfaces per second Total number of inbound packets on all interfaces per second
Total Inbound errors on all interfaces per second Total number of inbound errors on all interfaces per second
Total Outbound packets on all interfaces per second Total number of outbound packets on all interfaces per second
Total Outbound errors on all interfaces per second Total number of outbound errors on all interfaces per second
Total collisions seen on all interfaces per second Total number of collisions seen on all interfaces per second
Total context switches per second Total number of context switches per second
Average number of jobs in run queue (1 minute average) Number of jobs in the run queue averaged over the last 1 minute. See Note 2, below
Average number of jobs in run queue (5 minute average) Number of jobs in the run queue averaged over the last 5 minutes.
Average number of jobs in run queue (15 minute average) Number of jobs in the run queue averaged over the last 15 minutes.

Notes:

  1. Paging is a light-weight mechanism whereby the operating system reads/writes pages of memory from/to the disk; swapping is similar to paging, but is more heavy-weight in that entire processes can have all their pages read/written from/to disk.
  2. The jobs on the run queue are the processes that are ready to run on the processor.

Data collection

The rstat daemon is a native application that specifically operates on UNIX-based systems, such as Solaris, HP-UX, AIX, and Linux. Performance Tester integrates with the daemon through a Java-based RPC client, which is responsible for marshalling transactions to and processing the results from a rstat daemon. The RPC client is bundled within an Eclipse plug-in that extends the Resource Monitoring platform within Performance Tester.

When performance counters are selected to be monitored, a Remote Service Monitor (RSM) program polls the counters at an interval and sends alerts to the client upon detection. The RSM program sends resource data to the client at reporting intervals set by the client or system administrator.

Depending on the setup of your account and the UNIX-based operating system, the system administrator might control the location and maintenance of the RSM programs that monitor your account. The RSM program obtains the performance data from the rstatd server, which returns performance statistics obtained from the kernel.

The version of rstatd that is supported in Performance Tester is the RSTATVERS_TIME version, or version 3. The rstatd is backwards compatible, therefore, versions greater than 3 will support the RSTATVERS_TIME edition. Historically, this version has been in existence well before 1995.

System configuration

In this section we describe how to configure the system for the rstat daemon.

Installing rstatd on Linux

The rstat daemon is freely available on the Internet for the Linux platform, and is already bundled in most UNIX-based platforms. The following are installation instructions for rstatd on Linux using the open source version of rstatd available on SourceForge:

  • Go to the Rstatd 4 Linux site and download the latest version:
  • Extract the archive downloaded from the previous step, and follow the instructions in the install file. This file will be available after extracting the archive. Installation will require tools such as make and gcc, which are usually installed by default with most Linux distributions.
  • The next step requires verifying that the rstat daemon gets started through inetd daemon (or xinetd on some Linux distributions). Using a terminal, change directories to /etc/xinetd.d and create a file called rstatd, with the following contents:

Creating the rstatd file
                
     
service rstatd {
	type = RPC
	socket_type = dgram
	protocol = udp
	server = /usr/local/sbin/rpc.rstatd
	wait = yes
	user = root
	rpc_version = 2-4
	disable = no
}

Ensure that the value specified by the server parameter is actually where the rpc.rstatd daemon is installed. If it is not installed in /usr/local/sbin change the parameters above to reflect this configuration.

  • Restart the inetd (or xinetd) daemon:

Restarting the inetd daemon
                     
/etc/rc.d/init.d/xinetd restart

  • Now rstatd should be running. Running the following command will help verify that rstatd is actually started:

Verifying that the rstatd has started
                     
rpcinfo -p localhost

The daemon should be listed several times, which is due to the different RPC versions available for clients to use.

Portmapper service

The rstatd uses the Remote Procedure Call (RPC) system to facilitate connecting to the system portmapper daemon running on a well-known port. The rstatd will use RPC to create a listener that asks the portmapper for a port to use for communication with a remote client. The portmapper selects an unused port and assigns it to the listener.

Performance Tester requires the portmapper daemon to be running on the remote Linux/UNIX system. Run the following command to verify that the daemon is started:


Verifying that the rstatd has started
                     
rpcinfo -p localhost

This command will generate output similar to the following output:


Typical output of rpcinfo command
                     

program vers proto   port
100000    2   tcp    111  portmapper
100000    2   udp    111  portmapper
100024    1   udp  32768  status
100024    1   tcp  32769  status
100001    2   udp  32770  rstatd
100001    3   udp  32770  rstatd
100001    4   udp  32770  rstatd

The output indicates that the portmapper is operating using the TCP and UDP protocols using port 111, the default port.

If the portmapper daemon is not listed in the output of the rpcinfo command, as shown above, the daemon may not be installed or not configured to run. It is recommended to contact your system administrator and request that the portmapper RPC program be started.

To view the most commonly used RPC program numbers is held in a file called /etc/rpc on each system. The file consists of a series of entries, one per service, such as:


Typical RPC program numbers
                
	portmapper      100000  portmap sunrpc
	rstatd          100001  rstat rstat_svc rup perfmeter
	rusersd         100002  rusers
	nfs             100003  nfsprog
	ypserv          100004  ypprog
	mountd          100005  mount showmount
	ypbind          100007
	walld           100008  rwall shutdown
	yppasswdd       100009  yppasswd
	etherstatd      100010  etherstat
	rquotad         100011  rquotaprog quota rquota
	sprayd          100012  spray
	3270_mapper     100013
	rje_mapper      100014
	selection_svc   100015  selnsvc
	database_svc    100016

Configuring a performance schedule

To configure resource monitoring using UNIX rstatd monitor, create a new or add an existing location to a given performance schedule.

Enter a target Linux/UNIX host name from which to collect data from and select the UNIX rstatd monitor item from the list of available data sources (Figure 3).

On the Resource page, select the type of data you want to capture (Figure 12). The tree view shows all available performance counters, with a default set of counters pre-selected.


Figure 12. UNIX rstatd monitor Resource tab
image of dialog box

Clearing Show only selected counters allows you to see all available counters (Figure 13). Be selective; monitoring all possible resource data requires substantial amounts of memory. Hover over a counter with your mouse to see details about what that counter measures.


Figure 13. UNIX rstatd monitor Resource tab showing all available counters
image of dialog box

On the Options tab, the time interval properties can be configured (Figure 14):

  • Change the Connection information if needed. Typically, you only need to change this information if a firewall is blocking the default port. If the portmapper has be configured to use a different Protocol or Portmapper Port number, this area allows for custom configuration to be applied for the specific target Linux/UNIX host being monitored.
  • Enter the Polling Interval (in seconds), for collecting resource data. For example, if you accept the default of 5 seconds, counter information will be collected at 5-second intervals from the specified host during the schedule run.
  • Enter the Timeout Interval (in seconds). If the resource monitoring host does not respond within this amount of time during a schedule run, an error is logged.

Figure 14. UNIX rstatd monitor Options tab
image of dialog box

Once configuration is complete and saved, the resource monitoring location is added to the performance schedule (Figure 15).


Figure 15. Resource Monitoring configured using UNIX rstatd monitor
image of dialog box


Back to top


Monitoring using IBM Tivoli Monitoring

IBM Tivoli Monitoring monitors and manages system and network applications on a variety of platforms and keeps track of the availability and performance of all parts of your enterprise. IBM Tivoli Monitoring provides reports you can use to track trends and troubleshoot problems.

Components of IBM Tivoli Monitoring

The system architecture of IBM Tivoli Montoring is shown in Figure 16:


Figure 16. IBM Tivoli Monitoring system architecture
image of diagram

IBM Tivoli Monitoring consists of the following components:

  • A Tivoli Enterprise Monitoring Server (referred to as the monitoring server), which acts as a collection and control point for alerts received from the agents, and collects their performance and availability data. There are two types of monitoring servers: hub and remote.
  • A Tivoli Enterprise Portal Server (referred to as the portal server), placed between the client and the monitoring server, that enables retrieval, manipulation, and analysis of data from the agents.
  • A Tivoli Enterprise Portal Client, with a Java-based user interface for viewing and monitoring your enterprise. Tivoli Enterprise Portal offers two modes of operation: desktop and browser.
  • Monitoring Agents installed on the systems or subsystems you want to monitor. These agents collect and distribute data to a monitoring server.
  • Tivoli Data Warehouse for storing historical data collected from agents in your environment. The data warehouse is located on a DB2®, Oracle®, or Microsoft SQL Server database. To collect information to store in this database, you must install the Warehouse Proxy agent. To perform aggregation and pruning functions on the data, install the Warehouse Summarization and Pruning agent.
  • Tivoli Enterprise Console event synchronization component (not shown) for synchronizing the status of situation events that are forwarded to the event server. When the status of an event is updated because of Tivoli Enterprise Console rules or operator actions, the update is sent to the monitoring server, and the updated status is reflected in both the Situation Event Console and the Tivoli Enterprise Console event viewer.

Monitoring agents

Today, there are over one hundred ITM monitoring agents available from IBM and third-party software vendors. Each agent is responsible for data collection from systems, subsystems, or applications. The collected data is passed from the monitoring agents to the portal server through the monitoring server. An agent interacts with a single system or application and, in most cases, resides on the same computer where the system or application is running. There are two types of monitoring agents:

  • Operating system (OS) agents that monitor the availability and performance of the computers in your monitoring environment. An example of an OS agent is the Monitoring Agent for Windows OS, which monitors Windows XP, Windows 2000, and Windows 2003 operating systems.
  • Other agents (referred to as application agents or non-OS agents) that monitor the availability and performance of systems, subsystems, and applications. An example of a non-OS agent is IBM Tivoli Monitoring for Microsoft Exchange, which monitors the Microsoft Exchange Server.

IBM Tivoli Monitoring also provides a customized agent called the Universal Agent. The Universal Agent can monitor any data that is collectable in a given environment. For example, the Universal Agent can monitor the status of your company website to ensure that it is available.

Agents run on z/OS®, UNIX, Windows XP Professional Edition, Windows 2000, Windows 2003 Server, HP-UX®; however, not all agents are supported on all platforms.

Each agent has special metadata that describes itself. This metadata is required when integrating data collection into a custom client, such as Rational Performance Tester. Today, Performance Tester is capable of data collection from the following ITM Monitoring Agents:

  • Operating system agents
  • Monitoring Agent for Windows OS
  • Monitoring Agent for Linux OS
  • Monitoring Agent for UNIX OS
  • Monitoring Agent for z/OS
  • Application agents
  • Monitoring Agent for Citrix
  • Monitoring Agent for IBM DB2
  • Monitoring Agent for IBM Tivoli Composite Application Manager for WebSphere
  • Monitoring Agent for IBM WebSphere Application Server
  • Monitoring Agent for IBM WebSphere MQ
  • Monitoring Agent for Oracle Database
  • Monitoring Agent for SNMP-MIB2 (only)

Data collection

Retrieving data from ITM is accomplished by making Simple Object Access Protocol (SOAP) requests to the embedded SOAP server contained within the Tivoli Enterprise Monitoring Server. It is important to note that in the IBM Tivoli Monitoring documentation, the SOAP server is often referred to as a Web service, even though there is no Web Service Definition Language (WSDL) publicly published.

Once data is retrieved using SOAP requests, the well formed XML responses are transformed into the Eclipse Test and Performance Tools Platform (TPTP) statistical model format. This model format is based on the Eclipse Modeling Framework (EMF) and is designed as a generic method of storing and representing statistical data, despite which system or data collector the data points are observed from. The generic model format allows Performance Tester and other Eclipse-based views to render reports based on this data.

As each event (a statistical data point) is observed, it is transformed into a TPTP Statistical model event on-the-fly. This event is then loaded into the TPTP Data Loader, which persists the event and triggers any registered user interfaces to update based on this new event.

The integration between Performance Tester and IBM Tivoli Monitoring is shown in Figure 17.


Figure 17. IBM Rational Performance Tester integration with IBM Tivoli Monitoring
image of diagram

There are two mechanisms by which statistical data can be collected from ITM:

  • Real-time: Real-time data collection requires making SOAP requests to the monitoring server requesting for the most recently acquired data point.
  • Importing historically: Importing historical statistical data is possible because ITM is a managed solution, where the data observed over time is persisted to the Tivoli Data Warehouse.

A SOAP request to the monitoring server consists of:

  • Sending the XML fragment as the entity-body of an HTTP POST request to the URL: http://<MONITORING_SERVER>:1920///cms/soap/kshhsoap.htm
    • <MONITORING_SERVER> refers to the name of the host on which the monitoring server process is running.
    • The three slashes (///) in the URL are required by the monitoring server.
    • Setting the following properties in the HTTP header:
    • interfacename
    • CT_SOAP
    • messagetype: Call

Here is a sample HTTP POST request that can be used to query ITM for some data.


Sample HTTP POST request
                
POST http://localhost:1920///kdsmain/soap HTTP/1.0
Accept: */*
Accept-Language: en-us
Referer: http://localhost:1920///kdsmain/soap/kshhsoap.htm
interfacename: CT_SOAP
messagetype: Call
Content-Type: text/xml
Proxy-Connection: Keep-Alive
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;
 .NET CLR 1.1.4322;.NET CLR 1.0.3705)	
Host: localhost:1920
Content-Length: 96
Pragma: no-cache

	<CT_Get><userid>sysadmin</userid><password></password>
	<object>Web_Applications</object></CT_Get>
     

  • CT_Get is a command that the ITM SOAP server understands and declares the opening of a new transaction query.
  • userid and password are used for authenticating with the SOAP server.
  • object refers to the ITM object name for which we are querying (explained in more detail in the next section). In order to get the available agents, the ITM Object name is set to ManagedSystem.

Types of queries

In this section we list three types of queries.

Query monitor agents

Querying for monitoring agents that are available to the indicated monitoring server and collecting data:


Example monitoring agent query
                
<CT_Get><userid>sysadmin</userid><password></password>
	<object>ManagedSystem</object>
</CT_Get>
      


Example response
                
<?xml version="1.0" encoding="ISO-8859-1"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
  SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
 <SOAP-ENV:Body><SOAP-CHK:Success xmlns:SOAP-CHK = "http://soaptest1/soaptest/"
  xmlns="urn:candle-soap:attributes">
<TABLE name="O4SRV.INODESTS">
	<OBJECT>ManagedSystem</OBJECT>
	<DATA>
		<ROW>
		<Timestamp>1050620105552004</Timestamp>
		<Name>RDANEK:UA</Name>
		<Managing_System>HUB_RDANEK</Managing_System>
		<ORIGINNODE>RDANEK:UA</ORIGINNODE>
		<Reason>FA</Reason>
		<Status>*OFFLINE</Status>
		<Product>UM</Product>
		<Version>04.10.00</Version>
		<Type>V</Type>
		<HLOCFLAG>L</HLOCFLAG>
		<HHOSTINFO>WinXP~5.1-SP1</HHOSTINFO>
		<HHOSTLOC></HHOSTLOC>
		<CT_Host_Address>ip:#169.254.198.66[38213]
		&lt;NM&gt;RDANEK&lt;/NM&gt;</CT_Host_Address>
		</ROW>
	</DATA>
</TABLE>
</SOAP-CHK:Success></SOAP-ENV:Body></SOAP-ENV:Envelope>

The response is returned in a well-formed XML document structure. Now let us dissect the response:

  • The OBJECT element echoes the same ITM object name that was sent in the initial SOAP.
  • The DATA contains the result data of the query, which is of most interest to us. Each separate XML fragment of data is enclosed in a ROW element.
  • Within each ROW element are the attributes which are known about for the given object. For example, Timestamp, Name, Managing_System, are some of the available attributes.
  • The NAME element contains the name of the monitoring agent.
  • The CT_Host_Address contains the host and/or address on which the Monitoring Agent is running.

Query recent data

Querying for recent data allows clients to retrieve the most recently acquired statistical data point by the monitoring server for a monitoring agent. This query would be used in the live monitoring scenario where periodically the statistical model is updated with new data points:


Example data query
                

<CT_Get><userid>sysadmin</userid><password></password>
	<target>Primary:RDANEK:NT</target>
	<object>NT_Process</object>
	<attribute>Thread_Count</attribute>
	<afilter>ID_Process;EQ;988</afilter>
</CT_Get>

  • A TARGET is the name of the monitoring agent as returned in the NAME element when querying for the available agents on a monitoring server.
  • An OBJECT indicates the ITM Object that is being queried for data. This element is similar to a performance object in the Windows Management Infrastructure (WMI).
  • (optional) ATTRIBUTE elements are the Performance Counters belonging to the ITM object (or performance object). For example, the NT_Process object has attributes named Process Name, % Processor time, % User Time. By specifying one or more attributes, one limits the results returned to the specified attributes.
  • Filters, indicated by the AFILTER element, allow the query to specify the values of attributes for which the query should return data for. For example, when querying the NT_Process object, one may only want data for a certain process.

This example query returns the value of thread count for the process with an identifier of 988.


Example data query response
                
<?xml version="1.0" encoding="ISO-8859-1"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
  SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
   <SOAP-ENV:Body><SOAP-CHK:Success xmlns:SOAP-CHK = "http://soaptest1/soaptest/"
		 xmlns="urn:candle-soap:attributes">
	<TABLE name="KNT.WTPROCESS">
	<OBJECT>NT_Process</OBJECT>
		<DATA>
		<ROW>
		<Thread_Count dt="number">24</Thread_Count>
		</ROW>
		</DATA>
	</TABLE>
</SOAP-CHK:Success></SOAP-ENV:Body></SOAP-ENV:Envelope>

The response is returned in a well-formed XML document structure. Now let us dissect the response:

  • The OBJECT element echo's the same ITM Object name that was sent in the initial SOAP.
  • The DATA contains the result data of the query, which is of most interest to us. Each separate XML fragment of data is enclosed in a ROW element. In the example response, the value of the Thread count for the NT_Process object matching the process with an identifier of 988 is 24.

Query historical data

Querying for historical data allows the retrieval of statistical observations collected in the past for a monitoring agent. In order for this type of data collection to be enabled, historical data collection must have been configured for the desired monitoring agent through the monitoring server. The queries are identical to the preceding section except that we specify an additional tag in the query: <history>Y</history>:


Example historical data query
                
<CT_Get><userid>sysadmin</userid><password></password>
	<target>Primary:RDANEK:NT</target>
	<object>NT_Process</object>
	<attribute>Timestamp</attribute>
	<attribute>Thread_Count</attribute>
	<history>Y</history>
	<afilter>Timestamp;GE;1050620132400000</afilter>
</CT_Get>

This query will return the thread count for all observations after June 20th, 2005 at 1:24 PM. A filter is used to specify a timestamp in order to only retrieve the historical data for the time period that is of interest. Querying for historical data without a timestamp filter will result in the SOAP server returning all historical data and this can overwhelm the system resources.

The timestamp is of the format CYYMMDDHHMMSSmmm:

  • C is the century guard
  • YY is the year
  • First MM is the month
  • DD is the day
  • HH is the hour
  • Second MM is the minute
  • SS is the seconds
  • mmm is the milliseconds

Example historical data response
                
<?xml version="1.0" encoding="ISO-8859-1"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
  SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
	> <SOAP-ENV:Body><SOAP-CHK:Success xmlns:SOAP-CHK =
	 "http://soaptest1/soaptest/"xmlns="urn:candle-soap:attributes">
<TABLE name="KNT.WTPROCESS">
	<OBJECT>NT_Process</OBJECT>
	<DATA>
		<ROW>
		<Timestamp>1050620132453887</Timestamp>
		<Thread_Count dt="number">1</Thread_Count>
		</ROW>
		<ROW>
		<Timestamp>1050620132453887</Timestamp>
		<Thread_Count dt="number">64</Thread_Count>
		</ROW>
		<ROW>
		<Timestamp>1050620132453887</Timestamp>
		<Thread_Count dt="number">3</Thread_Count>
		</ROW>
	</DATA>
</TABLE>
</SOAP-CHK:Success></SOAP-ENV:Body></SOAP-ENV:Envelope>

Configuring a performance schedule

To configure resource monitoring using IBM Tivoli Monitoring, create a new or add an existing location to a given performance schedule.

Enter a target host name, where the ITM Monitoring Agent resides, from which to collect data from and select the IBM Tivoli Monitoring item from the list of available data sources (Figure 3).

On the Tivoli Enterprise Monitoring Server tab, specify the monitoring server that you want to use to capture resource monitoring data as shownin Figure 18:

  • Type the IP address or the fully qualified host name of the monitoring server in the Host field on the Tivoli Enterprise Monitoring Server tab. This is different from the Host field at the top of the Create and manage configurations wizard, which indicates the monitoring agent.
  • Type the user name and password for the monitoring server in Authentication.
  • Change the Connection information if needed. Typically, your Tivoli system administrator will specify this information.
  • Select Save Password to save your password locally. If you do not save your password, you might be prompted for it (depending on the host system configuration) when editing the configured location or when running test schedules that use the location.

After you have specified the monitoring server, you can select resources to capture. If the host is not managed by the monitoring server, you will see an error message.


Figure 18. IBM Tivoli Monitoring Tivoli Enterprise Monitoring Server tab
image of dialog box

On the Resource tab, select the type of data that you want to capture, as shown in Figure 19. The tree view shows the host and all of its available IBM Tivoli Monitoring agents, and their respective counter groups and counters.


Figure 19: IBM Tivoli Monitoring Resource tab
image of dialog box

Clearing Show only selected counters allows you to see all available counters (Figure 20). Be selective; monitoring all possible resource data requires substantial amounts of memory. Hover over a counter with your mouse to see details about what that counter measures.


Figure 20: IBM Tivoli Monitoring Resource tab showing all available counters
image of dialog box

On the Options tab, the time interval properties can be configured (Figure 21):

  • Enter the Polling Interval (in seconds), for collecting resource data. For example, if you accept the default of 5 seconds, counter information will be collected at 5-second intervals from the specified host during the schedule run.
  • Enter the Timeout Interval (in seconds). If the resource monitoring host does not respond within this amount of time during a schedule run, an error is logged.

Figure 21: IBM Tivoli Monitoring Options tab
image of dialog box

Once configuration is complete and saved, the resource monitoring location will be added to the performance schedule (Figure 22).


Figure 22: IBM Tivoli Monitoring configured for resource monitoring
image of dialog box

Profiling for real-time observation

Performance Tester has the ability to collect from ITM monitoring agents, in real-time, without having to configure a resource location through a performance schedule. This capability is valuable for large enterprise environments where different machines are used for performance testing and system monitoring. In addition, even though this resource data is collected outside of a performance schedule, you can overlay these counters using the Performance Test Runs view of an existing Performance Tester Report.

To initiate real-time data collection from an ITM monitoring agent, select Run -> Profile or use the toolbar button to open the launch configuration dialog (Figure 23). This profile action will ask you to switch into the Profile and Logging perspective of the Performance Tester product. This perspective enables profiling tools and has a different layout of the views on the workbench.


Figure 23: Profile launch configuration action
image of dialog box

Resource monitoring configuration using the performance schedule is similar to configuring real-time monitoring (Figure 24). One difference is that you can specify where the collected data should be stored using the Destination tab. Clicking Profile after the tab pages have been completed will initiate data collection.


Figure 24: Create and run a real-time IBM Tivoli Monitoring configuration
image of dialog box

As data is collected from the selected ITM Monitoring Agents, it is plotted on a graph display (Figure 25). As each new data point is observed and received by the client, the point will be plotted. The graph display is available by right-clicking on the agent in the Profiling Monitor view. Figure 25 illustrates the trend between the network traffic, disk activity, and processor activity.


Figure 25: Analyze statistical resource information collected in real-time using IBM Tivoli Monitoring
image of dialog box

Importing data from IBM Tivoli Monitoring

IBM Tivoli Monitoring is a managed solution that store all observed monitoring data in a data warehouse. This architecture allows you to query ITM for historical data from a particular agent, if needed.

In the workbench client, you can import the data using two methods:

  1. An import wizard that is accessible by selecting File -> Import (Figure 26>).
  2. An import wizard that is accessible by right-clicking on a performance report in the Performance Test Runs view (Figure 27) or on the resources report, which is the last tab in the view.

Figure 26: Performance Tester import wizard
image of dialog box

Figure 27: Performance result right-click menu
image of dialog box

Once the wizard is loaded, you must specify a resource location configured using the ITM data source (Figure 28).


Figure 28: Resource Monitoring import wizard
image of dialog box

From the three data collectors offered in Performance Tester, only ITM is capable of collecting historical data. If you try to add a resource location configured for any other data source the collection for these locations will be ignored during import and the user will be alerted (Figure 29).


Figure 29: Warning indicating that only historical data sources can be used for importing data
image of dialog box

The next stage of the import wizard is to specify the time period from which to import data (Figure 30). This can be specified as either the start and end times or as the amount of time you specify, counted backward from the time that you click Finish. You might need to adjust the time interval to compensate for clock discrepancies between the workbench and the monitoring server.

Note: When specifying the time in a specific number of units, note that, for consistency, month is defined as 30 days and year is defined as 365 days. Days refers to 24-hour periods, not calendar days. The units of time selected are subtracted from the point in time when you click Finish to import the data to give the start time. For example, if you select 2 months, the time period will be the 60 days (24-hour time periods) immediately prior to the point in time that you click Finish.


Figure 30: Resource Monitoring import constraints
image of dialog box

To import data from ITM, the ITM monitoring agent itself must be configured for historical data collection. If the agent is not properly configured, an error message will be displayed to the user when attempting the import, as shown in Figure 31.


Figure 31: Error that ITM Monitoring Agent is not configured for historical data collection
image of dialog box

After the import process has completed, the data is displayed in one of two ways, depending on how the import wizard was invoked:

  1. If the user invoked the wizard from the File -> Import menu, this action indicates that there is no association to a performance result and the data is displayed using the default statistical view in the Profile and Logging perspective (Figure 32).
  2. If the user invoked the wizard by right-clicking on a performance result, this action indicates the imported data is associated to the selected performance result. The data can be viewed using the Resource tab in the default report when the performance result is opened. Alternatively, if you manually switch to the Profile and Logging perspective, you can view and analyze the data using the default statistical view in the Profile and Logging perspective.

Figure 32: Display and analyze statistical resource information imported from IBM Tivoli Monitoring
image of dialog box


Back to top


Customizing reports and overlaying counters

Performance Tester's report infrastructure allows for display of rich interactive graphs and tables. To aid in problem determination, each graph in the report infrastructure has been enhanced to allow resource data to be placed over the already rendered graphs. This action allows one to correlate two separate data sets, such as page response time and % processor time. Correlation of resource data with response time data help an analyst to determine if there is an issue with the system, capacity of the environment, or the application itself.

There are two ways to overlay counters on a report:

  1. Add performance counters to an existing graph.
  2. Drag-and-drop counters onto an existing report.

Add performance counters to an existing graph

Right-click on an existing graph and select Add/Remove Performance Counters -> Resource Monitoring Counter (Figure 33) to display the Resource Counters configuration dialog. This menu item is available from any graph in the Performance Report view.


Figure 33: Overlay resource monitoring counter on an existing report
image of dialog box

The Resource Counters configuration dialog (Figure 34) allows you to select specific counters and adjust the scaling factor for each counter. Adjusting the scaling factor for a counter is important when trying to analyze correlation between counters that have different data types. For example, processor time is a percentage quantity where as memory is measured in bytes.


Figure 34: Resource Counters configuration dialog
image of dialog box

Once the counters are selected and the configuration dialog is complete, the counters are drawn on the selected graph (Figure 35). Hovering over any data point using the mouse will provide a tool-tip containing information about the data point, such as the value of the observation.


Figure 35: Response vs. Time summary correlated with the % Processor Time resource counter
image of dialog box

Drag-and-drop counters onto an existing report

Drag-and-drop counters from the Performance Test Run view onto an existing report, as shown in Figure 36. Expanding the tree control in this view allows the user to select specific counters. The tree is organized by the host name, under which are listed the data collection agents, such as UNIX rstatd monitor.


Figure 36: Drag-and-drop a resource counter from the Performance Test Run view
image of dialog box

A suggested report to modify with user-defined counters is the Resources report, the last tab in the Performance Report view, as shown in Figure 37. This report spans the entire view allowing more working area than other reports in the view.

The layout of the graph can also be customized using the Customize menu item when you right-click on the graph. This action allows you to set the X and Y axis limits, legend, and change the time range.


Figure 37: Resources report in a performance report
image of dialog box


Resources

Learn

Get products and technologies

Discuss


About the author

Russell Butek is one of the developers of IBM's WebSphere Web services engine. He is also IBM's representative on the JAX-RPC Java Specification Request (JSR) expert group. He was involved in the implementation of Apache's AXIS SOAP engine, driving AXIS 1.0 to comply with JAX-RPC 1.0. Previously, he was a developer of IBM's CORBA ORB and an IBM representative on a number of OMG task forces: the portable interceptor task force (of which he was chair), the core task force, and the interoperability task force.




Rate this page


Please take a moment to complete this form to help us better serve you.



YesNoDon't know
 


 


12345
Not
useful
Extremely
useful
 


Back to top