Editor's Note: This article applies to IBM Rational® Robot and IBM Rational® TestManager. It does not apply to IBM Rational® Performance Tester 6.1.

The business challenge

Rational Robot and Rational TestManager provide a majority of the features that are required to address the needs of an automated performance testing solution. Business transactions are automated using test scripts, end-user data access patterns are modeled effectively, and an application's workload profile is translated into maintainable and scalable test schedules that characterize an end-user's experience as related to the load placed against the application under test. End-user perceived response times are collected as related to the application's business processes and the reliability of the application's business processes is properly measured to determine if critical functions are failing under system load.

However, there remains a need to collect system resource usage data from all of the tiers within the application (for example, operating systems, application servers, database servers, network links, and so on), and these tools fall short on features. Software must be installed on any tier that is monitored and the operating system is the only layer that can be observed and collated along with the other performance related results. Ideally, a performance testing solution for monitoring application components should address the following requirements:

• Provide a mechanism for obtaining system resource usage from all the components within the application under test. This includes the operating system on each tier, as well as the software running on the different tiers; such as the vendor-specific Web server, application server, and database server.
• Support a non-intrusive or agentless solution for data collection, such that no additional software needs to be installed on the target tiers in the application.
• Present real-time updates while a test is running, so that the immediate visualization of any performance bottlenecks can be addressed, rather than waiting until a test is complete.
• The collected data should be persistent and stored along with the test results and the data should be effectively visualized to convey the characteristics of the monitored application layers.

This article addresses the needs for resource monitoring within the aforementioned IBM Rational performance testing tools and provides a solution for implementing a non-intrusive monitoring architecture that can be used when running automated performance tests. Specifically, this article provides a solution for monitoring the resource usage within any Apache Web server component.

Back to top

A strategic approach

Exploit remote performance interfaces

To implement a non-intrusive component monitoring solution, it is necessary to understand that many hardware and software components provide interfaces that allow other remote devices to connect to the component for the purpose of querying its status. For instance, a UNIX operating system publishes its resource usage, such as CPU usage and system load, to a well-known interface published by a daemon called rstad. Therefore, with the knowledge of the RSTAT (remote statistics) interface, a RSTAT client can be used to query any component running a UNIX operating system to obtain real-time status of the resource usage of the hardware.

The Apache Web server software provides the ability to publish its internal resource usage to a well-documented Web-based interface that can be viewed using a standard Web browser. As an example of this functionality, the Apache Foundation's Web site shows how this feature has been enabled within its own implementation. A detailed status of the Apache Foundation's Web site can be queried using the URL, http://www.apache.org/server-status. A summary status of the Apache Foundation's Web site can be queried using the URL, http://www.apache.org/server-status?auto. The latter URL provides a brief summary of the status of the Apache Web server software, including the following metrics:

• Total Accesses: The total requests processed since the last time the Web server was rebooted
• Total kBytes: The total amount of data that has been transferred since the last time the Web server was rebooted
• CPULoad: The percentage of the CPU that the Apache software is using
• Uptime: The amount of time the Web server has been running since its last reboot
• ReqPerSec: The number of HTTP requests per second that the Web server is processing
• BytesPerSec: The amount of data the Web server is transferring per second
• BytesPerReq: The average number of bytes being transferred per HTTP request
• BusyWorkers: The number of Apache processes actively processing an HTTP request
• IdleWorkers: The number of idle Apache processes awaiting an HTTP request

With this knowledge, it is possible to ascertain the capability of the Apache Web server software to handle the current load of incoming requests for whatever services it is providing. For instance, if IdleWorkers is consistently zero (0), there are probably not enough Apache processes available to efficiently serve the rate of incoming client requests. The net result is that most likely the application's end-users are experiencing longer than normal response times and system capacity should be addressed.

Automate performance data collection

With the knowledge and understanding of the remote statistics URL for an Apache Web server, it is rather straight forward to create an automated process using a test script with the sole purpose of intermittently accessing the statistics URL, scraping the statistics from the Web page, and returning them to the testing tool. The IBM Rational performance testing tools provide a straightforward capture mechanism to automate the access to the statistics URL and a set of functions and variables that allow you to parse and manipulate the content residing within a published Web page.

Display and publish the results

After the performance statistics have been scraped from the Apache statistics Web page, publishing the statistics to the IBM Rational TestManager interface can enable real-time monitoring. Because the data can also be written to a log file, the statistics information can be imported into a statistical package, such as Microsoft™ Excel, and visually presented using graphing techniques.

Back to top

Implementation

Enabling the Apache Software

The first step to enable the performance monitoring capabilities within the IBM Rational performance testing tools is to make sure that the target Apache Web server is publishing the performance statistics to the statistics URL. The default URL is /server-status?auto. For example, if the server that is being monitored has the domain name www.widgets.com, the performance statistics would be accessed using the URL www.widgets.com/sever-status?auto. However, you must enable the ability to publish performance statistics within the Apache configuration file (httpd.conf). For instance, the configuration may look like the following for a Web server accessed using the host name www.widgets.com:

<Location /server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from www.widgets.com
</Location>
ExtendedStatus On

You can find more information about enabling this feature on the Apache Web site

Automating the performance data collection

After the Apache configuration file has been updated, the statistics can be queried by accessing the statistics URL from within any Web browser. You can perform this process manually while a test is running, but the data collection process would be very limited because it would require constant intervention, especially logging the data manually into a data file or spreadsheet. If the IBM Rational performance tool installation supports HTTP capture and playback, you can automate the data collection process. The steps to automate the process are as follows:

1. Record Statistics URL-- To capture the Web client request for the statistics URL, the virtual user recorder must be used to record the interaction with the Apache Web server, as well as create and register the test script within the testing tool. In the majority of cases, API or network recording will suffice in enabling the recording of the Web client's HTTP request. Simply turn the VU recorder on, launch a Web browser, navigate to the appropriate statistics URL, and close the Web browser. The recorded HTTP conversation should translate into a single HTTP request/response sequence within the test script.
2. Parse Performance Data from Statistics Web Page-- The VU scripting language facilitates access to the server response data of any client request using the read-only variable _response. Because Apache publishes its statistics in a standard format (specific string tokens indicate the metric being displayed and its value being delimited and terminated by a linefeed), each of its resource statistics can be scraped or parsed from the server response data using VU string parsing functions, such as the match() subroutine. With the aide of the match() subroutine, statistics, such as the number of Busy Servers, can be saved to variables that can be either published to the User Message View, or written to a data file, or both.
3. Publish Real-time Performance Data-- The VU scripting language also provides the display() subroutine that allows each virtual user to write transient data to the User Message View. Using this subroutine, a virtual user can write the current resource usage within Apache to the User Message View to provide real-time statistics in textual format. This message will remain within the User Message View until the virtual user writes another message at which point the previously displayed message will be overwritten.
4. Write Performance Data to a Log File-- After the virtual user has collected the current resource data, it can be easily written to a mutually exclusive log file using the printf() subroutine. This VU subroutine writes a message to the virtual user's standard out file that can be accessed after the test has completed within the test results directory. Each virtual user creates a log file with the naming convention o00N, where N is equal to the virtual user's unique ID. Therefore, if a 100-user test was executed, there would be log files called o001 through o100. Because this is a flat ASCII file, it is possible to import the log file into ancillary tools for analysis.

An example of the data collection automation is presented in the following code listings. Each listing is a different segment of the same VU test script.

#include <VU.h>

/* DEFINE PSEUDO DATA STRUCTURE TO HOLD STATS */
#define NUM_APACHE_STATS    9
string apacheStats[NUM_APACHE_STATS];

#define APACHE_TOTAL_ACCESSES     0
#define APACHE_TOTAL_BYTES        1
#define APACHE_CPU_LOAD           2
#define APACHE_UPTIME             3
#define APACHE_REQ_PER_SEC        4
#define APACHE_BYTES_PER_SEC      5
#define APACHE_BYTES_PER_REQUEST  6
#define APACHE_BUSY_WORKERS       7
#define APACHE_IDLE_WORKERS       8

/* BEGIN MAIN FUNCTION */
{
string serverResponse = "";
string realTimeStats = "";
int currentIndex = -1;
persistent int currentIteration = 0;

push Timeout_scale = 200; /* Set timeouts to 200% of maximum response time */
push Think_def = "LR";
Min_tmout = 120000;       /* Set minimum Timeout_val to 2 minutes          */
push Timeout_val = Min_tmout;
push Think_avg = 0;

for(currentIndex = 0;currentIndex < NUM_APACHE_STATS;currentIndex++)
{
    apacheStats[currentIndex] = "NULL";
}

Listing 2 presents the first portion of the automated test script used for the Apache performance monitoring. Here, variables are initialized including a string array that is used to store all of the Apache system statistics.

Listing 3 shows the actual HTTP client request for the Apache statistics URL. In this example, the automated test script is accessing the system statistics from the Apache Foundation's Web site. The server response is assigned to a local string variable that is parsed using the match() subroutine. The string token for each Apache statistic is used within the regular expression as the left boundary. The regular expression instructs the match() subroutine to grab all of the text up until, but not including, the linefeed and assigns the extracted value to the appropriate element within the array.

www_apache_org = http_request ["Apache001"] "www.apache.org:80",
        HTTP_CONN_DIRECT,
   "GET /server-status?auto HTTP/1.1\r\n"
   "Accept: */*\r\n"
   "Accept-Language: en-us\r\n"
   "Accept-Encoding: gzip, deflate\r\n"
   "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR"
   " 1.0.3705; .NET CLR 1.1.4322)\r\n"
   "Host: www.apache.org\r\n"
   "Connection: Keep-Alive\r\n"
   "\r\n";

set Server_connection = www_apache_org;
http_header_recv ["Apache002"]  200;    
http_nrecv ["Apache003"]  100 %% ;      
serverResponse = _response;
http_disconnect(www_apache_org);

/* PARSE STATS FROM THE STATISTICS URL & ASSIGN TO ARRAY */
match("Total Accesses: ([^\n]*)$0",serverResponse,&apacheStats[APACHE_TOTAL_ACCESSES]);
match ("Total kBytes: ([^\n]*)$0",serverResponse,&apacheStats[APACHE_TOTAL_BYTES]);
match ("CPULoad: ([^\n]*)$0",serverResponse,&apacheStats[APACHE_CPU_LOAD]);
match ("Uptime: ([^\n]*)$0",serverResponse,&apacheStats[APACHE_UPTIME]);
match ("ReqPerSec: ([^\n]*)$0",serverResponse,&apacheStats[APACHE_REQ_PER_SEC]);
match ("BytesPerSec: ([^\n]*)$0",serverResponse,&apacheStats[APACHE_BYTES_PER_SEC]);
match ("BytesPerReq: ([^\n]*)$0",serverResponse,&apacheStats[APACHE_BYTES_PER_REQUEST]);
match ("BusyWorkers: ([^\n]*)$0",serverResponse,&apacheStats[APACHE_BUSY_WORKERS]);
match ("IdleWorkers: ([^\n]*)$0",serverResponse,&apacheStats[APACHE_IDLE_WORKERS]);	
	

Listing 4 presents the portion of the automated test script that performs the real-time monitoring and writes the Apache statistics to a test log. After this code is in place for the appropriate Web servers, it can be scheduled to run within Rational TestManager.

			
/* WRITE REAL-TIME STATS TO USER MESSAGE VIEW IN TEST MANAGER */        
sprintf(&realTimeStats,"Idle[%s]Busy[%s]",
           apacheStats[APACHE_IDLE_WORKERS],apacheStats[APACHE_BUSY_WORKERS]);
display(realTimeStats);

/* WRITE THE COLUMN HEADERS TO LOG FILE ONCE */
if(currentIteration == 0)
{
    printf("CURRENT_TIME,TOTAL_ACCESSES,TOTAL_BYTES,CPU_LOAD,UPTIME,"
             "REQ_PER_SEC,BYTES_PER_SEC,BYTES_PER_REQUEST,"
             "BUSY_WORKERS,IDLE_WORKERS\n");
    currentIteration++;
}

/* WRITE STATISTICS TO LOG FILE */        
printf("%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n",
        tod(),
        apacheStats[APACHE_TOTAL_ACCESSES],
        apacheStats[APACHE_TOTAL_BYTES],
        apacheStats[APACHE_CPU_LOAD],
        apacheStats[APACHE_UPTIME],
        apacheStats[APACHE_REQ_PER_SEC],
        apacheStats[APACHE_BYTES_PER_SEC],
        apacheStats[APACHE_BYTES_PER_REQUEST],
        apacheStats[APACHE_BUSY_WORKERS],
        apacheStats[APACHE_IDLE_WORKERS]);

pop [Think_def, Think_avg, Timeout_val, Timeout_scale];

}/* END OF MAIN FUNCTION */			
			

Advanced Tip: This automated test script can be transformed to a subroutine that can be used to monitor any Apache Web server. For instance, the host name and statistics URL can be parameterized and passed as arguments to a function or procedure that access the statistics URL, parses and audits the performance data.

Scheduling the performance data collection

After the performance data collection has been automated, configuring any test schedule to implement the monitoring solution is straightforward:

1. Create a Monitoring Scenario-- Within the particular test schedule, create a specialized Scenario that will execute the Apache performance data collection automated test script. Assign the automated test script to a Sequential Selector configured to execute a large number of iterations, so that the script is continually executed throughout the test.
2. Configure the Polling Interval-- Because monitoring any component or system adds some form of overhead, the virtual user should be configured to poll the statistics URL at fixed intervals, rather than as many times as it could. By adding a fixed Delay immediately following the execution of the automated test script, the virtual user will then survey the statistics URL at a well-known interval.
3. Create a Fixed User Group-- Within the test schedule, create a specialized User Group that will execute the Apache performance data collection Scenario. Configure this user group as Fixed and assign one virtual user to the user group. This single virtual user will have then specific job of sampling the Apache Web server, collecting the resource usage data, updating the User Message View within TestManager, and logging the data to a file.

Figure 1 shows an example of a Test Schedule that implements the automated performance data collection of Apache outlined within this article. A single Fixed User Group called ApplicationMonitor executes a Scenario called ApachePerformanceMonitor. The sampling interval is set for 15 seconds.

Advanced Tip: Rather than setting the polling interval using a hard-coded delay within the test schedule, a shared variable could be used to control a polling variable set within the test script. This would also allow the polling interval to be changed while a test is running to change the frequency of data collection, if needed.

Real-time Apache performance monitoring

The performance data collection automation outlined previously uses the display() subroutine that allows the automated test script to write transient data to the User Message View within the IBM Rational TestManager interface. When a test is executing, obtaining real-time performance data for Apache is straight forward and accessed by opening the User Message View. Within this view the Message column will display the current Apache statistics. Figure 2 shows an example of what an implementation of the automated data collection script could provide. The window within the Rational TestManager interface titled User View-- Message denotes that the Apache server being monitored had 60 Idle Servers and 147 Busy Servers.

Analysis

The User Message View provides real-time updates of the Apache server statistics. However, because the data is transient there is no way to visualize the data produced through Rational TestManager until the test is complete. Because the data was logged to the virtual user's log file, it can be imported into an ancillary tool, such as Microsoft Excel to create graphs that depict trends and patterns. Table 1 represents a portion of what the test log will contain.

Table 1. Test log data

Each of the rows within Table 1 provides a cross-section of each metric that was collected at a specific time during a test. Therefore, it is possible to visualize the values of the Apache metrics as related to the elapsed time within the particular test. By importing the data into Microsoft Excel, you can create a scatter-plot that plots each of the Apache metrics and can provide trends regarding Apache system resource usage.

Figure 3 provides a time-lapsed scatter plot of the number of Idle and Busy Apache Servers during a particular test. Based on the information in this chart, it is possible to deduce that there was a period during the particular test when the Apache Web server was overloaded, because there were no available Apache processes to handle any incoming requests. This data could be used to correlate business process response time and reliability trends that were collected during the test and aide in performing root cause analysis.

Advanced Tip: The raw data regarding the business processes response times could be exported to Microsoft Excel as well and graphed along with the Apache performance statistics.

Back to top

Summary

Moving forward, it is possible to access the remote performance interfaces of many well-known software implementations and provide the automated performance data collection processes outlined for Apache within this article. For instance, the following software implementations have well-documented interfaces for querying internal resource usage:

• Oracle RDBMS-- Statistics are published to Oracle's 'V$' tables that can be queried.
• IBM WebSphere-- Statistics are published to a URL.
• Sybase Enterprise Adaptive Server RDBMS-- Statistics are published to a Sybase instance that can be queried.
• Netscape/iPlanet-- Statistics are published to a URL.

If the implementation of IBM Rational performance testing products supports the necessary protocol used to query the remote statistics interface, it is quite possible that the data collection process can be automated with any of the previous software in addition to scores of other implementations.

Resources

• Additional information about Apache Web servers can be found on the Apache Foundation's Web site.
  
  
• Share your questions and views on this article with the author and other readers in the Rational discussion forums.
  
  
• To learn more, visit the Rational Robot product area and the Rational TestManager product area on developerWorks Rational. You'll find technical documentation, how-to articles, education, downloads, product information, and more.
  
  
• Get involved in the developerWorks community by participating in developerWorks blogs.
  
  
• Browse for books on these and other technical topics.
  
  

About the author

Rate this article

Comments

Back to top

Table of contents

• The business challenge
• A strategic approach
• Implementation
• Summary
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

Dig deeper into Rational on developerWorks

Discover knowledge paths

Skill-building guides for Rational, Linux, open source, cloud, Java, business analytics, and more.

Special offers