Application management for Web servers: Developing management policies for plug-ins

Improving Web Application Performance with IBM Tivoli Composite Application Manager 6.0

Learn how to customize transaction management capabilities for IBM® HTTP Server plug-in modules. By using the Application Response Measurement (ARM) libraries you will learn how to change an IBM HTTP Server module in to monitor usage and track performance using the IBM Tivoli® Component Application Management (ITCAM) products. This is the second in a series of articles demonstrating the development of a managed Web server plug-in. This article expands the management capabilities of the application and delves into the creation of the management policy using ITCAM for Response Time Tracking (ITCAMfRTT). Related developerWorks content.


Dan Mullen (, Software Engineer, IBM

Holds MS in Computer Science Engineering and 20 years experience in application development for business systems and real-time computing. Has spent seven years with IBM Tivoli software testing and development organization. Most recently writing software automation for the validation of Tivoli's composite application management products.

21 November 2005


The first article. explained how to develop a managed Web server plug-in using calls to the ARM V4 library. Here , extend the management capabilities of the sample application to incorporate more of the desired business logic. Using ARM V4 APIs, we refine the sample application and the definition of the transactions to achieve the following.

  • Provide more flexibility and control over the transactions being monitored
  • Identify transaction bottle-necks in the business-logic
  • Isolate slow performing transactions
  • Recognize changing performance trends over time

In this column, IBM Tivoli Composite Application Manager for Response Time Tracking (ITCAMfRTT) is used to identify and manage the transactions.

This column builds on code that was introduced in the previous column. It is assumed that the reader has reviewed the first article. In order to follow along with the examples presented here, the reader should have ITCAMfRTT V6.0 installed and also IBM HTTP Server V2.0 (or greater) installed on a managed agent.

Redefining the application

As discussed in the first column, when an application registers with the arm engine, ITCAMfRTT detects the application and gathers information about it. Up to now, you have not provided ITCAMfRTT any information regarding your application. Before doing so however, you need to understand how the application fits into the business process, so that the properties used to define the application make it suitable for management within ITCAMfRTT. Figure 1 shows a process diagram of how the application will interact with user requests.

Figure 1. Selecting the Application Properties.
Overview of the Business Logic

Identifying the application

The business model in Figure 1 shows that there will be multiple instances of the application running; we want to monitor some of the transactions and not others. Also, the application might or might not be part of an application group - meaning you need to leave open the possibility that your application might be monitored by a policy which covers several other applications as well. Note that there may be several instances of the application running at the same time; for the business logic. It does not matter if these are on the same server or on different servers, the management policy will be the same. By using agent groups, ITCAMfRTT can abstract separate host machines into a single management policy.

ARM allows for the static identification of the application as well as a contextual identification. The static identification will be the same each time the application is run. These are used to distinguish the application from other ARM-instrumented applications as well as group together applications related to the same business process. The contextual identification can change each time the application is started -- even if the application is re-started within the same process -- as if creating a new thread for example. Exactly what properties are used to describe the static and contextual identification of the application is up to the application developer. Listing 1 shows how to define the application properties.

Listing 1. Defining the application properties
static void get_app_identity(arm_buffer4_t *buf,
			     int cnt, 
			     const arm_char_t *names[],
			     const arm_char_t *values[],
			     apr_pool_t *p) {

  arm_subbuffer_app_identity_t  *appIdSubBuf;
  arm_property_t *props;
  arm_subbuffer_t **tbuf; 

  /* allocate the sub-buffer property values */ 
  props = (arm_property_t *) apr_pcalloc(p, cnt*sizeof(arm_property_t));
  if (buf != NULL && props != NULL) {
   /* allocate the sub-buffer */ 
   appIdSubBuf = (arm_subbuffer_app_identity_t *)
     apr_pcalloc(p, sizeof(arm_subbuffer_app_identity_t));

   /* allocate the sub-buffer array - 1 element */
   tbuf = (arm_subbuffer_t **) apr_pcalloc(p, sizeof(*tbuf));
   (props + 0)->name  = names[0];
   (props + 0)->value = values[0]; 
   (props + 1)->name  = names[1];
   (props + 1)->value = values[1]; 

   appIdSubBuf->header.format = ARM_SUBBUFFER_APP_IDENTITY;
   appIdSubBuf->identity_property_count = cnt;
   appIdSubBuf->identity_property_array = props;

   /* initialize the contextual property names */ 
   appIdSubBuf->context_name_count = 1;
   appIdSubBuf->context_name_array =
                     (char **)apr_pcalloc(p,sizeof(char *)); 
   appIdSubBuf->context_name_array[0] = 
   tbuf[0] = (arm_subbuffer_t *) appIdSubBuf;
   buf->count = 1;
   buf->subbuffer_array = tbuf; 

  static const arm_char_t *application_property_names[] = {
  static const arm_char_t *application_property_values[] = {
    "Http Module",

  arm_buffer4_t *app_buf4d = (arm_buffer4_t *) apr_pcalloc(pool, sizeof(arm_buffer4_t)); 

The routine get_app_identity accepts a list of static names and values; these are the static properties of the application. These are passed to the arm engine when the application is registered and they will never change. Also passed to the arm engine during registration is the context name AppId. Since this is a contextual property, the value of this property will not be determined until the application is started. For more information about the arm_buffer4_t *buf parameter, refer to the Resources at the end of the article.

Setting the application context

In this example, the AppID property will be used to set the context of the application. The value of the AppID context property is set by a directive in the httpd.conf file so you can manually configure the application after it has been installed. This will provide a little extra flexibility in the configuration of the management policy. Listing 2 shows how to modify the sample module to read the directive.

Listing 2. Reading the application context from the httpd.conf file
static const char *set_app_identity(cmd_parms *cmd,
                                    void *mconfig,
				    const char *msg)
  server_config_t *conf;
  conf= ap_get_module_config(cmd->server->module_config , &sample_module);
  if (msg != NULL) {
    conf->app_identity = apr_pstrdup(cmd->pool, msg);
  } else {
    return "Missing Directive : ArmApplicationIdentity"; 
  return NULL;

static const command_rec config_vars[] =
                  "ARM Application Identifier"),

module AP_MODULE_DECLARE_DATA sample_module =
    config_vars,   /* configure module directives */                      

Once these changes are made, the following line should be added to the httpd.conf file.

ArmApplicationIdentity     ArmApp01

Now, when you define the management policy in ITCAMfRTT, you will add the AppId property to the transaction definition. Figure 2 shows how the application properties can be selected in ITCAMfRTT. By setting the value appId the management policy definition, you can control which applications are managed and which ones are not managed by setting this value within the configuration file and restarting the server. This will be useful if you want to monitor a select number of servers at a time or when servers are added and removed from the environment. In these situations, you can control which application instances get monitored without having to change or redistribute the management policy.

Figure 2. Selecting the Application Properties.
Setting Application Properties

Defining the transactions

Transactions can also be defined in a static and contextual manner. Most of the interesting properties of a transaction are contextual and you tend to focus more on the context of transactions. However, for performance reasons, defining static properties might be preferable if it supports the business logic. Whether to use static or contextual properties - or both - depends on how the transaction is used and what data is associated with the transaction. ITCAMfRTT can manage transactions with static and contextual properties and treats the properties the same. Therefore, the application developer can choose to assist in the transaction management by using a naming convention that denotes the contextual properties. Since the code used to define the transaction properties is almost identical to the code used to set the application properties, it is not listed here to save space. Refer to the sample code in the download section for a full description on setting the transaction properties.

Once the transaction properties are defined for the module they can be incorporated into the management policy. Figure 3 shows an example of how to set the policy to handle only transactions with a filename that with the .gif extension.

Figure 3. Selecting the transaction properties.
Setting Application Properties

The properties that are defined for a transaction can serve two purposes. First, as you have just seen, the transaction properties, along with the application properties, are used to define the management policy being used to model the desired business logic. Second, sub-transaction properties can provide details about the instance of each transaction generated by the plug-in. Details such as the name of the requested file or the IP address of the requesting client machine can serve as an important element in the business logic. You will see how this all fits together in the next example.

Building the business logic

When the transaction is defined with the desired properties, it needs to be incorporated into the business logic. ITCAMfRTT specializes in tracking the business logic across the enterprise, therefore its necessary to define where the transaction falls within the business process. This is done with the use of correlators that are transferred from one transaction to another. If you consider the business model as a tree where each node is a single transaction, then the correlators are what connect the nodes together. Each node, e.g. transaction, can have zero to n child transactions and zero or one parent transaction. A transaction without a parent is the root transaction of the business logic -- sometimes referred to as the edge transaction. The root transaction acts as the filter, which identifies the sub-transactions as belonging to a particular element of the business model. In ITCAMfRTT, the properties of the root transaction are mapped to the properties of the management policy. This has important consequences on how you want to manage the business logic as demonstrated below.

Policy vs. transaction

In the above example, a management policy was created to identify transactions that have the property cFilename equal to the regular expression *\.gif. Figure 4 shows how this transaction appears when viewed in the ITCAMfRTT topology view. Since you have not defined any correlators, it is currently behaving as the root, or edge, transaction and thus displaying the properties of the policy - not the properties of the transaction itself. To correct this situation to monitor the individual transaction properties, the current edge transaction must be demoted to a sub-transaction.

Figure 4. GIF request as an edge
Figure X

Demoting the edge

The first step in demoting the edge transaction to a sub-transaction is to create a new transaction, which is the place-holder for the new edge policy. The HTTP server creates a TCP/IP connection for each request. Because of the server keep-alive functionality, several requests can be handled under a single connection. So the HTTP connection is a logical parent transaction for the GIF request. Listing 3 shows the adjustments to the sample module that identifies the current connection then creates a new edge transaction if it is a new connection.

Listing 3. Creating the HTTP connect transaction.
typedef struct connection_info {
  long id;                           // The connection ID
  arm_tran_start_handle_t handle;  
  unsigned char     destroyed;   
  arm_correlator_t  root_correlator; // ARM correlator for edge. 
  struct connection_info *next;      
  struct server_config *config;      
} connection_info_t;

static int sample_fixup(request_rec *r)
  int rc;
  sample_ctx * ctx; 
  arm_correlator_t *main_correlator = NULL ;
  arm_correlator_t  sub_correlator;
  struct connection_info *cConnInfo;
  server_config_t *sconf =


  cConnInfo = get_connection_info(sconf, r);
  if (cConnInfo == NULL) { // this is a new connection
    XML_LOG_DEBUG("Starting HTTP Connect transaction");
    cConnInfo = add_connection_info(sconf, r); 
    if (cConnInfo != NULL) {
      rc = arm_start_transaction(sconf->_aHndl,

                                (void *)cConnInfo,
  if (cConnInfo != NULL) {
    main_correlator = &(cConnInfo->root_correlator);

Up to now, memory has been allocated from the request pool. However, the connection data must survive past the end of each request so it's necessary to use the connection pool to store the connection data. Listing 4 show how the connection data is allocated from the connection pool.

Listing 4. Creating and storing the connection data
static struct connection_info * add_connection_info(
                                     server_config_t *sconf,
				     request_rec *r
  struct connetion_info *cConnInfo = (struct connection_info *)
                apr_pcalloc(r->connection->pool, sizeof(*cConnInfo)); 
  cConnInfo->id = r->connection->id;
  return cConnInfo; 

The other note-worthy part of Listing 3 is the call to apr_pool_cleanup_register. This API function registers a callback in the code that is invoked when the connection pool - and thus the connection - is destroyed. This is where the HTTP Connect transaction meets its logical end. Listing 5 shows how this callback is used to stop the transaction.

Listing 5. Stopping the new edge transaction
static apr_status_t remove_connection_info(void *data ) {
  connection_info_t *tConnInfo = (connection_info_t *) data;
  int rc;
  if (tConnInfo != NULL &&
      tConnInfo->destroyed == (unsigned char) 0) {
    rc = arm_stop_transaction(tConnInfo->handle,
  return APR_SUCCESS; 

The arm engine automatically creates a correlator for the start of each transaction and passes it back to the application. (See Listing 3). So the only remaining code change is to take this correlator and pass it to the start transaction call for the GIF Request - thus connecting the nodes of the business model.

The New Edge Policy

You can now define a new edge policy using the HTTP Connect transaction. After re-running the transaction discovery policy the HTTP Connect transaction shows up on the policy configuration panel as a new transaction pattern. Be sure to select this as the transaction pattern for the policy as shown in figure 5.

Figure 5. The Http connect pattern
HTTP Connection

Figure 6 shows the topology view of the new policy. Note that since the GIF Request has been demoted from the root node of the business logic, you can now see the individual transaction properties for each request.

Figure 6. The Http connect topology View
Figure X

Expanding the model

Business models are constantly changing and the number of monitored applications is increasing. So as is often the case, the business process you defined could likely be part of a larger process. In order to prepare for such a circumstance, you should modify the application to accept a parent correlator.

When the edge is not an edge

Since what is now considered an edge might be a sub-process of a larger model, the edge transaction would actually be a sub-transaction with a parent correlator. But where would this correlator come from? Because ARM correlators are transportable between all types of applications, platforms, and languages, the answer is practically anywhere. However, since the edge transaction responds to an HTTP connection, assume that the correlator is transmitted inside the HTTP header information.

Getting the parent correlator

In order to accommodate the situation where the edge policy might be invoked as a sub-transaction from another business process, examine the HTTP request header for a parent correlator. Listing 6 shows how the ARM_CORRELATOR header is retrieved and converted to a format that can be passed to arm_start_transaction. Refer to the Resources section for articles regarding the format of ARM V4 correlators and restrictions on their use.

Listing 6. Retrieving the optional parent correlator
  correlator_string = (unsigned char *)
                     apr_table_get(r->headers_in, "ARM_CORRELATOR");
  if (correlator_string != NULL) {
    // Our application is being called as a subtransaction. 
    main_correlator = str2correlator(correlator_string,r->pool);
  } else {
    // This is the edge.  
    main_correlator = NULL; 

Putting it all together

Figure 7 shows the HTTP connect transaction when invoked as a sub-transaction. Note that under these circumstances, the edge policy defined in the previous step is not used to filter the incoming transactions. This is because the transaction filtering is done only at the policy level - or essentially the root transaction.

Figure 7. The overall process topology
Figure 7

Optional transaction metrics

In addition to transaction properties, ARM V4 has the capability to assign additional metrics about transaction instances. These metrics cannot be used to define the management policy but can provide additional information the user of the ITCAMfRTT interface. Metrics can be used to correlate data across transaction instances and even across application instances.

Listing 7. Setting the transaction metrics
  arm_subbuffer_metric_bindings_t *cSubMetBnds;
  arm_metric_binding_t            metric_bindings[2];

    rc = arm_register_metric(&(sconf->_appId),

    rc |= arm_register_metric(&(sconf->_appId),
    metric_bindings[0].slot = 6; 
    metric_bindings[1].slot = 1;

    cSubMetBnds = apr_pcalloc(pool, sizeof(*cSubMetBnds));
    cSubMetBnds->header.format = ARM_SUBBUFFER_METRIC_BINDINGS; 
    cSubMetBnds->count = 2; 
    cSubMetBnds->metric_binding_array = &metric_bindings[0]; 
    tran_buf4->count = 2;
    tran_buf4->subbuffer_array = (arm_subbuffer_t **)
      apr_pcalloc(pool, 2*sizeof(arm_subbuffer_t *)); 

    tran_buf4->subbuffer_array[1] = (arm_subbuffer_t *) cSubTranId;
    tran_buf4->subbuffer_array[0] = (arm_subbuffer_t *) cSubMetBnds; 
    rc = arm_register_transaction(&(sconf->_appId),
				  "GIF Request", 

In the above listing, two metrics are defined; mFilesize and mRequests. As with properties, ITCAMfRTT treats the metrics as it does all other instance properties so use a naming convention to distinguish the metrics from other properties. The value of each metric must be captured when the transaction is stopped. Refer to the downloaded source code for a full description on setting the metric values. Tracking the metric values can then be done by viewing the transaction properties in the ITCAMfRTT topology view as shown in figure 8.

Figure 8. The transaction metric properties as viewed in the topology
Figure 8


The simple examples shown in this column are a small part of defining on overall management policy. However, as demonstrated, using managed plug-ins to customize Web applications to fit into the business model is a simple, yet effective way to define and manage the logic of a business process servicing user request via HTTP. But this is just a slice of the capabilities of ARM V4 and ITCAMfRTT. You can find more information on these topics in the resources below.


Source code for sample plug-in.sample2.zip13 KB



  • For more information about ARM go to The Open Group web site.
  • For more information about about the APR library and to download latest version, visit the Apache Portable Runtime web site.
  • To learn more about Tivoli products, visit the developerWorks Tivoli zone. You'll find technical documentation, how-to articles, education, downloads, product information, and more.

Get products and technologies

  • ITCAM for SOA: Get the latest on what Tivoli is doing for service-oriented architectures


developerWorks: Sign in

Required fields are indicated with an asterisk (*).

Need an IBM ID?
Forgot your IBM ID?

Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.


All information submitted is secure.

Dig deeper into Tivoli (service management) on developerWorks

Zone=Tivoli (service management), Tivoli
ArticleTitle=Application management for Web servers: Developing management policies for plug-ins