Whereas the base Glassbox Inspector I left off with in Part 1 of this article could monitor only simple Servlets-based applications (as you saw in the Duke's Bookstore example), a more pragmatic framework would include support for popular Web application frameworks such as Struts and Spring. Modern Web applications rarely use Servlets to handle requests directly. These frameworks typically delegate all dynamic page requests to a gateway Servlet, such as the Spring Framework’s DispatcherServlet, which then delegates request processing to various controllers. Similarly, a Struts ActionServlet delegates to subclasses of Action that act as controllers.

In this second half of my foray into performance monitoring with AspectJ, I extend the Glassbox Inspector to provide more meaningful information about application performance by adding monitors that track the interaction of controllers for the Struts and Spring Web application frameworks, as well as serving and requesting Web services operations. I also extend the system to support multiple applications and add an error-handling layer and the ability to easily enable and disable monitoring at run time. I conclude the article by showing you how to deploy the Glassbox Inspector using load-time weaving and how to measure the resulting overhead.

You need some familiarity with Spring MVC, Struts, and Apache Axis for Web Services to follow the examples in this article. See Download for the complete code for the Glassbox Inspector monitoring infrastructure. See Resources to download AspectJ, JMX, and Tomcat 5, which you need to run the examples.

Reusable operation monitors

To accommodate a broader range of monitoring functionality, I've pulled most of the reusable Servlet-monitoring code from Part 1, Listing 5 into an AbstractOperationMonitor aspect. Often it is important to track nested requests, both to track controllers at different levels of a framework as well as to track performance in delegated requests (e.g., from forwarding to a nested request). Listing 1 shows how I've extended the Glassbox Inspector to track nested requests for operations:

public abstract aspect AbstractOperationMonitor extends AbstractRequestMonitor {  
      
  protected abstract class OperationRequestContext extends RequestContext {
      /**
       * Find the appropriate statistics collector object for this
       * operation.
       * @param operation an instance of the operation being 
       *         monitored
       */
      public PerfStats lookupStats() {                
          if (getParent() != null) {
              // nested operation
              OperationStats parentStats =
                      (OperationStats)getParent().getStats();
              return parentStats.getOperationStats(getKey());
          }
          return getTopLevelStats(getKey());
      }
        
      /**
       * Determine the top-level statistics for a given operation 
       * key. This also looks up the context name for the
       * application from the operation monitor: 
       * @see AbstractOperationMonitor#getContextName(Object)
       * For a Web application, top-level statistics are normally
       * all Servlets, and the key is the Servlet name. 
       * @param key An object to uniquely identify the 
       *         operation being performed.
       */
      protected OperationStats getTopLevelStats(Object key) {
          OperationStats stats;
          synchronized(topLevelOperations) {
              stats = (OperationStats)topLevelOperations.get(key);
              if (stats == null) {                
                  stats =
                    perfStatsFactory.createTopLevelOperationStats(key,
                            getContextName(controller));
                  topLevelOperations.put(key, stats);
              }
          }
          return stats;        
      }
        
      /** 
       * @return An object that uniquely identifies the operation
       *          being performed. 
       */
      protected abstract Object getKey();     
        
      /** The current controller object executing, if any. */
      protected Object controller;
  };

  /** 
   * This advice stores the controller object whenever we construct a
   * request context. 
   */
  after(Object controller) returning (OperationRequestContext ctxt) : 
    cflow(adviceexecution() && args(controller, ..) &&
    this(AbstractOperationMonitor)) &&
    call(OperationRequestContext+.new(..)) {
      ctxt.controller = controller;
  }    

...

The first part of the AbstractOperationMonitor extends the original Servlet monitor from Part 1 to look up statistics for nested operations. The use of nested operations lets me track resource use in cases like dispatched JSPs or break down different methods for Web-service or multiple-method application controllers. The original lookupStats() method now checks for a parent request context. If there is one, it calls the new method getOperationStats() to retrieve it. Otherwise, it calls the new method getTopLevelStats(), which invokes a factory to create a new OperationStats. Using the factory ensures that my monitoring infrastructure isn't dependent on the implementation of statistics classes.

Back to top

Monitoring multiple applications

In Listing 1, I've also included support for monitoring multiple applications running in a single application server. I did this mainly by adding the provision to look up an application context name. When looking up top-level statistics, I invoke a template method on the monitor, getContextName(), to determine what application the operation is associated with. You'll see below how this is handled for Servlets. Note that the getContextName() method is passed an instance of the controller so I can look up the associated application context.

The abstract operation monitor also provides pointcuts with concrete advice that define common ways to monitor requests. These are extended to allow monitoring of incoming operations like Struts actions and incoming Web services requests. Listing 2 shows a representative example:

  /** 
   * This defaults to no join points. If a concrete aspect overrides
   * classControllerExec with a concrete definition,
   * then the monitor will track operations at matching join points
   * based on the class of the controller object.
   */ 
  protected pointcut classControllerExec(Object controller);
    
  Object around(final Object controller) :
    classControllerExec(controller) {
      RequestContext rc = new OperationRequestContext() {
          public Object doExecute() {
              return proceed(controller);
          }
                       
          protected Object getKey() {
              return controller.getClass();
          }                        
      };
      return rc.execute();        
  }    

  // Controller where the name of the signature at the monitored join point
  // determines what is being executed, for example, the method name
  /** 
   * This defaults to no join points. If a concrete monitor overrides
   * methodSignatureControllerExec with a concrete
   * definition, then it will track operations at matching join points
   * based on the run-time class of the executing controller instance
   * and the method signature at the join point.
   */ 
  protected pointcut methodSignatureControllerExec(Object controller);
    
  Object around(final Object controller) :
    methodSignatureControllerExec(controller) {
      RequestContext rc = new OperationRequestContext() {
          public Object doExecute() {
              return proceed(controller);
          }
                       
          protected Object getKey() {
              return concatenatedKey(controller.getClass(),
                thisJoinPointStaticPart.getSignature().getName());
          }
            
      };
      return rc.execute();        
  }   

The pointcut for classControllerExec() captures any point where a class controller is handling requests such as a Servlet do method execution or the normal Struts action execute method, where the class of the object servicing the request determines the operation being performed. More precisely, the classControllerExec() pointcut defines an empty pointcut (one that will never match any join point). It then provides concrete advice that sets up the worker object and returns the right key value for this case. This is similar to using an abstract pointcut, where subaspects must override the pointcut to apply advice. In this case, however, I provide a default definition that never matches. If a subaspect of the AbstractOperationMonitor doesn't need to monitor class controllers, it simply doesn't override the pointcut. If it does need to monitor class controllers, it provides a definition of when to monitor a point.

Back to top

Concrete operation monitors

The methodSignatureControllerExec() pointcut and associated advice are similar: they provide a means for concrete operation monitor aspects to match controllers that dispatch to different methods, based on the signature at the join point.

Listing 3 shows the concrete aspects that extend the AbstractOperationMonitor to monitor Struts and Spring MVC operations:

public aspect StrutsMonitor extends AbstractOperationMonitor {
    /** 
     * Marker interface that allows explicitly _excluding_ classes
     * from this monitor: not used by default. If using Java™ 5, an 
     * annotation would be better.
     */
    public interface NotMonitoredAction {}

  /** 
   * Matches execution of any method defined on a Struts action or
   * any subclass, which has signature of an action execute (or 
   * perform) method, including methods dispatched to in a
   * DispatchAction or template methods with the same signature.
   */ 
  public pointcut actionMethodExec() : 
      execution(public ActionForward Action+.*(ActionMapping,
              ActionForm, ServletRequest+, ServletResponse+)) &&
      !within(NotMonitoredAction);

  /** 
   * Matches execution of an action execute (or perform) method for
   * a Struts action. Supports the Struts 1.0 API (using the perform
   * method) as well as the Struts 1.1 API (using the execute method).
   */ 
  public pointcut rootActionExec() : 
      actionMethodExec() && (execution(* Action.execute(..)) ||
      execution(* Action.perform(..)));
    
  /** @Override */
  protected pointcut classControllerExec(Object controller) :
      rootActionExec() && this(controller);

  protected pointcut dispatchActionMethodExec() :
      actionMethodExec() && execution(* DispatchAction+.*(..));

  protected pointcut methodSignatureControllerExec(Object controller):
      dispatchActionMethodExec() && this(controller);
}

public aspect SpringMvcMonitor extends AbstractOperationMonitor {
  /** 
   * marker interface that allows explicitly _excluding_ classes
   * from this monitor: not used by default 
   */
  public interface NotMonitoredController {}
    
  public pointcut springControllerExec() :
      execution(public ModelAndView Controller+.*(HttpServletRequest,
              HttpServletResponse)) &&
      !within(NotMonitoredController+);

  protected pointcut classControllerExec(Object controller) :
      springControllerExec() && execution(* handleRequest(..)) &&
      this(controller);

  protected pointcut methodSignatureControllerExec(Object controller):
      springControllerExec() && 
      execution(* MultiActionController+.*(..)) && this(controller);    
}

The first thing to note about both of these aspects is how concise they are. They simply extend two pointcuts from the operation monitor to concretely monitor their specific APIs. Because of their simplicity, these concrete monitors could also be defined in XML without requiring compilation. See Listing 10 for an example of this feature of AspectJ 5.

About StrutsMonitor and SpringMvcMonitor

The StrutsMonitor aspect in Listing 3 is designed to work with both old and new versions of the Struts API: Struts 1.0 actions are invoked by calling perform() whereas Struts 1.1 actions are invoked by calling execute(). I track the concrete subclass of normal action classes as a Class controller: only the class of the executing object matters. However, Struts allows for controllers that dispatch to multiple methods in a single class if the class extends DispatchAction. I monitor the individual methods being executed in these dispatch actions by matching any methods with the signature of a Struts action in a subclass of a DispatchAction. This approach lets me track statistics for each different controller method separately.

I have tested the StrutsMonitor against the iBatis JPetStore 1.3 sample application (see Resources). Like many applications, it extends Struts with a little framework of its own: the common base action has a method called perform() that dispatches to helper actions using doPerform() as a template method. However, there's no need to track execution of these template methods: The class-level controller will identify the specific subclass of Action being executed in the execute() method, which is sufficient to distinguish them.

The SpringMvcMonitor aspect is quite similar to the StrutsMonitor in that it monitors any Controller object as a class controller when it executes handleRequest(). It further monitors the execution of public methods with the signature of a Spring controller method in a MultiActionController or any subclass thereof. For example, I would like to monitor the execution of welcomeHandler() and ownerHandler() in this code snippet separately:

public class ClinicController extends MultiActionController 
  implements InitializingBean {
...
  public ModelAndView welcomeHandler(HttpServletRequest request, 
    HttpServletResponse response) throws ServletException {
      return new ModelAndView("welcomeView");
  }
...
  public ModelAndView ownerHandler(HttpServletRequest request, 
    HttpServletResponse response) throws ServletException {
       Owner owner = clinic.loadOwner(
         RequestUtils.getIntParameter(request, "ownerId", 0));
...
       model.put("owner", owner);
       return new ModelAndView("ownerView", "model", model);
  }
...

Naturally, it is easy to extend this approach to handling other frameworks, including custom frameworks, even using an XML-defined aspect as described previously.

Writing portable aspects

Note that the StrutsMonitor aspect, like most of my monitoring aspects, is designed to work against multiple versions of common components. Its pointcuts will match for different versions of Struts; they reference types without requiring them to be available at weaving time (for example, at load-time). You can often write pointcuts this way by only using class names in a statically resolvable context such as in a field signature or type pattern. If you use the class names in a dynamic context (such as this, target, or args), then the weaver requires access to the type. Note that the Struts monitor binds the this pointcut designator to an instance of Object, which does not require the Struts JAR to be present. While a bit ugly (because it loses the advantages of compile-time type checking), this approach is preferable to requiring the presence of many dependent libraries when deploying reusable aspects.

However, advice or other executable code that refers to a class or accesses its members (for example, by invoking a method) needs to follow the usual rules for portability in Java code. If the advice must invoke changing methods on different versions of an API, it is possible to use Java reflection to test for the presence of different methods and execute them; for example, to call perform() or execute() on an action method explicitly. Note that if I simply want to proceed with the original execute() or perform() method at a join point in an around advice, AspectJ handles this situation for me without requiring any special effort. In practice, many APIs provide backward compatibility so I can compile against the oldest version I want to support, and the result will work with newer versions.

Typically, the aspect won't execute code that triggers loading a class unless the class would be loaded by the program's execution anyhow; for example, advice that references a class will be triggered when a method for that class is going to execute. This property is important to preserve for library aspects that refer to optional classes. For the Glassbox Inspector, I don't want to require monitored applications to include a version of Struts or other libraries that it can monitor.

Monitoring JSPs and application names

Listing 4 shows the ServletMonitor from Part 1 refactored to use the abstract operation monitor as a base aspect. I continue to track Servlets based on their class by extending the classControllerExec() pointcut. I've added support for monitoring JSPs by monitoring the jspService() method. Because I want to be able to determine the name of a Web application, I've set the ServletMonitor to override the getContext() method to look up the name of a Servlet context.

All the other Web application controllers I've implemented so far are ultimately called from a Servlet, so only this controller needs to provide an implementation of the application context. In particular, my Web application framework monitors are nested within their dispatching Servlet invocations. When I further extend the Glassbox Inspector to monitor other operations such as incoming Web services requests, I will need to provide the operation name as a different kind of application context. When I want to extend the framework to monitor incoming JMS messages or EJB requests, I will have a framework to supply application contexts for those resources as well.

 
public aspect ServletMonitor extends AbstractOperationMonitor {
    
  /** 
   * Execution of any Servlet method: typically not overridden in
   * HttpServlets. 
   */
  public pointcut ServletService(Servlet Servlet) :
      execution(void Servlet.service(..)) && this(Servlet);
        
  /** Execution of any Servlet request methods. */
  public pointcut httpServletDo(HttpServlet Servlet) :
      execution(void HttpServlet.do*(..)) && this(Servlet);
    
  /** Execution of any JSP page service method. */
  public pointcut jspService(JspPage page) : 
      execution(* _jspService(..)) && this(page);
    
  protected pointcut classControllerExec(Object controller) :
      (ServletService(*) || httpServletDo(*) || jspService(*)) &&
      this(controller);

  /** 
   * Create statistics for this object: looks up Servlet context to
   * determine application name.
   */
  protected String getContextName(Object controller) {
      Servlet Servlet = (Servlet)controller;
      return Servlet.getServletConfig().getServletContext().
              getServletContextName();
  }
}

Back to top

Updating JMX for nested statistics

In Part 1, I extended the Glassbox monitoring infrastructure to provide nested statistics, such as JDBC statements within connections within Servlet requests. Here, I look at how the StatsJmxManagement aspect can be updated to expose a compound name for these nested statistics. I also show how to integrate the application name into this compound name. For example, the statistics for a database statement might be named with the following string:

application=Spring Petclinic,operation=org.springframework.samples.
petclinic.web.ClinicController,database=jdbc:hsqldb:hsql://localhost:
9001,statement=SELECT id;name from types ORDER BY name

This string uses the descriptions and names for the performance statistics of all the ancestor statistics for the given information. Naming the statistics this way lets JMX tools group and display related information naturally. JMX tools like the JConsole use structured names to group common elements, making hierarchical browsing easier, as you will see in Figure 1. Listing 5 shows the necessary updates to StatsJmxManagement to support this feature:

public aspect StatsJmxManagement {

  private String PerfStats.cachedOperationName;
    
  /** JMX operation name for this performance statistics bean. */    
  public String PerfStats.getOperationName() {
      // look up from cache
      // else 
...
      appendOperationName(buffer);
...
      return operationName;
  }
    
  /** Add bean's JMX operation name to a StringBuffer. */    
  public void PerfStats.appendOperationName(StringBuffer buffer) {
      if (cachedOperationName != null) {
          buffer.append(cachedOperationName);
      } else {
          aspectOf().nameStrategy.appendOperationName(this, buffer);
      }
  }    

  public void PerfStats.appendName(StringBuffer buffer) {
      // append the appropriate name & JMX encode
...
  }

  public StatsJmxNameStrategy getNameStrategy() {...}
 
  public void setNameStrategy(StatsJmxNameStrategy nameStrategy) { ... }
    
  private StatsJmxNameStrategy nameStrategy;
}

public interface StatsJmxNameStrategy {
  void appendOperationName(PerfStats stats, StringBuffer buffer);
}

public class GuiFriendlyStatsJmxNameStrategy extends 
  AbstractStatsJmxNameStrategy {
    
  public void appendOperationName(PerfStats stats, 
          StringBuffer buffer) {        
      PerfStats parent = stats.getParent(); 
      if (parent != null) {
          appendOperationName(parent, buffer);
          buffer.append(',');
        else {
          if (stats instanceof OperationStats) {
              OperationStats opStats = (OperationStats)stats;
              String contextName = opStats.getContextName();
              if (contextName != null) {
                  buffer.append("application=\"");
                  int pos = buffer.length();
                  buffer.append(contextName);
                  JmxManagement.jmxEncode(buffer, pos);
                  buffer.append("\",");
              }
          }
      }
      buffer.append(stats.getDescription());
      buffer.append('=');
      stats.appendName(buffer);
  }
} 

Listing 5 shows how I've set up StatsJmxManagement to allow different naming strategies. For use with GUI tools like the JConsole, I've listed a GuiFriendlyStatsJmxNameStrategy, which returns names as shown in Listing 5 and in Figure 1. In addition to this, the complete code includes an alternative CanonicalStatsJmxNameStrategy, which adheres to the JMX recommendations for naming MBeans, but which doesn't display as well in any JMX tool I've tried.

The basic approach used to build up these names is to create a string buffer and recursively get the operation name for the parent statistic, then add the name afterwards. For top-level operation statistics, the context (application) name is added first. To support this approach, I added a getDescription() method to PerfStats. I also updated the operation and resource statistics implementations to set appropriate values (such as operation, operation1, .., resource, and request). Download the Glassbox Inspector source to see all the details and the CanonicalStatsJmxNameStrategy alternative.

Back to top

Monitoring Web service calls

The Glassbox Inspector framework is now quite extensible to support new types of incoming requests (operations) and also new resources. Monitoring provided (incoming) remote service operations is quite similar to monitoring Web application frameworks: the article source gives an example of monitoring services implemented with Apache Axis.

With just a few minor adjustments I can also extend the Glassbox Inspector to monitor the consumption of remote services that are external to an application. As applications with service-oriented architectures become increasingly common, it is becoming important to track the reliability of externally controlled services and determine correlations with application problems. Correlating such data with application context (such as the Web operation being performed and time spent in other resources) is an important way to quickly identify root causes of application failures. This type of analysis is quite different from using SOAP handlers to monitor Web services calls in isolation because it connects implementation information with performance on external requests.

Listing 6 shows a remote-call monitor that monitors Web services calls made through the JAX-RPC API for calling Web Services, as well as RMI calls (which are themselves normally remote):

public aspect RemoteCallMonitor extends AbstractResourceMonitor {
  /** Call to remote proxy: RMI or JAX-RPC */
  public pointcut remoteProxyCall(Object recipient) : 
      call(public * Remote+.*(..) throws RemoteException) &&
      target(recipient) && !within(glassbox.inspector..*);
    
  /** Monitor remote proxy calls based on called class and method */
  Object around(final Object recipient) : 
    remoteProxyCall(recipient) {
      RequestContext requestContext = new ResourceRequestContext() {
            
          public Object doExecute() {
              return proceed(recipient);
          }
            
          public PerfStats lookupStats() {
              String key = "jaxrpc:"+recipient.getClass().getName()+
                "."+thisJoinPointStaticPart.getSignature().getName();
              key = key.intern();
                
              return lookupResourceStats(key);
          }
      };
      return requestContext.execute();
  }
}

Notice how little I had to do to support this feature; most of the support is contained in the AbstractRequestMonitor. I simply defined remote proxy calls to be calls to public methods on any object implementing the Remote interface where the method can throw a RemoteException. With that small change in place, I can use the Worker Object pattern (see Part 1) to monitor remote calls, providing a name based on the class and method name of the object being invoked. The RemoteCallMonitor aspect uses a helper method to find the top-level resource statistics, which were extracted from the worker object in the JdbcConnectionMonitor and pulled into the new common base aspect: AbstractResourceMonitor.

Naturally, I could extend this approach further to monitor other Web services calls, as well as the execution of methods based on popular frameworks such as Apache Axis. I could also use this technique to monitor the time spent in XML processing, which would be useful for any XML-intensive application. Many applications that utilize Web services would also benefit from this type of monitoring.

Back to top

Monitoring for failure

Extending the Glassbox Inspector system to monitor application failures is a natural next step. I start by recording a failure whenever the system exits a monitored point by throwing an exception (more precisely, any Throwable). Failure recording provides useful information for tracking Web operations and for database connection attempts: throwing a Throwable from either of these is problematic in almost all cases. For JDBC statements and other resource accesses, a throwable often indicates a true application failure, but in some cases, it is part of the normal program logic. For example, trying to register with a name that is already taken can trigger an exception on insert. To account for this case, I use a configurable strategy for each monitor to determine whether a given throwable is actually an error or should be accepted as a normal exit condition.

Listing 7 contains the advice and changes that allow me to configure the AbstractRequestMonitor to determine whether a Throwable is a failure:

public aspect AbstractRequestMonitor {

  /** 
   * Record an error if this request exited by throwing a 
   * Throwable.
   */
  after(RequestContext requestContext) throwing (Throwable t) : 
    requestExecution(requestContext) {
      PerfStats stats = requestContext.getStats();
      if (stats != null) {
          if (failureDetectionStrategy.isFailure(t,
                  thisJoinPointStaticPart)) {
              requestContext.recordFailure(t);
          } else {
              requestContext.recordExecution();
          }
      }
  }
  public void setFailureDetectionStrategy(...) { ... }
  public FailureDetectionStrategy getFailureDetectionStrategy() { ... }
  protected FailureDetectionStrategy failureDetectionStrategy =
      new FailureDetectionStrategy() {
        public boolean isFailure(Throwable t, StaticPart staticPart) {
            return true;
        }
      };
...
}   

public interface FailureDetectionStrategy {
  boolean isFailure(Throwable t, StaticPart staticPart);
}

The net effect of Listing 7 is to record a count whenever I exit a monitored join point by throwing an exception. Note that I definitely want to identify cases where I exit by throwing an Error, as they are almost certainly failures! From here, I could easily extend my AbstractRequestMonitor to also track some or all of the exceptions, stack traces, and values of the current object and parameters at the point where the exception occurred. If I did this, I would need a means to be sure I did not record sensitive information such as passwords, credit card numbers, or personally identifiable information. To complete this integration, I made a few minor revisions to my AbstractRequestMonitor. See the article source for details.

Using exception conversion for errors

Given the previous discussion, you may be wondering how you could configure the Glassbox Inspector to identify some throwables as being not errors. The easiest way would be to pass in a strategy that always returns false for a given monitoring aspect (such as the JdbcStatementMonitor). Another simple solution is to detect certain exception types, such as NumberFormatException, and indicate that they are not failures. This could be combined with the Exception Conversion pattern (see Resources) to convert certain exceptions into a meaningful hierarchy. Listing 8 is an example of how an application might apply exception conversion along with a failure detection strategy:

/** 
 * Softens and uses Spring's exception translator to convert from
 * SQLException to unchecked, meaningful exception types 
 */
aspect DataAccessErrorHandling {
  declare soft: SQLException: jdbcCall();

  after() throwing (SQLException e) : jdbcCall() {
      throw sqlExceptionTranslator.translate("dbcall", getSql(), e);
  }

  declare precedence: ModelErrorHandling, JdbcStatementMonitor;
}

/** 
 * Conservative detection strategy: failures from SQL that often
 * indicate valid business conditions are not flagged as failures.
 */
public class DataAccessDetectionStrategy implements FailureDetectionStrategy {
  public boolean isFailure(Throwable t, StaticPart staticPart) {
      return !(t instanceof ConcurrencyFailureException) &&
         !(t instanceof DataIntegrityViolationException);
  }
}

The first part of Listing 8 shows a simple error-handling aspect that softens SQLExceptions from JDBC calls and then uses a Spring framework SQLExceptionTranslator (such as SQLErrorCodeSQLExceptionTranslator) to convert the exceptions into meaningful (unchecked) exceptions. This aspect is also declared to take precedence over the JdbcStatementMonitor using AspectJ's declare precedence form. This ensures that the DataAccessErrorHandling aspect has already converted exceptions before the JdbcStatementMonitor tracks the exception type being returned.

The other part of Listing 8 shows an example strategy that indicates failures only for conditions that are virtually guaranteed to indicate a failure (like the failure to access a resource). In particular, it excludes concurrency failures (which can happen in well-designed, multi-user applications) and data-integrity violations (which often happen when registering new users, for example).

For even more control over failure detection, you might rely on having a marker interface or an annotation on a method or class to indicate that certain parts of a system can throw an Exception as a normal course of action. This could be combined with the use of declare parents or AspectJ 5's new declare annotation form to capture modular rules indicating whether throwing a Throwable indicates a failure or not. In most cases, the simple rules I've indicated are a good match for coarse-grained monitoring, but it's nice to have more flexibility in cases where you want to avoid spurious warnings.

Detecting common Web failures

An application can fail even if it doesn't throw an exception. One of the most important cases occurs when a controller handles an exception and forwards the response to an error page. Another case is when the HTTP response for the page indicates an error. Listing 9 updates the ServletMonitor from Listing 4 to detect failures for such cases. I also made a small update to the AbstractRequestMonitor.RequestContext worker object to allow setting an error context and to record failures if an error context has been set.

  /** Call of send error in the Servlet API */
  public pointcut sendingErrorResponse(HttpServletResponse response,
          int sc) : 
      target(response) && call(* sendError(..)) && args(sc, ..);    
  /** Track a failure after sending an error response */
  after(HttpServletResponse response, RequestContext requestContext,
          int sc) returning : 
    sendingErrorResponse(response, sc) && inRequest(requestContext) {
      requestContext.setErrorContext(new Integer(sc));
  }

  /** Execute handle page exception in the JSP API */
  public pointcut handlePageException(PageContext pageContext,
          Throwable t) :
      call(* PageContext.handlePageException(*)) &&
      target(pageContext) && args(t);

  /** Track a failure when showing an error page */
  after(PageContext pageContext, RequestContext requestContext,
          Throwable t) returning : 
    handlePageException(pageContext, t) && inRequest(requestContext) {
      requestContext.setErrorContext(t);
  }    

Back to top

Extending library aspects

Several times, as I've built up the Glassbox Inspector, I've needed to make my library aspects more extensible. I've listed some of the common options available below. I've already demonstrated some of these options, and all can be usefully applied:

• Provide an abstract pointcut to define where the aspect applies (for example, a scope() definition).
  
  
• Provide an empty pointcut that must be overridden for certain advice to apply (as in the AbstractOperationMonitor in Listing 2). In many cases, a minority of concrete aspects will need to apply a given piece of advice, so having an empty default is more useful. Having an empty default implementation for a template pointcut like this is quite similar to having an empty default implementation for a template method.
  
  
• Rely on marker interfaces to indicate in which types the aspect should apply. Because interfaces are inherited, any subtypes of these types will also be included.
  
  
• Rely on Java 5 annotations to indicate in which types and/or methods the aspect should apply. This approach gives you more fine-grained control over where aspects should apply, allowing definition by method, as well as inclusion of a class but not its subclasses.
  
  
• Provide run-time configuration of behavior, with logic to test whether a capability is enabled. See Run-time control for an example.

With these extensions integrated into the application, I'm done extending the Glassbox Inspector for monitoring (at least for the purpose of this article -- the open source project begins where this article ends!). I now have a useful view of application performance organized logically. With this in place, I can use standard JMX clients to review overall performance statistics and errors by request in a Web application. I can also drill down into the performance and reliability of database or Web services for any that are suspect. Figure 1 shows the Glassbox Inspector monitoring tool at work on the Spring Petclinic, iBatis JPetstore, and Duke's Bookstore application examples:

Back to top

Error handling

From here forward, I'll focus on making it easy to deploy the extended Glassbox Inspector monitoring infrastructure. My first step toward an enterprise-ready management infrastructure is to extend the Glassbox Inspector for error isolation. Traditionally, developers do their best to ensure that functions like performance monitoring do not create errors that impact monitored applications. However, unexpected behavior in monitored code (for example, unanticipated null values, proxies that throw exceptions, etc.) occasionally exposes bugs in the monitoring code itself. In these cases, having the ability to isolate errors raises confidence in the robustness of the monitoring implementation. While problems from initializing the system can occasionally cause failures that the error isolation aspect can't buffer, this type of problem normally occurs immediately and is easy to catch during development, making it much less of a risk.

My basic error-isolation strategy is to catch and handle any Throwables that might be thrown from an advice-execution join point so they don't affect the underlying application. Note that the advice-execution pointcut matches join points where some advice is executing, so this aspect will affect the behavior of other aspects. This strategy works well for before and after advice, but implementing error isolation is a little more complicated for around advice. If an exception is thrown before reaching the proceed statement, I'd like to still proceed with the original join point. Conversely, if an exception is thrown by the proceed statement, I want to allow that exception to flow out and not "isolate" it (that is, not swallow it).

Because there's no join point for invoking proceed inside advice, I can't write fully generic advice that does what I want. However, I can use the structure of the Glassbox Inspector to provide error isolation for the one type of around advice that it uses. The only around advices in the Glassbox Inspector always construct a worker object and then invoke the execute() template method on it, which invokes doExecute() to proceed with the original join point. So I will handle any exceptions returned by a helper method in a monitor (AbstractRequestMonitor or any subtype) or that are returned by calls within execute or doExecute(). Listing 10 shows how I've extended the Glassbox Inspector to handle errors:

public aspect ErrorHandling {
  public pointcut scope() : 
      within(glassbox.inspector..*) && !within(ErrorHandling+);

  public pointcut inMonitor() : 
      within(AbstractRequestMonitor+) || within(RequestContext+);

  public pointcut voidReturnAdviceExecution() :
      adviceexecution() &&
      if(((AdviceSignature)thisJoinPointStaticPart.getSignature()).
             getReturnType() == void.class);

  protected pointcut monitorHelperExec() : 
      inMonitor() && execution(* *(..)) && 
      !execution(* execute(..)) && !execution(* doExecute(..));
    
  protected pointcut monitorExecuteCall() : 
      (inMonitor() && withincode(* execute(..)) || 
      withincode(* doExecute(..))) && 
      (call(* *(..)) && !call(* doExecute(..)) || call(new(..)));
    
  public pointcut handlingScope() : 
       scope() && 
         voidReturnAdviceExecution() || monitorHelperExec() || monitorExecuteCall();

  /**
   * This advice ensures that errors in the monitoring code will not
   * poison the underlying application code.
   */
  Object around() : handlingScope() {
      try {
          return proceed();
      } catch (Throwable e) {
          handleError(e, thisJoinPointStaticPart);
          return null;
      }
  }

  public synchronized void handleError(Throwable t, 
          StaticPart joinPointStaticPart) {
      // log 1 in 1000 but don't rethrow
      ...
  }

  /** 
   * Count of errors: a weak map so we don't leak memory after apps
   * have been disposed of.
   */
  private Map/* <JoinPoint.StaticPart, MutableInteger> */errorCount =
      new WeakHashMap();
    
...
}

About the error-isolation layer

Looking at Listing 10, you can see that I first define a pointcut for scope to limit advice in ErrorHandling to apply only to code in the Glassbox Inspector but not to the ErrorHandling aspect itself. Then, I define the inMonitor() pointcut to define code that is lexically within the request monitor (including any worker objects). I define the voidReturnAdviceExecution() pointcut to match advice executions that return void by using an if test based on the signature of the advice execution. Before and after advice always returns void and none of the Glassbox Inspector around advices do. For my system, this is equivalent to matching advice execution for before and after advice, but not for around advice.

Next, I define monitorHelperExec() to match the execution of helper methods in the monitor (that is, any call other than execute() or doExecute()). I define monitorExecuteCalls() as any call from within the execute() or doExecute() methods in the monitor other than calls to doExecute() itself. These pointcuts are combined in handlingScope() to define the join points where I will handle exceptions.

The net effect is that I handle exceptions at any before or after advice and at the right points in the monitoring around advice to prevent errors from affecting application code, but not to swallow application exceptions. To handle the exceptions, I use around advice that catches any Throwable thrown by proceeding and then delegates to a helper method to log them when first encountered. The advice then returns without throwing an exception. The Glassbox Inspector also uses AspectJ's declare soft form to indicate to the compiler that it handles any checked exceptions inside of advice.

Overall, I've been able to create a very effective error-isolation layer. The changes required to add this to the Glassbox Inspector are low overhead, as indicated by the performance measurements you'll see below.

Back to top

Run-time control

Having built a useful monitoring infrastructure, I now want to add run-time control over the monitoring to be performed without restarting the server. The run-time control infrastructure will also let users of the Glassbox Inspector add higher overhead monitors to capture more detailed data when a system is having problems. I can dynamically enable or disable the monitor as a whole, or each advice individually, at run time by adding an if test for a simple flag to each advice's pointcut. To provide a common framework for control, I use the idiom in Listing 11 to enable and disable the advice for each aspect:

public aspect AbstractRequestMonitor {
...
    protected pointcut scope() : if(true);
    protected pointcut monitorEnabled() : isMonitorEnabled() && scope();
    protected abstract pointcut isMonitorEnabled();

    public boolean isEnabled() {
        return enabled;
    }
    public void setEnabled(boolean enabled) {
        this.enabled = enabled;
    }
    protected boolean enabled = true;
}

public aspect JdbcStatementMonitor extends AbstractResourceMonitor {
...
    /** Monitor performance for executing a JDBC statement. */
    Object around(final Statement statement) :
      statementExec(statement) && monitorEnabled() {
        ...
    }
...
    protected pointcut isMonitorEnabled() : if(aspectOf().isEnabled());
}

Naturally, I will use JMX to provide the necessary run-time control, as shown in Listing 12:

public aspect MonitorJmxManagement {
  /** 
   * Management interface for monitors allows enabling and disabling 
   * at runtime. 
   */
  public interface RequestMonitorMBean extends ManagedBean {
      public boolean isEnabled();
      public void setEnabled(boolean enabled);
  }

  /** 
   * Make the {@link AbstractRequestMonitor} aspect implement 
   * {@link RequestMonitorMBean}, so all instances can be managed 
   */
  declare parents: AbstractRequestMonitor implements MonitorMBean;

  public String AbstractRequestMonitor.getOperationName() {
      return "control=monitor,type="+getClass().getName();
  }
    
  public MBeanInfoAssembler AbstractRequestMonitor.getAssembler() {
      if (assembler == null) {
          initAssembler();
      }
      return assembler;
  }
...
}

The MonitorJmxManagement aspect defines the management interface of a monitor to consist of a single attribute, enabled. Most of the additional support required to manage the monitors through JMX comes from the JmxManagement aspect discussed in Part 1. I also moved some common code from StatsJmxManagement into that shared class. Figure 2 shows an example of controlling monitoring at run time with JMX:

Back to top

Deploying the Glassbox Inspector

At this point, I'm ready to deploy the Glassbox Inspector for application monitoring. I've written the code to allow it to be applied to just application code as much as possible. In many cases, however, applying aspects to library code is advantageous. In the case of the Petclinic and JPetstore sample applications I worked with, the JDBC access is handled by Spring and iBatis library code respectively, and in the Petclinic, the core controller logic is handled by the Spring library. So it is often advantageous for monitoring aspects to affect some of the library code used by applications. Sometimes, you can avoid advising execution of library code by advising calls from the application code or by adding explicit hooks into the application code that can be advised (for example, by subclassing methods defined in a library class).

I have a choice of two basic approaches to weaving monitoring into applications: at build time or at load time. If I use build-time weaving, it is much more convenient to process binary jars (for example, using the ajc -inpath command line tool to perform weaving), rather than rebuilding the libraries from source. I've used this approach in the past, and it does offer the best performance on application startup and it imposes minimal memory overhead. However, it can add significant complexity to the build environment. One issue is the requirement that the AspectJ weaver be able to resolve references to third party classes contained in the jar being woven. In the case of Spring, this means including JARs such as Hibernate, a variety of open source cache managers, and the Servlet API.

The other approach that has become practical with AspectJ 5 is load-time weaving. Using this approach, I can build monitoring aspects separately and include a small deployment descriptor to define where the aspects apply.

Load-time weaving

Listing 13 shows an example of an aop.xml deployment descriptor file that describes where load-time weaving should apply. AspectJ 5 load-time weaving agents find all META-INF/aop.xml resources in a directory or jar accessible by any ClassLoader in the system. These aspects can be loaded automatically by any Java ClassLoader agent (such as the Java 5 JVMTI -javaagent, the JRockIt JVM agent, or even the WebSphere or WebLogic ClassLoader plugins) to apply aspects to all or some of the code in a system. This lets me apply monitoring to an application or a server without building it in up front, and it avoids the need to change build processes.

Note that code in (for example) the Spring framework that refers to classes that aren't on the classpath will never be loaded, so there's no need to add extra jars with this approach. As of this writing in November 2005, AspectJ 5 Milestone 4 has been released and the final release of AspectJ 5 is anticipated by year-end. Load-time weaving is very useful in preserving module independence in larger systems; it provides the same kind of run-time resolution for aspects that Java class loading does for objects.

<aspectj>
    <weaver>
      <exclude within="org.springframework.jmx..*"/>
      <!--  don't reweave -->
      <exclude within="glassbox.inspector..*"/>
    </weaver>
    <aspects>
      <aspect
 name="glassbox.inspector.monitor.operation.AxisOperationMonitor"/>
      <aspect
 name="glassbox.inspector.monitor.operation.SpringMvcMonitor"/>
      <aspect
...
      <aspect
 name="glassbox.inspector.monitor.resource.JdbcStatementMonitor"/>
      <concrete-aspect
 name="glassbox.inspector.monitor.operation.CustomMvcMonitor "
 extends="glassbox.inspector.monitor.operation.TemplOperationMonitor">
        <pointcut name="classControllerExecTarget"
           expression="execution(* com.ibatis.struts.BaseBean..*(..))
&& cflow(execution(* com.ibatis.struts.BeanAction.execute(..)))"/>
      </concrete-aspect>
    </aspects>
</aspectj>

The aop.xml file in Listing 13 starts with a definition of where the AspectJ weaver should and should not apply aspects. By default, it affects any code loaded through the ClassLoader. I've added some exclusions for some parts of the system where load-time weaving is not required to improve start-up time.

The AOP configuration file then lists all the aspects to be applied. Currently, this list of aspects must be maintained by hand, which is a common source of bugs. Fortunately, it is planned that the final release of AspectJ 5 will include a facility to generate these lists when compiling aspects.

Finally, note that the example file includes an XML-defined aspect, CustomMVCMonitor. This concrete aspect extends an abstract aspect to monitor an application-specific framework. Basically, it just defines a pointcut for operations that should be monitored. This shows how you can extend the Glassbox Inspector to monitor custom code without having to compile any code. You can also prevent monitoring aspects from running by removing them from the aop.xml file. If you undeploy them in this way, you can reduce overhead but you cannot re-enable them without restarting the application (or server). This technique can also be used to limit monitoring. For example, you could define a more narrow pointcut for monitoring Struts actions in an XML-defined aspect and exclude the standard StrutsMonitor.

Running Tomcat 5.5 with load-time weaving

I'll demonstrate the deployment of Glassbox Inspector on the Tomcat 5.5 application server. This server implements Java 5 by default, so I use Java 5's -javaagent start-up parameter to invoke AspectJ 5 load-time weaving. (See Resources to learn about using AspectJ load-time weaving on earlier versions of the Java language.) To use this feature, you simply need to add a new java option flag for start up. This is typically done by editing a shell script. For example you could add the following line to the start of setclasspath.bat on a Windows machine:

 set JAVA_OPTS=-javaagent:%CATALINA_HOME%\common\lib\aspectjweaver.jar -Xmx256m 

You can make a similar addition to the setclasspath.sh script for Linux or other Unix operating systems.

This setup simply allows load-time weaving in the Tomcat VM. To monitor Web applications, I can build them as usual and add the glassboxInspector.jar file and its dependencies into their WEB-INF/lib directory subsequently. The AspectJ weaver then finds the aop.xml file that I've deployed and ensures that the aspects it defines are woven as applications are loaded. Alternately, I could add the glassboxInspector.jar and file to the shared/lib folder for Tomcat. This would add monitoring to all Web applications in the server without requiring me to add this jar to each application. This solution is analogous to adding any other library functionality to a single application or to all applications in an application server. I could even put the inspector on my system classpath or Tomcat's common/lib folder. This would let me monitor the internals of Tomcat, although I haven't found much need for that, and doing so would require additional start up time.

I prefer to deploy monitoring across the entire server at once. Typically, application servers are managed as a whole, and it's desirable to have consistent management and monitoring capabilities across all applications on the server.

Back to top

Performance and memory use

The trade-off for the simplicity of using load-time weaving is the additional time required to start an application. I have been working with Alexandre Vasseur and Matthew Webster to optimize the performance of load-time weaving before the AspectJ 5 release. Already the overhead is reasonable, and we have identified some helpful optimizations to integrate. I anticipate further optimizations of load-time weaving performance will continue over the next year. Likewise, these tests are on an alpha version of the Glassbox Inspector. I expect that the overhead will drop significantly with further tuning of the Glassbox Inspector and AspectJ load-time weaving. I plan to optimize the Glassbox Inspector to use the java.util.concurrent library, with aspects that can use the most efficient concurrent implementation depending on what Java VM the system is running.

I took some simple measurements of server start-up times and median time per request using an October 2005 development build of AspectJ (after 1.5.0 M4) on a Windows XP workstation. This system was a developer build (after alpha 1) of the Inspector. In my test, I ran Tomcat 5.5.9 on the JRockIt 1.5.0_03 Java VM and auto-started the iBatis 1.3 JPetstore, Spring 1.2.1 Petclinic and Duke's Bookstore applications.

The end-to-end response time overhead was minimal, once the system had started. With 50 simultaneous users running on the same machine, the median server response time increased by about 10 milliseconds. However, start-up time with load-time weaving took about 15 seconds, including weaving time. By contrast, starting up without weaving took about five seconds. Finally, the use of load-time weaving added significant memory overhead: The Java process used about 100 percent more memory with load-time weaving than without.

Table 1 shows the per request overhead, net startup time, and memory usage when the Glassbox Inspector is enabled with load-time weaving, deployed with load-time weaving but disabled, or not deployed:

Back to top

In conclusion

In this two-part article, I've shown you how you can use AspectJ to tackle a complex crosscutting problem (in this case application monitoring) and build a solution incrementally. Even if you choose not to take an aspect-oriented approach to your next monitoring solution, many of the lessons and techniques described here will prove valuable. I certainly have found them to be useful on my other AspectJ projects.

That said, I hope this article has made a good case for using aspect-oriented code for monitoring. The aspects I've written are much easier to extend and adapt than traditional scattered and tangled instrumentation code, and they provide a better way to get data about production applications. Without aspects, it would be hard to implement the advanced functionality described here just once, let alone adapt it as you learn more about a production application.

Development and analysis of the Glassbox Inspector monitoring infrastructure need not end with this article. The project is looking for contributions in many areas, such as:

• Capturing more context about what's happening (for example, stack traces and parameters), especially for unusual results such as failures or abnormal response times.
  
  
• Monitoring more operations, resources such as JMS and EJB, and expanding on the Inspector's basic support for monitoring XML document processing.
  
  
• Handling distributed monitoring, to track information across clustered applications and to correlate information across distributed calls.
  
  
• Using Java 5 management information, such as CPU times or thread-specific statistics.
  
  
• Using application server JMX statistics, such as thread pools.
  
  
• Capturing history and trends with persistent storage and reports.
  
  
• Using JMX to provide alerts and exposing statistical summaries. It would be very nice to roll-up key information about nested monitors in the JMX statistics.
  
  
• Monitoring top-level resource information (for example, total time spent calling services or connecting to databases). This could be provided with better summary data.
  
  
• Providing different degrees of statistical summarization (for example, histograms of times spent in a request).
  
  
• Adaptively discovering relevant parameters to track (for example, for unknown database queries or Servlet requests).
  
  
• Providing resource monitoring for higher-level database and service access frameworks (like Hibernate, TopLink, EJB 3 Persistence Managers, JAX-WS, etc.).
  
  
• Allowing sampling to vary the amount of data captured (across an entire request).
  
  
• Monitoring system events such as 404 errors for requests to a Web application that aren't bound to a Servlet. This is a good case for advising Servlet filters, including adding an empty Servlet filter to match any requests for an application.
  
  
• Monitoring business events such as customer purchases or abandoned shopping carts.

Now it's your turn. I encourage you to download the Glassbox Inspector and try it out. I also would love for you to contribute to the project, whether by offering feedback, developing another monitor, extending the system in one of the directions suggested above, or establishing a new direction for the project. See Download for the complete Glassbox Inspector source code. See Resources to learn more about the topics discussed here.

Acknowledgments

Thanks to Eugene Kuleshov, Nicholas Lesiecki, Eamonn McManus, Srinivas Narayanan, Ramnivas Laddad, Will Edwards, Matt Hutton, David Pickering, Rob Harrop, Alex Vasseur, Paul Sutter, and Mik Kersten for reviewing this article and giving such insightful comments.

Back to top

Download

Information about download methods

Resources

Learn

• "AOP@Work: Performance monitoring with AspectJ, Part 1" (Ron Bodkin, developerWorks, September 2005): First steps toward building an aspect-oriented monitoring system.
  
  
• "AOP@Work: Introducing AspectJ 5" (Adrian Colyer, developerWorks, July 2005): Describes load-time weaving and annotation support in AspectJ 5.
  
  
• Load-Time Weaving for WebSphere (Ron Bodkin's Blog, October 2005): Includes entries on load-time weaving for AspectJ, including how to use it with WebSphere.
  
  
• JRockit JVM Support for AOP (Jonas Bonér and Alex Vasseur with Joakim Dahlstedt; dev2dev, August 2005): Introduces JRockit JVM support for AOP and discusses in depth the pros and cons of such support.
  
  
• Glassbox home page: Learn more about the Glassbox Inspector and get involved!
  
  
• Zen and the Art of Aspect-Oriented Programming (Ron Bodkin and Ramnivas Laddad, Linux Magazine, 2004): Describes in detail the Exception Conversion pattern and error handling with AOP.
  
  
• AOP@Work: Incorporate AOP into your day-to-day Java programming.
  
  
• The Java technology zone: Hundreds of articles about every aspect of Java programming.
  
  

Get products and technologies

• Glassbox Inspector: An open source project hosted on Java.net.
  
  
• AspectJ homepage: Download AspectJ.
  
  
• JMX home page: Download JMX.
  
  
• The Apache Software Foundation: Download Tomcat 5.
  
  
• The Spring framework: Source of the Petclinic example used in this article (see Part 1 for the Duke's Bookstore example).
  
  

Discuss

• developerWorks blogs: Get involved in the developerWorks community.
  
  

About the author

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• Reusable operation monitors
• Monitoring multiple applications
• Concrete operation monitors
• Updating JMX for nested statistics
• Monitoring Web service calls
• Monitoring for failure
• Extending library aspects
• Error handling
• Run-time control
• Deploying the Glassbox Inspector
• Performance and memory use
• In conclusion
• Download
• Resources
• About the author
• Comments

Next steps from IBM

Spring, JRuby, and Ajax development is easier with WebSphere Application Server - a smart Java 5 and J2EE Web services-based application server.

---

• Try: The no-charge WebSphere Application Server Community Edition is a pre-integrated, lightweight Java 5 application server built on Apache Tomcat and other best-of-breed open source software such as OpenEJB, Apache Axis, and Apache Derby.
• Article: The article leverage the Spring Framework and the WebSphere Application Server to improve your J2EE project productivity.
• Tutorial: See how the free WebSphere Application Server and XML can improve the efficiency of your JRuby on Rails and Ajax development.
• Buy: WebSphere Application Server - Express

My developerWorks community

Dig deeper into Java on developerWorks

Rate a product. Write a review.

Write a review for Rational products today!

Special offers