Problem Determination through Defensive Infrastructure Architecture: Part 1: Runtime deployments with queues and ESBs

The widely known "keep it simple strategy" (KISS) is a wise practice to follow when building out an infrastructure. Application deployments, packaging and placement requires considerable thought and effort to execute well. However, sometimes what appears to be a simple implementation can hinder problem determination leaving a production runtime unstable or, worse, unavailable. What is simpler for one side of the IT development or operations staff makes troubleshooting and root cause analysis that much harder because it is difficult to separate out discrete events or the lack of valuable but missing data. This article looks at some real life examples where "more is actually less" when it comes to KISS.

Share:

Kyle Brown (brownkyl@us.ibm.com), Distinguished Engineer, IBM  

Kyle BrownKyle Brown is a Distinguished Engineer with IBM Software Services for WebSphere, specializing in SOA and Emerging Technologies. Kyle provides consulting services, education, and mentoring on SOA, object-oriented topics and J2EE technologies to Fortune 500 clients. He is a co-author of Enterprise Java Programming with IBM WebSphere and Persistence in the Enterprise. He is also a frequent conference speaker on the topics of SOA, Enterprise Java, OO design, and design patterns.


developerWorks Professional author
        level

Alexandre Polozoff (polozoff@us.ibm.com), Sr. Certified IT Specialist, IBM  

Alexandre PolozoffAlexandre Polozoff is an IBM Master Inventor and a Senior Certified IT Specialist in the Software Group Performance Technology Practice across the WebSphere product stacks (Application Server, Portal, Process Server, etc) at large topology, international customer sites. He has authored numerous papers and blogs related to performance problem determination around J2EE technologies and speaks at various conferences.



14 July 2009

Introduction

In a previous article, Kyle Brown and Carolyn Norton have written about the notion of Defensive Architecture, and shown some of the decisions that you can make in developing your overall application architecture and infrastructure architecture to be more resilient in the face of failure and change. In this article, we will further examine these notions by looking at two examples where you can improve problem determination through the use of defensive techniques. We will first examine message-driven beans (MDBs) and runtime deployments supporting them, and then look at how an enterprise service bus (ESB) can be helpful in building a defensive infrastructure.


Keeping it Simple

Keeping things simple is a two-edged sword. Often you can design an infrastructure with the goal of making it less complex; but in trying to keep things simple you make your application runtime more complex. Lets look at a simple problem that illustrates this principle:

In this example we have an application module (A) that not only takes input off a queue but also has data it needs to send to to another application (B) over the same queue in order to have that application process the data. B has mdb-x and mdb-y which process messages for specific destinations. In this case the flow is that Application A creates a message that is processed in Application B by mdb-x and which then creates another message that is processed by mdb-y.

There are, at least, two ways one can approach this deployment and how queues are utilized.

Figure 1. Process A sends message to Process B which puts another message onto the same queue
Process A sends message to Process B which puts another message onto the same queue
Figure 2. Process B1 outputs its message to a separate queue that flows to a B2 deployment
Process B1 outputs its message to a separate queue that flows to a B2 deployment

Let us examine both approaches and see compare and contrast the complexity of the runtime with a specific slant on problem determination in both environments.

Figure 1 shows what seems to be a very simple queue deployment. The application module is configured to feed data it needs to process in another step back to itself. There is less to deploy and configure because everything is self contained. It seems to adhere to our keep it simple philosophy.

Figure 2 shows what looks like a more complex administrative task. The same application module is deployed more than once; what's more there are several queues to maintain. Some will argue that having so many moving parts leaves things open for being mis-configured when deployed. But if administrators use scripts to build each environment (functional test, user acceptance test, performance test, production) that once the application is promoted to production the repeatable scripts will have been well tested and less likely to fail.

While on the surface Figure 2 looks more complex it actually makes problem determination simpler than the runtime depicted in Figure 1 for the following reasons:


Measuring throughput

When looking at end to end throughput of an environment a crucial measuring point would be the queue depths and how quickly they drain. In Figure 1 when the queue depth is measured it is measured as an aggregate where the combined postings from both the remote process and the application module itself thereby making it difficult to tell if the queue is filling up due to the remote process or the local. There is no way to differentiate between the messages coming from one source or another.

Using the runtime in Figure 2 makes several things easily obtained. First, the queue depths of both the remote process and those generated by messages from the application module itself are easily distinguished from each other. Since applications only listen to one queue at a time it does require the deployment of another MDB to catch those messages.


Easier Separation and Tuning

Figure 2 can easily have the 2nd deployment of the application MDB occur on separate hardware. This allows for handy separation should any capacity increases occur. It also allows for separately tuning the JVM to the specific characteristics of the application's function if the MDBs are deployed to separate JVMs. It is not unusual to have different MDBs exhibit different performance characteristics because they have wildly different function.

There is a third option sort of a cross between Figure 1 and 2 where the B deployment remains in a single JVM but uses a 2nd queue to separate the messages. This may also be considered for MDBs that co-exist peacefully in the same JVM.


A Similar Problem

This same issue that sometimes “less does not equal more” comes up again in the case of mediations in an ESB. Consider the following problem: We have two SOAP messages in our application. One needs to be translated from one XML schema to another between the service requestor and the service provider. The other one, however, does not need translation.

In this case the simplest solution would be to place a mediation on the first message, but not on the second one. In fact, you might go so far as to not even use an ESB for the second message since there's no obvious need for mediations. However, using the ESB and perhaps adding a mediation in the second case, even if it does not do any translation is still valuable.

Consider the issue of problem determination of the two messages; in the first case, it's simple to log the message that has been translated both on its incoming (untranslated) and outgoing (translated) versions. In the second case where the message does not go through the ESB, problem determination is more difficult since you cannot easily examine the message as it flows from requestor to provider. So, routing the message through the ESB allows us to add a simple logging mediation when it is needed. Even though you've added some complexity at deployment time, the benefits gained in problem determination make it worthwhile.


Conclusion

While a runtime environment can seem to have a simple configuration it may impede the ability to properly conduct problem determination and/or performance tuning. This is a reason to always be looking from a different angle at alternative architectures that provide defensive capabilities in terms of how the environment can be monitored, controlled and performance tuned. Commonly the performance test environment tends to be where such defensive architectures readily become apparent.

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into SOA and web services on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=SOA and web services
ArticleID=412936
ArticleTitle=Problem Determination through Defensive Infrastructure Architecture: Part 1: Runtime deployments with queues and ESBs
publish-date=07142009