Just-In-Time Throttler and Dispatcher for WebSphere ESB

This article describes the Just-In-Time Throttler and Dispatcher for WebSphere ESB, which enables requests to service providers emitted from WebSphere ESB to be dispatched between several endpoints implementing the same logical service, while controlling the number of concurrent requests for each endpoint.

Patrick Marie (pmarie@fr.ibm.com), WebSphere Client Technical Professional, IBM

Photo of Patrick MariePatrick Marie is a Client Technical Professional with the IBM Software Services for WebSphere team in France. He holds an engineering diploma from the Ecole Centrale de Paris, and has been working for IBM France for 28 years on various software products. For 17 years, he worked as a Client Technical Professional in the BPM, and connectivity and integration areas. He recently moved to WebSphere Services. You can contact Patrick at pmarie@fr.ibm.com.

15 June 2011

Also available in Chinese


The Just-In-Time Throttler and Dispatcher (hereafter called the JITT&D) extends IBM® WebSphere® ESB mediation to provide throttling and dispatching, so that requests to service providers can be better distributed and regulated in mediation flows.

You can download the following files at the bottom of the article:

  • JITT&D installation code, including samples
  • JITT&D installation and programming guide

Part 1. Throttling and dispatching with ESBs

Throttling and dispatching enables you to better control how service requests are addressed to target service providers by ESBs.


From an ESB perspective, throttling refers to controlling the number of requests addressed concurrently to target services. For example, if the number of requests processed concurrently by a service should not exceed three in order to obtain good performance, the ESB will not submit more than three requests to the service at the same time, and excess requests will be queued until some of the in-progress requests are completed. Throttling is important in order to avoid target services receiving more requests that they can fulfil, which can harm performance.

Figure 1.


Dispatching refers to distributing requests between several endpoints. In an SOA, several endpoints often implement the same logical service, and requests addressed to that service must be distributed between the endpoints. Some service hosting infrastructures, such as WebSphere Application Server, include native dispatching capabilities (workload balancing through plug-ins installed in HTTP servers), but others do not. Therefore, the ESB should also be able to dispatch requests between several endpoints.

Figure 2.

Dispatching is important in order to dynamically distribute requests in a balanced way between the various endpoints implementing the same service, so that good performance is maintained.

Part 2. Requirements and possible designs


The JITT&D was implemented as a proof-of-concept for a customer that required regulation of service calls. Here are some of the requirements:

  • Outgoing service requests must be controlled so that no more than a specified number of requests are sent concurrently to the same endpoint.
  • Outgoing service requests must be dispatched between several endpoints implementing the same logical service (and sharing common WSDL and a common implementation). When dispatching, the workload must be balanced between the available endpoints and when possible the requests should be addressed to the endpoint with the lowest activity rate. In addition, dispatching should take into account different endpoint capacities, since some endpoints may be able to process more concurrent requests than others.
  • Both request-response and one-way service calls must be throttled and dispatched.
  • The tool must provide high performances and impose minimal overhead on the ESB.
  • The tool must be able to throttle and dispatch the requests processed by several ESB servers running on different physical servers. And it should work in high availability environments, so that requests can always be throttled and dispatched.
  • The tool must enable users to change the configuration dynamically (while requests are being processed, including changing the maximum throughput allowed for each endpoint, and adding or removing endpoints for a service.
  • The tool must provide graphical capabilities for technical monitoring:
    • A dashboard must display key performance indicators such as the input and output throughputs, number of requests waiting and in process, and average waiting and processing times.
    • A graphical display should show how these indicators change over time.

Given the above requirements for a WebSphere ESB throttler and dispatcher, four solution designs were considered:

Pure Java™ solution

The throttler and dispatcher is implemented as Java classes running in the WebSphere ESB JVM, with the data needed to manage the endpoint states also stored in the JVM memory:

Figure 3. Pure Java solution
Pure Java solution

Advantages of this solution are homogeneity (the mediations and the throttler and dispatcher both run in the same environment) and high-performance design (entirely synchronous).

Database solution

The throttler and dispatcher is implemented as a database with logic implemented as stored procedures to service WebSphere ESB requests:

Figure 4. Database solution
Database soution

Advantages of this solution are that it facilitates the sharing of the throttling/dispatching functions between several WebSphere ESB servers and that the data to be managed is stored very safely.

JMS queue solution

Another way to throttle requests with WebSphere ESB is to split the mediation module and control throttling on the connection between the two resulting modules. If the connection is implemented with a JMS binding, then you can use the maximum concurrency parameter on entry of the second mediation module (Export with JMS binding). This parameter ensures that no more than <maximum_concurrency> instances of the second mediation will be executed concurrently, and therefore no more than <maximum_concurrency> requests will be addressed concurrently to the target endpoint:

Figure 5. JMS queue solution
JMS queue solution

This solution is easy to implement, with no programming needed. However, its performances may be poor, because the pattern is asynchronous pattern and the mediation application is split into two modules, thus causing some processing overhead. Moreover, it does not provide dispatching.

WebSphere Application Server native throttling

WebSphere Application Server provides a parameter (com.ibm.websphere.webservices.http.maxConnection), set in the JVM custom properties, that defines the maximum number of Web Service requests which can be submitted concurrently by all Web service client applications running in the application server. This parameter also applies to the Web service requests emitted by WebSphere ESB (imports with Web service binding) and therefore can be used for throttling with WebSphere ESB. Although easy to use, this parameter is global and does not let you specify a specific number of concurrent requests for each target service. Moreover, it does not provide dispatching.

Part 3. Implementation

The JITT&D was implemented according to the first pattern described above, the pure Java solution, with a strong tie with WebSphere ESB V7.0 and a focus on synchronization and high throughput. The solution involves a set of Java classes executing in the WebSphere ESB JVM, with data stored as in-memory data in the JVM.

Implementation details

The JITT&D controls and distributes the SOAP requests sent by WebSphere ESB mediations to service providers. The target endpoints are managed by groups: all endpoints implementing the same service belong to a group representing this service. Each endpoint in a group can be assigned a specific value for the maximum number of concurrent requests, to account for the fact that they are hosted by more or less powerful servers. The JITT&D can manage several such groups in parallel. All configuration data such as groups, endpoints and their assigned parameters are stored in a configuration file.

All ESB mediations that include service requests first contact the JITT&D to get a token for one of the endpoints of the group representing the target service. These requests for a token are serviced by a singleton class named Dashboard, which manages the status of the endpoints of one or several groups. The Dashboard class then assigns tokens to mediations based on the maximum number of concurrent requests per endpoint (which must not be exceeded) and on the need to dispatch requests between the endpoints of the group. These assigned tokens are released in later stages, so that other mediations can use them. The method of releasing tokens differs for request-response and one-way calls.

Figure 6. JITT&D global overview
JITT&D global overview

Request-response mode

In request-response mode, the mediation flow, just before invoking the target service, contacts the JITT&D (API call) to get a token for one of the endpoints of the group. The JITT&D manages a table representing the status of the endpoints, and it searches the table for a free token that is not currently assigned. Two search algorithms are provided:

  • Basic -- The first free token found is used, with "round robin" to get some balancing over the endpoints.
  • Optimized -- The endpoint with the lowest activity rate is searched for and used. The activity rate is the ratio between the number of tokens currently assigned and the total number of tokens for an endpoint. This algorithm can take into account different capacities for different endpoints in a group.

Depending on the result of this search:

  • If a free token is found, its status is changed to "assigned" and the URL of the corresponding endpoint is returned to the mediation flow, which invokes the Web service using the provided URL.
  • If all tokens of all endpoints in the group are assigned, the mediation flow request is stored in a FIFO queue until a token becomes free.

When processing the service response, the mediation flow contacts the JITT&D again in order to release the token. The JITT&D checks whether the queue contains pending requests:

  • If there are requests waiting, the token is assigned to the next request, which is removed from the queue.
  • If the queue is empty, the token status is simply changed to "free."
Figure 7. Request-response mode
Request-response mode

One-way mode

In one-way mode, the token is assigned following the same mechanism as for request-response. However, no response is processed and therefore the previous mechanism cannot be used to release the token. Instead, a token is assigned to a mediation flow for a time slot that represents the theoretical execution time of the target Web service. When this time slot ends, the token is released by the JITT&D and either assigned to another mediation flow or switched to "free" status:

Figure 8. One-way mode
One-way mode

Error processing

The target Web service invocation may lead to an error, of which there are two types:

  • Unrecoverable errors -- The request must be returned to the client. For example, a service call fails because the input data is corrupted.
  • Recoverable errors -- The request must be submitted to another endpoint. For example, a service call fails because the endpoint is not available.

The JITT&D manages a list of recoverable errors. When processing a response, it identifies such errors and triggers the resubmission of the request to another endpoint of the group. It also registers the failing endpoint as "Suspended," and no more requests are addressed to this endpoint for a period of time. After that period, the JITT&D unsuspends the endpoint and resumes sending requests to it.

Other functionalities

While managing token assignments to mediation flows, the JITT&D also computes statistics about its activity, which is done by the History class. You can display these statistics graphically through the user interface described below. With the user interface, the JITT&D configuration can be adjusted dynamically (while token assignments to requests are being done). You can tune the following parameters:

  • Maximum number of tokens associated to an endpoint.
  • Endpoints can be added to groups or removed from them.

Runtime configurations

Two versions of the JITT&D were implemented, one for single-server environments and one for clustered and multi-server environments.

Single-server architecture

In this case, the mediation applications and the JITT&D both run in the same WebSphere ESB JVM and the communication between the two components is direct, with no network layer involved.

Figure 9. Single-server architecture
Single-server architecture

Multi-server architecture

In the multi-server architecture, mediation applications needing common throttling and dispatching are executed on several servers. These servers may belong to a common cluster but do not have to. Configurations with completely independent servers can also be used. In the multi-server architecture, a single instance of the JITT&D executes on a single server and all mediation applications contact it through the network in order to request and release tokens. The network communication between the mediations and the JITT&D is implemented in REST.

For high availability, the JITT&D should be installed in a cluster and configured to be active on one server at a time. Then if the server on which the JITT&D is running breaks down, the JITT&D will automatically restart on another member of the cluster.

Figure 10. Multi-server architecture
Multi-server architecture

Part 4. Using the JITT&T

Integration with mediation applications

The JITT&D is invoked by mediation flows in order to request and release tokens for service endpoints. These invocations are done using a Java API.

It is recommended to invoke the JITT&D from a specific mediation flow, different from the one holding the core mediation functionality such as transformation and routing. This separation facilitates the resubmission of requests that end up with recoverable errors. Resubmission is triggered by a Callout node in the first mediation flow.

Figure 11.
Integration assembly diagram

The throttling & dispatching mediation contains the code related to the JITT&D invocation.

Request flow

In the request flow of the throttling & dispatching mediation, the correlation context must be populated with a business object that is used to propagate the request message ID to the response flow. By default, the request message ID is not propagated to the response flow, in order to enable the JITT&D to correlate the request flow and response flow calls. The correlation context business object must contain a string field named requestMessageID.

Figure 12. Input node detailed properties
Input node detailed properties

Then a custom mediation node is used to invoke the JITT&D in order to request a token for a group, whose name is specified as the first parameter of the API call (QueryPersonDispatchThrottleGroup in the example below). The JITT&D, after having assigned a token to the mediation, sets the URL of the assigned endpoint into the service message object (/headers/SMOHeader/Target/address field) and also sets the request message ID into the correlation context requestMessageID field:

Figure 13. Mediation flow request diagram
Mediation flow request diagram

Response flow

In the response flow, a custom mediation node is used to invoke the JITT&D in order to release the token. The token is retrieved using the group and the request message ID (found in the correlation context of the service message object). If the service invocation fails, the JITT&D checks the error and determines whether it is recoverable (must be submitted again to another endpoint of the group) or unrecoverable (must be returned to the client application). This decision is returned to the mediation flow as a boolean value.

Figure 14. Mediation flow response diagram
Mediation flow response diagram

When a resubmission is needed, the throttling mediation returns a modeled fault to the transformation and routing mediation. The Callout node of this mediation flow must be configured with retry on modeled faults.

Figure 15. Callout node retry settings
Callout node retry settings

Technical monitoring and tuning

While managing token assignments to mediation flows executing in WebSphere ESB, the JITT&D also gathers statistics and historical data that can be displayed through a user interface. This user interface first displays the list of groups managed by the JITT&D:

Figure 16. Group selection panel
Group selection panel

These groups, their endpoints, and all other runtime parameters are defined in a configuration file.

Group performance indicators

The selection of a group in the preceding panel displays key performance indicators about this group:

  • Throughput of mediation flows asking for a token (Inputs per second) and releasing a token (Outputs per second).
  • Average numbers of mediation flows waiting for a token (Waiting flows), currently using a token to call a Web service (In process flows) and both (All flows).
  • Average waiting duration, processing duration, and global duration (waiting duration plus processing duration) for the mediation flows requesting and using tokens.
  • Details about the group endpoints: URL, current workload (used tokens), maximum tokens.
  • Buttons are available to add tokens to endpoints (+), removing tokens from endpoints (-), removing endpoints from the group (Remove), and adding endpoints to the group (Add new endpoint). These tunings do not modify the configuration file and are therefore lost when the server restarts. To make permanent changes, use the configuration file instead of the user interface.
Figure 17. Group performance indicators panel
Group performance indicators panel

Performance indicator history

You can use the History button in the previous panel to display historical data about how the indicators of a group evolve over time. Historical data is plotted on line charts showing:

  • Evolution of the numbers of input and output flows per second
  • Evolution of the numbers of waiting, in processes, and total flows
  • Evolution of the waiting, processing, and global durations

Using buttons, you can move forward or backward in time, to go back to any date and time in the past and return to the current time.

Figure 18. Performance indicator history panel
Performance indicator history panel


This article showed you can extend ESBs with throttling and dispatching functions. It provided a generic implementation of these functions for WebSphere ESB with the JITT&D, so that requests to Web services emitted by WebSphere ESB mediations can be regulated and dispatched, and throughputs and durations can be monitored through a graphical dashboard. Advantages of this tool include "good enough" throttling and dispatching capability, high performance, and ease of use. This tool could be further enhanced with additional functions, such as priorities assigned to requests, and SLA checking of endpoints in WebSphere Service Registry and Repository. The concepts, architecture, and code explained in this article can be reused for other ESBs.

Configuration file

The groups, endpoints, and other runtime parameters of the JITT&D are defined in a configuration file. The available parameters and their syntax are described in the JITT&D Installation and Programming Guide. Here is an example of a configuration file:

# JIT Throttler & Dispatcher configuration properties

# Endpoints of group '2525'
Group1 = 2525
Group1_Mode = RR
Group1_Endpoints_MaxReqNb = 3
Group1_Endpoint1 = http://localhost:9080/gSOAP1/ServiceMos
Group1_Endpoint2 = http://localhost:9080/gSOAP2/ServiceMos
Group1_Endpoint3 = http://localhost:9080/gSOAP3/ServiceMos
Group1_Endpoint3_MaxReqNb = 6
Group1_History = 1

# Endpoints of group '9911'
Group2 = 9911
Group2_Mode = RR
Group2_Endpoints_MaxReqNb = 2
Group2_Endpoint1 = http://localhost:9080/gSOAP4/ServiceMos
Group2_Endpoint2 = http://localhost:9080/gSOAP5/ServiceMos
Group2_Endpoint3 = http://localhost:9080/gSOAP6/ServiceMos
Group2_History = 1

# List of recoverable faults
SuspendRetryFault1 = javax.xml.ws.WebServiceException: java.net.ConnectException: 
    HTTP (404) Not Found address:
SuspendRetryFault2 = javax.xml.ws.WebServiceException: java.net.SocketTimeoutException: 
    Async operation timed out

# Suspend duration (milliseconds)
SuspendDuration = 180000

# Token wait time (milliseconds)
TokenWaitTime = 60000

# Pending in process requests overduetime
PendingInProcessRequestsOverdueTime = 120000

# Pending in process requests cleaner frequency
PendingInProcessRequestsCleanerFrequency = 60000

# Number of last completed requests used for average response time calculation
ResponseTimeSampleSize = 5

# Time in seconds used to calculate the average throughput
ThroughputCalculationTime = 3

# Number of history recordings per minute
HistoryRecordingsNb = 10

# Period of time for displaying history recordings (minutes)
HistoryDisplayPeriod = 2

# Time in seconds to refresh the group view in the user interface
UIGroupViewRefreshTime = 12

# Time in seconds to refresh historical graphics in the user interface
UIHistoricCurvesRefreshTime = 6

# Directory to store historical data
HistoryDataStorageDirectory = D:/Temp/JITThrottler

# TraceLevel (0, 1 or 2)
TraceLevel = 0


JITTD installation and programming guideJITTD_programming_guide.pdf1129 KB
JITTD installation packageJITTD_installation_package.zip184 KB


  • WebSphere ESB resources
  • WebSphere resources
    • developerWorks WebSphere developer resources
      Technical information and resources for developers who use WebSphere products. developerWorks WebSphere provides product downloads, how-to information, support resources, and a free technical library of more than 2000 technical articles, tutorials, best practices, IBM Redbooks, and online product manuals.
    • developerWorks WebSphere application connectivity developer resources
      How-to articles, downloads, tutorials, education, product info, and other resources to help you build WebSphere application connectivity and business integration solutions.
    • developerWorks WebSphere SOA and Web services developer resources
      How-to articles, downloads, tutorials, education, product info, and other resources to help you design and build WebSphere SOA and Web services solutions.
    • Most popular WebSphere trial downloads
      No-charge trial downloads for key WebSphere products.
    • WebSphere forums
      Product-specific forums where you can get answers to your technical questions and share your expertise with other WebSphere users.
    • WebSphere on-demand demos
      Download and watch these self-running demos, and learn how WebSphere products and technologies can help your company respond to the rapidly changing and increasingly complex business environment.
    • developerWorks WebSphere weekly newsletter
      The developerWorks newsletter gives you the latest articles and information only on those topics that interest you. In addition to WebSphere, you can select from Java, Linux, Open source, Rational, SOA, Web services, and other topics. Subscribe now and design your custom mailing.
    • WebSphere-related books from IBM Press
      Convenient online ordering through Barnes & Noble.
    • WebSphere-related events
      Conferences, trade shows, Webcasts, and other events around the world of interest to WebSphere developers.
  • developerWorks resources
    • Trial downloads for IBM software products
      No-charge trial downloads for selected IBM® DB2®, Lotus®, Rational®, Tivoli®, and WebSphere® products.
    • developerWorks blogs
      Join a conversation with developerWorks users and authors, and IBM editors and developers.
    • developerWorks tech briefings
      Free technical sessions by IBM experts to accelerate your learning curve and help you succeed in your most challenging software projects. Sessions range from one-hour virtual briefings to half-day and full-day live sessions in cities worldwide.
    • developerWorks podcasts
      Listen to interesting and offbeat interviews and discussions with software innovators.
    • developerWorks on Twitter
      Check out recent Twitter messages and URLs.
    • IBM Education Assistant
      A collection of multimedia educational modules that will help you better understand IBM software products and use them more effectively to meet your business requirements.


developerWorks: Sign in

Required fields are indicated with an asterisk (*).

Need an IBM ID?
Forgot your IBM ID?

Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.


All information submitted is secure.

Dig deeper into WebSphere on developerWorks

ArticleTitle=Just-In-Time Throttler and Dispatcher for WebSphere ESB