The Just-In-Time Throttler and Dispatcher (hereafter called the JITT&D) extends IBM® WebSphere® ESB mediation to provide throttling and dispatching, so that requests to service providers can be better distributed and regulated in mediation flows.
- Part 1. Throttling and dispatching with ESBs explains throttling and dispatching, and why they are important for ESBs.
- Part 2. Requirements and possible designs describes the requirements and possible solution designs to fulfill them.
- Part 3. Implementation explains the technical implementation and runtime configurations.
- Part 4. Using the JITT&D shows you usage details, including how to invoke it from mediation modules, and how to use its interface for technical monitoring.
- JITT&D installation code, including samples
- JITT&D installation and programming guide
Part 1. Throttling and dispatching with ESBs
Throttling and dispatching enables you to better control how service requests are addressed to target service providers by ESBs.
From an ESB perspective, throttling refers to controlling the number of requests addressed concurrently to target services. For example, if the number of requests processed concurrently by a service should not exceed three in order to obtain good performance, the ESB will not submit more than three requests to the service at the same time, and excess requests will be queued until some of the in-progress requests are completed. Throttling is important in order to avoid target services receiving more requests that they can fulfil, which can harm performance.
Dispatching refers to distributing requests between several endpoints. In an SOA, several endpoints often implement the same logical service, and requests addressed to that service must be distributed between the endpoints. Some service hosting infrastructures, such as WebSphere Application Server, include native dispatching capabilities (workload balancing through plug-ins installed in HTTP servers), but others do not. Therefore, the ESB should also be able to dispatch requests between several endpoints.
Dispatching is important in order to dynamically distribute requests in a balanced way between the various endpoints implementing the same service, so that good performance is maintained.
Part 2. Requirements and possible designs
The JITT&D was implemented as a proof-of-concept for a customer that required regulation of service calls. Here are some of the requirements:
- Outgoing service requests must be controlled so that no more than a specified number of requests are sent concurrently to the same endpoint.
- Outgoing service requests must be dispatched between several endpoints implementing the same logical service (and sharing common WSDL and a common implementation). When dispatching, the workload must be balanced between the available endpoints and when possible the requests should be addressed to the endpoint with the lowest activity rate. In addition, dispatching should take into account different endpoint capacities, since some endpoints may be able to process more concurrent requests than others.
- Both request-response and one-way service calls must be throttled and dispatched.
- The tool must provide high performances and impose minimal overhead on the ESB.
- The tool must be able to throttle and dispatch the requests processed by several ESB servers running on different physical servers. And it should work in high availability environments, so that requests can always be throttled and dispatched.
- The tool must enable users to change the configuration dynamically (while requests are being processed, including changing the maximum throughput allowed for each endpoint, and adding or removing endpoints for a service.
- The tool must provide graphical capabilities for technical monitoring:
- A dashboard must display key performance indicators such as the input and output throughputs, number of requests waiting and in process, and average waiting and processing times.
- A graphical display should show how these indicators change over time.
Given the above requirements for a WebSphere ESB throttler and dispatcher, four solution designs were considered:
Pure Java™ solution
The throttler and dispatcher is implemented as Java classes running in the WebSphere ESB JVM, with the data needed to manage the endpoint states also stored in the JVM memory:
Figure 3. Pure Java solution
Advantages of this solution are homogeneity (the mediations and the throttler and dispatcher both run in the same environment) and high-performance design (entirely synchronous).
The throttler and dispatcher is implemented as a database with logic implemented as stored procedures to service WebSphere ESB requests:
Figure 4. Database solution
Advantages of this solution are that it facilitates the sharing of the throttling/dispatching functions between several WebSphere ESB servers and that the data to be managed is stored very safely.
JMS queue solution
Another way to throttle requests with WebSphere ESB is to split the mediation module and control throttling on the connection between the two resulting modules. If the connection is implemented with a JMS binding, then you can use the maximum concurrency parameter on entry of the second mediation module (Export with JMS binding). This parameter ensures that no more than <maximum_concurrency> instances of the second mediation will be executed concurrently, and therefore no more than <maximum_concurrency> requests will be addressed concurrently to the target endpoint:
Figure 5. JMS queue solution
This solution is easy to implement, with no programming needed. However, its performances may be poor, because the pattern is asynchronous pattern and the mediation application is split into two modules, thus causing some processing overhead. Moreover, it does not provide dispatching.
WebSphere Application Server native throttling
WebSphere Application Server provides a parameter (com.ibm.websphere.webservices.http.maxConnection), set in the JVM custom properties, that defines the maximum number of Web Service requests which can be submitted concurrently by all Web service client applications running in the application server. This parameter also applies to the Web service requests emitted by WebSphere ESB (imports with Web service binding) and therefore can be used for throttling with WebSphere ESB. Although easy to use, this parameter is global and does not let you specify a specific number of concurrent requests for each target service. Moreover, it does not provide dispatching.
Part 3. Implementation
The JITT&D was implemented according to the first pattern described above, the pure Java solution, with a strong tie with WebSphere ESB V7.0 and a focus on synchronization and high throughput. The solution involves a set of Java classes executing in the WebSphere ESB JVM, with data stored as in-memory data in the JVM.
The JITT&D controls and distributes the SOAP requests sent by WebSphere ESB mediations to service providers. The target endpoints are managed by groups: all endpoints implementing the same service belong to a group representing this service. Each endpoint in a group can be assigned a specific value for the maximum number of concurrent requests, to account for the fact that they are hosted by more or less powerful servers. The JITT&D can manage several such groups in parallel. All configuration data such as groups, endpoints and their assigned parameters are stored in a configuration file.
All ESB mediations that include service requests first contact the JITT&D to get a token for one of the endpoints of the group representing the target service. These requests for a token are serviced by a singleton class named Dashboard, which manages the status of the endpoints of one or several groups. The Dashboard class then assigns tokens to mediations based on the maximum number of concurrent requests per endpoint (which must not be exceeded) and on the need to dispatch requests between the endpoints of the group. These assigned tokens are released in later stages, so that other mediations can use them. The method of releasing tokens differs for request-response and one-way calls.
Figure 6. JITT&D global overview
In request-response mode, the mediation flow, just before invoking the target service, contacts the JITT&D (API call) to get a token for one of the endpoints of the group. The JITT&D manages a table representing the status of the endpoints, and it searches the table for a free token that is not currently assigned. Two search algorithms are provided:
- Basic -- The first free token found is used, with "round robin" to get some balancing over the endpoints.
- Optimized -- The endpoint with the lowest activity rate is searched for and used. The activity rate is the ratio between the number of tokens currently assigned and the total number of tokens for an endpoint. This algorithm can take into account different capacities for different endpoints in a group.
Depending on the result of this search:
- If a free token is found, its status is changed to "assigned" and the URL of the corresponding endpoint is returned to the mediation flow, which invokes the Web service using the provided URL.
- If all tokens of all endpoints in the group are assigned, the mediation flow request is stored in a FIFO queue until a token becomes free.
When processing the service response, the mediation flow contacts the JITT&D again in order to release the token. The JITT&D checks whether the queue contains pending requests:
- If there are requests waiting, the token is assigned to the next request, which is removed from the queue.
- If the queue is empty, the token status is simply changed to "free."
Figure 7. Request-response mode
In one-way mode, the token is assigned following the same mechanism as for request-response. However, no response is processed and therefore the previous mechanism cannot be used to release the token. Instead, a token is assigned to a mediation flow for a time slot that represents the theoretical execution time of the target Web service. When this time slot ends, the token is released by the JITT&D and either assigned to another mediation flow or switched to "free" status:
Figure 8. One-way mode
The target Web service invocation may lead to an error, of which there are two types:
- Unrecoverable errors -- The request must be returned to the client. For example, a service call fails because the input data is corrupted.
- Recoverable errors -- The request must be submitted to another endpoint. For example, a service call fails because the endpoint is not available.
The JITT&D manages a list of recoverable errors. When processing a response, it identifies such errors and triggers the resubmission of the request to another endpoint of the group. It also registers the failing endpoint as "Suspended," and no more requests are addressed to this endpoint for a period of time. After that period, the JITT&D unsuspends the endpoint and resumes sending requests to it.
While managing token assignments to mediation flows, the JITT&D also computes statistics about its activity, which is done by the History class. You can display these statistics graphically through the user interface described below. With the user interface, the JITT&D configuration can be adjusted dynamically (while token assignments to requests are being done). You can tune the following parameters:
- Maximum number of tokens associated to an endpoint.
- Endpoints can be added to groups or removed from them.
Two versions of the JITT&D were implemented, one for single-server environments and one for clustered and multi-server environments.
In this case, the mediation applications and the JITT&D both run in the same WebSphere ESB JVM and the communication between the two components is direct, with no network layer involved.
Figure 9. Single-server architecture
In the multi-server architecture, mediation applications needing common throttling and dispatching are executed on several servers. These servers may belong to a common cluster but do not have to. Configurations with completely independent servers can also be used. In the multi-server architecture, a single instance of the JITT&D executes on a single server and all mediation applications contact it through the network in order to request and release tokens. The network communication between the mediations and the JITT&D is implemented in REST.
For high availability, the JITT&D should be installed in a cluster and configured to be active on one server at a time. Then if the server on which the JITT&D is running breaks down, the JITT&D will automatically restart on another member of the cluster.
Figure 10. Multi-server architecture
Part 4. Using the JITT&T
Integration with mediation applications
The JITT&D is invoked by mediation flows in order to request and release tokens for service endpoints. These invocations are done using a Java API.
It is recommended to invoke the JITT&D from a specific mediation flow, different from the one holding the core mediation functionality such as transformation and routing. This separation facilitates the resubmission of requests that end up with recoverable errors. Resubmission is triggered by a Callout node in the first mediation flow.
The throttling & dispatching mediation contains the code related to the JITT&D invocation.
In the request flow of the throttling & dispatching mediation, the correlation context must be populated with a business object that is used to propagate the request message ID to the response flow. By default, the request message ID is not propagated to the response flow, in order to enable the JITT&D to correlate the request flow and response flow calls. The correlation context business object must contain a string field named requestMessageID.
Figure 12. Input node detailed properties
Then a custom mediation node is used to invoke the JITT&D in order to request a token for a group, whose name is specified as the first parameter of the API call (QueryPersonDispatchThrottleGroup in the example below). The JITT&D, after having assigned a token to the mediation, sets the URL of the assigned endpoint into the service message object (/headers/SMOHeader/Target/address field) and also sets the request message ID into the correlation context requestMessageID field:
Figure 13. Mediation flow request diagram
In the response flow, a custom mediation node is used to invoke the JITT&D in order to release the token. The token is retrieved using the group and the request message ID (found in the correlation context of the service message object). If the service invocation fails, the JITT&D checks the error and determines whether it is recoverable (must be submitted again to another endpoint of the group) or unrecoverable (must be returned to the client application). This decision is returned to the mediation flow as a boolean value.
Figure 14. Mediation flow response diagram
When a resubmission is needed, the throttling mediation returns a modeled fault to the transformation and routing mediation. The Callout node of this mediation flow must be configured with retry on modeled faults.
Figure 15. Callout node retry settings
Technical monitoring and tuning
While managing token assignments to mediation flows executing in WebSphere ESB, the JITT&D also gathers statistics and historical data that can be displayed through a user interface. This user interface first displays the list of groups managed by the JITT&D:
Figure 16. Group selection panel
These groups, their endpoints, and all other runtime parameters are defined in a configuration file.
Group performance indicators
The selection of a group in the preceding panel displays key performance indicators about this group:
- Throughput of mediation flows asking for a token (Inputs per second) and releasing a token (Outputs per second).
- Average numbers of mediation flows waiting for a token (Waiting flows), currently using a token to call a Web service (In process flows) and both (All flows).
- Average waiting duration, processing duration, and global duration (waiting duration plus processing duration) for the mediation flows requesting and using tokens.
- Details about the group endpoints: URL, current workload (used tokens), maximum tokens.
- Buttons are available to add tokens to endpoints (+), removing tokens from endpoints (-), removing endpoints from the group (Remove), and adding endpoints to the group (Add new endpoint). These tunings do not modify the configuration file and are therefore lost when the server restarts. To make permanent changes, use the configuration file instead of the user interface.
Figure 17. Group performance indicators panel
Performance indicator history
You can use the History button in the previous panel to display historical data about how the indicators of a group evolve over time. Historical data is plotted on line charts showing:
- Evolution of the numbers of input and output flows per second
- Evolution of the numbers of waiting, in processes, and total flows
- Evolution of the waiting, processing, and global durations
Using buttons, you can move forward or backward in time, to go back to any date and time in the past and return to the current time.
Figure 18. Performance indicator history panel
This article showed you can extend ESBs with throttling and dispatching functions. It provided a generic implementation of these functions for WebSphere ESB with the JITT&D, so that requests to Web services emitted by WebSphere ESB mediations can be regulated and dispatched, and throughputs and durations can be monitored through a graphical dashboard. Advantages of this tool include "good enough" throttling and dispatching capability, high performance, and ease of use. This tool could be further enhanced with additional functions, such as priorities assigned to requests, and SLA checking of endpoints in WebSphere Service Registry and Repository. The concepts, architecture, and code explained in this article can be reused for other ESBs.
The groups, endpoints, and other runtime parameters of the JITT&D are defined in a configuration file. The available parameters and their syntax are described in the JITT&D Installation and Programming Guide. Here is an example of a configuration file:
# JIT Throttler & Dispatcher configuration properties ######################################################### # Endpoints of group '2525' Group1 = 2525 Group1_Mode = RR Group1_Endpoints_MaxReqNb = 3 Group1_Endpoint1 = http://localhost:9080/gSOAP1/ServiceMos Group1_Endpoint2 = http://localhost:9080/gSOAP2/ServiceMos Group1_Endpoint3 = http://localhost:9080/gSOAP3/ServiceMos Group1_Endpoint3_MaxReqNb = 6 Group1_History = 1 # Endpoints of group '9911' Group2 = 9911 Group2_Mode = RR Group2_Endpoints_MaxReqNb = 2 Group2_Endpoint1 = http://localhost:9080/gSOAP4/ServiceMos Group2_Endpoint2 = http://localhost:9080/gSOAP5/ServiceMos Group2_Endpoint3 = http://localhost:9080/gSOAP6/ServiceMos Group2_History = 1 # List of recoverable faults SuspendRetryFault1 = javax.xml.ws.WebServiceException: java.net.ConnectException: HTTP (404) Not Found address: SuspendRetryFault2 = javax.xml.ws.WebServiceException: java.net.SocketTimeoutException: Async operation timed out # Suspend duration (milliseconds) SuspendDuration = 180000 # Token wait time (milliseconds) TokenWaitTime = 60000 # Pending in process requests overduetime PendingInProcessRequestsOverdueTime = 120000 # Pending in process requests cleaner frequency PendingInProcessRequestsCleanerFrequency = 60000 # Number of last completed requests used for average response time calculation ResponseTimeSampleSize = 5 # Time in seconds used to calculate the average throughput ThroughputCalculationTime = 3 # Number of history recordings per minute HistoryRecordingsNb = 10 # Period of time for displaying history recordings (minutes) HistoryDisplayPeriod = 2 # Time in seconds to refresh the group view in the user interface UIGroupViewRefreshTime = 12 # Time in seconds to refresh historical graphics in the user interface UIHistoricCurvesRefreshTime = 6 # Directory to store historical data HistoryDataStorageDirectory = D:/Temp/JITThrottler # TraceLevel (0, 1 or 2) TraceLevel = 0
|JITTD installation and programming guide||JITTD_programming_guide.pdf||1129 KB|
|JITTD installation package||JITTD_installation_package.zip||184 KB|
- WebSphere ESB resources
- WebSphere ESB developer resources page
Technical resources to help you use WebSphere ESB as a flexible connectivity infrastructure for integrating applications and services to support an SOA.
- WebSphere ESB product page
Product descriptions, product news, training information, support information, and more.
- WebSphere ESB information center
A single Web portal to all WebSphere ESB documentation, with conceptual, task, and reference information on installing, configuring, and using WebSphere ESB.
- WebSphere ESB documentation library
WebSphere ESB product manuals.
- WebSphere ESB FAQs
Basic questions and answers about the new WebSphere ESB product and its relationship to other WebSphere products.
- WebSphere ESB support
A searchable database of support problems and their solutions, plus downloads, fixes, and problem tracking.
- WebSphere ESB developer resources page
- WebSphere resources
- developerWorks WebSphere developer resources
Technical information and resources for developers who use WebSphere products. developerWorks WebSphere provides product downloads, how-to information, support resources, and a free technical library of more than 2000 technical articles, tutorials, best practices, IBM Redbooks, and online product manuals.
- developerWorks WebSphere application connectivity developer resources
How-to articles, downloads, tutorials, education, product info, and other resources to help you build WebSphere application connectivity and business integration solutions.
- developerWorks WebSphere SOA and Web services developer resources
How-to articles, downloads, tutorials, education, product info, and other resources to help you design and build WebSphere SOA and Web services solutions.
- Most popular WebSphere trial downloads
No-charge trial downloads for key WebSphere products.
- WebSphere forums
Product-specific forums where you can get answers to your technical questions and share your expertise with other WebSphere users.
- WebSphere on-demand demos
Download and watch these self-running demos, and learn how WebSphere products and technologies can help your company respond to the rapidly changing and increasingly complex business environment.
- developerWorks WebSphere weekly newsletter
The developerWorks newsletter gives you the latest articles and information only on those topics that interest you. In addition to WebSphere, you can select from Java, Linux, Open source, Rational, SOA, Web services, and other topics. Subscribe now and design your custom mailing.
- WebSphere-related books from IBM Press
Convenient online ordering through Barnes & Noble.
- WebSphere-related events
Conferences, trade shows, Webcasts, and other events around the world of interest to WebSphere developers.
- developerWorks WebSphere developer resources
- developerWorks resources
- Trial downloads for IBM software products
No-charge trial downloads for selected IBM® DB2®, Lotus®, Rational®, Tivoli®, and WebSphere® products.
- developerWorks blogs
Join a conversation with developerWorks users and authors, and IBM editors and developers.
- developerWorks tech briefings
Free technical sessions by IBM experts to accelerate your learning curve and help you succeed in your most challenging software projects. Sessions range from one-hour virtual briefings to half-day and full-day live sessions in cities worldwide.
- developerWorks podcasts
Listen to interesting and offbeat interviews and discussions with software innovators.
- developerWorks on Twitter
Check out recent Twitter messages and URLs.
- IBM Education Assistant
A collection of multimedia educational modules that will help you better understand IBM software products and use them more effectively to meet your business requirements.
- Trial downloads for IBM software products
Dig deeper into WebSphere on developerWorks
Get samples, articles, product docs, and community resources to help build, deploy, and manage your cloud apps.
Keep up with the best and latest technical info to help you tackle your development challenges.
Software development in the cloud. Register today to create a project.
Evaluate IBM software and solutions, and transform challenges into opportunities.