Improve response time and data availability with WebSphere eXtreme Scale multi-master capability

Use the multi-master replication feature to link and synchronize data grids

Clustering and load balancing techniques are important best practices for making web applications process more work and do it faster. But if the services are spread out on a network, those same techniques can create bottlenecks and add complexity in the services that provide the data used by the application. This article describes how the multi-master replication feature of WebSphere® eXtreme Scale adds data mirroring capability to existing ObjectGrid functions. Using this feature, data can be kept close to the services that need it while still maintaining a unified view. This article also describes an important workaround that enables HTTP session data to be mirrored in the multi-master grid.


Tony Efremenko (, Certified Master IT Specialist, IBM

Tony Efremenko photoTony Efremenko is a Certified Master IT Specialist with IBM, with over 24 years of experience in the software industry.

developerWorks Contributing author

Web Identity Team, Web Identity Team, IBM

The Web Identity Team works on IBMs applications for profile and security. They identified the use cases for Extreme scale in 2007, and are rolling out parts of the application in an ongoing effort. The Web Identity team includes Fred Meredith, Brady Cooper, Snehini Sreenivasan, and Steve Looney.

08 November 2011 (First published 12 October 2011)

Also available in Chinese Japanese


Clustering and load balancing

Here is a short summary on load balancers and clustering.

When you want to use a Load Balancer

  • Load Balancers provide greater overall throughput by sending new work to the least busy servers
  • There are many options for load balancer products, including hardware devices, WebSphere Edge Server, and WebSphere Application Server plug-ins to Web Servers
  • Load Balancers can still provide session affinity, but a good algorithm or device will avoid always sending a user to the same server (to avoid affinity to failing server)

When you want to cluster the servers

  • Clustering provides improved availability and throughput by duplicating server instances
  • Horizontal clustering is when the server instances are provided on different machines
  • Vertical scaling is when the instances are added to the same machine (useful if you have untapped server capacity)

Clustering and load balancing are great ways to improve the scalability and availability of services within your system. When applied to applications housed in an application server like WebSphere, these techniques are used to process more overall throughput and provide improved availability of the overall application. They provide these improvements in scalability and availability using straightforward approach. Do you need more throughput on your clustered solution? No problem: just add another instance of a server on your cluster, dial in the load balancer to include the new server, and you’re all set.

But having a scalable system implies that you are able to scale all parts of the application, including session persistence and other local cache data. If you don’t address the availability of the data, then you’ve just moved a bottleneck further back in your system. Server clustering and load balancing don’t address the data that’s consumed behind the scenes. If that data is a choke point for access, either because it’s too far away across a network or is being serviced by too many connections, then it will hinder any attempts to make the application process more load. If you try the simplistic approach of just duplicating the data to make “local” copies, you actually add more complexity. This is because any update to one copy needs to be made in all the other copies to keep all of the copies synchronized.

One way of solving the problem of data availability is attained by new feature of the WebSphere eXtreme Scale Object Grid (hereafter referred to as Object Grid) called multi-master. This feature allows your data to be mirrored to another remote location. This means that the data cached there can live as close to where it’s needed as possible, while still keeping one synchronized view of your data. In this way you can increase the number of copies of your data as you increase the number of servers, without having to worry about how to maintain a single, transactional view of that data. Plus, it’s done without having to rely on additional products or mechanisms – it’s all handled within a simple, elegant solution.

This article describes how to configure a multi-master grid and discusses some current restrictions for using multi-master version 7.1, particularly for HTTP session persistence. HTTP session persistence data is an important special case for data optimization, and for a certain class of applications, the out-of-the-box functionality poses limitations. This article describes how the author, working with colleagues on the IBM Web Identity team, devised a solution to circumvent this limitation.

The multi-master solution

With multi-master, you can store your data in Object Grid and mirror the data to ensure that the data is kept close to the servers that need it. Figure 1 illustrates a globally clustered application, with web and application servers in New York and Hong Kong. Updates made in the New York node copy of Object Grid are propagated by the multi-master feature to the nodes in Hong Kong. Any users that might somehow be routed by the load balancer to New York after logging into Hong Kong won’t notice any difference in their data presentation. The data updates made while on the New York servers will still be available to them in the Hong Kong servers as local data in Hong Kong.

Figure 1. Multi-master data mirroring
Multi-master data mirroring

Object Grid uses its internal, optimized transactional capabilities to do the mirroring. There is no need for you to configure or maintain the internals of how that’s done. To take advantage of the mirroring feature, just use the standard Object Grid Application Programming Interfaces. The product works to maintain the data copies for you. An update to one grid mirror is automatically propagated to its multi-master copy.

How to configure a multi-master grid

Configuring multi-master is easy to do -- all that’s needed is a simple properties file setup in the object grid configuration for WebSphere eXtreme Scale 7.1 or later. After installing your object grid installation (which is just a JSE installation) define the local and domain names, and link the end points. That’s it. The steps are as follows:

  1. Make sure your server knows the location of the file. So on the Object Grid startup script provided with the product, identify the path to the serverProps file, like this: -serverProps /usr/WebSphere/eXtremeScale/ObjectGrid/bin/
  2. Define the local domain name in the file: domainName = Local
  3. Define the foreign domains in the file: foreignDomains= Foreign
  4. Provide an optional list of endpoints for the foreign domains in the file: Foreign.endpoints=host1:2809,host2:2809

Current restrictions

The following restrictions apply when using a multi-master solution:

  • Grid name and map set name must match
  • Must be fixed partition and have same number of partitions
  • Same data types and templates
  • Some loader restrictions apply

Most of these restrictions are trivial compared to the value provided by multi-master. However, there is one current restriction that prevents multi-master from being used out-of-the-box for HTTP Session Persistence: You must use fixed partitions and have the same number of partitions in each multi-master copy.

Object Grid can in fact be used without modification to provide session persistence (more on that below). It uses a partitioning algorithm to do some internal housekeeping functions such that it has optimized HTTP Session persistence as per container, not per fixed partition. This may be sufficient for some applications, but in many use cases it’s worth the effort to do a small amount of programming to realize the full benefits of a multi-master grid. Those cases are:

  • You can access your data directly by a known key
  • You can predict how much data you want to hold in the grid, so can use fixed partitions

For HTTP Session data, you can meet those cases if you do two things.

  • Use SessionID plus the application context as the key
  • Understand your peak user load to predict a maximum grid size (in effect, sizing your grid for peak load

Handling HTTP Session data is a key requirement of many applications that also require data mirroring. The remainder of this paper discusses a way to circumvent these restrictions by using a standard J2EE extension of the web container.

Special Case: How to bypass the restriction on automatic session persistence

Note that you could easily code around this restriction by simply using grid APIs to store data in the multi-master grid. That would involve checking places where setAttribute and getAttribute are used and overloading them to include simple grid APIs. There’s nothing wrong with that approach, and it’s easy to do.

But suppose you want to use a more general approach. Assume you want to use your application's existing getAttribute and setAttribute APIs, with no modification (that is, you don’t want to code overrides to those methods), and you don’t want inspect your code to add Object APIs to the logic that would use it. In that case, you can extend your capabilities with some easy additions, without modifying your existing application. Here’s what you’ll need to code:

  • A new Servlet filter: MySessionOverrideFilter
  • A new Servlet Request Wrapper: MyHttpServletRequestWrapper
  • A new Session Attribute Listener: MySessionAttributeListener

These can easily be “bolted on” to provide the function you need without further changes to the underlying application. All of these capabilities are part of a standard Java Enterprise Edition offering – there’s nothing radical or unsupported about this approach. In fact, you’re probably using these facilities for other functions right now.

Description of the Program Flow of the new Modules

Figure 2 illustrates the flow of how the modules interact with each other.

Figure 2. Design overview showing components
Design overview showing components

The following describes the flow of how these modules interact with each other:

  1. A Browser makes a request of the application
  2. The filter class MySessionOverrideFilter is invoked by the WebSphere container prior to running the servlet or JSP. The filter routine overrides HTTPServletRequest with MyHTTPServletRequestWrapper, which overrides the getSession() method.
  3. The application servlet is invoked. Any changes to attributes via the setAttribute() method trigger the attribute methods in WISessionAttributeListener listener.logic
  4. If the session is new, then the session.getSession() method is overridden to use the Object Grid version of the data

As with all objects, the classes stored in the grid must be serializable, and the object signature must be understood by the Object Grid. One of the requirements of the serializable interface is that the class signature must be known and the object must implement a no-argument constructor. To enable this serialization for application classes, you may need to add your application JAR file to the –classpath argument for an object grid server.

Implementation detail for the workaround

Overriding the filter chain: MySessionOverrideFilter

A new Filter will be added to the application sessionManager.war file. Its purpose is to override the filter chain for normal HTTP Processing and instead invoke a new servlet wrapper. The new servlet wrapper will be used to override the HTTPSession.getSession() method.

The Filter can be added using a standard Rational® Application Developer wizard (Figure 3). (Highlight the WAR file, right click, and select New->Filter.)

Figure 3. Rational Wizard for creating a new servlet filter
Rational Wizard for creating a new servlet filter

Listing 1 shows the main method of the new Filter.

Listing 1. Code Snippet for Filter override
public void doFilter(ServletRequest request, 
ServletResponse response,
			FilterChain chain) throws, ServletException {
		// Instantiate MyHttpServletRequest to override "getSession"

The chain.doFilter method overrides the normal filter chain to invoke the customer wrapper WIHttpServletReqeuestWrapper.

To ensure that the filter is applied to your entire web application, add the filter mapping to your web.xml as shown in Listing 2.

Listing 2. Filter mapping XML

Using a wrapper to read attributes from the Object Grid: MyHttpServletRequestWrapper

As part of the Servlet 2.3 specification, the HTTPServletRequestWrapper interface was added to allow you to override certain web container methods. In this design, the HTTPServletRequestgetSession() method will be overridden to read attributes from the Object Grid.

Listing 3 contains code snippets highlighting the key parts of the solution.

Listing 3. Code snippet for override of getSession() method
public HttpSession getSession(boolean aCreateIfAbsent) {
  return getSession(false, httpRequest);
public HttpSession getSession(boolean aCreateIfAbsent, HttpServletRequest aRequest) {
  Session gridSess = null;
  String sessionId = null;
  ObjectMap map1 = null;
  // Get the new session
  HttpSession httpSess = super.getSession();
  // ..and get the old session from the JSESSIONID cookie
  String cookie = aRequest.getHeader("Cookie");
  int startIndex = cookie.indexOf("JSESSIONID=0000") + "JSESSIONID=0000".length();
  int endIndex = cookie.indexOf(":",startIndex);

  String cookieSession = aRequest.getHeader("Cookie").substring(startIndex, endIndex); 
  String contextRoot = httpSess.getServletContext().getContextPath();

  sessionId = cookieSession;
  Hashtable gridStoreObj = new Hashtable();
  ObjectGrid grid = OGServiceLocator

  try {
	  // Get a session with the grid
	  gridSess = grid.getSession();

	  // Get the ObjectMap
	  map1 = gridSess.getMap(OBJECT_GRID_MAP_NAME);
	  gridStoreObj = (Hashtable) map1.get(contextRoot
			  + sessionId);

  } catch (Exception e) {

	  // set attributes for each object in the grid
  Iterator iter = (Iterator) gridStoreObj.keySet().iterator();
  while (iter.hasNext()) {
		  String name = (String);

		  Object value = gridStoreObj.get(name);

		  httpSess.setAttribute(name, value);


The new MyHTTPServletRequestWrapper overrides HTTPServletRequest, which is the correct interface to override default behavior as described in the Servlet 2.3 specification.

The module gets the session from the superclass, as usual. But it also uses the JSESSIONID cookie to get the sessionId as it was stored in the grid in the last iteration of this module, before the cluster failed over. Then, all attributes from the Object Grid version of the session are applied to the memory version of the session object (httpSess variable) using the standard setAttribute method.

An important point here: if setAttribute is also overridden to enable persistence to the grid, there is a danger of continued recursive calls creating a circular logic bug. To avoid that, the Session Attribute listener is written to avoid such a call. See the MySessionAttributeListener code in Listing 3 on how that’s done.

An alternative solution is to ensure that this is the only occurrence of the container setAttribute method,and that the rest of application uses a new method to set attributes.

Adding a session attribute listener: MySessionAttributeListener

A session attribute listener can be added through a wizard in Rational to listen for changes to attributes. The class can be added using a standard Rational Wizard (Figure 4).

Figure 4. Rational Wizard for creating session attribute listener
Rational Wizard for creating session attribute listener

Listing 4 highlights the key logic for MySessionAttributeListener.

Listing 4. Code snippet for Session Attribute Listener
public void attributeAdded(HttpSessionBindingEvent se) {

  String contextRoot = se.getSession().getServletContext()

  if (thisClassCalledMe("MyHttpServletRequestWrapper")) {
	  // do nothing
  } else {

	 try {
		// Get a session with the grid
		gridSess = grid.getSession();

		// Get the ObjectMap
		map1 = gridSess.getMap(OBJECT_GRID_MAP_NAME);
		.println("Debug MySessionAttributeListener attribute added: "
					+ map1.getName());


		storeObj = (Hashtable) map1.get(contextRoot + sessionId);
	} catch (Exception e) {
	// if nothing in grid, stick regular session there..
	// if (storedSession == null) {
if (storeObj == null) {
	try {
		storeObj = new Hashtable();

		Enumeration attrlist = (Enumeration) currentSession.getAttributeNames();
        ("Debug MySessionAttributeListener attribute added: attrlist> " + attrlist);
		while (attrlist.hasMoreElements()) {
			String name = (String) attrlist.nextElement();
			Object value = currentSession.getAttribute(name);
            ("Debug MySessionAttributeListener attribute added:  \
                Name " + name  + "  value  " + value);
			storeObj.put(name, value);

		storeObj.put(se.getName(), se.getValue());
		map1.insert(contextRoot + sessionId, storeObj);

	} catch (Exception e) {
		try {
		} catch (Exception e2) {

	// if something in grid, use it...
} else {
	storeObj.put(se.getName(), se.getValue());
	try {

		// add the new attribute
		map1.update(contextRoot + sessionId, storeObj); // currentSession);

	} catch (Exception e) {

A session Attribute listener provides three methods to detect changes to HTTP Session name/value pairs. The methods are:

  • attributeRemoved(HTTPSessionBindingEvent)
  • attributeAdded(HTTPSessionBindingEvent)
  • AttributeReplaced(HTTPSessionBindingEvent)

The snippet focuses on the attributeAdded(HTTPSessionBindingEvent) method. This method is invoked any time the API session.setAttribute(name, value) is invoked.

In the class, the thisClassCalledMe method is used to evaluate the call stack. If the special WIHTTPServletRequestWrapper class is invoked, then logic to persist attributes in the Grid is ignored, since the attributes would already be there in that case. The logic then updates the data in the object grid using normal Object Grid get/put APIs.

Listing 5 shows the full thisClassCalledMe method. See Resources for details on this method.

Listing 5. Code Snippet for evaluating call stack
public static boolean thisClassCalledMe(String who) {
		StackTraceElement[] stackTraceElements = 
		for (int i = 5; i < stackTraceElements.length; 
i++) {
			StackTraceElement ste = 
			String classname = ste.getClassName();
			if (classname.equals(who)) {
				return true;
			String methodName = ste.getMethodName();
			int lineNumber = ste.getLineNumber();
		return false;

It uses thread context and getStackTrace methods to determine if the caller was a specific method, so that conditional overrides can be invoked on recursive calls.

Comparing the special case to other HTTP session persistence approaches

Of course, the special-case needs of session persistence have been around for a long time, and other approaches do offer some attractive features. Here’s a quick recap of some of them and how they differ from this approach:

  • Database Session Persistence. In database session persistence, a central database is used to hold session data. Database session persistence is a common and solid approach. WebSphere Application Server offers it as an easy option to use it with minimal configuration. It provides a single view of the data, and database high availability techniques remove the database as a single point of failure. IBM DB2 database also offers Queue Replication and High Availablity Database Replication to provide data replication features.
  • Memory to Memory (M2M) Session Persistence. WebSphere Application Server also provides Memory to Memory (M2M) session persistence. In this approach, Application Server pushes copies of session updates to another WebSphere Application Server instance. M2M solutions provide built-in fault tolerance due to their clustered design. They can also keep the data close to the server that uses it. Here, care should be taken to ensure that the duplication paths are set up to avoid a single point of failure on a single server and too much cross-network traffic.
  • Session Persistence with WebSphere eXtreme Scale or XC10. Of course, WebSphere eXtreme Scale provides its own session persistence capabilities. It can be offered on both general-purpose computers or appliance technology like the XC10 device. It provides built-in fault tolerance and is easy to install. (It’s offered as part of a configuration on the latest releases of WebSphere.) But keep in mind that it’s difficult to control data placement of the internals of the grid. Its internal algorithms control data placement, so parts of the data may still be remote from the server that uses it. And of course, right now, it’s not using multi-master.


So you can see that common techniques to “scale up” your web application can lead to problems with data availability. There are other ways to solve the problem, but things can get complicated quickly A new way to solve the problem of data availability is the multi- master feature of WebSphere eXtreme Scale version 7.1. Multi-master replication provides a way to make mirror-copies of your data without having to worry about the mechanisms that do the mirroring. And if you need to extend its capabilities to include HTTP Session Data, you have the steps to do it. The reliable, transactional design of eXtreme Scale makes the solution viable, and the simple steps involved make it an easy choice for data availability needs.


In addition to the Web Identity Team, the author wants to thank the following members of the Object Grid team: Billy Newport and Doug Berg.


Sample code for Multi-master solutionMulti_Master_JEE_PI.zip22KB



developerWorks: Sign in

Required fields are indicated with an asterisk (*).

Need an IBM ID?
Forgot your IBM ID?

Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.


All information submitted is secure.

Dig deeper into WebSphere on developerWorks

ArticleTitle=Improve response time and data availability with WebSphere eXtreme Scale multi-master capability