Super cluster to the rescue, Part 2: Maximum scalability using WebSphere DMZ Secure Proxy Server, ODR, and WebSphere eXtreme Scale

Because application scalability is an important quality of service in most enterprise software topologies, enterprise quality Java™ EE applications are commonly deployed and executed in IBM® WebSphere® Application Server Network Deployment clusters. Even though the practical size of a cluster can be limited. This limitation can be overcome with a useful technique for achieving extreme application scalability that we refer to as the "super cluster". Part 1 of this two-part article defined a super cluster and examined how it can be used with the HTTP plug-in and IBM WebSphere Proxy Server. In this conclusion, the discussion is expanded it to include the IBM WebSphere DMZ Secure Proxy Server V7, the on demand router in IBM WebSphere Virtual Enterprise, and IBM WebSphere eXtreme Scale. This content is part of the IBM WebSphere Developer Technical Journal.

Share:

Kevin Kepros, Advisory Software Engineer, IBM

Kevin Kepros is an advisory software engineer at the IBM Software Development Lab in Rochester, Minnesota. Kevin was a lead developer on the WebSphere High Availability Manager component and a member of the WebSphere Clustering development team. Recently Kevin has taken a new position working with the IBM SPSS team on predictive analytics solutions.



Dr. Debasish Banerjee, WebSphere Consultant, IBM  

Dr. Debasish Banerjee is presently a WebSphere consultant in IBM Software Services. He started his WebSphere career as the WebSphere internationalization architect. Extreme transaction processing, distributed cache, elastic computing, and cloud computing are his current areas of interest. Debasish received his Ph. D. in the field of combinator-based Functional Programming languages.



22 July 2009

Also available in Chinese Japanese

Introduction

From the discussion in Part 1, we concluded that most application scalability issues can be addressed by making use of an IBM WebSphere Application Server cluster. While it is not very common for an application’s scalability requirements to exceed the ability that can be handled by a single WebSphere Application Server cluster, it can happen. For these scenarios, one technique that can be used to overcome the implicit cluster size limit is to define a hierarchical cluster or “super cluster,” as shown in Figure 1.

Figure 1. Hierarchical super cluster
Figure 1. Hierarchical super cluster

The quintessence of super clustering is to:

  • Deploy an application to multiple clusters (a “cluster-of-clusters”).
  • Use an appropriate router to distribute the client requests so that, from the viewpoint of the client, the two-level hierarchical cluster appears like a flattened single level traditional WebSphere Application Server cluster.

The type of router that is used to distribute client requests directly influences the use of a super cluster and its associated limitations. Part 1 discussed how to use the HTTP plug-in or WebSphere Proxy Server -- or both in combination -- in a super cluster topology. Part 2 continues the discussion and expands it to include the IBM WebSphere DMZ Secure Proxy Server V7, IBM WebSphere Virtual Enterprise on demand router (ODR), and IBM WebSphere eXtreme Scale.


Using WebSphere DMZ Secure Proxy Server

For security reasons, you might not want to place proxy servers out in the demilitarized zone (DMZ). Part 1 explained that one way you could work around this issue is to make use of both the HTTP Plug-in (in the DMZ) and WebSphere Proxy Server (inside the protocol firewall) to perform the routing, as shown in Figure 2.

Figure 2. Web server in DMZ routing to proxy servers
Figure 2. Web server in DMZ routing to proxy servers

Another option is to make use of the WebSphere DMZ Secure Proxy Server V7, as shown in Figure 3.

Figure 3. WebSphere DMZ Secure Proxy Server
Figure 3. WebSphere DMZ Secure Proxy Server

The WebSphere DMZ Secure Proxy Server V7 is included as part of WebSphere Application Server Network Deployment. WebSphere DMZ Secure Proxy Server is installed separately, and provides a way for you to install and run a proxy server in the DMZ.

After you install WebSphere DMZ Secure Proxy Server to a machine located in the DMZ, create a secure proxy profile. The resulting topology will be a new cell that is different from the back-end cell that contains the application server clusters. Because the secure proxy server does not have a Web container, it cannot host an administrative console. Therefore, when you need to configure or perform administrative tasks on WebSphere DMZ Secure Proxy Server, you can do so using one of these options:

  • Scripting (no GUI): You can manage the server via scripting on the command line.
  • Admin agent: Create an administrative agent profile that you will run in the DMZ and register the secure proxy with an administrative agent.
  • Configuration-only secure proxy server profile: Create a configuration-only secure proxy server profile in the back-end WebSphere Application Server Network Deployment cell to customize the configuration.

With the latter approach, you would export the configuration from the configuration-only version of the secure proxy profile and then import it into the secure proxy profile in your DMZ.

Further details on administration and management of the secure proxy server are beyond the scope of this article. For more information, refer to the WebSphere Application Server Information Center.

Routing

A secure proxy server can route requests based on either static or dynamic information. The mechanism used to obtain routing information depends on which security mode the secure proxy server is configured to use.

  • Low security mode

    This type of proxy configuration was discussed in Part 1. In this mode, there is no separate installation or separate cell, and so on. Instead, you are simply using a regular proxy server (defined in the same cell as the application servers) and routing based on dynamic information. This mode requires full communication between the proxy servers and application servers, and is typically not suitable for use in a DMZ.

  • Medium security mode

    When configured in medium security mode, the dynamic routing information is obtained and used by the secure proxy server. However, since the secure proxy server and the back-end application servers are really different cells, the dynamic routing information is communicated to the secure proxy server via a special core group bridge HTTP tunnel connection. Essentially, you create a cross-cell core group bridge using HTTP ports. Thus, routing information (along with application traffic) is exchanged between the back-end application server cell and the secure proxy server cell.

    In both low and medium security modes, the routing information is obtained by the proxy server dynamically, so the steps required to make use of a super cluster are essentially the same as those described in Part 1 for the WebSphere Proxy Server.

  • High security mode

    When configured in high security mode, the secure proxy server will route application requests based on static route data. This is the default mode for WebSphere DMZ Secure Proxy Server, and the mode that the remainder of this article will explore in detail. Conceptually, this is similar to how the HTTP plug-in performs routing: you must generate a flat file that contains routing information, edit the file to create a super cluster, and copy it out to the secure proxy server located in the DMZ. By default, the routing file that you generate will have the name targetTree.xml.

WebSphere DMZ Secure Proxy Server and the super cluster

The best way to understand how a secure proxy can be used to route requests across a super cluster is to look at a simple example. Let’s walk through the steps required to configure a secure proxy server to route across a simple two cluster topology, as shown in Figure 4.

Figure 4. Super cluster with DMZ secure proxy server
Figure 4. Super cluster with DMZ secure proxy server

The sample topology described here is intentionally limited, but the techniques explained can be easily be extended for use in much larger topologies.

The basic steps involved in configuring a super cluster topology with the WebSphere DMZ Secure Proxy Server are:

  1. Create a WebSphere Application Server cluster.
  2. Install the application to the cluster.
  3. Test and verify the application.
  4. Create additional WebSphere Application Server clusters.
  5. Map application modules to each WebSphere Application Server cluster.
  6. Generate the secure proxy static routing file (targetTree.xml).
  7. Edit the targetTree.xml file to route requests across multiple clusters.
  8. Copy the modified targetTree.xml file to the secure proxy server staticRoutes directory.

Many of these steps were detailed in Part 1 and so won’t be repeated here, but steps 6 and 7 are specific to a DMZ secure proxy server and its ability to route across a super cluster. Let’s take a closer look at these tasks.

Generate the targetTree.xml file

With a DMZ secure proxy server in high security mode (the default), you are required to supply a file that contains static routing information to the secure proxy server. This file must be generated from the back-end cell that is hosting the applications. To do this, use the exportTargetTree command on the TargetTreeMbean that is running in the Dmgr process. For example, you can generate the static route file via scripting by using the Jython commands shown in Listing 1.

Listing 1. Script commands to generate targetTree.xml file
// Query for the TargetTreeMbean MBean
mbean=AdminControl.queryNames('*:*,type=TargetTreeMbean,process=dmgr')

// Invoke exportTargetTree method on the TargetTree MBean
AdminControl.invoke(mbean, 'exportTargetTree',
C:/WebSphere/AppServer/targetTree.xml')

For a non-super cluster topology, the resulting XML file can simply be transferred from the deployment manager over to the secure proxy server's <profile root>/staticRoutes directory. The secure proxy server will then use this static routing information to target application requests. In a super cluster topology, you must edit this static routing information to make a single flattened cluster that contains all of the members of the super cluster.

Edit the targetTree.xml file

The principle behind the editing of the targetTree.xml and the plugin-cfg.xml routing files to create a super cluster is the same: you want to present a flattened view of the two-level hierarchical cluster to the router.

Assuming that you have your application modules mapped to multiple clusters and generated the static route file, your next step is to edit the file. Figure 5 illustrates relevant code sections of a targetTree.xml generated from this simple example topology: two clusters are configured (Cluster1 and Cluster2) with two members defined for each.

Figure 5. targetTree.xml -- cluster definition
Figure 5. targetTree.xml -- cluster definition

Each server in the targetTree.xml file (including non-clustered application servers) is defined within a cluster element. Each cluster element contains a server section with one (or more) link element(s) that define each member of the cluster.

At startup, the on demand configuration code inside the secure proxy reads the static route file and builds the in-memory routing data. When an application request is received, the secure proxy will use this in-memory routing information to target the request. The router does not check whether the target server is indeed part a configured WebSphere cluster. When routing requests, WebSphere DMZ Secure Proxy Server will maintain the appropriate session affinity.

The main purpose of editing the targetTree.xml file is to make the secure proxy server believe that all the members of the hierarchical (super) cluster belong to a traditional (flattened) cluster. Therefore, editing this file means you must:

  1. Determine which cluster will be used as the default for your application requests.
  2. Move the members (link elements from the server section) of the other clusters to the cluster in the targetTree.xml file that was handling the client requests by default.

The easiest way to determine which cluster is the default target is to simply test your application with an unmodified targetTree.xml file. Even though your application modules are mapped to multiple clusters, only one of the clusters will actually be used to service the application requests by default. Typically, this will be the first cluster defined in targetTree.xml that is capable of servicing the given request. Once you know which cluster is the default target, move the members from the server section of the other clusters to the default cluster. Cut and paste the link elements from the server section of all the other cluster elements into the corresponding server section in the default target cluster’s cluster element.

Figure 6 shows part of the modified targetTree.xml file that shows the merger of the Cluster1 and Cluster2 members to create the super cluster.

Figure 6. targetTree.xml super cluster: Cluster1
Figure 6. targetTree.xml super cluster: Cluster1

In the example above, the default target cluster was Cluster1. Hence the link elements from the <!-- server section --> were moved from Cluster2 into Cluster1. With the modified targetTree.xml file placed in the Secure Proxy <profile home>/staticRoutes directory, WebSphere DMZ Secure Proxy Server will distribute requests across all four members of the super cluster.

Be aware that you must actually move the link elements; if you copy them and they still exist in two places, your routing will not function as expected. Unlike when you edit the HTTP plug-in routing file and can leave the unused cluster definition intact, the secure proxy is more restrictive when it comes to the format and content of the static routing file. If servers in targetTree.xml are moved incorrectly, client (browser) requests can fail with a 503 – Service Unavailable error, since the router might be unable to find an appropriate member of the target cluster.

Additional considerations

Here are some additional items to keep in mind when using a secure proxy to route across a super cluster.

  • The static route XML file can have any name. For example: targetTree.xml or <any name>.xml
  • Place the static route XML file in the secure proxy server’s <profile home>/staticRoutes directory.
  • Multiple static routing files (for example, xxx.xml) can be placed in the secure proxy server’s <profile home>/staticRoutes directory. They will automatically be merged to create a single in-memory route table. You can avoid this automatic merging by creating a subdirectory (for example, <profile home>/staticRoutes/saved ) for keeping prior versions of the static routing files.
  • There is no auto-reload of changes that are made to the static routing files (for example, xxx.xml). In order to pick up these changes, the secure proxy servers must be stopped and started.
  • When enabled, static routing always overrides dynamic routing. The secure proxy configuration is persisted at the server scope in a file called proxy-settings.xml. This configuration file contains the attribute enableStaticRouting="true” which forces static routing.

Limitations

As mentioned in Part 1, there are restrictions and limitations with any super cluster topology. When using the WebSphere DMZ Secure Proxy Server as the router, be aware of these limitations:

  • The secure proxy can only support routing HTTP protocol for a super cluster.
  • The scheme provides no session failover and no session replication support. You can use WebSphere eXtreme Scale or session persistence for session failover.
  • The technique requires manual administration of the static route files. Therefore, you must manually:
    • Generate the static routing file.
    • Edit the static routing file.
    • Propagate the static routing file.
    • Keep the static routing file in sync with topology changes.
  • The technique might require core group bridge service (CGBS) configuration:
    • CGBS is required in the back-end cell if the clusters (application deployment targets) are located in different core groups.
    • CGBS is required if you run the secure proxy server in medium security mode in order to dynamically obtain routing information.

The next section looks at the same example scenario, but uses the WebSphere Virtual Enterprise on demand router (proxy server) to distribute requests across a super cluster instead of WebSphere DMZ Secure Proxy Server.


Using WebSphere Virtual Enterprise

As an add-on to WebSphere Application Server Network Deployment, IBM WebSphere Virtual Enterprise can be used to virtualize the underlying WebSphere Application Server infrastructure. Dynamic operations based on a user-specified Service Level Agreement (SLA), health management, operation monitoring, and application editions are among the important and useful features of WebSphere Virtual Enterprise (formerly known as WebSphere Extended Deployment Operations Optimization).

On demand router

WebSphere Virtual Enterprise contains an on demand router (ODR) component, which plays a pivotal role in application request traffic shaping, interrupt-free application rollout, and other important functions. Thus, in a WebSphere Virtual Enterprise topology, HTTP traffic will be routed through ODRs.

From a configuration and topology point of view, an ODR is similar to a classic WebSphere Application Server Network Deployment proxy server. For scalability and high availability, ODRs should be clustered. A typical ODR cluster would contain at least two members, each of which should reside in a separate machine. In the present WebSphere Virtual Enterprise implementation, an ODR cluster is simply a collection of ODRs. Therefore, an ODR cluster is not explicitly created or administered from the admin console or through scripting: you will create and administer individual ODRs.

ODR and the super cluster

Like the classic proxy server, an ODR can also be used to route traffic to super clusters. The steps to do this, then, are essentially the same as those described for a proxy server in Part 1:

  1. Create a WebSphere Application Server cluster.
  2. Install the application to the cluster.
  3. Test and verify the application.
  4. Create additional WebSphere Application Server clusters.
  5. Map application modules to each WebSphere Application Server cluster.
  6. Bridge all of the core groups containing the clusters and the core group containing the ODRs.

As with the classic proxy server, there is no need to manually edit any static routing file. The ODR will obtain routing information dynamically. For this dynamic routing to work properly, the core groups containing the ODRs and the core groups containing the target cluster members must be bridged together using CGBS.


ODR super clustering topology

For security reasons, an ODR is not typically placed in the DMZ. Instead, Web servers (with plug-in) residing in the DMZ would route HTTP requests to the members of the ODR cluster located inside the firewall, as shown in Figure 7. Notice the similarity of this topology with the topology described in Part 1 for using both the HTTP plug-in and proxy servers with a super cluster.

Figure 7. ODR proxy
Figure 7. ODR proxy

The routing capabilities of ODRs are similar to those of classic proxy servers. The ODRs will route the incoming HTTP requests in an “application scoped” fashion to any target server capable of servicing the request. In a WebSphere Application Server cell, ODRs will distribute HTTP requests while maintaining session affinity to the WebSphere cluster members, as well as to the non-clustered WebSphere Application Server instances where the relevant application is deployed. Internally, the multi-cluster routing component treats a non-clustered application server as a singleton WebSphere Application Server cluster with the application server as the lone member.

As shown in Figure 7, the Web server and plug-in router combination will forward HTTP requests to all the ODR cluster members in a strict round-robin fashion. There is no need to maintain session affinity at the plug-in router level, as session affinities will be taken care of by the ODRs when the requests are forwarded on to the application servers.

Just like the HTTP plug-in and proxy server topology documented in Part 1, a special static routing file is needed to enable the HTTP plug-in component to forward requests to the ODR cluster. The ODR provides the mechanism for automatically generating the required plugin-cfg.xml file, which will reside in the <profile-home>/etc directory of the ODR. There are options to specify the scope of the generated plug-in data. In WebSphere Virtual Enterprise, this scope setting indirectly determines which ODRs the plug-in router considers belonging to the ODR cluster. Scripts can be configured to automatically copy the ODR generated plug-in file to the appropriate location on the Web server machine. When any relevant configuration changes occur (such as application addition, deletion, modification, change in the number of ODRs, change in the plug-in generation scope, and so on), a new plug-in file will be automatically generated. For details of the plug-in routing file generation from ODRs consult Figures 18 and 19 from Part 1 of this article, and the IBM Redbook Optimizing Operations with WebSphere Extended Deployment V6.1.

Additional considerations

When using a WebSphere Virtual Enterprise ODR to route across a super cluster, consider these items:

  • Compared to WebSphere Application Server Network Deployment, WebSphere Virtual Enterprise might seem to be a relatively heavy user of the high availability manager bulletin board service. For this reason, you should limit the core group size to 50 (as a conservative measure) when using WebSphere Virtual Enterprise.
  • With the classic proxy server, you can configure the plug-in routing file generation and distribution settings at the cluster level. In the case of the ODR, these settings can only be specified at the individual ODR level.
  • You can adjust how the Web server will distribute requests across the ODR cluster members by changing the values of the LoadBalanceWeight attributes used by the default round-robin load distribution algorithm, located in the ODR generated plugin-cfg.xml file. For example, this might be necessary if the ODR cluster members are not hosted on machines of similar capacity.
  • Presently, the generated plug-in routing file contains only the ODRs that are up and running at the time of generation.
  • WebSphere Virtual Enterprise provides the option of creating dynamic clusters. The multi-cluster routing ability of the ODR does not differentiate between traditional WebSphere Application Server static clusters and the WebSphere Virtual Enterprise-specific dynamic clusters. Hence, some or all the clusters of Figure 7 can be of the dynamic variety.

Limitations

These are limitations when using the WebSphere Virtual Enterprise ODR as the router:

  • The ODR can only support routing HTTP protocol for a super cluster.
  • The scheme provides no session failover and no session replication support. You can use WebSphere eXtreme Scale or session persistence for session failover.
  • The technique requires CGBS configuration if the ODRs and clusters (application deployment targets) are located in different core groups.

The next section examines how the concept of a super cluster relates to IBM WebSphere eXtreme Scale.


Using WebSphere eXtreme Scale

WebSphere eXtreme Scale provides an extremely powerful coherent distributed cache. WebSphere eXtreme Scale caches data in the virtualized space formed from the heap space of the hosting Java Virtual Machines (JVMs). WebSphere eXtreme Scale provides linear scalability with predictable throughout. It can be used to create an elastic grid that could potentially store massive amounts of partitioned data. Using replicas, WebSphere eXtreme Scale provides support for built-in high availability. It provides a number of user configurable plug-ins for altering the behavior of the grid.

Deployment environment

WebSphere eXtreme Scale does not require the use of any WebSphere Application Server product for its operation. Also, WebSphere eXtreme Scale does not need the presence of a Java EE JVM. It can be deployed in a non-WebSphere, Java Standard Edition (Java SE) JVM. WebSphere eXtreme Scale can be deployed in either 32 bit (the more popular environment) or 64 bit JVMs.

WebSphere eXtreme Scale and WebSphere Application Server

While WebSphere eXtreme Scale can be used in a standard Java SE environment, using WebSphere eXtreme Scale with WebSphere Application Server can provide enhanced administration capabilities, with improvements in testing, debugging, problem determination, and so on.

If you are already familiar with and running WebSphere Application Server Network Deployment, you might find deploying WebSphere eXtreme Scale in this environment appealing for several reasons. WebSphere Application Server Network Deployment provides a central point of administration, deployment, and configuration for the WebSphere eXtreme Scale grid and associated applications. The Performance Monitoring Infrastructure (PMI) service can be used to visualize and monitor various WebSphere eXtreme Scale related statistics from a central location.

Although WebSphere eXtreme Scale can be deployed in a single JVM, such a deployment would not provide any high availability. In addition, a single JVM is not sufficient for caching large amounts of data (for example, a 32 bit JVM can only guarantee a maximum of 2 GB of usable heap space). Thus, for caching a large amount of highly available partitioned data, WebSphere eXtreme Scale needs to be deployed in multiple JVMs. In simple Java SE environments, WebSphere eXtreme Scale deployment is performed using scripts provided with the product. The same scripts can be used to deploy WebSphere eXtreme Scale in non-clustered WebSphere Application Server instances. However, using WebSphere Application Server Network Deployment clusters can provide enhanced management and easier WebSphere eXtreme Scale deployment.

It should be noted that the WebSphere Application Server Network Deployment runtime does consume some heap space for execution. Therefore, deploying WebSphere eXtreme Scale in a Network Deployment environment will leave less heap space available for caching actual objects in the grid; this is the trade-off for the enhanced administration capabilities.

Catalog service

The design of WebSphere eXtreme Scale dictates the presence of the catalog service before the grid is started. WebSphere eXtreme Scale catalog servers are used to provide placement, location, and administration services. In a non-WebSphere Application Server environment, you designate special JVMs to be catalog servers, and you start those JVMs before starting the grid.

In a WebSphere Application Server Network Deployment environment, the catalog service starts on the deployment manager process by default. However, you can modify this default behavior by using a cell-scoped custom property to designate one or more application servers to host the catalog service.

Deployment in WebSphere Application Server clusters

A WebSphere eXtreme Scale grid is hosted by WebSphere eXtreme Scale containers. In a WebSphere Application Server environment, the presence of two configuration files -- objectGrid.xml and objectGridDeployment.xml in the META-INF directory (entity.xml might also be present in the META-INF directory, depending on the programming model used to access the cache) -- of any deployed application triggers WebSphere eXtreme Scale to start the container service and, hence, the grid.

Local or remote grid

The WebSphere eXtreme Scale grid and the application using it can be collocated in the same JVM, or the application could be running in one JVM and access the grid container running in a separate JVM (client-server remote mode). As you would expect, there are various trade-offs associated with each topology.

Figure 8 shows an example topology where the WebSphere eXtreme Scale grid and the application using it are located in the same JVM. There is a performance advantage here when accessing the grid: No network hop is required from the application to access the cached data.

Figure 8. Local WebSphere eXtreme Scale grid
Figure 8. Local WebSphere eXtreme Scale grid

In this local grid configuration, the WebSphere eXtreme Scale XML grid configuration files mentioned above can be packaged in the META-INF directory of the using application. The application can then be deployed and started from the admin console (or through wsadmin scripts). The WebSphere eXtreme Scale container -- and therefore the grid -- will be started automatically along with the application (it is assumed of course that the WebSphere eXtreme Scale catalog service is already up and running).

In a client-server remote mode, the WebSphere eXtreme Scale grid is hosted in a WebSphere cluster that is separate from the cluster that is hosting the application, as shown in Figure 9.

Figure 9. Remote WebSphere eXtreme Scale grid
Figure 9. Remote WebSphere eXtreme Scale grid

This remote grid configuration does require a network hop to access the grid data. However, the topology has the advantage of providing isolation, along with the ability to scale the grid up or down independent of the application’s scalability (for example, the number of JVMs in the WebSphere eXtreme Scale grid cluster can be different from the number of JVMs in the application cluster). In addition, this topology also enables multiple different applications (each in their own cluster) to share a single data grid. Another potential advantage is that this topology enables the application cluster and the WebSphere eXtreme Scale grid cluster to be hosted on different versions of WebSphere Application Server.

If the grid is only used to cache data that can be described by a Java primitive type, a simple application can be used to manage WebSphere eXtreme Scale containers. The simple application can be deployed in the grid cluster, and it can contain the two configuration files in its META-INF directory. Thus, starting the simple application will automatically start the grid containers.

If non-primitive data objects are cached in the grid, then the simple application must also contain the class definitions of the objects that will be stored in the grid, along with the configuration files in the META-INF directory.

The purpose of the simple application deployed in the grid cluster is not for performing any computational work, but to provide the WebSphere eXtreme Scale runtime with:

  • The grid configuration parameters.
  • The class definitions needed for the serialization and deserialization of the objects stored in the grid.

WebSphere eXtreme Scale grid in a super cluster

As described in Part 1, there are scalability restrictions associated with a core group that directly affect that maximum size of a WebSphere Application Server cluster. For WebSphere Application Server V6.1 and higher, the general recommendation was to limit your core group (and hence a cluster size) to a maximum of around 100 members (your actual limit will vary depending on your available resources, number of applications, network bandwidth, and so on).

Recall that the maximum size of your WebSphere eXtreme Scale grid would be the combined free heap space available in all of the members of the grid cluster. Therefore, what WebSphere eXtreme Scale deployment strategy should you follow if the desired grid/cache size requires a WebSphere Application Server cluster that exceeds the maximum recommended size? Again, the answer is to host the grid in a WebSphere Application Server super cluster. This will allow you to use the heap space of all the JVMs in the super cluster to form an extremely large grid space.

For the sake of clarification, WebSphere eXtreme Scale is not dependant upon core groups or clusters; cluster size considerations come into play WebSphere eXtreme Scale is deployed on WebSphere Application Server Network Deployment.

Figure 10 shows a client-server style WebSphere eXtreme Scale grid environment where the application is deployed in a traditional WebSphere cluster and it accesses the WebSphere eXtreme Scale grid deployed in a super cluster.

Figure 10. WebSphere eXtreme Scale deployed in a super cluster
Figure 10. WebSphere eXtreme Scale deployed in a super cluster

To create a WebSphere eXtreme Scale super cluster topology:

  1. From the expected size of your WebSphere eXtreme Scale grid, determine the number of WebSphere Application Server clusters n (where n >= 1) that will constitute the super cluster.
  2. Create n (where n >= 1) number of core groups
  3. Now you have (n >= 1) core groups including the DefaultCoregroup core group. For all of these core groups:
    • Follow the standard core group formation rules.
    • Best practices recommend that each core group be configured with two preferred coordinator servers, ideally located on non-clustered servers and on different physical machines.
    • Configure each to run the latest available versions of core group protocol.
  4. Create a horizontal WebSphere Application Server cluster (for example, CatalogCluster) for the WebSphere eXtreme Scale catalog service consisting of three members in the DefaultCoregroup:
    • Using a CatalogCluster ensures that all the catalog servers belong to the same core group, which is a requirement for WebSphere eXtreme Scale deployment.
    • Having each catalog server residing in different physical machines will provide high availability of the catalog service.
    • A reasonable number of catalog servers (three in the present example) will contribute to the high availability of catalog service.
    • The CatalogCluster WebSphere Application Server cluster will not be part of the WebSphere Application Server super cluster hosting the WebSphere eXtreme Scale grid.
  5. Designate the members of the CatalogCluster as the WebSphere eXtreme Scale catalog servers by properly defining the cell-scoped catalog.services.cluster custom property, which lists the host and port of each member in the CatalogCluster.
  6. Create the desired WebSphere Application Server clusters in all the (n >= 1) core groups, each with the appropriate number of cluster members.

With the WebSphere Application Server super cluster topology configured, you would likely want to create a simple application for configuring and starting the grid containers. To configure the simple WebSphere eXtreme Scale admin application:

  1. Create the dummy application (for example, GridApp) with the required class definitions and the required WebSphere eXtreme Scale configuration files in its META-INF directory.
  2. Deploy GridApp to any one of the WebSphere Application Server clusters intended to be part of the super cluster.
  3. Map the GridApp modules to all of the WebSphere Application Server clusters that make up the super cluster. Use the technique described in the Part 1 for mapping the modules to multiple clusters. (The application should not be mapped to CatalogCluster, since the members of CatalogCluster cluster will not provide any cache space by hosting the WebSphere eXtreme Scale container.)
  4. Make sure that the CatalogCluster cluster is running.
  5. Start the dummy application in each cluster that will automatically start the associated grid containers.

Using WebSphere Application Server clusters and this type of simple application eases WebSphere eXtreme Scale administration. By starting or stopping the dummy application in all of the relevant clusters, the grid containers can be easily started or stopped. Since any change in the grid configuration file requires a grid restart, the technique described here provides a simple and straightforward way to manage a large grid containing hundreds of JVMs. Thus, using the super clustering technique with WebSphere eXtreme Scale provides simplicity in grid management and monitoring from a central point (for example, the WebSphere Application Server deployment manager).

Additional considerations

Other items to consider when using WebSphere eXtreme Scale with a super cluster topology include:

  • A remote enterprise quality WebSphere eXtreme Scale grid should not be collocated with any other user application.
  • For such a pure WebSphere eXtreme Scale grid running in a super cluster, there is no need to bridge any of the core groups containing the catalog server cluster or the grid hosting clusters.
  • If you use non-clustered WebSphere Application Server instances in a Network Deployment environment to host WebSphere eXtreme Scale containers, the core group size restriction does not disappear. Every process (clustered or not) is a member of a core group, and you can run into issues if a core group consisting of non-clustered servers grows too large.

Application and WebSphere eXtreme Scale in a super cluster

It is possible to have both the application and the WebSphere eXtreme Scale grid deployed in super clusters, as shown in Figure 11 (in this example, a client-server configuration). This scenario combines the use of a super cluster for application scalability with the use of a super cluster for grid scalability, where both super clusters can be scaled independently.

Figure 11. Application and WebSphere eXtreme Scale grid in super clusters
Figure 11. Application and WebSphere eXtreme Scale grid in super clusters

HTTP session failover

As mentioned when describing the super cluster technique as it relates to application scalability and routers, there is a limitation in that HTTP session replication is not supported. This is because the replication services in WebSphere Application Server are currently restricted to a single core group. However, as also mentioned earlier, a WebSphere eXtreme Scale grid could be used to provide HTTP session replication across the super cluster, thereby enabling HTTP session failover.

WebSphere eXtreme Scale includes an HTTP servlet filter for providing HTTP session management. WebSphere eXtreme Scale provides a utility for properly injecting the filter into an application, enabling HTTP session data to be stored and replicated across a super cluster. Using a WebSphere eXtreme Scale grid for HTTP session data enables the storage and replication of HTTP sessions across core groups, a super cluster, or even across WebSphere Application Server cells.

For HTTP session management, it is common that the grid container be collocated in the same JVM as the application. Figure 12 shows an example of such a topology.

Figure 12. HTTP session management for a super cluster using WebSphere eXtreme Scale
Figure 12. HTTP session management for a super cluster using WebSphere eXtreme Scale

Of course, a remote WebSphere eXtreme Scale grid can also be used for HTTP session management, if desired. Thus, you can have the application and the grid in a standard cluster (Figure 8), the application and the WebSphere eXtreme Scale grid in a super cluster (Figure 12), or you could have a remote grid topology where either the application hosting cluster, the WebSphere eXtreme Scale grid hosting cluster, or both, are configured as a super cluster. The use of a remote grid for HTTP session management is required in scenarios where multiple applications plan to use the same WebSphere eXtreme Scale grid to store their session data, or the HTTP session state is shared across different applications.


Conclusion

The second installment of this two part article continued the discussion of the super cluster by expanding it to include IBM WebSphere DMZ Secure Proxy Server V7, the IBM WebSphere Virtual Enterprise on demand router, and IBM WebSphere eXtreme Scale. The article described the strategy behind the technique for manually editing the static routing files so that two-level hierarchical clusters can be represented as traditional flattened clusters to DMZ secure proxy servers, and explained how to use the super cluster technique with the on demand router.

The super clustering technique also has an interesting application for using WebSphere eXtreme Scale with WebSphere Application Server Network Deployment clusters. In this situation, if the expected size of the WebSphere eXtreme Scale grid exceeds that of the combined heap space provided by a single WebSphere Application Server cluster, you can use the super cluster technique to create an extremely large grid. In fact, by increasing the number of clusters (each with a reasonable number of members), you can form WebSphere eXtreme Scale grids to store massive amounts of data. This article described the deployment topology, configuration, and management for such grids. It also discussed the possible usage of a WebSphere eXtreme Scale grid to support HTTP session failover for topologies where Java EE applications are deployed in super clusters.

The need for a super cluster does not occur in everyday practice, but should it happen for HTTP clients or large WebSphere eXtreme Scale grids, there are ways you can address it. All super cluster options have some limitations, but the restrictions might very well be appropriate, depending on the scenario.

You might consider a super cluster when you have an immediate need for a cluster with more than 100 members (50 for WebSphere Virtual Enterprise), or your deployment architecture needs to be flexible enough to accommodate more than 100 application servers (50 for WebSphere Virtual Enterprise) in a future cluster.

There is no theoretical limit on the number WebSphere Application Server instances a super cluster can contain, but the super clustering technique should not be considered a panacea for scalability. Resources used by specific Java EE applications can impose restrictions on the scalability of those applications. For example, with a database related application, the maximum number of database connections that can be served by the underlying database server can limit the number of WebSphere Application Server instances in which the application can be simultaneously deployed, and, hence, the scalability of the application.

In general, the topic of hyper sclability can be challenging to address. In some cases, it might require a paradigm shift in the traditional data modelling and programming disciplines. These topics are beyond the scope of these two articles. See Resources for information on eXtreme Transaction Processing (XTP).


Acknowledgements

The authors are grateful to the following persons who contributed toward either the preparation or review of this article for its technical contents and accuracy: Bob Westland, the WebSphere Application Server Network Deployment workload manager architect; Benjamin Parees, WebSphere Virtual Enterprise development; Keith Smith, on demand router architect; Chris Johnson, eXtreme Scale architect and co-lead; Doug Berg, eXtreme Scale architect and co-lead; Robert Wisniewski, eXtreme Scale evangelist; Hao Wang, eXtreme Scale development; and Peter Van Sickel, IBM Software Services for WebSphere consultant.

Resources

Learn

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=415462
ArticleTitle=Super cluster to the rescue, Part 2: Maximum scalability using WebSphere DMZ Secure Proxy Server, ODR, and WebSphere eXtreme Scale
publish-date=07222009