Portal administration and performance

Introduction

Database access is usually costly. There is the cost of driving the Database engine. There is the cost of response time in the application waiting for database results. If a particular database result is used often, then caching the results of that SQL call in the application server is a natural requirement to reduce the effect of these costs.

Because the use of database calls is quite common in WebSphere Portal as well as WebSphere Application server, care must be taken to optimize all facets of these SQL calls. In Portal, it is common for the theme as well as Portlets to use database calls therefore caching is imperative here. The most common caching mechanism used in WebSphere is the Dynacache.

A Dynacache is simply a cluster aware Java Hashmap. This Hashmap is cluster aware in the sense that if a “key” is invalidated or updated on one cluster member, appropriate measures are taken on all cluster members to insure that the “value” in the Dynacache Hashmap is either updated or removed as well. The key/value pair can also be inserted into the Dynacache with a time-out value to insure freshness.

So, what are the best practice patterns for the use of a database backed Dynacache instance? This article will attempt to highlight those best practices.

Initial Code

Initially, one can use a simple “dao” (data access object) to fetch the database contents based on some key value. In this example, the key is the “uniqueID”.

DBResults results = dao.getRow(uniqueID);

Obviously, a call to “dao.getRow() “ results in a database access every call. If this call is frequent and uses the same parameter(s), consider caching the result in a Dynacache.

Initial Dynacache Implementation

First, we must create the Dynacache instance along with a JNDI name which applications will use to reference this instance. This is done on the Deployment Manager GUI under "object instances".

To use the new Dynacache instance, the simplest thing to do is to create a Dynacache "impl" class with a "get" method that looks exactly like the call to the database “dao” class' "get" method. It could use exactly the same parameters in most cases.

DBResults results = daoHelperImpl.getRow(uniqueID);

In the new Impl class, check first if the value is in the Dynacache. If not, fetch it. If so, return it:

public DBResults getRow(String uid) {
   if (aDynacache.keySet().contains(uid)) {
      System.out.println("UID was in the cache");
      }
   else {
      aDynacache.put(uid, dao.getRow(uid));
      }        
    return (DBResults) aDynacache.get(uid); }

Performance Improvements

This is perhaps the most important section of this paper. In a web based application, performance is very important. So, having a version of the Dynacache based database access that avoids repeated accesses to the database is important.

The “impl” class above is a good first pass. However, there is a problem that manifests under load. When there are multiple threads all using the same “uniqueID” and the Dynacache is not yet loaded, you could have lots of thread all trying to write the dao results into the cache if the cache was previously empty.

This problem is resolved by “synchronizing” on the moethod that is used for updating. Kudos to Antonio Fiol Bonnin for finding the error in my original pseudo code.

    public DBResults getRow(String uid) {
        
        if (aDynacache.keySet().contains(uid)) {
            System.out.println("UID was in the cache");
            return (DBResults) aDynacache.get(uid);
            }
        else {
            return checkAndSet(uid);
            }
        }
    
    private synchronized DBResults checkAndSet(String uid) {
        // Handle the case where another thread updated the value while you were waiting    
        System.out.println("UID was NOT in the cache. Wait for other threads, check to see if then added, otherwise must add");
        if (!(aDynacache.keySet().contains(uid)))
            aDynacache.put(uid, dao.getRow(uid));
        return (DBResults) aDynacache.get(uid);
    }

One might see that there is a redundancy here in that the “distributedMap.keySet().contains(uniqueID)” test is duplicated. The reason is simple. Running the test first without the “synchronized” block is much faster and should result in significant time savings if the cache is already loaded. If, however, the cache were empty and this thread was blocked waiting on another thread similarly blocked, it is not clear who wins the race and is first to access the database. You do not want both threads then hitting the database for the same key. It is sufficient for only one thread to actually read the key/value pair from the database and then to update the Dynacache instance.

This implementation insures that only one database access is used to update the key and provides the best performance for all threads as well as the database.

A common problem that customer's often face is having a large Portal cluster in one part of the world, but having to access it from another. Because of the large number of "GET" requests needed to render a Portal page along with the latency of transcontenental access, performance typically suffers.

One way to alleviate this latency issue is by caching the "statics" required to render a Portal page. Generally, only one of these 1 GET requests is required to render a Portal page needs to be calculated dynamically by the Portal server. The rest are typically CSS, Javascript and images that are invarient. These invarient statics could be rendered from a cache if available. By caching these statics locally (e.g. with low latency), one can greatly reduce Portal render times on average.

A typicaly Portal topology might look like this:

Client browser <-> WebServer <-> Portal Server

To improve the performance, a user can change the topology to this:

Client browser (in EMEA, for example) <-> WebServer (in EMEA) <-> Web Server (in USA) <-> Portal Cluster (co-located in USA)

In addition, the WebServer in EMEA would impliment a caching proxy as a part of the reverse proxy cache. So, the client brower would access the site via the EMEA WebServer. That EMEA webserver would serve content from it's local cache as able. If the content is not in the local cache, it would forward the request to the USA WebServer. Note that this would always happen for the base Portal page as well as any statics not already in the EMEA cache.

Instead of using the address of the server(s) in the USA, the EMEA customer would now request the page via the hostname of the EMEA WebServer. Something like this:

http://host.usa.com/wps/portal becomes http://host.emea.com/wps/portal

Users could either use the new hostname (as immediately above), or the DNS server in their geography could return the EMEA webserver instead of the USA version of it. The later would make this local proxy server strategy transparent to the user.

The EMEA Apache web server would use this in it's httpd.conf to proxy the request to the USA webserver:

ProxyRequests off
ProxyPass /wps/ http://host.usa.com/wps/
<location /wps>
ProxyPassReverse /
RequestHeader    unset  Accept-Encoding
</location>

Note also that the EMEA web server would also use mod_disk_cache for the local caching of static content.
 

Whenever you deploy a new WebSphere Portal instance you must tune the Portal in a manner appropriate to the environment. For example, out of the box, Portal is NOT tuned for production.

The IBM Portal Performance team has produced several documents that contain the tuning that should be done after installation and before deploying a Portal to production. This is the document for V7, the one for V8 and this one for V8.5. This document should be considered a prerequisite for the beginning of performance testing and certainly a prerequiste for production.

There are several components that should be tuned. These include the Portal itself, the LDAP, the database, the OS and the web server.

Mike White and I have written a ConfigEngine script that will apply the tunings for the Portal automatically. The script is included in Portal V7.0.0.2 CF22+, Portal V8 CF05+ and all levels of 8.5. Use of this script will improve the accuracy of applying the tuning changes as well as reduce the time needed to do the task.

The changes are driven by a properties file (tuning.properties) along with several resource environment providers files. Since the tuning.properties file assume you are a WCM rendering server that is only a subscriber (and NOT a syndicator), you may need to adjust some of the setting to match your environment. You can easily do this by copying the read-only tuning.properties to a local directory, updating the name/value pairs that need updating and point to the new copy when you run the task.

Note for zOS users: This task does run on zOS. Note however, that there were significant fixes put into the code to resolve zOS issues in March, 2014. These fixes wil be included in 8001, CF10 as well as the initial release of V8.5. If you are on zOS and not running at least 8001 CF10 level, please download the linked code immediately following.

The latest README file is linked here. It provides the details on using this new task. I have also included the (current) latest version of the code here. The latest version can be untarred in the PortalServer/installer/wp.config/config subdirectory as is. Note that the default the subdirectory just mentioned does NOT have write permission. The permission must be changed before trying to untar this file. As a best practice, download the latest from the provided link regardless of your level of Portal. The latest version works on all levels.

Please Note: These defaults in the tuning.properties file assume the server being tuned is a WCM rendering server that is NOT subscriber only. It will also set the WCM advanced cache. If you are running this on an authoring server, you should consider turning off WCM advanced caching.

Virtual portals are ideal for setting up many micro sites that all have the same sets of applications in common. The most common usecases involve central service providers hosting a specific application suite for a series of independent yet similar tenants (departments, sales teams, vendors, etc) that all need access to that same application but want a differently branded experience. This is the usecase for which virtual portal was designed. This allows the maximized use of costly infrastructure and allows more users through the system without impacting the system with too many disparate applications.

That being said, virtual portals can also be used to host totally different user communities with different application needs. However, you do have to be more careful with how having many different applications will impact JVM heap utilization. Applications are shared resources across all virtual portals, regardless of whether the VP is actually using a particular application or not. Therefore, users in VP A can experience the effects of applications running for users in VP B. You can isolate the application environment from the virtual portals using WSRP, running those applications in their own JVM, but otherwise there is no application isolation inherent in virtual portals, nor will there ever be (hence the term "virtual").

If all the tenants use a common set of applications, you can likely scale to a very large number of tenants and virtual portals. If there is very little in common between the virtual portals, then you can expect to only scale so far, how far will depend on the relative expense of the applications in terms of memory consumption.

We have documented in the past that we only support 1000 virtual portals. That isn't actually true. We have only tested (back in WP 5.1) up to 1000 virtual portals. There are no physical limits to the number of VPs we can support. The number of virtual portals a system can support is highly dependent on the size of the VP in terms of pages, the number of users having access to that VP, and the inherent overhead of the applications in the system. There is very little to no overhead in the usage of virtual portal by itself.

For a system with a large number of virtual portals, performance is optimized when:

• Most users have access to a small number of pages relative to the whole portal (e.g. access to only one VP)
• Multiple users have the same roles on the same sets of pages (optimizes the caching of common content)

Otherwise, a large amount of resource (memory and CPU) is consumed keeping up with every user's configuration.

Performance will eventually degrade with very large virtual portal deployments with little content commonality between users, as our caching mechanism becomes less efficient and too memory intensive. What "large" means will vary by implementation, but is most likely in the several thousand range. At that point, you should consider adding parallel portal clusters for future VP instances and federate between them using technology like WebSphere Application Server Extended Deployment, which knows how to route users based on virtual portal ID to the correct Portal cluster.

There are a lot of knobs and switches in the WebSphere Portal and Application Server software stack that you can play with in the process of tuning the environment. We provide guidance and recommendations of what to pay attention to through our Performance and Tuning Guide, but the fact is that what actual values work best for you depends highly on what type of applications you run in Portal and the size and activity of your user population. The only way I know of to accurately arrive at the correct set of tuning values is to follow this basic testing methodology:

1. Install and configure WebSphere Portal on a single system with some simple default content and perform the requisite tuning as perscribed by the Performance and Tuning guide.
2. Take a baseline performance measurement of this simple site, making adjustments to the infrastructure and software tuning until desirable results are obtained. Also get an idea for how many users the baseline system can sustain before the system runs out of resources (more on this later).
3. Start adding content and repeat the analysis. Adjust the tuning as necessary. At least now you know, based on the baseline, that as changes are made, if things go wrong, you have an idea of what change may have instigated it.

Performance and capacity testing is a long, highly iterative process. It is also resource intensive, as it requires dedicated systems for days on end, as well as enough people to manage the test environment, observe test results, and tune the system. Often times, this important process is the first thing cut from a deployment plan in jeapordy. In my experience, you either pay for this time up front, before you go live, or you pay for it after you go live. It must be done, and is a lot more expensive the later you do it.

But all this being said, what should your goal be? Obviously, you go into this process with certain metrics in mind. For instance, I want to be able to handle 400 concurrent users with no worse than a 5 second response time, and maybe that is only during login. That's fine, and simple to measure, but there will be days where, for some reason, you have a lot more than 400 concurrent users, or you have to take systems down and the remaining systems must take the load. It isn't enough to know if your environment can handle the typically load; you need to know if it can handle the atypical, or worst case, scenario as well. You may not know what the worst case scenario is up front. But what you do need to know is what the maximum capacity of your portal environment is, so as the usage approaches that number, you will know you are in trouble and need to add capacity.

To understand what your environment's capacity is, you need to drive utilization of your portal environment to the point of CPU exhaustion. Not memory exhaustion, or DB connection exhaustion - CPU exhaustion. The reason for that is that as CPU utilization approaches 100%, the system slows down to the point of nearly being unresponsive, but it doesn't fail. Under this condition, the Web servers managing the load across a portal cluster will mark such a system as down and route traffic elsewhere, giving the server a chance to recover, which it should once the requests have been processed. If you run out of some other resource, like memory or DB connections, before running out of CPU, then things really start to fail and you won't recover from that.

So, as you perform your maximum capacity tests, give the server instances a large enough Java heap size and request and DB connection pool sizes to allow for enough traffic through driving CPU to 100%. If you can't before running out of resources other than CPU, then it is time to scale vertically (creating vertical cluster members on the same physical system) or reallocate processors to other systems. If you can configure the system to meet this goal, then you have your "sweet spot".