WebSphere Portal at its core is a collection of enterprise archives files (.ear) and web application archive files (.war) deployed to a WebSphere Application Server. Most versions of WebSphere Portal ship over 100 applications to the Portal server. Each of these applications may require a location in the server memory to store data temporarily. Most commonly the location where this data is stored a session. For a single application on a WAS server - managing the session is a trivial exercise. However, on an application server with over a hundred applications - such as WebSphere Portal - managing each individual session can be a difficult task.
WebSphere Application Server - realizing that managing several hundred applications on a single application server would be difficult - extended the standard javax.servlet.http.HttpSession class with the IBMSessionExt class. Per the Javadoc - "The IBMSessionExt interface extends the javax.servlet.http.HttpSession interface of the Servlet API to Invalidate all sessions with the same session id as the current session." Over a period of time, several sessions can accumulate on a Portal server and with each application potentially needing a session to store data for each individual user- memory can be used up quickly. WebSphere Portal makes use of the invalidateAll() command within the IBMSessionExt class during a user logout to ensure session data gets cleaned up.
However, even with the use of the invalidateAll() command to cleanup session memory, session leaks can still occur within the Portal server. This can be due to an application setting the lifetime of the session to infinite such that it will never be garbage collected, the application storing an excessive amount of data within the session, and/or, the sheer number of sessions on a single Portal server. While monitoring tools exist that can help with tracking session usage - often they are not helpful with diagnosing leaks. This blog entry will discuss utilizing a tool which will help in determining where session leaks may exist.
IBM Monitoring and Diagnostic Tools for Java
IBM offers a a tool called IBM Support Assistant (ISA) to perform a wide variety of diagnostics against multiple IBM products. Prior versions of ISA required a download and installation to Portal servers - which often was not feasible for production environments. The current version of ISA at the time this blog entry was authored - v5 - may be downloaded and executed as a standalone program - no installation required. ISA may be downloaded from the following link:
http://www-01.ibm.com/support/docview.wss?uid=swg27039277. The download itself is over 1GB in size - be patient! For purposes of this blog entry - I downloaded the ISA5 tool to a Windows2008 system. Also ensure you have a version of Java installed to the system you will be executing the following steps on.
1) Once the download is completed, extract it and execute the "start.isa" executable.
2) Open ISA5 in a web browser - e.g.https://localhost:10943/isa5/
3) Click on the Tools tab and scroll down until you see the "Memory Analyzer [Desktop]"
4) Click the Launch command, leave the values in the boxes at their defaults.
Analyzing the Session Data
1) Ensure your Portal server(s) operating systems are configured to generate full core dumps. See the following document for details of how to configure the operating system for such a setup.
2a) For Portal servers v80 and later, generate a core dump utilizing the DMGR / WAS Admin Console (Troubleshooting > Java dumps and cores > System Dump)
2b) For Portal servers v70 and earlier, a kill -6 command (*NIX) or task manager (Windows) will be needed to generate the core dump.
3) Wait 2 hours. Generate a second core dump.
4) Copy the core dumps from the Portal server to the server where the ISA5 tool is running.
5) Click on File --> Open Heap dump
*Note: This is how the tool has its menus worded, the tool can open either a core dump or a heap dump and we want to open the core dump. A heap dump will not output the information needed to determine if a session is leaky.
*Note - the security username set to anonymous is correct behavior. Portal v8 in its default settings does not have the setting "Session Security Integration" enabled by default. Enabling this setting will show the userID associated with a given session. For more details on this setting - see the Portal Security Hardening Guide and the following blog entry.
For a Portal v85 system running on WAS v855, the following screen was seen.
*Note - for Portal v85 the Session Security Integration setting is enabled by default. Hence the Security User Name shows the "wpsadmin" user.
7) Sort by the various columns to analyze the data. Items which should generate red flags for review:
- Any sessions which have >1MB for session size (heap size in the screenshots) - most sessions do not grow this large in size.
- Any sessions which have -1 for the timeout value, and show up in BOTH of the core dumps - these likely will not be garbage collected and may be the sign of a leak.
- Any username with more than 5 session per application - is the user logging in multiples times? Why? Or is this the sign of a user trying to break the system?