Default behavior of managed connections in WebSphere Application Server
What if one size does not fit all?
Whether discussing automobiles, the company medical plan, or a personal computer, most of us could agree on one point: the default configuration -- while useful in many cases -- may not necessarily be right for everyone. So it is with managed connections in IBM WebSphere Application Server V5 and V6. We will look at various aspects of the default behavior of managed connections in WebSphere Application Server and identify some situations where you might want to consider changing the configuration to better suit your application requirements.
A transaction can be described as a unit of work in which multiple updates to resources are made atomically; meaning that either ALL the resource updates will be made permanent if the transaction is committed, or NONE of the resource updates will occur if the transaction is rolled back. Transactions can be broadly categorized as either global or local:
- Global transaction context -- A global transaction is also referred to as a distributed transaction, an XA transaction, a 2PC (2-phase commit) transaction, or just a transaction. This type of transaction is global, or distributed, in the sense that multiple resource managers may be accessed within the transaction and the transaction manager has the ability to coordinate among all the resource managers to ensure the atomicity of updates. Conversely, a local transaction involves only a single resource manager.
A global transaction context may be created automatically for Enterprise Java™Bean (EJB) methods by using container managed transaction support and specifying an appropriate transaction attribute for the method. The Java Transaction API (JTA) UserTransaction interface may also be used to create a global transaction context from within a servlet, a message driven bean, or an EJB component that specifies the use of bean managed transactions. Use of the UserTransaction interface is optional, however, so in its absence WebSphere Application Server provides an alternate transaction context.
- Local transaction containment -- The EJB 2.0 specification states that an application which operates outside of a global transaction will execute instead under an "unspecified" transaction context. Naturally, an implementation of a Java 2 Platform, Enterprise Edition (J2EE) application server cannot behave randomly, and so the application server must specify its implementation of the unspecified context. In the case of WebSphere Application Server, this context is defined as a local transaction containment (LTC). The LTC functions as an implicit, containing scope for any local transactions that occur within the LTC boundaries.
The initiator of a local transaction is expected to complete each local transaction within the LTC boundary. Typically, this is accomplished by having the connection's auto-commit mode set to true -- or, if auto-commit is set to false, the application would explicitly call Connection.commit() or Connection.rollback(). However, the (Web or EJB) container is charged with completing any outstanding local transactions at the end of the LTC boundary. This serves as a safety net for defective applications which fail to complete their own transactions. The container will also release to the free connection pool any outstanding connections associated with the LTC.
Connection pooling and shareable connections
Connection pooling in WebSphere Application Server is managed according to the Java Connector Architecture (JCA) and enables the use of shareable or unshareable connections. In WebSphere Application Server, connections are shareable by default. The use of shareable connections means that, if conditions allow it, different getConnection requests by the application will actually receive a handle (indirectly) for the same physical connection to the resource. The benefits of this are improved performance and a reduction in the number of physical connections that need to be managed.
The JCA 1.0 spec has the following to say about shareable connections:
When multiple shareable connections x and y acquired by an application are used within a global transaction scope (for instance, container-managed or bean-managed), the application server must provide a single shared connection behavior under the following conditions:
- x and y are collocated in a single Java Virtual Machine process address space.
- x and y are using a single transactional resource manager.
- x and y have identical properties.
- x and y are marked as shareable.
- x and y are used within a container-managed or bean-managed transaction scope.
The ability to share is unspecified for connections marked shareable that are used outside a global transaction scope. Sharing is not supported for connections obtained from a non-transactional resource adapter, that is, transaction support level is NoTransaction.
Once again, we have an "unspecified" case with regard to connection sharing in the absence of a global transaction scope; that is, within an LTC scope. WebSphere Application Server has defined its behavior in this case with the assumption of serial reuse for shareable connection requests. Serial reuse is also referred to as get/use/close, get/use/close, where the expectation is that the application will get a connection, use it, and then close it while doing other processing before (possibly) getting another connection to the same resource. One key point regarding this behavior is that when the application closes a shareable connection, the connection is not truly closed, nor is it returned to the Free pool. Rather, it remains in the Shared connection pool, ready for another request within the same LTC for a connection to the same resource.
The use of shareable connections can provide performance benefits in many situations, but you need to be aware of this default behavior and take it into account when developing applications. Otherwise, the application could experience problems under heavy workload.
When a servlet is invoked, the Web container in WebSphere Application Server automatically creates an LTC context. If a shareable Java Database Connectivity (JDBC) connection is obtained within that LTC, the connection pooling code allocates the managed connection in the shared pool, where it is marked as having an affinity to the current LTC. As stated earlier, when the application closes the connection, it is not returned to the free pool, but is kept reserved for possible further use in that LTC. The connection is not returned to the free pool until the LTC ends. The LTC does not end until the servlet doService() method ends.
If a servlet gets a connection, uses it, and closes it, but then remains in the servlet for a relatively long period of time, that connection remains unavailable for use by other threads. In times of heavy workload, such a situation can lead to a shortage of available connections.
Likewise, if the servlet obtains a connection and then passes control to another servlet (via forward or include), or to an EJB, then the connection remains unavailable for use until control returns to the first servlet and the first servlet ends. Furthermore, when control is passed out of the first servlet, a new transaction context is created. Thus if another connection to the same resource is requested within the new transaction context, the connection held by the first servlet cannot be shared. A new connection must be allocated with affinity to the new transaction context. The thread in this case is now holding two connections from the same pool. The outline below shows how this scenario might look, creating the danger of a deadlock on the connection pool if too many threads do this concurrently. A situation could develop where all the connections available in the pool are held by different threads, and each thread that is holding a connection is trying to get another connection -- when none are available. This deadlock will persist until ConnectionWaitTimeouts cause threads to return a failure response and release their connections.
- LTC A begin
- Servlet A begin
- get Connection A //shareable
- use Connection A
- close Connection A
- // Connection A remains in shared pool, associated with LTC A
- include or forward Servlet B
- LTC A suspend
- LTC B begin
- Servlet B begin
- get Connection B //shareable
- use Connection B
- close Connection B
- // Connection B remains in shared pool, associated with LTC B
- Servlet B end
- Servlet B begin
- LTC B end //Connection B released to free pool
- LTC A resume
- // possible Connection A re-use (get/use/close)
- Servlet A end
- Servlet A begin
- LTC A end //Connection A released to free pool
A similar scenario can occur when servlet filters are used. Filters are related to servlets and are inserted in the HTTP request/response flow by the Web container, such that the filter may access the request/response data before the servlet starts, and again after the servlet ends. Each filter will run in its own LTC context that will span the entire request/response life cycle. It is possible that a filter might obtain a connection to a JCA managed resource. As with normal servlets, if that connection is shareable, then even though the connection is closed before the filter passes control on to the next filter or servlet, the connection will not be released to the free pool until the LTC ends. The LTC does not end until control returns to the filter and the filter ends.
As before, if a filter gets a shareable connection from a given datasource, closes the connection, and passes control to another servlet or filter, which in turn gets a connection from the same datasource, there will now be multiple connections to the same datasource on that thread. No sharing or re-use is enabled because the connections were obtained in distinct transaction contexts, and once again there is the potential for deadlock on the connection pool during heavy workload.
Solutions and caveats
If an application is structured, as in the above scenarios, and is using multiple connections per thread, then take measures to minimize the number of connections needed and protect against the deadlock scenario. Some suggestions follow on how to do that.
The behaviors described above were all predicated on the use of shareable connections. When Connection.close() is called on the shareable connection obtained in an LTC, the connection remains in the shared pool. In this way, the application behavior of serial re-use is accommodated with minimum latency. Conversely, if an unshareable connection is used, there is no assumption that the connection may be re-used within the LTC. Indeed, the unshareable connection has no LTC affinity. A more conventional behavior occurs such that, when the application executing in an LTC closes the connection, the connection is immediately returned to the free pool, even though the current LTC has not ended. If control then passes to another servlet, for example, and a new LTC context is created, a connection obtained in the new context will be the only connection held by that thread.
If the application is correctly using resource references, then this solution is quite attractive, since the change from shareable to unshareable connections is as easy as modifying the appropriate resource references in the deployed application and then reinstalling the application. A possible downside to this approach is that, since the unshareable connection has no affinity to the LTC, the LTC is unable to provide the aforementioned "safety net" to prevent a leaked connection. However, this is much preferred over the potential for deadlock if the application would otherwise be using multiple connections per thread.
A way to make a shareable connection get released to the free pool before the LTC ends is to keep it from being associated with the LTC. This can be done by enclosing the use of the connection in a global transaction via the UserTransaction interface of the Java Transaction API. As before, the LTC will be active initially, but when the global transaction is started, the LTC will be suspended. A shareable connection obtained within the global transaction will be released to the free pool when the transaction is ended. The point where that occurs is now under the control of the application.
The use of global transactions within the LTC will incur more overhead in terms of application complexity and also in terms of performance. Additionally, the use of the UserTransaction interface from within a servlet filter is prohibited by the J2EE specification.
Another solution may be to restructure the application to isolate the use of shareable connections so that they become naturally associated with separate transaction contexts rather than the main path LTCs. As an example, a main path servlet could include a separate servlet or call an EJB to access a resource. When control returns to the main servlet, any connections used will have been released. In this way, the application can make the default behaviors work in its favor. Such structure is also good for modularity of the application and may enhance re-use and application longevity.
The disadvantage to restructuring the application is, well, the application needs to be restructured.
WebSphere Application Server provides local transaction containment as a default transaction context in the absence of a global transaction context. Shareable connections obtained by an application within an LTC are kept reserved in the shared pool for use within that LTC until the LTC ends, even if the application closes the connection. This behavior is beneficial to many applications, but can have unintended consequences for others. Depending on the application design, some modifications to the application or the configuration may be desirable to ensure the application's reliability in times of heavy workload.
- Connection pooling article from the WebSphere Application Server Information Center; see section "Avoiding a deadlock"