PM11713: Out Of Memory exceptions after large spikes in load with large tran..

Download Description

PM11713 resolves the following problem:

ERROR DESCRIPTION:
The primary symptom is an OutOfMemoryError. This may cause CNTR0019E or CNTR0020E errors, since the memory that is not being cleaned up is EJB cache memory.

The heapdump will resemble the following. com/ibm/ejs/util/cache/BackgroundLruEvictionStrategy, which has a com/ibm/ejs/util/cache/CacheElementEnumerator, which has an array of com/ibm/ejs/util/cache/Element objects.

Each com/ibm/ejs/util/cache/Element has a com/ibm/ejs/container/activator/TransactionKey, which has a com/ibm/ejs/container/ContainerTx, which then has an ArrayList which contains thousands of com/ibm/ejs/container/ContainerManagedBeanO objects.

The problem was reported using IBM WebSphere Commerce Suite, so the beans in the EJB cache are EJBs from that product. The issue is not with the CommerceSuite beans, but with how the WebSphere Application Server cache is handling them.

The key symptom in terms of the heap is the Array of thousands of ContainerManagedBeanO objects tied to a CacheElementEnumerator.

--------------------------------------------------------
This shows that the memory that is potentially being leaked is tied to the EJB cache that WebSphere AppServer maintains.

317,803,376 (33%) [176] 5
com/ibm/ejs/util/cache/BackgroundLruEvictionStrategy 0x72ebe0c8
317,803,064 (33%) [56] 3
com/ibm/ejs/util/cache/CacheElementEnumerator 0x8243f630
317,802,912 (33%) [472] 26 array of
java/lang/Object 0xa70484b8

--------------------------------------
Here is one of the elements in the EJB Cache.

238,307,992 (25%) [40] 2 com/ibm/ejs/util/cache/Element
0xb6ae1778
238,307,256 (25%) [24] 2
com/ibm/ejs/container/activator/TransactionKey 0xb6ae17a0
238,306,896 (25%) [136] 10
com/ibm/ejs/container/ContainerTx 0x88a960a0
164,283,672 (17%) [24] 1 java/util/ArrayList
0x88d8f5c8

Here is that giant array of beans. It is directly tied to a cached EJB. The array consists of lots and lots of com/ibm/commerce/order/objects/EJSCMPTradingPositionContainerXMLDefinHomeBean_c498873a beans.

What this says to me is that the order objects are not being cleaned up and are accumulating. There is an array of 185,387 ContainerManagedBeanO objects.

164,283,648 (17%) [794,328] 185,387 array of
java/lang/Object 0xa4bd5008
21,504 (0%) [176] 10
com/ibm/ejs/container/ContainerManagedBeanO 0x761eb0c8
21,504 (0%) [176] 10
com/ibm/ejs/container/ContainerManagedBeanO 0x96b5b6e0
21,504 (0%) [176] 10
com/ibm/ejs/container/ContainerManagedBeanO 0x9a4ea550
21,224 (0%) [48] 4
com/ibm/commerce/order/objects/TradingPositionContainerXMLDefini
tionBean 0x9a4ea600
104 (0%) [24] 1 com/ibm/ejs/j2c/HandleList 0x9a4ea538
825,919,352 (88%) [200] 33
com/ibm/ejs/container/EJSContainer 0x8240da48
959,128 (0%) [160] 18
com/ibm/commerce/order/objects/EJSCMPTradingPositionContainerXML
DefinHom eBean_c498873a 0x8f0e5d38
957,072 (0%) [56] 4
com/ibm/ejs/csi/PoolImplThreadSafe 0x8f0e8070
1,576 (0%) [248] 15
com/ibm/ws/pmi/server/modules/BeanModule 0x8f0e59c0
872 (0%) [80] 16 array of
java/lang/String 0x7a456348
128 (0%) [72] 5
com/ibm/commerce/order/objects/EJSJDBCPersisterCMPTradingPositio
nContainerXMLDefinitionBean_c498873a 0x8f0c2948
24 (0%) [24] 2
com/ibm/ejs/container/activator/OptCEntityActivationStrategy
0x82495070
16 (0%) [16] 0
com/ibm/ejs/container/lock/LockStrategy 0x9326ee08
21,504 (0%) [176] 10
com/ibm/ejs/container/ContainerManagedBeanO 0x8f0fe698
21,224 (0%) [48] 4
com/ibm/commerce/order/objects/TradingPositionContainerXMLDefini
tionBean 0x8f0fe748
104 (0%) [24] 1
com/ibm/ejs/j2c/HandleList 0x8f0fe680
825,919,352 (88%) [200] 33
com/ibm/ejs/container/EJSContainer 0x8240da48
959,128 (0%) [160] 18
com/ibm/commerce/order/objects/EJSCMPTradingPositionContainerXML
DefinHomeBean_c498873a 0x8f0e5d38

LOCAL FIX:
None

PROBLEM SUMMARY

USERS AFFECTED:
All users of IBM WebSphere Application Server containing large or long-running transactions.

PROBLEM DESCRIPTION:
Users can experience Out Of Memory exceptions following large spikes in load with large transactions.

RECOMMENDATION:
None

Out Of Memory issues may result after large spikes in load. Analysis of the heap dumps should expose large "inactive" transactions with EJB references being held by the BackgroundLruEvictionStrategy.
This APAR is a follow-on APAR to PK98076 that addresses a separate issue in the same area of code.

PROBLEM CONCLUSION:
Modified the EJBContainer code to completely clear the copy of the cache bucket so that all items will be garbage collected prior to the next valid sweep.

The fix for this APAR is currently targeted for inclusion in fix packs 7.0.0.11, 6.1.0.33, Feature Pack for EJB3.0 6.1.0.33, and 6.0.2.43.

Refer to the Recommended Updates page for delivery information:
http://www.ibm.com/support/docview.wss?uid=swg27004980