Troubleshooting
Problem
Java Out of Memory (OOM) errors cause MXServer termination under normal operating conditions once or twice a week.
Symptom
Several OOM errors are detected in the SystemOut log, for example:
[11/5/12 14:59:52:916 GMT+00:00] 0000082b SystemOut O
[TpmUIThreadPool : 627][INFO][SR ] OSXMLSerializer operation=Query
[11/5/12 15:00:09:891 GMT+00:00] 0000002c TimerImpl E ASYN0012E:
Alarm/Timer threw exception commonj.work.WorkCompletedException:
java.lang.RuntimeException: java.lang.OutOfMemoryError
at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java:1120)
at
com.ibm.ws.asynchbeans.timer.TimerImpl.runListenerAsCJWork(TimerImpl.jav
a:425)
at com.ibm.ws.asynchbeans.am._Alarm.fireAlarm(_Alarm.java:332)
at com.ibm.ws.asynchbeans.am._Alarm.run(_Alarm.java:229)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1497)
Caused by: java.lang.RuntimeException: java.lang.OutOfMemoryError
at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:124)
at
com.ibm.ws.asynchbeans.J2EEContext$DoAsProxy.run(J2EEContext.java:328)
at
java.security.AccessController.doPrivileged(AccessController.java:241)
at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java:1111)
... 4 more
Caused by: java.lang.OutOfMemoryError
at org.apache.xerces.dom.DeferredDocumentImpl.createChunk(Unknown
Source)
at org.apache.xerces.dom.DeferredDocumentImpl.ensureCapacity(Unknown
Source)
at org.apache.xerces.dom.DeferredDocumentImpl.createNode(Unknown
Source)
at
org.apache.xerces.dom.DeferredDocumentImpl.createDeferredDocument(Unknow
n Source)
at org.apache.xerces.parsers.AbstractDOMParser.startDocument(Unknown
Source)
at org.apache.xerces.impl.dtd.XMLDTDValidator.startDocument(Unknown
Source)
at org.apache.xerces.impl.XMLDocumentScannerImpl.startEntity(Unknown
Source)
at
org.apache.xerces.impl.XMLVersionDetector.startDocumentParsing(Unknown
Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at javax.xml.parsers.DocumentBuilder.parse(Unknown Source)
at
com.ibm.hipods.tpmpc.data.spec.ServiceSpec.<init>(ServiceSpec.java:105)
at
com.ibm.hipods.tpmpc.data.Project.getServiceSpecObj(Project.java:131)
at
com.ibm.hipods.tpmpc.action.ProcessProjectsTimerListener.processProject(
ProcessProjectsTimerListener.java:187)
at
com.ibm.hipods.tpmpc.action.ProcessProjectsTimerListener.processProject(
ProcessProjectsTimerListener.java:168)
at
com.ibm.hipods.tpmpc.action.ProcessProjectsTimerListener.access$000(Proc
essProjectsTimerListener.java:44)
at
com.ibm.hipods.tpmpc.action.ProcessProjectsTimerListener$1.timerExpired(
ProcessProjectsTimerListener.java:113)
at
com.ibm.ws.asynchbeans.timer.TimerImpl.callListenerMethod(TimerImpl.java
:298)
at com.ibm.ws.asynchbeans.timer.GenericTimer.run(GenericTimer.java:216)
at
com.ibm.ws.asynchbeans.J2EEContext$RunProxy.run(J2EEContext.java:257)
at
java.security.AccessController.doPrivileged(AccessController.java:214)
at javax.security.auth.Subject.doAs(Subject.java:495)
at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:118)
... 7 more
Cause
This problem is caused by a memory leak in TSAM identified in APAR IV30917, as well as another memory leak in db2 documented in APAR IV30917.
Resolving The Problem
The recommendation is to apply a fix in db2 followed by a fix in TSAM.
In DB2:
The db2 JCC APAR IZ36247 is fixed in db2 version 9.5 FP4, but the recommendation is to apply the latest JCC driver in 9.5 ( current one is 9.5 FP 10).
Please refer to the information in the following link for JCC driver version
and its corresponding db2level:
http://www-01.ibm.com/support/docview.wss?uid=swg21363866
Please also note that DB2 JCC is downlevel compatible with the DB2 server version.
For the recommended JCC upgrade, the client can just upgrade the JDBC
driver now and do the client upgrade later via the following alternative methods:
a) The procedure to upgrade the driver is to make sure the new jdbc driver
is downloaded and then set the classpath to point to the jar file. Basically the classpath of the application
environment should have the new driver , jcc driver should pick the correct version.
Ensure CLASSPATH is correctly set and includes db2jcc.jar and the
corresponding license files.
Please place the new jar file path first in the classpath followed by
others so that the application makes use of that jar file.
To know the version of your JCC driver:
java com.ibm.db2.jcc.DB2Jcc -version
b) You can just replace the existing db2jcc.jar file (i.e. rename your
original to db2jcc_old.jar) and copy this new jar into the same
location. so that you do not need to update the classpath. Please check to ensure this works.
Also, the jdbc driver needs to be copied to the following additional
locations:
./opt/IBM/SMP/maximo/tools/maximo/iscintegration/libs/db2jcc_license_cu.jar
./opt/IBM/SMP/maximo/tools/maximo/iscintegration/libs/db2jcc.jar
./opt/IBM/SMP/maximo/applications/maximo/lib/db2jcc_license_cu.jar
./opt/IBM/SMP/maximo/applications/maximo/lib/db2jcc.jar
A build and redeploy of the maximo.ear is required.
In TSAM:
The permanent fix for APAR IV30917 is included in TSAM 7.2.4.
LA fix 7.2.2.2-TIV-TSAM-LA0011 was provided to client, this can be
applied to a TSAM 7221 system as well.
Was this topic helpful?
Document Information
Modified date:
17 June 2018
UID
swg21621448