Topic
  • 3 replies
  • Latest Post - ‏2013-06-07T11:23:37Z by bpaskin
SystemAdmin
SystemAdmin
37421 Posts

Pinned topic Seeing deadlocks when requesting a connection from a connection pool - Help

‏2011-06-16T16:34:58Z |
I java a java EAR application deployed in WebSphere.
When I restart the server, it often hangs because the various parts of the code are all attempting to get database connections.
The threads seem to be deadlocking, and from the thread dumps, the deadlocks seem to be around synchronized sets within the pool management code:


Stack trace #1:

Name: DEAG_ISO20022FileIOGateway - Outbound Thread #0
State: BLOCKED on java.util.Collections$SynchronizedSet@2d682d68 owned by: OFCLVGateway - Inbound Thread #0
Total blocked: 14 Total waited: 2

Stack trace:
java.util.Collections$SynchronizedSet.hashCode(Collections.java:837)
com.ibm.ejs.j2c.PoolManager$SubjectHashCode.run(PoolManager.java:4853)
com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:63)
com.ibm.ejs.j2c.PoolManager.computeHashCode(PoolManager.java:4528)
com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2165)
com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1059)
com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:696)
com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:668)
com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:635)
...


Stack Trace #2:

Name: OFCLVGateway - Inbound Thread #0
State: BLOCKED on java.util.Collections$SynchronizedSet@7c657c65 owned by: OFCLVGateway - Inbound Thread #4
Total blocked: 4 Total waited: 0

Stack trace:
java.util.Collections$SynchronizedCollection.size(Collections.java:478)
java.util.AbstractSet.equals(AbstractSet.java:57)
java.util.Collections$SynchronizedSet.equals(Collections.java:831)
- locked java.util.Collections$SynchronizedSet@2d682d68
com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl$Equals.checkPrivateCredentials(WSManagedConnectionFactoryImpl.java:3315)
com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl$Equals.run(WSManagedConnectionFactoryImpl.java:3275)
com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:63)
com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.matchManagedConnections(WSManagedConnectionFactoryImpl.java:2253)
com.ibm.ejs.j2c.FreePool.getMCWrapperFromMatch(FreePool.java:1892)
com.ibm.ejs.j2c.FreePool.getFreeConnection(FreePool.java:882)
com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2184)
com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1059)
com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:696)
com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:668)
com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:635)
...


Stack Trace #3:

Name: OFCLVGateway - Inbound Thread #4
State: BLOCKED on java.util.Collections$SynchronizedSet@2d682d68 owned by: OFCLVGateway - Inbound Thread #0
Total blocked: 3 Total waited: 0

Stack trace:
java.util.Collections$SynchronizedCollection.size(Collections.java:478)
java.util.AbstractSet.equals(AbstractSet.java:57)
java.util.Collections$SynchronizedSet.equals(Collections.java:831)
- locked java.util.Collections$SynchronizedSet@7c657c65
com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl$Equals.checkPrivateCredentials(WSManagedConnectionFactoryImpl.java:3315)
com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl$Equals.run(WSManagedConnectionFactoryImpl.java:3275)
com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:63)
com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.matchManagedConnections(WSManagedConnectionFactoryImpl.java:2253)
com.ibm.ejs.j2c.FreePool.getMCWrapperFromMatch(FreePool.java:1892)
com.ibm.ejs.j2c.FreePool.getFreeConnection(FreePool.java:882)
com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2184)
com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1059)
com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:696)
com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:668)
com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:635)
...

(full thread dumps for these threads are attached)


My setup is the following:
Integrated Solutions Console, 7.0.0.13
Build Number: cf131039.07
Build Date: 10/2/10
Base version, 32-bit, running on 64-bit Windows Vista

JDBC Provider related to the pool:
uses ojdbc16.jar
implementation class name: oracle.jdbc.xa.client.OracleXADataSource

Data source:
set to use Oracle 11g data store helper

Connection Pool:
Connection timeout: 300 (I changed this from default)
Max connections: 100 (I changed this from default)
Min connections: 5 (I changed this from default)
Reap time: 180
Unused timeout: 1800
aged timeout: 0
purge policy: failing connection only
no advanced/custom pool properties changed

Current performance monitor metrics on this pool indicates that the pool size is only about 50 connections, well below max connections.
DB:
Oracle version 11.2.0.1.0
(The DB is 64 bit, also running on my Windows Vista machine)


The problem comes and goes. Mostly, it appears.
I've tried re-deploying my application, and also removing everything under the temp and wstemp directories before restarting WebSphere. Sometimes that does the trick, sometimes it doesn't.

I have looked at the fix list for 7.0.0.15 and 17, nothing seems to be similar to this problem.
I have also tried to upgrade to version 17, but deploying my application there fails when rmic runs to generate the EJB stubs. This failure does not happen in version 13.

Any help would be greatly appreciated.

Thanks,
-Ian
Updated on 2011-06-20T22:57:38Z at 2011-06-20T22:57:38Z by bkail
  • bkail
    bkail
    372 Posts

    Re: Seeing deadlocks when requesting a connection from a connection pool - Help

    ‏2011-06-20T22:57:38Z  
    This looks like PK69110 (http://www-01.ibm.com/support/docview.wss?uid=swg1PK69110&wv=1). Once an appropriate fixpak is installed, I believe the useRRASetEquals system property must be set in order to enable the fix.
  • Ajay Vasu
    Ajay Vasu
    2 Posts

    Re: Seeing deadlocks when requesting a connection from a connection pool - Help

    ‏2013-06-07T10:21:23Z  
    • bkail
    • ‏2011-06-20T22:57:38Z
    This looks like PK69110 (http://www-01.ibm.com/support/docview.wss?uid=swg1PK69110&wv=1). Once an appropriate fixpak is installed, I believe the useRRASetEquals system property must be set in order to enable the fix.

    Hi Friends,

    We are facing  Lock time out period Exceeded issue using IBM Websphere.We tried setting lock time out property to 2000 still we are facing lock request time out exception.

    com.microsoft.sqlserver.jdbc.SQLServerException: Lock request time out period exceeded.
        at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:196)
        at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1454)
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:388)
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:338)
        at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4026)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1416)

    Can you guys help us and say whether we have any specific setting in websphere to avoid these exceptions when executing huge number of transactions?

    We tried setting lock timeout property to -1 which causes deadlock and all transactions were gettng rollbacked.

    Do we need to update any drivers for this? we are using sqljdbc driver.

    This error is specific in IBM WAS plus SQL Server 2008. IBM WAS Version 21 and 15.

    This issue specifically occurs in transactions using JMS Listeners.

    Mail ID : ajay.nair42@gmail.com,ajay.vasu@exterro.com

    Please provide your valuable comments.

    Updated on 2013-06-07T10:27:46Z at 2013-06-07T10:27:46Z by Ajay Vasu
  • bpaskin
    bpaskin
    5960 Posts

    Re: Seeing deadlocks when requesting a connection from a connection pool - Help

    ‏2013-06-07T11:23:37Z  
    • Ajay Vasu
    • ‏2013-06-07T10:21:23Z

    Hi Friends,

    We are facing  Lock time out period Exceeded issue using IBM Websphere.We tried setting lock time out property to 2000 still we are facing lock request time out exception.

    com.microsoft.sqlserver.jdbc.SQLServerException: Lock request time out period exceeded.
        at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:196)
        at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1454)
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:388)
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:338)
        at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4026)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1416)

    Can you guys help us and say whether we have any specific setting in websphere to avoid these exceptions when executing huge number of transactions?

    We tried setting lock timeout property to -1 which causes deadlock and all transactions were gettng rollbacked.

    Do we need to update any drivers for this? we are using sqljdbc driver.

    This error is specific in IBM WAS plus SQL Server 2008. IBM WAS Version 21 and 15.

    This issue specifically occurs in transactions using JMS Listeners.

    Mail ID : ajay.nair42@gmail.com,ajay.vasu@exterro.com

    Please provide your valuable comments.

    Hi, You should have started a new thread for this problem.

    Lock timeouts are with the Database and not the JMS Listeners.  This means that the application was calling the database and the locking on the database to retrieve data was taking too long and timed out.  This is a problem either with the query and may indicate that you need to somehow reorg your database to get faster results.

    Regards,

    Brian