Topic
3 replies Latest Post - ‏2013-03-08T11:49:56Z by SystemAdmin
ahm.m
ahm.m
1 Post
ACCEPTED ANSWER

Pinned topic Websphere FileNet Content Engine JVMs hangs

‏2013-03-07T09:14:35Z |
Hi
We have WAS 7.0.0.19 on windows 2008 x64 ,we have DB=MS SQL server 2008 ,
we recently face WAS JVMs hangs every day here below we have error at the time of hang.
Looking for suggestions

[ ] 00000003 ThreadMonitor W WSVR0605W: Thread "ORB.thread.pool : 569" (00000494) has been active for 614171 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:140)
at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1525)
at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:3258)
at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:4433)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:386)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:338)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4026)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1416)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:185)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:160)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.execute(SQLServerPreparedStatement.java:320)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.pmiExecute(WSJdbcPreparedStatement.java:961)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.execute(WSJdbcPreparedStatement.java:637)
at com.filenet.engine.dbpersist.DBExecutionElement.execute(DBExecutionElement.java:218)
at com.filenet.engine.dbpersist.DBExecutionContext.getNextResult(DBExecutionContext.java:106)
at com.filenet.engine.dbpersist.DBStatementBase.execute(DBStatementBase.java:249)
at com.filenet.engine.retrieve.AdHocClassRetriever.executeSQLStatement(AdHocClassRetriever.java:1783)
at com.filenet.engine.retrieve.AdHocClassRetriever.doSearchImpl(AdHocClassRetriever.java:1446)
at com.filenet.engine.retrieve.AdHocClassRetriever.doSearch(AdHocClassRetriever.java:1066)
at com.filenet.engine.retrieve.AdHocClassRetriever.executeSearch(AdHocClassRetriever.java:602)
at com.filenet.engine.jca.impl.RequestBrokerImpl.executeSearch(RequestBrokerImpl.java:890)
at com.filenet.engine.jca.impl.RequestBrokerImpl.executeSearch(RequestBrokerImpl.java:813)
at com.filenet.engine.ejb.EngineCoreBean._executeSearch(EngineCoreBean.java:307)
at com.filenet.engine.ejb.EngineCoreBean.executeSearch(EngineCoreBean.java:273)
at com.filenet.engine.ejb.EJSLocalStatelessEngineCore_22877cb1.executeSearch(Unknown Source)
at com.filenet.engine.ejb.EngineBean.executeSearch(EngineBean.java:428)
at com.filenet.apiimpl.transport.ejbstubs.EJSRemoteStatelessEngine_2e64c374.executeSearch(Unknown Source)
at com.filenet.apiimpl.transport.ejbstubs._EJSRemoteStatelessEngine_2e64c374_Tie.executeSearch(_EJSRemoteStatelessEngine_2e64c374_Tie.java:194)
at com.filenet.apiimpl.transport.ejbstubs._EJSRemoteStatelessEngine_2e64c374_Tie._invoke(_EJSRemoteStatelessEngine_2e64c374_Tie.java:114)
at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:623)
at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:476)
at com.ibm.rmi.iiop.ORB.process(ORB.java:513)
at com.ibm.CORBA.iiop.ORB.process(ORB.java:1574)
at com.ibm.rmi.iiop.Connection.respondTo(Connection.java:2845)
at com.ibm.rmi.iiop.Connection.doWork(Connection.java:2718)
at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:63)
at com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:118)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1604)
[] 000004fc WSRdbManagedC W DSRA1300E: Feature is not implemented: javax.sql.PooledConnection.addStatementEventListener
[] 000004fa WSRdbManagedC W DSRA1300E: Feature is not implemented: javax.sql.PooledConnection.addStatementEventListener
[] 00000007 TimeoutManage I WTRN0006W: Transaction 0000013D4338B4DF00000002000266757C0715E2903294674B6C35D69DC9F9EB788FB6C10000013D4338B4DF00000002000266757C0715E2903294674B6C35D69DC9F9EB788FB6C100000001 has timed out after 300 seconds.
[] 00000007 TimeoutManage I WTRN0124I: When the timeout occurred the thread with which the transaction is, or was most recently, associated was ThreadORB.thread.pool : 578,5,main. The stack trace of this thread when the timeout occurred was:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:140)
com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1525)
com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:3258)
com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:4433)
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:386)
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:338)
com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4026)
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1416)
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:185)
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:160)
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.execute(SQLServerPreparedStatement.java:320)
com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.pmiExecute(WSJdbcPreparedStatement.java:961)
com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.execute(WSJdbcPreparedStatement.java:637)
com.filenet.engine.dbpersist.DBExecutionElement.execute(DBExecutionElement.java:218)
com.filenet.engine.dbpersist.DBExecutionContext.getNextResult(DBExecutionContext.java:131)
com.filenet.engine.dbpersist.DBStatementList.getNextResult(DBStatementList.java:587)
com.filenet.engine.persist.DBStatementList2.getNextResult(DBStatementList2.java:86)
com.filenet.engine.dbpersist.DBStatementList.checkNextResult(DBStatementList.java:605)
com.filenet.engine.persist.DBStatementList2.checkNextResult(DBStatementList2.java:118)
com.filenet.engine.persist.IndependentPersister.postExecuteChange(IndependentPersister.java:524)
com.filenet.engine.persist.SubscribablePersister.postExecuteChange(SubscribablePersister.java:325)
com.filenet.engine.persist.ReplicablePersister.postExecuteChange(ReplicablePersister.java:127)
com.filenet.engine.persist.WithContentPersister.postExecuteChange(WithContentPersister.java:523)
com.filenet.engine.persist.VersionablePersister.postExecuteChange(VersionablePersister.java:249)
com.filenet.engine.persist.IndependentPersister.executeChangeWork(IndependentPersister.java:454)
com.filenet.engine.persist.IndependentPersister.executeChange(IndependentPersister.java:242)
com.filenet.engine.persist.SubscribablePersister.executeChange(SubscribablePersister.java:155)
com.filenet.engine.jca.impl.RequestBrokerImpl.executeChanges(RequestBrokerImpl.java:1244)
com.filenet.engine.jca.impl.RequestBrokerImpl.executeChanges(RequestBrokerImpl.java:1126)
com.filenet.engine.ejb.EngineCoreBean._executeChanges(EngineCoreBean.java:601)
com.filenet.engine.ejb.EngineCoreBean.executeChanges(EngineCoreBean.java:567)
com.filenet.engine.ejb.EJSLocalStatelessEngineCore_22877cb1.executeChanges(Unknown Source)
com.filenet.engine.ejb.EngineBean.executeChanges(EngineBean.java:668)
com.filenet.apiimpl.transport.ejbstubs.EJSRemoteStatelessEngine_2e64c374.executeChanges(Unknown Source)
com.filenet.apiimpl.transport.ejbstubs._EJSRemoteStatelessEngine_2e64c374_Tie.executeChanges(_EJSRemoteStatelessEngine_2e64c374_Tie.java:214)
com.filenet.apiimpl.transport.ejbstubs._EJSRemoteStatelessEngine_2e64c374_Tie._invoke(_EJSRemoteStatelessEngine_2e64c374_Tie.java:118)
com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:623)
com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:476)
com.ibm.rmi.iiop.ORB.process(ORB.java:513)
com.ibm.CORBA.iiop.ORB.process(ORB.java:1574)
com.ibm.rmi.iiop.Connection.respondTo(Connection.java:2845)
com.ibm.rmi.iiop.Connection.doWork(Connection.java:2718)
com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:63)
com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:118)
com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1604)
Updated on 2013-03-08T11:49:56Z at 2013-03-08T11:49:56Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    693 Posts
    ACCEPTED ANSWER

    Re: Websphere FileNet Content Engine JVMs hangs

    ‏2013-03-07T19:10:01Z  in response to ahm.m
    Perhaps you've already done this, but I'll just state the most obvious thing ... have you checked that MS SQL JDBC driver version is among those supported by P8?
    • SystemAdmin
      SystemAdmin
      693 Posts
      ACCEPTED ANSWER

      Re: Websphere FileNet Content Engine JVMs hangs

      ‏2013-03-08T10:57:08Z  in response to SystemAdmin
      We're on AIX and Oracle but were facing similar issues. (WSJdbcPreparedStatement.execute)

      In our case it turned out to be caused by stored searches on Workplace that had a sort order set under format results.
      Users would sometimes hit the search button without completing any of the search criteria.
      Search is pretty generic giving them various properties to search on, therefore we had no mandatory fields.
      The sort caused a full table scan on docversion.
      What made it worse is that the search button stays active, so when the user saw no result coming back, he/she would repeatedly click on the search button, each click causing another full table scan.
      We had one instance where around 30 off these full table scans were happening, tying up all the connections in the Oracle connection pool, effectively bringing down the CMs until we could get the DBAs to kill the sessions.

      Even after we removed the sort we still had this happening every now and then.
      Turned out to be caused by users that saved the URL to the old search from the browser's address bar.
      That URL contains the object_id of the superseded version.
      To overcome that we had to delete the old versions via FEM and educate the users to use the stored searches we setup for them under My Workplace.

      So in short, check your database for long running sql statements.

      Good luck, hope you find the cause.
  • SystemAdmin
    SystemAdmin
    693 Posts
    ACCEPTED ANSWER

    Re: Websphere FileNet Content Engine JVMs hangs

    ‏2013-03-08T11:49:56Z  in response to ahm.m
    You should also check the general capacity of the database. This really looks like undercapacity either from long running threads or just too much system load. The thread hang as there is no capacity on the database to accept new connections thus they have to wait, after a while the thread times out and gets killed by websphere.