Topic
  • 9 replies
  • Latest Post - ‏2011-06-09T03:41:40Z by SystemAdmin
SystemAdmin
SystemAdmin
462 Posts

Pinned topic Error message in derby.log for DM in ND-Cell z/OS

‏2011-05-31T16:01:21Z |
Hi,

Is running ND-cell on z/OS (drop3) seems that DM are creating a lot of errors in derby.log residing in DM's Servant region user's unix homedirectory.

Error seems to be written every 30th second, excerpt from derby.log:
2011-05-20 04:44:31.267 GMT ThreadThread-18,5,main Cleanup action starting
java.sql.SQLException: Database '/global/wikbndm/DeploymentManager/profiles/default/OTiS' not found.
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.newSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleDBNotFound(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(Unknown Source)
at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source)
at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
at org.apache.derby.jdbc.EmbeddedDriver.connect(Unknown Source)
at org.apache.openjpa.jdbc.schema.SimpleDriverDataSource.getSimpleConnection(SimpleDriverDataSource.java:96)
at org.apache.openjpa.jdbc.schema.SimpleDriverDataSource.getConnection(SimpleDriverDataSource.java:92)
at org.apache.openjpa.jdbc.schema.SimpleDriverDataSource.getConnection(SimpleDriverDataSource.java:72)
at org.apache.openjpa.lib.jdbc.DelegatingDataSource.getConnection(DelegatingDataSource.java:137)
at org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:112)
at org.apache.openjpa.jdbc.schema.DataSourceFactory.installDBDictionary(DataSourceFactory.java:239)
at org.apache.openjpa.jdbc.conf.JDBCConfigurationImpl.getConnectionFactory(JDBCConfigurationImpl.java:733)
at org.apache.openjpa.jdbc.conf.JDBCConfigurationImpl.getDataSource(JDBCConfigurationImpl.java:872)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.getDataSource(JDBCStoreManager.java:179)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.setContext(JDBCStoreManager.java:162)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.setContext(JDBCStoreManager.java:148)
at org.apache.openjpa.kernel.DelegatingStoreManager.setContext(DelegatingStoreManager.java:79)
at org.apache.openjpa.kernel.BrokerImpl.initialize(BrokerImpl.java:369)
at com.ibm.ws.persistence.kernel.WsJpaBrokerImpl.initialize(WsJpaBrokerImpl.java:271)
at org.apache.openjpa.kernel.BrokerImpl.initialize(BrokerImpl.java:323)
at com.ibm.ws.persistence.kernel.WsJpaBrokerImpl.initialize(WsJpaBrokerImpl.java:258)
at org.apache.openjpa.kernel.AbstractBrokerFactory.initializeBroker(AbstractBrokerFactory.java:229)
at org.apache.openjpa.kernel.AbstractBrokerFactory.newBroker(AbstractBrokerFactory.java:213)
at org.apache.openjpa.kernel.DelegatingBrokerFactory.newBroker(DelegatingBrokerFactory.java:156)
at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:227)
at com.ibm.ws.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:50)
at com.ibm.ws.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:35)
at com.ibm.ws.management.system.smgr.jpa.services.impl.EndpointServiceImpl.getEndpointsByType(EndpointServiceImpl.java:166)
at com.ibm.ws.management.system.smgr.host.RemoteDispatcher.processAllHosts(RemoteDispatcher.java:366)
at com.ibm.ws.management.system.smgr.host.RemoteDispatcher.access$000(RemoteDispatcher.java:70)
at com.ibm.ws.management.system.smgr.host.RemoteDispatcher$2.run(RemoteDispatcher.java:158)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:4678)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:4804)
at com.ibm.ws.management.system.smgr.host.RemoteDispatcher.run(RemoteDispatcher.java:156)
at java.lang.Thread.run(Thread.java:737)
Caused by: java.sql.SQLException: Database '/global/wikbndm/DeploymentManager/profiles/default/OTiS' not found.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
... 42 more
============= begin nested exception, level (1) ===========
java.sql.SQLException: Database '/global/wikbndm/DeploymentManager/profiles/default/OTiS' not found.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.newSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleDBNotFound(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(Unknown Source)
at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source)
at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
at org.apache.derby.jdbc.EmbeddedDriver.connect(Unknown Source)
at org.apache.openjpa.jdbc.schema.SimpleDriverDataSource.getSimpleConnection(SimpleDriverDataSource.java:96)
at org.apache.openjpa.jdbc.schema.SimpleDriverDataSource.getConnection(SimpleDriverDataSource.java:92)
at org.apache.openjpa.jdbc.schema.SimpleDriverDataSource.getConnection(SimpleDriverDataSource.java:72)
at org.apache.openjpa.lib.jdbc.DelegatingDataSource.getConnection(DelegatingDataSource.java:137)
at org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:112)
at org.apache.openjpa.jdbc.schema.DataSourceFactory.installDBDictionary(DataSourceFactory.java:239)
at org.apache.openjpa.jdbc.conf.JDBCConfigurationImpl.getConnectionFactory(JDBCConfigurationImpl.java:733)
at org.apache.openjpa.jdbc.conf.JDBCConfigurationImpl.getDataSource(JDBCConfigurationImpl.java:872)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.getDataSource(JDBCStoreManager.java:179)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.setContext(JDBCStoreManager.java:162)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.setContext(JDBCStoreManager.java:148)
at org.apache.openjpa.kernel.DelegatingStoreManager.setContext(DelegatingStoreManager.java:79)
at org.apache.openjpa.kernel.BrokerImpl.initialize(BrokerImpl.java:369)
at com.ibm.ws.persistence.kernel.WsJpaBrokerImpl.initialize(WsJpaBrokerImpl.java:271)
at org.apache.openjpa.kernel.BrokerImpl.initialize(BrokerImpl.java:323)
at com.ibm.ws.persistence.kernel.WsJpaBrokerImpl.initialize(WsJpaBrokerImpl.java:258)
at org.apache.openjpa.kernel.AbstractBrokerFactory.initializeBroker(AbstractBrokerFactory.java:229)
at org.apache.openjpa.kernel.AbstractBrokerFactory.newBroker(AbstractBrokerFactory.java:213)
at org.apache.openjpa.kernel.DelegatingBrokerFactory.newBroker(DelegatingBrokerFactory.java:156)
at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:227)
at com.ibm.ws.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:50)
at com.ibm.ws.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:35)
at com.ibm.ws.management.system.smgr.jpa.services.impl.EndpointServiceImpl.getEndpointsByType(EndpointServiceImpl.java:166)
at com.ibm.ws.management.system.smgr.host.RemoteDispatcher.processAllHosts(RemoteDispatcher.java:366)
at com.ibm.ws.management.system.smgr.host.RemoteDispatcher.access$000(RemoteDispatcher.java:70)
at com.ibm.ws.management.system.smgr.host.RemoteDispatcher$2.run(RemoteDispatcher.java:158)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:4678)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:4804)
at com.ibm.ws.management.system.smgr.host.RemoteDispatcher.run(RemoteDispatcher.java:156)
at java.lang.Thread.run(Thread.java:737)
============= end nested exception, level (1) ===========
Cleanup action completed

Tried to turn on "WAS.j2c=all"-tracing but that didn't show any activity

Have found a Derby JDBC provider at cell level, which seems to be created during customization, I've deleted that one now but haven't restarted the DN yet, so I don't know if that will stop the errors to occur.

Kind Rgds,
Peter
Updated on 2011-06-09T03:41:40Z at 2011-06-09T03:41:40Z by SystemAdmin
  • sutter
    sutter
    17 Posts

    Re: Error message in derby.log for DM in ND-Cell z/OS

    ‏2011-05-31T21:13:48Z  
    Hi PeterLar,
    From a purely JPA perspective, this error message is indicating that a datasource defined for a given persistence unit can not be found. By looking at the call stack, this does not look to be one of your application-defined datasources, but rather this looks to be a datasource used by the WebSphere Systems Mgmt code. I have no idea whether this particular database is supposed to be there or not. I have reached out to the Systems Mgmt team responsible for this area of the code. Hopefully, we'll get some type of input later today or tomorrow.

    Removing that JDBC provider definition at the cell level should not make a difference, but posting your results after your testing would still be good.

    Just curious, are you noticing other errors? Or, is just that your logs are filling up with these error messages?

    Thanks,
    Kevin
  • SystemAdmin
    SystemAdmin
    462 Posts

    Re: Error message in derby.log for DM in ND-Cell z/OS

    ‏2011-06-01T03:29:18Z  
    • sutter
    • ‏2011-05-31T21:13:48Z
    Hi PeterLar,
    From a purely JPA perspective, this error message is indicating that a datasource defined for a given persistence unit can not be found. By looking at the call stack, this does not look to be one of your application-defined datasources, but rather this looks to be a datasource used by the WebSphere Systems Mgmt code. I have no idea whether this particular database is supposed to be there or not. I have reached out to the Systems Mgmt team responsible for this area of the code. Hopefully, we'll get some type of input later today or tomorrow.

    Removing that JDBC provider definition at the cell level should not make a difference, but posting your results after your testing would still be good.

    Just curious, are you noticing other errors? Or, is just that your logs are filling up with these error messages?

    Thanks,
    Kevin
    Hi Kevin,

    No I have not noticed any odd errormessages in the joblogs for DM, neither any FFDC related to this nor any other erratic behaviour.

    In drop 1 & 2 we noticed a memory leak in DM, so we had to recycle it once a week at least, but that's seems to be gone in drop 3.

    I'll update this thread after I'v recycled the DM to see if the removal of the JDBC provider did any difference.

    Kind Rgds,
    Peter
  • sutter
    sutter
    17 Posts

    Re: Error message in derby.log for DM in ND-Cell z/OS

    ‏2011-06-01T13:36:59Z  
    Hi Kevin,

    No I have not noticed any odd errormessages in the joblogs for DM, neither any FFDC related to this nor any other erratic behaviour.

    In drop 1 & 2 we noticed a memory leak in DM, so we had to recycle it once a week at least, but that's seems to be gone in drop 3.

    I'll update this thread after I'v recycled the DM to see if the removal of the JDBC provider did any difference.

    Kind Rgds,
    Peter
    Hi Peter,
    I heard from one of the Systems Mgmt developers that there was a problem with this OTiS database in Beta 1 and 2, but supposedly this was resolved in Beta 3. I'm assuming that your reference to "drop 3" means Beta 3, so we might be looking at a different problem. He should be posting follow-up questions shortly...

    Thanks,
    Kevin
  • sutter
    sutter
    17 Posts

    Re: Error message in derby.log for DM in ND-Cell z/OS

    ‏2011-06-02T13:13:58Z  
    • sutter
    • ‏2011-06-01T13:36:59Z
    Hi Peter,
    I heard from one of the Systems Mgmt developers that there was a problem with this OTiS database in Beta 1 and 2, but supposedly this was resolved in Beta 3. I'm assuming that your reference to "drop 3" means Beta 3, so we might be looking at a different problem. He should be posting follow-up questions shortly...

    Thanks,
    Kevin
    Hi Peter,
    Me again. I mis-understood Roger's comment to me. I thought this OTiS database problem was resolved in Beta 3, but it was resolved after Beta 3... Roger also mentioned that the configuration issue that they were aware of and resolved centered around cell profiles, as opposed to dmgr profiles. The dmgr profiles are configured correctly.

    From your description, it sounds like you are having issued with a cell profile configuration. Correct? I'm just trying to verify that you are indeed running into the same issue that Roger is aware of (and has resolved for the GA).

    I'm also trying to determine if there's any type of workaround for this issue while you continue to experiment with the Beta 3 drop.

    Thanks,
    Kevin
  • SystemAdmin
    SystemAdmin
    462 Posts

    Re: Error message in derby.log for DM in ND-Cell z/OS

    ‏2011-06-02T18:24:41Z  
    • sutter
    • ‏2011-06-02T13:13:58Z
    Hi Peter,
    Me again. I mis-understood Roger's comment to me. I thought this OTiS database problem was resolved in Beta 3, but it was resolved after Beta 3... Roger also mentioned that the configuration issue that they were aware of and resolved centered around cell profiles, as opposed to dmgr profiles. The dmgr profiles are configured correctly.

    From your description, it sounds like you are having issued with a cell profile configuration. Correct? I'm just trying to verify that you are indeed running into the same issue that Roger is aware of (and has resolved for the GA).

    I'm also trying to determine if there's any type of workaround for this issue while you continue to experiment with the Beta 3 drop.

    Thanks,
    Kevin
    Hi Kevin,

    Actually I realize now that it looks like more profiles in the ND-cell has a derby.log, however only DM og seems to be that big, maybe due to that we restart Appservers more frequently.

    But the derby.log that we have uploaded is from the DM-profile and it's by far the largest of them

    Kind Rgds,
    Peter
  • rcundiff
    rcundiff
    2 Posts

    Re: Error message in derby.log for DM in ND-Cell z/OS

    ‏2011-06-03T20:02:32Z  
    Hi Kevin,

    Actually I realize now that it looks like more profiles in the ND-cell has a derby.log, however only DM og seems to be that big, maybe due to that we restart Appservers more frequently.

    But the derby.log that we have uploaded is from the DM-profile and it's by far the largest of them

    Kind Rgds,
    Peter
    Hi Peter,

    The messages you posted earlier are specific to the OTiS datasource, which is only configured and used at the job manager and deployment manager. OTiS is an internal component used only in these servers, which explains why you only see these messages in your deployment manager. The question at hand is how you created the profile? There was a problem in beta 3 with configuring OTiS correctly in cell profiles, but in deployment manager profiles we wouldn't expect to see this problem.

    Thanks,
    Roger
  • SystemAdmin
    SystemAdmin
    462 Posts

    Re: Error message in derby.log for DM in ND-Cell z/OS

    ‏2011-06-07T22:16:19Z  
    • rcundiff
    • ‏2011-06-03T20:02:32Z
    Hi Peter,

    The messages you posted earlier are specific to the OTiS datasource, which is only configured and used at the job manager and deployment manager. OTiS is an internal component used only in these servers, which explains why you only see these messages in your deployment manager. The question at hand is how you created the profile? There was a problem in beta 3 with configuring OTiS correctly in cell profiles, but in deployment manager profiles we wouldn't expect to see this problem.

    Thanks,
    Roger
    Hi Roger,

    Sorry for the delay, I'm not sure what you mean by cell profiles ?
    I've created the cell with the Profile Management tool as a "Cell (Deployment Manager and an application server)"
    However reading the derby.log I can see:
    java.sql.SQLException: Database '/global/wikbndm/DeploymentManager/profiles/default/OTiS' not found.
    Which obviously points out DM profile, but it has been created in the above mentioned "environment" in the Profile management tool in the "Websphere Customization Toolbox 8.0"

    Kind Rgds,
    Peter
  • rcundiff
    rcundiff
    2 Posts

    Re: Error message in derby.log for DM in ND-Cell z/OS

    ‏2011-06-08T18:43:01Z  
    Hi Roger,

    Sorry for the delay, I'm not sure what you mean by cell profiles ?
    I've created the cell with the Profile Management tool as a "Cell (Deployment Manager and an application server)"
    However reading the derby.log I can see:
    java.sql.SQLException: Database '/global/wikbndm/DeploymentManager/profiles/default/OTiS' not found.
    Which obviously points out DM profile, but it has been created in the above mentioned "environment" in the Profile management tool in the "Websphere Customization Toolbox 8.0"

    Kind Rgds,
    Peter
    Hi Peter,

    Thanks for confirming that you created a "Cell" via the Profile Management tool, which I had referred to as a cell profile. This means that the problem you're seeing is fixed. The Profile Management tool creates the deployment manager profile in a "Cell" in a different fashion than it does when it creates a "Deployment Manager". Only the "Cell" environment has the problem you encountered.

    Thanks,
    Roger
  • SystemAdmin
    SystemAdmin
    462 Posts

    Re: Error message in derby.log for DM in ND-Cell z/OS

    ‏2011-06-09T03:41:40Z  
    • rcundiff
    • ‏2011-06-08T18:43:01Z
    Hi Peter,

    Thanks for confirming that you created a "Cell" via the Profile Management tool, which I had referred to as a cell profile. This means that the problem you're seeing is fixed. The Profile Management tool creates the deployment manager profile in a "Cell" in a different fashion than it does when it creates a "Deployment Manager". Only the "Cell" environment has the problem you encountered.

    Thanks,
    Roger
    Thanks Roger !
    Then I won't worry about this anymore !

    Kind Rgds,
    Peter