DSI exceptions: Is it a false alarm or something that needs my attention
Introduction
First and foremost, it is important not to ignore error messages and to pay attention to them when you see them. There are many different types of DSI messages especially if you include WebSphere eXtreme Scale (WXS). WXS is used to distribute the runtime across a cluster of servers. DSI and WXS use exceptions to draw attention to a disruption of its servers that run in a cluster. Exceptions often indicate that a system is not in a stable condition, but they can also occur during a transitional period from which the system can recover. When a system does recover, some exceptions can be identified as false positive errors (wrongly indicating that a problem is present). In some cases, the exceptions can be used to help you to avoid an unrecoverable situation in the future.
The severity of the exceptions can be grouped into the following categories. The severity of the exceptions increase the lower down the list you go.
- Indicates a given condition exists, when it does not
- Worth following up on to know why it occurred and how to prevent it
- Temporary issue that might repeat and cause problems later on
- Likely configuration and/or start up error
1. Indicates a given condition exists, when it does not
1.1. Server versions are incompatible
com.ibm.websphere.objectgrid.ObjectGridRuntimeException "Stack Dump = com.ibm.websphere.objectgrid.ObjectGridRuntimeException: DefaultDomain:com.ibm.ia:iaMaps:2 com.ibm.websphere.objectgrid.TargetNotAvailableException: Retrying the request expired, the current timeout is set to 30000 (ms). DefaultDomain:com.ibm.ia:iaMaps:2 forward: true num fowards:70 lastIP: xxxx.xxxx.com last request started at 05/02/18 08:23:20.200 GST expire time set at 05/02/18 08:23:49.691 GST Number retries 2 Number retries 2 at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessageInternal(XIOClientCoreMessageHandler.java:963) at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessage(XIOClientCoreMessageHandler.java:144) at com.ibm.ws.objectgrid.client.CommonClientCoreMessageHandler.sendGetRequest(CommonClientCoreMessageHandler.java:229) at com.ibm.ws.objectgrid.client.RemoteTransactionCallbackImpl.mapGet(RemoteTransactionCallbackImpl.java:1953) at com.ibm.ws.objectgrid.client.RemoteCacheLoader.get(RemoteCacheLoader.java:497) at com.ibm.ws.objectgrid.map.BaseMap.getRaw(BaseMap.java:4844) at com.ibm.ws.objectgrid.DiffMap.lookForKey(DiffMap.java:1380) at com.ibm.ws.objectgrid.DiffMap.get(DiffMap.java:978) at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:451) at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:411) at com.ibm.ia.cluster.ClusterBean.getStoredCompatibilityMode(ClusterBean.java:1520) at com.ibm.ia.cluster.ClusterBean.doCompatibilityCheck(ClusterBean.java:1463) at com.ibm.ia.cluster.ClusterBean.checkCompatibility(ClusterBean.java:1422) at com.ibm.ia.cluster.ClusterBean.doTokenUpdate(ClusterBean.java:900) at com.ibm.ia.cluster.ClusterBean.transactionEnd(ClusterBean.java:814) at com.ibm.ws.objectgrid.OGEventListenerHandler.fireTransactionEnd(OGEventListenerHandler.java:539) at com.ibm.ws.objectgrid.SessionImpl.afterCompletion(SessionImpl.java:2555) at com.ibm.ws.objectgrid.SessionImpl.commit(SessionImpl.java:2298) at com.ibm.ia.cluster.ClusterBean$1.call(ClusterBean.java:641) at com.ibm.ia.cluster.ClusterBean$1.call(ClusterBean.java:1) at java.util.concurrent.FutureTask.run(FutureTask.java:267) at com.ibm.ia.cluster.ClusterBean.setServerToken(ClusterBean.java:657) at com.ibm.ia.cluster.ClusterBean.activate(ClusterBean.java:225) at Proxy993908b1_b421_462b_9e5c_4d9aac3ceee5.activate(Unknown Source) at Proxycc18703d_0d76_4185_acee_a9d5d92083f0.activate(Unknown Source) at com.ibm.ia.wxs.WXSServerManager.shardActivated(WXSServerManager.java:161) at Proxyfda1ec54_3147_4020_9526_1b33e485cf02.shardActivated(Unknown Source) at com.ibm.ia.wxs.GridServiceManagerCallback.shardActivated(GridServiceManagerCallback.java:87) at com.ibm.ws.objectgrid.OGEventListenerHandler.fireActivated(OGEventListenerHandler.java:587) at com.ibm.ws.objectgrid.ObjectGridImpl.activate(ObjectGridImpl.java:1215) at com.ibm.ws.objectgrid.ObjectGridImpl.activate(ObjectGridImpl.java:1003) at com.ibm.ws.objectgrid.replication.PrimaryShardImpl.activateObjectGrid(PrimaryShardImpl.java:6767) at com.ibm.ws.objectgrid.replication.PrimaryShardActor.activateObjectGrid(PrimaryShardActor.java:441) at com.ibm.ws.objectgrid.replication.ReplicatedPartition.becomePrimary(ReplicatedPartition.java:1381) at com.ibm.ws.objectgrid.replication.ReplicatedPartition.processPrimaryWork(ReplicatedPartition.java:641) at com.ibm.ws.objectgrid.replication.ReplicatedPartition.handleMigrationEvent(ReplicatedPartition.java:542) at com.ibm.ws.objectgrid.server.impl.ShardImpl.handleRoleTransition(ShardImpl.java:338) at com.ibm.ws.objectgrid.server.impl.ShardActor.handleRoleTransition(ShardActor.java:1097) at com.ibm.ws.objectgrid.server.container.ObjectGridContainerImpl.doContainerWork(ObjectGridContainerImpl.java:889) at com.ibm.ws.objectgrid.server.container.ContainerActor$2.run(ContainerActor.java:345) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618) at com.ibm.ws.objectgrid.thread.XSThreadPool$Worker.run(XSThreadPool.java:309) Caused by: com.ibm.websphere.objectgrid.TargetNotAvailableException: Retrying the request expired, the current timeout is set to 30000 (ms). DefaultDomain:com.ibm.ia:iaMaps:2 forward: true num fowards:70 lastIP: xxxx.xxxx.com last request started at 05/02/18 08:23:20.200 GST expire time set at 05/02/18 08:23:49.691 GST Number retries 2 at com.ibm.ws.objectgrid.client.XIOClientCoreMessageHandler.sendMessageInternal(XIOClientCoreMessageHandler.java:744) ... 42 more
Diagnosis: This runtime exception reports a failed operation to check that all the cluster members have a compatible DSI version. The exception is known to occur when a system is temporarily overused or there is a temporary issue. It occurs after the commit of the transaction, so it does not affect the consistency of the DSI data. If the check fails and throws an exception when the system temporarily prevents its success, the DSI cluster is not impacted.
1.2. Unable to calculate a global aggregate
com.ibm.websphere.objectgrid.ObjectGridException "Stack Dump = com.ibm.websphere.objectgrid.ObjectGridException: java.lang.InterruptedException at com.ibm.ws.objectgrid.map.GetNextKeyQueue.atomicWait(GetNextKeyQueue.java:348) at com.ibm.ws.objectgrid.map.GetNextKeyQueue.next(GetNextKeyQueue.java:236) at com.ibm.ws.objectgrid.map.BaseMap.getNextKey(BaseMap.java:5611) at com.ibm.ws.objectgrid.DiffMap.getNextKey(DiffMap.java:3116) at com.ibm.ws.objectgrid.ObjectMapImpl.getNextKey(ObjectMapImpl.java:2099) at com.ibm.ia.wxs.WXSTransactionStore.take(WXSTransactionStore.java:1300) at com.ibm.ia.runtime.PartialResultQueueService.take(PartialResultQueueService.java:95) at com.ibm.ia.runtime.PartialResultQueueService.take(PartialResultQueueService.java:1) at com.ibm.ia.runtime.BaseGetNextPartialResult.run(BaseGetNextPartialResult.java:100) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:267) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618) at java.lang.Thread.run(Thread.java:785) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1040) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1329) at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:410) at com.ibm.ws.objectgrid.map.GetNextKeyQueue.atomicWait(GetNextKeyQueue.java:336) ... 13 more
Diagnosis: It is a temporary failure, DSI will retry the operation later. If the global aggregate feature of DSI is not used, it will not have any impact.
1.3. Shard unable to serve requests
com.ibm.ws.xsspi.xio.exception.TransportException "Stack Dump = com.ibm.ws.xsspi.xio.exception.TransportException$Transient [originating=10.50.88.166:7118;exid=0]: Partition 10 for com.ibm.ia in map set iaMaps is temporarily not taking new transactions on xxxx.xxxx.com-ContainerSrvr03_C-13 at sun.reflect.GeneratedConstructorAccessor74.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57) at java.lang.reflect.Constructor.newInstance(Constructor.java:436) at com.ibm.ws.xsspi.xio.exception.XIOExceptionFactory.createException(XIOExceptionFactory.java:58) at com.ibm.ws.xsspi.xio.exception.ServerExceptionFactory.createTransient(ServerExceptionFactory.java:156) at com.ibm.ws.objectgrid.server.impl.ServerCoreEventProcessor.createInitialReadWriteTransaction(ServerCoreEventProcessor.java:734) at com.ibm.ws.objectgrid.server.impl.ServerCoreEventProcessor.processGetNextRequest(ServerCoreEventProcessor.java:1359) at com.ibm.ws.objectgrid.server.impl.ServerCoreEventProcessor.processClientServerRequest(ServerCoreEventProcessor.java:2679) at com.ibm.ws.objectgrid.server.impl.ShardImpl.processMessage(ShardImpl.java:1469) at com.ibm.ws.objectgrid.server.impl.ShardActor.handleContainerMessage(ShardActor.java:513) at com.ibm.ws.objectgrid.server.impl.ShardActor.receive(ShardActor.java:339) at com.ibm.ws.xs.xio.actor.impl.XIOReferableImpl.dispatch(XIOReferableImpl.java:110) at com.ibm.ws.xsspi.xio.actor.XIORegistry.sendToTarget(XIORegistry.java:987) at com.ibm.ws.xs.xio.transport.channel.XIORegistryRunnable.run(XIORegistryRunnable.java:88) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618) at com.ibm.ws.objectgrid.thread.XSThreadPool$Worker.run(XSThreadPool.java:309)
Diagnosis: This exception indicates that the client reached the shard, but the shard is in a transitional state, and cannot serve the request. The shard did not complete work to promote to the primary shard. A shard might be experiencing a demotion or the shard might be in the process of stopping.
2. Worth following up on to know why it occurred and how to prevent it
2.1. Event submission rate exceeded capacity
com.ibm.ia.common.EventCapacityExceededException: CWMBD9692E: Event submission rate {0} reached system capacity of {1}
at com.ibm.ia.gateway.impl.SolutionGatewayImpl.submitEvent(SolutionGatewayImpl.java:287)
at com.ibm.ia.gateway.impl.SolutionGatewayImpl.submit(SolutionGatewayImpl.java:204)
at com.ibm.ia.connectivity.inbound.impl.ReusableChannel.submitEvent(ReusableChannel.java:322)
at com.ibm.ia.connectivity.inbound.impl.xml.XMLChannel.submit(XMLChannel.java:185)
at com.ibm.ia.connectivity.inbound.impl.xml.XMLChannel.submit(XMLChannel.java:159)
at com.ibm.ia.connectivity.inbound.impl.GatewayChannelImpl.submit(GatewayChannelImpl.java:94)
at com.ibm.ia.connectivity.inbound.jms.InboundJmsEndpointMDB.onMessage(InboundJmsEndpointMDB.java:197)
at com.ibm.ia.connectivity.inbound.jms.MDBProxycncInboundEndPoint_dfc99253.onMessage(MDBProxycncInboundEndPoint_dfc99253.java)
Diagnosis: This exception reports that a DSI protection mechanism is triggered. The role of the mechanism is to slow the flow of incoming events until the grid is able to process at full capacity again. This might indicate that the number and speed at which events are received is too high for the system in general. Check the configuration of the servers and adjust accordingly.
3. Temporary issue that might cause problems later on
3.1. The grid is not online
com.ibm.websphere.objectgrid.ClientServerLoaderException "Stack Dump = com.ibm.websphere.objectgrid.ClientServerLoaderException: Client Services - Received exception from remote server at com.ibm.ws.objectgrid.client.RemoteTransactionCallbackImpl.processGetResponse(RemoteTransactionCallbackImpl.java:1384) at com.ibm.ws.objectgrid.client.RemoteTransactionCallbackImpl.getGetResponseWithAuthenticationRetry(RemoteTransactionCallbackImpl.java:1971) at com.ibm.ws.objectgrid.client.RemoteTransactionCallbackImpl.mapGet(RemoteTransactionCallbackImpl.java:1957) at com.ibm.ws.objectgrid.client.RemoteCacheLoader.get(RemoteCacheLoader.java:497) at com.ibm.ws.objectgrid.map.BaseMap.getRaw(BaseMap.java:4847) at com.ibm.ws.objectgrid.DiffMap.lookForKey(DiffMap.java:1388) at com.ibm.ws.objectgrid.DiffMap.get(DiffMap.java:978) at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:451) at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:411) at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:458) at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:412) at com.ibm.ia.runtime.stores.TransactionStoreCache.get(TransactionStoreCache.java:103) at com.ibm.ia.runtime.EntityStore.getEntity(EntityStore.java:235) at com.ibm.ia.runtime.EntityStore.getEntity(EntityStore.java:197) at com.ibm.ia.runtime.EntityStore.getEntity(EntityStore.java:190) at com.ibm.ia.runtime.CISEntityResolver.resolveEntity(CISEntityResolver.java:116) at com.ibm.ia.runtime.CISEntityResolver.getEntity(CISEntityResolver.java:106) at com.ibm.ia.runtime.AgentEntityResolver.getEntity(AgentEntityResolver.java:82) at com.ibm.ia.runtime.ObjectEntityResolverImpl.resolve(ObjectEntityResolverImpl.java:140) at com.ibm.ia.runtime.AgentObjectEntityResolver.resolve(AgentObjectEntityResolver.java:62) at com.ibm.ia.model.impl.RelationshipImpl.resolve(RelationshipImpl.java:56) at com.ibm.ia.model.impl.Utilities.resolve(Utilities.java:29) ... at com.ibm.rules.engine.rete.runtime.state.AbstractConditionExecEnv.evaluateBooleanMethod(AbstractConditionExecEnv.java:75) at com.ibm.rules.engine.rete.runtime.network.impl.IlrAbstractGeneratorJoinNode.evaluateDiscTest(IlrAbstractGeneratorJoinNode.java:471) at com.ibm.rules.engine.rete.runtime.network.impl.IlrAbstractGeneratorJoinNode.updateInternal(IlrAbstractGeneratorJoinNode.java:179) at com.ibm.rules.engine.rete.runtime.network.impl.IlrAbstractGeneratorJoinNode.update(IlrAbstractGeneratorJoinNode.java:211) at com.ibm.rules.engine.rete.runtime.network.impl.IlrAbstractTupleMemNode.notifyUpdate(IlrAbstractTupleMemNode.java:96) at com.ibm.rules.engine.rete.runtime.network.impl.IlrStandardJoinNode.updateInternal(IlrStandardJoinNode.java:138) at com.ibm.rules.engine.rete.runtime.network.impl.IlrStandardJoinNode.update(IlrStandardJoinNode.java:162) at com.ibm.rules.engine.rete.runtime.network.impl.IlrAbstractTupleMemNode.notifyUpdate(IlrAbstractTupleMemNode.java:96) at com.ibm.rules.engine.rete.runtime.network.impl.IlrStandardAlphaNode.update(IlrStandardAlphaNode.java:87) at com.ibm.rules.engine.rete.runtime.network.impl.IlrAbstractObjectMemNode.notifyUpdate(IlrAbstractObjectMemNode.java:74) at com.ibm.rules.engine.rete.runtime.network.impl.IlrAbstractGeneratorDiscNode.updateInternal(IlrAbstractGeneratorDiscNode.java:256) at com.ibm.rules.engine.rete.runtime.network.impl.IlrAbstractGeneratorDiscNode.updateEngineData(IlrAbstractGeneratorDiscNode.java:304) at com.ibm.rules.engine.rete.runtime.network.impl.IlrStandardWorkingMemoryNode.updateEngineData(IlrStandardWorkingMemoryNode.java:119) at com.ibm.rules.engine.rete.runtime.AbstractReteEngine.updateData(AbstractReteEngine.java:339) ... at com.ibm.rules.engine.rete.runtime.state.AbstractBodyExecEnv.executeBody(AbstractBodyExecEnv.java:82) at com.ibm.rules.engine.rete.runtime.state.AbstractNetworkState.executeRule(AbstractNetworkState.java:204) at com.ibm.rules.engine.rete.runtime.AbstractReteEngine.executeOneRule(AbstractReteEngine.java:470) at com.ibm.rules.engine.rete.runtime.AbstractReteEngine.stepOneRule(AbstractReteEngine.java:632) at com.ibm.ia.engine.runtime.impl.EventEngineImpl.executeRules(EventEngineImpl.java:467) at com.ibm.ia.engine.runtime.impl.EventEngineImpl.insertEventInternal(EventEngineImpl.java:422) at com.ibm.ia.engine.runtime.impl.EventEngineImpl.insertEvent(EventEngineImpl.java:314) at com.ibm.ia.agent.BaseRuleAgent.insertEvent(BaseRuleAgent.java:187) at com.ibm.ia.agent.RuleAgent.processEvent2(RuleAgent.java:253) at com.ibm.ia.agent.RuleAgent.processEvent(RuleAgent.java:224) at com.ibm.ia.agent.RuleAgent.doProcess(RuleAgent.java:119) at com.ibm.ia.agent.BaseRuleAgent.process(BaseRuleAgent.java:124) at com.ibm.ia.runtime.EventProcessor.executeTrackedService(EventProcessor.java:1403) at com.ibm.ia.runtime.EventProcessor.onEventDoAgents(EventProcessor.java:1107) at com.ibm.ia.runtime.EventProcessor.onEvent(EventProcessor.java:758) at com.ibm.ia.runtime.EventQueueMgr.post(EventQueueMgr.java:220) at com.ibm.ia.runtime.EventQueueMgr.post(EventQueueMgr.java:168) at com.ibm.ia.runtime.GatewayListener.processEvent(GatewayListener.java:79) at com.ibm.ia.runtime.OutboundEventDispatcher.dispatch(OutboundEventDispatcher.java:239) at Proxy398b549f_02dc_4448_951c_6f0f0780fbc1.dispatch(Unknown Source) at com.ibm.ia.runtime.AgentContext.dispatch(AgentContext.java:229) at com.ibm.ia.common.BaseComponent.processOutboundEvents(BaseComponent.java:351) at com.ibm.ia.agent.Agent.processOutboundEvents(Agent.java:535) at com.ibm.ia.runtime.EventProcessor.onEventDoTimeTriggers(EventProcessor.java:1229) at com.ibm.ia.runtime.EventProcessor.onEvent(EventProcessor.java:754) at com.ibm.ia.runtime.EventProcessor.run(EventProcessor.java:383) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:267) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618) at java.lang.Thread.run(Thread.java:785) Caused by: com.ibm.websphere.objectgrid.TransactionException: com.ibm.websphere.objectgrid.TransactionException:transaction rolled backed by another thread at RemovedByServerSerialization.RemovedByServerSerialization(RemovedByServerSerialization:0) Caused by: com.ibm.websphere.objectgrid.TransactionException: com.ibm.websphere.objectgrid.TransactionException:The current state of the ObjectGrid instance caused the transaction to end. ... 1 more and com.ibm.ia.common.AgentException "Stack Dump = com.ibm.ia.common.AgentException: com.ibm.ia.common.AgentException: com.ibm.ia.common.AgentException: com.ibm.ia.common.StoreException: com.ibm.websphere.objectgrid.TransactionException: transaction rolled backed by another thread at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngine(FactoryEngineStore.java:1093) at com.ibm.ia.agent.RuleAgent.retrieveEngine(RuleAgent.java:467) at com.ibm.ia.agent.RuleAgent.processEvent(RuleAgent.java:186) at com.ibm.ia.agent.RuleAgent.doProcess(RuleAgent.java:119) at com.ibm.ia.agent.BaseRuleAgent.process(BaseRuleAgent.java:124) at com.ibm.ia.runtime.EventProcessor.executeTrackedService(EventProcessor.java:1403) at com.ibm.ia.runtime.EventProcessor.onEventDoAgents(EventProcessor.java:1107) at com.ibm.ia.runtime.EventProcessor.onEvent(EventProcessor.java:758) at com.ibm.ia.runtime.EventProcessor.run(EventProcessor.java:383) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:267) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618) at java.lang.Thread.run(Thread.java:785) Caused by: com.ibm.ia.common.AgentException: com.ibm.ia.common.AgentException: com.ibm.ia.common.StoreException: com.ibm.websphere.objectgrid.TransactionException: transaction rolled backed by another thread at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngineState(FactoryEngineStore.java:794) at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngine(FactoryEngineStore.java:937) ... 13 more Caused by: com.ibm.ia.common.AgentException: com.ibm.ia.common.StoreException: com.ibm.websphere.objectgrid.TransactionException: transaction rolled backed by another thread at com.ibm.ia.runtime.stores.MementoStore.getMemento(MementoStore.java:241) at com.ibm.ia.runtime.stores.FactoryEngineStore.getEngineState(FactoryEngineStore.java:801) at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngineState(FactoryEngineStore.java:769) ... 14 more Caused by: com.ibm.ia.common.StoreException: com.ibm.websphere.objectgrid.TransactionException: transaction rolled backed by another thread at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:468) at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:412) at com.ibm.ia.runtime.stores.TransactionStoreCache.getObject(TransactionStoreCache.java:85) at com.ibm.ia.runtime.stores.MementoStore.getMemento(MementoStore.java:238) ... 16 more Caused by: com.ibm.websphere.objectgrid.TransactionException: transaction rolled backed by another thread at com.ibm.ws.objectgrid.SessionImpl.mapPostInvoke(SessionImpl.java:3724) at com.ibm.ws.objectgrid.ObjectMapImpl.postInvoke(ObjectMapImpl.java:883) at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:476) at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:411) at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:458) ... 19 more Caused by: com.ibm.websphere.objectgrid.TransactionException: The current state of the ObjectGrid instance caused the transaction to end. at com.ibm.ws.objectgrid.transaction.TransactionManager.processAsyncRollback(TransactionManager.java:606) at com.ibm.ws.objectgrid.transaction.TransactionManager.rollbackOutlierBucket(TransactionManager.java:558) at com.ibm.ws.objectgrid.transaction.TransactionManager.rollbackNextBucket(TransactionManager.java:531) at com.ibm.ws.objectgrid.transaction.TransactionManager.drainAll(TransactionManager.java:780) at com.ibm.ws.objectgrid.transaction.TransactionManager.destroy(TransactionManager.java:399) at com.ibm.ws.objectgrid.ObjectGridImpl.destroy(ObjectGridImpl.java:3315) at com.ibm.ws.objectgrid.catalog.placement.balance.disk.DiskCatalogUtil.destroyGrid(DiskCatalogUtil.java:176) at com.ibm.ws.objectgrid.server.impl.ShardImpl.destroy(ShardImpl.java:800) at com.ibm.ws.objectgrid.server.impl.ShardActor.destroy(ShardActor.java:1104) at com.ibm.ws.objectgrid.server.container.ObjectGridContainerImpl.deactivate(ObjectGridContainerImpl.java:1343) at com.ibm.ws.objectgrid.server.container.ObjectGridContainerImpl.destroyShard(ObjectGridContainerImpl.java:2394) at com.ibm.ws.objectgrid.server.container.ObjectGridContainerImpl.destroyShard(ObjectGridContainerImpl.java:2367) at com.ibm.ws.objectgrid.server.container.ContainerActor.destroyShard(ContainerActor.java:406) at com.ibm.ws.objectgrid.replication.PrimaryShardImpl.destroyShard(PrimaryShardImpl.java:2353) at com.ibm.ws.objectgrid.replication.PrimaryShardImpl.abdicateLeadershipRelayReplicas3Common(PrimaryShardImpl.java:2686) at com.ibm.ws.objectgrid.replication.PrimaryShardActor.abdicateLeadershipRelayReplicas3Common(PrimaryShardActor.java:308) at com.ibm.ws.objectgrid.replication.PrimaryShardActor.abdicateLeadershipReceive(PrimaryShardActor.java:163) at com.ibm.ws.objectgrid.replication.PrimaryShardActor.receive(PrimaryShardActor.java:115) at com.ibm.ws.xs.xio.actor.impl.XIOReferableImpl.dispatch(XIOReferableImpl.java:110) at com.ibm.ws.xsspi.xio.actor.XIORegistry.sendToTarget(XIORegistry.java:987) at com.ibm.ws.xs.xio.transport.channel.XIORegistryRunnable.run(XIORegistryRunnable.java:88) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618) at com.ibm.ws.objectgrid.thread.XSThreadPool$Worker.run(XSThreadPool.java:309)
Diagnosis: The grid is not online. It typically occurs when a shutdown is in progress or in case of major system issue.
4. Likely configuration and/or start up error
4.1. Failed to obtain a database connection
java.lang.reflect.InvocationTargetException "Stack Dump = java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor162.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) at java.lang.reflect.Method.invoke(Method.java:507) at com.ibm.ws.xs.osgi.service.XSServiceHandler.invoke(XSServiceHandler.java:87) at com.ibm.ws.xs.osgi.service.ObjectGridServiceHandler.invoke(ObjectGridServiceHandler.java:76) at com.sun.proxy.$Proxy60.populateTXSlots(Unknown Source) at com.ibm.ia.persistence.impl.SQLPersistenceManager.get(SQLPersistenceManager.java:644) at sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) at java.lang.reflect.Method.invoke(Method.java:507) at com.ibm.ws.xs.osgi.service.XSServiceHandler.invoke(XSServiceHandler.java:87) at com.ibm.ws.xs.osgi.service.BackingMapServiceHandler.invoke(BackingMapServiceHandler.java:74) at com.sun.proxy.$Proxy427.get(Unknown Source) at com.ibm.ws.objectgrid.map.BaseMap.getRaw(BaseMap.java:4844) at com.ibm.ws.objectgrid.DiffMap.lookForKey(DiffMap.java:1380) at com.ibm.ws.objectgrid.DiffMap.get(DiffMap.java:978) at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:451) at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:411) at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:458) at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:412) at com.ibm.ia.runtime.stores.TransactionStoreCache.getObject(TransactionStoreCache.java:85) at com.ibm.ia.runtime.stores.MementoStore.getMemento(MementoStore.java:238) at com.ibm.ia.runtime.stores.FactoryEngineStore.getEngineState(FactoryEngineStore.java:801) at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngineState(FactoryEngineStore.java:769) at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngine(FactoryEngineStore.java:937) at com.ibm.ia.agent.RuleAgent.retrieveEngine(RuleAgent.java:467) at com.ibm.ia.agent.RuleAgent.processEvent(RuleAgent.java:186) at com.ibm.ia.agent.RuleAgent.doProcess(RuleAgent.java:119) at com.ibm.ia.agent.BaseRuleAgent.process(BaseRuleAgent.java:124) at com.ibm.ia.runtime.EventProcessor.executeTrackedService(EventProcessor.java:1403) at com.ibm.ia.runtime.EventProcessor.onEventDoAgents(EventProcessor.java:1107) at com.ibm.ia.runtime.EventProcessor.onEvent(EventProcessor.java:758) at com.ibm.ia.runtime.EventProcessor.run(EventProcessor.java:383) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:267) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618) at java.lang.Thread.run(Thread.java:785) Caused by: com.ibm.ia.common.PersistenceException: com.ibm.ia.common.PersistenceSQLException: Connection not available, Timed out waiting for 180001 at com.ibm.ia.persistence.impl.SQLCallbackImpl.populateTXSlots(SQLCallbackImpl.java:160) at com.ibm.ia.persistence.impl.SQLCallbackImpl.populateTXSlots(SQLCallbackImpl.java:105) ... 38 more Caused by: com.ibm.ia.common.PersistenceSQLException: Connection not available, Timed out waiting for 180001 ... 40 more
and
com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException "Stack Dump = com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 180001 at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1442) at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3050) at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2335) at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1121) at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:688) at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:354) at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:325) at com.ibm.ia.persistence.impl.SQLCallbackImpl.putConnection(SQLCallbackImpl.java:117) at com.ibm.ia.persistence.impl.SQLCallbackImpl.populateTXSlots(SQLCallbackImpl.java:147) at com.ibm.ia.persistence.impl.SQLCallbackImpl.populateTXSlots(SQLCallbackImpl.java:105) at sun.reflect.GeneratedMethodAccessor162.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) at java.lang.reflect.Method.invoke(Method.java:507) at com.ibm.ws.xs.osgi.service.XSServiceHandler.invoke(XSServiceHandler.java:87) at com.ibm.ws.xs.osgi.service.ObjectGridServiceHandler.invoke(ObjectGridServiceHandler.java:76) at com.sun.proxy.$Proxy60.populateTXSlots(Unknown Source) at com.ibm.ia.persistence.impl.SQLPersistenceManager.get(SQLPersistenceManager.java:644) at sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) at java.lang.reflect.Method.invoke(Method.java:507) at com.ibm.ws.xs.osgi.service.XSServiceHandler.invoke(XSServiceHandler.java:87) at com.ibm.ws.xs.osgi.service.BackingMapServiceHandler.invoke(BackingMapServiceHandler.java:74) at com.sun.proxy.$Proxy427.get(Unknown Source) at com.ibm.ws.objectgrid.map.BaseMap.getRaw(BaseMap.java:4844) at com.ibm.ws.objectgrid.DiffMap.lookForKey(DiffMap.java:1380) at com.ibm.ws.objectgrid.DiffMap.get(DiffMap.java:978) at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:451) at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:411) at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:458) at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:412) at com.ibm.ia.runtime.stores.TransactionStoreCache.getObject(TransactionStoreCache.java:85) at com.ibm.ia.runtime.stores.MementoStore.getMemento(MementoStore.java:238) at com.ibm.ia.runtime.stores.FactoryEngineStore.getEngineState(FactoryEngineStore.java:801) at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngineState(FactoryEngineStore.java:769) at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngine(FactoryEngineStore.java:937) at com.ibm.ia.agent.RuleAgent.retrieveEngine(RuleAgent.java:467) at com.ibm.ia.agent.RuleAgent.processEvent(RuleAgent.java:186) at com.ibm.ia.agent.RuleAgent.doProcess(RuleAgent.java:119) at com.ibm.ia.agent.BaseRuleAgent.process(BaseRuleAgent.java:124) at com.ibm.ia.runtime.EventProcessor.executeTrackedService(EventProcessor.java:1403) at com.ibm.ia.runtime.EventProcessor.onEventDoAgents(EventProcessor.java:1107) at com.ibm.ia.runtime.EventProcessor.onEvent(EventProcessor.java:758) at com.ibm.ia.runtime.EventProcessor.run(EventProcessor.java:383) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:267) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618) at java.lang.Thread.run(Thread.java:785)
and
java.sql.SQLTransientConnectionException "Stack Dump = java.sql.SQLTransientConnectionException: Connection not available, Timed out waiting for 180001 at com.ibm.ws.rsadapter.AdapterUtil.toSQLException(AdapterUtil.java:1100) at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:368) at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:325) at com.ibm.ia.persistence.impl.SQLCallbackImpl.putConnection(SQLCallbackImpl.java:117) at com.ibm.ia.persistence.impl.SQLCallbackImpl.populateTXSlots(SQLCallbackImpl.java:147) at com.ibm.ia.persistence.impl.SQLCallbackImpl.populateTXSlots(SQLCallbackImpl.java:105) at sun.reflect.GeneratedMethodAccessor162.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) at java.lang.reflect.Method.invoke(Method.java:507) at com.ibm.ws.xs.osgi.service.XSServiceHandler.invoke(XSServiceHandler.java:87) at com.ibm.ws.xs.osgi.service.ObjectGridServiceHandler.invoke(ObjectGridServiceHandler.java:76) at com.sun.proxy.$Proxy60.populateTXSlots(Unknown Source) at com.ibm.ia.persistence.impl.SQLPersistenceManager.get(SQLPersistenceManager.java:644) at sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) at java.lang.reflect.Method.invoke(Method.java:507) at com.ibm.ws.xs.osgi.service.XSServiceHandler.invoke(XSServiceHandler.java:87) at com.ibm.ws.xs.osgi.service.BackingMapServiceHandler.invoke(BackingMapServiceHandler.java:74) at com.sun.proxy.$Proxy427.get(Unknown Source) at com.ibm.ws.objectgrid.map.BaseMap.getRaw(BaseMap.java:4844) at com.ibm.ws.objectgrid.DiffMap.lookForKey(DiffMap.java:1380) at com.ibm.ws.objectgrid.DiffMap.get(DiffMap.java:978) at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:451) at com.ibm.ws.objectgrid.ObjectMapImpl.get(ObjectMapImpl.java:411) at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:458) at com.ibm.ia.wxs.WXSTransactionStore.get(WXSTransactionStore.java:412) at com.ibm.ia.runtime.stores.TransactionStoreCache.getObject(TransactionStoreCache.java:85) at com.ibm.ia.runtime.stores.MementoStore.getMemento(MementoStore.java:238) at com.ibm.ia.runtime.stores.FactoryEngineStore.getEngineState(FactoryEngineStore.java:801) at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngineState(FactoryEngineStore.java:769) at com.ibm.ia.runtime.stores.FactoryEngineStore.retrieveEngine(FactoryEngineStore.java:937) at com.ibm.ia.agent.RuleAgent.retrieveEngine(RuleAgent.java:467) at com.ibm.ia.agent.RuleAgent.processEvent(RuleAgent.java:186) at com.ibm.ia.agent.RuleAgent.doProcess(RuleAgent.java:119) at com.ibm.ia.agent.BaseRuleAgent.process(BaseRuleAgent.java:124) at com.ibm.ia.runtime.EventProcessor.executeTrackedService(EventProcessor.java:1403) at com.ibm.ia.runtime.EventProcessor.onEventDoAgents(EventProcessor.java:1107) at com.ibm.ia.runtime.EventProcessor.onEvent(EventProcessor.java:758) at com.ibm.ia.runtime.EventProcessor.run(EventProcessor.java:383) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:267) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618) at java.lang.Thread.run(Thread.java:785) Caused by: com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 180001 at com.ibm.ws.rsadapter.AdapterUtil.toSQLException(AdapterUtil.java:1097) ... 43 more Caused by: com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 180001 at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1442) at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3050) at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2335) at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1121) at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:688) at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:354) ... 42 more
Diagnosis: These exceptions are indicating that DSI has failed to obtain a database connection.
It might occur for multiple reasons: Insufficient connection pool size, database slowness, or network issues.
4.2. Outbound servers started before runtime servers
com.ibm.ia.connectivity.outbound.sender.PermanentEventSenderException: Unable to access configuration for solution Solution_Name at com.ibm.ia.connectivity.outbound.sender.impl.SolutionSenders.getConfigurationSource(SolutionSenders.java:359) at com.ibm.ia.connectivity.outbound.sender.impl.SolutionSenders.getEventSender(SolutionSenders.java:80) at com.ibm.ia.connectivity.outbound.sender.impl.EventSenderManagerImpl.getEventSender(EventSenderManagerImpl.java:362) at com.ibm.ia.connectivity.outbound.async.impl.AsynchronousWorkerThread.run(AsynchronousWorkerThread.java:263) Caused by: com.ibm.ia.connectivity.config.ConfigurationException: Unable to access solution configuration from the grid: Solution_Name at com.ibm.ia.connectivity.config.grid.impl.ConfiguredGridConnectionManager.getSolutionConfiguration(ConfiguredGridConnectionManager.java:680) at com.ibm.ia.connectivity.config.grid.impl.ConfiguredGridConnectionManager.getConfigurationSource(ConfiguredGridConnectionManager.java:635) at com.ibm.ia.connectivity.outbound.sender.impl.SolutionSenders.getConfigurationSource(SolutionSenders.java:355) ... 3 more Caused by: com.ibm.ia.gateway.GridConnectionException: Grid is not available at com.ibm.ia.wxs.client.impl.ObjectGridEnvironment.getSolutionStore(ObjectGridEnvironment.java:305) at com.ibm.ia.gateway.spi.ClientConnectionImpl.getSolutionConfiguration(ClientConnectionImpl.java:169) at com.ibm.ia.connectivity.config.grid.impl.ConfiguredGridConnectionManager.getSolutionConfiguration(ConfiguredGridConnectionManager.java:668) ... 5 more
Diagnosis: This exception reports an outbound server is started before all of the runtime servers are ready. This is a transient condition until the grid is ready. The error disappears when all of the servers are started. Make sure that you start the servers in the following order: Catalog servers, Runtime servers, Outbound servers, Inbound servers.
Leave a Reply