Topic
3 replies Latest Post - ‏2013-12-10T12:41:10Z by smart_dev
Dev_Dhoot
Dev_Dhoot
42 Posts
ACCEPTED ANSWER

Pinned topic Container JVM process ended because a replacement JVM has started.?

‏2013-10-10T11:03:33Z |

Hi,

I am using WXS 8.6.0.3. I have experienced something unusual from WXS. The container server got shut down abruptly and never came up.

==

Executed the following command: [/www/IBM/WebSphere/eXtremeScale8.6/java/jre/bin/java, -D__xsAutoRestartPPID__=25086, -D__xsAutoRestarts__=1, -Xoptionsfile=/www/IBM/WebSphere/eXtremeScale8.6/java/jre/lib/amd64/compressedrefs/options.default, -Xlockword:mode=default,noLockword=java/lang/String,noLockword=java/util/MapEntry,noLockword=java/util/HashMap$Entry,noLockword=org/apache/harmony/luni/util/ModifiedMap$Entry,noLockword=java/util/Hashtable$Entry,noLockword=java/lang/invoke/MethodType,noLockword=java/lang/invoke/MethodHandle,noLockword=java/lang/invoke/CollectHandle,noLockword=java/lang/invoke/ConstructorHandle,noLockword=java/lang/invoke/ConvertHandle,noLockword=java/lang/invoke/ArgumentConversionHandle,noLockword=java/lang/invoke/AsTypeHandle,noLockword=java/lang/invoke/ExplicitCastHandle,noLockword=java/lang/invoke/FilterReturnHandle,noLockword=java/lang/invoke/DirectHandle,noLockword=java/lang/invoke/ReceiverBoundHandle,noLockword=java/lang/invoke/DynamicInvokerHandle,noLockword=java/lang/invoke/FieldHandle,noLockword=java/lang/invoke/FieldGetterHandle,noLockword=java/lang/invoke/FieldSetterHandle,noLockword=java/lang/invoke/StaticFieldGetterHandle,noLockword=java/lang/invoke/StaticFieldSetterHandle,noLockword=java/lang/invoke/IndirectHandle,noLockword=java/lang/invoke/InterfaceHandle,noLockword=java/lang/invoke/VirtualHandle,noLockword=java/lang/invoke/InvokeExactHandle,noLockword=java/lang/invoke/InvokeGenericHandle,noLockword=java/lang/invoke/VarargsCollectorHandle,noLockwor

 

[10/9/13 23:19:40:833 CDT] 000008b5 UnixRestart   I   CWOBJ1224I: The JVM process is ending because a replacement JVM has started.

Below is the detailed description of the logs.

==

23:12:07:979 CDT] 000000e3 LRUEvictor2   W   CWOBJ0002W: ObjectGrid component, LRUEvictor2, is ignoring an unexpected exception: com.ibm.websphere.objectgrid.ObjectGridRuntimeException: java.lang.ClassNotFoundException: CWOBJ6324E: Class definition is null for object com.***.lines.domain.LandAssets 
at com.ibm.ws.objectgrid.plugins.io.dataobject.values.ValueDataImpl.getObject(ValueDataImpl.java:336)
at com.ibm.ws.objectgrid.DiffMapValue.getCurrentValue(DiffMapValue.java:1050)
at com.ibm.ws.objectgrid.DiffMapValue.getCurrentValue(DiffMapValue.java:939)
at com.ibm.softmach.LRUEvictor2.apply(LRUEvictor2.java:95)
at com.ibm.ws.objectgrid.map.BaseMap.afterCompletion(BaseMap.java:2962)
at com.ibm.ws.objectgrid.SessionImpl.afterCompletion(SessionImpl.java:2470)
at com.ibm.ws.objectgrid.SessionImpl.commit(SessionImpl.java:2239)
at com.ibm.ws.objectgrid.server.impl.ServerCoreEventProcessor.processLogSequence(ServerCoreEventProcessor.java:1983)
at com.ibm.ws.objectgrid.server.impl.ServerCoreEventProcessor.processReadWriteTransactionRequest(ServerCoreEventProcessor.java:1668)
at com.ibm.ws.objectgrid.server.impl.ServerCoreEventProcessor.processClientServerRequest(ServerCoreEventProcessor.java:2566)
at com.ibm.ws.objectgrid.server.impl.ShardImpl.processMessage(ShardImpl.java:1498)
at com.ibm.ws.objectgrid.server.impl.ShardActor.handleContainerMessage(ShardActor.java:469)
at com.ibm.ws.objectgrid.server.impl.ShardActor.receive(ShardActor.java:323)
at com.ibm.ws.xsspi.xio.actor.XIOReferable.dispatch(XIOReferable.java:114)
at com.ibm.ws.xsspi.xio.actor.XIORegistry.sendToTarget(XIORegistry.java:968)
at com.ibm.ws.xs.xio.transport.channel.XIORegistryRunnable.run(XIORegistryRunnable.java:84)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1156)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:626)
at com.ibm.ws.objectgrid.thread.XSThreadPool$Worker.run(XSThreadPool.java:309)
Caused by: com.ibm.websphere.objectgrid.ObjectGridRuntimeException: java.lang.ClassNotFoundException: CWOBJ6324E: Class definition is null for object com.***.lines.domain.LandAssets 
at com.ibm.ws.objectgrid.xdf.XDFDescriptor.createNewObjectForSerialization(XDFDescriptor.java:1077)
at com.ibm.ws.objectgrid.xdf.serializers.GenericClassSerializer.deserializeObject(GenericClassSerializer.java:191)
at com.ibm.ws.objectgrid.xdf.XDFSerializerPlugin.inflateDataObject(XDFSerializerPlugin.java:319)
at com.ibm.ws.objectgrid.plugins.io.dataobject.values.ValueDataImpl.getObject(ValueDataImpl.java:328)
... 18 more
Caused by: java.lang.ClassNotFoundException: CWOBJ6324E: Class definition is null for object com.***.lines.domain.LandAssets 
... 22 more
.
[10/9/13 23:12:09:053 CDT] 00000084 LRUEvictor2   W   CWOBJ0002W: ObjectGrid component, LRUEvictor2, is ignoring an unexpected exception: com.ibm.websphere.objectgrid.ObjectGridRuntimeException: java.lang.ClassNotFoundException: CWOBJ6324E: Class definition is null for object com.***.lines.domain.LandAssets 
at com.ibm.ws.objectgrid.plugins.io.dataobject.values.ValueDataImpl.getObject(ValueDataImpl.java:336)
at com.ibm.ws.objectgrid.DiffMapValue.getCurrentValue(DiffMapValue.java:1050)
at com.ibm.ws.objectgrid.DiffMapValue.getCurrentValue(DiffMapValue.java:939)
at com.ibm.softmach.LRUEvictor2.apply(LRUEvictor2.java:95)
at com.ibm.ws.objectgrid.map.BaseMap.afterCompletion(BaseMap.java:2962)
at com.ibm.ws.objectgrid.SessionImpl.afterCompletion(SessionImpl.java:2470)
at com.ibm.ws.objectgrid.SessionImpl.commit(SessionImpl.java:2239)
at com.ibm.ws.objectgrid.replication.SynchronousReplicaRevisionShardImpl.commit(SynchronousReplicaRevisionShardImpl.java:802)
at com.ibm.ws.objectgrid.replication.SynchronousReplicaRevisionShardImpl.commit(SynchronousReplicaRevisionShardImpl.java:704)
at com.ibm.ws.objectgrid.replication.SynchronousReplicaRevisionShardActor.commitParamsReceive(SynchronousReplicaRevisionShardActor.java:512)
at com.ibm.ws.objectgrid.replication.SynchronousReplicaRevisionShardActor.receive(SynchronousReplicaRevisionShardActor.java:198)
at com.ibm.ws.xsspi.xio.actor.XIOReferable.dispatch(XIOReferable.java:114)
at com.ibm.ws.xsspi.xio.actor.XIORegistry.sendToTarget(XIORegistry.java:968)
at com.ibm.ws.xs.xio.transport.channel.XIORegistryRunnable.run(XIORegistryRunnable.java:84)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1156)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:626)
at com.ibm.ws.objectgrid.thread.XSThreadPool$Worker.run(XSThreadPool.java:309)
Caused by: com.ibm.websphere.objectgrid.ObjectGridRuntimeException: java.lang.ClassNotFoundException: CWOBJ6324E: Class definition is null for object com.***.lines.domain.LandAssets 
at com.ibm.ws.objectgrid.xdf.XDFDescriptor.createNewObjectForSerialization(XDFDescriptor.java:1077)
at com.ibm.ws.objectgrid.xdf.serializers.GenericClassSerializer.deserializeObject(GenericClassSerializer.java:191)
at com.ibm.ws.objectgrid.xdf.XDFSerializerPlugin.inflateDataObject(XDFSerializerPlugin.java:319)
at com.ibm.ws.objectgrid.plugins.io.dataobject.values.ValueDataImpl.getObject(ValueDataImpl.java:328)
... 16 more
Caused by: java.lang.ClassNotFoundException: CWOBJ6324E: Class definition is null for object com.***.lines.domain.LandAssets 
... 20 more
.
 
.
[10/9/13 23:14:05:147 CDT] 0000000f XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 34377 ms.
[10/9/13 23:14:05:147 CDT] 0000001e XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 34377 ms.
[10/9/13 23:14:05:148 CDT] 0000001d XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 34379 ms.
[10/9/13 23:14:05:147 CDT] 00000018 XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 34377 ms.
[10/9/13 23:14:05:149 CDT] 00000048 HAControllerI W   HMGR0152W: CPU Starvation detected. Current thread scheduling delay is 6 seconds.
[10/9/13 23:14:05:149 CDT] 0000001f XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 34379 ms.
[10/9/13 23:14:05:148 CDT] 00000020 XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 34396 ms.
[10/9/13 23:14:05:149 CDT] 00000032 XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 34379 ms.
[10/9/13 23:14:05:148 CDT] 00000055 XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 34379 ms.
[10/9/13 23:14:05:147 CDT] 00000033 XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 34395 ms.
[10/9/13 23:14:31:647 CDT] 00000072 SynchronousRe I   CWOBJ1524I: Replica listener azLWEB_SESSIONS:SessionMapSet:0 must register again with the primary. Reason: Replica was disconnected from primary on 86cond4Ldap_C-19 for an unknown length of time and must be reregistered to restart replication
[10/9/13 23:14:31:990 CDT] 000000da SynchronousRe I   CWOBJ1526I: Replica azLWEB_SESSIONS:SessionMapSet:0:xsastats_SessionMapSet_grid entering peer mode after 0.321 seconds, replicating from primary on 86cond4Ldap_C-19
[10/9/13 23:14:31:990 CDT] 000000da SynchronousRe I   CWOBJ1526I: Replica azLWEB_SESSIONS:SessionMapSet:0:xsastats_SessionMapSet_map entering peer mode after 0.321 seconds, replicating from primary on 86cond4Ldap_C-19
[10/9/13 23:14:31:991 CDT] 000000da SynchronousRe I   CWOBJ1526I: Replica azLWEB_SESSIONS:SessionMapSet:0:objectgridSessionMetadata entering peer mode after 0.322 seconds, replicating from primary on 86cond4Ldap_C-19
[10/9/13 23:14:31:991 CDT] 000000da SynchronousRe I   CWOBJ1526I: Replica azLWEB_SESSIONS:SessionMapSet:0:xsastats_info_SessionMapSet entering peer mode after 0.322 seconds, replicating from primary on 86cond4Ldap_C-19
[10/9/13 23:14:31:991 CDT] 000000da SynchronousRe I   CWOBJ1526I: Replica azLWEB_SESSIONS:SessionMapSet:0:objectgridSessionAttribute entering peer mode after 0.322 seconds, replicating from primary on 86cond4Ldap_C-19
[10/9/13 23:14:31:991 CDT] 000000da SynchronousRe I   CWOBJ1526I: Replica azLWEB_SESSIONS:SessionMapSet:0:objectgridSessionTTLAttributeEvicted entering peer mode after 0.322 seconds, replicating from primary on 86cond4Ldap_C-19
[10/9/13 23:14:31:991 CDT] 000000da SynchronousRe I   CWOBJ1526I: Replica azLWEB_SESSIONS:SessionMapSet:0:objectgridSessionAttributeEvicted entering peer mode after 0.322 seconds, replicating from primary on 86cond4Ldap_C-19
[10/9/13 23:14:31:991 CDT] 000000da SynchronousRe I   CWOBJ1526I: Replica azLWEB_SESSIONS:SessionMapSet:0:objectgridSessionTTLAttributeEvicted/mmsTest entering peer mode after 0.322 seconds, replicating from primary on 86cond4Ldap_C-19
[10/9/13 23:14:32:327 CDT] 0000009f ClusterStore  I   CWOBJ1132I: An updated routing entry for domain:grid:epoch DefaultDomain:azLWEB_SESSIONS:1381378471999 was obtained from the catalog server.
[10/9/13 23:16:49:702 CDT] 00000048 HAControllerI W   HMGR0152W: CPU Starvation detected. Current thread scheduling delay is 44 seconds.
[10/9/13 23:16:49:705 CDT] 00000055 XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 52013 ms.
[10/9/13 23:16:49:704 CDT] 00000020 XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 52012 ms.
[10/9/13 23:16:49:706 CDT] 0000000f XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 52014 ms.
[10/9/13 23:16:49:704 CDT] 0000001f XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 52012 ms.
[10/9/13 23:16:49:706 CDT] 0000001e XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 52014 ms.
[10/9/13 23:16:49:705 CDT] 00000033 XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 52013 ms.
[10/9/13 23:16:49:706 CDT] 00000018 XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 52014 ms.
[10/9/13 23:16:49:705 CDT] 00000032 XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 52013 ms.
[10/9/13 23:16:49:704 CDT] 0000001d XSThreadPool  W   CWOBJ7852W: Thread starvation detected.  Thread scheduling delay is 52012 ms.
[10/9/13 23:16:49:710 CDT] 0000086b ApplicationMo W   DCSV0004W: DCS Stack ObjectGridCoreGroup at Member aaa.sdde.***.com:56829: Did not receive adequate CPU time slice. Last known CPU usage time at 23:15:35:155 CDT. Inactivity duration was 44 seconds. 
[
10/9/13 23:19:40:783 CDT] 00000046 DiscoveryRcv  W   DCSV1115W: DCS Stack ObjectGridCoreGroup at Member aaa.sdde.***.com:56829: Member aaa.sdde.***.com:42228 connection  was closed. Member will  be removed from view. DCS connection status is Discovery|Ptp, receiver closed.
[10/9/13 23:19:40:783 CDT] 00000038 RmmPtpGroup   W   DCSV1115W: DCS Stack ObjectGridCoreGroup at Member aaa.sdde.***.com:56829: Member aaa.sdde.***.com:42228 connection  was closed. Member will  be removed from view. DCS connection status is View|Ptp, receiver closed.
[10/9/13 23:19:40:791 CDT] 00000037 RoleViewLeade I   DCSV8053I: DCS Stack ObjectGridCoreGroup at Member aaa.sdde.***.com:56829: View change in process. Excluded members are [aaa.sdde.***.com:42228].
[10/9/13 23:19:40:797 CDT] 00000038 VSyncAlgo1    I   DCSV2004I: DCS Stack ObjectGridCoreGroup at Member aaa.sdde.***.com:56829: View synchronization completed successfully. The View Identifier is (2:0.aaa.sdde.***.com:42228). The internal details are None.
[10/9/13 23:19:40:801 CDT] 0000086b ViewReceiver  I   DCSV1033I: DCS Stack ObjectGridCoreGroup at Member aaa.sdde.***.com:56829: Confirmed all new view members in view identifier (3:0.aaa.sdde.***.com:56829). View channel type is View|Ptp.
[10/9/13 23:19:40:804 CDT] 00000037 CoordinatorIm I   HMGR0218I: A new core group view has been installed. The core group is ObjectGridCoreGroup. The view identifier is (3:0.aaa.sdde.***.com:56829). The number of members in the new view is 1.
[10/9/13 23:19:40:806 CDT] 00000037 CoreGroupMemb I   DCSV8050I: DCS Stack ObjectGridCoreGroup at Member aaa.sdde.***.com:56829: New view installed, identifier (3:0.aaa.sdde.***.com:56829), view size is 1 (AV=1, CD=1, CN=1, DF=2)
[10/9/13 23:19:40:811 CDT] 00000037 CoordinatorIm I   HMGR0206I: The Coordinator is an Active Coordinator for core group ObjectGridCoreGroup. The active coordinator set is [aaa.sdde.***.com:56829].
[10/9/13 23:19:40:812 CDT] 00000049 PeerManager   I   CWOBJ8601I: The PeerManager found peers of size 1.
[10/9/13 23:19:40:812 CDT] 00000049 ServerAgent   I   CWOBJ1772I: The high availability (HA) manager and Distribution and Consistency Services (DCS) have notified eXtreme Scale that the list of servers that are running in this core group has changed to aaa.sdde.***.com:56829.
[10/9/13 23:19:40:813 CDT] 00000049 ServerAgent   I   CWOBJ1770I: This process is now the core group leader for the ObjectGridCoreGroup core group.
[10/9/13 23:19:40:816 CDT] 000008b5 ServerAgent   I   CWOBJ1227I: The server was disconnected from the primary catalog server, which will be restarted to reconnect.
[10/9/13 23:19:40:832 CDT] 000008b5 UnixRestart   I   Executed the following command: [/www/IBM/WebSphere/eXtremeScale8.6/java/jre/bin/java, -D__xsAutoRestartPPID__=25086, -D__xsAutoRestarts__=1, -Xoptionsfile=/www/IBM/WebSphere/eXtremeScale8.6/java/jre/lib/amd64/compressedrefs/options.default, -Xlockword:mode=default,noLockword=java/lang/String,noLockword=java/util/MapEntry,noLockword=java/util/HashMap$Entry,noLockword=org/apache/harmony/luni/util/ModifiedMap$Entry,noLockword=java/util/Hashtable$Entry,noLockword=java/lang/invoke/MethodType,noLockword=java/lang/invoke/MethodHandle,noLockword=java/lang/invoke/CollectHandle,noLockword=java/lang/invoke/ConstructorHandle,noLockword=java/lang/invoke/ConvertHandle,noLockword=java/lang/invoke/ArgumentConversionHandle,noLockword=java/lang/invoke/AsTypeHandle,noLockword=java/lang/invoke/ExplicitCastHandle,noLockword=java/lang/invoke/FilterReturnHandle,noLockword=java/lang/invoke/DirectHandle,noLockword=java/lang/invoke/ReceiverBoundHandle,noLockword=java/lang/invoke/DynamicInvokerHandle,noLockword=java/lang/invoke/FieldHandle,noLockword=java/lang/invoke/FieldGetterHandle,noLockword=java/lang/invoke/FieldSetterHandle,noLockword=java/lang/invoke/StaticFieldGetterHandle,noLockword=java/lang/invoke/StaticFieldSetterHandle,noLockword=java/lang/invoke/IndirectHandle,noLockword=java/lang/invoke/InterfaceHandle,noLockword=java/lang/invoke/VirtualHandle,noLockword=java/lang/invoke/InvokeExactHandle,noLockword=java/lang/invoke/InvokeGenericHandle,noLockword=java/lang/invoke/VarargsCollectorHandle,noLockword=java/lang/invoke/ThunkTuple, -Xjcl:jclse7b_26, -Dcom.ibm.oti.vm.bootstrap.library.path=/www/IBM/WebSphere/eXtremeScale8.6/java/jre/lib/amd64/compressedrefs:/www/IBM/WebSphere/eXtremeScale8.6/java/jre/lib/amd64, -Dsun.boot.library.path=/www/IBM/WebSphere/eXtremeScale8.6/java/jre/lib/amd64/compressedrefs:/www/IBM/WebSphere/eXtremeScale8.6/java/jre/lib/amd64, -Djava.library.path=/www/IBM/WebSphere/eXtremeScale8.6/java/jre/lib/amd64/compressedrefs:/www/IBM/WebSphere/eXtremeScale8.6/java/jre/lib/amd64:/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/native::/usr/lib, -Djava.home=/www/IBM/WebSphere/eXtremeScale8.6/java/jre, -Djava.ext.dirs=/www/IBM/WebSphere/eXtremeScale8.6/java/jre/lib/ext, -Duser.dir=/wxs/devl/config/merge***, -Djava.runtime.version=pxa6470sr4fp1ifx-20130624_01 (SR4 FP1+IV40575+IV42295+IV37797), -Djava.class.path=., -DXS_TEST_HINT=true, -Djava.class.path=/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/../java/../lib/tools.jar:/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/properties:/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/lib/objectgrid.jar:/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/session/lib/sessionobjectgrid.jar:/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/dynacache/lib/wxsdynacache.jar:/wxs/shared/evictors/***Evictor86.jar, -Djava.endorsed.dirs=/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/lib/endorsed:/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/../java/jre/lib/endorsed, -Xmx4G, -Djava.security.auth.login.config=/wxs/devl/security/xsldap.config, -Djava.security.auth.policy=/wxs/devl/security/jaasAuth.xml, -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager, -Djava.util.logging.configureByServer=true, -Dobjectgrid.home=/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid, -Dsun.java.command=com.ibm.ws.objectgrid.InitializationService 86cond6Ldap -transport XIO -objectGridFile /wxs/devl/config/merge***/objectGrid.xml -deploymentPolicyFile /wxs/devl/config/merge***/objectGridDeployment.xml -serverProps /wxs/devl/config/86cond6LdapServer.properties -clusterSecurityFile /wxs/devl/security/globalSecurityLdap.xml, -Dsun.java.launcher=SUN_STANDARD, -Dsun.java.launcher.pid=25086, -cp, /www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/../java/../lib/tools.jar:/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/properties:/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/lib/objectgrid.jar:/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/session/lib/sessionobjectgrid.jar:/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/dynacache/lib/wxsdynacache.jar:/wxs/shared/evictors/***Evictor86.jar, com.ibm.ws.objectgrid.InitializationService, 86cond6Ldap, -transport, XIO, -objectGridFile, /wxs/devl/config/merge***/objectGrid.xml, -deploymentPolicyFile, /wxs/devl/config/merge***/objectGridDeployment.xml, -serverProps, /wxs/devl/config/86cond6LdapServer.properties, -clusterSecurityFile, /wxs/devl/security/globalSecurityLdap.xml]
[10/9/13 23:19:40:833 CDT] 000008b5 UnixRestart   I   CWOBJ1224I: The JVM process is ending because a replacement JVM has started.
[10/9/13 23:19:40:834 CDT] 00000053 JVMShutdownHo I   CWOBJ2523I: Stopping this catalog or container server due to an external signal from the operating system.
[10/9/13 23:19:40:834 CDT] 00000053 ServerImpl    I   CWOBJ2510I: Stopping ObjectGrid server 86cond6Ldap.
[10/9/13 23:19:40:836 CDT] 000008b7 RestartStream I   Starting restart process stream gobbler.
[10/9/13 23:19:43:804 CDT] 000008b7 RestartStream I   CWOBJ1230I: During restart, the child Java virtual machine (JVM) produced the following output: [10/9/13 23:19:43:790 CDT] 00000001 ObjectGridRAS I   CWOBJ2507I: Trace specification is set to *=all=disabled.
[10/9/13 23:19:43:814 CDT] 000008b7 RestartStream I   CWOBJ1230I: During restart, the child Java virtual machine (JVM) produced the following output: [10/9/13 23:19:43:812 CDT] 00000001 ManagerAdmin  I   TRAS0018I: The trace state has changed. The new trace state is *=info.
************ Start Display Current Environment ************
WebSphere WebSphere eXtreme Scale v7.0.3 (8.6.0.3) [cf31329.18064852] running with process name 86cond6Ldap and process id 1580
Host Operating System is Linux, version 2.6.32-220.7.1.el6.x86_64
Java version = 1.7.0, Java Compiler = j9jit26, Java VM name = IBM J9 VM
was.install.root = null
user.install.root = null
Java Home = /www/IBM/WebSphere/eXtremeScale8.6/java/jre
ws.ext.dirs = null
Classpath = /www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/../java/../lib/tools.jar:/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/properties:/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/lib/objectgrid.jar:/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/session/lib/sessionobjectgrid.jar:/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/dynacache/lib/wxsdynacache.jar:/wxs/shared/evictors/***Evictor86.jar
Java Library path = /www/IBM/WebSphere/eXtremeScale8.6/java/jre/lib/amd64/compressedrefs:/www/IBM/WebSphere/eXtremeScale8.6/java/jre/lib/amd64:/www/IBM/WebSphere/eXtremeScale8.6/ObjectGrid/native::/usr/lib
************* End Display Current Environment *************
[10/9/13 23:19:44:032 CDT] 000008b7 RestartStream I   CWOBJ1230I: During restart, the child Java virtual machine (JVM) produced the following output: [10/9/13 23:19:44:030 CDT] 00000001 RuntimeInfo   I   CWOBJ0903I: The internal version of WebSphere eXtreme Scale is v7.0.3 (8.6.0.3) [cf31329.18064852].
[10/9/13 23:19:44:030 CDT] 00000001 RuntimeInfo   I   CWOBJ0903I: The internal version of WebSphere eXtreme Scale is v7.0.3 (8.6.0.3) [cf31329.18064852].
[10/9/13 23:19:44:037 CDT] 000008b7 RestartStream I   CWOBJ1230I: During restart, the child Java virtual machine (JVM) produced the following output: [10/9/13 23:19:44:035 CDT] 00000001 WXSProperties I   CWOBJ0054I: The value of the "com.ibm.websphere.objectgrid.container.reconnect.block.reconnect.time" property is "30000".
[10/9/13 23:19:44:035 CDT] 00000001 WXSProperties I   CWOBJ0054I: The value of the "com.ibm.websphere.objectgrid.container.reconnect.block.reconnect.time" property is "30000".

==

 

Can anyone let me know the reason behind the above exception?

 

Thanks,

Devendra

 

  • jhanders
    jhanders
    234 Posts
    ACCEPTED ANSWER

    Re: Container JVM process ended because a replacement JVM has started.?

    ‏2013-10-10T11:22:42Z  in response to Dev_Dhoot

    There are two issues that I think you want answers to if I am not mistaken.

    The ClassNotFoundException is becuse the value class is not on the server side and your LRUEvictor2 calls LogElement.getCurrentValue.  With XDF by default that will try to inflate the value unless you specific a PluginOutputFormat annotation of RAW so that you get the SerializedValue instead of the POJO.  An example of the annotation format that you would add to your class is:

    @PluginOutputFormat(keyFormat=OutputFormat.RAW, valueFormat=OutputFormat.RAW)
     

    Regarding the restart issue.  The server restarts because the container became disconnected from the catalog server and when it was able to be seen by the catalog server again it was told to restart in order to restore the full capacity of the grid.  You see the CPU starvation messages in the logs.  Since the peer servers could not get a results from the container while it was in a CPU starvation state it told the catalog server it wasn't there any longer.  A lot of times that has to do with a long GC pause.  Some things I see is that you don't have your min heap set.  I would recommend setting it to -Xms4G to avoid heap thrashing.  I do not have verbose gc details to show if that is what is happening, but that is common.

    I hope that helps explain both issues.

    Jared Anderson

    • Dev_Dhoot
      Dev_Dhoot
      42 Posts
      ACCEPTED ANSWER

      Re: Container JVM process ended because a replacement JVM has started.?

      ‏2013-10-10T14:03:03Z  in response to jhanders

      Jared,

       

      Thanks for the reply and useful explanation :)

       

      I just want to understand one more thing here.. If the catalog server prompts the container to restart..the container hasn't restarted successfully in our case.

      I would like to confirm that in such a case the container should come up cleanly.So, why in our case it didn't came up clean as seen in the above logs. Can you please help me identify the root cause.

       

      --Devendra

    • smart_dev
      smart_dev
      54 Posts
      ACCEPTED ANSWER

      Re: Container JVM process ended because a replacement JVM has started.?

      ‏2013-12-10T12:41:10Z  in response to jhanders

      Hi Jared,

      I am also facing similar problem and have seen such exceptions in the logs & FFDC. I have configured -Xmx4G as a heap when the server is started. I can see the server is being stopped by the catalog but not able to come up clean and throwing error msg " TCPPort E TCPC0003E: TCP Channel XIOInboundTCP initialization failed. The socket bind failed for host xxxcwxs8.abc.xyz.com and port 4009. The port may already be in use"

      [12/8/13 2:35:47:715 CST] 0000a0f1 FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /logs/ffdc/conc5Ldap_925999bd_13.12.08_02.35.47.5544999250199189725953.txt com.ibm.ws.objectgrid.replication.PrimaryShardImpl.foundLostContainer 2044 

       

      The FFDC as mentioned in the msg above shows this exception.

      [12/8/13 2:35:47:557 CST]     FFDC Exception:com.ibm.ws.xsspi.xio.exception.InvalidXIORefException SourceId:com.ibm.ws.objectgrid.replication.PrimaryShardImpl.foundLostContainer ProbeId:2044 Reporter:com.ibm.ws.objectgrid.replication.PrimaryShardImpl@b8049526

      com.ibm.ws.xsspi.xio.exception.InvalidXIORefException [originating=xxx.xx.xxx.xx:4009;causedby=xxx.xx.xxx.xx:4009;reqId=8987733;exid=376]: com.ibm.ws.xsspi.xio.exception.InvalidXIORefException:Tombstone(SynchronousReplicaRevisionShardActor id: 495 index: 232)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
       

      I have attached log file and FFDC file for your reference.

      Thanks for your help.