Topic
  • 3 replies
  • Latest Post - ‏2010-05-12T18:26:35Z by Chris.D.Johnson
rajivpa78
rajivpa78
13 Posts

Pinned topic Out of Memory on grid client ?

‏2010-05-07T20:18:04Z |
Hi
We are doing some benchmarking on eXtreme Scale.

We have a servlet which calls invokes a grid client, which in turn connects to grid, gets the ObjectMap and retrieves data. Using Jmeter we are generating 100+ concurrent requests to the servlet.

We are consistently getting out of memory error on the server that hosts the grid client in a webmodule. This happend when we run the 100+ user test 3-4 times.

Any ideas why out of memory happens on client ? pasting the stack trace for reference.

5/7/10 12:10:05:774 EDT 00000188 ServletWrappe E SRVE0068E: Uncaught exception thrown in one of the service methods of the servlet: PortalEntitlementGrid. Exception thrown : java.lang.OutOfMemoryError
at java.lang.Object.clone(Native Method)
at java.lang.Class.getDeclaredMethod(Class.java:582)
at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1444)
at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:109)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:511)
at java.security.AccessController.doPrivileged(AccessController.java:192)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:490)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:386)
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1588)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1502)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1735)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:378)
at com.ibm.ws.objectgrid.util.SerializationHelper.readNullableObject(SerializationHelper.java:79)
at com.ibm.ws.objectgrid.config.cluster.ServerImpl.readExternal(ServerImpl.java:312)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1791)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1670)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1335)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1755)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1755)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:378)
at com.ibm.ws.objectgrid.objectMapping.ObjectBytes.bytesToObject(ObjectBytes.java:133)
at com.ibm.ws.objectgrid.naming.LocationServiceFactory.bootstrap(LocationServiceFactory.java:194)
at com.ibm.ws.objectgrid.ObjectGridManagerImpl.connect(ObjectGridManagerImpl.java:2191)
at com.poc.controller.ObjectGridHelper.getObjectGrid(ObjectGridHelper.java:72)
at com.poc.controller.ObjectGridHelper.getPortalResourcesForUser(ObjectGridHelper.java:27)
at com.poc.controller.PortalEntitlementGrid.doGet(PortalEntitlementGrid.java:34)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1146)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:592)
at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:525)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:90)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:751)
at com.ibm.ws.wswebcontainer.WebContainer.handleRequest(WebContainer.java:1478)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:126)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:458)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:387)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:267)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1037)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:644)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1818)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:136)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:196)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:751)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:881)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1497)
Updated on 2010-05-12T18:26:35Z at 2010-05-12T18:26:35Z by Chris.D.Johnson
  • Chris.D.Johnson
    Chris.D.Johnson
    480 Posts

    Re: Out of Memory on grid client ?

    ‏2010-05-07T21:09:47Z  
    So you are:
    1. ccc = objectGridManager.connect(...)
    2. og = objectGridManager.getObjectGrid("mygrid")
    3. use the grid
    4. exit

    You need to make sure you destroy the grid or disconnect the ClientClusterContext by one of the two methods:
    • ObjectGridManager.disconnect(ccc)
    • ObjectGrid.destroy()

    I would avoid connecting each time, however, and instead cache the ObjectGrid client instance in a variable in your servlet. The ObjectGrid serves as your client connection and is thread safe. I would also suggest using a pool of Session objects, since getSession() is a bit expensive.

    We typically suggest that you do the connect/getObjectGrid in your Servlet.initialize() method or equivalent to avoid a sync around it. Then in your Servlet.destroy() method, call ObjectGrid.destroy() to clean up teh connection.

    Then for your Session pool, simply use a get/use/return pattern for each session. If your pool is empty, then just create a new session by calling ObjectGrid.getSession().
  • rajivpa78
    rajivpa78
    13 Posts

    Re: Out of Memory on grid client ?

    ‏2010-05-12T17:55:34Z  
    So you are:
    1. ccc = objectGridManager.connect(...)
    2. og = objectGridManager.getObjectGrid("mygrid")
    3. use the grid
    4. exit

    You need to make sure you destroy the grid or disconnect the ClientClusterContext by one of the two methods:
    • ObjectGridManager.disconnect(ccc)
    • ObjectGrid.destroy()

    I would avoid connecting each time, however, and instead cache the ObjectGrid client instance in a variable in your servlet. The ObjectGrid serves as your client connection and is thread safe. I would also suggest using a pool of Session objects, since getSession() is a bit expensive.

    We typically suggest that you do the connect/getObjectGrid in your Servlet.initialize() method or equivalent to avoid a sync around it. Then in your Servlet.destroy() method, call ObjectGrid.destroy() to clean up teh connection.

    Then for your Session pool, simply use a get/use/return pattern for each session. If your pool is empty, then just create a new session by calling ObjectGrid.getSession().
    Thanks, not connecting everytime improved the performance numbers drastically.

    I also implemented SessionPool. I kep around 20 sessions in the pool, for 300 concurrent threads, I find that lot of threads spent time waiting for the session object to be returned to pool, which increased the response time slightly as compared to having no Session Pool, am I doing it right ?
  • Chris.D.Johnson
    Chris.D.Johnson
    480 Posts

    Re: Out of Memory on grid client ?

    ‏2010-05-12T18:26:35Z  
    • rajivpa78
    • ‏2010-05-12T17:55:34Z
    Thanks, not connecting everytime improved the performance numbers drastically.

    I also implemented SessionPool. I kep around 20 sessions in the pool, for 300 concurrent threads, I find that lot of threads spent time waiting for the session object to be returned to pool, which increased the response time slightly as compared to having no Session Pool, am I doing it right ?
    Increase your pool size to 300 and change your implementation to no longer wait for a session to be returned. If the pool is empty, just return a new session. If your pool is full upon return, just throw away the session.

    Another alternative is to cache the Session in a ThreadLocal variable. This works well if you only plan on using one session at a time for that thread.