PH39123: HTTP SESSIONS SHOULD NOT BE USING DRS' THREAD POOL.

APAR status

• Closed as program error.

Error description

• Outofmemory
  
  One instance of "com.ibm.ws.hamanager.impl.WorkQueue" loaded by
  "com.ibm.ws.runtime" occupies 2,920,624,312 (72.64%) bytes. The
  memory is accumulated in one instance of "java.util.LinkedList",
  
  The dominator tree is showing -
  
  In the inspector window under value it also show the details of
  the the drs cached objects.
  The HA manager thread which is in the path to gc roots for this
  work queue is showing as -
  
  [HUNG] HAManager.thread.pool : 1
   at java.lang.Object.wait(JI)V (Native Method)
   at java.lang.Object.wait(J)V (Object.java:218(Compiled Code))
   at com.ibm.ws.util.BoundedBuffer.waitPut_(J)V
  (BoundedBuffer.java:493)
   at com.ibm.ws.util.BoundedBuffer.put(Ljava/lang/Object;)V
  (BoundedBuffer.java:622)
   at com.ibm.ws.util.ThreadPool.execute(Ljava/lang/Runnable;IJ)L
  java/lang/Runnable;(ThreadPool.java:1504(Compiled Code))
   at com.ibm.ws.util.ThreadPool.execute(Ljava/lang/Runnable;)V
  (ThreadPool.java:1271(Compiled Code))
   at com.ibm.ws.drs.ha.DRSAgentClassEvents.agentClassMessageRece
  ived(Lcom/ibm/wsspi/hamanager/agent/AgentClass;Lcom/ibm/wsspi/h
  amanager/GroupMemberId;[B)VDRSAgentClassEvents.java:902(Compiled
   at com.ibm.ws.hamanager.agent.AgentClassImpl.doDataStackMessag
  eReceived(Lcom/ibm/wsspi/hamanager/GroupMemberId;Ljava/lang/Str
  ing;[B)V(AgentClassImpl.java:1184(Compiled Code))
   at com.ibm.ws.hamanager.agent.AgentClassImpl$ACCallback.doCall
  back()V(AgentClassImpl.java:1360(Compiled Code))
   at com.ibm.ws.hamanager.impl.Worker.run()V
  (Worker.java:64(Compiled Code))
   at com.ibm.ws.util.ThreadPool$Worker.run()V
  (ThreadPool.java:1892(Compiled Code))
  
  Ondeeper investigation it was observed
  All (or most) the DRS threads are consumed by http sessions.
  This is causing an OOM because the DRS threads are not available
  to handle incoming replication messages.
  Http sessions should not be using DRS' thread pool.
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  All users of IBM WebSphere Application      *
  *                  Server using Memory-to-memory session       *
  ****************************************************************
  * PROBLEM DESCRIPTION: The data replication service (DRS)      *
  *                      threads are consumed by http sessions   *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  When using Memory-to-memory session replication, http sessions
  may consume all of the threads in DRS thread pool.
  

Problem conclusion

• Code changes were made to use session threadpool instead of
  the DRS threadpool.
  
  Memory-to-memory (MTM) session thread pool custom properties
  can be set through the following Java virtual machine custom
  properties:
  
  Property name           Default value
  MTM_THREADPOOL_MINSIZE 40
  MTM_THREADPOOL_MAXSIZE  40
  
  The fix for this APAR is targeted for inclusion in fix pack
  8.5.5.21 and 9.0.5.10. For more information, see 'Recommended
  Updates for WebSphere Application Server':
  https://www.ibm.com/support/pages/node/715553
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WEBS APP SERV N

• Fixed component ID

  5724H8800

Applicable component levels

• R850 PSY

     UP

• R900 PSY

     UP