PH66923: OOM HEAP ISSUE WHILE NAVIGATING ON ADMIN CONSOLE-LOGGING PANEL

APAR status

• Closed as program error.

Error description

• OOM in Admin Console Trace Panel due to logger preprocessing
  with classpath-embedded logger names.
  
  WebSphere Version: 9.0.5.20 BASE
  Java Version: IBM J9 VM, 1.8.0_421 (8.0.8.30 - SR8 FP30)
  OS: Linux 4.18.0-553.52.1.el8_10.x86_64
  
  The Admin agent dumps shows majority of heap (2.9G) used by:
  The java stack associated with above is:
  
  WebContainer : 4
   at java.lang.String.<init>([CII)V (String.java:687(Compiled
  Code))
   at java.lang.String.<init>([C)V (String.java:622(Compiled
  Code))
   at java.lang.Integer.toString(I)Ljava/lang/String;
  (Integer.java:415(Compiled Code))
   at java.lang.String.valueOf(I)Ljava/lang/String;
  (String.java(Compiled Code))
   at com.ibm.ws.console.probdetermination.trace.tree.AsynchTrace
  TreeServlet.populateTraceComponentStrings(Ljava/util/List;)Ljav
  a/util/TreeMap;(AsynchTraceTreeServlet.java:379(Compiled Code))
   at com.ibm.ws.console.probdetermination.trace.tree.TraceTreeCa
  cheInit.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/S
  ervletResponse;Ljavax/servlet/FilterChain;)V(TraceTreeCacheInit.
   at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilt
  er(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse
  ;Ljavax/servlet/FilterChain;)V(FilterInstanceWrapper.java:197)
   at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(L
  javax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(W
   at com.ibm.ws.console.core.servlet.WSCUrlFilter.setUpCommandAs
  sistance(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/
  http/HttpServletResponse;Ljavax/servlet/FilterChain;)V(WSCUrlFil
   at com.ibm.ws.console.core.servlet.WSCUrlFilter.continueStorin
  gTaskState(Ljavax/servlet/http/HttpSession;Ljava/lang/String;Lj
  ava/lang/String;Ljavax/servlet/http/HttpServletRequest;Ljavax/s
  ervlet/http/HttpServletResponse;Ljavax/servlet/FilterChain;)V(WS
   at com.ibm.ws.console.core.servlet.WSCUrlFilter.doFilter(Ljava
  x/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/
  servlet/FilterChain;)V(WSCUrlFilter.java:352)
   at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilt
  er(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse
  ;Ljavax/servlet/FilterChain;)V(FilterInstanceWrapper.java:197)
   at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(L
  javax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(W
   at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter
  (Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;L
  com/ibm/wsspi/webcontainer/RequestProcessor;Lcom/ibm/ws/webcont
  ainer/webapp/WebAppDispatcherContext;)V(WebAppFilterManager.java
   at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFi
  lters(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletRespo
  nse;Lcom/ibm/wsspi/webcontainer/servlet/IServletContext;Lcom/ib
  m/wsspi/webcontainer/RequestProcessor;Ljava/util/EnumSet;)Z(WebA
   at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(Ljavax/
  servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(WebApp.
   at com.ibm.ws.webcontainer.webapp.WebAppImpl.handleRequest(Lja
  vax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(Web
   at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(Ljava
  x/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V(WebGr
   at com.ibm.ws.webcontainer.WebContainer.handleRequest(Lcom/ibm
  /websphere/servlet/request/IRequest;Lcom/ibm/websphere/servlet/
  response/IResponse;)V(WebContainer.java:1033)
   at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(Lcom/i
  bm/websphere/servlet/request/IRequest;Lcom/ibm/websphere/servle
  t/response/IResponse;)V(WSWebContainer.java:1817)
   at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(Lcom/ib
  m/wsspi/channel/framework/VirtualConnection;)V(WCChannelLink.jav
   at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handle
  Discrimination()V(HttpInboundLink.java:465)
   at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handle
  NewRequest()V(HttpInboundLink.java:532)
   at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.proces
  sRequest()V(HttpInboundLink.java:318)
   at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.co
  mplete(Lcom/ibm/wsspi/channel/framework/VirtualConnection;Lcom/
  ibm/wsspi/tcp/channel/TCPReadRequestContext;)V(HttpICLReadCallba
   at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCo
  mpletedCallback.complete(Lcom/ibm/wsspi/channel/framework/Virtu
  alConnection;Lcom/ibm/wsspi/tcp/channel/TCPReadRequestContext;)V
   at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futur
  eCompleted(Lcom/ibm/io/async/IAbstractAsyncFuture;Ljava/lang/Ob
  ject;)V(AioReadCompletionListener.java:175)
   at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(Lcom/ib
  m/io/async/ICompletionListener;Lcom/ibm/io/async/AbstractAsyncF
  uture;Ljava/lang/Object;)V(AbstractAsyncFuture.java:217)
   at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions()V
  (AsyncChannelFuture.java:161)
   at com.ibm.io.async.AsyncFuture.completed(J)V
  (AsyncFuture.java:138(Compiled Code))
   at com.ibm.io.async.ResultHandler.complete(Lcom/ibm/io/async/A
  syncFuture;II)V(ResultHandler.java:204)
   at com.ibm.io.async.ResultHandler.runEventProcessingLoop(Z)V
  (ResultHandler.java:775(Compiled Code))
   at com.ibm.io.async.ResultHandler$2.run()V
  (ResultHandler.java:905)
   at com.ibm.ws.util.ThreadPool$Worker.run()V
  (ThreadPool.java:1909)
  
  The gc data shows the memory shot up in less than 2 seconds:
  

Local fix

• Avoid the trace specification page.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:  All users of IBM WebSphere Application      *
  *                  Server                                      *
  *                  with large amount of jars in application    *
  *                  class path w/ PARENT_LAST classloader       *
  *                  setting                                     *
  ****************************************************************
  * PROBLEM DESCRIPTION: OOM issue occurs when loading the       *
  *                      "Change log details" page.              *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  When using a PARENT_LAST classloader with an application
  classpath that has a large
  amount of jars in it (i.e., 50+).
  It is possible that `org.apache.cxf.*` classes are loaded with
  the
  com.ibm.ws.classloader.CompoundClassLoader. The logger for these
  classes are created
  with their classpath appended to the name used to register the
  logger. The compound
  class loader in this case, as part of it's toString(), will list
  the entire classpath
  (of jars).
  The "Change log details" page before it loads pre-processes the
  the list or registered
  loggers and breaks down the logger name into it's component
  hierarchical parts. This
  is split through the periods (.) that are found in the name.
  With multiple
  org.apache.cxf.* classes loaded by the compound class loader and
  combined with the
  classpath of many jars, this results in the underlying data
  structure used for
  processing to consume a large amount of memory leading to an OOM
  issue.
  

Problem conclusion

• The pre-processing will avoid incorporating the class path as
  part of the logger name.
  
  The fix for this APAR is targeted for inclusion in fix pack
  9.0.5.26 and 8.5.5.29. For more information, see 'Recommended
  Updates for WebSphere Application Server':
  https://www.ibm.com/support/pages/node/715553
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WEBSPHERE APP S

• Fixed component ID

  5724J0800

Applicable component levels