JR51271: Out-of-memory issue happens when an extension index contains a l arge number of dynamic columns

APAR status

• Closed as program error.

Error description

• The following exception will occur during server startup when
  there is a large number of dynamic columns in an extension
  index:
  
  [8/25/14 1:05:21:240 EDT] 0000000b ServletWrappe E
  com.ibm.ws.webcontainer.servlet.ServletWrapper init SRVE0100E:
  Uncaught init() exception created by servlet SolrSearchServlet
  in application Search: java.lang.OutOfMemoryError: Java heap
  space
   at
  java.lang.StringCoding$StringDecoder.decode(StringCoding.java:72
  3)
   at java.lang.StringCoding.decode(StringCoding.java:748)
   at java.lang.String.<init>(String.java:2053)
   at
  org.apache.lucene.store.DataInput.readString(DataInput.java:191)
   at
  org.apache.lucene.store.DataInput.readStringStringMap(DataInput.
  java:218)
   at
  org.apache.lucene.codecs.lucene42.Lucene42FieldInfosReader.read(
  Lucene42FieldInfosReader.java:88)
   at
  org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreRea
  ders.java:121)
   at
  org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:
  56)
   at
  org.apache.lucene.index.StandardDirectoryReader$1.doBody(Standar
  dDirectoryReader.java:62)
   at
  org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(Segmen
  tInfos.java:783)
   at
  org.apache.lucene.index.StandardDirectoryReader.open(StandardDir
  ectoryReader.java:52)
   at
  org.apache.lucene.index.DirectoryReader.open(DirectoryReader.jav
  a:88)
   at
  org.apache.solr.core.StandardIndexReaderFactory.newReader(Standa
  rdIndexReaderFactory.java:34)
   at
  com.ibm.commerce.foundation.internal.server.services.search.comp
  onent.solr.SolrSearchMultipleQueryComponent$CacheReader.generate
  SubCacheProvider(SolrSearchMultipleQueryComponent.java:2282)
   at
  com.ibm.commerce.foundation.internal.server.services.search.comp
  onent.solr.SolrSearchMultipleQueryComponent$CacheReader.warmCach
  e(SolrSearchMultipleQueryComponent.java:1808)
   at
  com.ibm.commerce.foundation.internal.server.services.search.comp
  onent.solr.SolrSearchMultipleQueryComponent$CacheReader.initCach
  e(SolrSearchMultipleQueryComponent.java:1788)
   at
  com.ibm.commerce.foundation.internal.server.services.search.comp
  onent.solr.SolrSearchMultipleQueryComponent.process(SolrSearchMu
  ltipleQueryComponent.java:659)
   at
  org.apache.solr.handler.component.SearchHandler.handleRequestBod
  y(SearchHandler.java:208)
   at
  org.apache.solr.handler.RequestHandlerBase.handleRequest(Request
  HandlerBase.java:135)
   at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
   at
  org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request
  (EmbeddedSolrServer.java:150)
   at
  org.apache.solr.client.solrj.request.QueryRequest.process(QueryR
  equest.java:90)
   at
  org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:31
  0)
   at
  com.ibm.commerce.foundation.internal.server.services.search.util
  .SearchQueryHelper.query(SearchQueryHelper.java:204)
   at
  com.ibm.commerce.foundation.internal.server.services.search.conf
  ig.solr.SolrSearchConfigurationRegistry.initializeServer(SolrSea
  rchConfigurationRegistry.java:1320)
   at
  com.ibm.commerce.foundation.internal.server.services.bootstrap.S
  earchServiceWarmStarter.start(SearchServiceWarmStarter.java:89)
   at
  com.ibm.commerce.foundation.internal.server.services.search.serv
  let.solr.SolrSearchServlet.init(SolrSearchServlet.java:138)
   at
  com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapp
  er.java:363)
   at
  com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.init(ServletW
  rapperImpl.java:171)
   at
  com.ibm.ws.webcontainer.servlet.ServletWrapper.initialize(Servle
  tWrapper.java:1844)
   at
  com.ibm.wsspi.webcontainer.extension.WebExtensionProcessor.creat
  eServletWrapper(WebExtensionProcessor.java:98)
   at
  com.ibm.ws.webcontainer.webapp.WebApp.getServletWrapper(WebApp.j
  ava:1048)
   at
  com.ibm.ws.webcontainer.webapp.WebApp.getServletWrapper(WebApp.j
  ava:969)
   at
  com.ibm.ws.webcontainer.webapp.WebApp.initializeTargetMappings(W
  ebApp.java:648)
   at
  com.ibm.ws.webcontainer.webapp.WebApp.commonInitializationFinall
  y(WebApp.java:450)
   at
  com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.
  java:304)
   at
  com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(We
  bGroupImpl.java:100)
   at
  com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(Virtua
  lHostImpl.java:166)
   at
  com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.
  java:732)
   at
  com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebCo
  ntainer.java:617)
   at
  com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebCo
  ntainerImpl.java:376)
   at
  com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebCont
  ainerImpl.java:668)
   at
  com.ibm.ws.runtime.component.ApplicationMgrImpl.start(Applicatio
  nMgrImpl.java:1128)
   at
  com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeploye
  dObjectStart(DeployedApplicationImpl.java:1319)
   at
  com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedMo
  duleImpl.java:611)
   at
  com.ibm.ws.runtime.component.DeployedApplicationImpl.start(Deplo
  yedApplicationImpl.java:944)
   at
  com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication
  (ApplicationMgrImpl.java:741)
   at
  com.ibm.ws.runtime.component.ApplicationMgrImpl$3.run(Applicatio
  nMgrImpl.java:2056)
   at
  com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManager
  Impl.java:5305)
   at
  com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextM
  anagerImpl.java:5483)
   at
  com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityCon
  text.java:255)
   at
  com.ibm.ws.runtime.component.ApplicationMgrImpl.start(Applicatio
  nMgrImpl.java:2061)
   at
  com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(Compos
  itionUnitMgrImpl.java:389)
   at
  com.ibm.ws.runtime.component.CompositionUnitImpl.start(Compositi
  onUnitImpl.java:123)
   at
  com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(Compos
  itionUnitMgrImpl.java:332)
   at
  com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$300(C
  ompositionUnitMgrImpl.java:117)
   at
  com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitialize
  r.run(CompositionUnitMgrImpl.java:899)
   at
  com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializ
  er.run(WsComponentImpl.java:496)
   at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1656)
  

Local fix

• Perform index optimization - this can reduce the overall number
  of index segment files.
  

Problem summary

• USERS AFFECTED:
  Shopper
  
  PROBLEM ABSTRACT:
  Out-of-memory issue happens when an extension index contains a
  large number of dynamic columns
  
  BUSINESS IMPACT:
  Site outage
  
  RECOMMENDATION:
  

Problem conclusion

• A custom implementation of codec field reader has been created
  to optimize the logic in Solr when initializing an index with
  many dynamic columns.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WC BUS EDITION

• Fixed component ID

  5724I3800

Applicable component levels

• R700 PSY

     UP