JR51767: LARGE NUMBER OF THREADS CREATED, CAUSING OUTOFMEMORYERROR, BECAUSE SEARCH TIMER NOT RESET

8.0.1.3-WS-BPM-IFJR51767
8.0.1.2-WS-BPM-IFJR51767
8.5.0.1-WS-BPM-IFJR51767
Version 8.5 Refresh Pack 6 for the IBM Business Process Manager products
Version 8.5.0 Fix Pack 2 for the IBM Business Process Manager products

APAR status

• Closed as program error.

Error description

• On IBM Business Process Manager (BPM) V8.5, a timer that resets
  the business data aliases cache was introduced. This timer is
  created in a separate thread during the cache initialization,
  which should happen only once per server. The cache
  initialization is invoked upon first access, either through the
  Process Admin Console or on the first run of a TWSearch API
  query.
  
  However, because of multiple entry points, cache initialization
  might be invoked at the same time from different areass and
  multiple reset threads might be created.
  
  As a result, thousands of threads might be created, which causes
  a java.lang.OutOfMemoryError.
  .--------------------------------------------
  
     The javastack of the threads would be similar to this
  I see several hundreds of
  threads with the same stack like:
  
  Java callstack:
  at java/lang/Object.wait(Native Method)
  at java/lang/Object.wait(Object.java:167(Compiled Code))
  at com/ibm/ws/sib/jfapchannel/impl/ExchangeReceiveListener.
     waitToComplete(ExchangeReceiveListener.java:227(Compiled
  Code))
     (entered lock: com/ibm/ws/sib/jfapchannel/impl/
     ExchangeReceiveListener@0x0000000743AE1038, entry count: 1)
  at com/ibm/ws/sib/jfapchannel/impl/ConversationImpl.exchange
     (ConversationImpl.java:988(Compiled Code))
  at com/ibm/ws/sib/comms/common/JFAPCommunicator.jfapExchange
     (JFAPCommunicator.java:386)
  at
  com/ibm/ws/sib/comms/client/ConnectionProxy._createProducerSessi
  on
     (ConnectionProxy.java:1074)
     (entered lock: java/lang/Object@0x0000000743AE0500, entry
  count: 1)
  at
  com/ibm/ws/sib/comms/client/ConnectionProxy.createProducerSessio
  n (ConnectionProxy.java:935)
  at com/ibm/ws/sib/api/jms/impl/JmsMsgProducerImpl.<init>
     (JmsMsgProducerImpl.java:371(Compiled Code))
  at com/ibm/ws/sib/api/jms/impl/JmsTopicPublisherImpl.<init>
     (JmsTopicPublisherImpl.java:59(Compiled Code))
  at
  com/ibm/ws/sib/api/jms/impl/JmsTopicSessionImpl.instantiateProdu
  cer (JmsTopicSessionImpl.java:231)
  at com/ibm/ws/sib/api/jms/impl/JmsSessionImpl.createProducer
     (JmsSessionImpl.java:866)
  at
  com/ibm/ws/sib/api/jms/impl/JmsTopicSessionImpl.createPublisher
     (JmsTopicSessionImpl.java:97)
  at com/lombardisoftware/core/cache/CacheMessenger.
     sendCacheMessageInTransaction(CacheMessenger.java:263)
  at com/lombardisoftware/core/cache/CacheMessenger.access$000
     (CacheMessenger.java:49)
  at
  com/lombardisoftware/core/cache/CacheMessenger$1.doInTransaction
     (CacheMessenger.java:144)
  at
  com/lombardisoftware/server/core/TXCommand$2.call(TXCommand.java
  :88 (Compiled Code))
  at
  com/lombardisoftware/utility/spring/ProgrammaticTransactionSuppo
  rt$1.oInTransaction(ProgrammaticTransactionSupport.java:409
     (Compiled Code))
  at org/springframework/transaction/jta/
     WebSphereUowTransactionManager$UOWActionAdapter.run
     (WebSphereUowTransactionManager.java:306(Compiled Code))
  at com/ibm/ws/uow/EmbeddableUOWManagerImpl.runUnderNewUOW
     (EmbeddableUOWManagerImpl.java:787(Compiled Code))
  at com/ibm/ws/uow/EmbeddableUOWManagerImpl.runUnderUOW
     (EmbeddableUOWManagerImpl.java:366(Compiled Code))
  at
  org/springframework/transaction/jta/WebSphereUowTransactionManag
  er.execute(WebSphereUowTransactionManager.java:252(Compiled
  Code))
  at
  com/lombardisoftware/utility/spring/ProgrammaticTransactionSuppo
  rt.executeInNewTransaction(ProgrammaticTransactionSupport.
  java:404 (Compiled Code))
  at
  com/lombardisoftware/utility/spring/ProgrammaticTransactionSuppo
  rt.execute(ProgrammaticTransactionSupport.java:267)
  at
  com/lombardisoftware/server/core/TXCommand.executeInDeadlockRetr
  yLoop (TXCommand.java:86(Compiled Code))
  at
  com/lombardisoftware/server/core/TXCommand.execute(TXCommand.jav
  a:75 (Compiled Code))
  at
  com/lombardisoftware/core/cache/CacheMessenger.sendCacheMessage
     (CacheMessenger.java:142)
  at
  com/lombardisoftware/core/cache/CacheMessenger.sendCacheMessage
     (CacheMessenger.java:130)
  at com/lombardisoftware/server/core/cache/ServerCacheManager.
     resetCache(ServerCacheManager.java:74)
  at
  com/lombardisoftware/server/ejb/psapi/BusinessDataAliasesCache$1
  .run(BusinessDataAliasesCache.java:58)
  at java/util/Timer$TimerImpl.run(Timer.java:296(Compiled Code))
  
  
     These errors may also be observed in the logs
  [08/06/15 06:08:50:695 BST] 00008252 FreePool      E
  J2CA0045E:
  Connection not available while invoking method
  createOrWaitForConnection
  for resource jdbc/TeamWorksDB.
  [08/06/15 06:08:50:697 BST] 00002f21 wle           E   Error
  loading
  cache data, returning empty result.
  
  com.lombardisoftware.core.TeamWorksException: Error compiling
  BusinessDataAliases
          at
  com.lombardisoftware.server.ejb.psapi.BusinessDataAliasesBuilder
  .buildBusinessDataAliases(BusinessDataAliasesBuilder.java:219)
          at
  com.lombardisoftware.server.ejb.psapi.BusinessDataAliasesCacheHe
  lper.fin
  dBusinessDataAliases(BusinessDataAliasesCacheHelper.java:20)
          at
  com.lombardisoftware.server.ejb.psapi.BusinessDataAliasesCache.l
  oadCacheData(BusinessDataAliasesCache.java:73)
          at
  com.lombardisoftware.server.ejb.psapi.BusinessDataAliasesCache.l
  oadCacheData(BusinessDataAliasesCache.java:18)
          at
  com.lombardisoftware.core.cache.GenericCache.getCacheData(Generi
  cCache.java:184)
          at
  com.lombardisoftware.server.ejb.psapi.BusinessDataAliasesCache.g
  etBusinessDataAliases(BusinessDataAliasesCache.java:67)
          at
  com.lombardisoftware.server.ejb.psapi.BusinessDataAliasesCache$1
  .run(BusinessDataAliasesCache.java:60)
  
  PRODUCTS AFFECTED:
  IBM Business Process Manager (BPM) Advanced
  IBM BPM Standard
  IBM BPM Express
  

Local fix

Problem summary

• The thread that is created goes through these calls:
  
  3XMTHREADINFO      "Thread-19058" J9VMThread:0x00000000C0757E00,
  j9thread_t:0x00000100AD93D520,
  java/lang/Thread:0x00000005D1578AE8, state:B, prio=5
  ,,,
  4XESTACKTRACE                at
  com/lombardisoftware/server/core/cache/ServerCacheManager.resetC
  ache(ServerCacheManager.java:73)
  4XESTACKTRACE                at
  com/lombardisoftware/server/ejb/psapi/BusinessDataAliasesCache$1
  .run(BusinessDataAliasesCache.java:58)
  4XESTACKTRACE                at
  java/util/Timer$TimerImpl.run(Timer.java:296(Compiled Code))
  3XMTHREADINFO3           No native callstack available for this
  thread
  
  This issue is related to the interim fix JR48886. Although
  JR48886 partially addresses the issue, JR48886 introduced a
  check that ensures a new reset timer is not created if a reset
  timer is already present. This check should prevent multiple
  reset threads on the cache by making sure only one exists.
  However, in a race condition, even with JR48886 applied, this
  problem still occurs.
  

Problem conclusion

• A fix is available for IBM BPM V8.5.0.1 that verifies whether a
  reset thread exists before creating one and improves the system
  configuration parameter reading.
  
  On Fix Central (http://www.ibm.com/support/fixcentral), search
  for JR51767:
  
  1. Select IBM Business Process Manager with your edition from
  the product selector, the installed version to the fix pack
  level, and your platform, and then click Continue.
  2. Select APAR or SPR, enter JR51767, and click Continue.
  
  When you download fix packages, ensure that you also download
  the readme file for each fix. Review each readme file for
  additional installation instructions and information about the
  fix.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  BPM STANDARD

• Fixed component ID

  5725C9500

Applicable component levels

• R850 PSY

     UP