IBM Support

JR51767: LARGE NUMBER OF THREADS CREATED, CAUSING OUTOFMEMORYERROR, BECAUSE SEARCH TIMER NOT RESET

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • On IBM Business Process Manager (BPM) V8.5, a timer that resets
    the business data aliases cache was introduced. This timer is
    created in a separate thread during the cache initialization,
    which should happen only once per server. The cache
    initialization is invoked upon first access, either through the
    Process Admin Console or on the first run of a TWSearch API
    query.
    
    However, because of multiple entry points, cache initialization
    might be invoked at the same time from different areass and
    multiple reset threads might be created.
    
    As a result, thousands of threads might be created, which causes
    a java.lang.OutOfMemoryError.
    .--------------------------------------------
    
       The javastack of the threads would be similar to this
    I see several hundreds of
    threads with the same stack like:
    
    Java callstack:
    at java/lang/Object.wait(Native Method)
    at java/lang/Object.wait(Object.java:167(Compiled Code))
    at com/ibm/ws/sib/jfapchannel/impl/ExchangeReceiveListener.
       waitToComplete(ExchangeReceiveListener.java:227(Compiled
    Code))
       (entered lock: com/ibm/ws/sib/jfapchannel/impl/
       ExchangeReceiveListener@0x0000000743AE1038, entry count: 1)
    at com/ibm/ws/sib/jfapchannel/impl/ConversationImpl.exchange
       (ConversationImpl.java:988(Compiled Code))
    at com/ibm/ws/sib/comms/common/JFAPCommunicator.jfapExchange
       (JFAPCommunicator.java:386)
    at
    com/ibm/ws/sib/comms/client/ConnectionProxy._createProducerSessi
    on
       (ConnectionProxy.java:1074)
       (entered lock: java/lang/Object@0x0000000743AE0500, entry
    count: 1)
    at
    com/ibm/ws/sib/comms/client/ConnectionProxy.createProducerSessio
    n (ConnectionProxy.java:935)
    at com/ibm/ws/sib/api/jms/impl/JmsMsgProducerImpl.<init>
       (JmsMsgProducerImpl.java:371(Compiled Code))
    at com/ibm/ws/sib/api/jms/impl/JmsTopicPublisherImpl.<init>
       (JmsTopicPublisherImpl.java:59(Compiled Code))
    at
    com/ibm/ws/sib/api/jms/impl/JmsTopicSessionImpl.instantiateProdu
    cer (JmsTopicSessionImpl.java:231)
    at com/ibm/ws/sib/api/jms/impl/JmsSessionImpl.createProducer
       (JmsSessionImpl.java:866)
    at
    com/ibm/ws/sib/api/jms/impl/JmsTopicSessionImpl.createPublisher
       (JmsTopicSessionImpl.java:97)
    at com/lombardisoftware/core/cache/CacheMessenger.
       sendCacheMessageInTransaction(CacheMessenger.java:263)
    at com/lombardisoftware/core/cache/CacheMessenger.access$000
       (CacheMessenger.java:49)
    at
    com/lombardisoftware/core/cache/CacheMessenger$1.doInTransaction
       (CacheMessenger.java:144)
    at
    com/lombardisoftware/server/core/TXCommand$2.call(TXCommand.java
    :88 (Compiled Code))
    at
    com/lombardisoftware/utility/spring/ProgrammaticTransactionSuppo
    rt$1.oInTransaction(ProgrammaticTransactionSupport.java:409
       (Compiled Code))
    at org/springframework/transaction/jta/
       WebSphereUowTransactionManager$UOWActionAdapter.run
       (WebSphereUowTransactionManager.java:306(Compiled Code))
    at com/ibm/ws/uow/EmbeddableUOWManagerImpl.runUnderNewUOW
       (EmbeddableUOWManagerImpl.java:787(Compiled Code))
    at com/ibm/ws/uow/EmbeddableUOWManagerImpl.runUnderUOW
       (EmbeddableUOWManagerImpl.java:366(Compiled Code))
    at
    org/springframework/transaction/jta/WebSphereUowTransactionManag
    er.execute(WebSphereUowTransactionManager.java:252(Compiled
    Code))
    at
    com/lombardisoftware/utility/spring/ProgrammaticTransactionSuppo
    rt.executeInNewTransaction(ProgrammaticTransactionSupport.
    java:404 (Compiled Code))
    at
    com/lombardisoftware/utility/spring/ProgrammaticTransactionSuppo
    rt.execute(ProgrammaticTransactionSupport.java:267)
    at
    com/lombardisoftware/server/core/TXCommand.executeInDeadlockRetr
    yLoop (TXCommand.java:86(Compiled Code))
    at
    com/lombardisoftware/server/core/TXCommand.execute(TXCommand.jav
    a:75 (Compiled Code))
    at
    com/lombardisoftware/core/cache/CacheMessenger.sendCacheMessage
       (CacheMessenger.java:142)
    at
    com/lombardisoftware/core/cache/CacheMessenger.sendCacheMessage
       (CacheMessenger.java:130)
    at com/lombardisoftware/server/core/cache/ServerCacheManager.
       resetCache(ServerCacheManager.java:74)
    at
    com/lombardisoftware/server/ejb/psapi/BusinessDataAliasesCache$1
    .run(BusinessDataAliasesCache.java:58)
    at java/util/Timer$TimerImpl.run(Timer.java:296(Compiled Code))
    
    
       These errors may also be observed in the logs
    [08/06/15 06:08:50:695 BST] 00008252 FreePool      E
    J2CA0045E:
    Connection not available while invoking method
    createOrWaitForConnection
    for resource jdbc/TeamWorksDB.
    [08/06/15 06:08:50:697 BST] 00002f21 wle           E   Error
    loading
    cache data, returning empty result.
    
    com.lombardisoftware.core.TeamWorksException: Error compiling
    BusinessDataAliases
            at
    com.lombardisoftware.server.ejb.psapi.BusinessDataAliasesBuilder
    .buildBusinessDataAliases(BusinessDataAliasesBuilder.java:219)
            at
    com.lombardisoftware.server.ejb.psapi.BusinessDataAliasesCacheHe
    lper.fin
    dBusinessDataAliases(BusinessDataAliasesCacheHelper.java:20)
            at
    com.lombardisoftware.server.ejb.psapi.BusinessDataAliasesCache.l
    oadCacheData(BusinessDataAliasesCache.java:73)
            at
    com.lombardisoftware.server.ejb.psapi.BusinessDataAliasesCache.l
    oadCacheData(BusinessDataAliasesCache.java:18)
            at
    com.lombardisoftware.core.cache.GenericCache.getCacheData(Generi
    cCache.java:184)
            at
    com.lombardisoftware.server.ejb.psapi.BusinessDataAliasesCache.g
    etBusinessDataAliases(BusinessDataAliasesCache.java:67)
            at
    com.lombardisoftware.server.ejb.psapi.BusinessDataAliasesCache$1
    .run(BusinessDataAliasesCache.java:60)
    
    PRODUCTS AFFECTED:
    IBM Business Process Manager (BPM) Advanced
    IBM BPM Standard
    IBM BPM Express
    

Local fix

Problem summary

  • The thread that is created goes through these calls:
    
    3XMTHREADINFO      "Thread-19058" J9VMThread:0x00000000C0757E00,
    j9thread_t:0x00000100AD93D520,
    java/lang/Thread:0x00000005D1578AE8, state:B, prio=5
    ,,,
    4XESTACKTRACE                at
    com/lombardisoftware/server/core/cache/ServerCacheManager.resetC
    ache(ServerCacheManager.java:73)
    4XESTACKTRACE                at
    com/lombardisoftware/server/ejb/psapi/BusinessDataAliasesCache$1
    .run(BusinessDataAliasesCache.java:58)
    4XESTACKTRACE                at
    java/util/Timer$TimerImpl.run(Timer.java:296(Compiled Code))
    3XMTHREADINFO3           No native callstack available for this
    thread
    
    This issue is related to the interim fix JR48886. Although
    JR48886 partially addresses the issue, JR48886 introduced a
    check that ensures a new reset timer is not created if a reset
    timer is already present. This check should prevent multiple
    reset threads on the cache by making sure only one exists.
    However, in a race condition, even with JR48886 applied, this
    problem still occurs.
    

Problem conclusion

  • A fix is available for IBM BPM V8.5.0.1 that verifies whether a
    reset thread exists before creating one and improves the system
    configuration parameter reading.
    
    On Fix Central (http://www.ibm.com/support/fixcentral), search
    for JR51767:
    
    1. Select IBM Business Process Manager with your edition from
    the product selector, the installed version to the fix pack
    level, and your platform, and then click Continue.
    2. Select APAR or SPR, enter JR51767, and click Continue.
    
    When you download fix packages, ensure that you also download
    the readme file for each fix. Review each readme file for
    additional installation instructions and information about the
    fix.
    

Temporary fix

Comments

APAR Information

  • APAR number

    JR51767

  • Reported component name

    BPM STANDARD

  • Reported component ID

    5725C9500

  • Reported release

    850

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2014-11-11

  • Closed date

    2014-12-11

  • Last modified date

    2016-03-22

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    BPM STANDARD

  • Fixed component ID

    5725C9500

Applicable component levels

  • R850 PSY

       UP

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSFTDH","label":"IBM Business Process Manager Standard"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.5","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
22 March 2016