IBM Support

IT26168: MESSAGE FABRIC SERVICE CAUSES AN OUT OF MEMORY ERROR

Subscribe to this APAR

By subscribing, you receive periodic emails alerting you to the status of the APAR, along with a link to the fix after it becomes available. You can track this item individually or track all items by product.

Notify me when this APAR changes.

Notify me when an APAR for this component changes.

 

APAR status

  • Closed as program error.

Error description

  • Error Description
    
    MessageFabricServiceImpl has used occupies 374,049,712 (71.61%)
    bytes memory heap and has caused OOM error which kept bringing
    the node down.
    
    
    The heap dump analysis shows major suspect was message fabric
    
    
    The class
    "com.ibm.b2b.osgi.bundles.msgfabric.RequestReplySessionManager",
    loaded by "<system class loader>", occupies 374,049,712 (71.61%)
    bytes. The memory is accumulated in one instance of
    "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by
    "<system class loader>".
    
    
    Keywords
    
    com.ibm.b2b.osgi.bundles.msgfabric.RequestReplySessionManager
    
    java.util.concurrent.ConcurrentHashMap$Segment[]
    
    
    
    
    There were log messages like
    
    
    [1557496804124]java.nio.file.ClosedWatchServiceException
    
        at sun.nio.fs.AbstractPoller.invoke(AbstractPoller.java:229)
    
        at
    sun.nio.fs.AbstractPoller.register(AbstractPoller.java:141)
    
        at
    sun.nio.fs.WindowsWatchService.register(WindowsWatchService.java
    :86)
    
        at sun.nio.fs.WindowsPath.register(WindowsPath.java:888)
    
        at sun.nio.fs.AbstractPath.register(AbstractPath.java:117)
    
        at
    com.ibm.b2b.storage.service.adapter.StorageAdapterDirectoryWatch
    er.registerPathForWatching(StorageAdapterDirectoryWatcher.java:2
    59)
    
        at
    com.ibm.b2b.storage.service.adapter.StorageAdapterDirectoryWatch
    er.registerBaseFilePath(StorageAdapterDirectoryWatcher.java:223)
    
        at
    com.ibm.b2b.storage.service.adapter.StorageAdapterDirectoryWatch
    er.registerBucketBaseFilePath(StorageAdapterDirectoryWatcher.jav
    a:210)
    
        at
    com.ibm.b2b.osgi.bundles.megstorage.MEGStoreContextManager.getSt
    oreContext(MEGStoreContextManager.java:147)
    
        at
    com.ibm.b2b.osgi.bundles.megstorage.MEGStorageServiceImpl.getSto
    reContext(MEGStorageServiceImpl.java:400)
    
        at
    com.ibm.b2b.osgi.bundles.megstorage.MEGStorageServiceImpl.getDat
    a(MEGStorageServiceImpl.java:211)
    
        at
    com.ibm.b2b.services.businessdoc.StorageKeeper.getData(StorageKe
    eper.java:126)
    
        at
    com.ibm.b2b.osgi.bundles.msgfabric.BusinessDocCallback.extractDo
    c(BusinessDocCallback.java:604)
    
        at
    com.ibm.b2b.osgi.bundles.msgfabric.BusinessDocCallback.toProcess
    Data(BusinessDocCallback.java:317)
    

Local fix

  • STRRTC - 573125
    DW / DW
    Circumvention :
    
    Please check the messagefabric.log if you have the following
    error repeatedly occurring in the log:
    [2018-07-27 18:21:16.151] ERROR BusinessDocCallback.onMessage
    caught throwable.
    [2018-07-27 18:21:16.151] ERROR [1532679676151]
    com.ibm.b2b.apiint.messagedefinitions.messages.xml.exchange.noti
    fication.NotifyErrorType incompatible with
    com.ibm.b2b.apiint.messagedefinitions.messages.xml.exchange.as4n
    otification.NotifyErrorMessage
    [2018-07-27 18:21:16.151] ERRORDTL
    [1532679676151]java.lang.ClassCastException:
    com.ibm.b2b.apiint.messagedefinitions.messages.xml.exchange.noti
    fication.NotifyErrorType incompatible with
    com.ibm.b2b.apiint.messagedefinitions.messages.xml.exchange.as4n
    otification.NotifyErrorMessage
    at
    com.ibm.b2b.osgi.bundles.msgfabric.BusinessDocCallback.toProcess
    Data(BusinessDocCallback.java:367)
    at
    com.ibm.b2b.osgi.bundles.msgfabric.ProcessLauncherCallback.onMes
    sage(ProcessLauncherCallback.java:142)
    at
    com.ibm.b2b.osgi.bundles.msgfabric.BusinessDocCallback.onMessage
    (BusinessDocCallback.java:157)
    at
    com.ibm.b2b.osgi.bundles.msgfabric.CompositeCallback.onMessage(C
    ompositeCallback.java:183)
    at
    com.ibm.b2b.osgi.bundles.msgfabric.AsyncMessageHandler.onMessage
    (AsyncMessageHandler.java:112)
    
    If yes, you need to:
    1. Identify the BDO Adapter that triggers the error.
    2. Disable the adapter and check the last message (invalid
    message) from the queue.
    3. Remove the message from the queue and restart the adapter.
    Ensure the error message disappears from the log.
    

Problem summary

  • Users Affected:
    All
    
    Problem Description:
    The Message Fabric Service causes an out of memory error in the
    adapter container due to ClassCastException.
    
    Platforms Affected:
    All
    

Problem conclusion

  • Resolution Summary:
    
    A code fix is provided.
    
    Fix for OOM is to make the unregister operation when there is a
    failure in Notification message:
    
    BusinessDocCallback.toProcessData
    
    
    Any faulty message processing which goes for loop back message
    handling need to unregister itself first before opening a new
    session to process.
    
    Delivered In:
    5020603_11
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT26168

  • Reported component name

    STR B2B INTEGRA

  • Reported component ID

    5725D0600

  • Reported release

    526

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2018-09-04

  • Closed date

    2019-08-30

  • Last modified date

    2019-09-06

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    STR B2B INTEGRA

  • Fixed component ID

    5725D0600

Applicable component levels

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS3JSW","label":"IBM Sterling B2B Integrator"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"5.2.6","Edition":"","Line of Business":{"code":"LOB02","label":"AI Applications"}}]

Document Information

Modified date:
06 September 2019