IBM Support

IT20636: High virtual memory utilization reported for IBM MQ processes

Subscribe to this APAR

By subscribing, you receive periodic emails alerting you to the status of the APAR, along with a link to the fix after it becomes available. You can track this item individually or track all items by product.

Notify me when this APAR changes.

Notify me when an APAR for this component changes.

 

APAR status

  • Closed as program error.

Error description

  • IBM MQ processes may be seen to be using a large amount of
    virtual memory (VSZ), for example:
    
    S   100 15572     1  814m 30888 0:0   04:31 00:00:00
    /opt/mqm/bin/amqzxma0 -m
    S   100 15580 15572  353m  8376 0:0   04:31 00:00:00
    /opt/mqm/bin/amqzfuma -m
    S   100 15585 15572 1962m 14476 0:0   04:31 00:00:00
    /opt/mqm/bin/amqzmuc0 -m
    S   100 15604 15572  735m  7748 0:0   04:31 00:00:00
    /opt/mqm/bin/amqzmur0 -m
    S   100 15609 15572  803m 10360 0:0   04:31 00:00:00
    /opt/mqm/bin/amqzmuf0 -m
    S   100 15613 15572  289m 10616 0:0   04:31 00:00:00
    /opt/mqm/bin/amqrrmfa -m
    S   100 15630 15572  185m  6004 0:0   04:31 00:00:00
    /opt/mqm/bin/amqzmgr0 -m
    S   100 15632 15609  416m 10324 0:0   04:31 00:00:00
    /opt/mqm/bin/amqfqpub -mT
    S   100 15642 15630  343m  6856 0:0   04:31 00:00:00
    /opt/mqm/bin/runmqchi -m
    S   100 15643 15630  201m  6288 0:0   04:31 00:00:00
    /opt/mqm/bin/amqpcsea TES
    S   100 15644 15632  673m 11160 0:0   04:31 00:00:00
    /opt/mqm/bin/amqfcxba -m
    S   100 15649 15572  797m  9964 0:0   04:31 00:00:00
    /opt/mqm/bin/amqzlaa0 -mT
    

Local fix

  • It may be possible to work around the issue by exporting this
    environment variable before starting an IBM MQ queue manager:
    
    MALLOC_ARENA_MAX=1
    
    Some malloc() implementations however will ignore the
    environment variable for setuid/setgid binaries such as those
    used by IBM MQ.
    

Problem summary

  • ****************************************************************
    USERS AFFECTED:
    Users running IBM MQ on an OS that uses an implementation of
    malloc() which utilises per-thread memory arenas may be affected
    by this issue.  This issue may particularly affect customers
    that run IBM MQ on a machine with many CPU cores.
    
    
    Platforms affected:
    Linux on x86-64, Linux on x86, Linux on S390, Linux on Power,
    Linux on zSeries
    
    ****************************************************************
    PROBLEM DESCRIPTION:
    This URL gives an overview of the issue caused by assigning
    threads their own memory pools:
    
    https://www.ibm.com/developerworks/community/blogs/kevgrig/entry
    /linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memor
    y_usage?lang=en
    
    IBM MQ processes such as the IBM MQ agent (amqzlaa0) and channel
    (amqrmppa) are highly multi-threaded.  This makes IBM MQ
    processes particularly vulnerable to this issue on a machine
    with a large number of cores as there is high contention for the
    malloc() lock across the threads in an IBM MQ process.
    
    When this issue occurs then pmap output will show many "arena"
    memory segments and these are normally 64MB in size, for
    example:
    
    Address	  Kbytes     RSS   Dirty    Swap  Mode  Mapping
    ...
    80064000   65136       0       0       0  ---p    [ anon ]
    84000000     132      12      12       0  rw-p    [ anon ]
    84021000   65404       0       0       0  ---p    [ anon ]
    88000000     132      16      16       0  rw-p    [ anon ]
    88021000   65404       0       0       0  ---p    [ anon ]
    8c000000     132      12      12       0  rw-p    [ anon ]
    8c021000   65404       0       0       0  ---p    [ anon ]
    ...
    --------  ------  ------  ------  ------
    total	 2009720    5924    3224       0
    
    The majority of the addressed memory segments should not be in
    the resident set (RSS) and should therefore not contribute to
    the commit charge of the process.  IBM has however seen evidence
    that some customers with particular memory allocation patterns
    and large numbers of arenas may suffer from fragmentation within
    arenas and this can lead to actual memory resource issues.
    

Problem conclusion

  • The majority of users should not be affected by this issue and
    so the default behaviour of the IBM MQ code is unchanged.
    
    Two new environment variables have been added which can be used
    to tune the behaviour of malloc():
    
    AMQ_MALLOC_ARENA_SET - if this is set, then MQ will call
    mallopt(M_ARENA_MAX, 1) when MQ processes start, in order to
    force a single arena
    AMQ_MALLOC_ARENA_COUNT - this can be set to a number in order to
    tune the number of arenas in use by the mallopt() call.  The
    default value is 1 and the maximum value is 8.
    
    Performance testing did not show any degradation with the number
    of arenas set to 1 but users should run their own tests before
    committing this change in a production environment.
    
    ---------------------------------------------------------------
    The fix is targeted for delivery in the following PTFs:
    
    Version    Maintenance Level
    v8.0       8.0.0.8
    v9.0 CD    9.0.5
    v9.0 LTS   9.0.0.3
    
    The latest available maintenance can be obtained from
    'WebSphere MQ Recommended Fixes'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
    
    If the maintenance level is not yet available information on
    its planned availability can be found in 'WebSphere MQ
    Planned Maintenance Release Dates'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
    ---------------------------------------------------------------
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT20636

  • Reported component name

    IBM MQ APPL M20

  • Reported component ID

    5725S1400

  • Reported release

    800

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2017-05-17

  • Closed date

    2017-10-25

  • Last modified date

    2018-02-06

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    IBM MQ APPL M20

  • Fixed component ID

    5725S1400

Applicable component levels

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SS5K6E","label":"IBM MQ Appliance"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.0","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
06 February 2018