Checking for malloc heap contention when running Db2 on AIX

Condition

Intermittent performance degradation on (AIX®) might be observed in high concurrency scenarios due to malloc heap contention.

You might notice an overall slowdown on (AIX) when one or more of the following features are used during high concurrency scenarios:

  • Native encryption, including encryption of the database, transaction log files and backups.
  • Client/server communication that uses Transport Layer Security (TLS).
  • Authentication that uses the LDAP plug-ins when the connection to the LDAP server uses TLS.
  • The ENCRYPT, DECRYPT_BIN, and DECRYPT_CHAR built-in functions.
  • Db2 Text Search queries that use contains() or score().

Run the following command to determine whether you are suffering a performance hit from running the malloc function:

db2sysc_PID=`db2pd -edus | awk '/db2sysc PID/{print $3}'`
procstack $db2sysc_PID > procstack.db2sysc.txt

On the collected stack dump, you might see a statement where it is acquiring a global lock for the malloc and free functions. The greater the number of stacks containing the global lock for the malloc or free functions, the more contention there is.

Stack:
======================

 0x090000000054183C _global_lock_common + 0x4BC
 0x09000000000412C8 free_y + 0x108
...

Stack:
======================

0x090000000054183C _global_lock_common + 0x4BC
 0x090000000004341C malloc_y + 0x79C
...

Cause

When the default (AIX) memory allocation setting is used at higher concurrency levels, it can result in serialization during memory allocation, which can affect throughput.

The core of the IBM® DB2® database engine allocates private memory that uses IBM DB2’s own memory manager, which adds a layer of caching and high-performing concurrency management on top of the standard (AIX) malloc API. However, some products that are integrated within IBM DB2, such as IBM IBM Global Security Kit (GSKit), interface directly with (AIX) malloc and are subject to the performance capabilities of the (AIX) malloc manager setting.

Note: All versions of Db2® are affected by this performance hit. AIX provides an option to use multiple heaps to improve the performance of multi-threaded applications that use malloc-based memory allocation and deallocation.

Remedy

Procedure

  1. Tune (AIX) memory allocation by adding the following 'export' commands into sqllib/db2profile or sqllib/userprofile. 
    export MALLOCOPTIONS=buckets,multiheap:4
export MALLOCBUCKETS=number_of_buckets:128,bucket_sizing_factor:64,blocks_per_bucket:1024
  2. MALLOCBUCKETS setting increases the process heap size, which might cause malloc failures in environments where "ulimit -d" is set to a low value. The MALLOC setting is effective for all processes started from the shell of the instance owner, so one should set the limit for "data area" (ulimit -d) to unlimited (-1) for that user via /etc/security/limits.

    For example:

    default:
        fsize = 2097151
        core = 2097151
        cpu = -1
        data = 262144
        rss = 65536
        stack = 65536
        nofiles = 2000

db2inst1:
		data = -1
    Or, set ulimit directly with the following command:
    ulimit -d unlimited
  3. Finally, update the IBM DB2 registry and restart the instance: 
    db2set DB2ENVLIST="MALLOCOPTIONS MALLOCBUCKETS"
db2stop
db2start

    For applications that use TLS for connections to the database server, and that have many concurrent connections, the malloc settings must be configured within the application environment.

    The enhanced malloc options provide better performance for multi-threaded applications performing frequent malloc operations. Be aware of the following dependencies:
    • Enhanced malloc options require extra real memory, which varies depending on the environment.
    • Extra private virtual memory requirements demand that you to use caution when running the following serviceability tools: