Troubleshooting
Problem
Intermittent performance degradation on AIX might be observed in high concurrency scenarios due to malloc heap contention.
Symptom
You might notice an overall slowdown on AIX when one or more of the following features are used during high concurrency scenarios:
- Native Encryption, including encryption of the database, transaction log files and backups
- Client/server Communication that uses SSL
- Authentication using the LDAP plug-ins when the connection to the LDAP server uses SSL
- ENCRYPT/DECRYPT_BIN/DECRYPT_CHAR built-in functions
- Db2 Text Search queries that use contains() or score()
db2sysc_PID=`db2pd -edus | awk '/db2sysc PID/{print $3}'`
procstack $db2sysc_PID > procstack.db2sysc.txt
Stack:
======================
0x090000000054183C _global_lock_common + 0x4BC
0x09000000000412C8 free_y + 0x108
...
Stack:
======================
0x090000000054183C _global_lock_common + 0x4BC
0x090000000004341C malloc_y + 0x79C
...
Cause
The core of the DB2 database engine allocates private memory using DB2’s own memory manager, which adds a layer of caching and high-performing concurrency management on top of the standard AIX malloc API. However, some products integrated within DB2, such as IBM GSKit, interface directly with AIX malloc and are subject to the performance capabilities of the AIX malloc manager setting.
Environment
Resolving The Problem
AIX provides an option to use multiple heaps to improve the performance of multi-threaded applications that use malloc-based memory allocation and deallocation.
1. Tune AIX memory allocation by adding the following 'export' commands into sqllib/db2profile or sqllib/userprofile.
export MALLOCOPTIONS=buckets,multiheap:4
export MALLOCBUCKETS=number_of_buckets:128,bucket_sizing_factor:64,blocks_per_bucket:1024
2. MALLOCBUCKETS setting increases the process heap size, which might cause malloc failures in environments where "ulimit -d" is set to a low value. The MALLOC setting is effective for all processes started from the shell of the instance owner, so one should set the limit for "data area" (ulimit -d) to unlimited (-1) for that user via /etc/security/limits
For example:
default:
fsize = 2097151
core = 2097151
cpu = -1
data = 262144
rss = 65536
stack = 65536
nofiles = 2000
db2inst1:
data = -1
Or, set ulimit directly with the following command:
ulimit -d unlimited
3. Finally, update the DB2 registry and restart the instance:
db2set DB2ENVLIST="MALLOCOPTIONS MALLOCBUCKETS"
db2stop
db2start
For applications that use SSL for connections to the database server, and have a many concurrent connections, the malloc settings will also need to be configured within the application environment.
The enhanced malloc options provide better performance for multi-threaded applications performing frequent malloc operations. Users should be aware of the following:
1. Enhanced malloc options require additional real memory, which will vary depending on the environment.
2. Due to additional private virtual memory requirements, the following serviceability tools should be used with extra caution:
DB2: db2pd -memblocks pid=<process id> ( https://www.ibm.com/support/pages/node/480925 )
AIX: gencore
Related Information
Was this topic helpful?
Document Information
Modified date:
25 June 2024
UID
ibm10741831