Troubleshooting
Problem
The server no longer responds to commands and client backups fail. The server must be restarted for operations to resume.
Symptom
The server activity log shows the following error prior to being unresponsive :
ANR0171I dbiconn.c(1844): Error detected on 0:xxx, database in evaluation mode. (SESSION: nnn)
Diagnosing The Problem
Examine the instance db2diag.x.log at around the time of the failure. For example :
<timestamp>.990535-300 E96097030A5757 LEVEL: Error (OS)
PID : 8257674 TID : 1224927 PROC : db2sysc 0
INSTANCE: tsminst1 NODE : 000 DB : TSMDB1
APPHDL : 0-48134 APPID: 127.0.0.1.48266.160212195300
AUTHID : TSMINST1 HOSTNAME: axpmgtsm01
EDUID : 1224927 EDUNAME: db2agent (TSMDB1) 0
FUNCTION: DB2 UDB, SQO Memory Management, sqloLogMemoryCondition, probe:100
CALLED : OS, -, malloc
OSERR : ENOMEM (12) "Not enough space"
MESSAGE : Private memory and/or virtual address space exhausted,
or data ulimit exceeded
DATA #1 : Soft data resource limit, PD_TYPE_RLIM_DATA_CUR, 8 bytes
9223372036854775807
DATA #2 : Requested size, PD_TYPE_MEM_REQUESTED_SIZE, 8 bytes
1052672
DATA #3 : Current set size, PD_TYPE_SET_SIZE, 8 bytes
4207280128
CALLSTCK: (Static functions may not be resolved correctly, as they are
resolved to the nearest symbol)
[0] 0x090000000AE38844 sqloLogMemoryCondition + 0x1B8
[1] 0x090000000DD0D3C8 sqlogmblkEx + 0xC4
...
<timestamp>.992155-300 E96102788A764 LEVEL: Warning
PID : 8257674 TID : 1224927 PROC : db2sysc 0
INSTANCE: tsminst1 NODE : 000 DB : TSMDB1
APPHDL : 0-48134 APPID: 127.0.0.1.48266.160212195300
AUTHID : TSMINST1 HOSTNAME: axpmgtsm01
EDUID : 1224927 EDUNAME: db2agent (TSMDB1) 0
FUNCTION: DB2 UDB, SQO Memory Management, sqloMemLogPoolConditions, probe:30
DATA #1 : <preformatted>
Out of memory failure for Private Heap on node 0.
Requested block size : 37808 bytes.
Physical heap size : 393216 bytes.
Configured heap size : 1099511627776 bytes.
Unreserved memory used by heap : 0 bytes.
Unreserved memory left in set : 0 bytes.
<timestamp>.538935-300 E96103553A493 LEVEL: Error (OS)
PID : 8257674 TID : 2828 PROC : db2sysc 0
INSTANCE: tsm1mtl NODE : 000
HOSTNAME: axpmgtsm01
EDUID : 2828 EDUNAME: db2tcpcm 0
FUNCTION: DB2 UDB, oper system services, sqloSpawnEDU, probe:80
MESSAGE : ZRC=0x8300000B=-2097151989
CALLED : OS, -, pthread_create
OSERR : EAGAIN (11) "Resource temporarily unavailable"
DATA #1 : Codepath, 8 bytes
5:10:25:35
Resolving The Problem
Ensure that the AIX ulimits are set to unlimited in the script that is used to start the server instance. For example :
ulimit -c unlimited
ulimit -f unlimited
ulimit -m unlimited
ulimit -n unlimited
ulimit -t unlimited
ulimit -d unlimited
ulimit -u unlimited
ulimit -r unlimited
Also, upgrade the Tivoli Storage Manager server to the latest supported level to get fixes for known memory leaks, such as apar IT11271.
Was this topic helpful?
Document Information
Modified date:
17 June 2018
UID
swg21977052