IZ62724: DATABASE CRASHES DUE TO MEMORY CORRUPTION IN STATEMENT HEAP

DB2 Version 9.5 Fix Pack 6a for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 7 for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 8 for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 9 for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 10 for Linux, UNIX, and Windows

APAR status

• Closed as duplicate of another APAR.

Error description

• db2 may trap due to memory corruption in statement heap while
  compiling a SQL if the Real Time Statistics(RTS) is turned ON,
  i.e
  
  Automatic maintenance                      (AUTO_MAINT) = ON
     Automatic table maintenance          (AUTO_TBL_MAINT) = ON
       Automatic runstats                  (AUTO_RUNSTATS) = ON
         Automatic statement statistics  (AUTO_STMT_STATS) = ON
  
  When the crash occurs, the db2diag.log may show the memory
  corruption messages as follows:
  
  2009-09-20-10.37.15.347782+000 E55366A1372        LEVEL: Severe
  
  PID     : 1318934              TID  : 23850       PROC : db2sysc
  0
  INSTANCE: db2inst1        NODE : 000         DB   : DB1
  APPHDL  : 0-980                APPID:
  *LOCAL.db2inst1.090920103627
  
  AUTHID  : db2inst1
  EDUID   : 23850                EDUNAME: db2agent (DB1) 0
  
  FUNCTION: DB2 UDB, SQO Memory Management,
  sqloDiagnoseFreeBlockFailure, probe:999
  
  MESSAGE : Memory validation failure, diagnostic file dumped.
  
  DATA #1 : String, 28 bytes
  
  Corrupt pool free tree node.
  
  DATA #2 : File name, 33 bytes
  
  1318934.23850.mem_diagnostics.txt
  
  CALLSTCK:
  
    [0] 0x090000000ACBF128 pdLog + 0x88
  
    [1] 0x090000000AA6E8E8
  diagnoseMemoryCorruptionAndCrash__13SQLO_MEM_POOLFUlCPCc + 0x250
  
    [2] 0x090000000AA6E5BC
  diagnoseMemoryCorruptionAndCrash__13SQLO_MEM_POOLFUlCPCc@glue693
  + 0x74
    [3] 0x090000000AE2575C
  .MemTreePut__13SQLO_MEM_POOLFP8SMemNodeUlP17SqloChunkSubgroup_fd
  prpro_clone_83
  + 0xC
    [4] 0x090000000AE2A328 sqlofmblkEx + 0x244
  
    [5] 0x090000000AEAFBC0 sqlofmblkEx@glue687 + 0x74
  
    [6] 0x090000000AEAF738 set_fcsptrs__9sqlnq_ftbFCP9sqlnq_fcs +
  0x280
    [7] 0x090000000AEAF0BC __ct__9sqlnq_fcsFP9sqlnq_ftbiRiP3locb +
  0x1DC
    [8] 0x090000000AEAE9E4
  add_column__9sqlnq_ftbFPiPP9sqlnq_fcsP3loc + 0xB8
  
    [9] 0x090000000AEAA0E0
  add_head_exp__9sqlnq_qtbFP9sqlnq_pidiP12sqlnq_stringPUcPiPP9sqln
  q_qclPP9sqlnq_expT2b
  + 0xD0
  
  and the stack of the agent thread which reports the corruption
  might look like:
  
  0x090000000AA6F344 sqloCrashOnCriticalMemoryValidationFailure +
  0x1C
  0x090000000AA6E8F0
  diagnoseMemoryCorruptionAndCrash__13SQLO_MEM_POOLFUlCPCc + 0x258
  0x090000000AA6E5BC
  diagnoseMemoryCorruptionAndCrash__13SQLO_MEM_POOLFUlCPCc@glue693
  + 0x74
  0x090000000AE2575C
  .MemTreePut__13SQLO_MEM_POOLFP8SMemNodeUlP17SqloChunkSubgroup_fd
  prpro_clone_83
  + 0xC
  0x090000000AE2A328 sqlofmblkEx + 0x244
  0x090000000AEAFBC0 sqlofmblkEx@glue687 + 0x74
  0x090000000AEAF738 set_fcsptrs__9sqlnq_ftbFCP9sqlnq_fcs + 0x280
  0x090000000AEAF0BC __ct__9sqlnq_fcsFP9sqlnq_ftbiRiP3locb + 0x1DC
  0x090000000AEAE9E4 add_column__9sqlnq_ftbFPiPP9sqlnq_fcsP3loc +
  0xB8
  0x090000000AEAA0E0
  add_head_exp__9sqlnq_qtbFP9sqlnq_pidiP12sqlnq_stringPUcPiPP9sqln
  q_qclPP9sqlnq_expT2b
  + 0xD0
  0x090000000AEA942C
  __ct__9sqlnq_hxpFP9sqlnq_oprP9sqlnq_quniT3P12sqlnq_stringPUcPiPP
  9sqlnq_qclRiP3loc
  + 0x204
  0x090000000AEA4BB0
  add2output__9sqlnq_qncFP9sqlnq_opriPP9sqlnq_qcl15sqlnq_usageType
  - 0x224
  0x090000000AEA45C0
  sqlnq_handle_qnc_sel_star__FP9sqlnq_qunbT2P9sqlnq_oprPiP9sqlnq_q
  urN42
  + 0x474
  0x090000000AECC978
  sqlnq_sel_star__FP9sqlnq_oprbP9sqlnq_qunPiP9sqlnq_qurN22 + 0x210
  0x090000000AECC664
  sqlnq_sem_sel_star__FPP8stknode_i10actiontypePUcP3loc + 0xB8
  0x090000000AE837A4 sqlnp_smactn__FP8sqlnp_cbi + 0x320
  0x090000000AE838C8 sqlnp_parser__FP8sqlnp_cb + 0x7C
  0x090000000AF306A8 sqlnp_main__FP12sqlnq_stringsPP9sqlnq_qur +
  0x33C
  0x090000000AF2DFCC
  sqlnn_cmpl__FP8sqeAgentP11sqlrrstrings17sqlnn_compileModeT3P14sq
  lrr_cmpl_enviT7PP9sqlnq_qur
  + 0x828
  0x090000000AEDFAA4
  sqlnn_cmpl__FP8sqeAgentP11sqlrrstrings17sqlnn_compileModeT3P14sq
  lrr_cmpl_env
  + 0x24
  0x090000000AEDC7A4
  sqlra_compile_var__FP8sqlrr_cbP14sqlra_cmpl_envPUciUsN54P16sqlra
  _cached_varPiPUl
  + 0x60C
  0x090000000AF845C8
  sqlra_find_var__FP8sqlrr_cbP17sqlra_cached_stmt13sqlra_stmt_idUi
  T4PUcT4UsUcP14sqlra_cmpl_enviPiT11_T12_N311_T12_P14SQLP_LOCK_INF
  OPP16sqlra_cached_varT12_PUlb
  + 0x400
  0x090000000ACDF750 sqlra_get_var__FP8sqlrr_cbiT2bPbT5 + 0x440
  0x090000000CBE516C
  sqlrr_prepare__FP14db2UCinterfaceP16db2UCprepareInfo + 0x138
  0x090000000B0A9158
  sqljs_ddm_prpsqlstt__FP14db2UCinterfaceP13sqljDDMObject + 0x428
  0x090000000A52B3EC
  sqljsParseRdbAccessed__FP13sqljsDrdaAsCbP13sqljDDMObjectP14db2UC
  interface
  + 0x10C
  0x090000000AD981FC
  .sqljsParse__FP13sqljsDrdaAsCbP14db2UCinterface_fdprpro_clone_21
  6
  + 0x278
  
  and the corrupted memory block might look like:
  
  07000013008ffe88 : -- BLOCK: code: 0xdb2cafe0 size: 24 file:
  sqlnq_ftb.C line: 3962 seq: 30 CSG: 0xfab07000013008f0 --
  -- CORRUPT NODE --
  07000013008ffec8 : 00 00 00 00 00 00 00 0d 00 00 00 05 00 00 00
  00 ................
  07000013008ffed8 : ff ff ff ff ff ff ff ff 00 00 00 00 00 00 00
  0e ................
  07000013008fff68 : -- BLOCK: code: 0xdb2cafe0 size: 96 file:
  sqlnq_ftb.C line: 7003 seq: 64 CSG: 0xfab07000013008f0 --
  

Local fix

• This issue only exists if Real Time Statistics(RTS) is turned
  on, i.e
  
  Automatic maintenance      (AUTO_MAINT) = ON
     Automatic table maintenance  (AUTO_TBL_MAINT) = ON
       Automatic runstats    (AUTO_RUNSTATS) = ON
         Automatic statement statistics  (AUTO_STMT_STATS) = ON
  
  To  workaround, you can turn off the RTS under the set of above
  db cfg.
  

Problem summary

Problem conclusion

Temporary fix

Comments

• This APAR is a duplicate of IZ58996
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  IC65793

Fix information

Applicable component levels