IBM Support

IT22260: INSTANCE CRASH AFTER RUNNING OUT OF MEMORY

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • Db2 instance crashed on one partition after running out of
    memory with the following stack:
    
    CALLSTCK: (Static functions may not be resolved correctly, as
    they are resolved to the nearest symbol)
      [0] 0x00007FEBD152AEEC sqloPGRPRegisterOneCrash + 0x20C
      [1] 0x00007FEBD167EDEF sqloEDUCodeTrapHandler + 0xB2F
      [2] 0x00007FEBD8EED7E0 /lib64/libpthread.so.0 + 0xF7E0
      [3] 0x00007FEBCFF9F742 _Z8sqlkqopnP17sqlkqBqsOpenParms + 0xC12
      [4] 0x00007FEBCFFBC65C
    _Z8sqlktopnP8sqeAgentP9sqlri_taoP9SQLE_KRCB + 0x12AC
      [5] 0x00007FEBD2213F9B _Z8sqlriopnP8sqlrr_cbP9sqlri_taoPi +
    0x44B
      [6] 0x00007FEBD2399C2D _Z8sqlritqbP8sqlrr_cb + 0x22D
      [7] 0x00007FEBD22A5017
    _Z15sqlriSectInvokeP8sqlrr_cbP12sqlri_opparm + 0x4B7
      [8] 0x00007FEBD1CE21C9
    _Z21sqlrr_subagent_routerP8sqeAgentP12SQLE_DB2RA_T + 0xDE9
      [9] 0x00007FEBCF87761F
    /db2home/db2edwp1/sqllib/lib64/libdb2e.so.1 + 0x4FDF61F
      [10] 0x00007FEBCF875EF0 _Z21sqleProcessSubRequestP8sqeAgent +
    0x3E0
      [11] 0x00007FEBCF8AA477 _ZN8sqeAgent6RunEDUEv + 0x5D7
      [12] 0x00007FEBD30C2FB6 _ZN9sqzEDUObj9EDUDriverEv + 0x116
      [13] 0x00007FEBD167B5B8 sqloEDUEntry + 0x578
      [14] 0x00007FEBD8EE5AA1 /lib64/libpthread.so.0 + 0x7AA1
      [15] 0x00007FEBC9916BCD clone + 0x6D
    
    Messages in db2diag.log will be similar to:
    
    2017-09-04-04.30.39.732652+000 E110068359E706        LEVEL:
    Warning
    PID     : 25249                TID : 140471707887360 PROC :
    db2sysc 0
    INSTANCE: db2inst1             NODE : 000            DB   :
    SAMPLE
    APPHDL  : 0-15638              APPID:
    10.55.76.184.51809.170904030603
    AUTHID  : AUTHID             HOSTNAME: partition1
    EDUID   : 165886               EDUNAME: db2agnts (SAMPLE) 0
    FUNCTION: DB2 UDB, SQO Memory Management,
    SqloMemController::getPartitionStats, probe:10
    DATA #1 : <preformatted>
    Instance Memory Controller statistics.
    Instance Memory Automatic: No.
    Maximum size = 118111600 KB
    Current size = 118111424 KB
    Usage HWM = 118111552 KB
    Cached memory = 216384 KB
    Cached DB memory = 0 KB
    
    2017-09-04-04.30.39.748838+000 I110072547E1086       LEVEL:
    Warning
    PID     : 25249                TID : 140471707887360 PROC :
    db2sysc 0
    INSTANCE: db2inst1             NODE : 000            DB   :
    SAMPLE
    APPHDL  : 0-15638              APPID:
    10.55.76.184.51809.170904030603
    AUTHID  : AUTHID             HOSTNAME: partition1
    EDUID   : 165886               EDUNAME: db2agnts (SAMPLE) 0
    FUNCTION: DB2 UDB, SQO Memory Management,
    SqloMemController::requestMemory, probe:50
    MESSAGE : ZRC=0x8B0F0000=-1961951232=SQLO_NOMEM "No Memory
    Available"
              DIA8300C A memory heap error has occurred.
    DATA #1 : String, 36 bytes
    OOM - Instance memory request failed
    DATA #2 : String, 35 bytes
    Logging disabled until next success
    DATA #3 : unsigned integer, 8 bytes
    1048576
    DATA #4 : unsigned integer, 8 bytes
    0
    DATA #5 : String, 13 bytes
    APPL-SAMPLE
    DATA #6 : unsigned integer, 8 bytes
    9202171904
    DATA #7 : unsigned integer, 8 bytes
    215547904
    DATA #8 : unsigned integer, 8 bytes
    120946098176
    DATA #9 : unsigned integer, 8 bytes
    120946278400
    DATA #10: unsigned integer, 8 bytes
    215547904
    
    2017-09-04-04.30.42.489568+000 E111194474E766        LEVEL:
    Warning
    PID     : 25249                TID : 140471854688000 PROC :
    db2sysc 0
    INSTANCE: db2inst1             NODE : 000            DB   :
    SAMPLE
    APPHDL  : 0-26595              APPID:
    10.120.16.118.50278.171112235219
    AUTHID  : EDUVDM3P             HOSTNAME: partition1
    EDUID   : 166932               EDUNAME: db2agent (SAMPLE) 0
    FUNCTION: DB2 UDB, SQO Memory Management,
    sqloMemLogPoolConditions, probe:30
    DATA #1 : <preformatted>
    Out of memory failure for Statement Heap on node 0.
    Requested block size           : 888 bytes.
    Physical heap size             : 0 bytes.
    Configured heap size           : 327680000 bytes.
    Unreserved memory used by heap : 0 bytes.
    Unreserved memory left in set  : 0 bytes.
    
    2017-09-04-04.36.13.550198+000 I124014097E1036       LEVEL:
    Severe
    PID     : 25249                TID : 140496517195520 PROC :
    db2sysc 0
    INSTANCE: db2inst1             NODE : 000            DB   :
    SAMPLE
    APPHDL  : 0-46106              APPID:
    10.120.16.120.60306.171017052444
    AUTHID  : EDUCMP1P             HOSTNAME: partition1
    EDUID   : 167545               EDUNAME: db2agnts (SAMPLE) 0
    FUNCTION: DB2 UDB, relation data serv, sqlrr_dump_ffdc,
    probe:250
    MESSAGE : ZRC=0x87120007=-2028863481=SQLR_SEVERE_PGM_ERROR
              "Severe programming error"
              DIA8516C A severe internal processing error has
    occurred.
    DATA #1 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes
    sqlcaid : SQLCA     sqlcabc: 136   sqlcode: 0   sqlerrml: 0
    sqlerrmc:
    sqlerrp : SQL11012
    sqlerrd : (1) 0x00000000      (2) 0x00000000      (3) 0x00000000
               (4) 0x00000000      (5) 0x00000000      (6)
    0x00000000
    sqlwarn : (1)      (2)      (3)      (4)        (5)       (6)
               (7)      (8)      (9)      (10)        (11)
    sqlstate:
    
    2017-09-04-04.41.09.321155+000 E125395774E610        LEVEL:
    Severe
    PID     : 25242                TID : 139940373456640 PROC :
    db2wdog 0 [db2inst1]
    INSTANCE: db2inst1             NODE : 000
    HOSTNAME: partition1
    EDUID   : 2                    EDUNAME: db2wdog 0 [db2inst1]
    FUNCTION: DB2 UDB, base sys utilities, sqleWatchDog, probe:20
    MESSAGE : ADM0503C  An unexpected internal processing error has
    occurred. All
              database manager processes associated with this
    instance have been
              shutdown. Diagnostic information has been recorded.
    Contact IBM
              Support for further assistance.
    
    This was caused by poor error handling in Db2.
    

Local fix

  • N/A
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:                                              *
    * ALL                                                          *
    ****************************************************************
    * PROBLEM DESCRIPTION:                                         *
    * See Problem Description above.                               *
    ****************************************************************
    * RECOMMENDATION:                                              *
    * Upgrade to DB2 Version 11.1.3 Fix Pack 3                     *
    ****************************************************************
    

Problem conclusion

  • First fixed in DB2 Version 11.1.3 Fix Pack 3
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT22260

  • Reported component name

    DB2 FOR LUW

  • Reported component ID

    DB2FORLUW

  • Reported release

    B10

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2017-09-04

  • Closed date

    2018-03-16

  • Last modified date

    2019-12-16

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    DB2 FOR LUW

  • Fixed component ID

    DB2FORLUW

Applicable component levels

  • RB10 PSN

       UP

[{"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSEPGG","label":"DB2 for Linux, UNIX and Windows"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"11.1","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
16 December 2019