IBM Support

IT32593: DISK FULL ERROR ON SMS SYSTEM TEMPORARY TABLE SPACE MAY CAUSE BAD CONTENTION TO DB2DIAG.LOG BETWEEN PAGE CLEANER & APPLICATIONS

Subscribe to this APAR

By subscribing, you receive periodic emails alerting you to the status of the APAR, along with a link to the fix after it becomes available. You can track this item individually or track all items by product.

Notify me when this APAR changes.

Notify me when an APAR for this component changes.

 

APAR status

  • Closed as program error.

Error description

  • Applications using SMS system temporary table space will hit an
    SQLO_DISK (disk full) error when the file system becomes full,
    page cleaners running at the background trying to write out
    dirty pages of this temp table space will hit the disk full
    error as well, both of them will write error analysis for the
    disk full situation to db2diag.log.
    
    Depending on the number of applications and page cleaners, a bad
    contention to db2diag.log may arise between them, this
    contention may result in halted applications, and in
    turn, slow the process to release space back to file system.
    
    Stack dumps collected during this time, may show the following:
    
    0x00002B6E802E3E3D __nanosleep + 0x002d
    0x00002B6E9245AC95 ossLockGetConflict + 0x01c5
    0x00002B6E82B0F3FD
    _ZN13PDLogFacility4openEbPcPKcPbP8PDLogFacP10PD_CONTEXTjjbmmP17P
    DDumpCleanupCB_t + 0x011d
    address: 0x00002B6E82ACEE4E ; dladdress: 0x00002B6E8089C000 ;
    offset in lib: 0x0000000002232E4E ;
    0x00002B6E82AEF5CB pdLogSysRC + 0x071b
    0x00002B6E899A3ED1 sqloSystemErrorHandler + 0x04f1
    0x00002B6E89ABE21E sqloseekwrite64 + 0x07ae
    0x00002B6E89ABD7A5 sqloWriteBlocks + 0x0245
    0x00002B6E83081A0C
    _Z19sqlbWritePageToDiskP11SQLO_IO_REQP16SqlbOpenFileInfoP12SQLB_
    GLOBALSjth + 0x043c
    0x00002B6E8307BD93 sqlbWritePage + 0x10b3
    0x00002B6E8307A77D
    _Z17sqlbWritePageSyncP8SQLB_BPDjbP15SQLB_BufferPoolP12SQLB_GLOBA
    LS + 0x008d
    address: 0x00002B6E8303BA05 ; dladdress: 0x00002B6E8089C000 ;
    offset in lib: 0x000000000279FA05 ;
    0x00002B6E83039687
    _Z14sqlbFreeUpSlotP11SQLB_FIX_CBPjPP8SQLB_BPDP13SQLB_PAGE_KEYm +
    0x0667
    0x00002B6E83037B0B
    _Z17sqlbGetVictimSlotP11SQLB_FIX_CBiPjPP8SQLB_BPDmP13SQLB_PAGE_K
    EY + 0x007b
    0x00002B6E830346E0 _Z19sqlbGetPageFromDiskP11SQLB_FIX_CBi +
    0x00b0
    0x00002B6E8302D0C8
    _Z16sqlbExtendObjectP12SQLB_GLOBALSP16SQLB_OBJECT_DESCjjjPFvP8sq
    eAgentP9SQLB_PAGEPvES7_jPjP11SQLB_FIX_CB + 0x1488
    0x00002B6E83339100
    _Z17sqldAppendTempRowP8sqeAgentPciiP8SQLZ_RIDP8SQLD_TCB + 0x0220
    address: 0x00002B6E83335E4B ; dladdress: 0x00002B6E8089C000 ;
    offset in lib: 0x0000000002A99E4B ;
    0x00002B6E8333246A
    _Z13sqldRowInsertP8sqeAgenttthmiPP10SQLD_VALUEP8SQLZ_RIDPPv +
    0x01aa
    0x00002B6E83E5CE5C
    _ZN7ibm_cde4data17DatabaseTempTable11writeBufferEPKvj + 0x015c
    .
    
    Sample messages in db2diag.log written by applications growing
    sys temp:
    
    EDUID   : 1014                 EDUNAME: db2agent (XXXXXX)
    FUNCTION: DB2 UDB, oper system services, sqloseekwrite64,
    probe:40
    MESSAGE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full."
              DIA8312C Disk was full.
    CALLED  : OS, -, pwrite                           OSERR: ENOSPC
    (28)
    DATA #1 : File handle, PD_TYPE_SQO_FILE_HDL, 8 bytes
      File Handle              = 1330
      File System Block Size   = 4096 bytes
      File System Type         = ext4
    <..skipped..>
    
    EDUID   : 1014                 EDUNAME: db2agent (XXXXXXX)
    FUNCTION: DB2 Common, OSSe, ossErrorIOAnalysis, probe:100
    CALLED  : OS, -, pwrite                           OSERR: ENOSPC
    (28)
    DATA #1 : String, 148 bytes
    A total of 5 analysis will be performed :
     - User info
     - ulimit info
     - Target file info
     - I/O attempt
     - File system
    
     Target file handle = 1330
    <..skipped..>
    CALLSTCK: (Static functions may not be resolved correctly, as
    they are resolved to the nearest symbol)
      [0] 0x00007FFFEBDA5DCB
    /UDB/apf01pi5/home/sqllib/lib64/libdb2osse.so.1 + 0x223DCB
      [1] 0x00007FFFEBDA7353 ossLogSysRC + 0xB3
      [2] 0x00007FFFEBD991F4
    /UDB/apf01pi5/home/sqllib/lib64/libdb2osse.so.1 + 0x2171F4
      [3] 0x00007FFFEBD964BD ossErrorAnalysis + 0x2D
      [4] 0x00007FFFF249125B sqloSystemErrorHandler + 0x71B
      [5] 0x00007FFFF045CCAD sqloseekwrite64 + 0x69D
      [6] 0x00007FFFF045C3DD sqloWriteBlocks + 0x24D
      [7] 0x00007FFFED58F290
    _Z19sqlbWritePageToDiskP11SQLO_IO_REQP16SqlbOpenFileInfoP12SQLB_
    GLOBALSjth + 0x1F0
      [8] 0x00007FFFED58DEEF sqlbWritePage + 0x74F
      [9] 0x00007FFFED58D1AB
    _Z17sqlbWritePageSyncP8SQLB_BPDjbP15SQLB_BufferPoolP12SQLB_GLOBA
    LS + 0x5B
      [10] 0x00007FFFF33A8E9C
    /UDB/apf01pi5/home/sqllib/lib64/libdb2e.so.1 + 0x70ABE9C
      [11] 0x00007FFFF33A8217
    _Z14sqlbFreeUpSlotP11SQLB_FIX_CBPjPP8SQLB_BPDP13SQLB_PAGE_KEYm +
    0x147
      [12] 0x00007FFFF33A63A2
    _Z17sqlbGetVictimSlotP11SQLB_FIX_CBiPjPP8SQLB_BPDmP13SQLB_PAGE_K
    EY + 0x92
      [13] 0x00007FFFF33A3BC9 _Z19sqlbGetPageFromDiskP11SQLB_FIX_CBi
    + 0xC9
      [14] 0x00007FFFED57544A
    _Z16sqlbExtendObjectP12SQLB_GLOBALSP16SQLB_OBJECT_DESCjjjPFvP8sq
    eAgentP9SQLB_PAGEPvES7_jPjP11SQLB_FIX_CB + 0xE3A
      [15] 0x00007FFFF33DADC9
    _Z17sqldAppendTempRowP8sqeAgentPciiP8SQLZ_RIDP8SQLD_TCB + 0x1089
      [16] 0x00007FFFF0A4B8B0
    /UDB/apf01pi5/home/sqllib/lib64/libdb2e.so.1 + 0x474E8B0
      [17] 0x00007FFFF0A4A803
    _Z13sqlsSpillSortP8sqeAgentP10SQLS_SLDESP10SQLS_SORTSi + 0x293
      [18] 0x00007FFFF0A49BA3
    _Z8sqlswritP8sqeAgentP10SQLS_SLDESP10SQLS_SORTSi + 0x153
      [19] 0x00007FFFF3508868
    /UDB/apf01pi5/home/sqllib/lib64/libdb2e.so.1 + 0x720B868
      [20] 0x00007FFFF2E05FD6 _Z18sqlsBufferedInsertP8sqeAgentPcS1_i
    + 0x2F6
      [21] 0x00007FFFF082BC6C _Z8sqlrifisP8sqlrr_cb + 0xC2C
      [22] 0x00007FFFF0820D07 _Z14sqlriNljnPipedP8sqlrr_cb + 0x2A7
      [23] 0x00007FFFF34E7CD1
    _Z15sqlriSectInvokeP8sqlrr_cbP12sqlri_opparm + 0x1B1
      [24] 0x00007FFFF34CBAB1
    _Z29sqlrr_process_execute_requestP8sqlrr_cbib + 0x1701
      [25] 0x00007FFFF34C8DFC
    _Z13sqlrr_executeP14db2UCinterfaceP9UCstpInfo + 0x26C
      [26] 0x00007FFFED452C99
    _ZN10pvmPackage14executeSectionEP5sqlcajb + 0x439
      [27] 0x00007FFFED451111 _ZN3PVM3runEv + 0xE21
      [28] 0x00007FFFED45CC76 pvm_entry + 0x206
      [29] 0x00007FFFF03FB19C sqloInvokeFnArgs + 0x700C
      [30] 0x00007FFFF34FAEAC
    _Z19sqlriInvokerTrustedP10sqlri_ufobP21sqlriRoutineErrorIntfb +
    0x211C
      [31] 0x00007FFFF34F6BE7 _Z18sqlriInvokeInvokerP10sqlri_ufobb +
    0x217
      [32] 0x00007FFFF085487A _Z9sqlricallP8sqlrr_cb + 0xDA
      [33] 0x00007FFFF34E7CD1
    _Z15sqlriSectInvokeP8sqlrr_cbP12sqlri_opparm + 0x1B1
      [34] 0x00007FFFF34CBAB1
    _Z29sqlrr_process_execute_requestP8sqlrr_cbib + 0x1701
      [35] 0x00007FFFF34C8DFC
    _Z13sqlrr_executeP14db2UCinterfaceP9UCstpInfo + 0x26C
      [36] 0x00007FFFF340F176
    _Z19sqljs_ddm_excsqlsttP14db2UCinterfaceP13sqljDDMObject + 0x4C6
      [37] 0x00007FFFF340E6FE
    _Z21sqljsParseRdbAccessedP13sqljsDrdaAsCbP13sqljDDMObjectP14db2U
    Cinterface + 0x7E
      [38] 0x00007FFFEF78966B
    _Z10sqljsParseP13sqljsDrdaAsCbP14db2UCinterfaceP8sqeAgentb +
    0x36B
      [39] 0x00007FFFEF783C6F
    /UDB/apf01pi5/home/sqllib/lib64/libdb2e.so.1 + 0x3486C6F
      [40] 0x00007FFFEF7820EC
    /UDB/apf01pi5/home/sqllib/lib64/libdb2e.so.1 + 0x34850EC
      [41] 0x00007FFFEF77F139
    /UDB/apf01pi5/home/sqllib/lib64/libdb2e.so.1 + 0x3482139
      [42] 0x00007FFFEF77ED2B
    _Z17sqljsDrdaAsDriverP18SQLCC_INITSTRUCT_T + 0xEB
      [43] 0x00007FFFEF47C6AF _ZN8sqeAgent6RunEDUEv + 0xACF
      [44] 0x00007FFFF0CB4C77 _ZN9sqzEDUObj9EDUDriverEv + 0xF7
      [45] 0x00007FFFF0464D93 sqloEDUEntry + 0x303
      [46] 0x00007FFFF7BC6AA1 /lib64/libpthread.so.0 + 0x7AA1
      [47] 0x00007FFFEB136C4D clone + 0x6D
    
    
    Sample messages in db2diag.log written by page cleaners:
    
    EDUID   : 551                  EDUNAME: db2pclnr (XXXXXX)
    FUNCTION: DB2 UDB, oper system services, sqloLioAIOCollect,
    probe:100
    MESSAGE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full."
              DIA8312C Disk was full.
    CALLED  : OS, -, aio_return                       OSERR: ENOSPC
    (28)
    DATA #1 : File descriptor, 4 bytes
    1330
    DATA #2 : unsigned integer, 8 bytes
    32768
    DATA #3 : signed integer, 8 bytes
    3173622972416
    DATA #4 : String, 105 bytes
    Search for ossError*Analysis probe point after this log entry
    for further
    self-diagnosis of this problem.
    
    EDUID   : 551                  EDUNAME: db2pclnr (XXXXXX)
    FUNCTION: DB2 Common, OSSe, ossErrorIOAnalysis, probe:100
    CALLED  : OS, -, aio_return                       OSERR: ENOSPC
    (28)
    DATA #1 : String, 133 bytes
    A total of 4 analysis will be performed :
     - User info
     - ulimit info
     - Target file info
     - File system
    
     Target file handle = 1330
    <..skipped..>
    CALLSTCK: (Static functions may not be resolved correctly, as
    they are resolved to the nearest symbol)
      [0] 0x00007FFFEBDA5DCB
    /UDB/apf01pi5/home/sqllib/lib64/libdb2osse.so.1 + 0x223DCB
      [1] 0x00007FFFEBDA7353 ossLogSysRC + 0xB3
      [2] 0x00007FFFEBD991F4
    /UDB/apf01pi5/home/sqllib/lib64/libdb2osse.so.1 + 0x2171F4
      [3] 0x00007FFFEBD964BD ossErrorAnalysis + 0x2D
      [4] 0x00007FFFF249125B sqloSystemErrorHandler + 0x71B
      [5] 0x00007FFFF349D7BB sqloLioCollectNBlocks + 0x58B
      [6] 0x00007FFFF339B22D _Z18sqlbClnrWritePagesP12SQLB_CLNR_CB +
    0xED
      [7] 0x00007FFFF33976BE _Z16sqlbClnrFindWorkP12SQLB_CLNR_CB +
    0x182E
      [8] 0x00007FFFED518992 _Z18sqlbClnrEntryPointP12sqbPgClnrEdu +
    0x102
      [9] 0x00007FFFED5187F5 _ZN12sqbPgClnrEdu6RunEDUEv + 0x25
      [10] 0x00007FFFF0CB4C77 _ZN9sqzEDUObj9EDUDriverEv + 0xF7
      [11] 0x00007FFFF0464D93 sqloEDUEntry + 0x303
      [12] 0x00007FFFF7BC6AA1 /lib64/libpthread.so.0 + 0x7AA1
      [13] 0x00007FFFEB136C4D clone + 0x6D
    

Local fix

  • Force application off to release space.  If some applications
    are not able to be forced off, restart db2 to release the
    contention and free temp space.
    As proactive measures, consider to use temp space thresholds
    and/or
    registry variables
    DB2_SMS_TRUNC_TMPTABLE_THRESH and/or DB2_OPT_MAX_TEMP_SIZE to
    control space consumption on sys temp table spaces.
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:                                              *
    * All Users on DB2 v111m4FP6 and earlier                       *
    ****************************************************************
    * PROBLEM DESCRIPTION:                                         *
    * See Error Description                                        *
    ****************************************************************
    * RECOMMENDATION:                                              *
    * Upgrade to DB2 v111m4FP6 or apply Local Fix if available     *
    ****************************************************************
    

Problem conclusion

  • First Fixed in DB2 v111m4FP6
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT32593

  • Reported component name

    DB2 FOR LUW

  • Reported component ID

    DB2FORLUW

  • Reported release

    B10

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2020-04-17

  • Closed date

    2021-03-15

  • Last modified date

    2021-03-15

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

    IT32650

Fix information

  • Fixed component name

    DB2 FOR LUW

  • Fixed component ID

    DB2FORLUW

Applicable component levels

[{"Line of Business":{"code":"LOB10","label":"Data and AI"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSEPGG","label":"Db2 for Linux, UNIX and Windows"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"11.1"}]

Document Information

Modified date:
16 March 2021