APAR status
Closed as program error.
Error description
Applications using SMS system temporary table space will hit an SQLO_DISK (disk full) error when the file system becomes full, page cleaners running at the background trying to write out dirty pages of this temp table space will hit the disk full error as well, both of them will write error analysis for the disk full situation to db2diag.log. Depending on the number of applications and page cleaners, a bad contention to db2diag.log may arise between them, this contention may result in halted applications, and in turn, slow the process to release space back to file system. Stack dumps collected during this time, may show the following: 0x00002B6E802E3E3D __nanosleep + 0x002d 0x00002B6E9245AC95 ossLockGetConflict + 0x01c5 0x00002B6E82B0F3FD _ZN13PDLogFacility4openEbPcPKcPbP8PDLogFacP10PD_CONTEXTjjbmmP17P DDumpCleanupCB_t + 0x011d address: 0x00002B6E82ACEE4E ; dladdress: 0x00002B6E8089C000 ; offset in lib: 0x0000000002232E4E ; 0x00002B6E82AEF5CB pdLogSysRC + 0x071b 0x00002B6E899A3ED1 sqloSystemErrorHandler + 0x04f1 0x00002B6E89ABE21E sqloseekwrite64 + 0x07ae 0x00002B6E89ABD7A5 sqloWriteBlocks + 0x0245 0x00002B6E83081A0C _Z19sqlbWritePageToDiskP11SQLO_IO_REQP16SqlbOpenFileInfoP12SQLB_ GLOBALSjth + 0x043c 0x00002B6E8307BD93 sqlbWritePage + 0x10b3 0x00002B6E8307A77D _Z17sqlbWritePageSyncP8SQLB_BPDjbP15SQLB_BufferPoolP12SQLB_GLOBA LS + 0x008d address: 0x00002B6E8303BA05 ; dladdress: 0x00002B6E8089C000 ; offset in lib: 0x000000000279FA05 ; 0x00002B6E83039687 _Z14sqlbFreeUpSlotP11SQLB_FIX_CBPjPP8SQLB_BPDP13SQLB_PAGE_KEYm + 0x0667 0x00002B6E83037B0B _Z17sqlbGetVictimSlotP11SQLB_FIX_CBiPjPP8SQLB_BPDmP13SQLB_PAGE_K EY + 0x007b 0x00002B6E830346E0 _Z19sqlbGetPageFromDiskP11SQLB_FIX_CBi + 0x00b0 0x00002B6E8302D0C8 _Z16sqlbExtendObjectP12SQLB_GLOBALSP16SQLB_OBJECT_DESCjjjPFvP8sq eAgentP9SQLB_PAGEPvES7_jPjP11SQLB_FIX_CB + 0x1488 0x00002B6E83339100 _Z17sqldAppendTempRowP8sqeAgentPciiP8SQLZ_RIDP8SQLD_TCB + 0x0220 address: 0x00002B6E83335E4B ; dladdress: 0x00002B6E8089C000 ; offset in lib: 0x0000000002A99E4B ; 0x00002B6E8333246A _Z13sqldRowInsertP8sqeAgenttthmiPP10SQLD_VALUEP8SQLZ_RIDPPv + 0x01aa 0x00002B6E83E5CE5C _ZN7ibm_cde4data17DatabaseTempTable11writeBufferEPKvj + 0x015c . Sample messages in db2diag.log written by applications growing sys temp: EDUID : 1014 EDUNAME: db2agent (XXXXXX) FUNCTION: DB2 UDB, oper system services, sqloseekwrite64, probe:40 MESSAGE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full." DIA8312C Disk was full. CALLED : OS, -, pwrite OSERR: ENOSPC (28) DATA #1 : File handle, PD_TYPE_SQO_FILE_HDL, 8 bytes File Handle = 1330 File System Block Size = 4096 bytes File System Type = ext4 <..skipped..> EDUID : 1014 EDUNAME: db2agent (XXXXXXX) FUNCTION: DB2 Common, OSSe, ossErrorIOAnalysis, probe:100 CALLED : OS, -, pwrite OSERR: ENOSPC (28) DATA #1 : String, 148 bytes A total of 5 analysis will be performed : - User info - ulimit info - Target file info - I/O attempt - File system Target file handle = 1330 <..skipped..> CALLSTCK: (Static functions may not be resolved correctly, as they are resolved to the nearest symbol) [0] 0x00007FFFEBDA5DCB /UDB/apf01pi5/home/sqllib/lib64/libdb2osse.so.1 + 0x223DCB [1] 0x00007FFFEBDA7353 ossLogSysRC + 0xB3 [2] 0x00007FFFEBD991F4 /UDB/apf01pi5/home/sqllib/lib64/libdb2osse.so.1 + 0x2171F4 [3] 0x00007FFFEBD964BD ossErrorAnalysis + 0x2D [4] 0x00007FFFF249125B sqloSystemErrorHandler + 0x71B [5] 0x00007FFFF045CCAD sqloseekwrite64 + 0x69D [6] 0x00007FFFF045C3DD sqloWriteBlocks + 0x24D [7] 0x00007FFFED58F290 _Z19sqlbWritePageToDiskP11SQLO_IO_REQP16SqlbOpenFileInfoP12SQLB_ GLOBALSjth + 0x1F0 [8] 0x00007FFFED58DEEF sqlbWritePage + 0x74F [9] 0x00007FFFED58D1AB _Z17sqlbWritePageSyncP8SQLB_BPDjbP15SQLB_BufferPoolP12SQLB_GLOBA LS + 0x5B [10] 0x00007FFFF33A8E9C /UDB/apf01pi5/home/sqllib/lib64/libdb2e.so.1 + 0x70ABE9C [11] 0x00007FFFF33A8217 _Z14sqlbFreeUpSlotP11SQLB_FIX_CBPjPP8SQLB_BPDP13SQLB_PAGE_KEYm + 0x147 [12] 0x00007FFFF33A63A2 _Z17sqlbGetVictimSlotP11SQLB_FIX_CBiPjPP8SQLB_BPDmP13SQLB_PAGE_K EY + 0x92 [13] 0x00007FFFF33A3BC9 _Z19sqlbGetPageFromDiskP11SQLB_FIX_CBi + 0xC9 [14] 0x00007FFFED57544A _Z16sqlbExtendObjectP12SQLB_GLOBALSP16SQLB_OBJECT_DESCjjjPFvP8sq eAgentP9SQLB_PAGEPvES7_jPjP11SQLB_FIX_CB + 0xE3A [15] 0x00007FFFF33DADC9 _Z17sqldAppendTempRowP8sqeAgentPciiP8SQLZ_RIDP8SQLD_TCB + 0x1089 [16] 0x00007FFFF0A4B8B0 /UDB/apf01pi5/home/sqllib/lib64/libdb2e.so.1 + 0x474E8B0 [17] 0x00007FFFF0A4A803 _Z13sqlsSpillSortP8sqeAgentP10SQLS_SLDESP10SQLS_SORTSi + 0x293 [18] 0x00007FFFF0A49BA3 _Z8sqlswritP8sqeAgentP10SQLS_SLDESP10SQLS_SORTSi + 0x153 [19] 0x00007FFFF3508868 /UDB/apf01pi5/home/sqllib/lib64/libdb2e.so.1 + 0x720B868 [20] 0x00007FFFF2E05FD6 _Z18sqlsBufferedInsertP8sqeAgentPcS1_i + 0x2F6 [21] 0x00007FFFF082BC6C _Z8sqlrifisP8sqlrr_cb + 0xC2C [22] 0x00007FFFF0820D07 _Z14sqlriNljnPipedP8sqlrr_cb + 0x2A7 [23] 0x00007FFFF34E7CD1 _Z15sqlriSectInvokeP8sqlrr_cbP12sqlri_opparm + 0x1B1 [24] 0x00007FFFF34CBAB1 _Z29sqlrr_process_execute_requestP8sqlrr_cbib + 0x1701 [25] 0x00007FFFF34C8DFC _Z13sqlrr_executeP14db2UCinterfaceP9UCstpInfo + 0x26C [26] 0x00007FFFED452C99 _ZN10pvmPackage14executeSectionEP5sqlcajb + 0x439 [27] 0x00007FFFED451111 _ZN3PVM3runEv + 0xE21 [28] 0x00007FFFED45CC76 pvm_entry + 0x206 [29] 0x00007FFFF03FB19C sqloInvokeFnArgs + 0x700C [30] 0x00007FFFF34FAEAC _Z19sqlriInvokerTrustedP10sqlri_ufobP21sqlriRoutineErrorIntfb + 0x211C [31] 0x00007FFFF34F6BE7 _Z18sqlriInvokeInvokerP10sqlri_ufobb + 0x217 [32] 0x00007FFFF085487A _Z9sqlricallP8sqlrr_cb + 0xDA [33] 0x00007FFFF34E7CD1 _Z15sqlriSectInvokeP8sqlrr_cbP12sqlri_opparm + 0x1B1 [34] 0x00007FFFF34CBAB1 _Z29sqlrr_process_execute_requestP8sqlrr_cbib + 0x1701 [35] 0x00007FFFF34C8DFC _Z13sqlrr_executeP14db2UCinterfaceP9UCstpInfo + 0x26C [36] 0x00007FFFF340F176 _Z19sqljs_ddm_excsqlsttP14db2UCinterfaceP13sqljDDMObject + 0x4C6 [37] 0x00007FFFF340E6FE _Z21sqljsParseRdbAccessedP13sqljsDrdaAsCbP13sqljDDMObjectP14db2U Cinterface + 0x7E [38] 0x00007FFFEF78966B _Z10sqljsParseP13sqljsDrdaAsCbP14db2UCinterfaceP8sqeAgentb + 0x36B [39] 0x00007FFFEF783C6F /UDB/apf01pi5/home/sqllib/lib64/libdb2e.so.1 + 0x3486C6F [40] 0x00007FFFEF7820EC /UDB/apf01pi5/home/sqllib/lib64/libdb2e.so.1 + 0x34850EC [41] 0x00007FFFEF77F139 /UDB/apf01pi5/home/sqllib/lib64/libdb2e.so.1 + 0x3482139 [42] 0x00007FFFEF77ED2B _Z17sqljsDrdaAsDriverP18SQLCC_INITSTRUCT_T + 0xEB [43] 0x00007FFFEF47C6AF _ZN8sqeAgent6RunEDUEv + 0xACF [44] 0x00007FFFF0CB4C77 _ZN9sqzEDUObj9EDUDriverEv + 0xF7 [45] 0x00007FFFF0464D93 sqloEDUEntry + 0x303 [46] 0x00007FFFF7BC6AA1 /lib64/libpthread.so.0 + 0x7AA1 [47] 0x00007FFFEB136C4D clone + 0x6D Sample messages in db2diag.log written by page cleaners: EDUID : 551 EDUNAME: db2pclnr (XXXXXX) FUNCTION: DB2 UDB, oper system services, sqloLioAIOCollect, probe:100 MESSAGE : ZRC=0x850F000C=-2062614516=SQLO_DISK "Disk full." DIA8312C Disk was full. CALLED : OS, -, aio_return OSERR: ENOSPC (28) DATA #1 : File descriptor, 4 bytes 1330 DATA #2 : unsigned integer, 8 bytes 32768 DATA #3 : signed integer, 8 bytes 3173622972416 DATA #4 : String, 105 bytes Search for ossError*Analysis probe point after this log entry for further self-diagnosis of this problem. EDUID : 551 EDUNAME: db2pclnr (XXXXXX) FUNCTION: DB2 Common, OSSe, ossErrorIOAnalysis, probe:100 CALLED : OS, -, aio_return OSERR: ENOSPC (28) DATA #1 : String, 133 bytes A total of 4 analysis will be performed : - User info - ulimit info - Target file info - File system Target file handle = 1330 <..skipped..> CALLSTCK: (Static functions may not be resolved correctly, as they are resolved to the nearest symbol) [0] 0x00007FFFEBDA5DCB /UDB/apf01pi5/home/sqllib/lib64/libdb2osse.so.1 + 0x223DCB [1] 0x00007FFFEBDA7353 ossLogSysRC + 0xB3 [2] 0x00007FFFEBD991F4 /UDB/apf01pi5/home/sqllib/lib64/libdb2osse.so.1 + 0x2171F4 [3] 0x00007FFFEBD964BD ossErrorAnalysis + 0x2D [4] 0x00007FFFF249125B sqloSystemErrorHandler + 0x71B [5] 0x00007FFFF349D7BB sqloLioCollectNBlocks + 0x58B [6] 0x00007FFFF339B22D _Z18sqlbClnrWritePagesP12SQLB_CLNR_CB + 0xED [7] 0x00007FFFF33976BE _Z16sqlbClnrFindWorkP12SQLB_CLNR_CB + 0x182E [8] 0x00007FFFED518992 _Z18sqlbClnrEntryPointP12sqbPgClnrEdu + 0x102 [9] 0x00007FFFED5187F5 _ZN12sqbPgClnrEdu6RunEDUEv + 0x25 [10] 0x00007FFFF0CB4C77 _ZN9sqzEDUObj9EDUDriverEv + 0xF7 [11] 0x00007FFFF0464D93 sqloEDUEntry + 0x303 [12] 0x00007FFFF7BC6AA1 /lib64/libpthread.so.0 + 0x7AA1 [13] 0x00007FFFEB136C4D clone + 0x6D
Local fix
Force application off to release space. If some applications are not able to be forced off, restart db2 to release the contention and free temp space. As proactive measures, consider to use temp space thresholds and/or registry variables DB2_SMS_TRUNC_TMPTABLE_THRESH and/or DB2_OPT_MAX_TEMP_SIZE to control space consumption on sys temp table spaces.
Problem summary
**************************************************************** * USERS AFFECTED: * * All Users on DB2 v111m4FP6 and earlier * **************************************************************** * PROBLEM DESCRIPTION: * * See Error Description * **************************************************************** * RECOMMENDATION: * * Upgrade to DB2 v111m4FP6 or apply Local Fix if available * ****************************************************************
Problem conclusion
First Fixed in DB2 v111m4FP6
Temporary fix
Comments
APAR Information
APAR number
IT32593
Reported component name
DB2 FOR LUW
Reported component ID
DB2FORLUW
Reported release
B10
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2020-04-17
Closed date
2021-03-15
Last modified date
2021-03-15
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
DB2 FOR LUW
Fixed component ID
DB2FORLUW
Applicable component levels
[{"Line of Business":{"code":"LOB10","label":"Data and AI"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSEPGG","label":"Db2 for Linux, UNIX and Windows"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"11.1"}]
Document Information
Modified date:
16 March 2021