IV95270: DB2 IS DOWN WHEN AIO FAILED TO COLLECT ALL OUTSTANDING IO REQUESTS.

Db2 Version 11.1 Mod 3 Fix Pack 3 for Linux, UNIX, and Windows
Db2 Version 11.1 Mod3 Fix Pack3 iFix001 for Linux, UNIX, and Windows
Db2 Version 11.1 Mod3 Fix Pack3 iFix002 for Linux, UNIX, and Windows
Db2 Version 11.1 Mod 4 Fix Pack 6 for Linux, UNIX, and Windows
Db2 Version 11.1 Mod 4 Fix Pack 5 for Linux, UNIX, and Windows
Db2 Version 11.1 Mod 4 Fix Pack 7 for Linux, UNIX, and Windows

APAR status

• Closed as fixed if next.

Error description

• db2diag.log showed that there was still one page outstanding
  and hadn't been collected, which led  assertion failure / panic.
  2017-04-07-12.59.44.093685+480 I13323030E34369   LEVEL:Severe
  PID     : 42704                TID : 140384869017344 PROC :
  db2sysc 1
  INSTANCE: db2inst1               NODE : 001            DB   :
  MYDB1
  HOSTNAME: MYHOST1
  EDUID   : 129                  EDUNAME: db2pclnr (MYDB1) 1
  FUNCTION: DB2 UDB, buffer pool services, sqlbClnrCollectAllAIO,
  probe:1960
  DATA #1 : String, 146 bytes
  ASSERTION FAILED!!!
  ASSERTION EXPRESSION: ((clnrCB->lioCB.numAIOOutstanding == 0))
  SOURCE FILE NAME: sqlbclnr_core.C
  SOURCE FILE LINE NUMBER: 1960
  DATA #2 : Codepath, 8 bytes
  0
  DATA #3 : SQLB_CLNR_CB, PD_TYPE_SQLB_CLNR_CB, 6480 bytes
  clnrIndex: 0
  parentGroup: 0x00007fc473c89000
  eduCB: 00007fc473cd64e0
  Cleaner Latch (clnrLatch):
  {
     lock          = { 0x00 [ unlocked ] }
     fighterCount  = 0
     identity      = SQLB_PAGE_CLEANER::clnrLatch (277)
  }
  clnrState: 2
  To Cleaner WaitPost (toClnrWP):
  {
     postcode  = 0x0
     cs        = {
     lock          = { 0x00 [ unlocked ] }
     fighterCount  = 0
     identity      = sqlo_waitpost::cs (6)
  }
     state     = 0x0 (Not Posted)
     guard     = initialized
     initflags = 0x1
     first = 0x0000000000000000
  }
  From Cleaner WaitPost (fromClnrWP):
  {
     postcode  = 0x0
     cs        = {
     lock          = { 0x00 [ unlocked ] }
     fighterCount  = 0
     identity      = sqlo_waitpost::cs (6)
  }
     state     = 0x0 (Not Posted)
     guard     = initialized
     initflags = 0x1
     first = 0x0000000000000000
  }
  SQLB_CLNR_LIO_CB:
  {
     numAIOOutstanding: 1
     maxAIO: 256
     numToCollect: 128
     lioFlags: 4
     batchIORequestList
     {
        IOReqList: 00007fc473ca91e0
        numPagesInList: 0
        lowPos: 231
        highPos: 232
        appendToHighPos: true
     }
     serializeBatchIORequestList
     {
        IOReqList: 00007fc473c88040
        numPagesInList: 0
        lowPos: 0
        highPos: 0
        appendToHighPos: false
     }
     serializeIORequestList
     {
        IOReqList: 00007fc473c5a3a0
        numPagesInList: 0
        lowPos: 0
        highPos: 0
        appendToHighPos: false
     }
     completedIOReqScratch: 00007fc473c3b3a0
  }
  -- Start of warmCB --
  0x00007FC473CAA180 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAA190 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAA1A0 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  <skip>
  -- End of castoutCB --
  SQLB_GLOBALS: 7FC98CED5080
  terminating: false
  forcedTerminate false:
  currentGatherTask: 9
  -- Start of currentGatherInfo --
  0x00007FC473CAB750 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAB760 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAB770 : 0000 0000 0000 0000
  ........
  -- End of currentGatherInfo --
  -- Start of gatherStatics --
  0x00007FC473CAB778 : 0100 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAB788 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAB798 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAB7A8 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAB7B8 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAB7C8 : 0000 0000 0000 0000 0100 0000 0000 0000
  ................
  0x00007FC473CAB7D8 : 0100 0000 0000 0000 0200 0000 0000 0000
  ................
  0x00007FC473CAB7E8 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAB7F8 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAB808 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAB818 : 0000 0000 0000 0000 0100 0000 0000 0000
  ................
  0x00007FC473CAB828 : 0100 0000 0000 0000 0100 0000 0000 0000
  ................
  0x00007FC473CAB838 : 0000 0000 0000 0000
  ........
  -- End of gatherStatics --
  curTime: 0
  softMax: 240
  softMaxUnitsTime: true
  alternatePageCleaning: true
  pleaseTerminate: false
  markGBPDep: false
  outOfFileHandlesOccurred: false
  -- Start of stats --
  0x00007FC473CAB858 : 0080 CD73 C47F 0000 0090 CD73 C47F 0000
  ...s.......s....
  0x00007FC473CAB868 : 0A22 1A26 0704 E107 20FE 0500 0000 0000
  .".&.... .......
  0x00007FC473CAB878 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAB888 : 0000 0000 0000 0000 9B2E 0100 0000 0000
  ................
  0x00007FC473CAB898 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAB8A8 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAB8B8 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAB8C8 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAB8D8 : 0000 0000 0000 0000
  ........
  -- End of stats --
  -- Start of lsnHistoryTable --
  0x00007FC473CAB8E0 : E8F5 0300 0000 0000 0100 0000 0000 0000
  ................
  0x00007FC473CAB8F0 : D102 0000 0000 0000 8F02 0000 0000 0000
  ................
  0x00007FC473CAB900 : D102 0000 0000 0000 A0B0 B3DB C47F 0000
  ................
  -- End of lsnHistoryTable --
  -- Start of initInfo --
  0x00007FC473CAB910 : 00
  .
  -- End of initInfo --
  skippedForSuspend: false
  -- Start of dispatchStats --
  0x00007FC473CAB918 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAB928 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  0x00007FC473CAB938 : 0000 0000 0000 0000 0000 0000 0000 0000
  ................
  -- End of dispatchStats --
  clnrGlobalIndex: 0
  MESSAGE : AIO failed to collect all outstanding IO requests.
  This is a fatal
            error which must bring down the database.
  CALLSTCK: (Static functions may not be resolved correctly, as
  they are resolved to the nearest symbol)
    [0] 0x00007FC9CEA24B29
  _Z21sqlbClnrCollectAllAIOP12SQLB_CLNR_CB + 0x209
    [1] 0x00007FC9CEAC4689 _Z13sqlbgbCastoutP12SQLB_CLNR_CBb +
  0xF59
    [2] 0x00007FC9CEA16641 _Z19sqlbClnrGatherPagesP12SQLB_CLNR_CB
  + 0x3E1
    [3] 0x00007FC9CEA22198 _Z18sqlbClnrEntryPointP12sqbPgClnrEdu
  + 0x178
    [4] 0x00007FC9CEA21F94 _ZN12sqbPgClnrEdu6RunEDUEv + 0x24
    [5] 0x00007FC9D58EB796 _ZN9sqzEDUObj9EDUDriverEv + 0x116
    [6] 0x00007FC9D3F3E688 sqloEDUEntry + 0x578
    [7] 0x0000003326407A51 /lib64/libpthread.so.0 + 0x7A51
    [8] 0x00000033260E896D clone + 0x6D
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * DB2 is down when AIO failed to collect all outstanding IO    *
  * requests.                                                    *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Upgrade to db2_v111m3fp3                                     *
  ****************************************************************
  

Problem conclusion

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels

• RB10 PSN

     UP