IT22759: DB2 AGENT/DB2FMP CAN HANG DUE TO INCOMPLETE HANDSHAKE DURING INTERRUPT

Db2 Version 11.1 Mod 3 Fix Pack 3 for Linux, UNIX, and Windows
Db2 Version 11.1 Mod3 Fix Pack3 iFix001 for Linux, UNIX, and Windows
Db2 Version 11.1 Mod3 Fix Pack3 iFix002 for Linux, UNIX, and Windows

APAR status

• Closed as program error.

Error description

• db2 agent/db2fmp can hang due to incomplete handshake during
  interrupt.  This can result in any agent looking to call a
  routine to get stuck waiting on the
  SQLO_LT_sqlerFmpTable__fmpTableLatch latch result in a hang for
  that agent as well.
  
  db2diag.log:
  
  2017-08-23-00.16.02.999306+000 I20742564A998        LEVEL: Error
  PID     : 2557828              TID : 23697          PROC :
  db2sysc 24
  INSTANCE: db2inst1             NODE : 024           DB   :
  DBNAME
  APPHDL  : 999-5773             APPID:
  *N999.db2inst1.170823001545
  AUTHID  : db2inst1             HOSTNAME: db2host1
  EDUID   : 23697                EDUNAME: db2agntp (DBNAME) 24
  FUNCTION: DB2 UDB, routine_infrastructure,
  sqlerInvokeFencedRoutine, probe:1220
  MESSAGE : ZRC=0x804B0003=-2142568445=SQLJS_INTRPT "INTERRUPT
  DETECTED"
            DIA8003C The interrupt  has been received.
  DATA #1 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes
  sqlcaid : SQLCA     sqlcabc: 136   sqlcode: 0   sqlerrml: 0
  sqlerrmc:
  sqlerrp : SQL11012
  sqlerrd : (1) 0x00000000      (2) 0x00000000      (3) 0x00000000
             (4) 0x00000000      (5) 0x00000000      (6)
  0x00000018
  sqlwarn : (1)      (2)      (3)      (4)        (5)       (6)
             (7)      (8)      (9)      (10)        (11)
  sqlstate:
  
  2017-08-23-00.16.02.689867+000 I20743563A899        LEVEL: Error
  PID     : 8913656              TID : 772            PROC :
  db2fmp (C) 9
  INSTANCE: db2inst1             NODE : 009           DB   :
  DBNAME
  APPID   : *N999.db2inst1.170823000555
  HOSTNAME: db2host
  EDUID   : 772                  EDUNAME: db2fmp (C) 9
  FUNCTION: DB2 UDB, routine_infrastructure, sqlerFmpReceive,
  probe:1259
  MESSAGE : ZRC=0x8024006D=-2145124243=SQLER_CA_BUILT
            "SQLCA has already been built"
  DATA #1 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes
  sqlcaid : SQLCA     sqlcabc: 136   sqlcode: -1131   sqlerrml: 0
  sqlerrmc:
  sqlerrp : SQLERRCV
  sqlerrd : (1) 0x00000000      (2) 0x00000000      (3) 0x00000000
             (4) 0x00000000      (5) 0x00000000      (6)
  0x00000009
  sqlwarn : (1)      (2)      (3)      (4)        (5)       (6)
             (7)      (8)      (9)      (10)        (11)
  sqlstate:
  
  2017-08-23-00.16.02.691142+000 I20744463A556        LEVEL:
  Severe
  PID     : 8913656              TID : 1              PROC :
  db2fmp (C) 9
  INSTANCE: db2inst1             NODE : 009
  HOSTNAME: db2host
  EDUID   : 1                    EDUNAME: db2fmp (C) 9
  FUNCTION: DB2 UDB, trace services, sqlt_logerr_data (secondary
  logging function), probe:50
  MESSAGE : Library still in use in sqlerLibraryUnloadAll
            /db2home/db2inst1/sqllib/bin/routine/db2monudf 3
  DATA #1 : Hexdump, 4 bytes
  0x0FFFFFFFFFFFDDB0 : 0000 0000
  ....
  
  2017-08-23-00.16.02.519850+000 E20745020A4253       LEVEL:
  Severe
  PID     : 10159080             TID : 10285          PROC :
  db2sysc 57
  INSTANCE: db2inst1             NODE : 057           DB   :
  DBNAME
  APPHDL  : 999-5773             APPID:
  *N999.db2inst1.170823001545
  AUTHID  : db2inst1             HOSTNAME: db2host
  EDUID   : 10285                EDUNAME: db2agntp (DBNAME) 57
  FUNCTION: DB2 UDB, routine_infrastructure, sqlerMasterThreadReq,
  probe:910
  DATA #1 : String, 23 bytes
  Marking fmp as unstable
  DATA #2 : String, 8 bytes
  Fmp TID:
  DATA #3 : Hexdump, 4 bytes
  0x0780000002AFFEB8 : 0000 0607
  ....
  DATA #4 : String, 8 bytes
  Fmp Row:
  DATA #5 : sqlerFmpRow, PD_SQLER_TYPE_FMP_ROW, 856 bytes
  fmpPid: 5636556
  fmpPoolList Ptr: 0x0780000002a3f740    fmpForcedList Ptr:
  0x0000000000000000
  nextFmpCB Ptr: 0x0780000000da5360      prevFmpCB Ptr:
  0x0000000000000000
  fmpIPCList Ptr: 0x0780000002a3ffc0
  stateFlags: 0x00000012 numFmp32Attaches: 0
  numActiveThreads: 1    numPoolThreads: 3
  fmpCodePage: 0 fmpRowUseCount: 1
  active: 0x01   rowLoaderValidate: 0x00
  ipcLatch:
  0x0780000002A3EB64 : 1000 0000 003F 0000
  .....?..
  rowLatch:
  0x0780000002A3EB6C : 1000 0000 01DB 0000
  ........
  fmpAgentList:
  
  2017-08-23-00.16.02.999763+000 I20755577A495        LEVEL:
  Severe
  PID     : 10421710             TID : 1              PROC :
  db2fmp (C) 24
  INSTANCE: db2inst1              NODE : 024
  HOSTNAME: db2host1
  EDUID   : 1                    EDUNAME: db2fmp (C) 24
  FUNCTION: DB2 UDB, routine_infrastructure,
  sqlerMasterThreadListener, probe:400
  RETCODE : ZRC=0xFFFFFB95=-1131
            SQL1131N  A stored procedure process has been
  terminated abnormally.
            Routine name: "". Specific name: "".
  
  Agent Stack could look like:
  
  *** Stack ***
  
  Timestamp: 2017-08-23-17.55.48.247784
  0x09000000001F5C10 semop + 0xB0
  0x090000000E118664 sqloSSemP + 0x1224
  0x0900000012C7AD14
  IPRA.$sqlccipcWaitSynch__FP18SQLCC_INITSTRUCT_T + 0x254
  0x0900000012C74254 sqlccipcinit + 0x8D4
  0x090000000E10F94C
  sqlccinit__FP18SQLCC_INITSTRUCT_TPP17SQLCC_COMHANDLE_TP12SQLCC_C
  OND_TP13SQLO_MEM_POOL + 0x5CC
  0x090000000EBF7EBC
  IPRA.$sqlerInitCommsLayer__FP14sqlerFmpHandleP8sqeAgentb + 0x11C
  0x090000000EBF34D8
  sqlerMasterThreadReq__FP17sqlerFmpParmsBaseP13sqlerFmpTableP14sq
  lerFmpHandleP18sqlerFmpThreadListP8sqeAgentUicT7P5sqlcab+ 0x858
  0x090000000EBF6340
  sqlerInterruptThreadedFmp__FP14sqlerFmpHandleP17sqlerFmpParmsBas
  e + 0xC0
  0x090000000EBE2E1C
  sqlerReturnFmpToPool__FcT1P14sqlerFmpHandleP8sqeAgent + 0x1DDC
  0x090000000EBDED24 sqlerInvokeFencedRoutine__FP13sqlerFmpParms +
  0xB44
  0x0900000011D54F54 sqlriInvokeInvoker__FP10sqlri_ufobb + 0x1CF4
  0x0900000011D56C9C sqlriutf__FP8sqlrr_cb + 0x13C
  0x09000000145156B8 sqlri_tfopn__FP8sqlrr_cbP9sqlri_tao + 0x1F8
  0x09000000101296A4 sqlriopn__FP8sqlrr_cbP9sqlri_taoPi + 0x5A4
  0x090000001204C750 sqlrita__FP8sqlrr_cb + 0xEF0
  0x090000000FCDC898 sqlriSectInvoke__FP8sqlrr_cbP12sqlri_opparm +
  0x618
  0x090000000F634380 sqlrr_dss_router__FP8sqlrr_cb + 0x9A0
  0x090000000F633054
  sqlrr_subagent_router__FP8sqeAgentP12SQLE_DB2RA_T + 0xF54
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * N/A                                                          *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Upgrade to db2_v111m3fp3.                                    *
  ****************************************************************
  

Problem conclusion

• db2_v111m3fp3 contains fix.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  DB2 FOR LUW

• Fixed component ID

  DB2FORLUW

Applicable component levels

• RB10 PSY

     UP