IZ58302: DB2WDOG PROCESS TRAP IN SQLECREATEFMP AND BRING DOWN THE INSTANCE

DB2 Version 9.5 Fix Pack 6a for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 7 for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 8 for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 9 for Linux, UNIX, and Windows
DB2 Version 9.5 Fix Pack 10 for Linux, UNIX, and Windows

APAR status

• Closed as program error.

Error description

• Users effected:  DB2 UDB v9.5 and v9.7 users
  Problem description:
  
  Instance trap due to db2wdog process receives signal #11
  
  sample db2diag.log entry:
  
  2009-06-06-13.27.15.061362-240 E66710914A2199     LEVEL:
  Critical
  PID     : 8573                 TID  : 2           PROC : db2wdog
  INSTANCE: db2inst1             NODE : 000
  EDUID   : 2                    EDUNAME: db2wdog
  FUNCTION: DB2 UDB, oper system services, sqloEDUCodeTrapHandler,
  probe:10
  MESSAGE : ADM14001C  An unexpected and critical error has
  occurred:
  "Trap". The
           instance may have been shutdown as a result.
  "Automatic" FODC
  (First
           Occurrence Data Capture) has been invoked and
  diagnostic
  information
           has been recorded in directory
  
  "/db2home/db2inst1/sqllib/db2dump/FODC_Trap_2009-06-06-13.27.05.
  64599
           2/". Please look in this directory for detailed
  evidence about
  what
           happened and contact IBM support if necessary to
  diagnose the
           problem.
  DATA #1 : Signal Number Recieved, 4 bytes
  11
  DATA #2 : Siginfo, 256 bytes
  
  Trap file shows:
  
  EDU name     : db2wdog
  EDU ID       : 2
  Signal #11
  uname: S:SunOS R:5.10 V:Generic_120011-14 M:sun4u
  N:db2sunprod01.rack.peerplace.com
  process id: 8573
  thread id : 2 (0x2)
  </Header>
  
  <StackTrace>
  ##### Object: /db2home/db2inst1/sqllib/adm/db2syscr
  __1cNsqleCreateFmp6FpcpnQSQdDLO_EDUWAITPOST_pi_i_(0xffffffff607f
  d240,0x2
  02593918,0x1000009fde0,0x10238c,0x0,0x1c00) + 0x1c
  __1cMsqleWatchDog6FL_i_(0x0,0x1,0xffffffff7e983c70,0x2,0x1001117
  90,0xfff
  fffff607fd1e8) + 0x938
  DB2main(0x0,0x100111120,0x49,0x18280000,0x1,0x10010e2c8) + 0xabc
  ##### Object: /opt/IBM/db2/V9.5/lib64/libdb2e.so.1
  sqloEDUEntry(0xffffffff7ffffa20,0x2,0xa10210,0xffffffff7f09e928,
  0x187800
  00,0xffffffff7e615ce0) + 0x130
  ##### Object: /lib/sparcv9/libc.so.1
  _thr_slot_offset(0x0,0x0,0x0,0x0,0x0,0x0) + 0x484
  </StackTrace>
  
  Root cause is due to db2wdog process failed to attach to the
  newly created DBMS shared memory set.  So when fmp tries to
  allocate a memory block, wrong memory address is being accessed.
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * DB2 UDB v9.5 and v9.7 users                                  *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * Instance trap due to db2wdog process receives signal         *
  * #11sample db2diag.log entry:2009-06-06-13.27.15.061362-240   *
  * E66710914A2199    LEVEL:CriticalPID    : 8573                *
  * TID  : 2          PROC :db2wdogINSTANCE: db2inst1            *
  * NODE : 000EDUID  : 2                    EDUNAME:             *
  * db2wdogFUNCTION: DB2 UDB, oper system                        *
  * services,sqloEDUCodeTrapHandler,probe:10MESSAGE : ADM14001C  *
  * An unexpected and critical error hasoccurred:"Trap".         *
  * Theinstance may have been shutdown as a result."Automatic"   *
  * FODC(FirstOccurrence Data Capture) has been invoked          *
  * anddiagnosticinformationhas been recorded in                 *
  * directory"/db2home/db2inst1/sqllib/db2dump/FODC_Trap_2009-06-0
  * Please look in this directory for detailedevidence           *
  * aboutwhathappened and contact IBM support if necessary       *
  * todiagnose theproblem.DATA #1 : Signal Number Recieved, 4    *
  * bytes11DATA #2 : Siginfo, 256 bytesTrap file shows:EDU name  *
  *   : db2wdogEDU ID      : 2Signal #11uname: S:SunOS R:5.10    *
  * V:Generic_120011-14                                          *
  * M:sun4uN:db2sunprod01.rack.peerplace.comprocess id:          *
  * 8573thread id : 2 (0x2)</Header><StackTrace>##### Object:    *
  * /db2home/db2inst1/sqllib/adm/db2syscr__1cNsqleCreateFmp6FpcpnQ
  * +                                                            *
  * 0x1c__1cMsqleWatchDog6FL_i_(0x0,0x1,0xffffffff7e983c70,0x2,0x1
  * +                                                            *
  * 0x938DB2main(0x0,0x100111120,0x49,0x18280000,0x1,0x10010e2c8)
  * +0xabc##### Object:                                          *
  * /opt/IBM/db2/V9.5/lib64/libdb2e.so.1sqloEDUEntry(0xffffffff7ff
  * + 0x130##### Object:                                         *
  * /lib/sparcv9/libc.so.1_thr_slot_offset(0x0,0x0,0x0,0x0,0x0,0x0
  * + 0x484</StackTrace>Root cause is due to db2wdog process     *
  * failed to attach to thenewly created DBMS shared memory set. *
  *  So when fmp tries toallocate a memory block, wrong memory   *
  * address is beingaccessed.                                    *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Upgrade to DB2 version 9.5 Fix Pack 6                        *
  ****************************************************************
  

Problem conclusion

• Problem was first fixed in Version 9.5 Fix Pack 6
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  IC62523

Fix information

• Fixed component name

  DB2 UDB WSE SOL

• Fixed component ID

  5765F3502

Applicable component levels

• R950 PSN

     UP