IT23954: HADR STANDBY FAILS WITH DBHEAP OUT OF MEMORY OR MEMORY CORRUPTION

Db2 Version 11.1 Mod3 Fix Pack3 iFix001 for Linux, UNIX, and Windows
Db2 Version 11.1 Mod3 Fix Pack3 iFix002 for Linux, UNIX, and Windows

APAR status

• Closed as program error.

Error description

• The memory assigned to DB2_HADR_BUF_SIZE db2 registry is being
  allocated from dbheap during database activation/connection.
  Any changed to this registry become effective after db2 restart.
  
  This defect can cause a) unnecessary re-allocation of the
  receive
  buffer leading to the Standby running out of memory or b) memory
  corruption due to not allocating memory for the receive buffer
  when
  necessary.
  
  Symptom a) DBHEAP OUT OF MEMORY
  
  This defect causes re-allocation of DB2_HADR_BUF_SIZE memory
  during standby db re-connect after communication failure when
  there is no change to the setting. This causes double allocation
  of memory from dbheap.
  
  And since some customers set the value of this DB2_HADR_BUF_SIZE
  registry to a higher number than the default of 2Xlogbufsz, it
  may cause out of dbheap memory errors on Standby causing db2 to
  go down.
  
  the messages in db2diag.log may look like this:
  
  2018-01-21-01.31.15.322975-300 E12975694A696        LEVEL:
  Warning
  PID     : 18612304             TID : 20562          PROC :
  db2sysc 0
  INSTANCE: db2inst1             NODE : 000           DB   :
  SAMPLE
  HOSTNAME: standbyhost
  EDUID   : 20562                EDUNAME: db2hadrs.0.0 (SAMPLE) 0
  FUNCTION: DB2 UDB, SQO Memory Management,
  sqloMemLogPoolConditions, probe:30
  DATA #1 : <preformatted>
  Out of memory failure for Database Heap (DBHEAP) on node 0.
  Requested block size &#160; &#160; &#160; &#160; &#160; : 4295032808 bytes.
  Physical heap size &#160; &#160; &#160; &#160; &#160; &#160; : 5139988480 bytes.
  Configured heap size &#160; &#160; &#160; &#160; &#160; : 9320792064 bytes.
  Unreserved memory used by heap : 0 bytes.
  Unreserved memory left in set &#160;: 470089728 bytes.
  
  This is most likely to occur when using a very large
  DB2_HADR_BUF_SIZE up to the 4GB recommended limit, in
  conjunction with low db memory.
  
  With this fix, we will not re-allocate memory if no change to
  the setting.
  
  Symptom b) MEMORY CORRUPTION
  
  Under the following conditions:
  
  1. logbufsz on the Primary database is exactly 32768
  
  2. DB2_HADR_BUF_SIZE is unset on the Standby
  
  memory corruption occurs and the Standby database
  crashes with an FODC_Trap or FODC_Panic as a result.
  

Local fix

• Symptom a) DBHEAP OUT OF MEMORY
  
  As a temporary workaround, please apply one of the following:
  
  1. Increase database memory
  
  or
  
  2. Use log spooling instead of using large DB2_HADR_BUF_SIZE
  
  Symptom b) MEMORY CORRUPTION
  
  Set the DB2_HADR_BUF_SIZE registry variable on the Standby to
  any value other than the default of 16, e.g.
  
  db2set DB2_HADR_BUF_SIZE=65536
  
  The Standby receive buffer will be correctly allocated when the
  Standby is brought back up.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * ALL                                                          *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * The complete fix for this problem first appears in DB2       *
  * Version 11.1.3.3 iFix001 and all the subsequent Fix Packs.   *
  ****************************************************************
  

Problem conclusion

• The complete fix for this problem first appears in DB2 Version
  11.1.3.3 iFix001 and all the subsequent Fix Packs.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  IT23988

Fix information

• Fixed component name

  DB2 FOR LUW

• Fixed component ID

  DB2FORLUW

Applicable component levels

• RB10 PSN

     UP