IT21939: In HADR, primary database could be brought down due to no heap memory is available

Db2 Version 11.1 Mod 3 Fix Pack 3 for Linux, UNIX, and Windows
Db2 Version 11.1 Mod3 Fix Pack3 iFix001 for Linux, UNIX, and Windows
Db2 Version 11.1 Mod3 Fix Pack3 iFix002 for Linux, UNIX, and Windows
Db2 Version 11.1 Mod 4 Fix Pack 6 for Linux, UNIX, and Windows
Db2 Version 11.1 Mod 4 Fix Pack 5 for Linux, UNIX, and Windows
Db2 Version 11.1 Mod 4 Fix Pack 7 for Linux, UNIX, and Windows

APAR status

• Closed as program error.

Error description

• In HADR system, primary may request LFR to scan logs. While HADR
  primary edu is opening the LFR scan request, it needs to
  allocate an LFR request structure of a given type.
  If there is no heap memory available at that time, the dynamic
  allocation would fail and HADR primary process would bring down
  the database.
  
  Please look for following db2diag.log messages to identify the
  problem.
  
  2017-06-16-17.17.24.336504+480 I145800777A432       LEVEL: Error
  PID     : 14880082             TID : 156749         PROC :
  db2sysc 0
  INSTANCE: db2inst1             NODE : 000           DB   :
  MYDB1
  HOSTNAME: MYHOST1
  EDUID   : 156749               EDUNAME: db2hadrp.0.2 (MYDB1) 0
  FUNCTION: DB2 UDB, recovery manager, sqlplfrReqAllocate,
  probe:1200
  MESSAGE : Error: mem alloc for LFR request of type 0 failed, rc
  = 0x8b0f0002.
  
  2017-06-16-17.17.24.336877+480 I145801210A497       LEVEL: Error
  PID     : 14880082             TID : 156749         PROC :
  db2sysc 0
  INSTANCE: db2inst1             NODE : 000           DB   :
  MYDB1
  HOSTNAME: MYHOST1
  EDUID   : 156749               EDUNAME: db2hadrp.0.2 (MYDB1) 0
  FUNCTION: DB2 UDB, recovery manager, sqlplfrScanOpen, probe:3200
  MESSAGE : Error: ScanNext req alloc failed, rc =
  DATA #1 : Hexdump, 4 bytes
  0x0A000000713F7D50 : 8B0F 0002
  ....
  
  2017-06-16-17.17.24.337197+480 I145801708A535       LEVEL: Error
  PID     : 14880082             TID : 156749         PROC :
  db2sysc 0
  INSTANCE: db2inst1             NODE : 000           DB   :
  MYDB1
  HOSTNAME: MYHOST1
  EDUID   : 156749               EDUNAME: db2hadrp.0.2 (MYDB1) 0
  FUNCTION: DB2 UDB, High Availability Disaster Recovery,
  hdrEdu::hdrEduP, probe:20450
  MESSAGE : ZRC=0x8B0F0002=-1961951230=SQLO_NOMEM_DBH
            "No memory available in 'Database Heap'"
            DIA8302C No memory available in the database heap.
  

Local fix

• Please upgrade to Db2 V10.5 Fixpack 9.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * primary database could be brought down                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Please upgrade to Db2 V10.5 Fixpack 9.                       *
  ****************************************************************
  

Problem conclusion

• In HADR system, primary may request LFR to scan logs. While HADR
  primary edu is opening the LFR scan request, it needs to
  allocate an LFR request structure of a given type.
  If there is no heap memory available at that time, the dynamic
  allocation would fail and HADR primary process would bring down
  the database.
  
  Please look for following db2diag.log messages to identify the
  problem.
  
  2017-06-16-17.17.24.336504+480 I145800777A432       LEVEL: Error
  PID     : 14880082             TID : 156749         PROC :
  db2sysc 0
  INSTANCE: db2inst1             NODE : 000           DB   :
  KGDBDG
  HOSTNAME: JZJYDGDB1
  EDUID   : 156749               EDUNAME: db2hadrp.0.2 (KGDBDG) 0
  FUNCTION: DB2 UDB, recovery manager, sqlplfrReqAllocate,
  probe:1200
  MESSAGE : Error: mem alloc for LFR request of type 0 failed, rc
  = 0x8b0f0002.
  
  2017-06-16-17.17.24.336877+480 I145801210A497       LEVEL: Error
  PID     : 14880082             TID : 156749         PROC :
  db2sysc 0
  INSTANCE: db2inst1             NODE : 000           DB   :
  KGDBDG
  HOSTNAME: JZJYDGDB1
  EDUID   : 156749               EDUNAME: db2hadrp.0.2 (KGDBDG) 0
  FUNCTION: DB2 UDB, recovery manager, sqlplfrScanOpen, probe:3200
  MESSAGE : Error: ScanNext req alloc failed, rc =
  DATA #1 : Hexdump, 4 bytes
  0x0A000000713F7D50 : 8B0F 0002
  ....
  
  2017-06-16-17.17.24.337197+480 I145801708A535       LEVEL: Error
  PID     : 14880082             TID : 156749         PROC :
  db2sysc 0
  INSTANCE: db2inst1             NODE : 000           DB   :
  KGDBDG
  HOSTNAME: JZJYDGDB1
  EDUID   : 156749               EDUNAME: db2hadrp.0.2 (KGDBDG) 0
  FUNCTION: DB2 UDB, High Availability Disaster Recovery,
  hdrEdu::hdrEduP, probe:20450
  MESSAGE : ZRC=0x8B0F0002=-1961951230=SQLO_NOMEM_DBH
            "No memory available in 'Database Heap'"
            DIA8302C No memory available in the database heap.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

  IT21312

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  DB2 FOR LUW

• Fixed component ID

  DB2FORLUW

Applicable component levels

• RA50 PSN

     UP

• RA50 PSY

     UP