PI70933: MEMORY LEAK IN BIGSQL DFSRW (READER/WRITER) DB2FMP PROCESS RESULTS IN VARIOUS SYMPTOMS (PAGING,OOM,HANG,ERROR)

APAR status

• Closed as program error.

Error description

• Memory is leaked in the BigSQL C++ DFSRW process on each query
  execution where a predicate is pushed down for processing in
  impala.
  The memory leak results in growing system memory usage over
  time, which can result in symptoms associated with memory over-
  commitment (performance degradation associated with paging,
  oom errors).
  The memory usage from this process is also tracked within the
  Database Memory Controller, which attempts to keep BigSQL's
  memory usage within an "INSTANCE_MEMORY" limit.  The DFSRW C++
  process private memory usage is part of the DFSRW_PRIVATE
  consumer. Once the amounts tracked by the Controller exceed
  the limit, various OOM-related errors will occur, resulting in
  symptoms such as SQL5199 rc=2 and hangs.  The db2diag.log on
  the node will contain db2diag.log messages such as :
  
  FUNCTION: DB2 UDB, SQO Memory Management,
  SqloMemController::requestMemory, probe:50
  MESSAGE : ZRC=0x8B0F0000=-1961951232=SQLO_NOMEM "No Memory
  Available"
            DIA8300C A memory heap error has occurred.
  DATA #1 : String, 28 bytes
  Attempt to get memory failed
  
  
  
  
  Issuing the following command and monitoring the growth of the
  DFSRW_PRIVATE memory consumers is the easiest method of
  tracking this memory usage, which should not normally exceed
  half of the instance memory limit.
     db2_all "db2pd -dbptnmem"
  
  Database Member Memory Controller Statistics
  
  Controller Automatic: N
  
  
  Controller Automatic: N
  Memory Limit:         24791332 KB
  Current usage:        20690592 KB
  HWM usage:            20690592 KB
  Cached memory:        1727360 KB
  
  Individual Memory Consumers:
  
  
  
  Name             Mem Used (KB) HWM Used (KB) Cached (KB)
  ========================================================
  APPL-BIGSQL             160000        160000      158208
  DBMS-mcornish           128512        128512        6144
  FMP_RESOURCES            90112         90112       89152
  PRIVATE                  19392         19392           0
  FCM_RESOURCES           296992        296992           0
  DFSRW_PRIVATE          5257408       5257408           0
  DB-BIGSQL             14737920      14737920     1473856
  
  
  In the above example, DFSRW_PRIVATE "Mem Used" is at a
  reasonable level.  If it were above, say, 12400000 KB (more
  than half of the Memory Limit), and continuously growing over
  time, this would be indicative of abnormal memory usage that
  should be addressed.
  Before considering this APAR, it is recommended that the Linux
  malloc settings suggested in APAR PI61701 are attempted first.
  

Local fix

Problem summary

• See Error Description
  

Problem conclusion

• The problem is fixed in Version 4.2.0.0 December 2016 Refresh (
  Level s161128 & Bigsql-dist version 5.78.5.310) and later fix
  packs.
  For more info, see:
  See: http://ibm.co/2iuUmYV
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

  PI70925

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• NA
  

Fix information

• Fixed component name

  INFO BIGINSIGHT

• Fixed component ID

  5725C0900

Applicable component levels

• R420 PSY

     UP