IT35629: MEMORY LEAK IN DSMSERV PROCESS AT SERVER VERSION 8.1.10 AND ABOVE WHEN USING TIERING

8.1.10.300-IBM-SPSRV-WindowsX64
8.1.10.300-IBM-SPSRV-Linuxx86_64
8.1.10.300-IBM-SPSRV-Linuxs390x
8.1.10.300-IBM-SPSRV-Linuxppc64le
8.1.10.300-IBM-SPSRV-AIX
8.1.12.000-IBM-SPSRV-WindowsX64
8.1.12.000-IBM-SPSRV-Linuxx86_64
8.1.12.000-IBM-SPSRV-Linuxs390x
8.1.12.000-IBM-SPSRV-Linuxppc64le
8.1.12.000-IBM-SPSRV-AIX
IBM Spectrum Protect Server V8.1.11.X interim fix downloads
IBM Spectrum Protect Server V8.1 Fix Pack 12 (V8.1.12) Downloads

APAR status

• Closed as program error.

Error description

• You might be exposed to the symptom described with this APAR, if
  you are using tiering and upgraded to Spectrum Protect server
  version 8.1.10.000 and above.
  The memory increase for the dsmserv process can be seen after
  tiering is finished. After the TIER operation is finished the
  memory is held by the TierStreamHelper thread and increases with
  each new tiering operation started.
  
  This can be seen from the 'SHOW ALLOC' collected from dsmadmc or
  from the servermon *show.txt output, for example:
  
  Individual Memory Allocations:
  20201208-show.txt:sctierbatch line  3726:     60201 entries for
  7556236528 bytes (TierStreamHelper) thread 248513()
  20201209-show.txt:sctierbatch line  3726:   150475 entries for
  17342703952 bytes (TierStreamHelper) thread 248513()
  20201210-show.txt:sctierbatch line  3726:   275805 entries for
  30260307568 bytes (TierStreamHelper) thread 248513()
  20201211-show.txt:sctierbatch line  3726:   408805 entries for
  43581244752 bytes (TierStreamHelper) thread 248513()
  20201212-show.txt:sctierbatch line  3726:   548276 entries for
  57528680688 bytes (TierStreamHelper) thread 248513()
  
  20201206-show.txt:There are 1,031,465 memory allocations for
  1,426,242,368 total bytes
  20201207-show.txt:There are 1,345,162 memory allocations for
  16,701,377,072 total bytes
  20201208-show.txt:There are 1,590,655 memory allocations for
  20,633,548,384 total bytes
  20201209-show.txt:There are 2,189,683 memory allocations for
  22,313,614,976 total bytes
  20201210-show.txt:There are 2,826,314 memory allocations for
  33,997,961,792 total bytes
  20201211-show.txt:There are 3,153,871 memory allocations for
  45,374,051,744 total bytes
  
  Customer/Support Diagnostics:
  
  The memory increase will be seen in an error path when a chunk
  in a non-fragmented object is not found and it can occur in any
  tiering path that moves non-fragmented objects.
  If you are affected by this APAR you will see an entry about
  skipped objects due to rc 1116 in a Spectrum Protect server
  trace and in the dsmffdc.log.
  
  A Spectrum Protect server trace with trace flag SC might show:
  [sctierbatch.c][2744][ScTierData]:Skipping current object
  without abort txn with rc=1116
  
  [ffdcutil.c][432][FFDCLogThread]:[01-25-2021 12:39:12.211][
  FFDC_GENERAL_SERVER_ERROR ]: (sctierbatch.c:2731) Skipped object
  6 during tier operation due to rc=1116
  
  The dsmffdc.log might show:
  [ FFDC_GENERAL_SERVER_ERROR ]: (scutil.c:2622) Repeated FFDCLog
  for tering process 55: file sctierbatch.c, line 2737, rc 1116,
  count 3~
  
  [ FFDC_GENERAL_SERVER_ERROR ]: (sctierbatch.c:2737) Skipped
  object 6 during tier operation due to rc=1116
  
  
  IBM Spectrum Protect Server Version Affected:
  Version 8.1.10.000 and above on all supported platforms
  
  | MDVREGR 8.1.10-TIV_5698MSV |
  | MDVREGR 8.1.11-TIV_5698MSV |
  
  Initial Impact:
  Medium
  
  Additional Keywords:
  TSM server Spectrum Protect TS004574971 tiering memory
  

Local fix

• Restart the Spectrum Protect server to release the memory.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 8.1.11.100 and 8.1.12. Note  *
  * that this is subject to change at the discretion of IBM.     *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, Linux, and
  Windows.
  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R81A PSY

     UP

• R81L PSY

     UP

• R81W PSY

     UP