IT41720: MEMORY LEAK IN SDEXTENDCLOUDOBJECTLOCKTHREAD LEADING TO CRASH

8.1.15.100-IBM-SPSRV-WindowsX64
8.1.15.100-IBM-SPSRV-Linuxx86_64
8.1.15.100-IBM-SPSRV-Linuxs390x
8.1.15.100-IBM-SPSRV-Linuxppc64le
8.1.15.100-IBM-SPSRV-AIX
8.1.16.000-IBM-SPCMS-WindowsX64
8.1.16.000-IBM-SPCMS-WindowsI32
8.1.16.000-IBM-SPCMS-Linuxx86_64
8.1.16.000-IBM-SPOC-WindowsX64
8.1.16.000-IBM-SPOC-Linuxx86_64
8.1.16.000-IBM-SPOC-Linuxs390x
8.1.16.000-IBM-SPOC-LinuxPPC64le
8.1.16.000-IBM-SPOC-AIX
8.1.16.000-IBM-SPSRV-WindowsX64
8.1.16.000-IBM-SPSRV-Linuxx86_64
8.1.16.000-IBM-SPSRV-Linuxs390x
8.1.16.000-IBM-SPSRV-Linuxppc64le
8.1.16.000-IBM-SPSRV-AIX

APAR status

• Closed as program error.

Error description

• When using a cloud container storage pool with cloud data lock
  enabled, the SdExtendCloudObjectLockThread leaks memory over
  time and will lead to an out of memory condition.
  
  Customer/L2 Diagnostics:
  Server will run out of memory and crash. Before that the leak
  can be seen in The primary symptom is shown in SHOW ALLOC with
  these entries:
     sdcntr.c line 14873:       44 entries for          3520 bytes
  (ExtendCloudObjectLock) thread 65(SdExtendCloudObjectLockThread
  parent=1)
     sdcntr.c line  4448:       44 entries for      14085632 bytes
  (BigLinkedListAllocTracked) thread
  65(SdExtendCloudObjectLockThread parent=1)
     sdcntr.c line  4455:       44 entries for       8805632 bytes
  (BigLinkedListAllocTracked) thread
  65(SdExtendCloudObjectLockThread parent=1)
      pkmon.c line   509:       44 entries for          5632 bytes
  (pkCreateNamedMutex) (descP->rangeMutex)
     sdcntr.c line  4681:       44 entries for          3520 bytes
  (SdGetContainerDescTracked) thread
  65(SdExtendCloudObjectLockThread parent=1)
     sdcntr.c line  4708:       44 entries for          2112 bytes
  (SdGetContainerDescTracked) thread
  65(SdExtendCloudObjectLockThread parent=1)
  
  Over time these SdExtendCloudObjectLockThread should remain
  relatively low and not increase. In fact, If cloud containers
  are not currently needing their lock extended, these entries
  should not be seen in SHOW ALLOC. This output can be viewed on a
  running server or is recorded by always on servermon.
  
  IBM Spectrum Protect Versions Affected: 8.1.15
  
  Initial Impact: Server running out of memory.
  
  Additional Keywords:
  Cloud Data Lock, AWS Object Lock
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 8.1.15.100 and 8.1.16. Note  *
  * that this is subject to change at the discretion of IBM.     *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, Linux, and
  Windows.
  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels