IT41803: SPECTRUM PROTECT SERVER MAY FREEZE BECAUSE OF FILE HANDLE LEAK IF CLOUD READCACHE IS ENABLED

8.1.17.000-IBM-SPOC-WindowsX64
8.1.17.000-IBM-SPOC-Linuxx86_64
8.1.17.000-IBM-SPOC-Linuxs390x
8.1.17.000-IBM-SPOC-LinuxPPC64le
8.1.17.000-IBM-SPOC-AIX
8.1.17.000-IBM-SPSRV-WindowsX64
8.1.17.000-IBM-SPSRV-Linuxx86_64
8.1.17.000-IBM-SPSRV-Linuxs390x
8.1.17.000-IBM-SPSRV-Linuxppc64le
8.1.17.000-IBM-SPSRV-AIX
8.1.16.100-IBM-SPSRV-WindowsX64
8.1.16.100-IBM-SPSRV-Linuxx86_64
8.1.16.100-IBM-SPSRV-Linuxs390x
8.1.16.100-IBM-SPSRV-Linuxppc64le
8.1.16.100-IBM-SPSRV-AIX
8.1.15.100-IBM-SPSRV-WindowsX64
8.1.15.100-IBM-SPSRV-Linuxx86_64
8.1.15.100-IBM-SPSRV-Linuxs390x
8.1.15.100-IBM-SPSRV-Linuxppc64le
8.1.15.100-IBM-SPSRV-AIX

APAR status

• Closed as program error.

Error description

• Spectrum Protect Server may experience hang during read from
  cloud if the cloud readcache is enabled.
  Support/customer diagnostics:
  
  1: When the problem happens, the following message may be logged
  in actlog:
  
  << Message number XXXX not available for language AMENG:  read
  error YYYY at outmsg.c(681) >>
  ANR9999D_2048560863 FFDCLogThread(ffdcutil.c:465)
  Thread<3>: Error 9 writing to FFDCLOG. Msg:
  com.tivoli.dsm.cloud.api.ProviderS3Retry handleException
  com.amazonaws.SdkClientException Unable to execute HTTP
  request: Too many open files
  
  2: using OS command to monitor the open files can catch the
  handle increase :
  
  For example
  procfiles pid_of_dsmserv , run every 1 hour can show when the
  problem happens, dsmserv process has 10s of thousands of files
  open with READONLY:
  
     20128: S_IFREG mode:0444 dev:48,9 ino:40767 uid:1001 gid:1000
  rdev:0,0 O_RDONLY size:50091458
     20129: S_IFREG mode:0444 dev:48,7 ino:47085 uid:1001 gid:1000
  rdev:0,0 O_RDONLY size:7751442
     20130: S_IFREG mode:0444 dev:48,9 ino:45319 uid:1001 gid:1000
  rdev:0,0 O_RDONLY size:170532380
     20131: S_IFREG mode:0444 dev:48,7 ino:46934 uid:1001 gid:1000
  rdev:0,0 O_RDONLY size:1073739144
     20132: S_IFREG mode:0444 dev:48,9 ino:40597 uid:1001 gid:1000
  rdev:0,0 O_RDONLY size:24897933
     20133: S_IFREG mode:0444 dev:48,1 ino:1282 uid:1001 gid:1000
  rdev:0,0  O_RDONLY size:1073733584
     20134: S_IFREG mode:0444 dev:48,7 ino:46958 uid:1001 gid:1000
  rdev:0,0 O_RDONLY size:50883323
  
  Version/Platform affected:
  Spectrum Protect server on all supported platform with cloud
  readcache enabled
  

Local fix

• Disable CloudReadCache for the cloud container storage pool.
  Restart instance if the server is affected by IT41584
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in 8.1.15.100, 8.1.16.100, and 8.1.17. *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release: AIX, Linux, and
  Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels