IT37019: ANR3701E CANNOT CONNECT TO THE CLOUD SERVICE PROVIDER DUE TO JAVA OUT OF MEMORY ERROR

8.1.12.200-IBM-SPSRV-WindowsX64
8.1.12.200-IBM-SPSRV-Linuxx86_64
8.1.12.200-IBM-SPSRV-Linuxs390x
8.1.12.200-IBM-SPSRV-Linuxppc64le
8.1.12.200-IBM-SPSRV-AIX
8.1.12.100-IBM-SPCMS-WindowsX64
8.1.12.100-IBM-SPCMS-WindowsI32
8.1.12.100-IBM-SPCMS-Linuxx86_64
8.1.12.100-IBM-SPOC-WindowsX64
8.1.12.100-IBM-SPOC-Linuxx86_64
8.1.12.100-IBM-SPOC-Linuxs390x
8.1.12.100-IBM-SPOC-LinuxPPC64le
8.1.12.100-IBM-SPOC-AIX
8.1.12.100-IBM-SPSRV-WindowsX64
8.1.12.100-IBM-SPSRV-Linuxx86_64
8.1.12.100-IBM-SPSRV-Linuxs390x
8.1.12.100-IBM-SPSRV-Linuxppc64le
8.1.12.100-IBM-SPSRV-AIX
IBM Spectrum Protect Server V8.1.12.X interim fix downloads
IBM Spectrum Protect Server V8.1 Fix Pack 13 (V8.1.13) Downloads

APAR status

• Closed as program error.

Error description

• When sending a request to a cloud service provider requests fail
  with ANR3701E indicating that a connection cannot be
  established. Upon investigation in the dsmffdc.log, messages
  there indicate "java.lang.IllegalStateException: Connection pool
  shut down" and/or "java.lang.OutOfMemoryError: Java heap space".
  
  Other operations that use IBM Spectrum Protect's embedded Java
  Virtual Machine may also see the "java.lang.OutOfMemoryError:
  Java heap space". These could include any cloud container
  storage pool request, Operation Center requests to the Spectrum
  Protect server, and Db backup to cloud operations.
  
  Customer/L2 Diagnostics:
  A common symptom presented to the user is "ANR3701E Cannot
  connect to the cloud service provider for the delete container
  operation on the SOMESTORAGEPOOL connection or storage pool."
  Other messages indicating cloud operations failed are also
  possible.
  
  The primary indicator of the problem shows up in the
  dsmffdc.log:
  [04-21-2021 19:27:06.377][ FFDC_GENERAL_SERVER_ERROR ]:
  (jvm.c:1854) ANR0161I Displaying segment 1 for message insert
  [ID1:19:27:06.372] with  text
  com.tivoli.dsm.cloud.api.ProviderS3Retry$5 singleAttempt
  Throwable out of callOp that wasn't handled
  java.lang.OutOfMemoryError: Java heap space
     at
  java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
     at
  java.io.BufferedInputStream.read1(BufferedInputStream.java:297)
     at
  java.io.BufferedInputStream.read(BufferedInputStream.java:356)
     at
  com.amazonaws.internal.SdkBufferedInputStream.read(SdkBufferedIn
  putStream.java:76)
  
  And a side effect dsmffdc.log message is this:
  20:05:43.042 [3][ffdcutil.c][432][FFDCLogThread]:[03-15-2021
  20:05:43.042][ FFDC_GENERAL_SERVER_ERROR ]: (jvm.c:1854)
  ANR0161I Displaying segment 1 for message insert
  [ID1:20:05:43.040] with  text com.tivoli.dsm
  .cloud.api.ProviderS3Retry$13 run Error in the ProviderS3Retry
  run method.
  java.lang.IllegalStateException: Connection pool shut down
     at org.apache.http.util.Asserts.check(Asserts.java:34)
     at
  org.apache.http.impl.conn.PoolingHttpClientConnectionManager.req
  uestConnection(PoolingHttpClientConnectionManager.java:269)
     at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
  Source)
     at
  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod
  AccessorImpl.java:55)
     at java.lang.reflect.Method.invoke(Method.java:508)
  
  This second message may also flood the dsmffdc.log files making
  it impossible to see the originating out of memory error.
  
  IBM Spectrum Protect Versions Affected:
  8.1.12
  
  Initial Impact:
  IBM Spectrum Protect will no longer be able to perform cloud
  requests until the server is restarted.
  
  Additional Keywords:
  OutOfMemory, Cloud Object Storage, Connection pool shut down
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 8.1.12.200 and 8.1.13. Note  *
  * that this is subject to change at the discretion of IBM.     *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, Linux, and
  Windows.
  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels