IT37419: SLOW COPY TO CLOUD OPERATION FROM S3 CLIENTS TO THE IBM SPECTRUM PROTECT SERVER ON AIX

8.1.12.200-IBM-SPSRV-WindowsX64
8.1.12.200-IBM-SPSRV-Linuxx86_64
8.1.12.200-IBM-SPSRV-Linuxs390x
8.1.12.200-IBM-SPSRV-Linuxppc64le
8.1.12.200-IBM-SPSRV-AIX
IBM Spectrum Protect Server V8.1 Fix Pack 13 (V8.1.13) Downloads
IBM Spectrum Protect Server V8.1.12.X interim fix downloads

APAR status

• Closed as program error.

Error description

• Starting with AIX Version 7.2 TL 1 and SP 4 or later operating
  system, running on an IBM POWER8® or later server the version
  8.1.9 or later of the IBM Spectrum Protect Object Agent running
  on AIX is supported.
  
  Poor HTTP PUT request operation performance may be experienced
  on AIX with the IBM Spectrum Protect Object Agent that provides
  an Amazon S3 (Simple Storage Service) API interface for storing
  data into IBM Spectrum Protect.
  This includes "copy" operations which copy data to IBM Spectrum
  Protect as a S3 object storage endpoint using IBM Spectrum
  Protect Plus SLA.
  
  This performance issue is due to the use of
  non-hardware-accelerated function calls in the Golang language
  library for performing data payload SHA256 hashing operations.
  When the "x-amz-content-sha256" header is included in a
  HTTP PUT operation to a S3 endpoint, that endpoint must do
  SHA256 hashing verification on the data payload included with
  that HTTP PUT operation.
  These operations can perform poorly on AIX since this operation
  is done without hardware acceleration on the "ppc64" system
  architecture.
  
  In addition, the default IBM Spectrum Protect Object Agent
  configuration values for session pooling towards the IBM
  Spectrum Protect Server need extended timeout values to reduce
  the number of used sessions for better performance.
  
  IBM Spectrum Protect Versions Affected:
  IBM Spectrum Protect Server 8.1.9 and later on AIX
  
  Additional Keywords: TS004042513, SPP, backup, offload
  

Local fix

• There is no workaround for the slow HTTP PUT operations but the
  IBM Spectrum Protect Object Agent session timeout values can be
  changed to have a moderate positive impact on performance.
  
  1. The Object Agent configuration file will exist after it has
     been started a first time.
     Update the "timeout" and "maximumTime" options in the
     configuration file <ObjectAgentName>_config.json located in
     /home/<InstanceUser>/<InstanceUser>/SPPOBJAGENT/ as follows:
        From :     "timeout":   30000000000
               "maximumTime":  240000000000
          to :     "timeout":  120000000000
               "maximumTime": 1680000000000
     The maximumTime parameter controls how long a session can
     receive data before closing and a new session is opened.
     The timeout parameter controls how long the SPPOBJAGENT
     server sessions are kept before timing out.
  2. Apply the changes:
     a. Wait for an opportunity where no SPP copy to SP jobs are
        in progress.
     b. Determine the object agent subsystem name
          lssrc -a | grep SPP
     c. Stop the object agent sub system
          stopsrc -s <ObjectAgentName>
     d. Backup the orignal conf file
          cd /home/<InstanceUser>/<InstanceUser>/SPPOBJAGENT
          cp <ObjectAgentName>_config.json
  <ObjectAgentName>_config.json_sav
     e. Edit <ObjectAgentName>_config.json, and apply changes
        discussed above the the two parameters in the sessionPool
        stanza
     f. Restart the object agent
          startsrc -s <ObjectAgentName>
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users using the Object Agent *
  * on AIX.                                                      *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 8.1.12.200 and 8.1.13. Note  *
  * that this is subject to change at the discretion of IBM.     *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX.
  Platforms fixed:  AIX.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R81A PSY

     UP