IT38103: POOR PERFORMANCE OF THE OBJECT AGENT DUE TO LOCK CONTENTION WHEN UNDER LOAD

8.1.12.200-IBM-SPSRV-WindowsX64
8.1.12.200-IBM-SPSRV-Linuxx86_64
8.1.12.200-IBM-SPSRV-Linuxs390x
8.1.12.200-IBM-SPSRV-Linuxppc64le
8.1.12.200-IBM-SPSRV-AIX
IBM Spectrum Protect Server V8.1 Fix Pack 13 (V8.1.13) Downloads
IBM Spectrum Protect Server V8.1.12.X interim fix downloads

APAR status

• Closed as program error.

Error description

• The IBM Spectrum Protect Object Agent was introduced starting
  with version 8.1.7.
  When there are many concurrent Amazon S3 protocol requests
  (S3 requests) that are being made on the same bucket which
  exists as a filespace on the IBM Spectrum Protect server, the
  server experiences lock contention which leads to an overall
  slow down in processing S3 requests.
  The external symptoms can be seen by a client using Amazon S3
  protocol as a general slowdown of its S3 requests, which can
  also include requests taking so long that they would time out.
  On the IBM Spectrum Protect server, investigating the servermon
  data collection, further evidence of this can be seen by
  examining the SHOW THREADS output and noting the number of
  threads in the tmLockTracked function.
  The problem is seen if there are a great number of threads as
  seen in the example stacks below:
  
  About 50% of the time, the sessions in TmLockWait were here:
        0x000000010007fb30 tmLockTracked
        0x0000000100216aa0 ImLockFileSpaceTracked
        0x00000001002187e8 ImLockCopyType
        0x0000000100235308 IPRA.$LockFilespaces
        0x00000001002332c4 imUpdateInventory
        0x000000010022659c imPrepareTxn
        0x0000000100074054 CollectVotes
  
  About 25% of the time they were here:
        0x000000010007fb30 tmLockTracked
        0x00000001002166e4 ImLockFileSpaceTracked
        0x0000000100b33630 OpenBackupQuery
        0x00000001007752a4 IPRA.$DoBackQry
        0x000000010075d94c SmNodeSession
        0x000000010074e0d8 IPRA.$HandleNodeSession
        0x0000000100738850 smExecuteSession
  
  And some less common stacks (<10% each):
        0x000000010007fb30 tmLockTracked
        0x00000001002166e4 ImLockFileSpaceTracked
        0x0000000100bc2ee8 imIsGroupLeader
        0x0000000100763898 SmNodeSession
        0x000000010074e0d8 IPRA.$HandleNodeSession
        0x0000000100738850 smExecuteSession
        0x0000000100544328 psSessionThread
  
        0x000000010007fb30 tmLockTracked
        0x00000001002166e4 ImLockFileSpaceTracked
        0x0000000100b3e500 imRetrieveBackup
        0x000000010077a3c8 SmDoObjRtrv
        0x000000010075db84 SmNodeSession
        0x000000010074e0d8 IPRA.$HandleNodeSession
        0x0000000100738850 smExecuteSession
  
          0x000000010007fb30 tmLockTracked
        0x00000001001f2804 ImLockNodeIdTracked
        0x00000001001d3c00 imGetFsId
        0x000000010079d024 SmParseBackInsNormE
        0x000000010077057c SmDoBackInsNormEnhanced
        0x000000010076206c SmNodeSession
        0x000000010074e0d8 IPRA.$HandleNodeSession
  
          0x000000010007fb30 tmLockTracked
        0x00000001002166e4 ImLockFileSpaceTracked
        0x00000001001d776c imGetFsNameEx
        0x00000001001f4190 IPRA.$ResolveObject
        0x00000001001f3c10 imResolveObjectx
        0x000000010079f600 SmReportRetrieve
        0x0000000100779c78 SmDoObjRtrv
  
  IBM Spectrum Protect Versions Affected:
  IBM Spectrum Protect Server (Object Agent) 8.1.7 and later
  
  Additional Keywords: TS006309420, SPP, offload, copy to cloud
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 8.1.12.200 and 8.1.13. Note  *
  * that this is subject to change at the discretion of IBM.     *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, Linux, and
  Windows.
  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R81A PSY

     UP