IT38104: OBJECT AGENT GET REQUESTS POOR PERFORMANCE

8.1.12.200-IBM-SPSRV-WindowsX64
8.1.12.200-IBM-SPSRV-Linuxx86_64
8.1.12.200-IBM-SPSRV-Linuxs390x
8.1.12.200-IBM-SPSRV-Linuxppc64le
8.1.12.200-IBM-SPSRV-AIX
IBM Spectrum Protect Server V8.1 Fix Pack 13 (V8.1.13) Downloads
IBM Spectrum Protect Server V8.1.12.X interim fix downloads

APAR status

• Closed as program error.

Error description

• The IBM Spectrum Protect Object Agent was introduced starting
  with version 8.1.7.
  The GET requests sent through the Object Agent Amazon S3
  protocol use two different IBM Spectrum Protect sessions to
  complete the operation.
  The request queries for information about the requested object
  by calling the GetObjectInfo function in a session and then
  makes the actual request for the data in another session.
  This leads to slowdown in handling the request as the request
  has to get back in the session queue to handle the actual read
  of the data.
  When the system is under load this waiting for a second session
  can greatly impact the overall response time and eventually see
  client requests timing out.
  
  This "two requests per GET operation" behaviour can be seen in
  the Object Agent protect.log by the following pattern:
  
  Here the GET request <request_1> is received:
  
  <timestamp>  INFO  gateway-protect.go GetObject:394,
                  access:12345ABCD, node: <ObjectClientNodeName>,
                 request: <request_1>, remote: <ClientIP>:<port>.
  
  This immediately is followed by a high level GetObjectInfo with
  a different request ID <request_2>:
  
  <timestamp>  INFO  gateway-protect.go GetObjectInfo:437,
                  access:12345ABCD, node: <ObjectClientNodeName>,
                 request: <request_2>, remote: <ClientIP>:<port>.
  
  The GetObjectInfo request <request_2> is then handled:
  
  <timestamp>  INFO  Session selected.
       {"requestID": "<request_2>",
            "class": "pooling",
          "session": {"node": "<ObjectClientNodeName>",
                  "endpoint":"<ServerIP>:<port>",
                        "id": "40MCMA",
                  "requests": 954,
                 "requestId":"<request_2>"},
          "elapsed": "285.672ms"}
  <timestamp>  INFO  -> Entry at client.go ObjectInfo:679.
       {"requestID":"<request_2>",
            "class": "entry",
           "object": {"object": "SPS3TOOL/ingestSKYWALKER01/
                                 THREAD.18.FILE.249-- head",
                "storageclass": "",
                       "range": "0 for 0 bytes"}}
  <timestamp>  INFO  <- Exit at client.go ObjectInfo:686.
       {"requestID":"<request_2>",
            "class": "exit",
          "objMeta": {"ObjectName": {"HL":"SPS3TOOL/
                                           ingestSKYWALKER01/
                                           THREAD.18.FILE.249",
                                           "LL": ""},
                       "FileSpace": {"node":
                                        "<ObjectClientNodeName>",
                                        "fsName": "testbucket",
                                        "fsID": 1},
               "ProtectObjectInfo": {"Size": 16777216,
                       "ETag":"3ce1ad78fe1eb2cee2a4cc3dbd8efe05",
                              "IsMultipart": false,
                              "IsCommitted": false,
                             "StorageClass":"STANDARD"},
                         "InsDate": "2021-07-27T13:39:16.000Z",
                        "ObjectID": 60812808,
                           "IsDir": false,
                         "InGroup": false,
                     "ObjectState": {"objectState": 1},
                          "OnTape": false}}
  
  The session <request_1> for the GET request is then selected:
  
  <timestamp>  INFO  Session selected.
       {"requestID": "<request_1>",
            "class": "pooling",
          "session": {"node": "<ObjectClientNodeName>",
                  "endpoint":"<ServerIP>:<port>",
                        "id": "40MCMA",
                  "requests": 962,
                 "requestId":"<request_1>"},
                   "elapsed": "292.312ms"}
  
  And the remainder of the GET request is handled.
  
  IBM Spectrum Protect Versions Affected:
  IBM Spectrum Protect Server (Object Agent) 8.1.7 and later
  
  Additional Keywords: TS006558881, cloud, SPP, offload,
                       copy to cloud
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 8.1.12.200 and 8.1.13. Note  *
  * that this is subject to change at the discretion of IBM.     *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, Linux, and
  Windows.
  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R81A PSY

     UP