IT36312: OBJECT AGENT PANIC - "INVALID MEMORY ADDRESS OR NIL POINTER DEREFERENCE" WHEN DELETING OBJECTS

IBM Spectrum Protect Server V8.1 Fix Pack 13 (V8.1.13) Downloads

APAR status

• Closed as program error.

Error description

• Objects that are being offloaded to an IBM Spectrum Protect
  Server using the IBM Spectrum Protect Object Agent can consist
  of multiple parts. One example is an archive operation on IBM
  Spectrum Protect Plus. This operation is offloading 128MB
  Objects that are grouped together when they are part of the same
   disk of the source Virtual Machine being archived.
  When the IBM Spectrum Protect Object Agent tries to delete such
  a multipart object that is composed of more than the IBM
  Spectrum Protect Server TXNGroupMax setting (4096 by default),
  the Object Agent will panic with following error in the
  protect.log:
  
  <timestamp>    INFO    http: panic serving <ip>:<port>: runtime
  error: invalid memory address or nil pointer dereference
  goroutine 31 [running]:
  net/http.(*conn).serve.func1(0xc000792000)
    /usr/local/go1.14.4/src/net/http/server.go:1772 +0x139
  panic(0x11c36c0, 0x21db3c0)
    /usr/local/go1.14.4/src/runtime/panic.go:975 +0x3e3
  github.ibm.com/Spectrum-Protect/protect-api/api.(*Session).Delet
   e(0xc000516b40, 0x1589e00, 0xc0010144b0, 0xc000387c80, 0x7f,
   0x0, 0x0, 0x0, 0xc00048bab0, 0x8, ...)
    /extracts/diag/sppsup-2400/srv_git/src/github.ibm.com/Spectrum
   -Protect/protect-api/api/client.go:1780 +0xd35
  github.com/minio/minio/cmd/gateway/protect.(*protectObjects).Del
   eteObject(0xc00013e4b0, 0x1589e00, 0xc0010144b0, 0xc0000de786,
   0x1a, 0xc000387c80, 0x7f, 0x0, 0x0)
    /extracts/diag/sppsup-2400/srv_git/src/github.com/minio/minio/
   cmd/gateway/protect/gateway-protect.go:557 +0x344
  github.com/minio/minio/cmd.objectAPIHandlers.DeleteMultipleObje
   ctsHandler(0x13d77d8, 0x13d77d0, 0x157f440, 0xc00000cae0,
   0xc00037e700)
  
  The key in the protect.log is the panic in the delete call.
  
  At the same time, the IBM Spectrum Protect Server's activity log
  will show:
  ANR0445W Protocol error on session <SessionID> for node
   <ObjectClientNodeName> (Linux x86-64) - maximum group
   transaction size exceeded.
  ANR0484W Session <SessionID> for node <ObjectClientNodeName>
   (Linux x86-64) terminated - protocol violation detected.
  
  IBM Spectrum Protect Server Versions Affected: IBM Spectrum
  Protect Server 8.1.7 and newer
  
  Initial Impact: Low
  
  Additional Keywords: SPP, Cloud, Spectrum Protect Plus, S3,
  TS004995615
  

Local fix

• Update TXNGroupMax for the Spectrum Protect Server to 10002.
  Note this is a serverwide setting.
    setopt TXNGroupMax 10002
  Note that the S3 protocol that is used during offload allows a
  maximum of 10000 objects grouped together. Setting to
  TXNGroupMax to 10002 (10000 plus some overhead) should avoid
  this error.
  Setting the option on the Object Client will fail with "ANR2465E
  UPDATE NODE: The TXNGROUPMAX parameter is invalid for nodes with
  TYPE=OBJECTCLIENT."
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users of the S3 Object       *
  * Agent.                                                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in level 8.1.13. Note that this is     *
  * subject to change at the discretion of IBM.                  *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, Linux, and
  Windows.
  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R81A PSY

     UP

• R81L PSY

     UP

• R81W PSY

     UP