IT35733: VSNAP CLOUD MAINTENANCE SESSIONS INDEFINITELY STUCK IN PENDING STATE WHEN USING WORM BUCKET

APAR status

• Closed as program error.

Error description

• Buckets created on IBM Cloud Object Storage (COS) can have
  active WORM (Write Once Read Many) retention rules applied to
  the stored objects.
  
  When a backup copy stored in the cloud endpoint is eligible for
  expiration, the IBM Spectrum Protect Plus Maintenance job
  informs the vSnap server to expire the copy.
  At the vSnap level, the incremental block level data is stored
  across many cloud objects.
  The vSnap sends object deletion requests to the cloud endpoint
  to remove objects that are no longer needed.
  If some of the objects are prevented from deletion due to the
  WORM retention rules, the Maintenance job reflects that the
  recovery point was correctly expired, but at the vSnap level
  the failed objects are added to a background maintenance queue.
  
  The vSnap processes the maintenance queue and keeps retrying
  the cloud object deletion every day until it is successful.
  The list of pending sessions in the queue can be seen using the
  following command on the vSnap:
  vsnap cloud maint show
  
  Once the WORM retention lock is lifted and the objects are
  successfully deleted, the vSnap is supposed to mark the
  session as completed.
  Due to a defect, the session on the vSnap remains in PENDING
  state even after the objects were successfully deleted.
  This causes a large number of PENDING sessions to build up
  unnecessarily in the queue.
  
  An additional symptom is that the following warnings may be
  seen in /opt/vsnap/log/maint.log:
  WARNING pid-xxxx vsnap.maint Failed to delete cloud objects
                               from partner id <CloudPartnerID>:
                               'str' object has no attribute
                               'object_path'
  
  IBM Spectrum Protect Versions Affected:
  IBM Spectrum Protect Plus 10.1.6, 10.1.7
  
  | MDVREGR 10.1.5.0-TIV_5737SPLUS |
  
  Initial Impact:
  Medium
  
  Additional Keywords: SPP, SPPLUS, TS004671157, worm
                       retention, maintenance, pending
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus levels 10.1.6 and 10.1.7.          *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply the fixing level when available. This problem is       *
  * projected to be fixed in IBM Spectrum Protect Plus level     *
  * 10.1.8. Note that this is subject to change at the           *
  * discretion of IBM.                                           *
  ****************************************************************
  

Problem conclusion

• The vSnap maintenance code was fixed to ensure that sessions are
  correctly marked as completed instead of remaining stuck in
  pending state. Upon upgrade to the fix level, old sessions that
  were stuck in pending state will gradually go into completed
  state. Note that the change may not be reflected immediately.
  Existing pending sessions are retried after seven days after the
  last attempt, so it may take a week or more before old pending
  sessions are cleared. Also note that many maintenance sessions
  may legitimately remain in pending state for extended periods of
  time because they are associated with cloud objects that
  WORM-locked and not yet eligible for deletion.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels