IBM Support

IT35733: VSNAP CLOUD MAINTENANCE SESSIONS INDEFINITELY STUCK IN PENDING STATE WHEN USING WORM BUCKET

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • Buckets created on IBM Cloud Object Storage (COS) can have
    active WORM (Write Once Read Many) retention rules applied to
    the stored objects.
    
    When a backup copy stored in the cloud endpoint is eligible for
    expiration, the IBM Spectrum Protect Plus Maintenance job
    informs the vSnap server to expire the copy.
    At the vSnap level, the incremental block level data is stored
    across many cloud objects.
    The vSnap sends object deletion requests to the cloud endpoint
    to remove objects that are no longer needed.
    If some of the objects are prevented from deletion due to the
    WORM retention rules, the Maintenance job reflects that the
    recovery point was correctly expired, but at the vSnap level
    the failed objects are added to a background maintenance queue.
    
    The vSnap processes the maintenance queue and keeps retrying
    the cloud object deletion every day until it is successful.
    The list of pending sessions in the queue can be seen using the
    following command on the vSnap:
    vsnap cloud maint show
    
    Once the WORM retention lock is lifted and the objects are
    successfully deleted, the vSnap is supposed to mark the
    session as completed.
    Due to a defect, the session on the vSnap remains in PENDING
    state even after the objects were successfully deleted.
    This causes a large number of PENDING sessions to build up
    unnecessarily in the queue.
    
    An additional symptom is that the following warnings may be
    seen in /opt/vsnap/log/maint.log:
    WARNING pid-xxxx vsnap.maint Failed to delete cloud objects
                                 from partner id <CloudPartnerID>:
                                 'str' object has no attribute
                                 'object_path'
    
    IBM Spectrum Protect Versions Affected:
    IBM Spectrum Protect Plus 10.1.6, 10.1.7
    
    | MDVREGR 10.1.5.0-TIV_5737SPLUS |
    
    Initial Impact:
    Medium
    
    Additional Keywords: SPP, SPPLUS, TS004671157, worm
                         retention, maintenance, pending
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:                                              *
    * IBM Spectrum Protect Plus levels 10.1.6 and 10.1.7.          *
    ****************************************************************
    * PROBLEM DESCRIPTION:                                         *
    * See Error Description.                                       *
    ****************************************************************
    * RECOMMENDATION:                                              *
    * Apply the fixing level when available. This problem is       *
    * projected to be fixed in IBM Spectrum Protect Plus level     *
    * 10.1.8. Note that this is subject to change at the           *
    * discretion of IBM.                                           *
    ****************************************************************
    

Problem conclusion

  • The vSnap maintenance code was fixed to ensure that sessions are
    correctly marked as completed instead of remaining stuck in
    pending state. Upon upgrade to the fix level, old sessions that
    were stuck in pending state will gradually go into completed
    state. Note that the change may not be reflected immediately.
    Existing pending sessions are retried after seven days after the
    last attempt, so it may take a week or more before old pending
    sessions are cleared. Also note that many maintenance sessions
    may legitimately remain in pending state for extended periods of
    time because they are associated with cloud objects that
    WORM-locked and not yet eligible for deletion.
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT35733

  • Reported component name

    SP PLUS

  • Reported component ID

    5737SPLUS

  • Reported release

    A17

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2021-01-29

  • Closed date

    2021-03-25

  • Last modified date

    2021-03-25

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    SP PLUS

  • Fixed component ID

    5737SPLUS

Applicable component levels

[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSNQFQ","label":"IBM Spectrum Protect Plus"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"A17","Line of Business":{"code":"LOB26","label":"Storage"}}]

Document Information

Modified date:
31 January 2024