APAR status
Closed as program error.
Error description
When using the following IBM Spectrum Protect Plus vSnap restore from archive option value 'TRUE' with the command : vsnap system pref set --name archiveDownloadBeforeExtraction --value true as workaround referenced for APAR IT34012, the restore from archive can stop after many hours for guests disks of 1TB or more. In the job log the error can be seen after many hours of processing : SUMMARY,<timestamp>,CTGGA2398,Starting job for policy onDemandRestore_12345678912 (ID:1234). id -> <JobId>. IBM Spectrum Protect Plus version 10.1.7-3102. ... INFO,<timestamp>,2,CTGGA2589,Creating clones of vSnap volumes. DETAIL,<timestamp>,2,CTGGA2173,Cloning volume (<vSnapVolumeName>) from snapshot (<SnapshotName>) of archive volume (<ArchivedVolumeName>) DETAIL,<timestamp>,2,CTGGA0980,Starting to create volume clone (<vSnapVolumeName>) from snapshot (<SnapshotId>) using session ID (<ReplicationSessionId>). INFO,<timestamp>,2,CTGGA2287,Archive volume clone creation in progress: Clone (<vSnapVolumeName>) Data transferred (37893345871) Status message (Data transfer in progress. Downloaded 40106.95 MB. Throughput: 68.96 MB/s) ... INFO,<timestamp>,2,CTGGA2287,Archive volume clone creation in progress: Clone (<vSnapVolumeName>) Data transferred (4312463141953) Status message (Data transfer in progress. Downloaded 4148823.25 MB. Throughput: 18.09 MB/s) ERROR,<timestamp>,2,CTGGA0986,Async volume clone creation failed for volume (<ArchivedVolumeName>) snapshot (<SnapshotName>) error (RetriesExceededError: Max Retries Exceeded). ERROR,<timestamp>,2,CTGGA1110,No selected item is recoverable In the replication log, the actual error is displayed and after 5 retries, the replication session is aborted : [<timestamp>] INFO pid-1234 vsnap.repld Session <ReplicationSessionId> :worker started ... [<timestamp>] INFO pid-1234 vsnap.archive.mover Waiting for files from the archive provider. This may take several hours. .. Started restoring 16 files from from 16 objects .. Files will be downloaded before extracting. .. Session <ReplicationSessionId>: size received = 37893345871 (35.29GB) .. Session <ReplicationSessionId>: message = Data transfer in progress. Downloaded 40106.95 MB. Throughput: 159.60 MB/s ... <after a long time> [<timestamp>] WARNING pid-1234 vsnap.linux.system Ouput: ['star: Trying to access sparse aray beyond end (index <xxxx>).', "star: Error writing'<VMGuestDiskName>-flat.vmdk '.", 'star: Tar file too small (amount: 0 bytes).', 'star: Unexpected EOF on input.', 'star: Cannot recover from error - exiting.'] [<timestamp>] WARNING pid-1234 vsnap.archive.util Download file /vsnap/vpool1/fs123/a1b2c3/<VMName>.vm- <VMManagedObjectId>/<VMGuestDiskName>-flat.vmdk failed, retrying 1/5. Error: Command failed: star: Trying to access sparse array beyond end (index <xxxx>).; star: Error writing '<VMGuestDiskName>-flat.vmdk'.; star: Tar file too small (amount: 0 bytes).; star: Unexpected EOF on input.; star: Cannot recover from error - exiting. ... [<timestamp>] INFO pid-1234 vsnap.common.model Session <ReplicationSessionId>: message = Data transfer in progress. Downloaded 3025245.12 MB. Throughput: 20.05 MB/s [<timestamp>] WARNING pid-1234 vsnap.linux.system Return code 255: star x -C '/vsnap/vpool1/fs123/a1b2c3/<VMName>.vm- <VMManagedObjectId>/tmp_<zzzzzzzzzz>' --compress-program="lz4" -sparse -silent -no-statistics -f '/vsnap/vpool1/fs123/a1b2c3/<VMName>.vm- <VMManagedObjectId>/tmp_<zzzzzzzzzz>/<VMGuestDiskName>- flat.vmdk.tar.lz4' [<timestamp>] WARNING pid-1234 vsnap.linux.system Ouput: ['star: Trying to access sparse aray beyond end (index <xxxx>).', "star: Error writing '<VMGuestDiskName>-flat.vmdk'.", 'star: Tar file too small (amount: 0 bytes).', 'star: Unexpected EOF on input.', 'star: Cannot recover from error - exiting.'] [<timestamp>] WARNING pid-1234 vsnap.archive.util Download file /vsnap/vpool1/fs123/a1b2c3/<VMName>.vm- <VMManagedObjectId>/<VMGuestDiskName>-flat.vmdk failed after 5 attempts. Error: Command failed: star: Trying to access sparse aray beyond end (index <xxxx>).; star: Error writing '<VMGuestDiskName>-flat.vmdk'.; star: Tar file too small (amount: 0 bytes).; star: Unexpected EOF on input.; star: Cannot recover from error - exiting. The archived data is consistent but the vSnap fails to get it restored as expected. | MDVRPARTL 5737SPLUS 10.1.6 | IT34012 IBM Spectrum Protect Plus Versions Affected: IBM Spectrum Protect Plus 10.1.7 and later Additional Keywords: SPP, SPPLUS, TS005816967, restore, start, compression, IT34012
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: * * IBM Spectrum Protect Plus level 10.1.7 and 10.1.8 * **************************************************************** * PROBLEM DESCRIPTION: * * See Error Description * **************************************************************** * RECOMMENDATION: * * Apply fixing level when available. This problem is currently * * projected to be fixed in IBM Spectrum Protect Plus level * * 10.1.9. Note that this is subject to change at the * * discretion of IBM. * ****************************************************************
Problem conclusion
The problem occurred because files larger than 1TB which were uploaded to archive as multiple parts were not reconstructed correctly during restore. The issue has been resolved by implementing code fixes to ensure parts of large files are correctly reconstructed and uncompressed during restore from archive to vSnap.
Temporary fix
Comments
APAR Information
APAR number
IT37272
Reported component name
SP PLUS
Reported component ID
5737SPLUS
Reported release
A17
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2021-06-15
Closed date
2021-09-29
Last modified date
2021-09-29
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Modules/Macros
vSnap Archive
Fix information
Fixed component name
SP PLUS
Fixed component ID
5737SPLUS
Applicable component levels
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSNQFQ","label":"IBM Spectrum Protect Plus"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"A17","Line of Business":{"code":"LOB26","label":"Storage"}}]
Document Information
Modified date:
31 January 2024