IBM Support

Flash: Potential undetected backup data corruption after adding a new virtual disk to a VMware VM: Subsequent incremental forever backups are corrupted and cannot be restored (APAR IT43031)

Flashes (Alerts)


Abstract

A defect in IBM Storage Protect for Virtual Environments: Data Protection for VMware 8.1.12.0 - 8.1.17.2 can lead to undetected data corruption in VM backups. Specifically, all incremental backups of a VMware virtual machine (VM) that are created after a new disk is added to the VM are potentially corrupted.

The result of this undetected data corruption is that the VM cannot be restored correctly from the affected backup versions. The restored VM might not boot. If the VM can be booted, at least some data is missing or corrupted.

A fix is available beginning with version 8.1.19.0 (June 2023). If you ever used an affected version of Data Protection for VMware in your environment, then IBM strongly recommends that you install the fix, create full backups of your VMs, and then resume incremental backups. Existing corrupted backup versions cannot be repaired.

Content

Problem Description
Note: The product now known as IBM Storage Protect was named IBM Spectrum Protect in levels earlier than 8.1.19. To learn more about the brand change, see IBM Spectrum Protect brand change to IBM Storage Protect.
After a new virtual disk (VMDK) is added to a VM, all subsequent incremental backups created by IBM Storage Protect for Virtual Environments: Data Protection for VMware 8.1.12.0 - 8.1.17.2 are corrupted. The first incremental backup that occurs after the disk was added shows that 0 (zero) bytes were sent for the new disk. Each subsequent incremental backup might show a different number of bytes backed up. However, all of the backups that are created after the disk is added are potentially corrupted.
After restore of an affected backup version, the restored VM might not boot. If the VM can be booted, then data might be missing or corrupted.
There is no deterministic method to identify which backup versions are corrupted. Corrupted backup versions cannot be repaired.
Important: The corrupted backup chain persists even after the fixing version is installed. A new full backup must be created to initiate a new incremental forever backup chain.
Affected Levels
  • IBM Storage Protect for Virtual Environments: Data Protection for VMware 8.1.12.0 - 8.1.17.2.
The problem occurs for the affected Data Protection for VMware versions without regard to the VMware version. The problem does not occur in other versions of Data Protection for VMware.
Immediate Action
IBM strongly recommends the following steps:
  1. Identify the version of Data Protection for VMware that you use.
  2. If you use version 8.1.11.0 or earlier, then no further action is needed. When you plan a future upgrade, then upgrade to version 8.1.19.0 or later.
  3. If you use version 8.1.19.0 or later and if you never previously used 8.1.12.0 - 8.1.17.2, then no further action is needed. If you are not sure whether you ever used 8.1.12.0 - 8.1.17.2, then proceed to step 5.
  4. If you use version 8.1.12.0 - 8.1.17.2, then install IBM Storage Protect for Virtual Environments: Data Protection for VMware 8.1.19.0 or later.
  5. Back up all VMs with option -mode=iffull to create full backups of each VM.
  6. Resume incremental backups of the VMs.
IBM understands that installing the fixed version might require change control planning. If the fixed version cannot be installed immediately, then follow these alternate steps:
  1. Back up all VMs with option -mode=iffull to create full backups of each VM.
  2. Resume incremental backups of the VMs.
  3. Important: If you add a disk to a VM, then immediately create a full backup of the VM to prevent a recurrence of the problem.
  4. Plan an upgrade to a fixed version as soon as possible.
Considerations for full backups:
  • Backup processing time is longer because all used blocks are read from each VM's disks.
  • If you use client-side deduplication, then only unique data is sent to the IBM Storage Protect Server. Client-side deduplication can mitigate the amount of data transferred over the network and the amount of data inserted into the server's storage pools.
  • If you use server-side deduplication, then all used blocks are sent across the network to the IBM Storage Protect server. However, the server does not insert the duplicate data into the storage pool. Server-side deduplication avoids the insertion of duplicate data into the server's storage pools.
  • If the backup data is stored in a nondeduplicated storage pool, then all used blocks are sent across the network to the IBM Storage Protect server, and stored in the server's storage pools. Ensure the IBM Storage Protect server has sufficient storage to ingest the full backups.
Fix
The problem is fixed beginning in IBM Storage Protect Backup-Archive Client 8.1.19.0.

Related Information

[{"Type":"MASTER","Line of Business":{"code":"LOB26","label":"Storage"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSERB6","label":"IBM Spectrum Protect for Virtual Environments"},"ARM Category":[{"code":"a8m3p000000hAaNAAU","label":"Data Protection-\u003EVirtual Environments (VE)"}],"Platform":[{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"8.1.12;8.1.13;8.1.14;8.1.15;8.1.17"}]

Document Information

Modified date:
16 January 2024

UID

ibm17105603