IT26969: INCORRECT BACKUP STATUS REPORTED BY VADP PROXY CAUSES DATA INTEGRITY ISSUE ON THE TARGET BACKUP VOLUME

APAR status

• Closed as program error.

Error description

• An IBM Spectrum Protect Plus backup may become unrecoverable if
  the VADP proxy incorrectly reports a backup to be complete. The
  incorrect status is reported when the VM has multiple disks and
  the last disk has no new changes since last backup, in this case
  the data transfer for the other disks are stopped prematurely
  and VADP treats the backup as completed. The next incremental
  will use the last changeId however since the last data transfer
  was not complete it will lead to data integrity issues since
  there a potentially missing blocks on the target volume.
  
  VMs with multiple VMDK disks can be impacted by this, it is more
  likely to happen on VMs with low change rates.
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus levels  10.1.0, 10.1.1 and 10.1.2  *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * This issue  is currently projected to be fixed in IBM        *
  * Spectrum Protect Plus 10.1.2.209 and 10.1.3. Note that this  *
  * is subject to change at the discretion of IBM.               *
  ****************************************************************
  

Problem conclusion

• The VADP proxy has been refactored to ensure that all disks
  correctly report the completed status. In addition, IBM Spectrum
  Protect Plus has been updated to verify the individual status of
  each disk. This is a root cause fix and does not correct any
  existing data integrity issues.
  
  IBM Spectrum Protect Plus can identify possible impacted servers
  by analyzing the proxy logs for the past x days where x is
  determined by the version of IBM Spectrum Protect Plus.
  For 10.1.1, log retention is 15 days, x is 15
  For 10.1.2, log retention is 30 days, x is 30
  
  To identify impacted virtual machines from the proxy logs, you
  should perform a log collect, upload the logs to a temporary
  location on the IBM Spectrum Protect Plus server and run the
  following script:  /opt/ECX/tools/scripts/proxyBackupAnalyzer.py
  
  To run enter commands:
  
  cd /opt/ECX/tools/scripts/
  python proxyBackupAnalyzer.py <path>
  
  e.g.,
  
  python proxyBackupAnalyzer.py log_<specific value representing
  log collect timestamp>.zip
  
  Any impacted virtual machines will be reported as:
  
  ('Incorrect status detected for VM', 'VMabc-vm-xyz')
  
  Any VM that is reported will need to be rebased via the User
  Interface
  
  If the issue occurred before the last x days, the script will
  not be able to determine if the problem occurred using the logs.
  It is recommended to perform either an instant virtualize
  operation of the last recovery point or an instant disk mapping
  of critical servers and verify that the data is accessible.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels