IT26261: BACKUP VM COMPLETES SUCCESSFULLY, BUT NO DATA SENT TO THE SERVER

APAR status

• Closed as program error.

Error description

• vCenter 6.5 doesn't return an error after Data Protection for
  VMware tries to get CBT data when CBT was enabled with a
  pre-existing snapshot.
  
  In case CBT was enabled with pre-existing snapshots, the CBT
  data returned by VMware must be considered invalid:
   - it may be empty causing no data being sent to the server.
   - it can represent some random blocks. In this case, invalid
  data is sent to the server.
  In both cases, the backup can not be considered as a valid
  backup.
  
  CBT can be enabled manually by Data Protection for VMware, by
  other products or manually by the user. However, it is
  impossible to detect afterwards if a snapshot existed when CBT
  was enabled or re-enabled.
  
  IBM Spectrum Protect Versions Affected:
  IBM Spectrum Protect for Virtual Environments - Data Protection
  for VMware 8.1.x on Windows and Linux
  
  Customer/L2 Diagnostics:
  
  In case empty CBT data is returned, the backup summary will show
  no disks being backed up and only configuration information is
  sent to the server.
  Example:
  
    Protect> backup vm 'testvm'
    Full on-demand BACKUP VM of virtual machines 'testvm'.
  
  
    Backup VM command started.  Total number of virtual machines
  to process: 1
    Accessing as node: DC_node
  
    Starting Full VM backup of VMware Virtual Machine 'testvm'
          mode:                        'Incremental Forever -
  Incremental'
          target node name:            'DC_node'
          data mover node name:        'DM_node'
          application protection type: 'VMware'
          application(s) protected:    'n/a'
    Creating "VMware Tools" snapshot for virtual machine 'testvm'
    Backing up Full VM configuration information for 'testvm'
           31,451 VM Configuration [Sent]
  
         >>>> Note missing disks during the backup of the VM <<<<<
  
    Successful Full VM backup of VMware Virtual Machine 'testvm'
          mode:                 'Incremental Forever -
  Incremental'
          target node name:     'DC_node'
          data mover node name: 'DM_node'
    Statistics for Virtual Machine 'testvm'.
  
    Total number of objects inspected:            1
    Total number of objects backed up:            1
    ...
    Total number of bytes inspected:          40.00 GB
    Total number of bytes processed:          31.25 KB
    ...
    Removing snapshot for virtual machine 'testvm'
  
    Backup VM command complete
    Total number of virtual machines backed up successfully: 1
     virtual machine testvm backed up to nodename DC_node
    Total number of application protection warnings: 0
    Total number of virtual machines failed: 0
    Total number of virtual machines skipped: 0
    Total number of virtual machines processed: 1
  
  When the vCenter returns invalid CBT data, disk data will be
  sent to the server. This will look like a normal, successful
  backup.
  
  Review a trace with service, vm traceflags:
  
  Only in case Data Protection for VMware tries to enable CBT and
  detect an existing snapshot while enabling CBT, the following
  line is added to the trace:
  
  "vmVddkFullVMPrePareToOpenVMDKs: pre-existing snapshot are not
  recommend when enabling CBT".
  
  *NOTE: If CBT was enabled outside Data Protection for VMware, it
  is impossible to detect if a snapshot existed or not. Backup
  data is invalid in this case, but this can not be detected by
  Data Protection for VMware.
  
  Products affected:
  IBM Spectrum Protect for Virtual Environments:
  Data Protection for VMware version 8.1 on Windows x64
  and Linux x86_64 platforms.
  
  This problem also affects IBM Spectrum Protect
  Backup-Archive Client which is the data mover for
  Data Protection for VMware version 7.1.
  If you are using Backup-Archive Client version 7.1
  refer to APAR IT28068
  
  Initial Impact: high
  
  Additional Keywords: TSM, Tivoli, Storage, Manager, TDP, TDP4VE,
  TS000844751
  

Local fix

• Verify if no snapshot exists when enabling CBT for a Virtual
  Machine.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect for Virtual Environments Data           *
  * Protection for VMware versions 7.1 and  8.1 running on all   *
  * Microsoft Windows x64 and Linux x86 platforms.               *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * This issue is projected to be fixed in the Data Protection   *
  * for VMware Data Mover version 8.1.7 on Windows x64 and Linux *
  * x86 platforms.                                               *
  * Note 1: This is subject to change at the discretion of IBM.  *
  ****************************************************************
  

Problem conclusion

• The issue is due to a bug in vSphere 6.5 and higher: when CBT is
  enabled with a pre-existing snpashot the QueryChangedDiskAreas
  might return invalid data. https://kb.vmware.com/kb/1031873.
  The code has been updated to perform extra verification of CBT
  data.
  Also, a new Vmnocbtcontinue option has been added to specify
  whether to back up a virtual machine (VM) without using the
  change block tracking function when one or more snapshots
  already exist on the VM and change block tracking must be
  enabled or reset.
  

Temporary fix

Comments

• The issue is due to a bug in vSphere 6.5 and higher: when CBT is
  enabled with a pre-existing snpashot the QueryChangedDiskAreas
  might return invalid data. https://kb.vmware.com/kb/1031873
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  IT28068

Modules/Macros

• dsmagent
  

Fix information

• Fixed component name

  TSM FOR VE DP V

• Fixed component ID

  5725TVEVM

Applicable component levels