IT19508: SESSION HANG AND UNDELETED SNAPSHOTS IN PARALLEL 'BACKUP VM'

Download Information: Version 8.1.0.x Data Protection for VMware
Download Information: Version 8.1.2 IBM Spectrum Protectâ„¢ for Virtual Environments

APAR status

• Closed as Vendor Solution.

Error description

• Backup jobs performing VM backups for multiple guests in
  parallel might complete the data transfer to the server and save
  consistent backup images but fail to terminate the session
  gracefully and
  leave VMware snapshots on the datastore.
  
  This defect is different from APAR IT16364.
  In this case here, the defect is in the VMware Virtual Disk
  Development Kit 6.0.1, 6.0.2, 6.5.0. as documented here :
  
  "Backup process fails while restoring virtual machine"
  https://kb.vmware.com/kb/2148333
  
  
  Products affected:
  IBM Spectrum Protect for Virtual Environments: Data Protection
  for VMware version 8.1.0 on Linux and Windows
  
  This problem also affects IBM Spectrum Protect Backup-Archive
  Client 7.1 which is the data mover for Data Protection for
  VMware 7.1. If you are using Data Protection for VMware 7.1,
  refer to APAR IT19189.
  
  Customer/L2 Diagnostics
  A client trace together with a process dump for the client
  while the hang is seen will help identify if a deadlock is
  observed between
  VixDiskLib_Disconnect in one guest backup and the initiation of
  VixDiskLib_ConnectEx/VixDiskLib_Open for another guest backup.
  
  The Data Mover Client dsmerror.log will display entries like :
  
  <timestamp> ANS0361I DIAG: The VDDK Utility Thread may be hung
                       processing another request.
  <timestamp> ANS0361I DIAG: Waiting up to another 3600000
  milliseconds to process a request to VddkFullVMCloseVMDKs
  for vm '<vmname>'.
  
  The client trace will display one of the backup threads
  abruptly ending in vddksdkDisconnect() like this :
  
  <timestamp> [PID] [TID] : ..\..\common\vm\vmvddksdk.cpp(1647):
           =========> Entering vddksdkDisconnect()
  <timestamp> [PID] [TID] : ..\..\common\vm\vmvddksdk.cpp(1216):
           VixDiskLib: VixDiskLib_Disconnect: Disconnect.
  <timestamp> [PID] [TID] : ..\..\common\vm\vmvddksdk.cpp(1216):
           <timestamp> info -[TID] [Originator@6876
  sub=ThreadPool]
           Thread enlisted
  <timestamp> [PID] [TID] : ..\..\common\vm\vmvddksdk.cpp(1216):
           <timestamp> info -[TID] [Originator@6876 sub=transport]
           Performing unmount of on
             C:\Windows\TEMP\vmware-SYSTEM\<uuid-vm-<vm id>\san
  ...      Deleted directory
             C:\Windows\TEMP\vmware-SYSTEM\<uuid-vm-<vm id>\san
  ...      Performing unmount of on
             C:\Windows\TEMP\vmware-SYSTEM\<uuid-vm-<vm id>\nbdssl
  ...      Deleted directory
             C:\Windows\TEMP\vmware-SYSTEM\<uuid-vm-<vm id>\nbdssl
  ...      Successfully released all resources.
  
  The process dump when the hang occurs also will display one of
  the
  threads stuck in vddksdkDisconnect():
  
      ntdll.dll!0000000076d3bb7a()
      ntdll.dll!0000000076d38db8()
      ntdll.dll!0000000076d38cb4()
      vmacore.dll!000007feedfc1c34()
      disklibplugin.dll!000007feee63bc9f()
      disklibplugin.dll!000007feee63bd34()
      vmacore.dll!000007feedf7d721()
      disklibplugin.dll!000007feee52f8af()
      disklibplugin.dll!000007feee530024()
      vmacore.dll!000007feedf7d721()
      disklibplugin.dll!000007feee526c6b()
      disklibplugin.dll!000007feee527004()
      disklibplugin.dll!000007feee523ebd()
      vixDiskLib.dll!000007feef6e5ec4()
  ==> tsmvisdk.dll!vddksdkDisconnect(VixConnection_t *
  vixConnectionP)
                   Line 1650  C++
      dsmcsvc.exe!vmVddkFullVMCloseVMDKs(Sess_o * sessP,
  dsVmEntry_t *
                  vmEntryP) Line 15403    C++
      dsmcsvc.exe!VddkUtilityThread() Line 383    C++
      dsmcsvc.exe!startThread2(Threadstart * startInfoP) Line
  1569    C++
      dsmcsvc.exe!startThread(Threadstart * startInfoP) Line 1530
  C++
      msvcr110.dll!_callthreadstartex() Line 354  C
      msvcr110.dll!_threadstartex(void * ptd) Line 332    C
      kernel32.dll!0000000076be59cd()
      ntdll.dll!0000000076d1a2e1()
  
  Initial Impact: High
  
  Additional Keywords: tsm tsm4ve tdp4ve sp4ve hang snapshot
  backup
  

Local fix

• Avoid parallel VM backups using 'VMMAXPARALLEL 1' in the client
  option file.
  

Problem summary

Problem conclusion

Temporary fix

Comments

• The issue is due to a defect in VMware VDDK:
  https://kb.vmware.com/kb/2148333
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

  IT19189

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels