IT19189: SESSION HANG AND UNDELETED SNAPSHOTS IN PARALLEL 'BACKUP VM'

IBM Spectrum Protect client 7.1.6.x interim fix downloads
IBM Spectrum Protectâ„¢ Client 7.1.8 Downloads and READMEs

APAR status

• Closed as Vendor Solution.

Error description

• Backup jobs performing VM backups for multiple guests in
  parallel might complete the data transfer to the server and save
  consistent backup images but fail to terminate the session
  gracefully and leave VMware snapshots on the datastore.
  This defect is different from APAR IT16364.
  In this case here, the defect is in the VMware Virtual Disk
  Development Kit 6.0.1, 6.0.2, 6.5.0. as documented here :
  
  "Backup process fails while restoring virtual machine"
  https://kb.vmware.com/kb/2148333
  
  IBM Spectrum Protect Data Mover Client Versions Affected:
  IBM Spectrum Protect Data Mover Client 7.1.3.2 and higher on
  Linux and Windows
  
  Products affected:
  IBM Spectrum Protect Backup-Archive Client version
  7.1.3.2 and later 7.1.x.x versions on Linux and Windows
  Note: The Backup-Archive Client is a prerequisite to using the
  Data Protection for VMware. In Data Protection for VMware
  environments, the Backup-Archive Client is also known as the
  data mover.
  
  This problem also affects IBM Spectrum Protect for Virtual
  Environments: Data Protection for VMware 8.1.0. If you are using
  Data Protection for VMware 8.1.0, refer to APAR IT19508.
  
  Customer/L2 Diagnostics
  A client trace together with a process dump for the client
  while the
  hang is seen will help identify if a deadlock is observed
  between
  VixDiskLib_Disconnect in one guest backup and the initiation of
  VixDiskLib_ConnectEx/VixDiskLib_Open for another guest backup.
  
  The Data Mover Client dsmerror.log will display entries like :
  
  <timestamp> ANS0361I DIAG: The VDDK Utility Thread may be hung
                       processing another request.
  <timestamp> ANS0361I DIAG: Waiting up to another 3600000
  milliseconds to
                       process a request to VddkFullVMCloseVMDKs
  for vm
                       '<vmname>'.
  
  The client trace will display one of the backup threads
  abruptly ending
  in vddksdkDisconnect() like this :
  
  <timestamp> [PID] [TID] : ..\..\common\vm\vmvddksdk.cpp(1647):
           =========> Entering vddksdkDisconnect()
  <timestamp> [PID] [TID] : ..\..\common\vm\vmvddksdk.cpp(1216):
           VixDiskLib: VixDiskLib_Disconnect: Disconnect.
  <timestamp> [PID] [TID] : ..\..\common\vm\vmvddksdk.cpp(1216):
           <timestamp> info -[TID] [Originator@6876
  sub=ThreadPool]
           Thread enlisted
  <timestamp> [PID] [TID] : ..\..\common\vm\vmvddksdk.cpp(1216):
           <timestamp> info -[TID] [Originator@6876 sub=transport]
           Performing unmount of on
             C:\Windows\TEMP\vmware-SYSTEM\<uuid-vm-<vm id>\san
  ...      Deleted directory
             C:\Windows\TEMP\vmware-SYSTEM\<uuid-vm-<vm id>\san
  ...      Performing unmount of on
             C:\Windows\TEMP\vmware-SYSTEM\<uuid-vm-<vm id>\nbdssl
  ...      Deleted directory
             C:\Windows\TEMP\vmware-SYSTEM\<uuid-vm-<vm id>\nbdssl
  ...      Successfully released all resources.
  
  The process dump when the hang occurs also will display one of
  the
  threads stuck in vddksdkDisconnect():
  
      ntdll.dll!0000000076d3bb7a()
      ntdll.dll!0000000076d38db8()
      ntdll.dll!0000000076d38cb4()
      vmacore.dll!000007feedfc1c34()
      disklibplugin.dll!000007feee63bc9f()
      disklibplugin.dll!000007feee63bd34()
      vmacore.dll!000007feedf7d721()
      disklibplugin.dll!000007feee52f8af()
      disklibplugin.dll!000007feee530024()
      vmacore.dll!000007feedf7d721()
      disklibplugin.dll!000007feee526c6b()
      disklibplugin.dll!000007feee527004()
      disklibplugin.dll!000007feee523ebd()
      vixDiskLib.dll!000007feef6e5ec4()
  ==> tsmvisdk.dll!vddksdkDisconnect(VixConnection_t *
  vixConnectionP)
                   Line 1650  C++
      dsmcsvc.exe!vmVddkFullVMCloseVMDKs(Sess_o * sessP,
  dsVmEntry_t *
                  vmEntryP) Line 15403    C++
      dsmcsvc.exe!VddkUtilityThread() Line 383    C++
      dsmcsvc.exe!startThread2(Threadstart * startInfoP) Line
  1569    C++
      dsmcsvc.exe!startThread(Threadstart * startInfoP) Line 1530
  C++
      msvcr110.dll!_callthreadstartex() Line 354  C
      msvcr110.dll!_threadstartex(void * ptd) Line 332    C
      kernel32.dll!0000000076be59cd()
      ntdll.dll!0000000076d1a2e1()
  
  Initial Impact: High
  
  Additional Keywords: tsm tsm4ve tdp4ve sp4ve hang snapshot
  backup
  

Local fix

• Avoid parallel VM backups using 'VMMAXPARALLEL 1' in the client
  option file.
  

Problem summary

Problem conclusion

Temporary fix

Comments

• The issue is due to a defect in VMware VDDK:
  https://kb.vmware.com/kb/2148333
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  IT19508

Fix information

Applicable component levels