IT22350: BACKUP VM OPERATIONS MAY HANG AND REPORT ANS0361I WHEN OPENING VIRTUAL DISKS

APAR status

• Closed as Vendor Solution.

Error description

• A backup of a VMware guest may hang when trying to open a
  virtual disk, due to a known VMware defect.
  
  External symptoms include :
  - Schedule processing will show some successful VM processes,
    but ending statistics will be missing and not all VMs are
    processed.
  - Sessions on the IBM Spectrum Protect Server may still be
    active with little to no progress.
  - There may still be running tasks visible in WMware Web
    Interface for IBM Spectrum Protect.
  
  The datamover error log will show the following messages :
  
  ANS0361I DIAG: The VDDK Utility Thread may be hung processing
  another request.
  ANS0361I DIAG: Waiting up to another 3600000 milliseconds to
  process a request to VddkFullVMOpenCloseSingleVMDK for vm 'XXX'.
  
  IBM Spectrum Protect for Virtual Environments Versions Affected:
  Data Protection for VMware Version 7.1, 8.1 on Windows
  and Linux.
  
  Customer/L2 Diagnostics :
  A datamover client trace will show that the call to open the
  disk hangs. For example :
  
  [time] [xxx] [yyy] : ..\..\common\vm\vmbackvddk.cpp(8004):
   VmSendData(): call vddkUtility thread to OpenSingleVMDK
  [time] [xxx] [yyy] : ..\..\common\vm\utilitythread.cpp( 190):
   HandleVddkUtilityCmd(): queuing VddkFullVMOpenSingleVMDK
   command for vm 'XXX'  to the utility thread
  [time] [xxx] [yyy] : ..\..\common\vm\utilitythread.cpp( 222):
  HandleVddkUtilityCmd(): waiting up to 3600000 milliseconds to
  process VddkFullVMOpenSingleVMDK for vm 'XXX'
  ...
  [time] [xxx] [yyy] : ..\..\common\vm\utilitythread.cpp( 241):
   The VDDK Utility Thread may be hung processing another
   request.
  [time] [xxx] [yyy] : ..\..\common\vm\utilitythread.cpp( 248):
   Waiting up to another 3600000 milliseconds to process a
   request to VddkFullVMOpenSingleVMDK for vm 'XXX'.
  
  The hang is caused by a known VMware defect, which exists in
  both VDDK 6.0 and 6.5. The hang is described in the VDDK 6.0
  and 6.5 release notes under section titled
   "Intermittent hang observed when opening virtual disks".
  The VMware defect is described as follows :
  "A customer observed intermittent hanging while opening virtual
  disks. The hang was found to occur inside the VixDiskLibVim
  module, in VixDiskLib_PrepareForAccess, VixDiskLib_EndAccess,
  or GetNfcTicket, and may be caused by network problems.
  Fixes have been identified and will appear in a future release.
  One fix is a timeout for HTTP requests, and another is a
  timeout for the open operation."
  
  Initial Impact: Medium
  Additional Keywords: TSM4VE SP4VE
  

Local fix

• - Restart the running Spectrum Protect client services
  - Restart the datamover system
  

Problem summary

Problem conclusion

Temporary fix

Comments

• The issue occurs due to a bug in Vmware VDDK 6.0 and 6.5:
  Intermittent hang observed when opening virtual disks.
  https://vdc-download.vmware.com/vmwb-repository/dcr-public/ac678
  25e-a82c-4821-9059-b1554d4889ca/8161f71b-1f2e-488a-b4de-872f6145
  066c/vddk-600-releasenotes.html
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels