IT15333: ANS2330E AND ANS4174E RC=6511 ON APPLICATION PROTECTION VM BACKUP

Download Information: Version 8.1.0 IBM Spectrum Protectâ„¢ for Virtual Environments
IBM Spectrum Protect client 7.1.6.x interim fix downloads

APAR status

• Closed as program error.

Error description

• VMware guest backups with application protection for Microsoft
  Exchange or Microsoft SQL servers may fail with :
  ANS2330E Failed to unfreeze the VSS writers because the
  snapshot time exceeded the 10 second timeout limitation.
  and
  ANS4174E Full VM backup of VMware Virtual Machine '<vmname>'
  failed with RC=6511 mode=Incremental Forever - Incremental,
  target node name='<target node>', data mover node name='<agent
  node>'
  
  This will be seen when
  1. VMware Tools 9.10.0 or higher are used in the guest
  AND
  2. The following guest user is not logged in the guest
     User name found in the data mover host registry key :
  [SOFTWARE\IBM\ADSM\CurrentVersion\Nodes\(VMGUEST)\<vmname>]UserI
  D
  
  Tivoli Storage Manager Versions Affected:
  Tivoli Storage Manager Client 6.4.x and 7.1.x on Windows
  
  Customer/L2 Diagnostics (If Applicable)
  
  In the SERVICE VM trace of the data mover node we will see :
  
  <timestamp> [PID] [TID] : ..\..\common\vm\wsops.cpp ( 896):
  WsGuestOperations::ReadEnvironmentVariable(): ENTER
     name = TSM_VM_SNAP_STATE
  <timestamp> [PID] [TID] : ..\..\common\vm\vmvisdk.cpp (9124):
  visdkReadEnvironmentVariableInGuest(): ENTER
  <timestamp> [PID] [TID] : ..\..\common\vm\vmvisdk.cpp (9195):
  visdkReadEnvironmentVariableInGuest(): ''
  <timestamp> [PID] [TID] : ..\..\common\vm\vmvisdk.cpp (9197):
  visdkReadEnvironmentVariableInGuest(): EXIT, rc = 0
  <timestamp> [PID] [TID] : ..\..\common\vm\wsops.cpp   ( 903):
  WsGuestOperations::ReadEnvironmentVariable(): Environment
  variable ''
  <timestamp> [PID] [TID] : ..\..\common\vm\wsops.cpp   ( 905):
  WsGuestOperations::ReadEnvironmentVariable(): EXIT, rc = 0
  <timestamp> [PID] [TID] : ..\..\common\vm\acm.cpp     (2761):
  ACM::MonitorVmEnv(): Environment variable ''
  ... ACM::MonitorVmEnv(): Operation timed out, elapsed time =
  0.000000
  ... ACM::MonitorVmEnv(): EXIT, rc = 6502
  ... ACM::Freeze(): EXIT, rc = 6502
  <timestamp> [PID] [TID] : ..\..\common\vm\vmbackvddk.cpp
  (11387): vmVddkFullVMPrePareToOpenVMDKs: Freezing exited, rc =
  6511
  
  before that, the origin of the timeout is found here :
  
  <timestamp> [PID] [TID] : ..\..\common\vm\wsops.cpp( 899):
  WsGuestOperations::ReadEnvironmentVariable(): ENTER
     name = TEMP
  <timestamp> [PID] [TID] : ..\..\common\vm\vmvisdk.cpp(9140):
  visdkReadEnvironmentVariableInGuest(): ENTER
  <timestamp> [PID] [TID] : ..\..\common\vm\vmvisdk.cpp(9211):
  visdkReadEnvironmentVariableInGuest(): 'C:\TEMP'
  <timestamp> [PID] [TID] : ..\..\common\vm\vmvisdk.cpp(9213):
  visdkReadEnvironmentVariableInGuest(): EXIT, rc = 0
  <timestamp> [PID] [TID] : ..\..\common\vm\wsops.cpp( 906):
  WsGuestOperations::ReadEnvironmentVariable(): Environment
  variable 'C:\TEMP'
  
  The problem arises when the %temp% environment variable is not
  the value expected for the guest user.
  When the above conditions are met, the default system user's
  %temp% value 'C:\TEMP' is returned instead of the expected
  'C:\Users\<username>\AppData\Local\Temp'
  This causes the communication between the guest and the data
  mover host to time out.
  
  Initial Impact: High
  
  Additional Keywords: spectrum protect freeze thaw vss timeout
  environment variable %temp%
  

Local fix

• Ensure the user specified with this registry key in the data
  mover host :
  [SOFTWARE\IBM\ADSM\CurrentVersion\Nodes\(VMGUEST)\<vmname>]UserI
  D
  is logged in the guest before and during the VM backup.
  That user is set with the client command : 'dsmc set password
  -type=vmguest'
  
  Another workaround can be found in this VMware KB:
  https://kb.vmware.com/kb/2146470
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * Tivoli Storage Manager for Virtual Environments Data         *
  * Protection for VMware versions 7.1 and 6.4 running on all    *
  * Microsoft Windows x64 and Linux x86_64 platforms             *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  

Problem conclusion

• System environment variables are now used instead of user
  environment variables.
  

Temporary fix

• A fix for this problem is currently targeted for Tivoli Storage
  Manager Client on Linux x86 and Windows x64 for interim fix
  package 7.1.6.3
  
  Note1: Until a Data Protection for VMware packaged fix of a
  fixing Tivoli Storage Manager Client level is available, Data
  Protection for VMware customers can install an use a fixed level
  of the the Tivoli Storage Manager Client in their environment.
  
  Note 2: This is subject to change at the discretion of IBM.
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• dsm
  

Fix information

• Fixed component name

  TSM CLIENT

• Fixed component ID

  5698ISMCL

Applicable component levels

• R71W PSY

     UP

• R71L PSY

     UP

• R64W PSY

     UP

• R64L PSY

     UP