Troubleshooting
Problem
A vCenter Server installed in a VM guest, might cause a BACKUP VM failure when included in a VM backup list with an API message "Operation interrupted or timed out".
Symptom
In the client dsmerror.log the following error is seen :
<timestamp> ANS9365E VMware vStorage API error for virtual machine '<VM guest name>'.
TSM function name : __ns2__RetrievePropertiesEx
TSM file : vmvisdk.cpp (1984)
API return code : -1
API error message : Error -1 fault: SOAP-ENV:Client[no subcode]
"End of file or no input: Operation interrupted or timed out (300 s receive delay) (300 s send delay)"
Detail: [no detail]
Cause
One possible cause for such a time out message beside eventual network problems is that when the vCenter Server is hosted in a VM guest and it is included in a backup job with other VM guests to run in parallel, the vCenter might not be responsive because a snapshot operation triggered by the backup request is freezing it.
Diagnosing The Problem
In addition to the time out message as seen in the Symptom section, the vmware.log for the VM guest will display message like :
<timestamp>| vmx| I120: Vix: [84689 vigorCommands.c:481]: VigorSnapshotManager_Take: takeOptions=8, clientFlags=0, displayName=TSM-VM Snapshot Wed Jun 18 19:03:34 2014
...
<timestamp>| vcpu-1| I120: SnapshotVMXTakeSnapshotComplete : done with snapshot 'TSM-VM Snapshot Wed Jun 18 19:03:34 2014': 80
...
<timestamp>| vmx| I120: OBJLIB-FILEBE :FileBEUnlink : Failed to unlink the file '/vmfs/volumes/bbf067e3-a51ccda4/vmname/vmname_3-000003-delta.vmdk' : 1048579
<timestamp>| vmx| I120: DISKLIB-LIB : Cannot remove extent `/vmfs/volumes/bbf067e3-a51ccda4/vmname/vmname_3-000003-delta.vmdk': Device or resource busy
...
<timestamp>| vcpu-0| I120: OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/bbf067e3-a51ccda4/vmname/vmname_3-flat.vmdk' : Failed to lock the file (262146).
<timestamp>| vcpu-0| I120: DISKLIB-VMFS : "/vmfs/volumes/bbf067e3-a51ccda4/vmname/vmname_3-flat.vmdk" : failed to open (Failed to lock the file): ObjLib_Open failed. Type 3
...
2<timestamp>| vcpu-0| I120: DISKLIB-LIB : Failed to open '/vmfs/volumes/bbf067e3-a51ccda4/vmname/vmname_3.vmdk' with flags 0x20a Failed to lock the file (16392).
...
<timestamp>| vcpu-0| I120: DISK: Failed to open disk for consolidate '/vmfs/volumes/bbf067e3-a51ccda4/vmname/vmname_3-000004.vmdk' : Failed to lock the file (16392) 6878
Resolving The Problem
Plan to backup the vCenter Server when no heavy activity is going on and separately from the other VM guests backups to avoid the vCenter unresponsiveness when the datamover node needs to query it for the backups.
Was this topic helpful?
Document Information
Modified date:
17 June 2018
UID
swg21677476