IBM Support

VMware VM backup fails with ANS0361I:Deadlock detected and RC=7305

Troubleshooting


Problem

Virtual machine backup using IBM Storage Protect for Virtual Environment fails with below error message -
ANS0361I DIAG: VmWaitBackupDoneAndStopMovers(): Deadlock detected on process backup VM 'ABC'. No disks finished backing up or data sent to server in last 70 minutes.

Symptom

The dsmerror.log file of the data mover logs the following messages:
ANS0361I DIAG: VmWaitBackupDoneAndStopMovers():
Deadlock detected on process backup VM 'VMname'. No disks finished backing up or data sent to server in last 100 minutes.
ANS4174E Full VM backup of VMware Virtual Machine 'ABC' failed with RC=7305 mode=Incremental Forever - Full, target node name='DCNAME', data mover node name='DMNAME'
Trace log with trace flag "service" shows the following messages:

Thread -1728063744 delaying for 1000 milliseconds.
[004351] [2575296256] : pkthread.cpp  ( 340): pkAcquireMutex(): Thread 2575296256 requesting lock of mutex 0x7f8fcf7e5220, mutex is currently locked by thread 2617243392, waiting for mutex to be released.
[004351] [2583688960] : psthread.cpp    (1313): Thread -1711278336 delaying for 60000 milliseconds.
004351] [2566903552] : vmOverlappedIO.cpp (5137): OverlappedIOMonitor::Monitor(): _kill_vm message from queue entry.
[004351] [2566903552] : vmOverlappedIO.cpp (3259): Entering OverlappedIOMonitor::KillVM():
[004351] [2566903552] : vmOverlappedIO.cpp (3261): OverlappedIOMonitor::KillVM(): Received a queue entry.
[004351] [2566903552] : vmOverlappedIO.cpp (3275): OverlappedIOMonitor::KillVM(): error happened on consumer thread, abandoning backup for vm ‘VM_1’
[004351] [2239756032] : HandlePoolTemplate.h( 173): HandlePool::getHandle(): no handles available, we must wait
[2541725440] : HandlePoolTemplate.h( 173): HandlePool::getHandle(): no handles available, we must wait
[004351] [2583688960] : vmOverlappedIO.cpp  (2644): VmWaitBackupDoneAndStopMovers(): Deadlock detected on process backup VM 'ABC'. No disks finished backing up or data sent to server in last 70 minutes.
[004351] [2583688960] : vmOverlappedIO.cpp  (2665): VmWaitBackupDoneAndStopMovers(): A deadlock has been detected, telling the IO Monitor to stop backing vm=ABC.

Cause

Performance of backup operations for virtual machines was not tuned which was causing deadlock.

In this case, below parameters were set at the maximum values -


VMLIMITPERHost 50
VMLIMITPERDatastore 50
VMMAXBACKUPSession 100

Environment

IBM Storage Protect for Virtual Environments: 8.1.26.0
Data Protection for VMware is installed on Linux or Microsoft Windows operating systems. 

Diagnosing The Problem

If the virtual machines tuning parameters is not optimized it may cause deadlock.

Resolving The Problem

To help optimize parallel virtual machine backups for IBM Storage Protect for Virtual Environments: Data Protection for VMware -
Adjust the settings for the vmlimitperhost, vmbackupsession and vmlimitperdatastore options in client options file.
In this case, the values were initially set to their maximum limits. We reduced these values, which resolved the backup issue.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB69","label":"Storage TPS"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSJ3XJ5","label":"IBM Storage Protect for Virtual Environments"},"ARM Category":[{"code":"a8m3p000000hAaNAAU","label":"Data Protection-\u003EVirtual Environments (VE)"}],"ARM Case Number":"TS019456778","Platform":[{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"8.1.19;8.1.20;8.1.21;8.1.22;8.1.23;8.1.24;8.1.25;8.1.26;8.1.27"}]

Historical Number

TS019456778

Product Synonym

IBM Storage Protect for Virtual Environments, TDPVE

Document Information

Modified date:
09 July 2025

UID

ibm17238820

Page Feedback