The 'deadlock' mutex ensures that there is never be more than one session checking for a deadlock situation at any given time. Whenever some lock is requested due to the LOCK MODE set to WAIT [seconds] statement used by a session, the code for deadlock checking get executed.
The deadlock checking code
starts with marking all the sessions in the instance with an internal flag
saying 'not in deadlock'. Then it walks through the list of all the locks held
by the current session (including the newly requested one) and for each lock it
identifies whether there are some sessions waiting for it. If such a session
(called waiter) is found, its internal flag changed to 'deadlock candidate' and
the waiter added into the so called 'deadlock list'. Each of the waiters in
this list is then checked in the same way - the code checks whether any of the
waiters holds any lock the current session is waiting for. If such a lock is
found a deadlock is detected.
Each execution of the deadlock checking code depends on a consistent state of the internal 'deadlock' flag in each session. In other words there can't be more sessions checking for the deadlock at the same time, as they would overwrite that internal flag mutually. To ensure this, each session has to acquire the 'deadlock' mutex before the deadlock checking code can be executed and release it once the deadlock detection is finished.