Watchdog stopping the Windows OS agent unexpectedly

Body

Most of the defects that caused unexpected behaviors of the Watchdog component have been already addressed many years ago.

There are anyway some scenarios where the Watchdog code is not directly responsible of problems with service management.

In this blog article, I will describe a scenario where a Windows OS Agent on a server was stopped by the Watchdog and not restarted, thus causing an unplanned outage of the monitoring service.

ITM watchdog feature uses two scripts, getstat.vbs and getinstances.vbs, to retrieve the status of the monitored processes.

In this scenario, we observed that from time to time, getstat.vbs failed and exited with < -1073741819 >

The logs showed a message sequence like:

---

(2011/07/29,15:45:48.007B-3EBC:kcacmd.cpp,178,"executeCmd") Executing command <cscript C:\IBM\ITM\TMAITM6\getstat.vbs 6g >.
(2017/09/19,15:45:48.007D-3EBC:kcaproclist.cpp,269,"unlockList") Exit: 0x1
(2017/09/19,15:45:48.007F-3EBC:kcacmdw.cpp,123,"serializeCommand") Exit:0x0
(2017/09/19,15:45:48.0081-3EBC:kcacmdw.cpp,194,"executeCmd") Waiting for command to complete for 30 seconds.
(2017/09/19,15:45:49.0000-3EBC:kcacmdw.cpp,267,"executeCmd") data available - 111
(2017/09/19,15:45:49.0002-3EBC:kcacmdw.cpp,305,"executeCmd") Exit: 0xC0000005
(2017/09/19,15:45:49.0003-3EBC:kcaproclist.cpp,228,"lockList") Entry
(2017/09/19,15:45:49.0005-3EBC:kcacmd.cpp,218,"executeCmd") Command rc <-1073741819 >.

---

In this case, the watchdog stops the monitored process and does not restart it, causing an unexpected outage.

The negative return code -1073741819 is actually the translation of 0xC0000005.
This code mean ACCESS_VIOLATION and it is returned directly from the cscript manager.

From previous experiences, we know that this kind of of problem can be  caused by an AntiVirus software running on the machine, it occurred often with McAfee antivirus.

If McAfee antivirus is installed, the problem is most likely caused by the SCRIPTSN.dll, which is delivered with McAfee Version 8.8 (this dll is responsible for the script virus scan) even if the ScriptScan component is disabled.

This McAfee technote explains the cause of the problem:

https://kc.mcafee.com/corporate/index?page=content&id=KB71660&actp=search&viewlocale=en_US&searchid=1313422826655

The problem occurs on Windows 2008 R2 and McAfee 8.8.
Basically, the SCRIPTSN.dll is hooked into the scripting engine (cscript) even if ScriptScan is disabled.
Watchdog on Windows platform strongly leverages on cscript engine, so it may be impacted by this configuration.

The McAfee technote above describes two possible workarounds:

1) Enable the ScriptScan
2) Unregister the SCRIPTSN.dll library as follow:

- Open a command prompt windows

- Change to the right directory:
32 bit computers:
cd c:\Program Files\Common Files\McAfee\SystemCore

64 bit computers:
cd c:\Program Files (x86)\Common Files\McAfee\SystemCore
cd c:\Program Files\Common Files\McAfee\SystemCore

- To completely disable the scriptscan dll, type the command below and press
ENTER:
regsvr32.exe /u SCRIPTSN.dll

In our environment we unregistered SCRIPTSN.dll and problem was solved.

Hope it helps

Subscribe and follow us for all the latest information directly on your social feeds: