A Domino 7.0.x (or higher version) server reports in its console log a message every minute that a specific Domino process has terminated abnormally. This message continues repeatedly.
08/08/2006 10:46:19 AM Process c:\lotus\domino\nleiact.exe (320/0x140) has terminated abnormally
08/08/2006 10:47:19 AM Process c:\lotus\domino\nleiact.exe (320/0x140) has terminated abnormally
08/08/2006 10:48:19 AM Process c:\lotus\domino\nleiact.exe (320/0x140) has terminated abnormally
08/08/2006 10:49:18 AM Process c:\lotus\domino\nleiact.exe (320/0x140) has terminated abnormally
Resolving The Problem
This is a new detection feature for Domino on Windows® 32-bit operating systems (W32) beginning in Domino Notes release 7. This message prints to the console any time Domino detects that one of its child processes has abnormally terminated.
In previous versions, Domino did not perform a check for this condition, and therefore an administrator would be unaware that any such termination occurred unless there was an outward manifestation of this problem (such as user connectivity problems).
Once a process has abnormally terminated, Domino will print this message to the console every 60 seconds until the server is restarted. Under such as case, the server will not shutdown cleanly, since the abnormally terminated task still has certain resources locked.
Some servers may continue to display this message persistently (through multiple server restarts) if shared memory is not properly cleaned up. If this message persists over multiple reboots, take the following action:
1. Start-->Run--> cmd
2. cd <server's data folder>
3. <path to nServer's folder> nsd -kill
What is an abnormal termination?
Strictly speaking, an abnormal termination is a case where a process terminates without properly cleaning up its resources; or, more accurately, it exits without executing its standard termination routines. In the case of Domino, we define this to mean that one child process has terminated without indication of an error, leaving Domino to believe that it is still running. On W32, this could result from several different underlying reasons.
- A third-party add-in task (or stand-alone program that has initialized the Notes API) terminates otherwise normally, but neglects to call the termination routines for the Notes API.
- A Domino add-in task (third party or core Domino task) encounters a severe stack overwrite, resulting in the abnormal (unintended) termination of the process, essentially a crash.
- A Domino add-in task throws a user-defined unhandled exception. This case differs slightly from the other two. When this occurs, a dialog box may appear with the message "process.exe encountered a c++ runtime exception." This dialog box will only contain an "OK" button, such that when you click "OK", the task will terminate. The main difference is that you get the dialog box, but other than that, the task exhibits the same abnormal termination behavior as the other two cases.
On the W32 platform, when a child process abnormally terminates, the parent process is not notified of the termination (as is the case under Unix). The result is that on W32, the process terminates without any outward signs of a termination (i.e. no error message or dialog box). The task simply disappears from the process list. Under prior versions of Domino, nSERVER is unaware of the termination, leading to a condition where the Domino Server indicates that the process is still running, but where the process is no longer listed under the OS process list.
In order to address the potential negative impacts of such an occurrence, the Domino 7.x server implements a check to confirm that each Domino add-in task is still operational; this includes any stand-alone programs that have initialized the Notes API using the server's notes.ini.
Since there are cases where such an abnormal termination may not be detrimental in the short-term, Domino only prints a message to the console that this condition has occurred; the Domino server does not take any action.
In the case where the terminated process is an add-in task provided as part of the installation of the Domino server, it is safe to assume that an abnormal termination should not be occurring. This condition should be investigated immediately with the help of IBM Support, as that task may leave shared memory resources in a compromised state.
Internal Use Only
As mentioned, above, there may be cases where an abnormal termination does not impact the server in the short-term, and the administrator can wait until a normal maintenance window to restart Domino. However, there may be a number of circumstances where the server begins to experience adverse reactions shortly after the abnormal termination (such as semaphore contention, etc).
Beginning in 7.0.2, customers have the option of setting a NOTES.INI parameter that will cause a PANIC to occur whenever an abnormal termination is encountered, with the message "process c:\lotus\domino\xxx.exe has terminated abnormally". It may take up to 60 seconds for the PANIC to occur. While this will not collect any data relevant to the abnormal termination, this will allow the server to be automatically restarted using the Fault Recovery feature. To enable this option, enter the following INI parameter:
Note: This parameter only allows for the automatic restart of the server via fault recovery. In order to troubleshoot the abnormal termination itself, you must employ other means such as using WinDbg or NSD to attach to the process up-front, and wait for the condition to occur in order to collect call stacks. For further assistance, see document #1253056 - How To Troubleshoot Notes/Domino Abnormal Terminations on W32 ().This document discusses the reasons that generate "process has terminated abnormally" errors in the Domino Server console log.
16 June 2018