By checking, I found two servers LT1 and LT2 that encountered same issue. Lots of database semaphore time outs on log.nsf (18251 times for LT1 and 13010 times for LT2). Both of them were compacting and never be completed.
2017/06/03 上午 09:31:29 ZE8 sq="00500F91" THREAD [0F88:0DDB-13E4] WAITING FOR READ LOCK ON FRWSEM 0x0244 database semaphore (@0C1B522C) (d:\Lotus\Domino\data\log.nsf) (R=0,W=2,WRITER=1328:18B4,1STREADER=0000:0000) FOR 30000 ms
Server.Task = Database Compactor: Compacting log.nsf (LT1's Log), -B: [06/03/2017 08:55:30 ZE8]
Server.Task = Database Compactor: Compacting log.nsf (LT2's Log), -B: [06/03/2017 09:37:57 ZE8]
My recommendations are as follows.
*Recreate log.nsf for both servers because they are too large file size (Around 4 GB).
*Change log= setting in notes.ini for both servers.
By default, the log file (LOG.NSF) records information about the IBM Domino system. Because the log file can become quite large, it is important to manage its size. You can control the size of the log file automatically, using NOTES.INI settings, user preferences, and other settings. For example, the Log setting in the NOTES.INI file determines how long documents are maintained before being deleted from the log file. By default, documents are deleted after 7 days. Please see Technote 1093081 for the detail.
If you are troubleshooting a system problem, you may want to record additional information in the log file. The log file becomes large quickly when you set a higher logging level for purposes of analyzing a system problem. For example, if you are troubleshooting a mail routing problem, you can set the logging level to verbose. When you do, the log file will contain a large amount of information regarding that activity. If you set a high logging level during troubleshooting, remember to reset the logging level after you solve the problem.
Controlling the size of the log file (LOG.NSF) at https://www.ibm.com/support/knowledgecenter/en/SSKTMJ_8.5.3/com.ibm.help.domino.admin85.doc/H_CONTROLLING_THE_SIZE_OF_THE_LOG_FILE_LOGNSF_OVERVIEW.html
Lotus Software Knowledge Base Document
Title: Limiting the size of log entries by using the Size parameter of the Log= notes.ini
Doc #: 1093081
Also suggest to change from
Log=log.nsf, 1, 0, 7, 40000
Log=log.nsf, 1, 0, 7, 20000
1. Shutdown server.
2. Delete log.nsf.
3. change notes.ini.
4. Start server.