APAR status
Closed as program error.
Error description
ERROR DESCRIPTION: Domino 9 server crashes on Error Message = PANIC: Error writing Checkpoint to System Log File with callstack similarities to SPR WWHN972KZR and VPRS943PMK. Customer has 9.0HF834, thus these SPRs have been addressed in this release. Error Message = PANIC: Error writing Checkpoint to System Log File ################################### ## thread 247/251 :: server pid=10616944, k-id=93978673 , pthr-id=63223 ## stack :: k-state=wait, stk max-size=1054856, cur-size=21576 ################################### raise.nsleep(??, ??) at 0x9000000000383e4 sleep(??) at 0x90000000014d208 OSRunExternalScript(0x121d66170, 0x100000001) at 0x900000003ae3010 OSFaultCleanupExt(0x0, 0x100000001000, 0x0, 0x0, 0x0, 0x0) at 0x900000003ae26a8 OSFaultCleanup(0x0, 0x100000001000, 0x0) at 0x900000003ae48d4 fatal_error(0xb0000000b, 0x121d66860, 0x121d665b0) at 0x900000004fa4898 pthread_kill(??, ??) at 0x900000000815c30 _p_raise(??) at 0x9000000008154e4 raise.raise(??) at 0x900000000038968 Panic(0x121d69d10) at 0x9000000039144bc NSFPanic(0x0, 0x900000006090174) at 0x900000003aff148 _RmCheckpoint__FUi(0x0) at 0x900000003d5d5c8 RmChkptThreadWrapper() at 0x900000003d5df28 RmCheckpointTask(0x17e48fb00000000, 0x5129800000000) at 0x1000cfd90 Scheduler(0x0) at 0x1000e5e54 ThreadWrapper(0x0) at 0x9000000038e26ec NOTES.INI PARAMETERS RELATED TO TRANSACTION LOG TRANSLOG_AutoFixup=1 TRANSLOG_UseAll=0 TRANSLOG_Style=0 TRANSLOG_Performance=2 TRANSLOG_Status=1 TRANSLOG_Path=/tlogm01 Previous_TRANSLOG_Path=/tlogm01/ Previous_TRANSLOG_Style=0 TRANSLOG_MaxSize=8000 Previous_TRANSLOG_Status=1 TXNS accessed by the time server crashed Crash Time = 04/02/2014 13:54:02 <@@ Directory Listings -> Transaction Log Full Listing By Time @@> /tlogm01: total 8258576 44 -rw-r--r-- 1 lnmail01 notes 67117056 Apr 2 13:51 S0000038.TXN 45 -rw-r--r-- 1 lnmail01 notes 67117056 Apr 2 13:51 S0000039.TXN 46 -rw-r--r-- 1 lnmail01 notes 67117056 Apr 2 13:51 S0000040.TXN 47 -rw-r--r-- 1 lnmail01 notes 67117056 Apr 2 13:51 S0000041.TXN 48 -rw-r--r-- 1 lnmail01 notes 67117056 Apr 2 13:51 S0000042.TXN 49 -rw-r--r-- 1 lnmail01 notes 67117056 Apr 2 13:51 S0000043.TXN 5 -rw-r--r-- 1 lnmail01 notes 12288 Apr 2 13:52 nlogctrl.lfh 50 -rw-r--r-- 1 lnmail01 notes 67117056 Apr 2 13:52 S0000044.TXN 51 -rw-r--r-- 1 lnmail01 notes 67117056 Apr 2 13:54 S0000045.TXN CONSOLE LOG MESSAGES [10616944:00538-63223] Warning: Recovery Manager - Error taking checkpoint. Status: Recovery Manager: Log scan in progress. [10616944:00538-63223] Warning: Recovery Manager - Error taking checkpoint. Status: Recovery Manager: Log scan in progress. [10616944:00538-63223] Warning: Recovery Manager - Error taking checkpoint. Status: Recovery Manager: Log scan in progress. [10616944:00538-63223] Warning: Recovery Manager - Error taking checkpoint. Status: Recovery Manager: Log scan in progress. [10616944:00538-63223] Warning: Recovery Manager - Error taking checkpoint. Status: Recovery Manager: Log scan in progress. [10616944:00538-63223] Warning: Recovery Manager - Error taking checkpoint. Status: Recovery Manager: Log scan in progress. [10616944:00538-63223] Warning: Recovery Manager - Error taking checkpoint. Status: Recovery Manager: Log scan in progress. [10616944:00538-63223] Warning: Recovery Manager - Error taking checkpoint. Status: Recovery Manager: Log scan in progress. [10616944:00538-63223] Warning: Recovery Manager - Error taking checkpoint. Status: Recovery Manager: Log scan in progress. [10616944:00526-60139] 04/02/2014 12:12:35 22305 Transactions/Minute, 3400 Notes UsersWarning: Recovery Manager - Error taking checkpoint. Status: Recovery Manager: Log scan in progress. OTHER RELATED CALLS TACKS ################################### ## thread 106/251 :: server pid=10616944, k-id=79167515 , pthr-id=26986 ## stack :: k-state=wait, stk max-size=1058952, cur-size=24952 ################################### .() at 0x90000004c81e580 _cond_wait(??, ??, ??) at 0x90000000081f5f4 pthread_cond_wait(??, ??) at 0x9000000008202dc WaitForThreadSem(0x11700000117, 0xffffffffffffffff) at 0x9000000038c199c WaitOnNativeSemaphore(0x117000000000117, 0x0, 0x0, 0x0) at 0x9000000038ca024 OSWaitEvent(0xa000000024a3ad4, 0x0) at 0x9000000038c8f7c hosWaitWaitPostArea(0xa000000024a3ad4, 0xffffffffffffffff, 0x117d07b90) at 0x900000003f2c198 sqlpgpst(0xa00000002492a00, 0x300000003) at 0x900000003f37184 StartRead(0xa00000002492a00) at 0x900000005cc24c4 RandomRead(0xa00000002492a00, 0x117d07dc8, 0x117d07dc0) at 0x900000005cc144c sqlpgrlg(0xa00000002492a00, 0x11bbd6820, 0x8000000080000, 0xa00000094b12a00, 0x117d07ee0, 0x117d07ed8, 0x117d086c0, 0x117d07ed0) at 0x900000005cc2d50 hlgScanLogRecord(0x11bbd6818, 0xa00000094b12a00, 0x8000000080000, 0x0, 0x117d086c0) at 0x900000005d58630 _RmDispatch__FR11MemGrowableiR10LSN_STRUCTRC10LSN_STRUCT10_RM_PH ASESP10LSN_STRUCTP16DBCONTEXT_STRUCTRUs(0x117d086a0, 0x0, 0x117d086c0, 0x117d086d0, 0x800000008, 0x0, 0x117d091e8, 0x117d086b0) at 0x900000005d56b1c RmRollforwardDbLoad(0x117d091e8, 0x117d08820) at 0x900000003d58eec DbLoad(0x117d091e8) at 0x900000003d10a30 NSFDbOpenExtended6(0xa0000002c8daaa8, 0x1002000010020000, 0x0, 0x0, 0x1f1000001f1, 0x0, 0x0, 0x117d0c414) at 0x900000003c7c878 ServerDbOpenExtended5(0xa0000002c8da930, 0xa0000002c8daaa8, 0x1000000010000000, 0x0, 0x1f1000001f1, 0x0, 0x0, 0x117d0c82c) at 0x10004240c ServerDbOpenExtended3(0xa0000002c8da930, 0xa0000002c8daaa8, 0x1000000010000000, 0x0, 0x1f1000001f1, 0x0, 0x0, 0x117d0c82c) at 0x1000425cc OpenDB(0x15ee54800000000, 0x43364000000000, 0xa0000002c8daaa0, 0x0) at 0x10004174c ServerOpenDb(0x15ee54800000000, 0x43364000000000) at 0x100041c9c DbServer(0xdc653d800000000, 0x15ee54800000000) at 0x10001cc68 WorkThreadTask(0x5248fa00000000, 0x0) at 0x1000cf5d4 Scheduler(0x0) at 0x1000e5e54 ThreadWrapper(0x0) at 0x9000000038e26ec OTHER RELATED CALLS TACKS ################################### ## thread 34/83 :: router pid=12845170, k-id=47776003 , pthr-id=8482 ## stack :: k-state=wait, stk max-size=1058952, cur-size=34712 ################################### .() at 0x90000004c81e580 _cond_wait(??, ??, ??) at 0x90000000081f5f4 pthread_cond_timedwait(??, ??, ??) at 0x90000000081ff60 WaitForThreadSem(0x32800000328, 0x753000007530) at 0x9000000038c186c WaitOnNativeSemaphore(0x328000000000328, 0xa000100024a3562, 0x0, 0x0) at 0x9000000038ca0a8 OSLockWriteFRWSemInt(0xa000100024a3560, 0x100000001, 0x0) at 0x9000000038c2828 hosRequestMutexSem(0xa000100024a3560) at 0x900000003f2c3e0 RandomRead(0xa00010002492a00, 0x1128306d8, 0x1128306d0) at 0x900000005cc1340 sqlpgrlg(0xa00010002492a00, 0x1126b5be0, 0x8000000080000, 0xa00010094c077a0, 0x1128307f0, 0x1128307e8, 0x112830fdc, 0x1128307e0) at 0x900000005cc2d50 hlgScanLogRecord(0x1126b5bd8, 0xa00010094c077a0, 0x8000000080000, 0x0, 0x112830fdc) at 0x900000005d58630 _RmDispatch__FR11MemGrowableiR10LSN_STRUCTRC10LSN_STRUCT10_RM_PH ASESP10LSN_STRUCTP16DBCONTEXT_STRUCTRUs(0x112830fc0, 0x0, 0x112830fdc, 0x112830ff0, 0x400000004, 0x0, 0x112835cc0, 0x112830fd0) at 0x900000005d56b1c RmRollforwardGranules(0x112835cc0, 0x112831120, 0x8875000088750, 0x8884700088847) at 0x900000003d581e0 DbBucketUnpinBuffer(0x112835cc0, 0x1128313a8, 0x100000001, 0x100000001) at 0x900000003d90034 AllocFromBucket(0x112835cc0, 0xeac00000eac, 0x1000000000001, 0xc3b00000c3b, 0x0, 0x1128316d0, 0x1128329f0, 0x1128326d8) at 0x900000003d82d10 AllocSpace(0x112835cc0, 0x1000000000001, 0xeac00000eac, 0xc3c00000000, 0xeb000000eb0, 0x0, 0x1128329f0, 0x0) at 0x900000003d7e88c DbBktAlloc(0x112835cc0, 0x1000000000001, 0xeac00000ea1, 0x0, 0x8000000000008, 0x0, 0x1128329f0, 0x0) at 0x900000003d898f0 ReallocNote(0x112835cc0, 0x0, 0x0, 0x0, 0x0, 0x100c0000100c, 0xea100000ea1, 0x112831ff8) at 0x900000003d6fd78 NoteUpdateImpl(0x112835cc0, 0x70700000707, 0x100c0000100c, 0x10400000104, 0x112832730, 0x0, 0x0, 0x112835700) at 0x9000000044f479c iNoteUpdate2(0x112835cc0, 0x70700000707, 0x100c0000100c, 0x10400000104, 0x0, 0x0) at 0x90000000450ba44 DispatchNoteUpdate(0x112835cc0, 0x70700000707, 0x100c0000100c, 0x10400000104, 0x0, 0x11ca000011ca, 0x0, 0x100000001) at 0x9000000040d0104 NSFNoteUpdateExtended3(0x70700000707, 0x100c0000100c, 0x10400000104, 0x11ca000011ca, 0x0, 0x100000001, 0x0) at 0x9000000040d23ac xNSFNoteUpdateExtended2(0x70700000707, 0x100c0000100c, 0x10400000104, 0x11ca000011ca, 0x0, 0x100000001) at 0x900000004cb2fd8 DbNoteUpdateAndAddToFolders(0x70700000707, 0x112836358, 0x1128363a8, 0x11ca000011ca, 0x0, 0x100c0000100c, 0x10400000104) at 0x90000000454e884 FoldWrapNoteUpdateAndAddToFolders(0x70700000707, 0x112836358, 0x1128363a8, 0x11ca000011ca, 0x0, 0x100c0000100c, 0x10400000104) at 0x90000000455d8f4 iNSFNoteUpdateAndAddToFolders(0x70700000707, 0x112836358, 0x1128363a8, 0x11ca000011ca, 0x0, 0x100c0000100c, 0x10000000100) at 0x900000003f98608 NSFNoteUpdateAndAddToFolders(0x70700000707, 0xa7200000a72, 0xa7200000a72, 0x11ca000011ca, 0x0, 0x100c0000100c, 0x10000000100) at 0x9000000045cc700 MailDeliverMessage(0xa00010021bbcd60, 0x70700000707, 0xa000100201fcc28, 0xa0001008a77a8fc, 0xa00010090058a90, 0x0, 0xa7200000a72, 0x0) at 0x9000000045a4b1c AttemptMessageDelivery(0xa00010021bbcd60, 0x0, 0xa7200000a72, 0x0, 0xa0001004c1e0060, 0xa000100868adec8, 0xa0001008a77a830, 0xa00010090058a58) at 0x1000482ac DeliverToDestination(0x110005048, 0x0, 0xa000100201fcc28, 0x112838658, 0xa0001004c1e0060, 0xa0001008a77a830, 0x1126b3598) at 0x100049678 Transfer(0x110005048, 0x121e000000000, 0x100000001, 0x0, 0xa000100201fcc28, 0x112838658, 0x1126b3598) at 0x10004d2ac TransferThread(0x1) at 0x100056124 ThreadWrapper(0x0) at 0x9000000038e26ec
Local fix
LOCAL FIX: None
Problem summary
Server crashed due to txn logging performing A oll Forward because we don't set log getting full if RmFlush is active. The fix was marking log critical even if rmflush was in progress so that these rollfowards would abort
Problem conclusion
Server crashed due to txn logging performing A oll Forward because we don't set log getting full if RmFlush is active. The fix was marking log critical even if rmflush was in progress so that these rollfowards would abort
Temporary fix
Comments
This APAR is associated with SPR# MJTM9HUT3C. Server crashed due to txn logging performing A oll Forward because we don't set log getting full if RmFlush is active. The fix was marking log critical even if rmflush was in progress so that these rollfowards would abort
APAR Information
APAR number
LO79864
Reported component name
DOMINO SERVER
Reported component ID
5724E6200
Reported release
900
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2014-04-04
Closed date
2014-10-22
Last modified date
2014-10-22
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
DOMINO SERVER
Fixed component ID
5724E6200
Applicable component levels
R900 PSN
UP
[{"Business Unit":{"code":"BU055","label":"Cognitive Applications"},"Product":{"code":"SSKTMJ","label":"Lotus Domino"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"9.0","Edition":"","Line of Business":{"code":"","label":""}}]
Document Information
Modified date:
22 October 2014