LO71832: DOMINO SERVER CRASHED ONE TIME ON NCHRONOS

APAR status

• Closed as returned (APAR cannot be resolved without additional info from IBM or customer.)

Error description

• Domino Server crashed one time on nChronos. A review of the
  chronos task does not indicate a specific database.
  
  Chronos is required to queue the update requests to the update
  task. Although the update task initiates the hourly updates,
  nChronos is required to tell it to do so.
  
  We can see the server crashed when the Chronos task ran. Since
  the Chronos task tells the server a database has to be updated
  we can see one database was updating the index of the following
  database:
  
  Server.Task = Indexer: Full text indexing documents in
  mail\testuser.nsf: [09/25/2012 09:12:04 EDT]
  
  whose full path is:
  
  Database: D:\Domino\Data01\mail\testuser.nsf
  
  After further investigation I found the following database was
  may be related to the server crash and I would suggest,
  completing the following against this database. Open
  mail\testuser.nsf from your Lotus Notes Client, Go to File --
  Application --> Properties -->  2nd to last tab and delete index
  
  Once the Index is deleted, I would then complete the same steps
  but Create Index and let a new index be created.
  
  Crash showed:
  
  ############################################################
  ### thread 1/3: [nChronos:  3460:  40e0] FATAL THREAD
  ### FP=0x00125298, PC=0x7782f6fa, SP=0x00125298
  ### stkbase=0x00130000, total stksize=53248, used stksize=44392
  ### EAX=0x00000041, EBX=0x00000000, ECX=0x00124282,
  EDX=0x7FEFD4DF3E7
  ### ESI=0x000927c0, EDI=0x000003c0, CS=0x00000033,
  SS=0x7FE0000002B
  ### DS=0x00000000, ES=0x00000000, FS=0x00000000,
  GS=0x7FE00000000 Flags=0x00000287
  ############################################################
   [ 1] 0x7782f6fa ntdll.ZwWaitForSingleObject+10
  (9CE3507FE6C5,122e59b9,3,13479d80)
   [ 2] 0x7FEFDAA10AC KERNELBASE.WaitForSingleObjectEx+156
  (12a652,12a652,0,3c0)
  @[ 3] 0x1009535f nnotes.FRSendCommandToService+1931
  (12a900,12ac18,0,100916a8)
  @[ 4] 0x10098b73 nnotes.OSRunExternalScript+5835
  (3,0,1235c9c4,3)
  @[ 5] 0x100924d5 nnotes.FRTerminateWindowsResources+2277
  (80,80,a9,220230)
  @[ 6] 0x1009bbe2 nnotes.OSFaultCleanupExt+622
  (0,1009186f,150014,12360830)
  @[ 7] 0x1009c609 nnotes.OSFaultCleanup+29
  (276630,777f45d0,0,2300000003)
  @[ 8] 0x100b665d nnotes.OSNTUnhandledExceptionFilter+677
  (12d470,0,1,1b00001b)
   [ 9] 0x776593e0 kernel32.UnhandledExceptionFilter+352
  (12d470,6,0,1)
   [10] 0x77876228 ntdll.RtlWow64EnableFsRedirectionEx+10232
  (2202d8,778321bc,220000,2)
   [11] 0x777f4f48 ntdll.__C_specific_handler+156
  (130000,12ff90,12ff90,77925940)
   [12] 0x77814f6d ntdll.RtlCompareUnicodeString+141
  (130000,776ce940,13128,100610000)
   [13] 0x777f5b2c ntdll.RtlTimeToSecondsSince1970+1596
  (12e0b0,12dbc0,0,0)
   [14] 0x7782f638 ntdll.KiUserExceptionDispatcher+46
  (100,12faec,0,1)
  @[15] 0x10037a38 nnotes.AllocOS+56
  (107db6c,107db98,107dbd4,107dc04)
  @[16] 0x1002674f nnotes.OSInitExt+547 (27f,0,0,FFFF00001F80)
  @[17] 0x10132b41 nnotes.NotesInitIni+21 (0,0,0,0)
  @[18] 0x10132c4d nnotes.NotesInitExtended+65 (0,0,276630,220000)
  @[19] 0x00401a82 nChronos.main+106 (0,0,0,0)
  @[20] 0x004020f1 nChronos.mainCRTStartup+369 (0,0,0,0)
   [21] 0x775df33d kernel32.BaseThreadInitThunk+13 (0,0,0,0)
   [22] 0x77812cc1 ntdll.RtlUserThreadStart+33 (0,0,0,0)
  

Local fix

• A Review of the Update task indicates GTR_MergePatt which we
  have seen crashes in the past so I had the customer use this to
  prevent that known crash from occurring.
  
  I would also add the following INI parameter via the server
  console:
  
  1. SET CONFIG FTG_ENABLE_GTR_KEY_CHECK=1
  
  
  **The above parameter is used to stop a particular tup of panic
  due to corrupt indexes. This parameter identifies corrupted
  data contained in any of the GTR KEY files. Without identifying
  the bad data, the updating process continues processing until
  the crash occurs. When bad data is identified, then processing
  stops and the index is automatically rebuilt. By default, this
  feature is off. Requires a Domino Server restart
  
  ############################################################
  ### thread 1/5: [ nUpdate:  31b0:  3188]
  ### FP=0x00122e98, PC=0x7782f9fa, SP=0x00122e98
  ### stkbase=0x00130000, total stksize=106496, used stksize=53608
  ############################################################
   [ 1] 0x7782f9fa ntdll.ZwDelayExecution+10
  (122f58,3df5a20,2ace40,1004b009)
   [ 2] 0x7FEFDAA1203 KERNELBASE.SleepEx+179 (3188,0,0,0)
  @[ 3] 0x1003911e nnotes.OSDelayThread+50 (e,122fd8,0,0)
  @[ 4] 0x10060b61 nnotes.StaticHangEnable+93 (3bb11000,2000,0,1)
  @[ 5] 0x1005466e nnotes.OSFDFileRead+86 (f01028f8,0,31b0,0)
  @[ 6] 0x10151fb5 nnotes.FTEXTENTReadPage+657
  (400011EAA,2f0000c4,54C5C6C69616D5C,56F52632F0000C4)
  @[ 7] 0x11a457e5 nnotes.UBMSetupPage+2405
  (1234d0,2f0000c4,fdf8360,2f0000c4)
  @[ 8] 0x11a45e01 nnotes.UBMReplaceBuffer+1265
  (400011EAA,1001234A0,1,0)
  @[ 9] 0x10002450 nnotes.UBMAutoIngestGroupPage+1684
  (5103e0c8,508d8546,0,0)
  @[10] 0x11a43278 nnotes.UBMClock+64 (0,0,0,100397dd)
  @[11] 0x11a42ea7 nnotes.UBMPinExtended+6083
  (FFFFFFFFFFFFFFE,0,0,0)
  @[12] 0x11a41681 nnotes.UBMPin+25 (0,0,0,1005140a)
  @[13] 0x10153f56 nnotes.CExtentMgrInternal::Pin+314
  (0,508c992e,FFFFFFFFFFFFFFE,afb39)
  @[14] 0x508d8487 gtr40nts.CGtrExtentMgr::Pin+183
  (50f8e790,50f8dea0,1,508bd88e)
  @[15] 0x508bd38d gtr40nts.CGtrBlockMgr::Pin+61
  (5104fbd0,50f8dea0,0,0)
  @[16] 0x508c91e0 gtr40nts.CGtrNode::LeafPin+20
  (2288,1263c8,1263c8,125db0)
  @[17] 0x508c8ec3 gtr40nts.CGtrLeaf::Unpin2+519
  (0,125930,FFFFFFFFFFFFFFE,508348a5)
  @[18] 0x508c8903 gtr40nts.CGtrLeaf::ExternalizeNode+39
  (0,508c3bf2,1263c8,12add8)
  @[19] 0x508c3088 gtr40nts.CGtrBranch::ExternalizeNode+156
  (1262d4,1262d0,0,0)
  @[20] 0x508ccb25 gtr40nts.CGtrPosHigh::Externalize+149
  (328ab,125e90,125e90,508ce928)
  @[21] 0x508d08e0 gtr40nts.CGtrPosBrokerMulti::Externalize+320
  (51046140,2,1,510424a0)
  @[22] 0x508cb43f gtr40nts.gtrMergeMerge+387 (0,508f7c48,1,2)
  @[23] 0x5086964d gtr40nts.gtr_MergePatt+4629 (0,12b9d0,12b9d0,0)
  @[24] 0x5086dff7 gtr40nts.GTR__mergeIndex+3523 (51040048,0,0,0)
  @[25] 0x5089a4ad gtr40nts.GTR_mergeIndex+405
  (50a50c88,503e5fb4,0,0)
  @[26] 0x503c6b5c nftgtr40.cGTRio::Merge+316
  (C74662E6C657373,E6F666E69677466,746164,12c3a0)
  @[27] 0x503c272a nftgtr40.FTGMergeGTRIndexes+114
  (20,113209,257A8400120D9C,4968)
  @[28] 0x503b3f3f nftgtr40.FTGIndex+8303 (0,0,1dc,0)
  @[29] 0x1013f4d9 nnotes.FTCallIndex+53 (c,334,100,0)
  @[30] 0x1013dbfd nnotes.FTIndexExt2+1653
  (334,12db50,257A8400120D9C,5257A840024C732)
  @[31] 0x1013fdbc nnotes.FTIndex+32 (3309,12f604,120d9c,257a84)
  @[32] 0x0040d2b6 nUpdate.UpdateFullTextIndex+458
  (3306,12f604,2,2)
  @[33] 0x0040c519 nUpdate.UpdateCollections+3313 (343305,a09,0,0)
  @[34] 0x004023fb nUpdate.Update+5115
  (72657865643304,3440a0,0,348240)
  @[35] 0x00404073 nUpdate.AddInMain+375 (1,10132b05,0,0)
  @[36] 0x00416716 nUpdate.NotesMain+74 (0,0,297450,240000)
  @[37] 0x0041681d nUpdate.main+245 (0,0,0,0)
  @[38] 0x00417331 nUpdate.mainCRTStartup+369 (0,0,0,0)
   [39] 0x775df33d kernel32.BaseThreadInitThunk+13 (0,0,0,0)
   [40] 0x77812cc1 ntdll.RtlUserThreadStart+33 (0,0,0,0)
  

Problem summary

Problem conclusion

Temporary fix

Comments

• This APAR is associated with SPR# DADS8YHRA8.
  The record was returned to the person who created it for more
   information.
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels