Fixes are available
Tivoli Log File Agent, Version 6.3.0 Fix Pack 01 (6.3.0-TIV-ITM_LFA-FP0001)
Tivoli Log File Agent, Version 6.3.0 Interim Fix 04 6.3.0-TIV-ITM_LFA-IF0004
Tivoli Log File Agent, Version 6.3.0 Fix Pack 02 (6.3.0-TIV-ITM_LFA-FP0002)
Tivoli Log File Agent, Version 6.3.0 Interim Fix 05 6.3.0-TIV-ITM_LFA-IF0005
APAR status
Closed as program error.
Error description
When the size of an event exceeds the maximum event size (EventMaxSize, default value 16384 bytes) and is truncated, the Log File Agent (LO) process may terminate with an Exception 0xC0000005 (ACCESS_VIOLATION). The size of the event includes the slotnames, equal signs, quotes, etc. The LO agent instance unexpectedly stops and goes offline. The crash occurs when the truncation occurs on a boundary condition, and the event is truncated at a single quote and has no ending semi-colon. It only occurs in the mapping of the attribute slots to CustomSlots when sending events from either a Windows event log or a file to IBM Tivoli Monitoring. The parsing of the log entry / event is controlled by stanza definitions in the format (.fmt) file. Additional symptoms that may be reported that are due to the event being larger than EventMaxSize: - truncated attribute values displayed in the CustomSlots in the TEP portal - blank attribute values displayed in the CustomSlots in the TEP portal - situations not triggering as expected due to unexpected attribute values since the fields can be truncated when they are parsed from the monitored log - off-line managed system for LO agent Affected Versions: The problem exists on Log File Agent v6.3.0 and prior releases. Problem Determination: On the LO agent system, enable: KBB_RAS1= ERROR (UNIT:LogMonitorQuery ALL) (UNIT:WinLogQuery ALL) (UNIT:kum0regx ALL) The agent RAS1 log will have a naming convention: <hostname>_lo_[instance]_kloagent_<timestamp>-<nn>.log The following trace points show the size/data of the incoming event <timestamp>::winlogquerylist.cpp,1134,"writeEventDataToPipe") Writing event of length ###### to pipe: <contents of the log entry / event> <timestamp>:winlogquerylist.cpp,1145,"writeEventDataToPipe") Node <.fmt stanza> wrote event to pipe for event log <logname>, event size = XXXXX , bytes written = YYYYY The following trace shows that the overall event exceeds EventMaxSize and will be truncated <timestamp>:kum0regx.c,1152,"KUM0_PerformStringReplace") Warning: Replacement string size ZZZZZ for attribute EIFEvent larger than available buffer, only copied 16384 bytes to .... . The value copied (in the above 16384) is the EventMaxSize the LO agent is using. The following trace points show the access violation <timestamp>:logmonitorqueryclass.cpp,1812,"LogMonitorQueryClass: :getSlotAttrAndValue") Entry <timestamp>:logmonitorqueryclass.cpp,1823,"LogMonitorQueryClass: :getSlotAttrAndValue") No ; at end, <slotname>=' must be last slot <timestamp>:logmonitorqueryclass.cpp,1847,"LogMonitorQueryClass: :getSlotAttrAndValue") Found slot <slotname> mapped to custom attribute CustomSlot# <timestamp>:kumpxtrt.c,51,"KUMP_PerformDataCallback") *************************************************************** <timestamp>:kumpxtrt.c,52,"KUMP_PerformDataCallback") ***** Data Callback Execution Exception Handler ***** <timestamp>:kumpxtrt.c,56,"KUMP_PerformDataCallback") ***** Data Callback Function - EXCEPTION_ACCESS_VIOLATION ***** <timestamp>:kumpxtrt.c,62,"KUMP_PerformDataCallback") *************************************************************** <timestamp>:kumpxtrt.c,65,"KUMP_PerformDataCallback") Exit: 0x0 Note: These entries might not be at the end of the log. There will not be the corresponding exit trace point: "LogMonitorQueryClass::getSlotAttrAndValue") Exit On Windows, when the agent experiences an access_violation, the details for the crash and call stack should be written to the kloagent_<instance>.ras file. The details of the call stack are variable, as it depends on the data written after the truncation point. Example tracebacks: Origin: ntdll.dll<>RtlInitUnicodeString+0x1F3 Call Stack IP:77D4E29B BP:0809FCF0 ntdll.dll<>RtlInitUnicodeString+0x1F3 IP:77D4DEA3 BP:0809FD08 ntdll.dll<>RtlFreeHeap+0x7E IP:75F414D1 BP:0809FD1C kernel32.dll<>HeapFree+0x14 IP:7C362385 BP:0809FD64 MSVCR71.dll<>free+0x39 IP:07E89061 BP:0809FE38 kloudp.dll<>KUMP_AreAllFileThreadsStartedTable+0x1FDC1 IP:07E80878 BP:0809FF1C kloudp.dll<>KUMP_AreAllFileThreadsStartedTable+0x175D8 IP:07ED31D4 BP:0809FF2C kloudp.dll<>KUMP_FlushCommonProviderTable+0xF34 IP:64545280 BP:0809FF54 KLX.dll<>pthread_yield+0x810 IP:7C3694EF BP:0809FF88 MSVCR71.dll<>endthread+0xAA IP:75F43677 BP:0809FF94 kernel32.dll<>BaseThreadInitThunk+0x12 IP:77D59D72 BP:0809FFD4 ntdll.dll<>RtlInitializeExceptionChain+0x63 IP:77D59D45 BP:0809FFEC ntdll.dll<>RtlInitializeExceptionChain+0x36 ... Origin: ntdll.dll<>RtlImageNtHeader+0x19B Call Stack IP:77D5317F BP:08BEF8F0 ntdll.dll<>RtlImageNtHeader+0x19B IP:77D53407 BP:08BEF9E8 ntdll.dll<>RtlImageNtHeader+0x423 IP:77D532F2 BP:08BEFA08 ntdll.dll<>RtlImageNtHeader+0x30E IP:75F414D1 BP:08BEFA1C kernel32.dll<>HeapFree+0x14 IP:7C362385 BP:08BEFA64 MSVCR71.dll<>free+0x39 IP:004D72EB BP:08BEFB00 kloagent_60120.exe<>KDY::Array<long>::init+0x352BB IP:004D5B2F BP:08BEFC80 kloagent_60120.exe<>KDY::Array<long>::init+0x33AFF IP:004CE3A5 BP:08BEFCC8 kloagent_60120.exe<>KDY::Array<long>::init+0x2C375 IP:7238833C BP:08BEFD0C Wevtapi.dll<>EvtRender+0x29E1 IP:723882C2 BP:08BEFD5C Wevtapi.dll<>EvtRender+0x2967 IP:72388174 BP:08BEFD94 Wevtapi.dll<>EvtRender+0x2819 IP:723880FB BP:08BEFDC0 Wevtapi.dll<>EvtRender+0x27A0 IP:77D982D4 BP:08BEFE00 ntdll.dll<>RtlRegisterWait+0x314 IP:77D8377D BP:08BEFE28 ntdll.dll<>RtlIsCriticalSectionLockedByThread+0x308 IP:77D83235 BP:08BEFF88 ntdll.dll<>TpCallbackIndependent+0x7DC IP:75F43677 BP:08BEFF94 kernel32.dll<>BaseThreadInitThunk+0x12 IP:77D59D72 BP:08BEFFD4 ntdll.dll<>RtlInitializeExceptionChain+0x63 IP:77D59D45 BP:08BEFFEC ntdll.dll<>RtlInitializeExceptionChain+0x36 ... Origin: ntdll.dll<>RtlIntegerToUnicodeString+0x2FC Call Stack IP:77D58C39 BP:0642FE38 ntdll.dll<>RtlIntegerToUnicodeString+0x2FC IP:77D58B48 BP:0642FE60 ntdll.dll<>RtlIntegerToUnicodeString+0x20B IP:619043A7 BP:0642FE6C KBB.dll<>BSS1_GetLock+0x37 IP:64A2D6E3 BP:0642FE7C kra.dll<>CtiraAutomationCapsule::CheckAndResetCommandConversion+ 0x3DD63 IP:64A2B60F BP:0642FE9C kra.dll<>CtiraAutomationCapsule::CheckAndResetCommandConversion+ 0x3BC8F IP:64A2CFC1 BP:0642FF2C kra.dll<>CtiraAutomationCapsule::CheckAndResetCommandConversion+ 0x3D641 IP:64545280 BP:0642FF54 KLX.dll<>pthread_yield+0x810 IP:7C3694EF BP:0642FF88 MSVCR71.dll<>endthread+0xAA IP:75F43677 BP:0642FF94 kernel32.dll<>BaseThreadInitThunk+0x12 IP:77D59D72 BP:0642FFD4 ntdll.dll<>RtlInitializeExceptionChain+0x63 IP:77D59D45 BP:0642FFEC ntdll.dll<>RtlInitializeExceptionChain+0x36 ... Origin: ntdll.dll<>RtlInitUnicodeString+0x1F3 Call Stack IP:77D4E29B BP:081FFCF0 ntdll.dll<>RtlInitUnicodeString+0x1F3 IP:77D4DEA3 BP:081FFD08 ntdll.dll<>RtlFreeHeap+0x7E IP:75F414D1 BP:081FFD1C kernel32.dll<>HeapFree+0x14 IP:7C362385 BP:081FFD64 MSVCR71.dll<>free+0x39 IP:07FE9061 BP:081FFE38 kloudp.dll<>KUMP_AreAllFileThreadsStartedTable+0x1FDC1 IP:07FE0878 BP:081FFF1C kloudp.dll<>KUMP_AreAllFileThreadsStartedTable+0x175D8 IP:080331D4 BP:081FFF2C kloudp.dll<>KUMP_FlushCommonProviderTable+0xF34 IP:64545280 BP:081FFF54 KLX.dll<>pthread_yield+0x810 IP:7C3694EF BP:081FFF88 MSVCR71.dll<>endthread+0xAA IP:75F43677 BP:081FFF94 kernel32.dll<>BaseThreadInitThunk+0x12 IP:77D59D72 BP:081FFFD4 ntdll.dll<>RtlInitializeExceptionChain+0x63 IP:77D59D45 BP:081FFFEC ntdll.dll<>RtlInitializeExceptionChain+0x36 ... Origin: kloagent_60120.exe<>KDY::Array<long>::init+0x2AE16 Call Stack IP:004CCE46 BP:08BFFC78 kloagent_60120.exe<>KDY::Array<long>::init+0x2AE16 IP:004CE1BC BP:08BFFCC8 kloagent_60120.exe<>KDY::Array<long>::init+0x2C18C IP:7254833C BP:08BFFD0C Wevtapi.dll<>EvtRender+0x29E1 IP:725482C2 BP:08BFFD5C Wevtapi.dll<>EvtRender+0x2967 IP:72548174 BP:08BFFD94 Wevtapi.dll<>EvtRender+0x2819 IP:725480FB BP:08BFFDC0 Wevtapi.dll<>EvtRender+0x27A0 IP:77D982D4 BP:08BFFE00 ntdll.dll<>RtlRegisterWait+0x314 IP:77D8377D BP:08BFFE28 ntdll.dll<>RtlIsCriticalSectionLockedByThread+0x308 IP:77D83235 BP:08BFFF88 ntdll.dll<>TpCallbackIndependent+0x7DC IP:75F43677 BP:08BFFF94 kernel32.dll<>BaseThreadInitThunk+0x12 IP:77D59D72 BP:08BFFFD4 ntdll.dll<>RtlInitializeExceptionChain+0x63 IP:77D59D45 BP:08BFFFEC ntdll.dll<>RtlInitializeExceptionChain+0x36 ... Origin: ntdll.dll<>RtlInitUnicodeString+0x196 Call Stack IP:77D4E23E BP:08AAEA90 ntdll.dll<>RtlInitUnicodeString+0x196 IP:77D4DEA3 BP:08AAEAA8 ntdll.dll<>RtlFreeHeap+0x7E IP:75F414D1 BP:08AAEABC kernel32.dll<>HeapFree+0x14 IP:7C362385 BP:08AAEB04 MSVCR71.dll<>free+0x39 IP:004CD36E BP:08AAEB54 kloagent_60120.exe<>KDY::Array<long>::init+0x2B33E IP:004CDC7C BP:08AAFC7C kloagent_60120.exe<>KDY::Array<long>::init+0x2BC4C IP:004CE1CB BP:08AAFCC8 kloagent_60120.exe<>KDY::Array<long>::init+0x2C19B IP:7238833C BP:08AAFD0C Wevtapi.dll<>EvtRender+0x29E1 IP:723882C2 BP:08AAFD5C Wevtapi.dll<>EvtRender+0x2967 IP:72388174 BP:08AAFD94 Wevtapi.dll<>EvtRender+0x2819 IP:723880FB BP:08AAFDC0 Wevtapi.dll<>EvtRender+0x27A0 IP:77D982D4 BP:08AAFE00 ntdll.dll<>RtlRegisterWait+0x314 IP:77D8377D BP:08AAFE28 ntdll.dll<>RtlIsCriticalSectionLockedByThread+0x308 IP:77D83235 BP:08AAFF88 ntdll.dll<>TpCallbackIndependent+0x7DC IP:75F43677 BP:08AAFF94 kernel32.dll<>BaseThreadInitThunk+0x12 IP:77D59D72 BP:08AAFFD4 ntdll.dll<>RtlInitializeExceptionChain+0x63 IP:77D59D45 BP:08AAFFEC ntdll.dll<>RtlInitializeExceptionChain+0x36 ... There may also be indications of "symInitialize() failed" messages: . Exception 0xC0000005 (ACCESS_VIOLATION) Program: C:\IBM\ITM\TMAITM6\kloagent_60120.exe symInitialize() failed RECREATE INSTRUCTIONS: *NOTE* Recreating the access_violation is very difficult, as the truncation point has to align on a boundary condition. In order to reproduce in the lab, a brute force script was written to incrementally test increasing messages 1 character at a time. 1 character more or less than a boundary condition will not result in an access_violation, although the other symptoms of truncated or blank attribute values will still occur. 1. In the .conf file, specify an EventMaxSize smaller than the average size of the events. 2. Start the kloagent. 3. Generate events larger than the EventMaxSize. Additional Keywords: core crash dump abend lfa
Local fix
Edit the .conf file. Specify "EventMaxSize=XXXXX" where "XXXXX" is based on the size of the largest event including slotnames, equal signs, quotes, etc. The maximum value is 100000. *Note* the LO agent uses the amount of memory specified in EventMaxSize for all events, even ones that are smaller. Therefore, do not use the maximum value unless it is required. See above trace points for an indication of the size of the events
Problem summary
The kloagent process may terminate with an Exception 0xC0000005 (ACCESS_VIOLATION), when the size of an event exceeds the maximum event size (EventMaxSize) and is truncated. The agent instance unexpectedly stops and goes offline. The crash occurs when the truncation occurs on a boundary condition, and the event is truncated at a single quote and has no ending semi-colon. It only occurs in the mapping of the attribute slots to CustomSlots when sending events to IBM Tivoli Monitoring from either a Windows event log or a file. The problem exists on the Log File Agent version 6.3.0 Interim Fix 0003 (6.3.0-TIV-ITM_LFA-IF0003) and earlier. It was seen on Windows but is platform independent.
Problem conclusion
Check the length of the event before accessing the data. The fix for this APAR is included in the following maintenance vehicle: | interim fix | 6.3.0-TIV-ITM_LFA-IF0004 available at http://www.ibm.com/support/docview.wss?uid=swg24039388
Temporary fix
Comments
APAR Information
APAR number
IV64093
Reported component name
ITM LOG FILE AG
Reported component ID
5724C04LF
Reported release
630
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2014-08-28
Closed date
2015-02-26
Last modified date
2015-02-26
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
ITM LOG FILE AG
Fixed component ID
5724C04LF
Applicable component levels
R630 PSY
UP
[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSTFXA","label":"Tivoli Monitoring"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"630"}]
Document Information
Modified date:
30 December 2022