Fixes are available
Tivoli Log File Agent, Version 6.3.0 Fix Pack 01 (6.3.0-TIV-ITM_LFA-FP0001)
Tivoli Log File Agent, Version 6.3.0 Interim Fix 04 6.3.0-TIV-ITM_LFA-IF0004
Tivoli Log File Agent, Version 6.3.0 Fix Pack 02 (6.3.0-TIV-ITM_LFA-FP0002)
Tivoli Log File Agent, Version 6.3.0 Interim Fix 05 6.3.0-TIV-ITM_LFA-IF0005
APAR status
Closed as program error.
Error description
All events are not matched and sent to the event receiver. The events that are received might contain garbled characters. Close examination of the agent log shows that parts of the events or lines are overwritten. This only occurs when translating the data from a multi-byte code page to UTF-8 and the conversion fails. The conversion fails because the agent was unable to read all the bytes required for a complete character into its buffer. This might occur if a group of messages are sent or when a message exceeds the maximum event size (EventMaxSize). Increasing the EventMaxSize might produce different results. Affected Versions: The problem exists on the following Log File Agent versions: - 6.3.0 interim fix 0003 and earlier - 6.2.3.2 and later - 6.2.2.4 Interim Fix 07 6.2.2.4-TIV-ITM_LFA-IF0007 and later. It is platform independent. Problem Determination: On the LO agent system, enable: KBB_RAS1= ERROR (UNIT:logmonitor all)(UNIT:kum0nget all) (UNIT:kumprmfr all) The agent RAS1 log <hostname>_lo_[instance]_kloagent_<timestamp>-<nn>.log might contain these error indicators: "partial data", "RESIDUAL" , or "*****Error: u_strFromUTF8 failed for string". The trace entries are similar to the following where the translation buffer ends 0x1A and contains partial data. <timestamp>:kum0nget.c,411,"TranslateStringToUTF8") Entry <timestamp>:kum0nget.c,417,"TranslateStringToUTF8") translateBuffer allocated 865 bytes at 43075E0 <timestamp>:kum0nget.c,419,"TranslateStringToUTF8") Converting string buffer from ibm-943_P15A-2003 to UTF-8 <timestamp>:kum0nget.c,467,"TranslateStringToUTF8") Input buffer 43E31C3 of length 288 when translated into 43075E0 of length 293 has partial data <timestamp>:kum0nget.c,469,"TranslateStringToUTF8") <0x43E31C3,0x120> +<timestamp> 00000000 23232323 3C323031 342F3132 2F303120 !!! ####<2014/12/01. . . . . . . +<timestamp> 00000100 31373430 39353439 3936383E 203C4245 17409549968>.<BE +<timestamp> 00000110 412D3030 30303030 3E203C83 81836783 A-000000>.<...g. <timestamp>:kum0nget.c,470,"TranslateStringToUTF8") <0x43075E0,0x125> +<timestamp> 00000000 23232323 3C323031 342F3132 2F303120 !!! ####<2014/12/01. . . . . . . +<timestamp> 00000100 3C313431 37343039 35343939 36383E20 <1417409549968>. +<timestamp> 00000110 3C424541 2D303030 3030303E 203CE383 <BEA-000000>.<.. +<timestamp> 00000120 A1E38388 1A <--- ..... !! >>> Note: When the failure occurs, the last three bytes of the translated data are 0xEF 0xBF0 0xBD or the last byte is a 0x1A. <timestamp>:kum0nget.c,479,"TranslateStringToUTF8") Buffersize 865 bytes; Translated size 292; Copied 865 characters to readBuffer 43E31C3 RESIDUAL 1 <---- !!!!!!!!!! <timestamp>:kum0nget.c,496,"TranslateStringToUTF8") translateBuffer freeing 43075E0 <timestamp>:kum0nget.c,498,"TranslateStringToUTF8") Exit: 0x43E31C3 . . . <timestamp>:kum0nget.c,377,"KUM0_Fgets") <0x43E32E7,0x49> buffer +<timestamp> 00000000 8A836283 4E955C22 646F6D61 696E5F6F ..b.N.\"domain_o +<timestamp> 00000010 7261636C 655F6F69 6D3A6A64 62632282 racle_oim:jdbc". +<timestamp> 00000020 C9834C81 5B97F182 AA82A082 E882DC82 ..L.[........... +<timestamp> 00000030 B982F181 428EFB8F 5782B382 EA82DC82 ....B...W....... +<timestamp> 00000040 B982F181 423E200D 0A >>> The data above is lost. . . . (<timestamp>:kumprmfr.c,449,"KUMP_ReadMonitorFileUnicodeRecord") <0x43DD208,0x6318> . . . . . . +<timestamp> 000060C0 34303935 34393936 383E203C 4245412D 409549968>.<BEA- +<timestamp> 000060D0 30303030 30303E20 3CE383A1 E3832323 000000>.<.....## +<timestamp> 000060E0 23233C32 3031342F 31322F30 31203133 ##<2014/12/01.13 ... (<timestamp>:kum0regx.c,286,"KUM0_IsRegExPatternMatch") *****Error:u_strFromUTF8 failed for string ... RECREATE INSTRUCTIONS: This issue is not easily reproduced. It depends on the size of the lines being read, and the EventMaxSize. The key to recreating this issue is: 1) Use a multi-line format in the format (.fmt) file, for example %n in a FORMAT statement and or 0x0D0A in REGEX statement. 2) Require translation from a multi-byte code page to UTF-8. 3) Specify an EventMaxSize parameter value, such that a complete multi-byte character is not read in and translated. We created test messages. It was 2000 records and 130 records should be detected by LFA. After starting LFA with default EventMaxSize, added the 2000 records to the monitored log. But no event was not detected by LFA. Using the following data inserted into the monitored log: testdata200-13A.txt or testdata2000-130A.txt. Approver Initials: LD
Local fix
Edit the .conf file. Specify "EventMaxSize=XXXXX" where "XXXXX" is based on the size of the largest event including slotnames, equal signs, quotes, etc. The maximum value is 100000. *Note* the LO agent uses the amount of memory specified in EventMaxSize for all events, even ones that are smaller. Therefore, do not use the maximum value unless it is required. See above trace points for an indication of the size of the events.
Problem summary
All events are not matched and sent to the event receiver. The events that are received might contain garbled characters. This only occurs when translating the data from a multi-byte code page to UTF-8 and the conversion fails because the agent was unable to read all the bytes required for a complete character into its buffer. This might occur if a group of messages are sent or when a message exceeds the maximum event size (EventMaxSize).
Problem conclusion
Properly handle the incomplete character read. The fix for this APAR is included in the following maintenance vehicle: | interim fix | 6.3.0-TIV-ITM_LFA-IF0004 available at http://www.ibm.com/support/docview.wss?uid=swg24039388
Temporary fix
Comments
APAR Information
APAR number
IV68490
Reported component name
ITM LOG FILE AG
Reported component ID
5724C04LF
Reported release
630
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2015-01-12
Closed date
2015-02-26
Last modified date
2015-02-26
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
ITM LOG FILE AG
Fixed component ID
5724C04LF
Applicable component levels
R630 PSY
UP
[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSTFXA","label":"Tivoli Monitoring"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"630"}]
Document Information
Modified date:
30 December 2022