IBM Support

How Log File Agent storing duplicate Events and Summary Events in Tivoli Data Warehouse

Technical Blog Post


Abstract

How Log File Agent storing duplicate Events and Summary Events in Tivoli Data Warehouse

Body

I recently received a question on how LFA handles Duplicate events and Summary events when they are being stored in TDW.

Event filtering and summarization configuration options are described  very good well  in the Technote "Disabling Summary Events with Log File Agent. "http://www-01.ibm.com/support/docview.wss?uid=swg21680594"
To create this instance I used the autodiscovery feature. I placed my duplicate.conf and duplicate.fmt files under KLO_FILE_DISCOVERY_DIR=$\{CANDLE_HOME\}/config/lo. When you are using this feature, you must make sure that create a pairs of configuration and format files. In my exampe I am using duplicate.fmt and duplicate.conf files.

 

Example of my duplicate.fmt file:

// Matches a simple error message like:
//  Error: C disk full
REGEX REBase
Error: ([A-Z])(.*)
severity $1 CustomSlot1
msg $2
END

 

 

Example of my duplicate.conf file:
LogSources=c:/test.log
DupDetectionKeyAttributes=msg,CustomSlot1
EventSummaryInterval=60

EventFloodThreshold=
# In examples below I will changing EventFloodThreshold tag for every recreate to send_none, send_first, send_all and n_integer 

 

 

Test with EventFloodThreshold=send_none tag:

I append below messages in test.log log.

Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full

 

When summary interval expired EventSummaryInterval=60 I run below command against short term history file

C:\IBM\ITM\TMAITM6\logs\History\KLO\duplicate>krarloff -h -d ";" -m KLOLOGPEVT.hdr -o KLOLOGPEVT_Send_None.out -s KLOLOGPEVT

 

From KLOLOGPEVT_Send_None.out can be seen that only Summary events were sent to TDW. Rest of duplicate events were dropped.

TECCLASSLOGNAMEEIFEVENTMSGCUSLOT1OCOUNTEVTYPESAMPLES
REBasetest.log  Disk FullE1110
REBasetest.log  Disk FullD4111
REBasetest.log  Disk FullC4112

 

 

 

Test with EventFloodThreshold=send_first.tag:

I append below messages in test.log log.

Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full

 

After that I run command:

C:\IBM\ITM\TMAITM6\logs\History\KLO\duplicate>krarloff -h -d ";" -m KLOLOGPEVT.hdr -o KLOLOGPEVT_Send_First.out -s KLOLOGPEVT

 

From KLOLOGPEVT_Send_First.out can be seen that only the first event is sent to TDW, rest of duplicate events are dropped. Summary event at the end of summary interval with count of all duplicated events for each message is also sent to TDW:

TECCLASSLOGNAMEEIFEVENTMSGCUSLOT1OCOUNTEVTYPESAMPLES
REBasetest.log  Disk FullC1010
REBasetest.log  Disk FullD1011
REBasetest.log  Disk FullE1012
REBasetest.log  Disk FullE1110
REBasetest.log  Disk FullD4111
REBasetest.log  Disk FullC4112

 

 

Test with EventFloodThreshold=send_all.tag:

I append below messages in test.log log.

Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full

 

After that I run command:

C:\IBM\ITM\TMAITM6\logs\History\KLO\duplicate>krarloff -h -d ";" -m KLOLOGPEVT.hdr -o KLOLOGPEVT_Send_All.out -s KLOLOGPEVT

 

From KLOLOGPEVT_Send_All.out can be seen that all event are sent to TDW and Summary event at the end for summary interval with count of all duplicated events for each message:

TECCLASSLOGNAMEEIFEVENTMSGCUSLOT1OCOUNTEVTYPESAMPLES
REBasetest.log  Disk FullC1010
REBasetest.log  Disk FullD1011
REBasetest.log  Disk FullE1012
REBasetest.log  Disk FullC1013
REBasetest.log  Disk FullD1014
REBasetest.log  Disk FullC1015
REBasetest.log  Disk FullD1016
REBasetest.log  Disk FullC1017
REBasetest.log  Disk FullD1018
REBasetest.log  Disk FullE1110
REBasetest.log  Disk FullD4111
REBasetest.log  Disk FullC4112

 

 

Test with EventFloodThreshold=5 tag:

I append below messages in test.log log.

Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full

 

After that I run command:

C:\IBM\ITM\TMAITM6\logs\History\KLO\duplicate>krarloff -h -d ";" -m KLOLOGPEVT.hdr -o KLOLOGPEVT_5.out -s KLOLOGPEVT

 

From KLOLOGPEVT_5.out can be seen that every 5th event is sent to TDW and Summary event at the end for summary interval with count of all duplicated events for each message:

TECCLASSLOGNAMEEIFEVENTMSGCUSLOT1OCOUNTEVTYPESAMPLES
REBasetest.log  Disk FullC1010
REBasetest.log  Disk FullD1011
REBasetest.log  Disk FullC1012
REBasetest.log  Disk FullD1013
REBasetest.log  Disk FullE3110
REBasetest.log  Disk FullD12111
REBasetest.log  Disk FullC12112

 

 

Tutorials Point

 

Subscribe and follow us for all the latest information directly on your social feeds:

 

 

image

 

image

 

image

 

 

  

Check out all our other posts and updates:

Academy Blogs: http://ow.ly/Otue0
Academy Videos: http://ow.ly/PIKFz
Academy Google+: http://ow.ly/Dj3nn
Academy Twitter Handle: http://ow.ly/Dj35c


image

[{"Business Unit":{"code":"BU025","label":"IBM Cloud and Cognitive Software"}, "Product":{"code":"SSVJUL","label":"IBM Application Performance Management"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"","Edition":""}]

UID

ibm11277368