How Log File Agent storing duplicate Events and Summary Events in Tivoli Data Warehouse

Body

I recently received a question on how LFA handles Duplicate events and Summary events when they are being stored in TDW.

Event filtering and summarization configuration options are described  very good well  in the Technote "Disabling Summary Events with Log File Agent. "http://www-01.ibm.com/support/docview.wss?uid=swg21680594"
To create this instance I used the autodiscovery feature. I placed my duplicate.conf and duplicate.fmt files under KLO_FILE_DISCOVERY_DIR=$\{CANDLE_HOME\}/config/lo. When you are using this feature, you must make sure that create a pairs of configuration and format files. In my exampe I am using duplicate.fmt and duplicate.conf files.

Example of my duplicate.fmt file:

// Matches a simple error message like:
//  Error: C disk full
REGEX REBase
Error: ([A-Z])(.*)
severity $1 CustomSlot1
msg $2
END

Example of my duplicate.conf file:
LogSources=c:/test.log
DupDetectionKeyAttributes=msg,CustomSlot1
EventSummaryInterval=60

EventFloodThreshold=
# In examples below I will changing EventFloodThreshold tag for every recreate to send_none, send_first, send_all and n_integer 

Test with EventFloodThreshold=send_none tag:

I append below messages in test.log log.

Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full

When summary interval expired EventSummaryInterval=60 I run below command against short term history file

C:\IBM\ITM\TMAITM6\logs\History\KLO\duplicate>krarloff -h -d ";" -m KLOLOGPEVT.hdr -o KLOLOGPEVT_Send_None.out -s KLOLOGPEVT

From KLOLOGPEVT_Send_None.out can be seen that only Summary events were sent to TDW. Rest of duplicate events were dropped.

Test with EventFloodThreshold=send_first.tag:

I append below messages in test.log log.

Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full

After that I run command:

C:\IBM\ITM\TMAITM6\logs\History\KLO\duplicate>krarloff -h -d ";" -m KLOLOGPEVT.hdr -o KLOLOGPEVT_Send_First.out -s KLOLOGPEVT

From KLOLOGPEVT_Send_First.out can be seen that only the first event is sent to TDW, rest of duplicate events are dropped. Summary event at the end of summary interval with count of all duplicated events for each message is also sent to TDW:

Test with EventFloodThreshold=send_all.tag:

I append below messages in test.log log.

Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full

After that I run command:

C:\IBM\ITM\TMAITM6\logs\History\KLO\duplicate>krarloff -h -d ";" -m KLOLOGPEVT.hdr -o KLOLOGPEVT_Send_All.out -s KLOLOGPEVT

From KLOLOGPEVT_Send_All.out can be seen that all event are sent to TDW and Summary event at the end for summary interval with count of all duplicated events for each message:

Test with EventFloodThreshold=5 tag:

I append below messages in test.log log.

Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: E Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full
Error: C Disk Full
Error: D Disk Full

After that I run command:

C:\IBM\ITM\TMAITM6\logs\History\KLO\duplicate>krarloff -h -d ";" -m KLOLOGPEVT.hdr -o KLOLOGPEVT_5.out -s KLOLOGPEVT

From KLOLOGPEVT_5.out can be seen that every 5th event is sent to TDW and Summary event at the end for summary interval with count of all duplicated events for each message:

Subscribe and follow us for all the latest information directly on your social feeds: