IZ41189: ALL PURE EVENTS NOT WRITTEN TO THE WAREHOUSE IN AN EVENT STORM

APAR status

• Closed as program error.

Error description

• Environment:
  TEMS/TEPS/TEPD/WPA/AgentBuilder/AgentBuilderAgent
  Win2k3 ITM6.2+FP02
  
  Problem Description:
  The customer created an agent in order to monitor a text log
  file by the Agent Builder.Then he tried to collect the
  history data from the agent.Initially, the monitored log
  file included 179,424 enties.
  
  The history collection file "K05SYSLOGC" under
  <ITMHOME>/tmaitm6/logs directory included all 179,424
  entries successfully.
  
  However, after the warehouse interval, in this case one hour,
  only 169,000 entries were stored into WAREHOUS database.
  
  The missing records are not on the tail end or not at
  the beginning but in the middle.
  
  Detailed Recreation Procedure:
  I could recreate the problem in my test environment.
  
  1) Install TEMS, TEPS, TEP Desktop Client, WPA, Windows OS
  agent and Agent Builder on a single win2k3 box.
  
  2) Apply FP0002 for each component.
  
  3) Launch the Agent Builder and click "Create New Agent"
  icon.
  
  4) On "New IBM Tivoli Monitoring Agent Project" window,
  specify the project name "Project-K05" and click "Next".
  
  5) On "Agent Information" window, specify the following
  values.
  
     Company identifier : IBM
     Agent identifier   : K05
     Display name       : K05 SYSLOG Agent
     Product code       : K05
     Version            : 620
     Copyright          : IBM
  
  Then click "Next".
  
  6) On "Monitoring Information" window, check
  "This agent will gather data from an external data source."
  
  7) On "Data Source Definition" window, click "New Data
  Source".
  
  8) On "Data Source Location" window, select "Collect data
  from a Log File."Then click Next.
  
  9) On "Log File Information" window, specify
  "C:\userfiles\SyslogCatchAll.txt.001" as Log file name
  and select "Tab separator".Then click "Advanced".
  
  10) On "Advanced Log File Data Source Information",
  select the followings.
  
     Process new records appended to the file
       Detect new records when the file size increases
          Ignore existing records
  
     Single line
  
     Then click OK.
  
  11) Back to "Log File Information" window, click finish.
  
  12) Add the following attributes.
  
      Attribute name : datetime
      Maximum size   : 64
      Help text      : datetime
      Recoed Field Information : Next field
      check "Display attribute in the Tivoli Enterprise Portal"
      check "Display string"
  
      Attribute name : priority
      Maximum size   : 64
      Help text      : priority
      Recoed Field Information : Next field
      check "Display attribute in the Tivoli Enterprise Portal"
      check "Display string"
  
      Attribute name : hostname
      Maximum size   : 64
      Help text      : hostname
      Recoed Field Information : Next field
      check "Display attribute in the Tivoli Enterprise Portal"
      check "Display string"
  
      Attribute name : message
      Maximum size   : 2000
      Help text      : message
      Recoed Field Information : Remainder of record
      check "Display attribute in the Tivoli Enterprise Portal"
      check "Display string"
  
      Then click finish and generate this agent.
  
  13) On TEP Desktop Client, click "History Configuration"
  bottun.
  
  14) On "History Collection Configuration" window, select
  "K05 SYSLOG Agent" as Select a Product field and configure
  "K05 SYSLOGCATCHALL TXT 001" group with the following
  values.
  
     Collection Interval : 30 minutes
     Collection Location : TEMA
     Warehouse Interval  : 1 hour
  
  Then click "Start Collection".
  
  15) From MTEMS, start the agent.
  
  16) Copy SyslogCatchAll.txt.001 sent from the customer
  to C:/userfiles directory.
  
  17) Start WPA and wait for a few hours.Then a new table
  named "K05_SYSLOGCATCHALL_TXT_001" is created into
  WAREHOUS database.
  
  18) After connecto to WAREHOUS database, run the following
  command.
  
  db2 -z select.out select * from
  itmuser.k05_syslogcatchall_txt_001
  
  19) Having a look at select.out, only 169,000 entries of
  179,424 were stored into WAREHOUS database.This is the
  problem.
  
  20) Having a look at K05SYSLOGC under <ITMHOME>/tmaitm6/logs
  directory using krarloff command, all 179,424 records were
  included.
  
  Related Files and Output:
  - history collection file
  - krarloff command output
  - db2 corresponding table export output
  

Local fix

• If WRITETIME value of n*1000 th row is different from
  WRITETIME value of n*1000+1 th row in the history
  collection file, this problem can be avoided.
  

Problem summary

• An agent that is generating more than 999 events per second does
  not get all entries uploaded to the WAREHOUS database at the
  warehouse interval, even though all events were correctly
  created in the agent historical data binary file.  The code was
  only processing the first 999 entries with the same timestamp
  and all events with the same timestamp after the initial 999
  were deemed to be duplicates and were discarded.
  

Problem conclusion

• The code was modified to increment the SAMPLES value for each
  row of data if its timestamp (up to seconds) is the same as that
  of the previous row.  The combination of timestamp and SAMPLES
  would ensure that each record is treated as an individual one
  rather than as a duplicate.  The fix has only been made for
  tables defined to use pure event sampling since the problem has
  been seen with pure events only.
  
  
  The fix for this APAR is contained in the following maintenance
  packages:
  
     | fix pack | 6.2.0-TIV-ITM-FP0003
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  OA28769 IZ50733 OA29273

Fix information

• Fixed component name

  TEMA

• Fixed component ID

  5724C04TE

Applicable component levels

• R620 PSY

     UP