Best practice: Tuning the event data loader

Tuning may help to prevent event data files from switching too quickly to be read by the data loader, resulting in lost records in the data being loaded.

Consider the following critical factors when tuning the event data loader:
  • Stream file size: This is the most critical factor. For a production cluster, check the stream file switch time. The average switch time must be greater than 3 minutes. If the switch time is too frequent, increase the file size. Note that the file size also impacts the stream file writing speed; so if the file is too large, writing speed will slow down.

    The stream file size, specified in Mbytes, is controlled by the parameter EGO_DATA_MAXSIZE in ego.conf.

  • Loader interval: This value, specified in seconds, must be less than the stream file switch time but greater than the database loading speed.
    To set the loader interval, open the plc_ego.xml file and change the Interval attribute for egoeventsloader. For example:
    <DataLoader Name="egoeventsloader" Interval="100" Enable="true"
     LoadXML="dataloader/egoevents.xml" />
    
Consider the following factors related to data-loading performance and ensure that they are optimized for speed:
  • Performance of the target database
  • Batch size of the loader
  • Performance of the host running the loader.

If tuning cannot provide satisfactory loader performance, you can create multiple loader controller service instances to balance the workload. Contact your sales representative for more information.