Configuring the Spark resource usage aggregation data loader

You can change the configuration of the sparkresusageaggloader (Spark resource usage aggregation data loader), if you want to change the behavior, how it aggregates data, and the data it aggregates.

Procedure

  1. Open the data loader configuration file: $EGO_CONFDIR/../../perf/conf/dataloader/sparkresusageaggloader.xml.
  2. Configure the following parameters for the Reader class:
    minMinutesToAggregateAfterHour
    Description: Minimum minutes past the current hour to aggregate metrics. The value is the minimum time to ensure that the metrics that are collected by the Elastic Stack have been indexed into Elasticsearch.
    Default: 10
    Valid values: 5 to 1440
    Example: <Value>10</Value>
    aggBucket
    Description: The seconds to bucket metrics to apply aggFunction, which is used for metrics with recurring frequencies; for example, memory and CPU.
    Default: 30
    Valid values: 30 to 3600
    Example: <Value>60</Value>
    aggIntervalMetric
    Description: Comma-delimited list of metrics to aggregate that are recorded on a regular interval.
    Valid values: cpu, memory, util_gpu, memory_gpu, or slots_cpu,slots_gpu
    Example: <Value>cpu,memory,util_gpu,memory_gpu,slots_cpu,slots_gpu</Value>
    aggIntervalFunction
    Description: Comma-delimited list of aggregate function to apply to aggIntervalMetric metrics within aggBucket. This parameter does affect the aggregation within Explorer, only how to calculate usage for metrics that have more than one metric with the aggBucket.
    Default: max
    Valid values: max, min, or avg
    Example: <Value>max,max,max,max,max,max</Value>
  3. Configure the following parameters for the Writer class that is used to load the output of the reader into Elasticsearch:
    bulkActions
    Description: When to flush a new bulk request (number of actions).
    Example: <Value>1000</Value>

What to do next

After editing the sparkresusageaggloader data loader controller configuration file, restart the loader controller service for your changes to take effect. Run:
egosh service stop plc
egosh service start plc