Data aggregation

The MDE utility supports data aggregation.

The DataChannel component of Tivoli® Netcool® Performance Manager is responsible for aggregating the raw metric values into daily, weekly, and monthly records. The aggregation is done for efficiency purpose to support the on-demand reporting. The pre-aggregated data can be used to produce reports more quickly than would be possible if large amounts of the raw data had to be aggregated for each query.

During this pre-aggregation process, the DataChannel component is able to do more complex calculations and aggregations on the raw data than at run time of the reports. Therefore, more aggregated/statistical values are supported for a time series with a granularity of a day, week, or month.

There are two sets of aggregation functions, these aggregation functions can be applied to any time period of that you choose. Also, for the time period which must conform to the aggregation periods, or granularity, as set by DataChannel, that is, greater than the period of one day.

The aggregated/statistical values that are supported by the MDE for granularities of one day or greater are as follows:
sampleQuality
Ratio of actual raw values that are collected to the expected number of raw values over the aggregation period. Sample quality is expressed as (actual count / expected count) * 100.

For example, with a polling interval of 15 minutes, the expected count of raw metric values for a one day period would be 96. If all 96 values are collected for that one day period, the sampleQuality would be 100, that is (96 / 96) * 100.

percentile
You can use a percentile value instead of the average or max statistics to better represent a metric that includes the occasional burst or spike. Occasional bursts or spikes that render a min or max value meaningless, and throw off average and mean calculations. In these cases, you can use a percentile calculation where you can see more accurately how the metric works over time.

The MDE API supports the following aggregation functions for all granularities:

  • avg: The average of all aggregated values.
  • min: The minimum of all aggregated values.
  • max: The maximum of all aggregated values.
  • sum: The sum of all aggregated values.
  • count: The number of all aggregated values.

Granularity

The granularity, or the time frame over which statistics are aggregated, can be set as follows:

  • When you use the command line, the granularity is defined in the query by using the <granularity> tag.

    For more information about how to set the granularity when you use MDE through the command line, see the command-line example Time Series: Raw.

  • When you use the API, the granularity is set as the granularity parameter of the getResourceAggregatedMetricTimeSeries()function.

    For more information about how to set the granularity when you use the MDE API, see the function description getResourceAggregatedMetricTimeSeries().

The possible levels of granularity are as follows:

5min
Sets the granularity of data aggregation to 5 minutes. This level of granularity can be used only if the start and end time do not span more than one day.
10min
Sets the granularity of data aggregation to 10 minutes. This level of granularity can be used only if the start and end time do not span more than one day.
15min
Sets the granularity of data aggregation to 15 minutes. This level of granularity can be used only if the start and end time do not span more than one day.
20min
Sets the granularity of data aggregation to 20 minutes. This level of granularity can be used only if the start and end time do not span more than one day.
30min
Sets the granularity of data aggregation to 30 minutes. This level of granularity can be used only if the start and end time do not span more than one day.
1hour
Sets the granularity of data aggregation to 1 hour. This level of granularity can be used only if the start and end time do not span more than one week.
2hour
Sets the granularity of data aggregation to 2 hour. This level of granularity can be used only if the start and end time do not span more than one week.
3hour
Sets the granularity of data aggregation to 3 hour. This level of granularity can be used only if the start and end time do not span more than one week.
day
Sets the granularity of data aggregation to one day. This level of granularity can be used only if the start and end time do not span more than one week.
week
Sets the granularity of data aggregation to seven days. This level of granularity can be used only if the start and end time do not span more than 90 days.
month
Sets the granularity of data aggregation to a month. This level of granularity can be used if only the start and end time do not span more than 365 days or one year.