Poll data aggregation

Edit online

The historical poll data processing and storage system is designed to accept data from all instances of the Polling engine, ncp_poller across all domains. Within these ncp_poller instances, only data from poll policies that are configured to store data will be aggregated into historical data.

The system automatically aggregates data into four collections, each representing a different time period:
  • Last day
  • Last week
  • Last month
  • Last year
Each of these collections, including an additional collection that receives the raw data, is automatically purged as data ages out of its time period. The raw data is collected in a database table that maintains the last hour's worth of data.

Raw polling data, and the results of the data aggregation for the four time periods, is stored in the NCPOLLDATA database. All of the Apache Storm processing required to produce this aggregated data is set up by default. This enables Network Manager to provide a view of polling data going back a full year.

Raw polling data, and the results of the data aggregation for the four time periods, are stored in the following database tables within the NCPOLLDATA database.
  • Raw polling data is stored in the pollData table.
  • There are four summary data tables, one for each time period. In order to populate these tables with historical poll data, Apache Storm continuously reads data from the raw poll data table, pollData, and calculates aggregated poll data based on a separate default intervals for each time period using an exponential weighted moving average (EWMA) of the raw poll data. Storm then writes this average data to the relevant aggregated poll data table. For example, for the last day time period Storm calculates an average of the raw poll data every 15 minutes and writes this average value to the pdEwmaForDay table. The following NCPOLLDATA database tables store historical poll data.
    Note: Aggregated data is stored in the NCPOLLDATA database without regard to domain.
    pdEwmaForDay table
    Stores the aggregated last day of poll data with a 15 minute moving average.
    pdEwmaForWeek table
    Stores the aggregated last week of poll data with a two hour moving average.
    pdEwmaForMonth table
    Stores the aggregated last month of poll data with an 8 hour moving average.
    pdEwmaForYear table.
    Stores the aggregated last year of poll data with a 4 day moving average.