Understanding historical data collection

Before you can configure historical data collection, understand the difference between short-term and long-term historical data and how both are collected.

Historical data collection is an optional feature that is enabled using the Tivoli Enterprise Portal or OMEGAMON® Enhanced 3270 User Interface. The OMEGAMON platform provides the following types of historical data collection:
  • Short-term historical data is stored in the persistent data store on z/OS® systems or in files on distributed systems. To optimize performance, configure the persistent data store at the monitoring agent, meaning that you will have a persistent data store on each z/OS system you are monitoring.

    Short-term historical data typically refers to data that is stored for 24 hours or less. However, the amount and age of the short-term data that is retained depends on the number of resources being monitored and the amount of disk space configured for use by the persistent data store.

  • Long-term historical data is stored in the Tivoli Data Warehouse. The long-term history database can retain data collected by IBM® Z OMEGAMON® AI for Networks monitoring agents for as long as you like (days, weeks, months or years).

Short-term historical data is best used for analysis during problem determination. Additional prerequisite software is not required for short-term historical data collection, however the data sets used by the persistent data store must be configured using the PARMGEN configuration method.

Long-term historical data is better used for trend analysis and to determine workload balance. See the chapter about configuring the warehouse proxy for Tivoli Data Warehouse in the IBM Tivoli Monitoring: Installation and Setup Guide for the list of supported databases, releases and operating system platforms. Long-term history also requires installation of the warehouse proxy software (provided) and configuration of an Open Database Connectivity (ODBC) connection. Use the warehouse proxy installation default support for defining database tablespaces and creating the ODB connection. Short-term historical data collection must be enabled and configured if you want to perform long-term historical data collection.

After historical data collection is enabled, an icon is displayed in qualifying views in Tivoli Enterprise Portal workspaces. Allow time for historical data to be stored, to produce meaningful reports. You can click this icon to extend any existing Tivoli Enterprise Portal view (also called a report) to include historical data. Tivoli Enterprise Portal reports automatically pull data from short-term and long-term history, based upon the time period you specify for the report.

The collection interval for historical data can be configured to be different from the collection interval for real-time data. To avoid excessive processing activities and decrease storage consumption, historical data collection is typically performed less frequently than real-time data collection. You can configure a short-term historical data collection interval of 1, 5, 15, 30 or 60 minutes or 1 day.

Writing the data to long-term history can be configured for 15 or 30 minutes or 1, 12, or 24 hours. If you configure long-term history, use a warehousing interval of 1 hour to avoid transferring 24 hours worth of historical data at one time. This shorter interval reduces the duration of CPU usage associated with writing data to the warehouse by spreading the writing across 24 periods.

The following example illustrates the rate of accumulation of historical records, assuming the following intervals have been specified:
  • Real-time data collection interval: 5 minutes
  • Short-term historical data collection interval: 15 minutes
  • Long-term warehousing interval: 1 hour
  1. At 1:57, data collection is initiated for 1 TCP/IP Address Space.
  2. One row of data (512 bytes) is collected at 1:57, another row at 2:02, another at 2:07, 2:12, 2:17, and so on. Only the current row of data is stored because the old row is discarded when a new row arrives. Because the real-time data collection interval is 5 minutes, 12 collections are made per hour.
  3. At 1:58, collection of historical data is initiated for 1 TCP/IP Address Space.
  4. One row of data (540 bytes) is stored in short-term history at 2:00, the second row is collected at 2:15, the third at 2:30, 2:45, 3:00, and so on. Short-term history uses the most recent collection and therefore does not initiate another data collection. The row stored at 2:00 would use the 1:57 collection. The row stored at 2:15 would use the 2:12 collection.
  5. Because the short-term historical data collection interval is 15 minutes, 4 collections are made per hour. All measurements are stored for future use.
  6. After one hour, all (4) rows of short-term historical data are transferred by the warehouse proxy to the long-term history SQL database.
  7. After 24 hours, 96 (24 x 4) rows of data will be stored in the Tivoli Data Warehouse.