About data retention and summarization

Learn about the timeline metrics shown in the Cloud App Management console Resource dashboards and how you can configure your IBM® Cloud App Management installation to retain historical data for a shorter or longer period of time than the default 8 days and to enable hourly and daily summarization.

Data sampling

The ICAM Agents and ICAM Data Collectors are monitoring your environment for early detection of performance and availability issues.

Performance data is collected as frequently as once per minute and stored on the Cloud App Management server for 8 days by default. After 8 days, as new data samples arrive, the oldest are removed.

Resource dashboards

Along with performance and other relevant metrics displayed in the Resource dashboards, you have an Events timeline. The initial display of the timeline and chart views present the past 12 hours. You can adjust the time span to show from 3 hours up to a week. If your Cloud App Management installation is configured for a data retention value of 2 days up to 32 days, the time span options reflect the value that was set.

Data retention and summarization

Data retention is the number of days that data samples are saved before the oldest data samples are deleted to make room for new data samples. The default is 8 days. During Cloud App Management server install or upgrade, you can reconfigure retention to be from 2 to 32 days. Any value beyond 32 days is not recommended and can degrade Cloud App Management performance. If you prefer to keep the prior release's data retention during server upgrade and make a later decision about reducing raw data retention, set rawMaxDays to 32. For instructions, see step 14 of Installing your Cloud App Management server or step 8 of Upgrading your server.

Summarization refers to the aggregation of the retained data into time-based categories: hourly for short-term recall; daily for long-term recall. By default, summarization is turned off. During Cloud App Management server installation or upgrade, you can enable summarization (step 14 of Installing your Cloud App Management server or step 8 of Upgrading your server). When summarization is enabled, samples of agent metrics are summarized once per hour for short-term history and once per day for long-term history. The server keeps the summarized metrics for recall: 60 days of short-term history and 6 months of long-term history. You can change the length of time short-term and long-term history data samples are saved by creating a Kubernetes ConfigMap, as described in Configuring summarization.

Raw data retention offers the greatest flexibility in data visualization and on-demand aggregation. The more days that are retained, the more storage that is needed to store large quantities of raw data. In addition, metric query requests can take longer to fulfill as the number of days of raw data increases.

The retention period for hourly and daily summarized data should be chosen based on your long term trend analysis needs. Although hourly summarized data offers a more granular look at metrics than daily summarization, a greater amount of storage is needed to retain than daily summarized data.

Agents that support metric retention summarization

The following agents support summarization for a limited set of metrics: Linux KVM agent, Linux OS agent, UNIX OS agent, and VMware VI agent.

The Cloud App Management console dashboards automatically present the correct visualization of the data based on the available summarized metrics and the time span chosen in the Events timeline. Regardless of the retention period that you can choose for hourly and daily summarizations, the metrics that can be summarized cannot be changed.

To get a list of the metrics that can be retained and summarized for each agent, go to developerWorks and download the ICAM Metric Summarization spreadsheet . (See also Load projection spreadsheet and Using the IBM Cloud App Management Database Load Projections Spreadsheet .)