Configuring data retention and index rollover time periods

You can set a time period for data retention and index rollover.

About this task

Each API event is recorded as a document in a shared index. When this index reaches a maximum size or age it is archived, and a new index is created for recording new API events. This is known as index rollover. When an archived index reaches a maximum age, which is known as the data retention maximum, it is deleted. You can specify index rollover and data retention settings that control how often new indexes are created, and how long archived indexes are stored.

The index rollover settings determine how long an index accumulates data before it "rolls over" and a new index is created. You can control the duration of an index two ways: by setting a maximum age for the index (default is 1 day), and by setting a maximum number of documents that can be recorded in the index (default is 25 million documents). When the rollover occurs, a new index is created and new API events are recorded there, while the previous index is archived and kept until its age exceeds the data retention setting. If you change the rollover settings, consider both the amount of data to be stored within each index and the total number of indexes. Allowing indexes to grow too large, or storing too many indexes, can cause performance problems.

The data retention setting determines how long the stored indexes are retained. When an index's creation date exceeds the retention period, that index and all its data is deleted, even if some of the data that is stored in that index is younger than the retention period.

The default retention period is 30 days for a one replica deployment, and 90 days for a three replica deployment. Reasons for changing this setting include storage constraints, and data retention requirements for your organization. You might want to set this value to be less than the default if many API events are stored, especially if payload logging is enabled on the APIs.

The most recently created index (the index that is currently being written to) is not deleted, even if you set the retention period as small as 1 hour. If you regularly need to delete data quickly, adjust the Index Rollover setting so that a new index is rolled over to sooner, and the old index can be deleted.

Note: The rollover and retention settings have a significant impact on the memory that is required by the analytics subsystem. If you have rollover set to one day (1d), then to maintain an acceptable level of performance the following retention settings are recommended for each deployment profile:

n1xc2.m16: This profile provides 3 GB of heap memory for storage on one node. Recommended retention is four days or less.
n1xc4.m32: This profile provides 11 GB of heap memory for storage on one node. Recommended retention is 15 days or less.
n1xc6.m48: This profile provides 19 GB of heap memory for storage on one node. Recommended retention is 25 days or less.
n3xc4.m16: This profile provides 3 GB of heap memory for storage on three nodes. Recommended retention is 12 days or less.
n3xc4.m32: This profile provides 11 GB of heap memory for storage on three nodes. Recommended retention is 45 days or less.
n3xc6.m48: This profile provides 19 GB of heap memory for storage on three nodes. Recommended retention is 76 days or less.
n3xc8.m64: This profile provides 27 GB of heap memory for storage on three nodes. Recommended retention is 108 days or less.

If rollover is set to higher than one day, then more analytics data can be stored before a performance deterioration is observed. However, a higher rollover setting has the disadvantage that more data is deleted when an index reaches the retention age. For example, if you set your rollover period to seven days and you have the default 90-day retention period, then when your oldest index reaches the 90 days since it was created, it is deleted and you lose your first seven days of event data.

To view and update the rollover and retention settings, from v10.0.5.3 you can use the Cloud Manager UI. For previous releases, use the toolkit CLI.

Procedure

From the Cloud Manager UI, click Topology.
Select the Analytics Service you want to view or edit.
Click Advanced settings.
Modify your rollover and retention settings as required and click Save.

v10.0.5.2 and previous releases: Use the toolkit CLI.

View the current retention and rollover settings by using the clustermgmt:getStorageSettings analytics CLI command:
```
apic -m analytics clustermgmt:getStorageSettings --server [SERVER] --analytics-service [ANALYTICS SERVICE] --format json
```
- <management server platform api endpoint> is the Management server platform api endpoint.
- <analytics service name> is the name of Analytics service as configured in the Cloud Manager.
Example output:
```
{
    "rollover": {
        "min_doc_count": 25000000,
        "min_index_age": "1d"
    },
    "retention": {
        "min_index_age": "30d"
    }
}
```
"min_" refers to the minimum age or number of documents that are reached before the action takes place.
To update the retention and rollover settings create a JSON file that is called retention_and_rollover_settings.json, paste in the output from step 5 and set the retention and rollover setting as required. Apply these new settings with: clustermgmt:putStorageSettings
```
apic -m analytics clustermgmt:putStorageSettings --server [SERVER] --analytics-service [ANALYTICS SERVICE] retention_and_rollover_settings.json
```
- <management server platform api endpoint> is the Management server platform api endpoint.
- <analytics service name> is the name of Analytics service as configured in the Cloud Manager.

If the retention or rollover process fails, the Cloud Manager and API Manager UIs show the following error message:

Error: Analytics internal storage rollover might be experiencing problems. Contact your system administrator to investigate further. Rollover failure means that data retention is not correctly enforced and storage usage continues to grow which can eventually lead to service shutdown.

If you see this error, then open a support case.