Retention and rollover

Manage how long your analytics subsystem stores data.

Each API event is recorded as a document in an OpenSearch index. When this index reaches a maximum size or age it is archived, and a new index is created for recording new API events. This is known as index rollover. When an archived index reaches a maximum age, which is known as the data retention maximum, it is deleted. You can specify the index rollover size and the data retention period in the Cloud Manager UI.

For more information about OpenSearch indices, see OpenSearch nodes, indices, shards, and replicas.

Note: If a rollover is in the failed state for more than 36 hours, an automated forced rollover is attempted. If this attempt is unsuccessful, the following error message is displayed in the analytics director pod logs:

Error triggering forced rollover

To learn more about OpenSearch indices, see OpenSearch nodes, indices, shards, and replicas.

The index rollover settings determine how long an index accumulates data before it "rolls over" and a new index is created. You can control the duration of an index two ways: by setting a maximum age for the index (default is 1 day), or by setting a maximum number of documents that can be recorded in the index (default is 25 million documents). When the rollover occurs, a new index is created for new API events to be stored in, while the previous index is archived and kept until its age exceeds the data retention setting. If you change the rollover settings, consider both the amount of data to be stored within each index and the total number of indices. Allowing indices to grow too large, or storing too many indices, can cause performance problems.

The data retention setting determines how long the stored indices are retained. When an index's creation date exceeds the retention period, that index and all its data is deleted.

Note: It is the creation date of the index that is used to determine retention, not the age of the newest record in the index.

The default retention period depends on the deployment profile, as shown in the following table:

Table 1. Default analytics retention periods
Profile	Retention (days)
n1xc2.m16	3
n1xc4.m32	30
n1xc6.m48	45
n3xc4.m16 - DEPRECATED	2
n3xc4.m32	30
n3xc6.m48	45
n3xc8.m64	45

Reasons for changing this setting might include storage constraints, and the data retention requirements for your organization. You might want to set this value to be less than the default if you have a high transaction load and limited persistent storage space, especially if payload logging is enabled on many of your APIs. Increasing the retention period above the default is not recommended, if you want to store your analytics data for longer, then Offload to third-party systems.

The most recently created index (the index that is currently being written to) is not deleted, even if you set the retention period as small as 1 hour. If you regularly need to delete data quickly, adjust the index rollover so that a new index is rolled over to sooner, and the old index can be deleted.

Before an index is deleted, the API event data in the index is summarized and stored in a separate summary index to provide long-term data. The summary index has a separate retention setting, which is set to one year by default.

Note: The rollover and retention settings have a significant impact on the memory that is required by the analytics subsystem. If rollover is set to higher than one day, then more analytics data can be stored before a performance deterioration is observed. However, a higher rollover setting has the disadvantage that more data is deleted when an index reaches the retention age. For example, if you set your rollover period to seven days and you have the default 90-day retention period, then when your oldest index reaches the 90 days since it was created, it is deleted and you are left with 83 days of API event data.

To view and update the rollover and retention settings, use the Cloud Manager UI.

From the Cloud Manager UI, click Topology.
Select the Analytics Service you want to view or edit.
Click Advanced settings.

Alternative method using the Analytics CLI.

View the current retention and rollover settings by using the clustermgmt:getStorageSettings analytics CLI command:
```
apic -m analytics clustermgmt:getStorageSettings --server <platform api endpoint> --analytics-service <analytics service name> --format json
```
- <platform api endpoint> is the platform API endpoint.
- <analytics service name> is the name of analytics service as configured in the Cloud Manager UI.
To update the retention and rollover settings create a JSON file that is called retention_and_rollover_settings.json, paste in the output from step 1 and set the retention and rollover setting as required. Apply these new settings with: clustermgmt:putStorageSettings
```
apic -m analytics clustermgmt:putStorageSettings --server <platform api endpoint> --analytics-service <analytics service name> retention_and_rollover_settings.json
```
- <platform api endpoint> is the platform API endpoint.
- <analytics service name> is the name of analytics service as configured in the Cloud Manager UI.

Note:

If the retention or rollover process fails, the Cloud Manager and API Manager UIs show the following error message:

Error: Analytics internal storage rollover might be experiencing problems. Contact your system administrator to investigate further. Rollover failure means that data retention is not correctly enforced and storage usage continues to grow which can eventually lead to service shutdown.

If you see this error, then open a support case.