Analytics running out of disk space on Kubernetes
Actions that you can take if your analytics data is using more persistent storage space than you expected.
Reduce retention period
The simplest remedy to running out of storage space is to reduce the data retention period.
By default, API event records are retained for 30 days on a one replica deployment, and 90 days on a three replica deployment.
If your analytics subsystem was deployed more recently than your retention period (for example, less that 30 days ago for a default one replica deployment), and you are running out of storage space, then there is a risk that your persistent storage fills up before the retention period is reached. Check in the Cloud Manager UI how many days back your API event data goes, and set the retention period to a lower number. For example, if you see API event records from 15 days ago, reduce your storage requirement by one-third by setting your retention period to 10 days. See Configuring data retention and index rollover time periods.
Reinstall with larger PVCs
- Take a backup of your analytics subsystem: Backing up and restoring.
- Reinstall your analytics subsystem with a larger PVC size: Deployment procedures.
- Restore your analytics data from your backup: Backing up and restoring.
Reduce space used by new API event data
payload
when it is not needed, then change logging
to activity
: Configuring activity logging.filter {
if "apicapievent" in [tags] {
mutate {
remove_field => ["response_body"]
}
}
}
For more information about filters, see: Planning to modify your incoming analytics data.What to do if your storage is 100% full
If your storage space is 100% full, your analytics subsystem might be unresponsive, so reducing the retention period is not possible. In this scenario, follow the disaster recovery steps: Disaster recovery.