Analytics running out of disk space on Kubernetes

Actions that you can take if your analytics data is using more persistent storage space than you expected.

Reduce retention period

The simplest remedy to running out of storage space is to reduce the data retention period.

By default, API event records are retained for 30 days on a one replica deployment, and 90 days on a three replica deployment.

If your analytics subsystem was deployed more recently than your retention period (for example, less that 30 days ago for a default one replica deployment), and you are running out of storage space, then there is a risk that your persistent storage fills up before the retention period is reached. Check in the Cloud Manager UI how many days back your API event data goes, and set the retention period to a lower number. For example, if you see API event records from 15 days ago, reduce your storage requirement by one-third by setting your retention period to 10 days. See Configuring data retention and index rollover time periods.

Reinstall with larger PVCs

It is not possible to increase the PVC size after installation. If you require more space for your analytics data, the analytics subsystem must be reinstalled. Follow these steps:
  1. Take a backup of your analytics subsystem: Backing up and restoring.
  2. Reinstall your analytics subsystem with a larger PVC size: Deployment procedures.
  3. Restore your analytics data from your backup: Backing up and restoring.

Reduce space used by new API event data

To reduce the storage required for new analytics data, update the activity log settings of your APIs. If you have logging set to payload when it is not needed, then change logging to activity: Configuring activity logging.
Note: If you're not able to update the activity log setting for your APIs, you can disable response payload logging for all APIs with ingestion filters. For example:
filter {
  if "apicapievent" in [tags] {
    mutate {
      remove_field => ["response_body"]
    }
  }
}
For more information about filters, see: Planning to modify your incoming analytics data.

What to do if your storage is 100% full

If your storage space is 100% full, your analytics subsystem might be unresponsive, so reducing the retention period is not possible. In this scenario, follow the disaster recovery steps: Disaster recovery.