Analytics running out of disk space on VMware

Actions that you can take if your analytics data is using more persistent storage space than you expected.

Reduce retention period

The simplest remedy to running out of storage space is to reduce the data retention period.

By default, API event records are retained for 30 days on a one replica deployment, and 90 days on a three replica deployment.

If your analytics subsystem was deployed more recently than your retention period (for example, less that 30 days ago for a default one replica deployment), and you are running out of storage space, then there is a risk that your persistent storage fills up before the retention period is reached. Check in the Cloud Manager UI how many days back your API event data goes, and set the retention period to a lower number. For example, if you see API event records from 15 days ago, reduce your storage requirement by one-third by setting your retention period to 10 days. See Configuring data retention and index rollover time periods.

Add extra disk

You can increase the space available for your analytics data by adding a disk, see: Adding disk space to a VMware appliance.

Warning: Do not attempt to increase the size of the existing data disk. Changing the disk size corrupts your existing analytics data.

Reduce space used by new API event data

To reduce the storage required for new analytics data, update the activity log settings of your APIs. If you have logging set to payload when it is not needed, then change logging to activity: Configuring activity logging.
Note: If you're not able to update the activity log setting for your APIs, you can disable response payload logging for all APIs with ingestion filters. For example:
filter {
  if "apicapievent" in [tags] {
    mutate {
      remove_field => ["response_body"]
    }
  }
}
For more information about filters, see: Planning to modify your analytics data.

What to do if your storage is 100% full

If your storage space is 100% full, your analytics subsystem might be unresponsive, so reducing the retention period is not possible. In this scenario, follow the disaster recovery steps: Disaster recovery.