Additional information
General tuning process
In general, the process for enabling this feature is turning it on for a storage pool, and then monitoring the system with these rates with expiration work running (i.e. expiration lifecycle rules configured on vaults or containers) for at least 48 hours, including end user request performance (using access logs to monitor latency/ops) and expiry performance (using reports and events/incidents). Gradually increase the rates if the expiry performance is too low to delete the requisite number of objects per day. Decrease the rates if end user request performance is unacceptable.
Keep in mind that the suggested rates are based on storage pool size, access pool size, and vault IDA. They do not take into account hardware differences, and thus assume relatively high-performance hardware.
Factors to consider
There are a number of factors that can affect expiry performance and what rates are appropriate for your system. Below are a number of different factors to consider, some of which provide formulas for computing rates, and some of which are general considerations.
- Expected number of objects to delete per day
- Used to compute the desired object deletion rate by dividing the expected number of objects to delete per day by 64800 seconds (18 hours, in seconds).
- Expected number of bytes to delete per day
- Used to compute the desired byte deletion rate by dividing the expected number of bytes to delete per day by 64800 seconds (18 hours, in seconds).
- Total number of objects in expiry-enabled buckets.
- Used to compute the desired object scanning rate by diving the approximate total number of objects in all expiration-enabled buckets by 64800 seconds (18 hours, in seconds).
- Accesser® capabilities
- Since all scanning/reclamation work is run by the Accessers, Accesser performance is the primary bottleneck of object expiration.
- Less powerful Accessers, embedded Accessers, or Accessers under high load in your access pool implies the rates should be lowered.
- Slicestor® capabilities
- Since the deletion work ultimately involves deleting data on disk, Slicestors may be a bottleneck for the deletion stage. If you have less powerful Slicestors (or Slicestors under high load) on your storage pool, consider lowering the rates.
- System max ops and end user ops
- If you have a measurement of the max ops/s that your system can handle, you can estimate the
maximum object deletion rate to be around 80% of
max ops - user request ops.
- If you have a measurement of the max ops/s that your system can handle, you can estimate the
maximum object deletion rate to be around 80% of
- End User latency requirements
- Running expiry close to the maximum ops your system is capable of can lead to degradation in end user request latency. If you have strict latency requirements, you need to lower your rates from the suggested values.
- Average Object Size
- Byte Deletion Rate should typically be set to around the same rate as Object Deletion Rate * Average Object Size.