Table Optimizer configuration options
Table Optimizer provides several configuration options to optimize your Iceberg tables. You can customize each configuration option with specific parameters to match your workload requirements.
Compaction
Compaction improves query performance by rewriting small or oversized data files into optimally sized files. Excessively small files increase query planning overhead and reduce scan efficiency, while overly large files may limit parallelism during query execution.
Compaction ensures data files remain close to the target file size, improving both performance and metadata efficiency.
Optional parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| Target object size | Size | 128 MB | Specify the desired size of data files after compaction. The system rewrites files to match this target size. |
| Small file size ratio | Ratio | 0.75 (75%) | Specify the ratio to identify small files. The system considers files smaller than target size × ratio as small. For example, 0.75 × 128MB = 96MB, so the system treats files under 96MB as small files. |
| Large file size ratio | Ratio | 1.8 (180%) | Specify the ratio to identify large files. The system considers files larger than target size × ratio as large. For example, 1.8 × 128MB = 230MB, so the system considers files larger than 230MB as oversized. |
| Small file count threshold | Ratio | 0.1 (10%) | Specify the minimum percentage of small files required to trigger compaction. Example: 0.1 means the system runs compaction if 10% or more files are small. |
| Large file count threshold | Ratio | 0.1 (10%) | Specify the minimum percentage of large files required to trigger compaction. Example: 0.1 means the system runs compaction if 10% or more files are oversized. |
Expire Snapshots
Iceberg creates a new snapshot for every table commit. Over time, these snapshots accumulate and increase metadata size and storage cost.
The Expire Snapshots optimization removes old snapshots while preserving a configurable number of recent snapshots for time-travel and recovery.
Required parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| Snapshot expiration threshold (days) | Number | 5 | Specify the number of days after which the system expires snapshots unless they are among the most recent snapshots retained. |
| Number of snapshots to retain | Number | 5 | Specify the minimum number of recent snapshots that the system must always preserve regardless of their age. |
Partition Evolution
Partition Evolution analyzes query patterns and table statistics to recommend optimal partitioning strategies. Proper partitioning helps reduce the amount of data scanned during queries and significantly improves query performance.
Required parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| Queries path | String | — | Specify the path to SQL query files used for analysis. These queries help the system identify common filtering patterns and recommend partitioning. |
Optional parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| Analysis sampling ratio | Ratio | 0.01 (1%) | Specify the percentage of table data to sample during analysis. Lower values speed up analysis but may be less accurate. 0.01 means the system analyzes 1% of data. |
| Target data file size | Size | 128 MB | Specify the desired file size used to estimate optimal partition granularity. |
| Minimum partition table cardinality | Number | 1000 | Specify the minimum number of rows a table must have to be considered for partitioning. Smaller tables don't benefit from partitioning. |
| Minimum column cardinality for identity partitioning | Number | 10 | Specify the minimum number of distinct values a column must have to be used for partitioning. Columns with fewer values won't provide meaningful partitions. |
| Maximum column cardinality for identity partitioning | Number | 4000 | Specify the maximum number of distinct values a column can have for partitioning. Columns with too many distinct values may produce excessive partitions. |
| Minimum data required under partition | Size | 32 MB | Specify the minimum data size required per partition to avoid creating excessively small partitions. |
| Minimum data required under bucket | Size | 100 MB | Specify the minimum amount of data required per bucket when bucketing is used. |
| Periodic data ingestion | Boolean | true | Specify whether the table receives data periodically. This helps the system determine whether time-based partitioning strategies may be beneficial. |
Remove Orphan Files
Orphan files are data files present in storage but not referenced by any Iceberg snapshot. These files typically occur due to failed or interrupted write operations.
Removing orphan files helps reclaim storage space and maintain a clean table state.
Optional parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
| Orphan files removal threshold (days) | Number | 5 | Specify the number of days after which the system removes orphan files. The system removes only orphan files older than this number of days. This safety buffer prevents accidental removal of files from ongoing operations. |