Data Retention and Compaction
Ice Flow automatically manages the lifecycle of its operational state to prevent unbounded storage growth. Different categories of data have different retention policies.
Retention Policies
| Data | Retention | Detail |
|---|---|---|
| Configuration (catalogs, scopes, warehouses, mappings) | Permanent | No TTL — retained until explicitly deleted |
| Checkpoints (replication state, last event, statistics) | Permanent | Only the latest value per key is kept (no revision history) |
| Monitoring events | 2 days | Individual events expire after 48 hours |
| Operations | 7 days | Completed operation records expire after 7 days |
| File transfers | 1 day | Individual file transfer records expire after 24 hours |
| Monitoring summaries | 1 year | Compacted daily summaries of monitoring events |
| Operations summaries | 1 year | Compacted daily summaries of completed operations |
How Compaction Works
To preserve long-term visibility, Ice Flow runs a compaction service every hour that aggregates detailed records into daily summaries before they expire.
The result:
- Recent data (last 1-7 days depending on type) is available at full detail, with individual events, operations, and file transfers visible in the UI
- Historical data (up to 1 year) is available as daily summaries that capture aggregate counts and totals — event counts, records added, files transferred, bytes transferred, and other aggregate metrics
Compaction Timing
Compaction cutoffs are set well inside the TTL windows to ensure data is always summarised before it expires:
| Data type | Compacted after | TTL expires after |
|---|---|---|
| Monitoring events | 24 hours | 48 hours |
| Operations | 48 hours | 7 days |
This overlap ensures no data is lost between the compaction window and the TTL expiry.
What Summaries Contain
Daily summaries aggregate the following per day:
Monitoring summaries:
- Total number of events detected
- Records added and removed
- Files and bytes observed
Operations summaries:
- Number of operations completed
- Total data files committed
- Total bytes transferred
- Total duration
No Configuration Required
Compaction runs automatically with no user configuration. The retention periods and compaction intervals are fixed. If you see older events or operations disappearing from the detailed views, this is expected behaviour — the data has been compacted into the daily summaries visible in the historical view.