Operations: configurable retention and safe pruning for run state, logs, cache, previews, rejects, and temporary artifacts

[LouisDeconinck]

Problem

A low-cost local-first platform must bound its operational metadata and intermediate storage.

Long-running Duckle servers can accumulate:

run history
run-scoped logs
async run records
preview/sample data
temporary/spill files left after abnormal termination
content-addressed stage cache
queued-work ledgers
reject/dead-letter data
compiled/built releases
local artifact copies

DuckLake table maintenance solves a different problem; it does not bound Duckle’s own control-plane and cache storage.

Proposed solution

Add explicit retention policies per category.

Example:

retention:
  runs:
    keepDays: 180
    keepLastPerPipeline: 50

  logs
## Scheduling

Allow retention as a normal low-priority maintenance task using resource pools.

## Policy

An authoritative server policy can enforce:

```text
minimum audit retention
maximum cache size
whether reject data may be deleted

Observability

Expose:

bytes by category
oldest retained item
last prune result
prune failures

Scope

This is not object-storage lifecycle management or DuckLake snapshot expiration.

It is bounded housekeeping for data Duckle itself owns.

Acceptance criteria

1. --dry-run accurately lists what would be deleted;
2. active/referenced data is protected;
3. cache/log/history growth can be bounded by age and/or size;
4. interrupted temporary files can be reclaimed safely;
5. pruning is auditable.:
   keepDays: 30

cache:
maxBytes: 500GB
keepDays: 45

previews:
keepHours: 24

interruptedTemp:
keepHours: 24

releases:
keepLast: 10

Reject/dead-letter and audit data should have conservative separate defaults.

## Commands

```bash
duckle retention status --json
duckle retention prune --dry-run
duckle retention prune

The dry run should report item counts and reclaimable bytes per category.

Safety

Never remove:

active-run files
queued-run state
artifacts referenced by retained manifests/releases
the active release
a cache entry currently being read/written
audit records unless an explicit policy permits it
user-owned source/raw/output data

Distinguish Duckle-managed files from arbitrary paths referenced by a pipeline.

Crash recovery

On startup or maintenance, detect abandoned run-scoped temp directories and incomplete cache writes using ownership metadata/leases rather than deleting files solely by name.

Scheduling

Allow retention as a normal low-priority maintenance task using resource pools.

Policy

An authoritative server policy can enforce:

minimum audit retention
maximum cache size
whether reject data may be deleted

Observability

Expose:

bytes by category
oldest retained item
last prune result
prune failures

Scope

This is not object-storage lifecycle management or DuckLake snapshot expiration.

It is bounded housekeeping for data Duckle itself owns.

Acceptance criteria

1. --dry-run accurately lists what would be deleted;
2. active/referenced data is protected;
3. cache/log/history growth can be bounded by age and/or size;
4. interrupted temporary files can be reclaimed safely;
5. pruning is auditable.

[SouravRoy-ETL]

Shipped on main (7c5036d): duckle-runner retention status|prune.

category          files        bytes  oldest
cache               412   1204338112  61d
logs                 38      1102944  90d
runs                  6        49122  12d
receipts             57        27991  12d

duckle-runner retention prune --cache-days 45 --logs-days 30 --receipts-keep 50 --dry-run

Opt-in per category. A bare prune with no limits removes nothing. Housekeeping that deletes by default is how a workspace loses something nobody meant to lose, and the default has to be the boring one.

AC2, and the reason this is one module rather than four ad-hoc deletions: .duckle/ is never touched. It holds the workspace encryption key, saved watermarks and resume positions, known host keys, alert state and the accepted XSD contracts. Losing a watermark does not lose history - it silently re-ingests or skips data, which is a correctness problem wearing a housekeeping costume. The prunable-root check runs when the plan is built and again when it is applied, so a caller that hands apply() something else still cannot delete state. There is a test that asks it to delete the workspace key directly and asserts the file survives.

The audit log is excluded from age pruning, because it is the record of the prune's own deletions. A retention policy that erased its own evidence would be the wrong shape of correct. Every prune appends to it (AC5):

{"action":"retention.prune","outcome":"allowed","detail":"removed 1 files, 491 bytes"}

AC1 is structural rather than careful: --dry-run and a real prune call the same plan() and differ only in whether the deletion runs. A dry run that computed its own list would eventually disagree with the prune, which is the one thing a dry run must never do.

Not in this slice:

• Size-based caps (maxBytes: 500GB). Age and count are in; bytes needs an eviction ORDER to be meaningful - least-recently-used is the obvious one and the cache does not record use, only mtime. Doable, but it is a cache-accounting change rather than a retention flag.
• Reject/dead-letter data. You said conservative separate defaults and I agree, but reject data is pipeline output living at a configured path rather than in a directory Duckle owns, so pruning it means understanding sink configuration. Different problem.
• Interrupted temp reclamation. Temp dirs are per-run and guarded, so the leak only happens on abnormal termination; identifying an orphan safely means telling it from a run still in progress, which wants the run addressability from Runs: retry a failed run from a safe node/checkpoint without repeating successful side effects #305 rather than a timestamp heuristic.
• Retention as a scheduled maintenance task, and the server-policy floor (minimum audit retention, whether reject data may be deleted at all). Both want the policy layer rather than the CLI.

Practical note: this got written the day a 475 GB disk here hit 100% with 77 GB of Rust build artifacts, so the problem is real even at one-developer scale.

[LouisDeconinck]

There is some new context for this issue from the features that landed after the first retention implementation.

The initial safety rule:

never touch .duckle/

was absolutely the right default when .duckle/ mostly contained correctness-critical state such as:

watermarks
resume positions
keys
XSD contracts
alert state

But Duckle now also has growing operational ledgers there, notably:

.duckle/materializations.ndjson
.duckle/deliveries.json

and the slice/backfill/chunk machinery has added more durable execution state elsewhere.

I think #303 should eventually distinguish:

correctness state
    -> generally never age-pruned automatically

operational history / ledgers
    -> explicitly retainable/prunable

The materialization/subscription case needs one semantic rule before pruning:

A subscription can only replay publication events still inside the configured retention horizon.

That should be explicit rather than an accidental consequence of deleting old events.

For example:

retention:
  materializations:
    keepDays: 180

  deliveries:
    keepDays: 180

with safety rules approximately like:

pending delivery                  -> never prune
failed/retryable delivery         -> never prune by ordinary age rule
event referenced by retained delivery -> protect
old delivered event               -> eligible after horizon

A newly created subscription would then deterministically mean:

replay matching retained materializations

rather than implicitly "replay forever".

I would also revisit interrupted-temp reclamation now that #259 provides durable run identity/reconciliation. We no longer need to infer liveness from file age alone; the run state can help establish that the owner is actually gone.

Likewise, active backfill/chunk slices should protect their parts/working files, while completed historical ledgers can have explicit retention.

This seems increasingly important for Duckle's single-machine model: the more durable the control plane becomes, the more important it is that durability does not mean unbounded disk growth.

[LouisDeconinck]

#325 surfaced an important new retention dependency that I think belongs here.

The materialization event obtains partition provenance from the producer run's receipt.

That means this can now happen:

producer run
    ↓
receipt contains partition = 2026-09-04
    ↓
materialization event retained
    ↓
retention prunes producer receipt
    ↓
event is later reconstructed/read without partition
    ↓
subscription using ${event.partition} can no longer resolve

So receipt retention can no longer be considered independently from the newer materialization/delivery state.

I think the general retention invariant should become:

Never prune an object that is still needed to interpret a retained durable object.

For the current model, that suggests protecting at least:

receipt referenced by retained materialization
receipt referenced by retained delivery
receipt needed by retry provenance
active release
active/queued run state
active slice/chunk/backfill artifacts

There are two ways to handle the specific materialization case.

Option A — make the event self-contained

When creating the event, copy all event-level provenance needed for future delivery:

partition_key
release_id
producer run_id
parameter-relevant metadata

into the materialization event.

Then the receipt can be pruned independently once nothing else references it.

This is probably my preference because an event should remain understandable throughout its own retention horizon.

Option B — reference-aware receipt retention

Keep the current lookup model, but retention must refuse to prune a receipt while a retained materialization/delivery refers to it.

This is safe but creates a longer reference graph for the retention planner.

Either way, I would add a regression test along the lines of:

publish partitioned run
-> create materialization
-> apply configured receipt retention
-> process/replay subscription
-> event.partition is still available

More broadly, I think #303 should now distinguish:

correctness state
operational history
durable event/delivery history

and prune the latter two with explicit reference-aware retention rather than relying only on directory boundaries.

The newer control-plane features make this more important than when the first retention implementation landed.

[SouravRoy-ETL]

Done, and the receipt/materialization interaction you raised at 07:27 was a live bug rather than a hazard. f28a830.

The bug

Sharper than "the event is later read without a partition". event_id is hashed over pipeline + run + commit time + release, and the release is read from the producer's receipt. So:

retention prunes the receipt
    -> the same publication hashes to a DIFFERENT id
    -> reconcile does not recognise the event already on disk
    -> it appends a second one, with no partition and no release
    -> every subscription delivers that publication twice

Reconcile now recognises a publication by what it is - pipeline, run, commit time - all of which the event carries itself, so it cannot drift with what retention has removed.

Worth saying: my first attempt to reproduce this was contaminated. append_run_record already emits the event, and my test appended a second one itself, so the first "two events" I saw was my own setup. The bug is real - reverting the fix brings back Event { ..., release_id: None, partition_key: None } as a duplicate - but I nearly reported a test artifact as a defect.

Option A was already half true

The event has carried partitionKey, releaseId and runId since #325, copied at publication time. So an event is self-contained for reading; what was not safe was reconstructing one. With that fixed, your regression passes literally: publish a partitioned run, prune every receipt (--receipts-keep 0), and ${event.partition} still resolves.

I did Option B as well, because they are not alternatives. A receipt named by a retained event or delivery survives the keep-count however old it is - not because the event needs it to be read, but because "which run produced this" must stay answerable for a record we are deliberately still keeping. That is your invariant applied to the one reference that exists today.

The ledgers

duckle-runner retention prune --materializations-days 180 --deliveries-days 180

Pruned by rewriting, never by deleting: one file holds the records inside the horizon and the ones outside it, and remove_file is not a retention policy. The distinction you asked for is enforced by shape rather than by care - plan/apply delete files and still cannot reach .duckle/ at all, while plan_ledgers/apply_ledgers rewrite two named ledgers and delete nothing.

Your safety rules, each with a test:

pending delivery                      -> never
failed delivery                       -> never BY AGE; it is retryable
event a retained delivery refers to   -> protected
delivered and past the horizon        -> history, and eligible

Plus one you did not list and I think belongs: a record nobody can date is kept. An empty or unparseable timestamp is not evidence that something is old.

And the semantic is stated rather than left to be an emergent property:

A subscription can only replay publication events still inside the configured retention horizon.

Live, on a real publication:

$ retention prune --materializations-days 1 --deliveries-days 1     # delivery PENDING
deliveries          0 record(s), 1 kept
materializations    0 record(s), 1 kept

$ ...                                                              # same, now DELIVERED
removed 1 deliveries record(s)
removed 1 materializations record(s)

$ audit
retention.prune.ledgers | removed 1 deliveries, 1 materializations

1886 tests pass.

Still not done, and I would rather say so than imply otherwise

• Interrupted-temp reclamation. You are right that Feature: Durable asynchronous server runs with run IDs, status, logs and cancellation #259 changes the picture - a temp file's owning run is now addressable, so liveness need not be inferred from age. I did not build it here; it is a separate piece from ledger retention and I would rather do it as its own slice than bolt it on.
• Active slice/chunk/backfill artifacts. Not yet protected, and they should be by the same invariant. A chunk's staged parts live under the workspace, and a running backfill's ledger names them. That is the next reference edge, and it is the same shape as the receipt one.
• Size caps, still. Bytes needs an eviction ORDER, and the cache records mtime rather than use.

The general form of the invariant now has three edges implemented (delivery -> event, event -> receipt, delivery -> receipt) and at least one known missing (backfill -> parts). If you would rather I generalise it into a reference graph the planner walks, rather than three hand-written protections, say so - I stopped at three because that is what exists today and a graph with one edge type is a framework in search of a second.

[LouisDeconinck]

a55153a introduces another retention invariant that I think should be captured here.

Freshness now correctly consults the materialization log because rolling run history is not durable enough for low-frequency assets produced by high-frequency pipelines.

That creates a direct relationship between:

freshness SLA horizon
and
materialization retention horizon

For example:

asset:
  freshness:
    maximumAge: 90d

retention:
  materializations:
    keepDays: 30

Consider:

last successful materialization = 45 days ago
maximumAge                     = 90 days

The asset is still legitimately fresh.

But if:

materialization event was pruned after 30 days
+
the producer has run >50 unrelated times since then

then neither the materialization log nor rolling run history necessarily retains evidence of that successful write.

Freshness could therefore regress to:

no known materialization
-> stale

45 days before its declared SLA actually expires.

I think the retention planner should protect publication evidence for as long as another retained contract still needs it.

For freshness, a useful invariant is:

Do not prune the latest successful materialization of an asset while that materialization can still satisfy the asset's configured freshness SLA.

Conceptually:

event is latest successful publication for asset
AND
event.committed_at + effective freshness horizon > now

-> protect event

For expectedAfterSchedule, the exact future deadline may be schedule-derived, so the general formulation might be:

freshness evaluator says this publication is still sufficient
-> retention must not remove the last evidence of it

An alternative is to persist a compact independent freshness checkpoint:

asset
last_successful_materialization
producer run/release

that survives event-history pruning.

Either design is fine; I would mainly avoid coupling correctness to an undocumented requirement such as:

materialization retention must always be longer than every freshness SLA

unless Duckle validates that explicitly.

A regression test could be:

asset maximumAge = 90d
last publication = 45d ago
>50 unrelated producer runs
materialization retention request = 30d

prune
-> asset must remain fresh

This feels like the same general AC2 principle already emerging in #303:

A retained durable contract must not lose the state required to interpret it correctly.