Skip to content

[long-run] Enforce deterministic per-owner retention budgets and pressure actions #217

Description

@chaoz23

Parent: #215. Depends on #216. Related: #112.

Product outcome

Prevent any one retained structure from consuming an unbounded share of a long run while preserving critical evidence and honest exact-search semantics.

Contract

  • Give pending/active frontier payload, exact dedupe, ancestry, semantic indexes, frontier references/node slots, and findings explicit count/byte ceilings plus one total retained watermark and checkpoint/finalization reserve.
  • Let owners borrow unused soft budget deterministically, but never cross the total hard envelope.
  • In exact mode, apply pressure in this order: compact stale references/tombstones, remove duplicate payload, dense-rebase live node IDs at an epoch boundary, checkpoint, spill through an exact checksummed store, or finish with retention_ceiling.
  • Never silently evict exact dedupe/frontier state or use an approximate membership result as proof.
  • Persist every pressure action, owner, threshold, bytes/counts reclaimed, policy version, and result.
  • Stream/persist critical evidence before compacting in-memory detail; retain stable replay coordinates.

Acceptance criteria

  • Every previously unbounded owner has an explicit configured hard ceiling and deterministic response.
  • Same source/config/seed/policy/event ledger makes the same structural pressure decisions.
  • A bound owner cannot starve checkpoint/finalization reserve or cause a hard OOM.
  • Exact mode either preserves split-run equivalence or stops partial with no completeness claim.
  • Critical findings and replay witnesses survive every compact/spill/stop path.
  • Tests cover low-dedup wide state, deep ancestry, large semantic indexes, many findings, stale frontier references, multiple simultaneous soft limits, no-reclaim pressure, and finalization near the cap.
  • Matched evaluation reports bytes reclaimed, action cost, early discovery, and post-discovery yield separately.

Non-goals

  • A universal fixed percentage allocation before evidence.
  • Lossy/yield-aware eviction.
  • Raising the default heap ceiling.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions