Repository navigation
Compact shared checkpoints before frontier artifacts dominate campaigns #156
Description
Product outcome
Let a large-story campaign retain a useful broad-search frontier and then schedule specialists without checkpoint artifacts becoming the binding memory or disk cost long before the declared state budget.
Measured problem
The 2026-07-14 campaign-child evaluation on an 8 GiB Apple Silicon machine used inkle's MIT-licensed The Intercept, depth 100, seed 7, story seed 1, and Inkcheck's default 1,996,488,704-byte campaign memory ceiling.
- Independent 5M base grants stopped safely on memory after 1,213,503 and 1,484,991 states, so specialists were correctly blocked.
- Independent 500K bases completed all 500,000 states and found identical broad evidence (29 knots and 1,174 terminal states), but each checkpoint exceeded the 512 MiB single-artifact cap and stopped at
disk_ceiling. - A 100K base persisted successfully and admitted 5M-grant assertion and goal children.
- Before the evaluation hardening, building the larger checkpoint could throw
RangeError: Invalid string length; the evaluator PR converts this into a clean partial stop but does not make the frontier economical.
Machine evidence: benchmarks/results/campaign-child-evaluation-v1.json, campaign-child-recovery-500k-blocked-v1.json, and campaign-child-recovery-100k-v1.json.
Contract
- Compactness must be measured against the full durable artifact, not only pending-state payload.
- The encoding remains source/configuration bound, private, atomic, checksummed, versioned, and exact enough for split-run equivalence.
- A checkpoint limit is a resumability limit, not story-coverage proof and not a reason to discard the latest report.
- Compaction or spill must not change deterministic frontier order, finding identities, or witness replay.
- A specialist may not claim broad yield merely because compact replay makes a base frontier available.
Acceptance criteria
- A preregistered Intercept depth-100, seed-7 500K base persists below the 512 MiB single-checkpoint cap on the reference machine, or a documented smaller envelope is justified by a matched value/cost study.
- Checkpoint size reports component bytes for pending states, ancestry, dedupe, semantic indexes, frontier views, findings, and serialization overhead.
- Serialization never constructs an unbounded duplicate JSON string before checking the artifact ceiling.
- Oversized or unstringifiable checkpoints retain an immutable partial report and return an explicit checkpoint/disk binding reason without throwing.
- Resume remains evidence-identical to one uninterrupted fixed-state run for the same source/config/seed.
- Tests cover low-dedup wide state, deep ancestry, large semantic indexes, source changes, cancellation, corruption, and retention pruning.
- Evaluation includes at least one additional public or consent-safe story family so the representation is not tuned only to The Intercept.
Non-goal
This does not raise default memory/disk ceilings or imply that a larger resumable frontier is always worth exploring. #93 still decides whether another window or specialist earns budget.
Activity
chaoz23 commentedon Jul 15, 2026
PR #158 implements the first measured compact-storage slice without raising either checkpoint ceiling.
Matched 2026-07-15 The Intercept gate (500K states, depth 100, search seed 7, story seed 1):
disk_ceiling; no checkpointThe selected fast encoding preserves 1,174 terminal states, 29 visited knots, and stable checkpoint ID
checkpoint-b2e41a8eadaae28907c07835. The campaign remains active withmaxStatesas the binding limit. It leaves roughly 9.5x headroom below 512 MiB, so raising the cap to 1 GiB is not justified by this cell.The PR also streams serialization without one artifact-sized duplicate string, retains legacy
.jsonreads/reuse, enforces ceilings against actual compressed bytes, and passes 211/211 tests.Remaining #156 acceptance work: component-level durable-byte attribution, adversarial wide/deep/index/corruption/cancellation/retention cells, and at least one second public or consent-safe story family.
chaoz23 commentedon Jul 16, 2026
#178 authored preflight found a new readback boundary on 2026-07-16.\n\nOn the same 8 GiB Apple Silicon machine, depth 100 / search seed 7 / story seed 1 / 4 GiB campaign ceiling:\n\n- a fresh 600,000-state The Intercept shared base completed exploration and wrote a compressed checkpoint;\n- the immediate retention/readback pass failed with while reopening that brand-new artifact;\n- the evaluator used a fresh temporary project, so this was not stale prior data;\n- the already-qualified 500,000-state base saved, reopened, hash-checked, and supported nine child windows successfully.\n\nThe current catch collapses gzip corruption, decode/output-size failures, and possibly parse-memory failures into one message. Follow-up here should distinguish those causes, stream or bound decompression/readback, and retain the partial report without calling a valid-but-too-large checkpoint corrupt. The successful matched #178 evaluation therefore pins a 500K bootstrap rather than raising limits.
chaoz23 commentedon Aug 25, 2026
Progress map — 2026-08-25
Merged foundations now on
main@52e1721:Acceptance remains partial. The current artifact is still one schema-v1 JSON/gzip value and reopen still materializes the whole decompressed/string/parsed graph. Remaining work includes a framed/checksummed artifact+manifest v2 with incremental bounded decode, v1 compatibility, stable logical IDs across physical layouts, prior-good crash/cancellation behavior, the adversarial wide/deep/index/finding matrix, exact split-run evidence, and a second public story family.
The next implementation branch is
agent/156-checkpoint-v2-foundation, based directly on52e1721. It will keep v2 opt-in until the Intercept and Heresy II gates support promotion. This issue remains open.