Stream framed checkpoints from the shared engine

.gitattributes

@@ -0,0 +1,3 @@
+benchmarks/authored/**/*.ink text eol=lf
+benchmarks/authored/**/LICENSE text eol=lf
+benchmarks/authored/**/LICENSE*.md text eol=lf

CHANGELOG.md

@@ -2,7 +2,8 @@
 
 ## Unreleased
 
-- Add an opt-in framed checkpoint artifact-v2 codec around the unchanged logical shared-checkpoint schema v1. Ordered configuration, scheduler, frontier, witness-ancestry, dedupe, semantic-index, finding, and metadata records are independently bounded and compressed; exact frame/component checksums and a terminal index support incremental readback without a whole stored artifact, decompressed artifact, or raw artifact string. Cross-format writers share one neutral no-clobber commit point, so a stable ID reuses exactly one verified `.json.gz` or `.inkcp` layout under the existing crash-recovery and retention protocol. Observed-before-work, preflight-without-output, and successfully encoded no-clobber losers reuse the winner; a private candidate that emitted bytes and then failed preserves its error for a later retry. Stable checkpoint IDs, exact resume state, default gzip writes, default CLI/MCP search behavior, search order, and resource policy are unchanged; storage listing and search-session disk accounting now recognize, reuse, and attribute either verified layout. This is a codec foundation only: framed v2 remains an explicit library opt-in pending Intercept plus second-family promotion evidence, and it does not complete #156, activate checkpoint/epoch/pressure policy, restart InkBench, or publish a release.
+- Add an opt-in engine-native producer for framed checkpoint v2. A shared-search engine can remain quiescent at its exact resumable boundary while a callback-scoped, repeatable source streams the unchanged canonical schema-v1 identity and ordered frame records; the source yields detached values, snapshots mutable caller inputs, prevents evidence callbacks from aliasing retained findings, invalidates at synchronous or awaited callback settlement, and never constructs or recursively clones a complete checkpoint graph. Write receipts distinguish zero materialized graphs, one identity pass, and separate zero-or-one candidate-encoding and reuse-verification record passes. Same-ID framed reuse bounded-stream-compares canonical records with the live source instead of trusting a recomputed manifest/payload pair; legacy reuse and sidecar-free retention inventory fail closed on this graph-free route while remaining available to the graph API. Framed read receipts expose their actual effective codec limits, including cumulative decoded bytes independently of the legacy whole-value string ceiling, and restore structurally clones the validated graph instead of recreating one artifact-sized JSON string, while stored, frame, record, depth, and runtime bounds remain enforced. Windows recovery-slot cleaning treats ordinary `EPERM`/`EBUSY`/`EACCES` namespace contention as failure to acquire—not ownership—without weakening durable claim or manifest writes. Existing synchronous and graph APIs, stable IDs, default legacy-v1 CLI/MCP writes, search order, and resource policy remain unchanged. The checked six-cell evaluation on candidate `9312811` records exact Heresy II 100K→150K and Intercept 600K→650K framed resumes, zero materialized checkpoint graphs on both framed writes, and the matched Intercept legacy decompression limit at 536,870,888 bytes without artifact mutation. This is still a partial #156 foundation: the evidence does not promote a default, stream provisional read records into a live engine, restart InkBench, establish portable performance or memory improvement, or publish a release.
+- Add an opt-in framed checkpoint artifact-v2 codec around the unchanged logical shared-checkpoint schema v1. Ordered configuration, scheduler, frontier, witness-ancestry, dedupe, semantic-index, finding, and metadata records are independently bounded and compressed; exact frame/component checksums and a terminal index support incremental readback without a whole stored artifact, decompressed artifact, or raw artifact string. Cross-format writers share one neutral no-clobber commit point, so a stable ID reuses exactly one verified `.json.gz` or `.inkcp` layout under the existing crash-recovery and retention protocol. Observed-before-work, preflight-without-output, and successfully encoded no-clobber losers reuse the winner; a private candidate that emitted bytes and then failed preserves its error for a later retry. Stable checkpoint IDs, exact resume state, default gzip writes, default CLI/MCP search behavior, search order, and resource policy are unchanged; storage listing and search-session disk accounting now recognize, reuse, and attribute either verified layout. This is a codec foundation only: the later engine-native evaluation supplies its declared Intercept plus second-family functional evidence, but framed v2 remains an explicit library opt-in and the work does not complete #156, activate checkpoint/epoch/pressure policy, restart InkBench, or publish a release.
 - Extend the partial owner ledger across checkpoint reopen and report enrichment/finalization without adding another payload read, decode, parse, or stable-ID traversal. Successful checkpoint opens and resumes expose bounded stage-by-stage receipts, while typed corrupt/unsupported/resource-limit failures retain the work reached before failure. The opt-in accounted report builder measures the source exploration graph, finding-identity strings already materialized by enrichment, and the returned report; report saves separately distinguish created envelope graphs from reuse-parsed artifact graphs. These compact-JSON serialized-view proxies and their conservative potential sums are not exclusive retained-owner bytes or observed heap peaks. Runtime receipts remain outside saved report/checkpoint payloads and IDs, and bounded JSON streaming still avoids the monolithic report graph. This adds no checkpoint v2/framed readback, allocation, eviction, compaction, stopping policy, benchmark restart, or release; #156 and #216 remain open.
 - Add versioned retained-owner and finalization receipts without changing search policy or the checksum-bound observability-v1/v2 shapes. Shared pass telemetry now accounts for the bounded observability ledger through a non-recursive canonical UTF-8 projection and preserves its known high-water state across resume; older checkpoints remain readable and explicitly report incomplete owner history. Checkpoint saves report logical graph bytes, streamed artifact/chunk bytes, compressed output, configured compression capacity, durable payload/manifest bytes, and whether encode/compress work was applied or an existing stable ID was reused. Report saves similarly expose canonical identity-string, artifact-serialization, conservative potential string, and durable bytes outside the content-derived report. Configured finalization headroom is labeled unallocated capacity and remains excluded from retained/process totals. Stable-ID algorithms are unchanged: deterministic owner state participates in newly emitted checkpoint content, while post-write receipts never feed back into the IDs or payloads they measure. These fields are observational/accounting inputs only: they do not emit the reserved checkpoint/epoch/pressure reasons, change frontier order, activate a budget or stopping policy, add checkpoint v2, or publish a release; #156 and #216 remain open.
 - Extend shared-search observability to nested schema v2 with monotonic sample sequences, canonical coalesced reason vectors, an exact 15-bit trigger mask, a fixed eight-count boundary-local trigger-yield tuple, category/frontier/cadence/termination triggers, and category-specific first/last discovery, current/longest dry-state, and identities-per-million-transition facts. The redundant vector/mask/tuple preserves exact retained-boundary triggers across compaction while aggregate interval deltas are rebuilt: adjacent complete-history samples require the tuple to equal the semantic interval delta, while genuine compaction gaps or incomplete v1 history allow only a componentwise-bounded tuple. Observability-v1 migration synthesizes masks only from explicit historical boundaries and uses a zero tuple to mean no v1 trigger was recorded, not that an interval was event-free. Sequence validation applies monotonic and feasible record/state/cadence bounds without claiming exact replay of discarded compaction history. Emitted v2 checkpoints add a domain-separated, key-order-independent canonical SHA-256 self-check binding every fixed known ledger field; it detects stale or accidental mutation but is not authentication because a writer can recompute it, and persisted payload/manifest digests remain the external artifact-corruption boundary. V1 and pre-ledger inputs gain the nested checksum on their first emitted v2 checkpoint. Live process sampling and progress output remain bounded to cadence and final termination while the compacted deterministic ledger retains event boundaries. Checkpoint construction remains side-effect-free and deterministic: the persisted ledger participates in stable checkpoint identity, while observability-v1 checkpoints migrate explicitly with incomplete event history and nullable facts rather than invented history. This remains an independent partial #216 slice: `checkpoint`, `epoch`, and generic `pressure` reasons are reserved but not emitted; retained-owner accounting remains incomplete; no resource, allocation, or stopping policy is activated; #216 remains open; and the earlier V1 overhead study is not an exact-head overhead claim for this V2 implementation.

benchmarks/checkpoint-v2-promotion-v1.json

@@ -0,0 +1,125 @@
+{
+  "schemaVersion": 1,
+  "kind": "checkpoint_v2_promotion_evaluation",
+  "id": "checkpoint-v2-promotion-v1",
+  "compiler": {
+    "name": "inklecate",
+    "version": "1.2.1"
+  },
+  "controls": {
+    "cellOrder": [
+      "heresy2-legacy-split",
+      "heresy2-framed-split",
+      "heresy2-uninterrupted",
+      "intercept-uninterrupted",
+      "intercept-framed-split",
+      "intercept-legacy-split"
+    ],
+    "maxDepth": 100,
+    "searchSeed": 7,
+    "storySeed": 1,
+    "concurrency": 1,
+    "minimizeRepros": false,
+    "stateSensitivity": "source_semantics",
+    "loopRiskDetection": "only_without_turns_randomness_visit_counts_or_externals",
+    "maxMemoryMb": 4096,
+    "maxTimeMs": 840000,
+    "workerTimeoutMs": 900000,
+    "coordinatorMaxOldSpaceMb": 6144,
+    "storage": {
+      "maxCheckpointBytes": 536870912,
+      "maxProjectBytes": 2147483648,
+      "framedV2": {
+        "maxTotalDecodedBytes": 2147483648
+      }
+    }
+  },
+  "sources": [
+    {
+      "id": "heresy2",
+      "story": "authored/heresy2/heresy2.ink",
+      "entrypointSha256": "49db768479e523f39ae9eb838f6daa0798244b7bbe266d686a7a967ccf66f7ea",
+      "compiledStorySha256": "5c65cf077478ad70985341e564695cb8ab35af20ec574d12b517d1e18f9a4bb7",
+      "closure": {
+        "includeCount": 20,
+        "fileCount": 21,
+        "bundleSha256": "2991d13fbf2d036c9bfb5cf4207c16509dfdf77dae24fa410ace1a0d3a36b626"
+      },
+      "provenance": {
+        "license": "CC-BY-4.0",
+        "licenseFile": "authored/heresy2/LICENSE",
+        "licenseSha256": "7e7170e3cebf88a9f60c7b8421418323c09304da1af4d5e90f4da1dc1c8a2661",
+        "upstream": "randall-frank/heresy2-assets",
+        "commit": "37b8a7804217bb40a9f69f6fd9c173f2017d550e"
+      }
+    },
+    {
+      "id": "intercept",
+      "story": "authored/the-intercept/TheIntercept.ink",
+      "entrypointSha256": "1d7b5653b689a9da6801bbfaaa3fbd20b0cc47113b28bb1b1774f6cd11e9eea4",
+      "compiledStorySha256": "2a441a97a345cfc7d3c944e758949d6c3b37ccfe88a75ddad06cf3ac83a637c6",
+      "closure": {
+        "includeCount": 0,
+        "fileCount": 1,
+        "bundleSha256": "c9865a7d2d334e39b68f788f790d202a7c1fbd9acb2bf2fc54fa1a4d549db6c9"
+      },
+      "provenance": {
+        "license": "MIT",
+        "licenseFile": "authored/the-intercept/LICENSE-AND-PROVENANCE.md",
+        "licenseSha256": "4b0d7e6b2ccac7794d0705ee40a6728d664ff1cd80994b5d31077a4cde38f3da",
+        "upstream": "inkle/the-intercept",
+        "commit": "2a816b56e61ce4bf02bec1c638074645bdd871e3"
+      }
+    }
+  ],
+  "cells": [
+    {
+      "id": "heresy2-legacy-split",
+      "sourceId": "heresy2",
+      "mode": "legacy-split",
+      "baseStates": 100000,
+      "targetStates": 150000,
+      "expected": "resume_exact"
+    },
+    {
+      "id": "heresy2-framed-split",
+      "sourceId": "heresy2",
+      "mode": "framed-split",
+      "baseStates": 100000,
+      "targetStates": 150000,
+      "expected": "resume_exact"
+    },
+    {
+      "id": "heresy2-uninterrupted",
+      "sourceId": "heresy2",
+      "mode": "uninterrupted",
+      "baseStates": 100000,
+      "targetStates": 150000,
+      "expected": "endpoint"
+    },
+    {
+      "id": "intercept-uninterrupted",
+      "sourceId": "intercept",
+      "mode": "uninterrupted",
+      "baseStates": 600000,
+      "targetStates": 650000,
+      "expected": "endpoint"
+    },
+    {
+      "id": "intercept-framed-split",
+      "sourceId": "intercept",
+      "mode": "framed-split",
+      "baseStates": 600000,
+      "targetStates": 650000,
+      "expected": "resume_exact"
+    },
+    {
+      "id": "intercept-legacy-split",
+      "sourceId": "intercept",
+      "mode": "legacy-split",
+      "baseStates": 600000,
+      "targetStates": 650000,
+      "expected": "legacy_readback_resource_limit"
+    }
+  ]
+}