Bound checkpoint readback and metadata listing

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased
 
+- Bound schema-v1 checkpoint readback by stored and decompressed bytes, classify reopen failures as corrupt, unsupported, or resource-limited, and add canonically self-bound private manifests so listing and retention use bounded metadata I/O without opening new frontiers. Full payload digests remain an open/resume boundary; no-clobber same-ID publication, payload-first crash recovery, and pair-inclusive quotas preserve existing v1 JSON/gzip reads, stable IDs, and exact resume pending framed schema v2.
 - Add an opt-in bounded NDJSON evidence stream for marathon-scale external consumers. It emits replayable numeric ending/runtime witnesses with global elapsed timestamps as they are retained, then a compact terminal summary without constructing the monolithic full-report JSON string.
 - Treat `--max-time` as a total CLI deadline and retain bounded time/heap headroom for clean report finalization. Explicit heap caps now expose the lower search watermark separately from the full process envelope.
 

docs/local-checkpoints.md

@@ -28,27 +28,47 @@ inkcheck checkpoints list --json
 inkcheck checkpoints show checkpoint-0123456789abcdef01234567 --json
 ```
 
-These commands return bounded metadata, not the frontier payload. `show` recompiles the project and reports:
+These commands return metadata, not the frontier payload. New saves also write a private, at-most-64-KiB `checkpoint-<hash>.meta.json` sidecar. `list` and retention read only that canonical self-checksummed manifest plus the payload's file size; they never open, hash, inflate, or JSON-parse a manifested frontier. Earlier schema-v1 `.json` and `.json.gz` artifacts without a sidecar still work, using the bounded full reader as a compatibility fallback.
+
+The manifest self-checksum binds every field, including the stored-payload digest, so an isolated edit to creation time, grant, state count, or another field fails before it can affect listing or retention. This detects accidental/local metadata corruption; it is not authentication against someone with write access who deliberately rewrites both the manifest and its checksum. `open` and resume are the full-integrity boundary: they validate the manifest, hash the complete stored payload and compare its digest, then decompress, parse, verify the stable logical ID/configuration, and check source freshness. `show` reports:
 
 - `current`: compiled story and knot/source map match the saved checkpoint.
 - `stale`: source exists but no longer matches or compiles.
 - `path_changed`: the project-relative entrypoint no longer exists.
 
 Resume requires `current`, the supported artifact and checkpoint schema versions, and every engine binding to match: story and knot hashes, depth, both seeds, hidden turn/random sensitivity, randomness detection, frontier envelopes, and external bindings. Corrupt content, metadata mismatch, unsupported versions, and a non-increasing grant fail closed.
 
+Library callers can distinguish three `CheckpointReadError.kind` values instead of parsing messages:
+
+- `corrupt`: invalid gzip/JSON, checksum or stable-ID mismatch, or malformed metadata;
+- `unsupported`: an artifact, manifest, or shared-checkpoint schema this Inkcheck cannot interpret;
+- `resource_limit`: stored or decompressed bytes exceed the bounded readback envelope. This does not label the checkpoint corrupt and never deletes or replaces it.
+
+`listCheckpointArtifacts`, `openCheckpointArtifact`, and `loadCheckpointForResume` accept optional `maxStoredBytes` and `maxDecompressedBytes` read limits. Defaults cap stored input at 512 MiB and schema-v1 decompression at the smaller of 512 MiB and the runtime's maximum string length. Listing uses those limits only for a sidecar-free compatibility fallback.
+
 ## Atomicity and retention
 
-New checkpoints live under `.inkcheck/checkpoints/checkpoint-<hash>.json.gz`. Their stable ID derives from the exact logical checkpoint content plus the project-relative entrypoint, not from compression bytes, so repeating the same deterministic boundary reuses one artifact. Existing schema-v1 `.json` artifacts remain readable and resumable.
+New checkpoints live under `.inkcheck/checkpoints/checkpoint-<hash>.json.gz`, with the bounded metadata sidecar beside them. Their stable ID derives from the exact logical checkpoint content plus the project-relative entrypoint, not from compression or manifest bytes, so repeating the same deterministic boundary reuses one artifact. Existing schema-v1 `.json` artifacts remain readable and resumable.
+
+The writer emits compact JSON through gzip directly into a private same-directory temporary file while computing the stored-byte digest in the same pass. It never constructs a second artifact-sized JSON string in memory, and it checks the final payload-plus-sidecar bytes before publication. The reader bounds stored input and gzip output before parsing and does not create a second decompressed string. Schema v1 is nevertheless still memory-heavy: the compressed buffer, decompressed buffer, one JSON string, and parsed frontier graph can overlap until garbage collection. A malformed gzip stream fails closed as storage corruption; a valid stream still passes the normal schema, stable-ID, source, and configuration checks after decompression.
 
-The writer emits compact JSON through gzip directly into the private same-directory temporary file. It never constructs a second artifact-sized JSON string in memory, and it enforces the single-artifact ceiling against bytes that would actually remain on disk. A malformed gzip stream fails closed as storage corruption; a valid stream still passes the normal schema, stable-ID, source, and configuration checks after decompression.
+Before exposing payload bytes, Inkcheck reserves one of 32 fixed, hidden recovery-manifest slots for that stable ID, writes the canonical at-most-64-KiB manifest there, flushes it, and directory-syncs the slot. A same-directory hard-link create is then the exclusive, no-clobber same-ID payload commit point on supported local filesystems, including ordinary NTFS/APFS/ext filesystems: concurrent losers reopen the winning bytes instead of overwriting them. The writer hashes the visible payload with a fixed-size buffer, selects only a recovery record whose size, digest, ID, and requested checkpoint summary match, and promotes that record to the canonical sidecar. Recovery therefore does not need to inflate or parse an artifact that exceeds the schema-v1 readback ceiling.
 
-Inkcheck writes a same-directory temporary file with mode `0600`, flushes it, atomically renames it, and removes temporary files on failure. Only after the new file is durable does retention remove older artifacts. Defaults are hard safety ceilings:
+The canonical sidecar is never pre-quarantined. A matching recovery record replaces an older orphan/corrupt sidecar only after this writer sees the published payload; POSIX uses atomic rename-over-existing, while the portable fallback retains bounded promotion and displaced companions so a crash in the replace gap can restore or complete the pair. Each slot has a nonce-bearing owner claim. Reservation and cleanup must first acquire the same fixed per-slot cleaning claim, which is removed last; a delayed cleaner therefore cannot delete a pathname after another writer has reused the slot. A transaction that returns before cleanup durably records that exact nonce as released, allowing a same-process or cross-process retry to reclaim it without confusing another live transaction for debris. Recovery records remain hidden from listing and retention. They are deleted only after the canonical manifest has been reread and matched against a second fixed-memory hash of the same visible payload; a crash before payload publication leaves ignored recovery-only metadata, and a crash after publication leaves enough metadata to finish without decoding. Sidecar reconstruction for legacy schema-v1 JSON also uses this fixed transaction namespace. The claim, release marker, cleaning claim, payload temporary, manifest, promotion, and displaced filenames are fixed and bounded per stable ID. A process that dies while holding a cleaning claim consumes that one slot until an operator removes it while no checkpoint save is active; it cannot expose or overwrite checkpoint evidence.
+
+This publication guarantee serializes writers for the same stable ID. Retention still validates the complete project set again after the pair is durable, but it is not a global transaction or recovery journal across simultaneous writers of different checkpoint IDs. Hidden files left by an abruptly terminated process are transaction debris, not listed checkpoint artifacts: they are excluded from retention and the project artifact byte ceiling, and the next successful save of that same stable ID cleans dead-owner slots. Repeated crashes across many distinct IDs can therefore leave additional hidden disk use; when no checkpoint save is active, an operator may remove those hidden recovery-slot files. Project-wide crash-debris accounting/recovery belongs with the framed checkpoint-v2 journal rather than this same-ID schema-v1 precursor. Only after the new pair is durable does retention remove older payloads and their sidecars. Defaults are hard safety ceilings for final checkpoint artifacts:
 
 - 512 MiB for one checkpoint;
 - 1 GiB across checkpoint artifacts in one project;
 - three generations per entrypoint.
 
-An individually oversized compressed checkpoint is rejected. Once a new generation is durable, oldest generations for that entrypoint are removed first, then the oldest project checkpoints if needed to satisfy the project byte ceiling. The saved generation is protected from that cleanup. `checkpoints list/show` reports `storageEncoding` and the actual durable `sizeBytes`; this is storage cost, not an estimate of process heap or future search value.
+An individually oversized payload-plus-sidecar pair is rejected. Once a new generation is durable, oldest generations for that entrypoint are removed first, then the oldest project checkpoints if needed to satisfy the project byte ceiling. The saved generation is protected from that cleanup. `checkpoints list/show` reports `payloadSizeBytes`, `metadataSizeBytes`, and their sum as the actual durable `sizeBytes`; this is storage cost, not an estimate of process heap or future search value.
+
+## Schema-v1 readback boundary
+
+This is the safe foundation for the observed 600,000-state boundary, not a claim that every such checkpoint can now resume. A gzip payload can be within the durable disk quota while its single logical JSON value is larger than V8 can represent. Inkcheck now returns `resource_limit` at that boundary, keeps the known-good bytes intact, and can still list/prune a manifested artifact without inflation. It does not misreport the file as corrupt or retry an unsafe allocation. A repeated same-ID save may recognize such an artifact only after its canonical manifest matches the requested checkpoint summary and its full stored-byte digest verifies; that preserves known bytes but does not claim the logical payload was decoded or resumable.
+
+Removing that format ceiling requires a framed artifact schema v2: independently bounded metadata and frontier frames, per-frame lengths/checksums, incremental decode into the resume structures, and a compatibility reader that leaves schema-v1 IDs and exact trajectories unchanged. Promotion should require split-run equality against uninterrupted search plus truncated-frame, oversized-frame, checksum, and mixed-v1/v2 retention tests.
 
 ## Privacy
 

docs/shared-checkpoint-schema-v1.md

@@ -44,8 +44,10 @@ See [local resumable checkpoints](local-checkpoints.md) for freshness, privacy,
 
 The logical checkpoint remains schema v1 JSON. New local artifacts stream that JSON through gzip as `.json.gz`; stable IDs still hash the same uncompressed logical checkpoint, and readers continue to accept earlier plain `.json` artifacts. Storage compression therefore changes neither deterministic frontier order nor split-run equivalence.
 
+New artifacts have a small canonical self-checksummed metadata sidecar, allowing list and retention operations to read bounded metadata plus payload file size without opening the frontier. That checksum detects isolated field corruption but is not authentication against a hostile local writer who can recompute it. Opening and resuming remain full-integrity boundaries: manifest validation and the full stored-payload digest precede bounded decompression, then the original stable-ID, envelope, configuration, and freshness checks run. Read failures explicitly distinguish corruption, unsupported schemas, and an artifact that cannot fit the schema-v1 readback envelope.
+
 ## Deliberate limits
 
-Schema v1 supports only base `shared:deep-novelty-v1`; assertions, goals, variable-aware steering, goal-aware steering, and the default portfolio are rejected rather than resumed approximately. Hosted/MCP resume, frontier partitioning, and cross-version migration remain future work.
+Schema v1 supports only base `shared:deep-novelty-v1`; assertions, goals, variable-aware steering, goal-aware steering, and the default portfolio are rejected rather than resumed approximately. Hosted/MCP resume, frontier partitioning, and cross-version migration remain future work. Because v1 is one JSON value, it also cannot safely reopen a payload above the runtime's maximum string length even when gzip keeps the stored file below quota. Its compressed buffer, decompressed buffer, JSON string, and parsed graph may overlap in memory. The reader reports an unsafe boundary as a resource limit and preserves the artifact; framed incremental schema v2 is required to remove both the string ceiling and this peak-memory shape.
 
 Checkpoint JSON can contain authored choice text, ending text, variable snapshots, serialized Ink runtime state, and exact witness paths. Treat it as sensitive project data and do not commit checkpoints by default.