Add bounded shared-search observability

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased
 
+- Add a bounded fixed-interval/termination `ResourceSampleV1` ledger for shared search with deterministic logical retention, category-specific interval yield, early-versus-post-first summaries, fail-closed exact-resume persistence, strict compact machine projection, privacy-safe live heap/RSS observations, and explicit monotonic run-wide positions across additive goal passes. This partial #216 slice changes no search policy, makes no complete owner-attribution claim, and leaves checkpoint/discovery/pressure-triggered sampling for later work.
 - Bound schema-v1 checkpoint readback by stored and decompressed bytes, classify reopen failures as corrupt, unsupported, or resource-limited, and add canonically self-bound private manifests so listing and retention use bounded metadata I/O without opening new frontiers. Full payload digests remain an open/resume boundary; no-clobber same-ID publication, payload-first crash recovery, and pair-inclusive quotas preserve existing v1 JSON/gzip reads, stable IDs, and exact resume pending framed schema v2.
 - Add an opt-in bounded NDJSON evidence stream for marathon-scale external consumers. It emits replayable numeric ending/runtime witnesses with global elapsed timestamps as they are retained, then a compact terminal summary without constructing the monolithic full-report JSON string.
 - Treat `--max-time` as a total CLI deadline and retain bounded time/heap headroom for clean report finalization. Explicit heap caps now expose the lower search watermark separately from the full process envelope.

README.md

@@ -134,7 +134,7 @@ For a new project containing one `.ink` file, `inkcheck init` creates this confi
 
 `--save-report` atomically stores a versioned report under `.inkcheck/reports/` and returns its stable content-and-entrypoint-derived ID. A later session can use `inkcheck artifacts list --json` and `inkcheck artifacts show <report-id> --json`; reopening reports whether the saved evidence is `current`, `stale`, or `path_changed` against the present entrypoint. Reports can contain story text, variables, and exact witnesses, so the agent kit ignores them by default. See [local report artifacts](docs/local-artifacts.md) for the trust, privacy, and compatibility contract.
 
-Long base-shared runs can also persist their exact live frontier locally. Start with `--search=shared --no-min-repro --save-checkpoint`, then continue later with `inkcheck resume <checkpoint-id> --max-states N`; `N` is the larger total grant, not extra hidden work. `inkcheck checkpoints list/show` reports bounded metadata, durable compressed size, storage encoding, and source freshness. New checkpoint files are streamed gzip artifacts; older plain schema-v1 JSON remains readable. Checkpoints are private, atomic, source/config-bound, ignored by default, and retention-capped; they may contain authored text and runtime state. See [local resumable checkpoints](docs/local-checkpoints.md). MCP agents can use the same exact foundation through durable [`start_search` / `inspect_search` / `continue_search` / `cancel_search` result windows](docs/mcp-search-sessions.md). They can also use `add_goal` for an explicit additive directed probe that starts from the story root and leaves that exact base frontier untouched. Portfolio, shared-variable, assertions, directed-frontier resume, and hosted jobs do not use this checkpoint contract yet.
+Long base-shared runs can also persist their exact live frontier locally. Start with `--search=shared --no-min-repro --save-checkpoint`, then continue later with `inkcheck resume <checkpoint-id> --max-states N`; `N` is the larger total grant, not extra hidden work. `inkcheck checkpoints list/show` reports bounded metadata, durable compressed size, storage encoding, and source freshness. New checkpoint files are streamed gzip artifacts; older plain schema-v1 JSON remains readable. Checkpoints are private, atomic, source/config-bound, ignored by default, and retention-capped; they may contain authored text and runtime state. See [local resumable checkpoints](docs/local-checkpoints.md). Shared passes also expose a bounded [resource/yield observability ledger](docs/shared-search-observability.md) that keeps deterministic logical accounting separate from live heap/RSS observations. MCP agents can use the same exact foundation through durable [`start_search` / `inspect_search` / `continue_search` / `cancel_search` result windows](docs/mcp-search-sessions.md). They can also use `add_goal` for an explicit additive directed probe that starts from the story root and leaves that exact base frontier untouched. Portfolio, shared-variable, assertions, directed-frontier resume, and hosted jobs do not use this checkpoint contract yet.
 
 ## Hosted checker
 

docs/progress-ndjson.md

@@ -16,7 +16,7 @@ Common fields:
 | --- | --- | --- |
 | `schemaVersion` | number | Progress schema version. Currently `1`. |
 | `sequence` | number | Monotonic event number starting at `1` for each CLI process. |
-| `type` | string | Event kind: `run_start`, `phase_start`, `progress`, `discovery`, `phase_end`, or `run_end`. |
+| `type` | string | Event kind: `run_start`, `phase_start`, `progress`, `discovery`, `resource`, `phase_end`, or `run_end`. |
 | `elapsedMs` | number | Milliseconds since the CLI run started. |
 | `statesExplored` | number | Total story states explored so far in this CLI process. |
 | `stateBudget` | number | Total configured work budget: baseline plus additional goal states. |
@@ -41,6 +41,7 @@ Optional fields:
 | `unvisitedKnots` | number | Knots not yet reached by any pass in this run. Non-increasing within a run. |
 | `knotsVisited` | number | Cumulative authored knots reached. Present on `discovery` events. |
 | `discoveries` | object | Numeric deltas first observed at this event: `endings`, `runtimeErrors`, `knotsVisited`, `visibleOutcomes`, `assertionViolations`, `goalsReached`, and `stagesReached`. Present only on `discovery` events. |
+| `sharedObservability` | object | Shared-search `resource` events only: one deterministic logical retention/yield sample paired with observed process heap/RSS. `runWideState` is the explicit outer-run position; nested `sample.state` remains local to its shared pass. See [shared-search observability](shared-search-observability.md). |
 | `status` | string | Terminal process status: `complete`, `cancelled`, or `error`. Hosted wrappers also use queue/job states. |
 | `stopReason` | string | Binding terminal reason such as `exhaustive`, `state_budget`, `depth_limit`, `time_limit`, `memory_limit`, `frontier_limit`, `worker_failure`, `compile_error`, `cancelled`, or `error`. |
 | `outcome` | string | Result classification separate from the stop cause: `clean`, `issues_found`, `review_required`, or `compile_error`. |
@@ -57,7 +58,7 @@ A normal complete run looks like this:
 2. `phase_start` for `compile`
 3. `phase_end` for `compile`
 4. `phase_start` / `phase_end` for source scanning and exploration phases as applicable
-5. zero or more `progress` activity events and `discovery` evidence events during exploration
+5. zero or more `progress` activity, `discovery` evidence, and shared-search `resource` events during exploration
 6. `phase_start` for `report`
 7. `phase_end` for `report`
 8. `run_end`
@@ -84,6 +85,12 @@ Privacy-safe discovery event:
 {"schemaVersion":1,"sequence":5,"type":"discovery","elapsedMs":611,"statesExplored":5200,"stateBudget":100000,"budgetFraction":0.052,"pass":"beam:w=64","endingsFound":4,"runtimeErrorsFound":1,"unvisitedKnots":7,"knotsVisited":12,"discoveries":{"endings":1,"runtimeErrors":1,"knotsVisited":2,"visibleOutcomes":1,"assertionViolations":0,"goalsReached":0,"stagesReached":0}}
 ```
 
+Shared-search resource event (abridged here; consumers should ignore unknown nested fields):
+
+```json
+{"schemaVersion":1,"sequence":6,"type":"resource","elapsedMs":702,"statesExplored":10000,"stateBudget":100000,"budgetFraction":0.1,"pass":"shared:deep-novelty-v1:seed=1","sharedObservability":{"schemaVersion":1,"pass":"shared:deep-novelty-v1:seed=1","runWideState":10000,"sample":{"schemaVersion":1,"boundary":"interval","state":10000,"retention":{"schemaVersion":1,"current":{"totalAccountedBytes":8388608}},"yield":{"schemaVersion":1,"fromStateExclusive":0,"throughState":10000,"delta":{"critical":{"runtimeErrors":0,"assertionViolations":0}}}},"process":{"schemaVersion":1,"scope":"process","heapUsedBytes":67108864,"heapTotalBytes":83886080,"rssBytes":104857600,"externalBytes":2097152,"arrayBuffersBytes":1048576,"comparedLogicalAccountedBytes":8388608,"unattributedBytes":58720256}}}
+```
+
 Terminal event:
 
 ```json
@@ -113,6 +120,8 @@ for await (const line of stderrLines) {
 
 `discovery` means that a cumulative run counter increased. It is useful for a concise terminal update, hosted status, or agent scheduling, but it is not a finding record and does not replace the final report. Counts stay privacy-safe by omitting identities, story labels, source locations, messages, paths, and variable data. A later bounded run can still find more.
 
+`resource` is emitted only by shared search at fixed transition boundaries and termination. Checkpoint operations, discovery events, and memory/frontier pressure do not add sampling boundaries in this partial #216 slice. `sample` contains deterministic aggregate counts and logical byte estimates. `process` contains nondeterministic Node process observations and must not participate in report identity, exact-resume comparison, frontier order, or a coverage claim. The outer `statesExplored` is CLI-process progress and equals the run base plus `sharedObservability.runWideState`. The nested `sample.state` always belongs to that shared pass. During additive goal work, Inkcheck advances `runWideState` by the general pass's consumed work instead of rewriting the directed pass's local sample position; consumers must not infer this offset from pass names.
+
 ## Privacy
 
 Progress events are intentionally telemetry-like. They must not contain:
@@ -124,7 +133,7 @@ Progress events are intentionally telemetry-like. They must not contain:
 - uploaded file contents;
 - runtime error messages or repro paths.
 
-Those can appear in the final report because the report is story material. Keep the final report wherever you would be comfortable storing project QA artifacts. Progress streams are safer for logs, status UIs, and agent orchestration, but they still reveal operational facts such as run duration, state budget, pass names, and counts.
+Those can appear in the final report because the report is story material. Keep the final report wherever you would be comfortable storing project QA artifacts. Progress streams are safer for logs, status UIs, and agent orchestration, but they still reveal operational facts such as run duration, state budget, pass names, counts, and process memory.
 
 ## Compatibility notes
 

docs/report-schema-v1.md

@@ -57,6 +57,8 @@ Portfolio reports record the resolved worker ceiling and `concurrencyMode` (`aut
 
 Shared-search pass telemetry includes `sharedMemory.current`, per-component `peak` values, configured pending-state/byte `limits`, `releasedNodes`, and `frontierCompactions`. Components cover pending and active state JSON/variable snapshots, retained witness ancestry, dedupe keys, semantic indexes, frontier references, and findings. Serialized strings use UTF-8 byte counts; structural bytes are documented estimates. `totalAccountedBytes` is deterministic retained-payload accounting, not process heap or RSS.
 
+Shared passes additionally expose `sharedObservability` schema v1: a deterministic ledger sampled every 10,000 transitions and at termination, bounded to 128 retained samples. Checkpoint, discovery, and pressure events are not sampling boundaries in this partial #216 slice. Each sample keeps current/per-field-peak logical retention and separate cumulative/interval yield categories for critical findings, intent, authored coverage, visible outcomes, bounded semantic transitions, exact terminal variants, and raw territory. `yieldSummary` keeps through-first-useful and post-first-useful category vectors separate and deliberately has no scalar score. Compact machine responses reconstruct a strict numeric whitelist containing only the latest sample plus the summary. Observed process heap/RSS appears only on live progress and bounded evidence-stream termination, never in this canonical report. See [shared-search observability](shared-search-observability.md) for the compatibility, privacy, and incomplete-owner-accounting boundaries.
+
 Portfolio pass telemetry additionally contains `portfolioMarginalCurve` and `portfolioMarginalSummary`. The pass-local curve answers “what did this explorer find itself?”; the marginal curve answers “which findings did this explorer add first to the combined portfolio?” Runtime and assertion credit uses stable identities, approximate runtime locations are conservatively normalized for allocation credit, and every exact ending, visible outcome, authored knot, goal/stage, or critical finding is paid once. Cross-pass state novelty remains zero because independent pass hashes are not comparable. Shadow allocation reads this marginal curve when present; diagnostics retain both.
 
 `discoverySummary` preserves factual distances that curve compaction must not lose: total discovery events, first/latest discovery states, current states since discovery, latest discovery gap, and longest observed gap. These fields intentionally contain no plateau probability, knee estimate, value score, or automatic decision.

docs/shared-checkpoint-schema-v1.md

@@ -20,7 +20,9 @@ const continued = exploreSharedResumable(storyJson, knots, externals, {
 
 ## Exact-resume contract
 
-Schema v1 stores the partially expanded choice cursor, pending nodes and witness ancestry, deep/novelty/seeded frontier internals, PRNG state, deduplication and semantic indexes, findings, coverage, discovery-curve state, counters, and deterministic memory accounting. Tests pause partway through a choice list, round-trip the checkpoint through JSON, and require the resumed result and next checkpoint to deep-equal uninterrupted execution at the same final grant.
+Schema v1 stores the partially expanded choice cursor, pending nodes and witness ancestry, deep/novelty/seeded frontier internals, PRNG state, deduplication and semantic indexes, findings, coverage, discovery-curve state, counters, deterministic memory accounting, and the additive deterministic resource/yield ledger. Tests pause partway through a choice list, round-trip the checkpoint through JSON, and require the resumed result and next checkpoint to deep-equal uninterrupted execution at the same final grant.
+
+The resource/yield fields are additive within schema v1. Older v1 checkpoints that lack them remain readable and resume the exact search frontier; their new telemetry reports `historyComplete: false` because Inkcheck does not reconstruct missing interval history. A checkpoint that contains the ledger must resume with the same sampling interval. Saving or reopening a checkpoint does not itself create a resource sample; this partial #216 slice samples only fixed transition intervals and termination. Live process heap/RSS is observational and is never written to checkpoint JSON or included in its stable ID. See [shared-search observability](shared-search-observability.md).
 
 The checkpoint is bound to: