Add deterministic shared observability boundaries

CHANGELOG.md

@@ -2,7 +2,8 @@
 
 ## Unreleased
 
-- Add a bounded fixed-interval/termination `ResourceSampleV1` ledger for shared search with deterministic logical retention, category-specific interval yield, early-versus-post-first summaries, fail-closed exact-resume persistence, strict compact machine projection, privacy-safe live heap/RSS observations, and explicit monotonic run-wide positions across additive goal passes. This partial #216 slice changes no search policy, makes no complete owner-attribution claim, and leaves checkpoint/discovery/pressure-triggered sampling for later work.
+- Extend shared-search observability to nested schema v2 with monotonic sample sequences, canonical coalesced reason vectors, an exact 15-bit trigger mask, a fixed eight-count boundary-local trigger-yield tuple, category/frontier/cadence/termination triggers, and category-specific first/last discovery, current/longest dry-state, and identities-per-million-transition facts. The redundant vector/mask/tuple preserves exact retained-boundary triggers across compaction while aggregate interval deltas are rebuilt: adjacent complete-history samples require the tuple to equal the semantic interval delta, while genuine compaction gaps or incomplete v1 history allow only a componentwise-bounded tuple. Observability-v1 migration synthesizes masks only from explicit historical boundaries and uses a zero tuple to mean no v1 trigger was recorded, not that an interval was event-free. Sequence validation applies monotonic and feasible record/state/cadence bounds without claiming exact replay of discarded compaction history. Emitted v2 checkpoints add a domain-separated, key-order-independent canonical SHA-256 self-check binding every fixed known ledger field; it detects stale or accidental mutation but is not authentication because a writer can recompute it, and persisted payload/manifest digests remain the external artifact-corruption boundary. V1 and pre-ledger inputs gain the nested checksum on their first emitted v2 checkpoint. Live process sampling and progress output remain bounded to cadence and final termination while the compacted deterministic ledger retains event boundaries. Checkpoint construction remains side-effect-free and deterministic: the persisted ledger participates in stable checkpoint identity, while observability-v1 checkpoints migrate explicitly with incomplete event history and nullable facts rather than invented history. This remains an independent partial #216 slice: `checkpoint`, `epoch`, and generic `pressure` reasons are reserved but not emitted; retained-owner accounting remains incomplete; no resource, allocation, or stopping policy is activated; #216 remains open; and the earlier V1 overhead study is not an exact-head overhead claim for this V2 implementation.
+- Add the initial bounded fixed-interval/termination `ResourceSampleV1` ledger for shared search with deterministic logical retention, category-specific interval yield, early-versus-post-first summaries, fail-closed exact-resume persistence, strict compact machine projection, privacy-safe live heap/RSS observations, and explicit monotonic run-wide positions across additive goal passes. The nested-v2 extension above adds category and frontier event boundaries; checkpoint, epoch, and generic pressure boundaries remain later work.
 - Bound schema-v1 checkpoint readback by stored and decompressed bytes, classify reopen failures as corrupt, unsupported, or resource-limited, and add canonically self-bound private manifests so listing and retention use bounded metadata I/O without opening new frontiers. Full payload digests remain an open/resume boundary; no-clobber same-ID publication, payload-first crash recovery, and pair-inclusive quotas preserve existing v1 JSON/gzip reads, stable IDs, and exact resume pending framed schema v2.
 - Add an opt-in bounded NDJSON evidence stream for marathon-scale external consumers. It emits replayable numeric ending/runtime witnesses with global elapsed timestamps as they are retained, then a compact terminal summary without constructing the monolithic full-report JSON string.
 - Treat `--max-time` as a total CLI deadline and retain bounded time/heap headroom for clean report finalization. Explicit heap caps now expose the lower search watermark separately from the full process envelope.

docs/progress-ndjson.md

@@ -41,7 +41,7 @@ Optional fields:
 | `unvisitedKnots` | number | Knots not yet reached by any pass in this run. Non-increasing within a run. |
 | `knotsVisited` | number | Cumulative authored knots reached. Present on `discovery` events. |
 | `discoveries` | object | Numeric deltas first observed at this event: `endings`, `runtimeErrors`, `knotsVisited`, `visibleOutcomes`, `assertionViolations`, `goalsReached`, and `stagesReached`. Present only on `discovery` events. |
-| `sharedObservability` | object | Shared-search `resource` events only: one deterministic logical retention/yield sample paired with observed process heap/RSS. `runWideState` is the explicit outer-run position; nested `sample.state` remains local to its shared pass. See [shared-search observability](shared-search-observability.md). |
+| `sharedObservability` | object | Shared-search `resource` events only: one deterministic `ResourceSampleV2` paired with observed process heap/RSS. `runWideState` is the explicit outer-run position; nested `sample.state` remains local to its shared pass. See [shared-search observability](shared-search-observability.md). |
 | `status` | string | Terminal process status: `complete`, `cancelled`, or `error`. Hosted wrappers also use queue/job states. |
 | `stopReason` | string | Binding terminal reason such as `exhaustive`, `state_budget`, `depth_limit`, `time_limit`, `memory_limit`, `frontier_limit`, `worker_failure`, `compile_error`, `cancelled`, or `error`. |
 | `outcome` | string | Result classification separate from the stop cause: `clean`, `issues_found`, `review_required`, or `compile_error`. |
@@ -50,6 +50,8 @@ Progress counts are cumulative over the run, not per-pass, so a consumer can ren
 
 Fields may be added in a future schema or minor version. Consumers should branch on `type`, use fields they understand, and ignore unknown fields.
 
+Nested contracts carry their own versions. A `resource` event still has progress `schemaVersion: 1`, while its `sharedObservability` live observation and `sample` use schema v2. Consumers must branch on each nested discriminator rather than assuming it matches the outer progress version; historical v1 observations remain valid recorded data.
+
 ## Lifecycle
 
 A normal complete run looks like this:
@@ -88,7 +90,7 @@ Privacy-safe discovery event:
 Shared-search resource event (abridged here; consumers should ignore unknown nested fields):
 
 ```json
-{"schemaVersion":1,"sequence":6,"type":"resource","elapsedMs":702,"statesExplored":10000,"stateBudget":100000,"budgetFraction":0.1,"pass":"shared:deep-novelty-v1:seed=1","sharedObservability":{"schemaVersion":1,"pass":"shared:deep-novelty-v1:seed=1","runWideState":10000,"sample":{"schemaVersion":1,"boundary":"interval","state":10000,"retention":{"schemaVersion":1,"current":{"totalAccountedBytes":8388608}},"yield":{"schemaVersion":1,"fromStateExclusive":0,"throughState":10000,"delta":{"critical":{"runtimeErrors":0,"assertionViolations":0}}}},"process":{"schemaVersion":1,"scope":"process","heapUsedBytes":67108864,"heapTotalBytes":83886080,"rssBytes":104857600,"externalBytes":2097152,"arrayBuffersBytes":1048576,"comparedLogicalAccountedBytes":8388608,"unattributedBytes":58720256}}}
+{"schemaVersion":1,"sequence":6,"type":"resource","elapsedMs":702,"statesExplored":10000,"stateBudget":100000,"budgetFraction":0.1,"pass":"shared:deep-novelty-v1:seed=1","sharedObservability":{"schemaVersion":2,"pass":"shared:deep-novelty-v1:seed=1","runWideState":10000,"sample":{"schemaVersion":2,"sequence":17,"reasons":["cadence","authored_knot"],"triggerMask":33,"triggerYield":[0,0,0,0,1,0,0,0],"state":10000,"retention":{"schemaVersion":1,"current":{"totalAccountedBytes":8388608}},"yield":{"schemaVersion":1,"fromStateExclusive":9999,"throughState":10000,"delta":{"critical":{"runtimeErrors":0,"assertionViolations":0},"authoredCoverage":{"knotsVisited":1}}}},"process":{"schemaVersion":1,"scope":"process","heapUsedBytes":67108864,"heapTotalBytes":83886080,"rssBytes":104857600,"externalBytes":2097152,"arrayBuffersBytes":1048576,"comparedLogicalAccountedBytes":8388608,"unattributedBytes":58720256}}}
 ```
 
 Terminal event:
@@ -120,7 +122,19 @@ for await (const line of stderrLines) {
 
 `discovery` means that a cumulative run counter increased. It is useful for a concise terminal update, hosted status, or agent scheduling, but it is not a finding record and does not replace the final report. Counts stay privacy-safe by omitting identities, story labels, source locations, messages, paths, and variable data. A later bounded run can still find more.
 
-`resource` is emitted only by shared search at fixed transition boundaries and termination. Checkpoint operations, discovery events, and memory/frontier pressure do not add sampling boundaries in this partial #216 slice. `sample` contains deterministic aggregate counts and logical byte estimates. `process` contains nondeterministic Node process observations and must not participate in report identity, exact-resume comparison, frontier order, or a coverage claim. The outer `statesExplored` is CLI-process progress and equals the run base plus `sharedObservability.runWideState`. The nested `sample.state` always belongs to that shared pass. During additive goal work, Inkcheck advances `runWideState` by the general pass's consumed work instead of rewriting the directed pass's local sample position; consumers must not infer this offset from pass names.
+The bounded shared-pass ledger records the fixed cadence; new runtime errors, assertion violations, goals, stages, authored knots, visible outcomes, bounded semantic transitions, or exact terminal variants; frontier compaction or the existing shared-frontier ceiling; and termination. If several triggers occur at one pass-local state, one v2 sample coalesces them into a deduplicated `reasons` vector in canonical schema order. Consumers should use the vector rather than infer a primary trigger from array position.
+
+The complete reason vocabulary, in order, is `cadence`, `runtime_error`, `assertion_violation`, `goal_reached`, `stage_reached`, `authored_knot`, `visible_outcome`, `semantic_transition`, `terminal_variant`, `frontier_compaction`, `frontier_ceiling`, `checkpoint`, `epoch`, `pressure`, and `termination`. `checkpoint`, `epoch`, and generic `pressure` are reserved but are not emitted by this slice. Checkpoint save/reopen and memory/time pressure therefore remain absent as resource-event boundaries; no allocation or stopping policy reads these events.
+
+`triggerMask` is the exact bounded numeric encoding of the same vector: bits 0 through 14 correspond to that canonical order, and a valid sample has a set bit exactly for each listed reason. The example's cadence and authored-knot reasons therefore produce mask `33`. This mask stays attached to the exact ending boundary when deterministic sample compaction rebuilds `yield.delta` across a wider retained interval; consumers must not infer boundary triggers from that aggregate delta.
+
+`triggerYield` supplies the exact boundary-local counts for reason bits 1 through 8: runtime errors, assertion violations, goals, stages, authored knots, visible outcomes, semantic transitions, and terminal variants. Its eight safe nonnegative integers have nonzero positions exactly matching those yield reasons. Between adjacent sequences with complete event history, this tuple equals the semantic-category portion of `yield.delta` exactly. Across a genuine compaction gap or incomplete v1 history, compaction/migration preserves the boundary tuple while the aggregate delta covers a wider or incompletely known interval, so each tuple component is only required to be no greater than its matching delta count. A v1-migrated all-zero tuple means v1 recorded no exact category trigger at that boundary; it does not prove the wider interval contained no category event.
+
+The nested checkpoint-only `integritySha256` self-check is intentionally absent from live progress. A `resource` event carries the validated sample projection and observational process values, not the persisted `SharedObservabilityCheckpointV2` envelope or its artifact-integrity metadata.
+
+Live `resource` events are intentionally sparser than that deterministic ledger: the CLI samples process memory and emits them only at the fixed cadence and final termination. A cadence observation is delayed until its pass-local state is closed, so a later same-state reason is coalesced before append-only output and no partial revision is emitted. With the production cadence and 100-million-state ceiling, one uninterrupted shared-pass execution emits at most 10,001 resource events; category-heavy stories do not increase the count.
+
+`sample` contains deterministic aggregate counts, canonical reason codes, their equivalent bounded numeric mask, the fixed boundary-local trigger tuple, transition positions, and logical byte estimates. The full pass telemetry's separate `timing` summary contains category dry-gap/rate facts; it is not duplicated into every live sample. `process` contains nondeterministic Node process observations and must not participate in report identity, checkpoint identity, exact-resume comparison, frontier order, or a coverage claim. The outer `statesExplored` is CLI-process progress and equals the run base plus `sharedObservability.runWideState`. The nested `sample.state` always belongs to that shared pass. During additive goal work, Inkcheck advances `runWideState` by the general pass's consumed work instead of rewriting the directed pass's local sample position; consumers must not infer this offset from pass names.
 
 ## Privacy