diff --git a/CHANGELOG.md b/CHANGELOG.md index 67b2cb1..9474660 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## Unreleased +- Add a bounded fixed-interval/termination `ResourceSampleV1` ledger for shared search with deterministic logical retention, category-specific interval yield, early-versus-post-first summaries, fail-closed exact-resume persistence, strict compact machine projection, privacy-safe live heap/RSS observations, and explicit monotonic run-wide positions across additive goal passes. This partial #216 slice changes no search policy, makes no complete owner-attribution claim, and leaves checkpoint/discovery/pressure-triggered sampling for later work. - Bound schema-v1 checkpoint readback by stored and decompressed bytes, classify reopen failures as corrupt, unsupported, or resource-limited, and add canonically self-bound private manifests so listing and retention use bounded metadata I/O without opening new frontiers. Full payload digests remain an open/resume boundary; no-clobber same-ID publication, payload-first crash recovery, and pair-inclusive quotas preserve existing v1 JSON/gzip reads, stable IDs, and exact resume pending framed schema v2. - Add an opt-in bounded NDJSON evidence stream for marathon-scale external consumers. It emits replayable numeric ending/runtime witnesses with global elapsed timestamps as they are retained, then a compact terminal summary without constructing the monolithic full-report JSON string. - Treat `--max-time` as a total CLI deadline and retain bounded time/heap headroom for clean report finalization. Explicit heap caps now expose the lower search watermark separately from the full process envelope. diff --git a/README.md b/README.md index 8cb25a3..cd36277 100644 --- a/README.md +++ b/README.md @@ -134,7 +134,7 @@ For a new project containing one `.ink` file, `inkcheck init` creates this confi `--save-report` atomically stores a versioned report under `.inkcheck/reports/` and returns its stable content-and-entrypoint-derived ID. A later session can use `inkcheck artifacts list --json` and `inkcheck artifacts show --json`; reopening reports whether the saved evidence is `current`, `stale`, or `path_changed` against the present entrypoint. Reports can contain story text, variables, and exact witnesses, so the agent kit ignores them by default. See [local report artifacts](docs/local-artifacts.md) for the trust, privacy, and compatibility contract. -Long base-shared runs can also persist their exact live frontier locally. Start with `--search=shared --no-min-repro --save-checkpoint`, then continue later with `inkcheck resume --max-states N`; `N` is the larger total grant, not extra hidden work. `inkcheck checkpoints list/show` reports bounded metadata, durable compressed size, storage encoding, and source freshness. New checkpoint files are streamed gzip artifacts; older plain schema-v1 JSON remains readable. Checkpoints are private, atomic, source/config-bound, ignored by default, and retention-capped; they may contain authored text and runtime state. See [local resumable checkpoints](docs/local-checkpoints.md). MCP agents can use the same exact foundation through durable [`start_search` / `inspect_search` / `continue_search` / `cancel_search` result windows](docs/mcp-search-sessions.md). They can also use `add_goal` for an explicit additive directed probe that starts from the story root and leaves that exact base frontier untouched. Portfolio, shared-variable, assertions, directed-frontier resume, and hosted jobs do not use this checkpoint contract yet. +Long base-shared runs can also persist their exact live frontier locally. Start with `--search=shared --no-min-repro --save-checkpoint`, then continue later with `inkcheck resume --max-states N`; `N` is the larger total grant, not extra hidden work. `inkcheck checkpoints list/show` reports bounded metadata, durable compressed size, storage encoding, and source freshness. New checkpoint files are streamed gzip artifacts; older plain schema-v1 JSON remains readable. Checkpoints are private, atomic, source/config-bound, ignored by default, and retention-capped; they may contain authored text and runtime state. See [local resumable checkpoints](docs/local-checkpoints.md). Shared passes also expose a bounded [resource/yield observability ledger](docs/shared-search-observability.md) that keeps deterministic logical accounting separate from live heap/RSS observations. MCP agents can use the same exact foundation through durable [`start_search` / `inspect_search` / `continue_search` / `cancel_search` result windows](docs/mcp-search-sessions.md). They can also use `add_goal` for an explicit additive directed probe that starts from the story root and leaves that exact base frontier untouched. Portfolio, shared-variable, assertions, directed-frontier resume, and hosted jobs do not use this checkpoint contract yet. ## Hosted checker diff --git a/docs/progress-ndjson.md b/docs/progress-ndjson.md index 6af0f31..6a528e5 100644 --- a/docs/progress-ndjson.md +++ b/docs/progress-ndjson.md @@ -16,7 +16,7 @@ Common fields: | --- | --- | --- | | `schemaVersion` | number | Progress schema version. Currently `1`. | | `sequence` | number | Monotonic event number starting at `1` for each CLI process. | -| `type` | string | Event kind: `run_start`, `phase_start`, `progress`, `discovery`, `phase_end`, or `run_end`. | +| `type` | string | Event kind: `run_start`, `phase_start`, `progress`, `discovery`, `resource`, `phase_end`, or `run_end`. | | `elapsedMs` | number | Milliseconds since the CLI run started. | | `statesExplored` | number | Total story states explored so far in this CLI process. | | `stateBudget` | number | Total configured work budget: baseline plus additional goal states. | @@ -41,6 +41,7 @@ Optional fields: | `unvisitedKnots` | number | Knots not yet reached by any pass in this run. Non-increasing within a run. | | `knotsVisited` | number | Cumulative authored knots reached. Present on `discovery` events. | | `discoveries` | object | Numeric deltas first observed at this event: `endings`, `runtimeErrors`, `knotsVisited`, `visibleOutcomes`, `assertionViolations`, `goalsReached`, and `stagesReached`. Present only on `discovery` events. | +| `sharedObservability` | object | Shared-search `resource` events only: one deterministic logical retention/yield sample paired with observed process heap/RSS. `runWideState` is the explicit outer-run position; nested `sample.state` remains local to its shared pass. See [shared-search observability](shared-search-observability.md). | | `status` | string | Terminal process status: `complete`, `cancelled`, or `error`. Hosted wrappers also use queue/job states. | | `stopReason` | string | Binding terminal reason such as `exhaustive`, `state_budget`, `depth_limit`, `time_limit`, `memory_limit`, `frontier_limit`, `worker_failure`, `compile_error`, `cancelled`, or `error`. | | `outcome` | string | Result classification separate from the stop cause: `clean`, `issues_found`, `review_required`, or `compile_error`. | @@ -57,7 +58,7 @@ A normal complete run looks like this: 2. `phase_start` for `compile` 3. `phase_end` for `compile` 4. `phase_start` / `phase_end` for source scanning and exploration phases as applicable -5. zero or more `progress` activity events and `discovery` evidence events during exploration +5. zero or more `progress` activity, `discovery` evidence, and shared-search `resource` events during exploration 6. `phase_start` for `report` 7. `phase_end` for `report` 8. `run_end` @@ -84,6 +85,12 @@ Privacy-safe discovery event: {"schemaVersion":1,"sequence":5,"type":"discovery","elapsedMs":611,"statesExplored":5200,"stateBudget":100000,"budgetFraction":0.052,"pass":"beam:w=64","endingsFound":4,"runtimeErrorsFound":1,"unvisitedKnots":7,"knotsVisited":12,"discoveries":{"endings":1,"runtimeErrors":1,"knotsVisited":2,"visibleOutcomes":1,"assertionViolations":0,"goalsReached":0,"stagesReached":0}} ``` +Shared-search resource event (abridged here; consumers should ignore unknown nested fields): + +```json +{"schemaVersion":1,"sequence":6,"type":"resource","elapsedMs":702,"statesExplored":10000,"stateBudget":100000,"budgetFraction":0.1,"pass":"shared:deep-novelty-v1:seed=1","sharedObservability":{"schemaVersion":1,"pass":"shared:deep-novelty-v1:seed=1","runWideState":10000,"sample":{"schemaVersion":1,"boundary":"interval","state":10000,"retention":{"schemaVersion":1,"current":{"totalAccountedBytes":8388608}},"yield":{"schemaVersion":1,"fromStateExclusive":0,"throughState":10000,"delta":{"critical":{"runtimeErrors":0,"assertionViolations":0}}}},"process":{"schemaVersion":1,"scope":"process","heapUsedBytes":67108864,"heapTotalBytes":83886080,"rssBytes":104857600,"externalBytes":2097152,"arrayBuffersBytes":1048576,"comparedLogicalAccountedBytes":8388608,"unattributedBytes":58720256}}} +``` + Terminal event: ```json @@ -113,6 +120,8 @@ for await (const line of stderrLines) { `discovery` means that a cumulative run counter increased. It is useful for a concise terminal update, hosted status, or agent scheduling, but it is not a finding record and does not replace the final report. Counts stay privacy-safe by omitting identities, story labels, source locations, messages, paths, and variable data. A later bounded run can still find more. +`resource` is emitted only by shared search at fixed transition boundaries and termination. Checkpoint operations, discovery events, and memory/frontier pressure do not add sampling boundaries in this partial #216 slice. `sample` contains deterministic aggregate counts and logical byte estimates. `process` contains nondeterministic Node process observations and must not participate in report identity, exact-resume comparison, frontier order, or a coverage claim. The outer `statesExplored` is CLI-process progress and equals the run base plus `sharedObservability.runWideState`. The nested `sample.state` always belongs to that shared pass. During additive goal work, Inkcheck advances `runWideState` by the general pass's consumed work instead of rewriting the directed pass's local sample position; consumers must not infer this offset from pass names. + ## Privacy Progress events are intentionally telemetry-like. They must not contain: @@ -124,7 +133,7 @@ Progress events are intentionally telemetry-like. They must not contain: - uploaded file contents; - runtime error messages or repro paths. -Those can appear in the final report because the report is story material. Keep the final report wherever you would be comfortable storing project QA artifacts. Progress streams are safer for logs, status UIs, and agent orchestration, but they still reveal operational facts such as run duration, state budget, pass names, and counts. +Those can appear in the final report because the report is story material. Keep the final report wherever you would be comfortable storing project QA artifacts. Progress streams are safer for logs, status UIs, and agent orchestration, but they still reveal operational facts such as run duration, state budget, pass names, counts, and process memory. ## Compatibility notes diff --git a/docs/report-schema-v1.md b/docs/report-schema-v1.md index 1ded5cd..671d75b 100644 --- a/docs/report-schema-v1.md +++ b/docs/report-schema-v1.md @@ -57,6 +57,8 @@ Portfolio reports record the resolved worker ceiling and `concurrencyMode` (`aut Shared-search pass telemetry includes `sharedMemory.current`, per-component `peak` values, configured pending-state/byte `limits`, `releasedNodes`, and `frontierCompactions`. Components cover pending and active state JSON/variable snapshots, retained witness ancestry, dedupe keys, semantic indexes, frontier references, and findings. Serialized strings use UTF-8 byte counts; structural bytes are documented estimates. `totalAccountedBytes` is deterministic retained-payload accounting, not process heap or RSS. +Shared passes additionally expose `sharedObservability` schema v1: a deterministic ledger sampled every 10,000 transitions and at termination, bounded to 128 retained samples. Checkpoint, discovery, and pressure events are not sampling boundaries in this partial #216 slice. Each sample keeps current/per-field-peak logical retention and separate cumulative/interval yield categories for critical findings, intent, authored coverage, visible outcomes, bounded semantic transitions, exact terminal variants, and raw territory. `yieldSummary` keeps through-first-useful and post-first-useful category vectors separate and deliberately has no scalar score. Compact machine responses reconstruct a strict numeric whitelist containing only the latest sample plus the summary. Observed process heap/RSS appears only on live progress and bounded evidence-stream termination, never in this canonical report. See [shared-search observability](shared-search-observability.md) for the compatibility, privacy, and incomplete-owner-accounting boundaries. + Portfolio pass telemetry additionally contains `portfolioMarginalCurve` and `portfolioMarginalSummary`. The pass-local curve answers “what did this explorer find itself?”; the marginal curve answers “which findings did this explorer add first to the combined portfolio?” Runtime and assertion credit uses stable identities, approximate runtime locations are conservatively normalized for allocation credit, and every exact ending, visible outcome, authored knot, goal/stage, or critical finding is paid once. Cross-pass state novelty remains zero because independent pass hashes are not comparable. Shadow allocation reads this marginal curve when present; diagnostics retain both. `discoverySummary` preserves factual distances that curve compaction must not lose: total discovery events, first/latest discovery states, current states since discovery, latest discovery gap, and longest observed gap. These fields intentionally contain no plateau probability, knee estimate, value score, or automatic decision. diff --git a/docs/shared-checkpoint-schema-v1.md b/docs/shared-checkpoint-schema-v1.md index aed2f2b..fd74fc1 100644 --- a/docs/shared-checkpoint-schema-v1.md +++ b/docs/shared-checkpoint-schema-v1.md @@ -20,7 +20,9 @@ const continued = exploreSharedResumable(storyJson, knots, externals, { ## Exact-resume contract -Schema v1 stores the partially expanded choice cursor, pending nodes and witness ancestry, deep/novelty/seeded frontier internals, PRNG state, deduplication and semantic indexes, findings, coverage, discovery-curve state, counters, and deterministic memory accounting. Tests pause partway through a choice list, round-trip the checkpoint through JSON, and require the resumed result and next checkpoint to deep-equal uninterrupted execution at the same final grant. +Schema v1 stores the partially expanded choice cursor, pending nodes and witness ancestry, deep/novelty/seeded frontier internals, PRNG state, deduplication and semantic indexes, findings, coverage, discovery-curve state, counters, deterministic memory accounting, and the additive deterministic resource/yield ledger. Tests pause partway through a choice list, round-trip the checkpoint through JSON, and require the resumed result and next checkpoint to deep-equal uninterrupted execution at the same final grant. + +The resource/yield fields are additive within schema v1. Older v1 checkpoints that lack them remain readable and resume the exact search frontier; their new telemetry reports `historyComplete: false` because Inkcheck does not reconstruct missing interval history. A checkpoint that contains the ledger must resume with the same sampling interval. Saving or reopening a checkpoint does not itself create a resource sample; this partial #216 slice samples only fixed transition intervals and termination. Live process heap/RSS is observational and is never written to checkpoint JSON or included in its stable ID. See [shared-search observability](shared-search-observability.md). The checkpoint is bound to: diff --git a/docs/shared-search-observability.md b/docs/shared-search-observability.md new file mode 100644 index 0000000..1ee6da5 --- /dev/null +++ b/docs/shared-search-observability.md @@ -0,0 +1,68 @@ +# Shared-search observability v1 + +Inkcheck's base shared search now records a bounded, versioned ledger of deterministic logical retention and category-specific yield. Live CLI progress can pair the same sample with observed Node process memory. This is a partial implementation slice tracked by [issue #216](https://github.com/chaoz23/inkcheck/issues/216), which remains open; it is not the complete long-run resource policy. + +## Deterministic ledger + +Each shared pass exposes `passes[].sharedObservability` with `schemaVersion: 1`. Inkcheck records a `ResourceSampleV1` every 10,000 completed transitions and at termination. Tests and embedders may request a different positive interval through the library option `sharedObservabilityIntervalStates`; CLI users receive the fixed default. + +Those are the only sampling boundaries in this slice. Checkpoint save/resume, discovery events, memory or frontier pressure, and other lifecycle events do not trigger an additional sample. Boundary-specific sampling and the policy for choosing it remain deferred under #216. + +The ledger retains at most 128 samples. If it grows beyond that bound, Inkcheck deterministically keeps the first and latest boundaries and downsamples the interior. `samplesRecorded`, `samplesRetained`, and `samplesCompacted` make that loss of interval resolution explicit. Every retained sample keeps cumulative counters, and its `delta` is recomputed over the retained interval. This compaction changes telemetry resolution only; it never changes frontier order or findings. + +`RetentionBreakdownV1` separates `current` from per-field `peak` values. The current structural subset accounts for: + +- pending and active serialized state and variable payloads; +- retained ancestry and node-table slots; +- exact dedupe keys; +- semantic indexes; +- frontier-view references; and +- retained findings. + +These are deterministic logical estimates compatible with the existing `sharedMemory` telemetry. Serialized strings use UTF-8 bytes and structural values use documented estimates. A peak object contains independent high-water values, so its components need not describe one simultaneous heap snapshot. + +The subset is not complete owner attribution. It does not yet account for Ink runtime objects, checkpoint encode/decode buffers, report serialization, process-tree memory, or reserved finalization headroom. It must not be described as total memory owned by search. + +## Yield vector + +`YieldIntervalV1` reports separate cumulative and interval counts; it never produces a weighted usefulness score: + +| Category | Version-1 meaning | +| --- | --- | +| `critical` | Distinct runtime errors and assertion violations. | +| `intent` | Approved goals and cumulative goal stages reached. | +| `authoredCoverage` | Distinct knots visited. | +| `visibleOutcomes` | Distinct normalized rendered endings; a fallback, not authored-ending identity. | +| `semanticTransitions` | First observation of a bounded Boolean toggle, bounded string/enum change, or numeric zero crossing. Ordinary numeric churn is excluded. | +| `terminalVariants` | Exact terminal states, kept separate from visible outcomes. | +| `rawTerritory` | Transitions, unique exact states, and dedupe hits. These are work facts, not useful yield by themselves. | + +`yieldSummary.firstUsefulAtState` marks the first critical, intent, authored-coverage, visible-outcome, or semantic-transition event. `firstCriticalAtState` is separate. `throughFirstUseful` and `afterFirstUseful` keep early value distinct from later yield without collapsing unlike categories. The existing discovery curve also samples assertion and goal changes after those trackers run, and visible-outcome-only changes are valid discovery boundaries. + +This slice does not yet expose kth/last event timing, full dry-gap history, rediscovery identities, throughput, retained GiB-minutes, campaign-new attribution, or checkpoint-, pressure-, and discovery-triggered samples. + +## Observed process memory + +`onSharedObservability` and CLI `resource` progress events pair a deterministic sample with `ProcessMemoryObservationV1`. Each observation also carries numeric `runWideState`. For a standalone or general shared pass it equals the pass-local `sample.state`; for additive directed-goal work it adds the completed general-pass work. The CLI uses `runWideState` for monotonic outer progress and never rewrites the nested pass-local sample or infers an offset from the pass name. + +Process fields are: + +- `heapUsedBytes` and `heapTotalBytes` from V8; +- process `rssBytes`; +- `externalBytes` and `arrayBuffersBytes`; +- `comparedLogicalAccountedBytes`; and +- `unattributedBytes`, calculated as `heapUsedBytes - comparedLogicalAccountedBytes`. + +The difference may be negative because the logical model and V8 heap measure different things. It is an observational comparison, not proof of ownership or a leak. Heap/RSS values vary with runtime, garbage collection, machine load, and Node version. + +For that reason, process observations are deliberately excluded from shared checkpoints, canonical JSON reports, report/checkpoint IDs, exact-resume comparisons, and compact machine summaries. They appear only on live CLI progress and the bounded `--json-stream` terminal resource summary. Turning these values into automatic stopping, eviction, or allocation decisions is outside this slice. + +## Resume and compatibility + +The deterministic ledger is additive inside shared checkpoint schema v1. A split run with the same source, configuration, seed, and sampling interval produces the same search result, ledger, and next checkpoint as one uninterrupted run. Saving a checkpoint preserves the latest fixed-cadence ledger state but does not create a checkpoint-boundary sample. Changing the interval while resuming fails closed. + +Resumable checkpoints contain interval samples only; termination samples belong to finalized report telemetry and live progress. For checkpoint state `S`, ledger base `B`, and cadence `I`, `samplesRecorded` is exactly `floor(S / I) - floor(B / I)`. Retained samples stay on cadence boundaries, preserve the first boundary after `B` and the latest completed boundary through `S`, and keep transition counters aligned with their sample/cursor states. Milestone fields must agree with cumulative critical/useful yield: samples before the first-useful state cannot already contain useful yield, and the milestone vector cannot exceed any retained cumulative sample at or after that state. A complete history has base state zero. These cross-field checks fail closed on tampering while still permitting useful evidence discovered before the first retained sample. + +Older schema-v1 checkpoints remain readable. Because they contain no historical interval ledger or meaningful-transition counter, resumed telemetry sets `historyComplete: false` and begins interval deltas from the reopen boundary. Search frontier and finding compatibility remain exact; Inkcheck does not invent missing historical telemetry. + +Samples and live resource events contain only aggregate counts, byte estimates, pass names, and process values. They contain no story source, choice prose, final text, variable names or values, runtime messages, or witness paths. diff --git a/package.json b/package.json index 2300e6f..be92354 100644 --- a/package.json +++ b/package.json @@ -74,6 +74,7 @@ "docs/local-artifacts.md", "docs/local-checkpoints.md", "docs/shared-checkpoint-schema-v1.md", + "docs/shared-search-observability.md", "docs/mcp-search-sessions.md", "docs/campaign-policy-contract.md", "docs/concurrency-evaluation.md", diff --git a/src/adaptive-concurrent-portfolio.ts b/src/adaptive-concurrent-portfolio.ts index 1e842d8..b8850c3 100644 --- a/src/adaptive-concurrent-portfolio.ts +++ b/src/adaptive-concurrent-portfolio.ts @@ -119,6 +119,7 @@ function sanitizedOptions(options: AdaptiveConcurrentOptions): ExploreOptions { onProgress: _onProgress, onSnapshot: _onSnapshot, onEvidence: _onEvidence, + onSharedObservability: _onSharedObservability, memoryGuard: _memoryGuard, timeGuard: _timeGuard, loopRiskRegistry: _loopRiskRegistry, diff --git a/src/cli.ts b/src/cli.ts index cfcb51d..2255c2b 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -19,6 +19,7 @@ import { ExploreResult, ExploreProgress, ExploreEvidence, + SharedResourceObservationV1, DEFAULT_STORY_SEED, MAX_STORY_SEED, PortfolioWeights, @@ -59,7 +60,7 @@ import { } from "./discovery"; import { findDefaultProjectConfig, loadProjectConfig } from "./config"; import { createAgentKit, initProject, renderScaffoldResult } from "./scaffold"; -import { createResourceGuards } from "./resource-guards"; +import { createResourceGuards, observeProcessMemory } from "./resource-guards"; import { explorePortfolioConcurrent, explorePortfolioPilotHandoffConcurrent, @@ -769,6 +770,7 @@ async function main() { }); let sequence = 0; let statesExplored = resumed?.checkpoint.state.statesExplored ?? 0; + let latestSharedObservation: SharedResourceObservationV1 | undefined; const selectedProgressMode = progressMode as "auto" | "human" | "ndjson" | "off"; const humanProgress = selectedProgressMode === "auto" || selectedProgressMode === "human" ? new HumanProgressRenderer(process.stderr, selectedProgressMode) @@ -793,7 +795,7 @@ async function main() { }; const discoveryKeys = Object.keys(discoveryTotals) as (keyof DiscoveryChanges)[]; const emitProgress = ( - type: "run_start" | "phase_start" | "progress" | "discovery" | "phase_end" | "run_end", + type: "run_start" | "phase_start" | "progress" | "discovery" | "resource" | "phase_end" | "run_end", details: { phase?: "compile" | "source_scan" | "explore" | "min_repro" | "report"; pass?: string; @@ -808,6 +810,7 @@ async function main() { statesSinceLastDiscovery?: number | null; knotsVisited?: number; discoveries?: DiscoveryChanges; + sharedObservability?: SharedResourceObservationV1; status?: ProgressStatus; stopReason?: ProgressStopReason; outcome?: ProgressOutcome; @@ -982,6 +985,18 @@ async function main() { detectLoopRisks: !semantics.usesTurns && !semantics.usesRandomness && !semantics.usesVisitCounts && externals.length === 0, randomnessDetected: semantics.usesRandomness, ...(asJsonStream ? { onEvidence: streamEvidence } : {}), + ...(selectedProgressMode === "off" ? {} : { + onSharedObservability: (observation: SharedResourceObservationV1) => { + latestSharedObservation = observation; + statesExplored = saveCheckpoint + ? observation.runWideState + : statesBase + observation.runWideState; + emitProgress("resource", { + pass: observation.pass, + sharedObservability: observation, + }); + }, + }), onProgress: (progress: ExploreProgress) => { statesExplored = saveCheckpoint ? progress.statesExplored : statesBase + progress.statesExplored; const progressDetails = { @@ -1231,6 +1246,11 @@ async function main() { deadlineMs: maxTimeSec === undefined ? null : startedAt + maxTimeSec * 1_000, searchDeadlineMs: deadlineMs ?? null, finalizationTimeReserveMs: finalizationTimeReserveMs ?? 0, + observedProcessAtTermination: observeProcessMemory( + latestSharedObservation?.sample.retention.current.totalAccountedBytes + ?? report.passes?.find((pass) => pass.sharedMemory)?.sharedMemory?.current.totalAccountedBytes + ?? 0 + ), }, evidence: { endingsEmitted: streamedEndings, runtimeErrorsEmitted: streamedRuntimeErrors, benchmarkSignalsEmitted: streamedBenchmarkSignals }, }); @@ -1387,6 +1407,7 @@ async function main() { endingsFound: report.endingsFound.length, runtimeErrorsFound: report.runtimeErrors.length, unvisitedKnots: report.unvisitedKnots.length, + ...(latestSharedObservation ? { sharedObservability: latestSharedObservation } : {}), }); process.exitCode = hardFail || softFail ? 1 : 0; } diff --git a/src/explore.ts b/src/explore.ts index a72ca61..43c32db 100644 --- a/src/explore.ts +++ b/src/explore.ts @@ -17,6 +17,7 @@ import { import { GoalDefinition, GoalResult, GoalTracker, validateGoals } from "./goals"; import { recommendShadowDecision, type ShadowDecision } from "./decision-policy"; import { CumulativeFloorAllocator, type FloorAllocation } from "./floor-allocator"; +import { observeProcessMemory, type ProcessMemoryObservationV1 } from "./resource-guards"; export const DEFAULT_STORY_SEED = 1; export const MAX_STORY_SEED = 2_147_483_646; @@ -791,6 +792,14 @@ export interface ExploreOptions { onSnapshot?: (result: ExploreResult) => void; /** Receive replayable findings without waiting for the final report. */ onEvidence?: (evidence: ExploreEvidence) => void; + /** + * Shared search only: receive process-scoped heap/RSS beside each bounded, + * deterministic resource/yield sample. Process values are observational and + * are deliberately excluded from checkpoints and canonical reports. + */ + onSharedObservability?: (observation: SharedResourceObservationV1) => void; + /** Internal/test override for the normal 10,000-transition sample cadence. */ + sharedObservabilityIntervalStates?: number; /** Internal/test override for the normal 10,000-state progress cadence. */ progressIntervalStates?: number; /** Internal/test override for the normal one-second progress heartbeat. */ @@ -829,6 +838,8 @@ export const DEFAULT_RANDOM_SEED = 1; export const DEFAULT_BEAM_WIDTH = 64; export const DEFAULT_PROGRESS_INTERVAL_STATES = 10_000; export const DEFAULT_PROGRESS_INTERVAL_MS = 1_000; +export const DEFAULT_SHARED_OBSERVABILITY_INTERVAL_STATES = 10_000; +export const MAX_SHARED_OBSERVABILITY_SAMPLES = 128; const RESOURCE_GUARD_INTERVAL = 512; const TIMED_RESOURCE_GUARD_INTERVAL = 64; @@ -939,6 +950,8 @@ export interface PassTelemetry { peakPendingBytes?: number; /** Shared search only: deterministic retained-payload accounting, not process heap usage. */ sharedMemory?: SharedMemoryTelemetry; + /** Shared search only: bounded deterministic retention and category-specific yield samples. */ + sharedObservability?: SharedObservabilityTelemetryV1; /** Shared search only: distinct variable snapshots observed. */ variableStatesObserved?: number; /** Shared search only: distinct variable changes observed. */ @@ -977,6 +990,154 @@ export interface SharedMemoryTelemetry { frontierCompactions: number; } +export interface SharedYieldCountsV1 { + critical: { + runtimeErrors: number; + assertionViolations: number; + }; + intent: { + goalsReached: number; + stagesReached: number; + }; + authoredCoverage: { + knotsVisited: number; + }; + /** Normalized rendered-text fallback, not authored-ending identity. */ + visibleOutcomes: number; + /** Version-1 Boolean/enum/zero-crossing variable transitions. */ + semanticTransitions: number; + /** Exact terminal states, deliberately separate from visible outcomes. */ + terminalVariants: number; + /** Work facts that are never treated as useful yield by themselves. */ + rawTerritory: { + transitions: number; + uniqueStates: number; + dedupeHits: number; + }; +} + +export interface RetentionBreakdownV1 { + schemaVersion: 1; + /** Deterministic logical/accounted bytes and counts at this boundary. */ + current: SharedRetainedMemory; + /** Per-component deterministic high-water observations through this boundary. */ + peak: SharedRetainedMemory; + releasedNodes: number; + frontierCompactions: number; +} + +export interface YieldIntervalV1 { + schemaVersion: 1; + /** Previous retained sample boundary; deltas cover work after this state. */ + fromStateExclusive: number; + throughState: number; + delta: SharedYieldCountsV1; + cumulative: SharedYieldCountsV1; +} + +export interface ResourceSampleV1 { + schemaVersion: 1; + boundary: "interval" | "termination" | "interval_and_termination"; + state: number; + retention: RetentionBreakdownV1; + yield: YieldIntervalV1; +} + +export interface SharedYieldPhaseSummaryV1 { + schemaVersion: 1; + /** First event in critical, intent, authored coverage, visible outcome, or semantic-transition categories. */ + firstUsefulAtState: number | null; + firstCriticalAtState: number | null; + /** Category vector accumulated through the first useful boundary. */ + throughFirstUseful: SharedYieldCountsV1; + /** Category vector accumulated strictly after that boundary. */ + afterFirstUseful: SharedYieldCountsV1; + cumulative: SharedYieldCountsV1; +} + +export interface SharedObservabilityTelemetryV1 { + schemaVersion: 1; + sampleIntervalStates: number; + samplesRecorded: number; + samplesRetained: number; + samplesCompacted: number; + /** False only when resuming an older checkpoint that predates this ledger. */ + historyComplete: boolean; + samples: ResourceSampleV1[]; + yieldSummary: SharedYieldPhaseSummaryV1; +} + +/** Live-only observation. The process fields never enter exact resume/report identity. */ +export interface SharedResourceObservationV1 { + schemaVersion: 1; + pass: string; + /** Run-wide work position for outer progress; the nested sample remains pass-local. */ + runWideState: number; + sample: ResourceSampleV1; + process: ProcessMemoryObservationV1; +} + +function emptySharedYieldCounts(): SharedYieldCountsV1 { + return { + critical: { runtimeErrors: 0, assertionViolations: 0 }, + intent: { goalsReached: 0, stagesReached: 0 }, + authoredCoverage: { knotsVisited: 0 }, + visibleOutcomes: 0, + semanticTransitions: 0, + terminalVariants: 0, + rawTerritory: { transitions: 0, uniqueStates: 0, dedupeHits: 0 }, + }; +} + +function cloneSharedYieldCounts(value: SharedYieldCountsV1): SharedYieldCountsV1 { + return { + critical: { ...value.critical }, + intent: { ...value.intent }, + authoredCoverage: { ...value.authoredCoverage }, + visibleOutcomes: value.visibleOutcomes, + semanticTransitions: value.semanticTransitions, + terminalVariants: value.terminalVariants, + rawTerritory: { ...value.rawTerritory }, + }; +} + +function subtractSharedYieldCounts( + current: SharedYieldCountsV1, + previous: SharedYieldCountsV1 +): SharedYieldCountsV1 { + return { + critical: { + runtimeErrors: Math.max(0, current.critical.runtimeErrors - previous.critical.runtimeErrors), + assertionViolations: Math.max(0, current.critical.assertionViolations - previous.critical.assertionViolations), + }, + intent: { + goalsReached: Math.max(0, current.intent.goalsReached - previous.intent.goalsReached), + stagesReached: Math.max(0, current.intent.stagesReached - previous.intent.stagesReached), + }, + authoredCoverage: { + knotsVisited: Math.max(0, current.authoredCoverage.knotsVisited - previous.authoredCoverage.knotsVisited), + }, + visibleOutcomes: Math.max(0, current.visibleOutcomes - previous.visibleOutcomes), + semanticTransitions: Math.max(0, current.semanticTransitions - previous.semanticTransitions), + terminalVariants: Math.max(0, current.terminalVariants - previous.terminalVariants), + rawTerritory: { + transitions: Math.max(0, current.rawTerritory.transitions - previous.rawTerritory.transitions), + uniqueStates: Math.max(0, current.rawTerritory.uniqueStates - previous.rawTerritory.uniqueStates), + dedupeHits: Math.max(0, current.rawTerritory.dedupeHits - previous.rawTerritory.dedupeHits), + }, + }; +} + +function usefulYieldObserved(value: SharedYieldCountsV1): boolean { + return value.critical.runtimeErrors > 0 + || value.critical.assertionViolations > 0 + || value.intent.goalsReached > 0 + || value.intent.stagesReached > 0 + || value.authoredCoverage.knotsVisited > 0 + || value.visibleOutcomes > 0 + || value.semanticTransitions > 0; +} + export interface DiscoveryCounts { endingsFound: number; runtimeErrorsFound: number; @@ -1022,6 +1183,8 @@ export interface DiscoveryCurveCheckpoint { latestGap: number | null; longestGap: number | null; previousCounts: DiscoveryCounts; + /** Additive marker for checkpoints written after visible outcomes became event boundaries. */ + countedVisibleOutcomes?: true; } export class DiscoveryCurveRecorder { @@ -1046,7 +1209,8 @@ export class DiscoveryCurveRecorder { constructor(checkpoint?: DiscoveryCurveCheckpoint) { if (!checkpoint) return; this.samples = checkpoint.samples.map((sample) => ({ ...sample })); - this.previousTotal = checkpoint.previousTotal; + this.previousTotal = checkpoint.previousTotal + + (checkpoint.countedVisibleOutcomes ? 0 : checkpoint.previousCounts.visibleOutcomes); this.previousState = checkpoint.previousState; this.firstState = checkpoint.firstState; this.eventCount = checkpoint.eventCount; @@ -1056,7 +1220,7 @@ export class DiscoveryCurveRecorder { } observe(state: number, counts: DiscoveryCounts): boolean { - const total = counts.endingsFound + counts.runtimeErrorsFound + counts.knotsVisited + const total = counts.endingsFound + counts.runtimeErrorsFound + counts.knotsVisited + counts.visibleOutcomes + counts.assertionViolations + counts.goalsReached + counts.stagesReached; if (total <= this.previousTotal) return false; const gap = this.previousState === null ? null : state - this.previousState; @@ -1103,6 +1267,7 @@ export class DiscoveryCurveRecorder { latestGap: this.latestGap, longestGap: this.longestGap, previousCounts: { ...this.previousCounts }, + countedVisibleOutcomes: true, }; } @@ -1122,6 +1287,19 @@ function visibleOutcomeKey(finalText: string): string { return finalText.trim().replace(/\s+/g, " "); } +/** Version-1 bounded semantic-transition filter; ordinary numeric churn is excluded. */ +function meaningfulVariableTransition(change: { before: unknown; after: unknown }): boolean { + if (typeof change.before === "boolean" && typeof change.after === "boolean") return change.before !== change.after; + if (typeof change.before === "string" && typeof change.after === "string") { + return change.before !== change.after && change.before.length <= 128 && change.after.length <= 128; + } + if (typeof change.before === "number" && typeof change.after === "number" + && Number.isFinite(change.before) && Number.isFinite(change.after)) { + return (change.before <= 0 && change.after > 0) || (change.before >= 0 && change.after < 0); + } + return false; +} + function stableObservedValues(values: Record): string { return JSON.stringify(Object.fromEntries(Object.entries(values).sort(([a], [b]) => a.localeCompare(b)))); } @@ -1451,7 +1629,6 @@ function createSearchEngine( }); s.warnings.forEach((w) => runtimeWarnings.add(w)); recordKnotCoverage(s); - noteDiscoveryProgress(); const ended = !s.story.canContinue && s.story.currentChoices.length === 0; const stateVariables = extractVariables(s.story); @@ -1469,10 +1646,11 @@ function createSearchEngine( const key = finalText + "|" + JSON.stringify(stateVariables); if (!endings.has(key)) { recordEnding(endings, visibleOutcomes, key, { path, choiceIndices, firstDiscoveredAtState: statesExplored, finalText, variables: stateVariables, foundBy }, opts.onEvidence); - noteDiscoveryProgress(); } + noteDiscoveryProgress(); return true; } + noteDiscoveryProgress(); const nextState = s.story.state.ToJson(); if (noteLoopControl(nextState, stateVariables, path, choiceIndices, statesExplored)) { truncated = true; @@ -1652,6 +1830,22 @@ export interface SharedCheckpointHeapItem { order: number; } +export interface SharedObservabilityCheckpointV1 { + schemaVersion: 1; + sampleIntervalStates: number; + samplesRecorded: number; + samples: ResourceSampleV1[]; + baseState: number; + baseYield: SharedYieldCountsV1; + previousSampleState: number; + previousYield: SharedYieldCountsV1; + nextSampleState: number; + firstUsefulAtState: number | null; + firstCriticalAtState: number | null; + throughFirstUseful: SharedYieldCountsV1; + historyComplete: boolean; +} + function cloneJsonValue(value: T): T { if (Array.isArray(value)) { return value.map((item) => item === undefined ? null : cloneJsonValue(item)) as T; @@ -1690,6 +1884,8 @@ export interface SharedSearchCheckpoint { seenChoiceSets: string[]; variableStateCounts: Array<[string, number]>; variableTransitionCounts: Array<[string, number]>; + /** Additive issue-216 semantic-yield counter; absent in older v1 checkpoints. */ + meaningfulVariableTransitions?: number; nodes: Array; deep: number[]; random: number[]; @@ -1724,6 +1920,8 @@ export interface SharedSearchCheckpoint { findingBytes: number; ancestryPayloadBytes: number; peakRetainedMemory: SharedRetainedMemory; + /** Additive deterministic ledger; live heap/RSS observations are never persisted. */ + sharedObservability?: SharedObservabilityCheckpointV1; }; } @@ -1809,6 +2007,85 @@ class CheckpointMulberry32 { } } +const SHARED_RETAINED_MEMORY_KEYS: Array = [ + "pendingStateBytes", "pendingVariableBytes", "activeStateBytes", "activeVariableBytes", + "ancestryBytes", "dedupeBytes", "semanticIndexBytes", "frontierReferenceBytes", + "findingBytes", "totalAccountedBytes", "pendingStates", "retainedNodes", "frontierReferences", +]; + +const SHARED_RETAINED_BYTE_KEYS: Array = [ + "pendingStateBytes", "pendingVariableBytes", "activeStateBytes", "activeVariableBytes", + "ancestryBytes", "dedupeBytes", "semanticIndexBytes", "frontierReferenceBytes", "findingBytes", +]; + +function validSharedYieldCounts(value: SharedYieldCountsV1 | undefined): boolean { + if (!value || !value.critical || !value.intent || !value.authoredCoverage || !value.rawTerritory) return false; + return [ + value.critical.runtimeErrors, value.critical.assertionViolations, + value.intent.goalsReached, value.intent.stagesReached, + value.authoredCoverage.knotsVisited, value.visibleOutcomes, value.semanticTransitions, + value.terminalVariants, value.rawTerritory.transitions, value.rawTerritory.uniqueStates, + value.rawTerritory.dedupeHits, + ].every((count) => Number.isSafeInteger(count) && count >= 0); +} + +function sharedYieldCountsAtMost(left: SharedYieldCountsV1, right: SharedYieldCountsV1): boolean { + return left.critical.runtimeErrors <= right.critical.runtimeErrors + && left.critical.assertionViolations <= right.critical.assertionViolations + && left.intent.goalsReached <= right.intent.goalsReached + && left.intent.stagesReached <= right.intent.stagesReached + && left.authoredCoverage.knotsVisited <= right.authoredCoverage.knotsVisited + && left.visibleOutcomes <= right.visibleOutcomes + && left.semanticTransitions <= right.semanticTransitions + && left.terminalVariants <= right.terminalVariants + && left.rawTerritory.transitions <= right.rawTerritory.transitions + && left.rawTerritory.uniqueStates <= right.rawTerritory.uniqueStates + && left.rawTerritory.dedupeHits <= right.rawTerritory.dedupeHits; +} + +function sharedYieldCountsEqual(left: SharedYieldCountsV1, right: SharedYieldCountsV1): boolean { + return sharedYieldCountsAtMost(left, right) && sharedYieldCountsAtMost(right, left); +} + +function validSharedRetainedMemory(value: SharedRetainedMemory | undefined): boolean { + return !!value && SHARED_RETAINED_MEMORY_KEYS.every( + (key) => Number.isSafeInteger(value[key]) && value[key] >= 0 + ); +} + +function sharedRetainedMemoryAtMost(left: SharedRetainedMemory, right: SharedRetainedMemory): boolean { + return SHARED_RETAINED_MEMORY_KEYS.every((key) => left[key] <= right[key]); +} + +function sharedRetainedCurrentTotalIsExact(value: SharedRetainedMemory): boolean { + const total = SHARED_RETAINED_BYTE_KEYS.reduce((sum, key) => sum + value[key], 0); + return Number.isSafeInteger(total) && value.totalAccountedBytes === total; +} + +function sharedCheckpointYieldCounts( + state: SharedSearchCheckpoint["state"] +): SharedYieldCountsV1 { + return { + critical: { + runtimeErrors: state.runtimeErrors.length, + assertionViolations: 0, + }, + intent: { + goalsReached: 0, + stagesReached: 0, + }, + authoredCoverage: { knotsVisited: state.visitedKnots.length }, + visibleOutcomes: state.visibleOutcomes.length, + semanticTransitions: state.meaningfulVariableTransitions ?? 0, + terminalVariants: state.endings.length, + rawTerritory: { + transitions: state.statesExplored, + uniqueStates: state.seenStates.length, + dedupeHits: state.dedupeHits, + }, + }; +} + /** * Experimental shared-state search. Several deterministic frontier views * select from one global state table, so a state referenced by multiple @@ -1854,6 +2131,10 @@ function validateSharedCheckpoint( state.findingBytes, state.ancestryPayloadBytes, ]; if (nonNegative.some((value) => !Number.isSafeInteger(value) || value < 0)) fail("counters must be non-negative safe integers"); + if (state.meaningfulVariableTransitions !== undefined + && (!Number.isSafeInteger(state.meaningfulVariableTransitions) || state.meaningfulVariableTransitions < 0)) { + fail("meaningful variable-transition count must be a non-negative safe integer"); + } if (state.statesExplored > state.totalGranted) fail("statesExplored cannot exceed totalGranted"); if (!Number.isSafeInteger(state.rngState) || state.rngState < 0 || state.rngState > 0xffffffff) fail("RNG state is malformed"); if (state.finished || state.truncatedBy?.maxStates || state.truncatedBy?.frontier || state.truncatedBy?.memory || state.truncatedBy?.time) { @@ -1877,6 +2158,149 @@ function validateSharedCheckpoint( fail(`active node ${id} is missing serialized state`); } } + if (state.sharedObservability !== undefined) { + const observation = state.sharedObservability; + const currentYield = sharedCheckpointYieldCounts(state); + if (observation.schemaVersion !== 1 + || !Number.isSafeInteger(observation.sampleIntervalStates) || observation.sampleIntervalStates < 1 + || observation.sampleIntervalStates > 10_000_000 + || !Number.isSafeInteger(observation.samplesRecorded) || observation.samplesRecorded < 0 + || !Array.isArray(observation.samples) || observation.samples.length > MAX_SHARED_OBSERVABILITY_SAMPLES + || observation.samplesRecorded < observation.samples.length + || !Number.isSafeInteger(observation.baseState) || observation.baseState < 0 || observation.baseState > state.statesExplored + || !Number.isSafeInteger(observation.previousSampleState) || observation.previousSampleState < observation.baseState + || observation.previousSampleState > state.statesExplored + || !Number.isSafeInteger(observation.nextSampleState) + || observation.nextSampleState !== (Math.floor(state.statesExplored / observation.sampleIntervalStates) + 1) + * observation.sampleIntervalStates + || typeof observation.historyComplete !== "boolean" + || !validSharedYieldCounts(observation.baseYield) + || !validSharedYieldCounts(observation.previousYield) + || !validSharedYieldCounts(observation.throughFirstUseful) + || !validSharedRetainedMemory(state.peakRetainedMemory) + || !sharedYieldCountsAtMost(observation.baseYield, observation.previousYield) + || !sharedYieldCountsAtMost(observation.previousYield, currentYield) + || !sharedYieldCountsAtMost(observation.throughFirstUseful, currentYield) + || (observation.firstUsefulAtState !== null + && (!Number.isSafeInteger(observation.firstUsefulAtState) + || observation.firstUsefulAtState < 0 || observation.firstUsefulAtState > state.statesExplored)) + || (observation.firstCriticalAtState !== null + && (!Number.isSafeInteger(observation.firstCriticalAtState) + || observation.firstCriticalAtState < 0 || observation.firstCriticalAtState > state.statesExplored))) { + fail("shared observability ledger is malformed"); + } + const baseState = observation.baseState; + const interval = observation.sampleIntervalStates; + const expectedSamplesRecorded = Math.floor(state.statesExplored / interval) + - Math.floor(baseState / interval); + const firstExpectedSampleState = (Math.floor(baseState / interval) + 1) * interval; + const lastExpectedSampleState = Math.floor(state.statesExplored / interval) * interval; + if (observation.samplesRecorded !== expectedSamplesRecorded + || observation.baseYield.rawTerritory.transitions !== baseState + || observation.previousYield.rawTerritory.transitions !== observation.previousSampleState + || (observation.historyComplete && baseState !== 0) + || (expectedSamplesRecorded === 0 && observation.samples.length !== 0) + || (expectedSamplesRecorded > 0 && observation.samples.length === 0) + || (expectedSamplesRecorded <= MAX_SHARED_OBSERVABILITY_SAMPLES + && observation.samples.length !== expectedSamplesRecorded) + || (observation.samples.length > 0 + && (observation.samples[0].state !== firstExpectedSampleState + || observation.samples.at(-1)!.state !== lastExpectedSampleState))) { + fail("shared observability cadence ledger is inconsistent"); + } + + const runtimeErrorStates = state.runtimeErrors.map(([, error]) => error?.firstDiscoveredAtState); + if (runtimeErrorStates.some((value) => !Number.isSafeInteger(value) + || value! < 0 || value! > state.statesExplored)) { + fail("runtime error discovery state is malformed"); + } + const expectedFirstCritical = runtimeErrorStates.length > 0 + ? Math.min(...runtimeErrorStates as number[]) + : null; + const currentUseful = usefulYieldObserved(currentYield); + const baseUseful = usefulYieldObserved(observation.baseYield); + const firstUseful = observation.firstUsefulAtState; + const firstCritical = observation.firstCriticalAtState; + const emptyYield = emptySharedYieldCounts(); + if (firstCritical !== expectedFirstCritical + || (firstCritical !== null && (firstUseful === null || firstUseful > firstCritical))) { + fail("shared observability critical milestone is inconsistent"); + } + if (!currentUseful) { + if (firstUseful !== null + || !sharedYieldCountsEqual(observation.throughFirstUseful, emptyYield)) { + fail("shared observability useful milestone is inconsistent"); + } + } else if (firstUseful === null) { + fail("shared observability useful milestone is missing"); + } else if (baseUseful) { + if (firstUseful > baseState + || !sharedYieldCountsEqual(observation.throughFirstUseful, observation.baseYield)) { + fail("shared observability baseline milestone is inconsistent"); + } + } else if (firstUseful <= baseState + || !usefulYieldObserved(observation.throughFirstUseful) + || observation.throughFirstUseful.rawTerritory.transitions !== firstUseful) { + fail("shared observability post-baseline milestone is inconsistent"); + } + + let previousState = observation.baseState; + let previousYield = observation.baseYield; + let previousPeak: SharedRetainedMemory | undefined; + let previousReleasedNodes = 0; + let previousFrontierCompactions = 0; + for (const sample of observation.samples) { + if (!sample || sample.schemaVersion !== 1 || !Number.isSafeInteger(sample.state) + || sample.state <= previousState || sample.state > state.statesExplored + || sample.boundary !== "interval" + || sample.state % interval !== 0 + || !sample.retention || sample.retention.schemaVersion !== 1 + || !validSharedRetainedMemory(sample.retention.current) + || !validSharedRetainedMemory(sample.retention.peak) + || !sharedRetainedCurrentTotalIsExact(sample.retention.current) + || !sharedRetainedMemoryAtMost(sample.retention.current, sample.retention.peak) + || (previousPeak !== undefined && !sharedRetainedMemoryAtMost(previousPeak, sample.retention.peak)) + || !Number.isSafeInteger(sample.retention.releasedNodes) || sample.retention.releasedNodes < 0 + || sample.retention.releasedNodes < previousReleasedNodes + || !Number.isSafeInteger(sample.retention.frontierCompactions) || sample.retention.frontierCompactions < 0 + || sample.retention.frontierCompactions < previousFrontierCompactions + || !sample.yield || sample.yield.schemaVersion !== 1 + || sample.yield.fromStateExclusive !== previousState + || sample.yield.throughState !== sample.state + || !validSharedYieldCounts(sample.yield.delta) + || !validSharedYieldCounts(sample.yield.cumulative) + || sample.yield.cumulative.rawTerritory.transitions !== sample.state + || !sharedYieldCountsAtMost(previousYield, sample.yield.cumulative) + || !sharedYieldCountsEqual( + sample.yield.delta, + subtractSharedYieldCounts(sample.yield.cumulative, previousYield) + )) { + fail("shared observability samples are malformed or out of order"); + } + if (!baseUseful && firstUseful !== null) { + if (sample.state < firstUseful && usefulYieldObserved(sample.yield.cumulative)) { + fail("shared observability useful milestone is later than retained useful yield"); + } + if (sample.state >= firstUseful + && !sharedYieldCountsAtMost(observation.throughFirstUseful, sample.yield.cumulative)) { + fail("shared observability useful milestone exceeds retained cumulative yield"); + } + } + previousState = sample.state; + previousYield = sample.yield.cumulative; + previousPeak = sample.retention.peak; + previousReleasedNodes = sample.retention.releasedNodes; + previousFrontierCompactions = sample.retention.frontierCompactions; + } + if (previousState !== observation.previousSampleState + || !sharedYieldCountsEqual(previousYield, observation.previousYield) + || (previousPeak !== undefined + && !sharedRetainedMemoryAtMost(previousPeak, state.peakRetainedMemory)) + || previousReleasedNodes > state.releasedNodes + || previousFrontierCompactions > state.frontierCompactions) { + fail("shared observability sample cursor does not match its ledger"); + } + } const childCounts = new Array(state.nodes.length).fill(0); for (const node of state.nodes) { if (node?.parent !== null && node?.parent !== undefined) childCounts[node.parent]++; @@ -1958,6 +2382,12 @@ function createSharedEngine( if (maxPendingBytes !== undefined && (!Number.isSafeInteger(maxPendingBytes) || maxPendingBytes < 1)) { throw new RangeError("sharedMaxPendingBytes must be a positive safe integer"); } + const observabilityIntervalStates = opts.sharedObservabilityIntervalStates + ?? DEFAULT_SHARED_OBSERVABILITY_INTERVAL_STATES; + if (!Number.isSafeInteger(observabilityIntervalStates) + || observabilityIntervalStates < 1 || observabilityIntervalStates > 10_000_000) { + throw new RangeError("sharedObservabilityIntervalStates must be an integer from 1 to 10000000"); + } const variableAware = opts.sharedVariableAware ?? false; const goalAware = opts.sharedGoalAware ?? false; if (checkpoint && (variableAware || goalAware || opts.assertions?.length || opts.goals?.length)) { @@ -1986,6 +2416,10 @@ function createSharedEngine( externals: [...externals], }; if (checkpoint) validateSharedCheckpoint(checkpoint, checkpointConfiguration); + if (checkpoint?.state.sharedObservability + && checkpoint.state.sharedObservability.sampleIntervalStates !== observabilityIntervalStates) { + throw new RangeError("Invalid shared checkpoint: observability interval changed"); + } const restored = checkpoint ? JSON.parse(JSON.stringify(checkpoint.state)) as SharedSearchCheckpoint["state"] : undefined; @@ -1999,6 +2433,7 @@ function createSharedEngine( const seenChoiceSets = new Set(restored?.seenChoiceSets ?? []); const variableStateCounts = new Map(restored?.variableStateCounts ?? []); const variableTransitionCounts = new Map(restored?.variableTransitionCounts ?? []); + let meaningfulVariableTransitions = restored?.meaningfulVariableTransitions ?? 0; const nonFunctionKnots = knots.filter((k) => !k.isFunction); const nodes: Array = restored?.nodes.map((node) => node ?? undefined) ?? []; const deep: number[] = restored?.deep ?? []; @@ -2120,8 +2555,159 @@ function createSharedEngine( })); }; + const restoredObservability = restored?.sharedObservability; + let observabilitySamples = restoredObservability?.samples.map((sample) => cloneJsonValue(sample)) ?? []; + let observabilitySamplesRecorded = restoredObservability?.samplesRecorded ?? 0; + let observabilityBaseState = restoredObservability?.baseState ?? statesExplored; + let observabilityBaseYield = restoredObservability + ? cloneSharedYieldCounts(restoredObservability.baseYield) + : emptySharedYieldCounts(); + let previousObservabilityState = restoredObservability?.previousSampleState ?? statesExplored; + let previousObservabilityYield = restoredObservability + ? cloneSharedYieldCounts(restoredObservability.previousYield) + : emptySharedYieldCounts(); + let nextObservabilityState = restoredObservability?.nextSampleState + ?? (Math.floor(statesExplored / observabilityIntervalStates) + 1) * observabilityIntervalStates; + let firstUsefulAtState = restoredObservability?.firstUsefulAtState ?? null; + let firstCriticalAtState = restoredObservability?.firstCriticalAtState ?? null; + let throughFirstUseful = restoredObservability + ? cloneSharedYieldCounts(restoredObservability.throughFirstUseful) + : emptySharedYieldCounts(); + const observabilityHistoryComplete = restoredObservability?.historyComplete ?? !checkpoint; + + const sharedYieldCounts = (): SharedYieldCountsV1 => ({ + critical: { + runtimeErrors: runtimeErrors.size, + assertionViolations: assertions.violationCount(), + }, + intent: { + goalsReached: goals.reachedGoalCount(), + stagesReached: goals.reachedStageCount(), + }, + authoredCoverage: { knotsVisited: visitedKnots.size }, + visibleOutcomes: visibleOutcomes.size, + semanticTransitions: meaningfulVariableTransitions, + terminalVariants: endings.size, + rawTerritory: { + transitions: statesExplored, + uniqueStates: seenStates.size, + dedupeHits, + }, + }); + + const observeYieldMilestones = (current = sharedYieldCounts()): void => { + if (firstUsefulAtState === null && usefulYieldObserved(current)) { + firstUsefulAtState = statesExplored; + throughFirstUseful = cloneSharedYieldCounts(current); + } + if (firstCriticalAtState === null + && (current.critical.runtimeErrors > 0 || current.critical.assertionViolations > 0)) { + firstCriticalAtState = statesExplored; + } + }; + + const rebuildRetainedYieldIntervals = (): void => { + let previousState = observabilityBaseState; + let previousYield = observabilityBaseYield; + for (const sample of observabilitySamples) { + sample.yield.fromStateExclusive = previousState; + sample.yield.delta = subtractSharedYieldCounts(sample.yield.cumulative, previousYield); + previousState = sample.state; + previousYield = sample.yield.cumulative; + } + }; + + const compactObservabilitySamples = (): void => { + if (observabilitySamples.length <= MAX_SHARED_OBSERVABILITY_SAMPLES) return; + const first = observabilitySamples[0]; + const last = observabilitySamples[observabilitySamples.length - 1]; + const interior = observabilitySamples.slice(1, -1).filter((_, index) => index % 2 === 1); + observabilitySamples = [first, ...interior, last]; + rebuildRetainedYieldIntervals(); + }; + + const recordObservabilitySample = ( + boundary: ResourceSampleV1["boundary"] + ): ResourceSampleV1 => { + refreshFindingBytes(); + observeRetainedMemory(); + const currentRetention = retainedMemory(); + const cumulativeYield = sharedYieldCounts(); + observeYieldMilestones(cumulativeYield); + const sample: ResourceSampleV1 = { + schemaVersion: 1, + boundary, + state: statesExplored, + retention: { + schemaVersion: 1, + current: { ...currentRetention }, + peak: { ...peakRetainedMemory }, + releasedNodes, + frontierCompactions, + }, + yield: { + schemaVersion: 1, + fromStateExclusive: previousObservabilityState, + throughState: statesExplored, + delta: subtractSharedYieldCounts(cumulativeYield, previousObservabilityYield), + cumulative: cloneSharedYieldCounts(cumulativeYield), + }, + }; + const latest = observabilitySamples.at(-1); + if (latest?.state === statesExplored && boundary === "termination") { + sample.boundary = latest.boundary === "interval" ? "interval_and_termination" : "termination"; + sample.yield.fromStateExclusive = latest.yield.fromStateExclusive; + sample.yield.delta = cloneSharedYieldCounts(latest.yield.delta); + observabilitySamples[observabilitySamples.length - 1] = sample; + } else { + observabilitySamples.push(sample); + observabilitySamplesRecorded++; + previousObservabilityState = statesExplored; + previousObservabilityYield = cloneSharedYieldCounts(cumulativeYield); + compactObservabilitySamples(); + } + opts.onSharedObservability?.({ + schemaVersion: 1, + pass: foundBy, + runWideState: statesExplored, + sample: cloneJsonValue(sample), + process: observeProcessMemory(currentRetention.totalAccountedBytes), + }); + return sample; + }; + + const maybeRecordObservabilityInterval = (): void => { + if (statesExplored < nextObservabilityState) return; + recordObservabilitySample("interval"); + nextObservabilityState = (Math.floor(statesExplored / observabilityIntervalStates) + 1) + * observabilityIntervalStates; + }; + + const sharedObservabilityTelemetry = (): SharedObservabilityTelemetryV1 => { + const cumulative = sharedYieldCounts(); + return { + schemaVersion: 1, + sampleIntervalStates: observabilityIntervalStates, + samplesRecorded: observabilitySamplesRecorded, + samplesRetained: observabilitySamples.length, + samplesCompacted: Math.max(0, observabilitySamplesRecorded - observabilitySamples.length), + historyComplete: observabilityHistoryComplete, + samples: observabilitySamples.map((sample) => cloneJsonValue(sample)), + yieldSummary: { + schemaVersion: 1, + firstUsefulAtState, + firstCriticalAtState, + throughFirstUseful: cloneSharedYieldCounts(throughFirstUseful), + afterFirstUseful: firstUsefulAtState === null + ? emptySharedYieldCounts() + : subtractSharedYieldCounts(cumulative, throughFirstUseful), + cumulative: cloneSharedYieldCounts(cumulative), + }, + }; + }; + const noteDiscoveryProgress = () => { - if (discoveryCurve.observe(statesExplored, { + const observed = discoveryCurve.observe(statesExplored, { endingsFound: endings.size, runtimeErrorsFound: runtimeErrors.size, knotsVisited: visitedKnots.size, @@ -2130,7 +2716,9 @@ function createSharedEngine( goalsReached: goals.reachedGoalCount(), stagesReached: goals.reachedStageCount(), uniqueStatesObserved: seenStates.size, - })) { + }); + observeYieldMilestones(); + if (observed) { lastDiscoveryAtState = statesExplored; refreshFindingBytes(); observeRetainedMemory(); @@ -2370,6 +2958,24 @@ function createSharedEngine( emitBenchmarkSignals(rootStep.tags, [], 0, opts.onEvidence); const assertions = assertionTracker(session.story, knots, opts.assertions, foundBy); const goals = new GoalTracker(opts.goals ?? [], foundBy); + if (restored && !restoredObservability) { + const baseline = sharedYieldCounts(); + observabilityBaseYield = cloneSharedYieldCounts(baseline); + previousObservabilityYield = cloneSharedYieldCounts(baseline); + const priorFirst = discoveryCurve.summary(statesExplored).firstDiscoveryAtState; + if (usefulYieldObserved(baseline)) { + // Older schema-v1 checkpoints may prove that useful evidence already + // exists without retaining its exact first boundary. In that case the + // reopen state is the conservative milestone and historyComplete stays + // false; no earlier interval history is invented. + firstUsefulAtState = priorFirst ?? statesExplored; + throughFirstUseful = cloneSharedYieldCounts(baseline); + } + firstCriticalAtState = [...runtimeErrors.values()] + .reduce((first, error) => first === null + ? error.firstDiscoveredAtState + : Math.min(first, error.firstDiscoveredAtState), null); + } if (!restored) { session.errors.forEach((message) => recordRuntimeError(runtimeErrors, message, { @@ -2415,6 +3021,10 @@ function createSharedEngine( addNode(rootState, rootVariables, null, undefined, undefined, 0, 1, 0, goals.priority(rootVariables)); } noteDiscoveryProgress(); + // State zero is the baseline for interval deltas. Root findings and the + // initial exact state remain visible in cumulative and early-yield fields. + observabilityBaseYield = sharedYieldCounts(); + previousObservabilityYield = cloneSharedYieldCounts(observabilityBaseYield); } const finishCurrent = () => { @@ -2537,11 +3147,14 @@ function createSharedEngine( const key = variableTransitionKey(change); const previousObservations = variableTransitionCounts.get(key) ?? 0; rarestTransitionWeight = Math.max(rarestTransitionWeight, rarityWeight(previousObservations)); + if (previousObservations === 0 && meaningfulVariableTransition(change)) { + meaningfulVariableTransitions++; + } observeSemanticKey(variableTransitionCounts, key); } - noteDiscoveryProgress(); if (session.errors.length > 0) { + noteDiscoveryProgress(); finishIfLast(); return true; } @@ -2561,12 +3174,14 @@ function createSharedEngine( const key = `${finalText}|${JSON.stringify(nextVariables)}`; if (!endings.has(key)) { recordEnding(endings, visibleOutcomes, key, { path, choiceIndices, firstDiscoveredAtState: statesExplored, finalText, variables: nextVariables, foundBy }, opts.onEvidence); - noteDiscoveryProgress(); } + noteDiscoveryProgress(); finishIfLast(); return true; } + noteDiscoveryProgress(); + if (path.length >= maxDepth) { truncated = true; truncatedBy.maxDepth = true; @@ -2658,6 +3273,7 @@ function createSharedEngine( seenChoiceSets: [...seenChoiceSets], variableStateCounts: [...variableStateCounts.entries()], variableTransitionCounts: [...variableTransitionCounts.entries()], + meaningfulVariableTransitions, nodes: nodes.map((node) => node ?? null), deep: [...deep], random: [...random], @@ -2692,6 +3308,21 @@ function createSharedEngine( findingBytes, ancestryPayloadBytes, peakRetainedMemory: { ...peakRetainedMemory }, + sharedObservability: { + schemaVersion: 1, + sampleIntervalStates: observabilityIntervalStates, + samplesRecorded: observabilitySamplesRecorded, + samples: observabilitySamples.map((sample) => cloneJsonValue(sample)), + baseState: observabilityBaseState, + baseYield: cloneSharedYieldCounts(observabilityBaseYield), + previousSampleState: previousObservabilityState, + previousYield: cloneSharedYieldCounts(previousObservabilityYield), + nextSampleState: nextObservabilityState, + firstUsefulAtState, + firstCriticalAtState, + throughFirstUseful: cloneSharedYieldCounts(throughFirstUseful), + historyComplete: observabilityHistoryComplete, + }, }, }; return cloneJsonValue(value); @@ -2720,6 +3351,7 @@ function createSharedEngine( // Compaction timing is part of frontier order. Tie it to cumulative // work, not caller chunk boundaries, so pause/resume cannot perturb it. if (statesExplored > 0 && statesExplored % 1_000 === 0) compactFrontiers(); + maybeRecordObservabilityInterval(); } return statesExplored - start; }, @@ -2741,7 +3373,7 @@ function createSharedEngine( truncatedBy.maxStates = true; } compactFrontiers(true); - observeRetainedMemory(); + recordObservabilitySample("termination"); return buildResult(); }, telemetry(): PassTelemetry { @@ -2776,6 +3408,7 @@ function createSharedEngine( releasedNodes, frontierCompactions, }, + sharedObservability: sharedObservabilityTelemetry(), variableStatesObserved: variableStateCounts.size, variableTransitionsObserved: variableTransitionCounts.size, rareVariableTransitions: [...variableTransitionCounts.values()].filter((count) => count === 1).length, @@ -2922,6 +3555,12 @@ export function exploreWithGoals( onProgress: opts.onProgress ? (progress) => opts.onProgress!({ ...progress, statesExplored: generalConsumed + progress.statesExplored }) : undefined, + onSharedObservability: opts.onSharedObservability + ? (observation) => opts.onSharedObservability!({ + ...observation, + runWideState: generalConsumed + observation.runWideState, + }) + : undefined, }); const directedConsumed = directed.statesExplored; const merged = mergeExploreResults(general, directed); @@ -3162,8 +3801,8 @@ function createRandomEngine( }); s.warnings.forEach((w) => runtimeWarnings.add(w)); recordKnotCoverage(s); - noteDiscoveryProgress(); if (s.errors.length > 0) { + noteDiscoveryProgress(); walkPath = null; walkChoiceIndices = null; return; @@ -3189,12 +3828,13 @@ function createRandomEngine( const key = finalText + "|" + JSON.stringify(stateVariables); if (!endings.has(key)) { recordEnding(endings, visibleOutcomes, key, { path: [...walkPath], choiceIndices: [...walkChoiceIndices!], firstDiscoveredAtState: statesExplored, finalText, variables: stateVariables, foundBy }, opts.onEvidence); - noteDiscoveryProgress(); } + noteDiscoveryProgress(); walkPath = null; walkChoiceIndices = null; return; } + noteDiscoveryProgress(); const nextState = s.story.state.ToJson(); const loopRisk = noteLoopControl(nextState, stateVariables, walkPath, walkChoiceIndices!, statesExplored); if (loopRisk) { @@ -3598,9 +4238,11 @@ function createBeamEngine( }); s.warnings.forEach((w) => runtimeWarnings.add(w)); recordKnotCoverage(s); - noteDiscoveryProgress(); const newKnots = visitedKnots.size - knotsBefore; - if (s.errors.length > 0) return true; + if (s.errors.length > 0) { + noteDiscoveryProgress(); + return true; + } const ended = !s.story.canContinue && s.story.currentChoices.length === 0; const stateVariables = extractVariables(s.story); @@ -3618,10 +4260,11 @@ function createBeamEngine( const key = finalText + "|" + JSON.stringify(stateVariables); if (!endings.has(key)) { recordEnding(endings, visibleOutcomes, key, { path, choiceIndices, firstDiscoveredAtState: statesExplored, finalText, variables: stateVariables, foundBy }, opts.onEvidence); - noteDiscoveryProgress(); } + noteDiscoveryProgress(); return true; } + noteDiscoveryProgress(); if (path.length >= maxDepth) { truncated = true; truncatedBy.maxDepth = true; diff --git a/src/machine-output.ts b/src/machine-output.ts index ac7c117..00e798c 100644 --- a/src/machine-output.ts +++ b/src/machine-output.ts @@ -167,6 +167,177 @@ function executionSummary(value: unknown) { }; } +const MAX_COMPACT_SHARED_OBSERVABILITY_PASSES = 8; +const MAX_SHARED_OBSERVABILITY_SAMPLES = 128; +const SHARED_RETAINED_MEMORY_FIELDS = [ + "pendingStateBytes", "pendingVariableBytes", "activeStateBytes", "activeVariableBytes", + "ancestryBytes", "dedupeBytes", "semanticIndexBytes", "frontierReferenceBytes", + "findingBytes", "totalAccountedBytes", "pendingStates", "retainedNodes", "frontierReferences", +] as const; + +function nonNegativeSafeInteger(value: unknown): value is number { + return typeof value === "number" && Number.isSafeInteger(value) && value >= 0; +} + +function sharedPass(value: unknown): string | undefined { + if (typeof value !== "string" || value.length > 64) return undefined; + const match = /^shared:(?:deep-novelty|variable-aware|goal-directed)-v1:seed=(\d{1,10})$/.exec(value); + if (!match) return undefined; + const seed = Number(match[1]); + return Number.isSafeInteger(seed) && seed >= 0 && seed <= 0xffffffff ? value : undefined; +} + +function sharedRetainedMemory(value: unknown): Record | undefined { + const memory = record(value); + if (!memory) return undefined; + const projected: Record = {}; + for (const field of SHARED_RETAINED_MEMORY_FIELDS) { + const count = memory[field]; + if (!nonNegativeSafeInteger(count)) return undefined; + projected[field] = count; + } + return projected; +} + +function sharedYieldCounts(value: unknown): Record | undefined { + const counts = record(value); + const critical = record(counts?.critical); + const intent = record(counts?.intent); + const authoredCoverage = record(counts?.authoredCoverage); + const rawTerritory = record(counts?.rawTerritory); + if (!counts || !critical || !intent || !authoredCoverage || !rawTerritory) return undefined; + const numeric = [ + critical.runtimeErrors, critical.assertionViolations, + intent.goalsReached, intent.stagesReached, + authoredCoverage.knotsVisited, counts.visibleOutcomes, counts.semanticTransitions, + counts.terminalVariants, rawTerritory.transitions, rawTerritory.uniqueStates, + rawTerritory.dedupeHits, + ]; + if (!numeric.every(nonNegativeSafeInteger)) return undefined; + return { + critical: { + runtimeErrors: critical.runtimeErrors, + assertionViolations: critical.assertionViolations, + }, + intent: { + goalsReached: intent.goalsReached, + stagesReached: intent.stagesReached, + }, + authoredCoverage: { knotsVisited: authoredCoverage.knotsVisited }, + visibleOutcomes: counts.visibleOutcomes, + semanticTransitions: counts.semanticTransitions, + terminalVariants: counts.terminalVariants, + rawTerritory: { + transitions: rawTerritory.transitions, + uniqueStates: rawTerritory.uniqueStates, + dedupeHits: rawTerritory.dedupeHits, + }, + }; +} + +function sharedResourceSample(value: unknown): Record | undefined { + const sample = record(value); + const retention = record(sample?.retention); + const interval = record(sample?.yield); + const current = sharedRetainedMemory(retention?.current); + const peak = sharedRetainedMemory(retention?.peak); + const delta = sharedYieldCounts(interval?.delta); + const cumulative = sharedYieldCounts(interval?.cumulative); + if (!sample || sample.schemaVersion !== 1 + || typeof sample.boundary !== "string" + || !["interval", "termination", "interval_and_termination"].includes(sample.boundary) + || !nonNegativeSafeInteger(sample.state) + || !retention || retention.schemaVersion !== 1 || !current || !peak + || !nonNegativeSafeInteger(retention.releasedNodes) + || !nonNegativeSafeInteger(retention.frontierCompactions) + || !interval || interval.schemaVersion !== 1 + || !nonNegativeSafeInteger(interval.fromStateExclusive) + || !nonNegativeSafeInteger(interval.throughState) + || interval.fromStateExclusive > interval.throughState + || interval.throughState !== sample.state + || !delta || !cumulative) { + return undefined; + } + return { + schemaVersion: 1, + boundary: sample.boundary, + state: sample.state, + retention: { + schemaVersion: 1, + current, + peak, + releasedNodes: retention.releasedNodes, + frontierCompactions: retention.frontierCompactions, + }, + yield: { + schemaVersion: 1, + fromStateExclusive: interval.fromStateExclusive, + throughState: interval.throughState, + delta, + cumulative, + }, + }; +} + +function sharedYieldSummary(value: unknown): Record | undefined { + const summary = record(value); + const throughFirstUseful = sharedYieldCounts(summary?.throughFirstUseful); + const afterFirstUseful = sharedYieldCounts(summary?.afterFirstUseful); + const cumulative = sharedYieldCounts(summary?.cumulative); + const firstUsefulAtState = summary?.firstUsefulAtState; + const firstCriticalAtState = summary?.firstCriticalAtState; + if (!summary || summary.schemaVersion !== 1 + || (firstUsefulAtState !== null && !nonNegativeSafeInteger(firstUsefulAtState)) + || (firstCriticalAtState !== null && !nonNegativeSafeInteger(firstCriticalAtState)) + || !throughFirstUseful || !afterFirstUseful || !cumulative) { + return undefined; + } + return { + schemaVersion: 1, + firstUsefulAtState, + firstCriticalAtState, + throughFirstUseful, + afterFirstUseful, + cumulative, + }; +} + +function sharedObservabilitySummary(value: unknown) { + const summaries: Record[] = []; + for (const passValue of array(value)) { + if (summaries.length >= MAX_COMPACT_SHARED_OBSERVABILITY_PASSES) break; + const pass = record(passValue); + const telemetry = record(pass?.sharedObservability); + const projectedPass = sharedPass(pass?.pass); + if (!pass || !telemetry || !projectedPass || telemetry.schemaVersion !== 1 + || !nonNegativeSafeInteger(telemetry.sampleIntervalStates) + || telemetry.sampleIntervalStates < 1 || telemetry.sampleIntervalStates > 10_000_000 + || !nonNegativeSafeInteger(telemetry.samplesRecorded) + || !nonNegativeSafeInteger(telemetry.samplesRetained) + || telemetry.samplesRetained > MAX_SHARED_OBSERVABILITY_SAMPLES + || !nonNegativeSafeInteger(telemetry.samplesCompacted) + || telemetry.samplesRecorded - telemetry.samplesRetained !== telemetry.samplesCompacted + || typeof telemetry.historyComplete !== "boolean") continue; + const samples = array(telemetry.samples); + if (samples.length !== telemetry.samplesRetained) continue; + const latest = samples.length ? sharedResourceSample(samples.at(-1)) : undefined; + const yieldSummary = sharedYieldSummary(telemetry.yieldSummary); + if ((samples.length && !latest) || !yieldSummary) continue; + summaries.push({ + pass: projectedPass, + schemaVersion: 1, + sampleIntervalStates: telemetry.sampleIntervalStates, + samplesRecorded: telemetry.samplesRecorded, + samplesRetained: telemetry.samplesRetained, + samplesCompacted: telemetry.samplesCompacted, + historyComplete: telemetry.historyComplete, + ...(latest ? { latestSample: latest } : {}), + yieldSummary, + }); + } + return summaries.length ? summaries : undefined; +} + function explorationSummary(explore: Record | undefined) { if (!explore) return undefined; const assertionResults = array(explore.assertionResults).map(record).filter(Boolean) as Record[]; @@ -174,6 +345,7 @@ function explorationSummary(explore: Record | undefined) { (total, result) => total + array(result.violations).length, 0 ); + const sharedObservability = sharedObservabilitySummary(explore.passes); return { statesExplored: explore.statesExplored, runtimeErrorCount: array(explore.runtimeErrors).length, @@ -191,6 +363,7 @@ function explorationSummary(explore: Record | undefined) { exhaustive: explore.exhaustive === true, limits: explore.limits, ...(explore.execution ? { execution: executionSummary(explore.execution) } : {}), + ...(sharedObservability ? { sharedObservability } : {}), }; } diff --git a/src/resource-guards.ts b/src/resource-guards.ts index ba0f985..93494ed 100644 --- a/src/resource-guards.ts +++ b/src/resource-guards.ts @@ -19,6 +19,40 @@ export interface ResourceGuards { peakMemoryBytes: () => number; } +/** + * Process-scoped memory observed from Node. These values are environmental + * facts, not deterministic search state or an ownership accounting model. + */ +export interface ProcessMemoryObservationV1 { + schemaVersion: 1; + scope: "process"; + heapUsedBytes: number; + heapTotalBytes: number; + rssBytes: number; + externalBytes: number; + arrayBuffersBytes: number; + /** Deterministic logical estimate used only for the adjacent comparison. */ + comparedLogicalAccountedBytes: number; + /** heapUsed minus the logical estimate; observational, not owner attribution. */ + unattributedBytes: number; +} + +export function observeProcessMemory(accountedLogicalBytes = 0): ProcessMemoryObservationV1 { + const usage = process.memoryUsage(); + const comparedLogicalAccountedBytes = Math.max(0, accountedLogicalBytes); + return { + schemaVersion: 1, + scope: "process", + heapUsedBytes: usage.heapUsed, + heapTotalBytes: usage.heapTotal, + rssBytes: usage.rss, + externalBytes: usage.external, + arrayBuffersBytes: usage.arrayBuffers ?? 0, + comparedLogicalAccountedBytes, + unattributedBytes: usage.heapUsed - comparedLogicalAccountedBytes, + }; +} + /** Build the same pre-OOM and wall-clock guards for every execution surface. */ export function createResourceGuards(options: ResourceGuardOptions = {}): ResourceGuards { const memoryCapBytes = options.maxMemoryMb === undefined diff --git a/src/terminal-progress.ts b/src/terminal-progress.ts index b0a744e..a41d4d2 100644 --- a/src/terminal-progress.ts +++ b/src/terminal-progress.ts @@ -1,3 +1,5 @@ +import type { SharedResourceObservationV1 } from "./explore"; + export type HumanProgressPhase = "compile" | "source_scan" | "explore" | "min_repro" | "report"; export type ProgressStatus = "complete" | "cancelled" | "error"; export type ProgressStopReason = @@ -17,7 +19,7 @@ export type ProgressStopReason = export type ProgressOutcome = "clean" | "issues_found" | "review_required" | "compile_error"; export interface HumanProgressEvent { - type: "run_start" | "phase_start" | "progress" | "discovery" | "phase_end" | "run_end"; + type: "run_start" | "phase_start" | "progress" | "discovery" | "resource" | "phase_end" | "run_end"; phase?: HumanProgressPhase; pass?: string; elapsedMs: number; @@ -41,6 +43,8 @@ export interface HumanProgressEvent { stopReason?: ProgressStopReason; outcome?: ProgressOutcome; exhaustive?: boolean; + /** Shared-search only; contains counts/bytes but no story content or identities. */ + sharedObservability?: SharedResourceObservationV1; } export interface TerminalWriter { @@ -133,6 +137,12 @@ export class HumanProgressRenderer { event.discoveries.knotsVisited ? `+${event.discoveries.knotsVisited} knot${event.discoveries.knotsVisited === 1 ? "" : "s"}` : "", ].filter(Boolean).join(", "); line = `Found ${found || "new story evidence"} at ${event.statesExplored.toLocaleString()} work states | ${duration(event.elapsedMs)} elapsed`; + } else if (event.type === "resource" && event.sharedObservability) { + const observation = event.sharedObservability; + const retained = observation.sample.retention.current.totalAccountedBytes / 1048576; + const heap = observation.process.heapUsedBytes / 1048576; + const rss = observation.process.rssBytes / 1048576; + line = `Shared search: ${event.statesExplored.toLocaleString()} states | logical ${retained.toFixed(1)} MiB, heap ${heap.toFixed(1)} MiB, RSS ${rss.toFixed(1)} MiB | ${duration(event.elapsedMs)} elapsed`; } else if (event.phase === "explore" || event.phase === "min_repro" || event.type === "progress") { const percent = event.stateBudget ? Math.floor((event.statesExplored / event.stateBudget) * 100) : 0; const throughput = this.rate >= 1 ? ` ${Math.round(this.rate).toLocaleString()} states/s` : ""; diff --git a/test/concurrent-portfolio.test.js b/test/concurrent-portfolio.test.js index ce15f47..502d8c7 100644 --- a/test/concurrent-portfolio.test.js +++ b/test/concurrent-portfolio.test.js @@ -199,6 +199,19 @@ test("concurrent workers stream monotonic aggregate budget progress before the f assert.strictEqual(updates.at(-1).runtimeErrorsFound, result.runtimeErrors.length); }); +test("concurrent workers exclude shared-only observation callbacks from worker data", async () => { + const compiled = await story(GRID); + let observations = 0; + const result = explorePortfolioConcurrent(compiled.storyJson, compiled.knots, [], { + maxStates: 100, + concurrency: 2, + memoryCapBytes: ONE_GIB, + onSharedObservability: () => observations++, + }); + assert.strictEqual(result.execution.mode, "concurrent"); + assert.strictEqual(observations, 0); +}); + test("concurrent portfolio falls back before spawning when global memory cannot safely fund two workers", async () => { const compiled = await story(GRID); const result = explorePortfolioConcurrent(compiled.storyJson, compiled.knots, [], { diff --git a/test/inkcheck.test.js b/test/inkcheck.test.js index 016f74f..527d39e 100644 --- a/test/inkcheck.test.js +++ b/test/inkcheck.test.js @@ -95,6 +95,7 @@ const CLI = path.join(__dirname, "..", "dist", "cli.js"); const CHECKPOINT_SAVE_WORKER = path.join(__dirname, "fixtures", "checkpoint-save-worker.js"); const ROOT = path.join(__dirname, ".."); const SEARCH_FIXTURES = path.join(__dirname, "fixtures", "search"); +const LOW_DEDUP_WIDE = path.join(SEARCH_FIXTURES, "low-dedup-wide.ink"); const INSPECT_PROJECT = path.join(__dirname, "fixtures", "inspect", "project.ink"); const DUPLICATE_CHOICE_TEXT = path.join(__dirname, "fixtures", "duplicate-choice-text.ink"); const ASSERTION_STORY = path.join(__dirname, "fixtures", "assertions.ink"); @@ -584,6 +585,49 @@ test("bounded goal search reaches targets with exact witnesses and protects gene assert.strictEqual(result.goalBudget.directedGranted, 25); }); +test("additive goal resource observations separate run-wide progress from pass-local samples", async () => { + const compiled = await compile(LOW_DEDUP_WIDE); + const knots = scanKnots(LOW_DEDUP_WIDE); + const goals = [{ + id: "unreachable_depth", + condition: { left: { variable: "depth" }, operator: "==", right: { literal: 999 } }, + }]; + for (const baseline of ["shared", "portfolio"]) { + const observations = []; + const result = exploreWithGoals(compiled.storyJson, knots, [], { + maxDepth: 150, + maxStates: 100, + goalMaxStates: 50, + goals, + sharedObservabilityIntervalStates: 25, + onSharedObservability: (observation) => observations.push(observation), + }, baseline); + assert.strictEqual(result.statesExplored, 150, baseline); + assert.deepStrictEqual(result.goalBudget, { + generalGranted: 100, + generalConsumed: 100, + directedGranted: 50, + directedConsumed: 50, + }, baseline); + assert.ok(observations.every((observation, index) => ( + index === 0 || observation.runWideState >= observations[index - 1].runWideState + )), baseline); + const directed = observations.filter((observation) => observation.pass.startsWith("shared:goal-directed")); + assert.deepStrictEqual(directed.map((observation) => observation.sample.state), [25, 50, 50], baseline); + assert.deepStrictEqual(directed.map((observation) => observation.runWideState), [125, 150, 150], baseline); + if (baseline === "shared") { + assert.deepStrictEqual( + observations.map((observation) => observation.sample.state), + [25, 50, 75, 100, 100, 25, 50, 50] + ); + assert.deepStrictEqual( + observations.map((observation) => observation.runWideState), + [25, 50, 75, 100, 100, 125, 150, 150] + ); + } + } +}); + test("every exploration engine records goals reached during general exploration", async () => { const compiled = await compile(ASSERTION_STORY); const knots = scanKnots(ASSERTION_STORY); @@ -3483,6 +3527,84 @@ test("discovery curves separate assertion, goal, stage, and visible-outcome valu assert.ok(latest.visibleOutcomes < report.endingsFound.length); }); +test("every engine samples assertion and goal discoveries before a bounded nonterminal stop", async () => { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), "inkcheck-discovery-boundary-")); + const story = path.join(directory, "story.ink"); + fs.writeFileSync(story, `VAR ready = false +VAR score = 1 + +Start. +* [Trigger] + ~ ready = true + ~ score = -1 + Still running. + * [Finish] + -> END +`); + try { + const compiled = await compile(story); + assert.strictEqual(compiled.success, true); + const knots = scanKnots(story); + const assertions = [{ + id: "score_nonnegative", + when: "always", + condition: { left: { variable: "score" }, operator: ">=", right: { literal: 0 } }, + }]; + const goals = [{ + id: "ready", + condition: { left: { variable: "ready" }, operator: "==", right: { literal: true } }, + }]; + const engines = [ + ["dfs", explore], + ["shared", exploreShared], + ["random", exploreRandom], + ["beam", exploreBeam], + ]; + for (const [name, run] of engines) { + const result = run(compiled.storyJson, knots, [], { + maxDepth: 10, + maxStates: 1, + seed: 7, + assertions, + goals, + }); + const latest = result.passes[0].discoveryCurve.at(-1); + assert.strictEqual(result.truncatedBy.maxStates, true, name); + assert.strictEqual(latest.state, 1, name); + assert.strictEqual(latest.assertionViolations, 1, name); + assert.strictEqual(latest.goalsReached, 1, name); + assert.strictEqual(result.passes[0].lastDiscoveryAtState, 1, name); + } + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +}); + +test("visible outcomes are discovery boundaries and older curve checkpoints migrate additively", () => { + const counts = (visibleOutcomes) => ({ + endingsFound: 0, + runtimeErrorsFound: 0, + knotsVisited: 0, + visibleOutcomes, + assertionViolations: 0, + goalsReached: 0, + stagesReached: 0, + uniqueStatesObserved: 0, + }); + const recorder = new DiscoveryCurveRecorder(); + assert.strictEqual(recorder.observe(0, counts(0)), false); + assert.strictEqual(recorder.observe(1, counts(1)), true); + assert.strictEqual(recorder.result().at(-1).visibleOutcomes, 1); + + const legacyCheckpoint = recorder.checkpoint(); + delete legacyCheckpoint.countedVisibleOutcomes; + legacyCheckpoint.previousTotal -= legacyCheckpoint.previousCounts.visibleOutcomes; + const restored = new DiscoveryCurveRecorder(legacyCheckpoint); + assert.strictEqual(restored.observe(2, counts(1)), false); + assert.strictEqual(restored.observe(3, counts(2)), true); + assert.strictEqual(restored.checkpoint().countedVisibleOutcomes, true); +}); + test("discovery summaries distinguish no evidence from a late recovery", async () => { const emptyCompiled = await compile(NO_DISCOVERY_BEFORE_DEPTH); const empty = explore(emptyCompiled.storyJson, scanKnots(NO_DISCOVERY_BEFORE_DEPTH), [], { @@ -4424,6 +4546,82 @@ test("CLI streams versioned progress to stderr without changing the final JSON r assert.strictEqual(final.outcome, "clean"); }); +test("shared CLI progress emits bounded resource observations without changing canonical JSON", () => { + const story = path.join(SEARCH_FIXTURES, "low-dedup-wide.ink"); + const args = [CLI, story, "--search=shared", "--max-states", "100", "--no-min-repro", "--json"]; + const plain = spawnSync(process.execPath, [...args, "--progress=off"], { encoding: "utf8" }); + const streamed = spawnSync(process.execPath, [...args, "--progress=ndjson"], { encoding: "utf8" }); + assert.strictEqual(streamed.status, plain.status, streamed.stderr); + assert.strictEqual(streamed.stdout, plain.stdout); + + const report = JSON.parse(streamed.stdout); + const events = streamed.stderr.trim().split("\n").map((line) => JSON.parse(line)); + const resources = events.filter((event) => event.type === "resource"); + assert.strictEqual(resources.length, 1); + const observation = resources[0].sharedObservability; + assert.strictEqual(observation.schemaVersion, 1); + assert.match(observation.pass, /^shared:/); + assert.strictEqual(observation.sample.boundary, "termination"); + assert.strictEqual(observation.sample.state, report.explore.statesExplored); + assert.strictEqual(observation.runWideState, report.explore.statesExplored); + assert.ok(observation.sample.retention.current.totalAccountedBytes > 0); + assert.strictEqual(observation.process.scope, "process"); + assert.ok(observation.process.heapUsedBytes > 0); + assert.ok(observation.process.rssBytes > 0); + assert.strictEqual( + observation.process.comparedLogicalAccountedBytes, + observation.sample.retention.current.totalAccountedBytes + ); + assert.deepStrictEqual(events.at(-1).sharedObservability, observation); + assert.doesNotMatch(JSON.stringify(resources), /path_code|wide tree leaf|"Left"|"Center"|"Right"/i); + assert.doesNotMatch(streamed.stdout, /heapUsedBytes|heapTotalBytes|rssBytes|unattributedBytes/); +}); + +test("CLI resource progress stays global across additive shared-goal work", () => { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "inkcheck-goal-resource-progress-")); + try { + fs.copyFileSync(LOW_DEDUP_WIDE, path.join(tmp, "story.ink")); + fs.writeFileSync(path.join(tmp, "inkcheck.yml"), require("yaml").stringify({ + schemaVersion: 1, + entrypoint: "story.ink", + ci: { maxDepth: 150, maxStates: 100, goalMaxStates: 50, minRepro: false }, + goals: [{ + id: "unreachable_depth", + condition: { left: { variable: "depth" }, operator: "==", right: { literal: 999 } }, + }], + })); + for (const baseline of ["shared", "portfolio"]) { + const checked = spawnSync(process.execPath, [ + CLI, `--search=${baseline}`, "--json", "--progress=ndjson", + ], { cwd: tmp, encoding: "utf8" }); + assert.strictEqual(checked.status, 0, checked.stderr); + const report = JSON.parse(checked.stdout); + const events = checked.stderr.trim().split("\n").map((line) => JSON.parse(line)); + const resources = events.filter((event) => event.type === "resource"); + assert.ok(resources.length > 0, baseline); + assert.ok(resources.every((event, index) => ( + index === 0 || event.statesExplored >= resources[index - 1].statesExplored + )), baseline); + assert.ok(resources.every((event) => ( + event.statesExplored === event.sharedObservability.runWideState + )), baseline); + const directed = resources.filter((event) => ( + event.sharedObservability.pass.startsWith("shared:goal-directed") + )); + assert.ok(directed.length > 0, baseline); + assert.strictEqual(directed.at(-1).sharedObservability.sample.state, 50, baseline); + assert.strictEqual(directed.at(-1).sharedObservability.runWideState, 150, baseline); + assert.strictEqual(report.explore.statesExplored, 150, baseline); + assert.strictEqual(events.at(-1).type, "run_end", baseline); + assert.strictEqual(events.at(-1).statesExplored, 150, baseline); + assert.strictEqual(events.at(-1).sharedObservability.sample.state, 50, baseline); + assert.strictEqual(events.at(-1).sharedObservability.runWideState, 150, baseline); + } + } finally { + fs.rmSync(tmp, { recursive: true, force: true }); + } +}); + test("--json-stream emits replayable evidence and a bounded terminal summary", () => { const proc = spawnSync( process.execPath, @@ -4445,9 +4643,68 @@ test("--json-stream emits replayable evidence and a bounded terminal summary", ( assert.strictEqual(terminal.explore.endingsFound, endings.length); assert.strictEqual(terminal.evidence.endingsEmitted, endings.length); assert.ok(terminal.resources.memorySearchLimitBytes < terminal.resources.memoryCapBytes); + assert.strictEqual(terminal.resources.observedProcessAtTermination.scope, "process"); + assert.ok(terminal.resources.observedProcessAtTermination.heapUsedBytes > 0); + assert.ok(terminal.resources.observedProcessAtTermination.rssBytes > 0); assert.ok(Buffer.byteLength(JSON.stringify(terminal)) < 16 * 1024); }); +test("--json-stream samples process memory at termination independently of progress mode", () => { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "inkcheck-terminal-memory-")); + const preload = path.join(tmp, "memory-calls.cjs"); + const heapUsedBase = 3_000_000; + fs.writeFileSync(preload, [ + 'const fs = require("node:fs");', + "let calls = 0;", + "process.memoryUsage = () => {", + " calls++;", + " return {", + " rss: 1_000_000 + calls,", + " heapTotal: 2_000_000 + calls,", + ` heapUsed: ${heapUsedBase} + calls,`, + " external: 4_000_000 + calls,", + " arrayBuffers: 5_000_000 + calls,", + " };", + "};", + "process.on(\"exit\", () => {", + " fs.writeFileSync(process.env.INKCHECK_MEMORY_CALLS_PATH, String(calls));", + "});", + ].join("\n")); + + try { + const observations = []; + for (const progress of ["off", "ndjson"]) { + const callsPath = path.join(tmp, `${progress}-calls.txt`); + const proc = spawnSync(process.execPath, [ + "--require", preload, CLI, CLEAN_BRANCH, + "--search=shared", "--max-states", "100", "--max-memory", "96", + "--concurrency", "1", "--json-stream", `--progress=${progress}`, + ], { + encoding: "utf8", + env: { ...process.env, INKCHECK_MEMORY_CALLS_PATH: callsPath }, + }); + assert.strictEqual(proc.status, 0, proc.stderr); + const terminal = proc.stdout.trim().split("\n").map((line) => JSON.parse(line)).at(-1); + const memoryCalls = Number(fs.readFileSync(callsPath, "utf8")); + const observed = terminal.resources.observedProcessAtTermination; + assert.strictEqual(observed.heapUsedBytes, heapUsedBase + memoryCalls, progress); + observations.push(observed); + + if (progress === "ndjson") { + const progressEvents = proc.stderr.trim().split("\n").map((line) => JSON.parse(line)); + assert.ok(progressEvents.some((event) => event.type === "phase_start" && event.phase === "min_repro")); + assert.ok(progressEvents.some((event) => event.type === "phase_end" && event.phase === "min_repro")); + } + } + assert.strictEqual( + observations[0].comparedLogicalAccountedBytes, + observations[1].comparedLogicalAccountedBytes + ); + } finally { + fs.rmSync(tmp, { recursive: true, force: true }); + } +}); + test("--json-stream requires a single worker and rejects monolithic report persistence", () => { const concurrent = spawnSync( process.execPath, @@ -4642,10 +4899,31 @@ test("NDJSON progress contract docs stay linked and privacy-focused", () => { assert.match(docs, /work-budget progress, not story coverage/); assert.match(docs, /"type":"progress"/); assert.match(docs, /"type":"discovery"/); + assert.match(docs, /"type":"resource"/); assert.match(docs, /"type":"run_end"/); assert.match(docs, /must not contain:[\s\S]*story source text[\s\S]*choice prose[\s\S]*variable names or values/); }); +test("shared observability contract is linked, packaged, and explicit about its partial boundary", () => { + const readme = fs.readFileSync(path.join(ROOT, "README.md"), "utf8"); + const docs = fs.readFileSync(path.join(ROOT, "docs", "shared-search-observability.md"), "utf8"); + const packageJson = JSON.parse(fs.readFileSync(path.join(ROOT, "package.json"), "utf8")); + assert.match(readme, /docs\/shared-search-observability\.md/); + assert.ok(packageJson.files.includes("docs/shared-search-observability.md")); + assert.match(docs, /issue #216/); + assert.match(docs, /which remains open/); + assert.match(docs, /not the complete long-run resource policy/); + assert.match(docs, /only sampling boundaries in this slice/); + assert.match(docs, /Checkpoint save\/resume, discovery events, memory or frontier pressure/); + assert.match(docs, /never produces a weighted usefulness score/); + assert.match(docs, /not complete owner attribution/); + assert.match(docs, /rediscovery identities, throughput, retained GiB-minutes/); + assert.match(docs, /excluded from shared checkpoints, canonical JSON reports/); + assert.match(docs, /runWideState/); + assert.match(docs, /nested pass-local sample/); + assert.match(docs, /historyComplete: false/); +}); + test("Rules That Matter contract stays linked, packaged, and bounded", () => { const fs = require("node:fs"); const readme = fs.readFileSync(path.join(ROOT, "README.md"), "utf8"); diff --git a/test/machine-output.test.js b/test/machine-output.test.js index de47606..cb84cce 100644 --- a/test/machine-output.test.js +++ b/test/machine-output.test.js @@ -92,6 +92,85 @@ function largeReport() { }; } +function sharedRetention(offset = 0) { + const retention = { + pendingStateBytes: 100 + offset, + pendingVariableBytes: 101 + offset, + activeStateBytes: 102 + offset, + activeVariableBytes: 103 + offset, + ancestryBytes: 104 + offset, + dedupeBytes: 105 + offset, + semanticIndexBytes: 106 + offset, + frontierReferenceBytes: 107 + offset, + findingBytes: 108 + offset, + totalAccountedBytes: 0, + pendingStates: 10 + offset, + retainedNodes: 11 + offset, + frontierReferences: 12 + offset, + }; + retention.totalAccountedBytes = Object.entries(retention) + .filter(([key]) => key.endsWith("Bytes") && key !== "totalAccountedBytes") + .reduce((total, [, value]) => total + value, 0); + return retention; +} + +function sharedYield(offset = 0) { + return { + critical: { runtimeErrors: offset, assertionViolations: offset }, + intent: { goalsReached: offset, stagesReached: offset }, + authoredCoverage: { knotsVisited: offset }, + visibleOutcomes: offset, + semanticTransitions: offset, + terminalVariants: offset, + rawTerritory: { transitions: offset, uniqueStates: offset, dedupeHits: offset }, + }; +} + +function sharedSample(state, boundary, offset) { + return { + schemaVersion: 1, + boundary, + state, + retention: { + schemaVersion: 1, + current: sharedRetention(offset), + peak: sharedRetention(offset + 10), + releasedNodes: offset, + frontierCompactions: offset, + }, + yield: { + schemaVersion: 1, + fromStateExclusive: state - 2_000, + throughState: state, + delta: sharedYield(1), + cumulative: sharedYield(offset), + }, + }; +} + +function validSharedObservability() { + return { + schemaVersion: 1, + sampleIntervalStates: 10_000, + samplesRecorded: 2, + samplesRetained: 2, + samplesCompacted: 0, + historyComplete: true, + samples: [ + sharedSample(10_000, "interval", 1), + sharedSample(12_000, "termination", 2), + ], + yieldSummary: { + schemaVersion: 1, + firstUsefulAtState: 4, + firstCriticalAtState: null, + throughFirstUseful: sharedYield(1), + afterFirstUseful: sharedYield(1), + cumulative: sharedYield(2), + }, + }; +} + test("default machine detail stays bounded and keeps response truncation separate from search truncation", () => { const report = largeReport(); const projected = projectMachineReport(report); @@ -151,6 +230,74 @@ test("summary, standard, and full detail form an explicit privacy and drill-down ]); }); +test("compact shared observability recursively whitelists its latest sample and yield summary", () => { + const report = largeReport(); + const telemetry = validSharedObservability(); + const expectedLatest = structuredClone(telemetry.samples.at(-1)); + const expectedYieldSummary = structuredClone(telemetry.yieldSummary); + const secret = "PRIVATE NESTED OBSERVABILITY SECRET ".repeat(10_000); + report.explore.passes[0].pass = "shared:deep-novelty-v1:seed=7"; + report.explore.passes[0].sharedObservability = telemetry; + telemetry.privateFuture = { secret, nested: [{ owner: secret }] }; + telemetry.samples.at(-1).privateFuture = secret; + telemetry.samples.at(-1).retention.current.privateOwnerMap = [secret]; + telemetry.samples.at(-1).retention.peak.privateFuturePeak = { secret }; + telemetry.samples.at(-1).yield.delta.critical.privateFinding = secret; + telemetry.samples.at(-1).yield.cumulative.rawTerritory.privateIdentity = secret; + telemetry.yieldSummary.privateScore = secret; + telemetry.yieldSummary.throughFirstUseful.intent.privateGoal = { secret }; + telemetry.yieldSummary.cumulative.rawTerritory.privateFuture = [secret]; + telemetry.observedProcess = { rssBytes: 999_999_999, heapUsedBytes: 888_888_888, secret }; + const projected = projectMachineReport(report, "summary"); + assert.deepStrictEqual(projected.explore.sharedObservability, [{ + pass: "shared:deep-novelty-v1:seed=7", + schemaVersion: 1, + sampleIntervalStates: 10_000, + samplesRecorded: 2, + samplesRetained: 2, + samplesCompacted: 0, + historyComplete: true, + latestSample: expectedLatest, + yieldSummary: expectedYieldSummary, + }]); + const serialized = JSON.stringify(projected); + assert.doesNotMatch(serialized, /PRIVATE NESTED OBSERVABILITY SECRET|privateFuture|privateOwnerMap|privateScore/); + assert.doesNotMatch(serialized, /rssBytes|heapUsedBytes|observedProcess/); + assert.ok(Buffer.byteLength(serialized, "utf8") <= MAX_STANDARD_MACHINE_RESPONSE_BYTES); +}); + +test("compact shared observability rejects invalid numbers, private pass names, and oversized collections", () => { + const report = largeReport(); + const secret = "PRIVATE OVERSIZED OBSERVABILITY SECRET ".repeat(10_000); + const badNumber = validSharedObservability(); + badNumber.samples.at(-1).retention.current.totalAccountedBytes = Number.MAX_SAFE_INTEGER + 1; + const oversized = validSharedObservability(); + oversized.samples = Array.from({ length: 10_000 }, () => ({ private: secret })); + oversized.samplesRecorded = 10_000; + oversized.samplesRetained = 10_000; + report.explore.passes = [ + { pass: `shared:deep-novelty-v1:seed=7:${secret}`, sharedObservability: validSharedObservability() }, + { pass: "shared:deep-novelty-v1:seed=7", sharedObservability: badNumber }, + { pass: "shared:deep-novelty-v1:seed=7", sharedObservability: oversized }, + ]; + const projected = projectMachineReport(report, "summary"); + const serialized = JSON.stringify(projected); + assert.strictEqual(projected.explore.sharedObservability, undefined); + assert.doesNotMatch(serialized, /PRIVATE OVERSIZED OBSERVABILITY SECRET/); + assert.ok(Buffer.byteLength(serialized, "utf8") <= MAX_STANDARD_MACHINE_RESPONSE_BYTES); +}); + +test("compact shared observability bounds the number of otherwise valid pass summaries", () => { + const report = largeReport(); + report.explore.passes = Array.from({ length: 1_000 }, (_, index) => ({ + pass: `shared:deep-novelty-v1:seed=${index}`, + sharedObservability: validSharedObservability(), + })); + const projected = projectMachineReport(report, "summary"); + assert.strictEqual(projected.explore.sharedObservability.length, 8); + assert.ok(Buffer.byteLength(JSON.stringify(projected), "utf8") <= MAX_STANDARD_MACHINE_RESPONSE_BYTES); +}); + test("machine response limits reject unsafe caller values", () => { const report = largeReport(); assert.throws(() => projectMachineReport(report, "standard", 0), /integer from 1 to 100/); diff --git a/test/search-benchmark.test.js b/test/search-benchmark.test.js index 85ce831..d3423d9 100644 --- a/test/search-benchmark.test.js +++ b/test/search-benchmark.test.js @@ -7,6 +7,7 @@ const { spawnSync } = require("node:child_process"); const { compile, scanKnots } = require("../dist/inklecate"); const { + MAX_SHARED_OBSERVABILITY_SAMPLES, SHARED_SEARCH_CHECKPOINT_SCHEMA_VERSION, explore, explorePortfolio, @@ -47,6 +48,7 @@ const FINITE_LOOP = path.join(FIXTURES, "finite-counter-loop.ink"); const GATED_ENDING = path.join(FIXTURES, "gated-ending.ink"); const LOW_DEDUP_WIDE = path.join(FIXTURES, "low-dedup-wide.ink"); const DEEP_BRANCHING = path.join(FIXTURES, "deep-branching.ink"); +const AUTHORED_DOG = path.join(__dirname, "..", "benchmarks", "authored", "dog-ink-adventure", "root.ink"); const PROMOTION_CLI = path.join(__dirname, "..", "dist", "promotion-benchmark-cli.js"); const EMPTY_TRUNCATION = { @@ -507,6 +509,59 @@ test("shared checkpoint envelopes bind cleanly on adversarial growth shapes", as assert.strictEqual(byteMemory.limits.maxPendingBytes, byteLimit); }); +test("shared observability keeps deterministic retention and yield separate from live process memory", async () => { + const compiled = await compile(LOW_DEDUP_WIDE); + const observations = []; + const report = exploreShared(compiled.storyJson, scanKnots(LOW_DEDUP_WIDE), [], { + maxDepth: 150, + maxStates: 300, + seed: 7, + sharedObservabilityIntervalStates: 1, + onSharedObservability: (observation) => observations.push(observation), + }); + const telemetry = report.passes[0].sharedObservability; + assert.strictEqual(telemetry.schemaVersion, 1); + assert.strictEqual(telemetry.sampleIntervalStates, 1); + assert.strictEqual(telemetry.samplesRecorded, report.statesExplored); + assert.ok(telemetry.samplesRetained <= MAX_SHARED_OBSERVABILITY_SAMPLES); + assert.strictEqual(telemetry.samplesCompacted, telemetry.samplesRecorded - telemetry.samplesRetained); + assert.ok(telemetry.samplesCompacted > 0); + assert.strictEqual(telemetry.samples[0].state, 1); + assert.strictEqual(telemetry.samples.at(-1).state, report.statesExplored); + assert.strictEqual(telemetry.samples.at(-1).boundary, "interval_and_termination"); + assert.deepStrictEqual( + telemetry.samples.at(-1).yield.cumulative, + telemetry.yieldSummary.cumulative + ); + assert.strictEqual(telemetry.yieldSummary.firstUsefulAtState, 0); + assert.ok(telemetry.yieldSummary.throughFirstUseful.authoredCoverage.knotsVisited > 0); + assert.ok(telemetry.yieldSummary.afterFirstUseful.rawTerritory.transitions > 0); + assert.ok(telemetry.yieldSummary.cumulative.semanticTransitions < report.passes[0].variableTransitionsObserved); + assert.strictEqual("score" in telemetry.yieldSummary, false); + + for (let index = 1; index < telemetry.samples.length; index++) { + assert.ok(telemetry.samples[index].state > telemetry.samples[index - 1].state); + assert.strictEqual( + telemetry.samples[index].yield.fromStateExclusive, + telemetry.samples[index - 1].state + ); + } + const latest = observations.at(-1); + assert.strictEqual(latest.schemaVersion, 1); + assert.strictEqual(latest.runWideState, latest.sample.state); + assert.strictEqual(latest.process.schemaVersion, 1); + assert.strictEqual(latest.process.scope, "process"); + assert.ok(latest.process.heapUsedBytes > 0); + assert.ok(latest.process.rssBytes > 0); + assert.strictEqual( + latest.process.comparedLogicalAccountedBytes, + latest.sample.retention.current.totalAccountedBytes + ); + assert.ok(Number.isInteger(latest.process.unattributedBytes)); + assert.doesNotMatch(JSON.stringify(observations), /path_code|wide tree leaf|"Left"|"Center"|"Right"/i); + assert.doesNotMatch(JSON.stringify(report), /heapUsedBytes|heapTotalBytes|rssBytes|unattributedBytes/); +}); + test("base shared search resumes from JSON with the exact uninterrupted result", async () => { const compiled = await compile(LOW_DEDUP_WIDE); const knots = scanKnots(LOW_DEDUP_WIDE); @@ -516,6 +571,7 @@ test("base shared search resumes from JSON with the exact uninterrupted result", seed: 7, preserveTurnState: false, preserveRandomState: false, + sharedObservabilityIntervalStates: 25, }; const uninterrupted = exploreSharedResumable(compiled.storyJson, knots, [], options); assert.deepStrictEqual(uninterrupted.result, exploreShared(compiled.storyJson, knots, [], options)); @@ -529,6 +585,8 @@ test("base shared search resumes from JSON with the exact uninterrupted result", assert.ok(first.checkpoint.state.current.cursor > 0, "fixture should pause partway through a choice list"); assert.strictEqual(first.checkpoint.state.truncatedBy.maxStates, false); assert.strictEqual(first.result.truncatedBy.maxStates, true); + assert.deepStrictEqual(first.checkpoint.state.sharedObservability.samples.map((sample) => sample.state), [25, 50]); + assert.doesNotMatch(JSON.stringify(first.checkpoint), /heapUsedBytes|heapTotalBytes|rssBytes|unattributedBytes/); const serialized = JSON.parse(JSON.stringify(first.checkpoint)); const resumed = exploreSharedResumable(compiled.storyJson, knots, [], options, serialized); @@ -536,6 +594,77 @@ test("base shared search resumes from JSON with the exact uninterrupted result", assert.strictEqual(resumed.checkpoint.state.totalGranted, 500); }); +test("shared checkpoints preserve useful milestones before the first retained sample", async () => { + const compiled = await compile(AUTHORED_DOG); + const knots = scanKnots(AUTHORED_DOG); + const options = { + maxDepth: 30, + maxStates: 100, + seed: 7, + storySeed: 1, + sharedObservabilityIntervalStates: 10, + }; + const first = exploreSharedResumable(compiled.storyJson, knots, [], options); + const observability = first.checkpoint.state.sharedObservability; + assert.strictEqual(first.checkpoint.state.statesExplored, 100); + assert.strictEqual(observability.baseState, 0); + assert.strictEqual(observability.sampleIntervalStates, 10); + assert.strictEqual(observability.firstUsefulAtState, 1); + assert.strictEqual(observability.samples[0].state, 10); + assert.ok(observability.samples[0].yield.cumulative.authoredCoverage.knotsVisited > 0); + assert.ok(observability.firstUsefulAtState < observability.samples[0].state); + assert.ok( + observability.throughFirstUseful.authoredCoverage.knotsVisited + <= observability.samples[0].yield.cumulative.authoredCoverage.knotsVisited + ); + + const resumed = exploreSharedResumable(compiled.storyJson, knots, [], { + ...options, + maxStates: 200, + }, JSON.parse(JSON.stringify(first.checkpoint))); + assert.ok(resumed.result.statesExplored >= first.result.statesExplored); + + const atFirstSample = structuredClone(first.checkpoint); + atFirstSample.state.sharedObservability.firstUsefulAtState = 10; + atFirstSample.state.sharedObservability.throughFirstUseful.rawTerritory.transitions = 10; + assert.doesNotThrow(() => exploreSharedResumable(compiled.storyJson, knots, [], { + ...options, + maxStates: 200, + }, atFirstSample)); + + const exceedsFirstSample = structuredClone(first.checkpoint); + const firstSampleUniqueStates = exceedsFirstSample.state.sharedObservability + .samples[0].yield.cumulative.rawTerritory.uniqueStates; + assert.ok( + firstSampleUniqueStates + < exceedsFirstSample.state.sharedObservability.previousYield.rawTerritory.uniqueStates + ); + exceedsFirstSample.state.sharedObservability.firstUsefulAtState = 10; + exceedsFirstSample.state.sharedObservability.throughFirstUseful.rawTerritory.transitions = 10; + exceedsFirstSample.state.sharedObservability.throughFirstUseful.rawTerritory.uniqueStates + = firstSampleUniqueStates + 1; + assert.throws( + () => exploreSharedResumable(compiled.storyJson, knots, [], { + ...options, + maxStates: 200, + }, exceedsFirstSample), + /useful milestone exceeds retained cumulative yield/ + ); + + for (const lateMilestone of [11, 50]) { + const tampered = structuredClone(first.checkpoint); + tampered.state.sharedObservability.firstUsefulAtState = lateMilestone; + tampered.state.sharedObservability.throughFirstUseful.rawTerritory.transitions = lateMilestone; + assert.throws( + () => exploreSharedResumable(compiled.storyJson, knots, [], { + ...options, + maxStates: 200, + }, tampered), + /useful milestone is later than retained useful yield/ + ); + } +}); + test("shared checkpoints fail closed on incompatible source, options, budget, and state", async () => { const compiled = await compile(LOW_DEDUP_WIDE); const knots = scanKnots(LOW_DEDUP_WIDE); @@ -552,6 +681,14 @@ test("shared checkpoints fail closed on incompatible source, options, budget, an () => exploreSharedResumable(compiled.storyJson, knots, [], { ...options, maxStates: 100, seed: 8 }, checkpoint), /source, strategy, limits, seeds/ ); + assert.throws( + () => exploreSharedResumable(compiled.storyJson, knots, [], { + ...options, + maxStates: 100, + sharedObservabilityIntervalStates: 1, + }, checkpoint), + /observability interval changed/ + ); for (const [changedOptions, changedExternals] of [ [{ maxDepth: 149 }, []], [{ storySeed: 2 }, []], @@ -597,6 +734,99 @@ test("shared checkpoints fail closed on incompatible source, options, budget, an () => exploreSharedResumable(compiled.storyJson, knots, [], { ...options, maxStates: 100 }, badParent), /invalid parent reference/ ); + const badObservability = clone(); + badObservability.state.sharedObservability.baseYield.rawTerritory.transitions = -1; + assert.throws( + () => exploreSharedResumable(compiled.storyJson, knots, [], { ...options, maxStates: 100 }, badObservability), + /shared observability ledger is malformed/ + ); + const observedOptions = { ...options, sharedObservabilityIntervalStates: 10 }; + const observedCheckpoint = exploreSharedResumable( + compiled.storyJson, + knots, + [], + observedOptions + ).checkpoint; + assert.ok(observedCheckpoint.state.sharedObservability.samples.length >= 2); + assert.strictEqual(observedCheckpoint.state.runtimeErrors.length, 0); + assert.ok(observedCheckpoint.state.sharedObservability.firstUsefulAtState !== null); + const rejectObservedTamper = (mutate) => { + const tampered = structuredClone(observedCheckpoint); + mutate(tampered.state.sharedObservability, tampered.state); + assert.throws( + () => exploreSharedResumable(compiled.storyJson, knots, [], { + ...observedOptions, + maxStates: 100, + }, tampered), + /shared observability/ + ); + }; + rejectObservedTamper((observability) => { + observability.samples[0].yield.delta.rawTerritory.transitions++; + }); + rejectObservedTamper((observability) => { + observability.samples[0].retention.current.totalAccountedBytes++; + }); + rejectObservedTamper((observability) => { + const sample = observability.samples[0]; + sample.retention.current.pendingStates = sample.retention.peak.pendingStates + 1; + }); + rejectObservedTamper((observability) => { + const [firstSample, secondSample] = observability.samples; + const high = Math.max( + firstSample.retention.current.findingBytes, + secondSample.retention.current.findingBytes, + firstSample.retention.peak.findingBytes, + secondSample.retention.peak.findingBytes + ) + 2; + firstSample.retention.peak.findingBytes = high; + secondSample.retention.peak.findingBytes = high - 1; + }); + rejectObservedTamper((observability, state) => { + const [firstSample, secondSample] = observability.samples; + assert.ok(state.releasedNodes > 0); + firstSample.retention.releasedNodes = state.releasedNodes; + secondSample.retention.releasedNodes = state.releasedNodes - 1; + }); + rejectObservedTamper((observability, state) => { + const [firstSample, secondSample] = observability.samples; + state.frontierCompactions = 1; + firstSample.retention.frontierCompactions = 1; + secondSample.retention.frontierCompactions = 0; + }); + rejectObservedTamper((observability) => { + observability.nextSampleState++; + }); + rejectObservedTamper((observability) => { + observability.firstCriticalAtState = 1; + }); + rejectObservedTamper((observability) => { + observability.samples[0].boundary = "termination"; + }); + rejectObservedTamper((observability) => { + observability.samplesRecorded = 1_000_000; + }); + rejectObservedTamper((observability) => { + observability.firstUsefulAtState = null; + }); + rejectObservedTamper((observability, state) => { + const latestPeak = observability.samples.at(-1).retention.peak; + assert.ok(latestPeak.pendingStates > 0); + state.peakRetainedMemory.pendingStates = latestPeak.pendingStates - 1; + }); + const legacyCheckpoint = clone(); + delete legacyCheckpoint.state.sharedObservability; + delete legacyCheckpoint.state.meaningfulVariableTransitions; + delete legacyCheckpoint.state.discoveryCurve.countedVisibleOutcomes; + const legacyResume = exploreSharedResumable( + compiled.storyJson, + knots, + [], + { ...options, maxStates: 100 }, + legacyCheckpoint + ); + assert.strictEqual(legacyResume.result.passes[0].sharedObservability.historyComplete, false); + assert.strictEqual(legacyResume.result.statesExplored, 100); assert.throws( () => exploreSharedResumable(compiled.storyJson, knots, [], { ...options, assertions: [{}] }), /only the base shared strategy/